diff --git a/404.html b/404.html new file mode 100644 index 0000000000..13f44ec1f4 --- /dev/null +++ b/404.html @@ -0,0 +1,9 @@ + + + + +Swift Evolution diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7817678224..dce8b59b25 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,16 +1,18 @@ -# Contributing +# Welcome to the Swift community! -## Participating in the Swift Evolution Process +Contributions to Swift are welcomed and encouraged! Please see the [Contributing to Swift guide](https://www.swift.org/contributing/) and check out the [structure of the community](https://www.swift.org/community/#community-structure). -See [Participating in the Swift Evolution Process](https://swift.org/contributing/#participating-in-the-swift-evolution-process) on Swift.org and [process.md](process.md). +To be a truly great community, Swift needs to welcome developers from all walks of life, with different backgrounds, and with a wide range of experience. A diverse and friendly community will have more great ideas, more unique perspectives, and produce more great code. We will work diligently to make the Swift community welcoming to everyone. -## Contributing to the status page -The [status page](https://apple.github.io/swift-evolution/) shows community activity related to the Swift Evolution Process. Changes to it should focus on supporting the efforts of the community in that process. +To give clarity of what is expected of our members, Swift has adopted the code of conduct defined by the Contributor Covenant. This document is used across many open source communities, and we think it articulates our values well. For more, see the [Code of Conduct](https://www.swift.org/code-of-conduct/). -Before making a pull request to change the status page, consider the following steps: - -- **Socialize the idea**: Is there a broader desire in the Swift community for the feature? Ask. Ensure that the feature can be implemented using data from the Swift project itself, rather than depending externally derived data. - -- **Develop a working copy**: Ideas are easiest to understand with a complete implementation. A quick prototype may be good enough for early stage feedback. Once the idea is understood, clean up the code, test it, and format it according to [JavaScript Standard](http://standardjs.com) style. - -- **Request a review**: Initiate a pull request to the [swift-evolution repository](https://github.com/apple/swift-evolution) when a proposed change has a complete implementation. Include a link to a working copy, then assign the review to @krilnon. When everything looks good, the pull request will be merged. \ No newline at end of file +## Contributing to Swift Evolution + +This repository is not your standard codebase. It houses Swift evolution proposals and related process documents, mostly composed of markdown and text files. Pull requests that make minor editorial and administrative changes are always welcome, including fixing typos and grammar mistakes, repairing links, and maintaining document and repository metadata. Other pull requests must follow the [Swift evolution process](process.md): + +- New proposals and substantive changes to existing proposals should be [pitched on the evolution forums](https://forums.swift.org/c/evolution/pitches/5) before a PR is opened here. +- Substantive changes to existing proposals require the permission of the proposal authors. +- Substantive changes to existing proposals require the approval of the appropriate evolution workgroup if the proposal is in an Active Review, Accepted, or Rejected state. +- New vision documents and substantive changes to existing vision documents require the approval of the appropriate evolution workgroup. +- Substantive changes to the evolution process require the approval of the Core Team. +- Conversations about the substance of a proposal should be held in an appropriate forums thread rather than in PR comments. This centralizes the discussion and allows more of the community to participate. diff --git a/README.md b/README.md index 80a3ca1bd4..47a172b81a 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,33 @@ -# Swift Programming Language Evolution +# Swift Evolution -**Before you initiate a pull request**, please read the [process](process.md) document. -Ideas should be thoroughly discussed on the [swift-evolution forums](https://swift.org/community/#swift-evolution) first. + -This repository tracks the ongoing evolution of Swift. It contains: - -* The [status page](https://apple.github.io/swift-evolution/), tracking proposals to change Swift. -* The [process](process.md) document that governs the evolution of Swift. -* [Commonly Rejected Changes](commonly_proposed.md), proposals that have been denied in the past. +This repository tracks the ongoing evolution of the Swift programming language, standard library, and package manager. ## Goals and Release Notes -* [On the road to Swift 6](https://forums.swift.org/t/on-the-road-to-swift-6/32862) +* [Swift Language focus areas heading into 2025](https://forums.swift.org/t/swift-language-focus-areas-heading-into-2025/76611) * [CHANGELOG](https://github.com/apple/swift/blob/main/CHANGELOG.md) -| Version | Announced | Released | -| :-------- | :----------------------------------------------------------------------- | :------------------------------------------------------- | -| Swift 5.6 | [2021-11-10](https://forums.swift.org/t/swift-5-6-release-process/53412) | -| Swift 5.5 | [2021-03-12](https://forums.swift.org/t/swift-5-5-release-process/45644) | [2021-09-20](https://swift.org/blog/swift-5-5-released/) | -| Swift 5.4 | [2020-11-11](https://forums.swift.org/t/swift-5-4-release-process/41936) | [2021-04-26](https://swift.org/blog/swift-5-4-released/) | -| Swift 5.3 | [2020-03-25](https://swift.org/blog/5-3-release-process/) | [2020-09-16](https://swift.org/blog/swift-5-3-released/) | -| Swift 5.2 | [2019-09-24](https://swift.org/blog/5-2-release-process/) | [2020-03-24](https://swift.org/blog/swift-5-2-released/) | -| Swift 5.1 | [2019-02-18](https://swift.org/blog/5-1-release-process/) | [2019-09-20](https://swift.org/blog/swift-5-1-released/) | -| Swift 5.0 | [2018-09-25](https://swift.org/blog/5-0-release-process/) | [2019-03-25](https://swift.org/blog/swift-5-released/) | -| Swift 4.2 | [2018-02-28](https://swift.org/blog/4-2-release-process/) | [2018-09-17](https://swift.org/blog/swift-4-2-released/) | -| Swift 4.1 | [2017-10-17](https://swift.org/blog/swift-4-1-release-process/) | [2018-03-29](https://swift.org/blog/swift-4-1-released/) | -| Swift 4.0 | [2017-02-16](https://swift.org/blog/swift-4-0-release-process/) | [2017-09-19](https://swift.org/blog/swift-4-0-released/) | -| Swift 3.1 | [2016-12-09](https://swift.org/blog/swift-3-1-release-process/) | [2017-03-27](https://swift.org/blog/swift-3-1-released/) | -| Swift 3.0 | [2016-05-06](https://swift.org/blog/swift-3-0-release-process/) | [2016-09-13](https://swift.org/blog/swift-3-0-released/) | -| Swift 2.2 | [2016-01-05](https://swift.org/blog/swift-2-2-release-process/) | [2016-03-21](https://swift.org/blog/swift-2-2-released/) | +| Version | Announced | Released | +| :-------- | :----------------------------------------------------------------------- | :----------------------------------------------------------- | +| Swift 6.2 | [2025-03-08](https://forums.swift.org/t/swift-6-2-release-process/78371) | [2025-09-15](https://www.swift.org/blog/swift-6.2-released/) | +| Swift 6.1 | [2024-10-17](https://forums.swift.org/t/swift-6-1-release-process/75442) | [2025-03-31](https://www.swift.org/blog/swift-6.1-released/) | +| Swift 6.0 | [2024-02-22](https://forums.swift.org/t/swift-6-0-release-process/70220) | [2024-09-17](https://www.swift.org/blog/announcing-swift-6/) | +| Swift 5.10 | [2023-08-23](https://forums.swift.org/t/swift-5-10-release-process/66911) | [2024-03-05](https://www.swift.org/blog/swift-5.10-released/) | +| Swift 5.9 | [2023-03-06](https://forums.swift.org/t/swift-5-9-release-process/63557) | [2023-09-18](https://www.swift.org/blog/swift-5.9-released/) | +| Swift 5.8 | [2022-11-19](https://forums.swift.org/t/swift-5-8-release-process/61540) | [2023-03-30](https://www.swift.org/blog/swift-5.8-released/) | +| Swift 5.7 | [2022-03-29](https://forums.swift.org/t/swift-5-7-release-process/56316) | [2022-09-12](https://www.swift.org/blog/swift-5.7-released/) | +| Swift 5.6 | [2021-11-10](https://forums.swift.org/t/swift-5-6-release-process/53412) | [2022-03-14](https://www.swift.org/blog/swift-5.6-released/) | +| Swift 5.5 | [2021-03-12](https://forums.swift.org/t/swift-5-5-release-process/45644) | [2021-09-20](https://www.swift.org/blog/swift-5.5-released/) | +| Swift 5.4 | [2020-11-11](https://forums.swift.org/t/swift-5-4-release-process/41936) | [2021-04-26](https://www.swift.org/blog/swift-5.4-released/) | +| Swift 5.3 | [2020-03-25](https://www.swift.org/blog/5.3-release-process/) | [2020-09-16](https://www.swift.org/blog/swift-5.3-released/) | +| Swift 5.2 | [2019-09-24](https://www.swift.org/blog/5.2-release-process/) | [2020-03-24](https://www.swift.org/blog/swift-5.2-released/) | +| Swift 5.1 | [2019-02-18](https://www.swift.org/blog/5.1-release-process/) | [2019-09-20](https://www.swift.org/blog/swift-5.1-released/) | +| Swift 5.0 | [2018-09-25](https://www.swift.org/blog/5.0-release-process/) | [2019-03-25](https://www.swift.org/blog/swift-5-released/) | +| Swift 4.2 | [2018-02-28](https://www.swift.org/blog/4.2-release-process/) | [2018-09-17](https://www.swift.org/blog/swift-4.2-released/) | +| Swift 4.1 | [2017-10-17](https://www.swift.org/blog/swift-4.1-release-process/) | [2018-03-29](https://www.swift.org/blog/swift-4.1-released/) | +| Swift 4.0 | [2017-02-16](https://www.swift.org/blog/swift-4.0-release-process/) | [2017-09-19](https://www.swift.org/blog/swift-4.0-released/) | +| Swift 3.1 | [2016-12-09](https://www.swift.org/blog/swift-3.1-release-process/) | [2017-03-27](https://www.swift.org/blog/swift-3.1-released/) | +| Swift 3.0 | [2016-05-06](https://www.swift.org/blog/swift-3.0-release-process/) | [2016-09-13](https://www.swift.org/blog/swift-3.0-released/) | +| Swift 2.2 | [2016-01-05](https://www.swift.org/blog/swift-2.2-release-process/) | [2016-03-21](https://www.swift.org/blog/swift-2.2-released/) | diff --git a/commonly_proposed.md b/commonly_proposed.md index b81f0d0ac9..d2d2bdc8ca 100644 --- a/commonly_proposed.md +++ b/commonly_proposed.md @@ -1,15 +1,15 @@ # Commonly Rejected Changes - + This is a list of changes to the Swift language that are frequently proposed but that are unlikely to be accepted. If you're interested in pursuing something in this space, please familiarize yourself with the discussions that we have already had. In order to bring one of these topics up, you'll be expected to add new information to the discussion, not just to say, "I really want this" or "this exists in some other language and I liked it there". -Additionally, proposals for out-of-scope changes will not be scheduled for review. The [readme file](README.md) identifies a list of priorities for the next major release of Swift, and the [proposal review status page](https://apple.github.io/swift-evolution/) includes a list of changes that have been deferred for future discussion because they were deemed to be out of scope at the time of review (in addition to a list of changes proposed and rejected after a formal review). +Additionally, proposals for out-of-scope changes will not be scheduled for review. The [readme file](README.md) identifies a list of priorities for the next major release of Swift, and the [dashboard](https://www.swift.org/swift-evolution/) includes a list of changes that have been rejected after a formal review. -Several of the discussions below refer to "C family" languages. This is intended to mean the extended family of languages that resemble C at a syntactic level, such as C++, C#, Objective-C, Java, and Javascript. Swift embraces its C heritage. Where it deviates from other languages in the family, it does so because the feature was thought actively harmful (such as the pre/post-increment `++`) or to reduce needless clutter (such as `;` or parentheses in `if` statements). +Several of the discussions below refer to "C family" languages. This is intended to mean the extended family of languages that resemble C at a syntactic level, such as C++, C#, Objective-C, Java, and JavaScript. Swift embraces its C heritage. Where it deviates from other languages in the family, it does so because the feature was thought actively harmful (such as the pre/post-increment `++`) or to reduce needless clutter (such as `;` or parentheses in `if` statements). ## Basic Syntax and Operators * [Replace `{}` brace syntax with Python-style indentation](https://forums.swift.org/t/brace-syntax/678/3): Surely a polarizing issue, but Swift will not change to use indentation for scoping instead of curly braces. - + * [Remove `;` semicolons](https://forums.swift.org/t/proposal-to-remove-semicolons/523/3): Semicolons within a line are an intentional expressivity feature. Semicolons at the end of the line should be handled by a linter, not by the compiler. * [Replace logical operators (`&&`, `||`, `!`, etc.) with words like "and", "or", "not"](https://forums.swift.org/t/change-the-name-of-the-boolean-operators/30/2), and [allow non-punctuation operators](https://forums.swift.org/t/allowing-characters-for-use-as-custom-operators/952) and infix functions: The operator and identifier grammars are intentionally partitioned in Swift, which is a key part of how user-defined overloaded operators are supported. Requiring the compiler to see the "operator" declaration to know how to parse a file would break the ability to be able to parse a Swift file without parsing all of its imports. This has a major negative effect on tooling support. While not needing infix support, `not` would need operator or keyword status to omit the parentheses as `!` can, and `not somePredicate()` visually binds too loosely compared to `!somePredicate()`. @@ -22,30 +22,16 @@ Several of the discussions below refer to "C family" languages. This is intended ## Control Flow, Closures, Optional Binding, and Error Handling - * [`if/else` and `switch` as expressions](https://forums.swift.org/t/control-flow-expressions/90/5): These are conceptually interesting things to support, but many of the problems solved by making these into expressions are already solved in Swift in other ways. Making them expressions introduces significant tradeoffs, and on balance, we haven't found a design that is clearly better than what we have so far. - * [Replace `continue` keyword with synonyms from other scripting languages (e.g. next, skip, advance, etc)](https://forums.swift.org/t/replace-continue-keyword/764/2): Swift is designed to feel like a member of the C family of languages. Switching keywords away from C precedent without strong motivation is a non-goal. - * [Remove support for `default:` in `switch` and just use `case _:`](https://forums.swift.org/t/remove-default-case-in-switch-case/360/4): `default` is widely used, `case _` is too magical, and `default` is widely precedented in many C family languages. - * [Rename `guard` to `unless`](https://forums.swift.org/t/rename-guard-to-unless/934/7): It is a common request that `guard` be renamed `unless`. People requesting this change argue that `guard` is simply a logically inverted `if` statement, and therefore `unless` is a more obvious keyword. However, such requests stem from a fundamental misunderstanding of the functionality provided by `guard`. Unlike `if`, `guard` *enforces* that the code within its curly braces provides an early exit from the codepath. In other words, a `guard` block **must** `return`, `throw`, `break`, `continue` or call a function that does not return, such as `fatalError()`. This differs from `if` quite significantly, and therefore the parallels assumed between `guard` and `if` are not valid. - * [Infer `return` for omitted `guard` body](https://forums.swift.org/t/inferred-return-for-guard-statement/12099/11): It has been proposed many times to allow omission of the `guard` body for the sake of brevity. However, a core principle of Swift is to make control flow explicit and visible. For example, the `try` keyword exists solely to indicate to the human reader where thrown errors can happen. Implicit returns would violate this principle, favoring terseness over clarity in a way that isn't typical of Swift. Furthermore, there are many ways of exiting the scope other than `return` (loops may want `break` or `continue`), and not every function has an obvious default value to return. - * [Change closure literal syntax](https://forums.swift.org/t/streamlining-closures/487/3): Closure syntax in Swift has been carefully debated internally, and aspects of the design have strong motivations. It is unlikely that we'll find something better, and any proposals to change it should have a very detailed understanding of the Swift grammar. - * [Use pattern-matching in `if let` instead of optional-unwrapping](https://forums.swift.org/t/obsoleting-if-let/1301/4): We actually tried this and got a lot of negative feedback, for several reasons: (1) Most developers don't think about things in "pattern matching" terms, they think about "destructuring". (2) The vastly most common use case for `if let` is actually for optional matching, and this change made the common case more awkward. (3) This change increases the learning curve of Swift, changing pattern matching from being a concept that can be learned late to something that must be confronted early. (4) The current design of `if case` unifies "pattern matching" around the `case` keyword. (5) If a developer unfamiliar with `if case` runs into one in some code, they can successfully search for it in a search engine or Stack Overflow. - - * [Syntactic sugar for `if let` self-assignment](https://forums.swift.org/t/new-feature-request-syntactic-sugar-for-if-let-scoped-self-assignment/3887/4): An alternative syntax (such as `if let foo? { ... }` or `if let foo=? { ... }`) to serve as a shorthand for `if let foo = foo { ... }` is often proposed and rejected because it is favoring terseness over clarity by introducing new magic syntactic sugar. - * [Remove or deprecate the force-unwrap operator `!`](https://forums.swift.org/t/moving-toward-deprecating-force-unwrap-from-swift/43455/82): Force-unwrap and force-try are legitimately useful parts of the language, and not just for source stability reasons. Therefore, proposals to deprecate or remove the force-unwrap operator (or `try!`), even in a mode enabled via compiler flag, will not be considered by the core team. Whether the Swift compiler should gain a more general "linting" capability to guide coding style remains a possible topic of discussion. - - * [Replace the `do`/`try`/`repeat` keywords with C++-style syntax](https://forums.swift.org/t/use-standard-syntax-instead-of-do-and-repeat/791/2): Swift's error handling approach is carefully designed to make it obvious to maintainers of code when a call can "throw" an error. It is intentionally designed to be syntactically similar in some ways, but different in other key ways, to exception handling in other languages. Its design is a careful balance that favors maintainers of code that uses errors, to make sure someone reading the code understands what can throw. Before proposing a change to this system, please read the [Error Handling Rationale and Proposal](https://github.com/apple/swift/blob/master/docs/ErrorHandlingRationale.rst) in full to understand why the current design is the way it is, and be ready to explain why your changes would be worth unbalancing this design. + * [Replace the `do`/`try`/`repeat` keywords with C++-style syntax](https://forums.swift.org/t/use-standard-syntax-instead-of-do-and-repeat/791/2): Swift's error handling approach is carefully designed to make it obvious to maintainers of code when a call can "throw" an error. It is intentionally designed to be syntactically similar in some ways, but different in other key ways, to exception handling in other languages. Its design is a careful balance that favors maintainers of code that uses errors, to make sure someone reading the code understands what can throw. Before proposing a change to this system, please read the [Error Handling Rationale and Proposal](https://github.com/swiftlang/swift/blob/main/docs/ErrorHandlingRationale.md) in full to understand why the current design is the way it is, and be ready to explain why your changes would be worth unbalancing this design. ## Miscellaneous * [Use garbage collection (GC) instead of automatic reference counting (ARC)](https://forums.swift.org/t/what-about-garbage-collection/1360): Mark-and-sweep garbage collection is a well-known technique used in many popular and widely used languages (e.g., Java and JavaScript) and it has the advantage of automatically collecting reference cycles that ARC requires the programmer to reason about. That said, garbage collection has a [large number of disadvantages](https://forums.swift.org/t/what-about-garbage-collection/1360/6) and using it would prevent Swift from successfully targeting a number of systems programming domains. For example, real-time systems (video or audio processing), deeply embedded controllers, and most kernels find GC to be generally unsuitable. Further, GC is only efficient when given 3–4× more memory to work with than the process is using at any time, and this tradeoff is not acceptable for Swift. - * [Disjunctions (logical ORs) in type constraints](https://forums.swift.org/t/returned-for-revision-se-0095-replace-protocol-p1-p2-syntax-with-any-p1-p2/2855): These include anonymous union-like types (e.g. `(Int | String)` for a type that can be inhabited by either an integer or a string). "[This type of constraint is] something that the type system cannot and should not support." - - * [Rewrite the Swift compiler in Swift](https://github.com/apple/swift/blob/2c7b0b22831159396fe0e98e5944e64a483c356e/www/FAQ.rst): This would be a lot of fun someday, but (unless you include rewriting all of LLVM) requires the ability to import C++ APIs into Swift. Additionally, there are lots of higher priority ways to make Swift better. diff --git a/index.css b/index.css deleted file mode 100644 index 164d1ec911..0000000000 --- a/index.css +++ /dev/null @@ -1,652 +0,0 @@ -/*===--- index.css - Swift Evolution --------------------------------------===// -// -// This source file is part of the Swift.org open source project -// -// Copyright (c) 2014 - 2020 Apple Inc. and the Swift project authors -// Licensed under Apache License v2.0 with Runtime Library Exception -// -// See http://swift.org/LICENSE.txt for license information -// See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors -// -//===----------------------------------------------------------------------===// -*/ - -:root { - color-scheme: light dark; - - --background: white; - --text: #333; - --link: #666; - --proposal-link: #888; - --footer-background: #333; - --footer-text: #ccc; - --header-background: rgba(248, 248, 248, 0.7); - --nav-background: rgba(248, 248, 248, 0.7); - --border: rgb(230, 230, 230); - --filter-link: rgb(0, 136, 204); -} - -@media (prefers-color-scheme: dark) { - :root { - --background: black; - --text: white; - --link: rgb(9, 132, 255); - --proposal-link: rgba(235, 235, 245, 0.6); - --footer-background: rgb(28, 28, 30); - --footer-text: rgb(255, 255, 255); - --header-background: rgba(44, 44, 46, 0.7); - --nav-background: rgba(28, 28, 30, 0.7); - --border: rgba(84, 84, 88, 0.6); - } -} - -*:focus { - outline: none; -} - -body { - font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", Helvetica, Arial, Verdana, sans-serif; - font-size: 18px; - -webkit-text-size-adjust: none; - line-height: 1.5; - color: var(--text); - background-color: var(--background); - margin: 0; -} - -h1 { - margin-left: 10px; - font-size: 54px; -} - -h4 { - margin-bottom: 0em; - font-size: 20px; - line-height: 1.2em; -} - -a:link { - color: var(--link); - text-decoration: none; -} - -a:visited { - color: var(--link); -} - -a:active { - color: var(--link); -} - -a:hover { - color: var(--link); - text-decoration: underline; -} - -ul { - list-style: none; - margin: 0; - padding: 0; -} - -main { - display: flex; - width: 95%; - flex: 1; - justify-content: center; -} - -header { - display: flex; - flex-wrap: wrap; - padding-top: 20px; - min-width: 100%; - height: 140px; - flex-direction: row; - text-align: left; - align-items: center; - justify-content: center; - background: var(--header-background); - -webkit-backdrop-filter: blur(10px); - backdrop-filter: blur(10px); - border-bottom: 1px solid var(--border); - padding-top: 0px; -} - -.header-contents { - flex-basis: 960px; -} - -#logo { - display: inline-block; - width: 60px; - height: 60px; - margin-bottom: -2px; - background-image: url("https://data.swift.org/swift-evolution/swift.svg"); -} - -#title { - display: inline-block; - line-height: 48px; - font-weight: 200; -} - -#title a:link { color: var(--text); } -#title a:active { color: var(--text); } -#title a:visited { color: var(--text); } -#title a:hover { text-decoration: none; } - -.app { - display: flex; - flex-direction: column; - align-items: center; - min-height: 100vh; -} - -nav { - top: 0px; - background: var(--nav-background); - -webkit-backdrop-filter: blur(10px); - backdrop-filter: blur(10px); - - border-bottom: 1px solid var(--border); - min-width: 100%; -} - -@supports ((position: sticky) or (position: -webkit-sticky)) { - nav { - position: -webkit-sticky; - position: sticky; - } -} - -.nav-contents { - width: 960px; - margin: 0 auto; - margin-top: 8px; - display: flex; - flex-direction: row; - flex-wrap: wrap; - justify-content: space-between; - min-height: 40px; -} - -footer { - background: var(--footer-background); - color: var(--footer-text); - font-size: 12px; - min-width: 100%; - justify-content: center; - display: flex; - flex-direction: row; -} - -footer a:link, footer a:active, footer a:visited, footer a:hover { - color: var(--footer-text); -} - -.footer-contents { - flex-basis: 960px; -} - -footer p { - margin: 1em 0em; -} - -article { - flex-basis: 960px; -} - - -section { - margin-bottom: 1.5em; -} - -section ul, section li { - display: inline; -} - -.section-title { - margin-top: 14px; -} - -.hidden { - display: none !important; -} - -#proposals-count { - line-height: 40px; - font-size: 16px; - color: var(--text); - padding: 4px 0px; -} - -/* - Individual proposals -*/ - -.proposal-id { - color: var(--proposal-link); - font-weight: 300; - font-size: 20px; - margin-right: 0.7em; - vertical-align: top; /* align with the proposal title */ - line-height: 1.2em; - display: inline-block; -} - -.proposal { - font-size: 14px; - display: flex; - flex-direction: row; - margin-bottom: 40px; -} - -.proposal-list, .proposal { - max-width: none; -} - -.proposal-header { - margin-bottom: 6px; -} - -.proposal-details { - -moz-column-count: 2; - column-count: 2; - -moz-column-width: 373px; - column-width: 373px; -} - -.proposal-detail { - break-inside: avoid; - display: flex; - max-width: 373px; -} - -.proposal-detail-label { - color: var(--proposal-link); - font-weight: 200; - display: inline-block; - padding-right: 6px; - white-space: nowrap; -} - -.proposal-detail-value { - display: inline-block; - color: var(--text); -} - -.proposal-title a:link, -.proposal-title a:visited, -.proposal-title a:active, -.proposal-detail-value a:link, -.proposal-detail-value a:visited, -.proposal-detail-value a:active { - color: var(--text); -} - -.authors .proposal-detail-value { - width: 280px; -} - -.authors a { - white-space: nowrap; -} - -.review-manager .proposal-detail-value { - width: 223px; -} - -.bug-list { - width: 310px; -} - -.bug-list a, .implementation-list a { - word-wrap: break-word; -} - -.proposal-title { - font-weight: 400; - margin-top: 0; - display: inline-block; - max-width: 650px; -} - -.status-pill-container { - margin-top: -2px; -} - -.status-pill { - display: inline-block; - border: 1px solid black; - border-radius: 4px; - padding: 0px; - white-space: nowrap; - font-size: 16px; - font-weight: 200; - line-height: 26px; - text-align: center; - width: 152px; - min-width: 152px; - max-width: 152px; - margin-right: 20px; -} - -/* - Filtering -*/ - -#search-filter { - background-color: var(--background); - border: 1px solid var(--border); - color: var(--text); -} - -.filter-by-status { - padding-bottom: 0.75em; - padding-top: 7px; - max-width: 800px; -} - -.filter-by-status li { - margin-right: 6px; - margin-bottom: 6px; -} - -.filter-by-status label { - border: 1px solid var(--proposal-link); - border-radius: 4px; - padding: 3px 16px; - font-size: 16px; - margin: 0 0px; - height: 28px; - cursor: pointer; -} - -.filter-by-status, .filter-by-status li { - display: inline-block; - -webkit-user-select: none; -} - -.filter-by-status input[type=checkbox] { - display: none; -} - -.filter-by-status input[type=checkbox] + label { - color: var(--proposal-link); -} - -.filter-by-status input[type=checkbox]:checked + label { - background: var(--proposal-link); - color: white; -} - -.filter-toggle a { - color: var(--filter-link); - font-weight: 400; - cursor: pointer; -} - -.filter-toggle-contents { - display: flex; - flex-direction: row; - justify-content: center; - align-items: center; - line-height: 20px; - height: 30px; -} - -.filter-button { - cursor: pointer; - line-height: 40px; - order: 3; - - -webkit-user-select: none; - -moz-user-select: none; - user-select: none; -} - -.filter-description { - font-size: 14px; - color: var(--proposal-link); - max-height: 70%; - margin-right: 10px; -} - -.toggle-filter-panel { - -webkit-user-select: none; - -moz-user-select: none; - user-select: none; -} - -.active .icon-line { - stroke: white; -} - -.active .icon-circle { - fill: var(--proposal-link); -} - -.expandable { - display: none; - max-height: 0px; - overflow: hidden; -} - -.expandable.expanded { - display: block; - flex-basis: 100%; - overflow: hidden; - max-height: 100vh; - order: 4; -} - -#filter-options-label, #version-options-label { - font-size: 14px; - font-weight: 300; - margin: 10px 0 0 0; -} - -#search-icon { - position: relative; - top: 4px; - left: 8px; - z-index: 1; -} - -#search-container { - flex-grow: 100; -} - -.filter { - -webkit-appearance: textfield; - font-size: 14px; - height: 28px; - max-height: 28px; - width: 200px; - padding: 0px 25px; - position: relative; - left: -20px; - font-family: -apple-system, BlinkMacSystemFont, "Helvetica Neue", Helvetica, Arial, Verdana, sans-serif; -} - -::-webkit-search-cancel-button { - -webkit-appearance: none; -} - -#clear-button { - position: relative; - top: 2px; - left: -45px; - z-index: 1; - cursor: pointer; -} - -@media (max-width: 1000px) { - body { - font-size: 16px; - } - - header { - height: auto; - } - - #logo { - width: 40px; - height: 40px; - background-size: 40px; - } - - .header-contents { - flex-basis: auto; - } - - .nav-contents { - width: 90%; - justify-content: center; - } - - .footer-contents { - flex-basis: 90%; - } - - #title { - font-size: 34px; - line-height: 75px; - margin: 5px 5px; - } - - h3 { - font-size: 14pt; - } - - .filter { - padding: 0px 25px; - } - - #search-container { - flex-grow: 100; - margin-top: 3px; - height: 38px; - } - - .filter-button { - margin-right: 10px; - margin-top: 3px; - } - - #search-filter { - width: calc(100% - 37px); - } - - #search-icon { - top: 3px; - width: 14px; - height: 14px; - } - - #clear-button { - left: -50px; - } - - #proposals-count { - margin-left: 6px; - } - - h4 a { - white-space: normal; - display: inline-block; - line-height: 24px; - -webkit-hyphens: auto; - -ms-hyphens: auto; - hyphens: auto; - } - - .status-pill { - width: inherit; - max-width: inherit; - min-width: inherit; - padding: 0px 8px; - margin-bottom: 6px; - } - - .authors a { - white-space: normal; - } - - .proposal { - flex-direction: column; - max-width: 100%; - padding: 0px 10px; - } - - .proposal-header { - flex-direction: column; - } - - .proposal-detail { - display: block; - } - - .proposal-detail-label, .proposal-detail-value { - display: inline; - } -} - -@media (max-width: 768px) { - h1 { - font-size: 54px; - } -} - -@media (max-width: 414px){ - .filter-button { - order: 0; - } -} - -/* status label colors */ - -.color-awaiting-review { - color: rgb(255, 149, 0); - border-color: rgb(255, 149, 0); -} - -.color-scheduled-for-review { - color: rgb(255, 149, 0); - border-color: rgb(255, 149, 0); -} - -.color-active-review { - color: rgb(255, 149, 0); - border-color: rgb(255, 149, 0); -} -.color-returned-for-revision { - color: rgb(88, 86, 214); - border-color: rgb(88, 86, 214); -} - -.color-deferred { - color: rgb(88, 86, 214); - border-color: rgb(88, 86, 214); -} -.color-accepted,.color-accepted-with-revisions { - color: rgb(76, 217, 100); - border-color: rgb(76, 217, 100); -} -.color-rejected { - color: rgb(255, 59, 48); - border-color: rgb(255, 59, 48); -} -.color-previewing { - color: rgb(0, 190, 180); - border-color: rgb(0, 190, 180); -} -.color-implemented { - color: rgb(0, 122, 255); - border-color: rgb(0, 122, 255); -} -.color-withdrawn { - color: rgb(255, 59, 48); - border-color: rgb(255, 59, 48); -} diff --git a/index.html b/index.html deleted file mode 100644 index ac99710440..0000000000 --- a/index.html +++ /dev/null @@ -1,97 +0,0 @@ - - - - - - - - - - - - - - - - - - - - Swift Evolution - - -
-
- - -

Swift Evolution

-
-
- -
-
-
-
    - -
-
-
-
- -
-
- - - diff --git a/index.js b/index.js deleted file mode 100644 index 78c7b3aff7..0000000000 --- a/index.js +++ /dev/null @@ -1,1064 +0,0 @@ -// ===--- index.js - Swift Evolution --------------------------------------===// -// -// This source file is part of the Swift.org open source project -// -// Copyright (c) 2014 - 2021 Apple Inc. and the Swift project authors -// Licensed under Apache License v2.0 with Runtime Library Exception -// -// See http://swift.org/LICENSE.txt for license information -// See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors -// -// ===---------------------------------------------------------------------===// - -'use strict' - -/** Holds the primary data used on this page: metadata about Swift Evolution proposals. */ -var proposals - -/** - * To be updated when proposals are confirmed to have been implemented - * in a new language version. - */ -var languageVersions = ['2.2', '3', '3.0.1', '3.1', '4', '4.1', '4.2', '5', '5.1', '5.2', '5.3', '5.4', '5.5', '5.6', 'Next'] - -/** Storage for the user's current selection of filters when filtering is toggled off. */ -var filterSelection = [] - -var GITHUB_BASE_URL = 'https://github.com/' -var REPO_PROPOSALS_BASE_URL = GITHUB_BASE_URL + 'apple/swift-evolution/blob/main/proposals' - -/** - * `name`: Mapping of the states in the proposals JSON to human-readable names. - * - * `shortName`: Mapping of the states in the proposals JSON to short human-readable names. - * Used for the left-hand column of proposal statuses. - * - * `className`: Mapping of states in the proposals JSON to the CSS class names used - * to manipulate and display proposals based on their status. - * - * `count`: Number of proposals that determine after all proposals is loaded - */ -var states = { - '.awaitingReview': { - name: 'Awaiting Review', - shortName: 'Awaiting Review', - className: 'awaiting-review', - count: 0 - }, - '.scheduledForReview': { - name: 'Scheduled for Review', - shortName: 'Scheduled', - className: 'scheduled-for-review', - count: 0 - }, - '.activeReview': { - name: 'Active Review', - shortName: 'Active Review', - className: 'active-review', - count: 0 - }, - '.returnedForRevision': { - name: 'Returned for Revision', - shortName: 'Returned', - className: 'returned-for-revision', - count: 0 - }, - '.withdrawn': { - name: 'Withdrawn', - shortName: 'Withdrawn', - className: 'withdrawn', - count: 0 - }, - '.deferred': { - name: 'Deferred', - shortName: 'Deferred', - className: 'deferred', - count: 0 - }, - '.accepted': { - name: 'Accepted', - shortName: 'Accepted', - className: 'accepted', - count: 0 - }, - '.acceptedWithRevisions': { - name: 'Accepted with revisions', - shortName: 'Accepted', - className: 'accepted-with-revisions', - count: 0 - }, - '.rejected': { - name: 'Rejected', - shortName: 'Rejected', - className: 'rejected', - count: 0 - }, - '.implemented': { - name: 'Implemented', - shortName: 'Implemented', - className: 'implemented', - count: 0 - }, - '.previewing': { - name: 'Previewing', - shortName: 'Previewing', - className: 'previewing', - count: 0 - }, - '.error': { - name: 'Error', - shortName: 'Error', - className: 'error', - count: 0 - } -} - -init() - -/** Primary entry point. */ -function init () { - var req = new window.XMLHttpRequest() - - req.addEventListener('load', function (e) { - proposals = JSON.parse(req.responseText) - - // don't display malformed proposals - proposals = proposals.filter(function (proposal) { - return !proposal.errors - }) - - // descending numeric sort based the numeric nnnn in a proposal ID's SE-nnnn - proposals.sort(function compareProposalIDs (p1, p2) { - return parseInt(p1.id.match(/\d\d\d\d/)[0]) - parseInt(p2.id.match(/\d\d\d\d/)[0]) - }) - proposals = proposals.reverse() - - render() - addEventListeners() - - // apply filters when the page loads with a search already filled out. - // typically this happens after navigating backwards in a tab's history. - if (document.querySelector('#search-filter').value.trim()) { - filterProposals() - } - - // apply selections from the current page's URI fragment - _applyFragment(document.location.hash) - }) - - req.addEventListener('error', function (e) { - document.querySelector('#proposals-count-number').innerText = 'Proposal data failed to load.' - }) - - document.querySelector('#proposals-count-number').innerHTML = 'Loading ...' - req.open('get', 'https://data.swift.org/swift-evolution/proposals') - req.send() -} - -/** - * Creates an Element. Convenience wrapper for `document.createElement`. - * - * @param {string} elementType - The tag name. 'div', 'span', etc. - * @param {string[]} attributes - A list of attributes. Use `className` for `class`. - * @param {(string | Element)[]} children - A list of either text or other Elements to be nested under this Element. - * @returns {Element} The new node. - */ -function html (elementType, attributes, children) { - var element = document.createElement(elementType) - - if (attributes) { - Object.keys(attributes).forEach(function (attributeName) { - var value = attributes[attributeName] - if (attributeName === 'className') attributeName = 'class' - element.setAttribute(attributeName, value) - }) - } - - if (!children) return element - if (!Array.isArray(children)) children = [children] - - children.forEach(function (child) { - if (!child) { - console.warn('Null child ignored during creation of ' + elementType) - return - } - if (Object.getPrototypeOf(child) === String.prototype) { - child = document.createTextNode(child) - } - - element.appendChild(child) - }) - - return element -} - -function determineNumberOfProposals (proposals) { - // reset count - Object.keys(states).forEach(function (state){ - states[state].count = 0 - }) - - proposals.forEach(function (proposal) { - states[proposal.status.state].count += 1 - }) - - // .acceptedWithRevisions proposals are combined in the filtering UI - // with .accepted proposals. - states['.accepted'].count += states['.acceptedWithRevisions'].count -} - -/** - * Adds the dynamic portions of the page to the DOM, primarily the list - * of proposals and list of statuses used for filtering. - * - * These `render` functions are only called once when the page loads, - * the rest of the interactivity is based on toggling `display: none`. - */ -function render () { - renderNav() - renderBody() -} - -/** Renders the top navigation bar. */ -function renderNav () { - var nav = document.querySelector('nav') - - // This list intentionally omits .acceptedWithRevisions and .error; - // .acceptedWithRevisions proposals are combined in the filtering UI - // with .accepted proposals. - var checkboxes = [ - '.awaitingReview', '.scheduledForReview', '.activeReview', '.accepted', - '.previewing', '.implemented', '.returnedForRevision', '.deferred', '.rejected', '.withdrawn' - ].map(function (state) { - var className = states[state].className - - return html('li', null, [ - html('input', { type: 'checkbox', className: 'filtered-by-status', id: 'filter-by-' + className, value: className }), - html('label', { className: className, tabindex: '0', role: 'button', 'for': 'filter-by-' + className, 'data-state-key': state }, [ - addNumberToState(states[state].name, states[state].count) - ]) - ]) - }) - - var expandableArea = html('div', { className: 'filter-options expandable' }, [ - html('h5', { id: 'filter-options-label' }, 'Status'), - html('ul', { id: 'filter-options', className: 'filter-by-status' }) - ]) - - nav.querySelector('.nav-contents').appendChild(expandableArea) - - checkboxes.forEach(function (box) { - nav.querySelector('.filter-by-status').appendChild(box) - }) - - // The 'Implemented' filter selection gets an extra row of options if selected. - var implementedCheckboxIfPresent = checkboxes.filter(function (cb) { - return cb.querySelector(`#filter-by-${states['.implemented'].className}`) - })[0] - - if (implementedCheckboxIfPresent) { - // add an extra row of options to filter by language version - var versionRowHeader = html('h5', { id: 'version-options-label', className: 'hidden' }, 'Language Version') - var versionRow = html('ul', { id: 'version-options', className: 'filter-by-status hidden' }) - - var versionOptions = languageVersions.map(function (version) { - return html('li', null, [ - html('input', { - type: 'checkbox', - id: 'filter-by-swift-' + _idSafeName(version), - className: 'filter-by-swift-version', - value: 'swift-' + _idSafeName(version) - }), - html('label', { - tabindex: '0', - role: 'button', - 'for': 'filter-by-swift-' + _idSafeName(version) - }, 'Swift ' + version) - ]) - }) - - versionOptions.forEach(function (version) { - versionRow.appendChild(version) - }) - - expandableArea.appendChild(versionRowHeader) - expandableArea.appendChild(versionRow) - } - - return nav -} - -/** Displays the main list of proposals that takes up the majority of the page. */ -function renderBody () { - var article = document.querySelector('article') - - var proposalAttachPoint = article.querySelector('.proposals-list') - - var proposalPresentationOrder = [ - '.awaitingReview', '.scheduledForReview', '.activeReview', '.accepted', '.acceptedWithRevisions', - '.previewing', '.implemented', '.returnedForRevision', '.deferred', '.rejected', '.withdrawn' - ] - - proposalPresentationOrder.map(function (state) { - var matchingProposals = proposals.filter(function (p) { return p.status && p.status.state === state }) - matchingProposals.map(function (proposal) { - var proposalBody = html('section', { id: proposal.id, className: 'proposal ' + proposal.id }, [ - html('div', { className: 'status-pill-container' }, [ - html('span', { className: 'status-pill color-' + states[state].className }, [ - states[proposal.status.state].shortName - ]) - ]), - html('div', { className: 'proposal-content' }, [ - html('div', { className: 'proposal-header' }, [ - html('span', { className: 'proposal-id' }, [ - proposal.id - ]), - html('h4', { className: 'proposal-title' }, [ - html('a', { - href: REPO_PROPOSALS_BASE_URL + '/' + proposal.link, - target: '_blank' - }, [ - proposal.title - ]) - ]) - ]) - ]) - ]) - - var detailNodes = [] - detailNodes.push(renderAuthors(proposal.authors)) - - if (proposal.reviewManager.name) detailNodes.push(renderReviewManager(proposal.reviewManager)) - if (proposal.trackingBugs) detailNodes.push(renderTrackingBugs(proposal.trackingBugs)) - if (state === '.implemented') detailNodes.push(renderVersion(proposal.status.version)) - if (state === '.previewing') detailNodes.push(renderPreview()) - if (proposal.implementation) detailNodes.push(renderImplementation(proposal.implementation)) - if (state === '.acceptedWithRevisions') detailNodes.push(renderStatus(proposal.status)) - - if (state === '.activeReview' || state === '.scheduledForReview') { - detailNodes.push(renderStatus(proposal.status)) - detailNodes.push(renderReviewPeriod(proposal.status)) - } - - if (state === '.returnedForRevision') { - detailNodes.push(renderStatus(proposal.status)) - } - - var details = html('div', { className: 'proposal-details' }, detailNodes) - - proposalBody.querySelector('.proposal-content').appendChild(details) - proposalAttachPoint.appendChild(proposalBody) - }) - }) - - // Update the "(n) proposals" text - updateProposalsCount(article.querySelectorAll('.proposal').length) - - return article -} - -/** Authors have a `name` and optional `link`. */ -function renderAuthors (authors) { - var authorNodes = authors.map(function (author) { - if (author.link.length > 0) { - return html('a', { href: author.link, target: '_blank' }, author.name) - } else { - return document.createTextNode(author.name) - } - }) - - authorNodes = _joinNodes(authorNodes, ', ') - - return html('div', { className: 'authors proposal-detail' }, [ - html('div', { className: 'proposal-detail-label' }, - authors.length > 1 ? 'Authors: ' : 'Author: ' - ), - html('div', { className: 'proposal-detail-value' }, authorNodes) - ]) -} - -/** Review managers have a `name` and optional `link`. */ -function renderReviewManager (reviewManager) { - return html('div', { className: 'review-manager proposal-detail' }, [ - html('div', { className: 'proposal-detail-label' }, 'Review Manager: '), - html('div', { className: 'proposal-detail-value' }, [ - reviewManager.link - ? html('a', { href: reviewManager.link, target: '_blank' }, reviewManager.name) - : reviewManager.name - ]) - ]) -} - -/** Tracking bugs linked in a proposal are updated via bugs.swift.org. */ -function renderTrackingBugs (bugs) { - var bugNodes = bugs.map(function (bug) { - return html('a', { href: bug.link, target: '_blank' }, [ - bug.id, - ' (', - bug.assignee || 'Unassigned', - ', ', - bug.status, - ')' - ]) - }) - - bugNodes = _joinNodes(bugNodes, ', ') - - return html('div', { className: 'proposal-detail' }, [ - html('div', { className: 'proposal-detail-label' }, [ - bugs.length > 1 ? 'Bugs: ' : 'Bug: ' - ]), - html('div', { className: 'bug-list proposal-detail-value' }, - bugNodes - ) - ]) -} - -/** Implementations are required alongside proposals (after Swift 4.0). */ -function renderImplementation (implementations) { - var implNodes = implementations.map(function (impl) { - return html('a', { - href: GITHUB_BASE_URL + impl.account + '/' + impl.repository + '/' + impl.type + '/' + impl.id - }, [ - impl.repository, - impl.type === 'pull' ? '#' : '@', - impl.id.substr(0, 7) - ]) - }) - - implNodes = _joinNodes(implNodes, ', ') - - var label = 'Implementation: ' - - return html('div', { className: 'proposal-detail' }, [ - html('div', { className: 'proposal-detail-label' }, [label]), - html('div', { className: 'implementation-list proposal-detail-value' }, - implNodes - ) - ]) -} - -/** For `.previewing` proposals, link to the stdlib preview package. */ -function renderPreview () { - return html('div', { className: 'proposal-detail' }, [ - html('div', { className: 'proposal-detail-label' }, [ - 'Preview: ' - ]), - html('div', { className: 'proposal-detail-value' }, [ - html('a', { href: 'https://github.com/apple/swift-standard-library-preview', target: '_blank' }, - 'Standard Library Preview' - ) - ]) - ]) -} - -/** For `.implemented` proposals, display the version of Swift in which they first appeared. */ -function renderVersion (version) { - return html('div', { className: 'proposal-detail' }, [ - html('div', { className: 'proposal-detail-label' }, [ - 'Implemented In: ' - ]), - html('div', { className: 'proposal-detail-value' }, [ - 'Swift ' + version - ]) - ]) -} - -/** For some proposal states like `.activeReview`, it helps to see the status in the same details list. */ -function renderStatus (status) { - return html('div', { className: 'proposal-detail' }, [ - html('div', { className: 'proposal-detail-label' }, [ - 'Status: ' - ]), - html('div', { className: 'proposal-detail-value' }, [ - states[status.state].name - ]) - ]) -} - -/** - * Review periods are ISO-8601-style 'YYYY-MM-DD' dates. - */ -function renderReviewPeriod (status) { - var months = ['January', 'February', 'March', 'April', 'May', 'June', 'July', - 'August', 'September', 'October', 'November', 'December' - ] - - var start = new Date(status.start) - var end = new Date(status.end) - - var startMonth = start.getUTCMonth() - var endMonth = end.getUTCMonth() - - var detailNodes = [months[startMonth], ' '] - - if (startMonth === endMonth) { - detailNodes.push( - start.getUTCDate().toString(), - '–', - end.getUTCDate().toString() - ) - } else { - detailNodes.push( - start.getUTCDate().toString(), - ' – ', - months[endMonth], - ' ', - end.getUTCDate().toString() - ) - } - - return html('div', { className: 'proposal-detail' }, [ - html('div', { className: 'proposal-detail-label' }, [ - 'Scheduled: ' - ]), - html('div', { className: 'proposal-detail-value' }, detailNodes) - ]) -} - -/** Utility used by some of the `render*` functions to add comma text nodes between DOM nodes. */ -function _joinNodes (nodeList, text) { - return nodeList.map(function (node) { - return [node, text] - }).reduce(function (result, pair, index, pairs) { - if (index === pairs.length - 1) pair.pop() - return result.concat(pair) - }, []) -} - -/** Adds UI interactivity to the page. Primarily activates the filtering controls. */ -function addEventListeners () { - var nav = document.querySelector('nav') - - // typing in the search field causes the filter to be reapplied. - nav.addEventListener('keyup', filterProposals) - nav.addEventListener('change', filterProposals) - - // clearing the search field also hides the X symbol - nav.querySelector('#clear-button').addEventListener('click', function () { - nav.querySelector('#search-filter').value = '' - nav.querySelector('#clear-button').classList.toggle('hidden') - filterProposals() - }) - - // each of the individual statuses needs to trigger filtering as well - ;[].forEach.call(nav.querySelectorAll('.filter-by-status input'), function (element) { - element.addEventListener('change', filterProposals) - }) - - var expandableArea = document.querySelector('.filter-options') - var implementedToggle = document.querySelector('#filter-by-implemented') - implementedToggle.addEventListener('change', function () { - // hide or show the row of version options depending on the status of the 'Implemented' option - ;['#version-options', '#version-options-label'].forEach(function (selector) { - expandableArea.querySelector(selector).classList.toggle('hidden') - }) - - // don't persist any version selections when the row is hidden - ;[].concat.apply([], expandableArea.querySelectorAll('.filter-by-swift-version')).forEach(function (versionCheckbox) { - versionCheckbox.checked = false - }) - }) - - document.querySelector('.filter-button').addEventListener('click', toggleFiltering) - - var filterToggle = document.querySelector('.filter-toggle') - filterToggle.querySelector('.toggle-filter-panel').addEventListener('click', toggleFilterPanel) - - // Behavior conditional on certain browser features - var CSS = window.CSS - if (CSS) { - // emulate position: sticky when it isn't available. - if (!(CSS.supports('position', 'sticky') || CSS.supports('position', '-webkit-sticky'))) { - window.addEventListener('scroll', function () { - var breakpoint = document.querySelector('header').getBoundingClientRect().bottom - var nav = document.querySelector('nav') - var position = window.getComputedStyle(nav).position - var shadowNav // maintain the main content height when the main 'nav' is removed from the flow - - // this is measuring whether or not the header has scrolled offscreen - if (breakpoint <= 0) { - if (position !== 'fixed') { - shadowNav = nav.cloneNode(true) - shadowNav.classList.add('clone') - shadowNav.style.visibility = 'hidden' - nav.parentNode.insertBefore(shadowNav, document.querySelector('main')) - nav.style.position = 'fixed' - } - } else if (position === 'fixed') { - nav.style.position = 'static' - shadowNav = document.querySelector('nav.clone') - if (shadowNav) shadowNav.parentNode.removeChild(shadowNav) - } - }) - } - } - - // on smaller screens, hide the filter panel when scrolling - if (window.matchMedia('(max-width: 414px)').matches) { - window.addEventListener('scroll', function () { - var breakpoint = document.querySelector('header').getBoundingClientRect().bottom - if (breakpoint <= 0 && document.querySelector('.expandable').classList.contains('expanded')) { - toggleFilterPanel() - } - }) - } -} - -/** - * Toggles whether filters are active. Rather than being cleared, they are saved to be restored later. - * Additionally, toggles the presence of the "Filtered by:" status indicator. - */ -function toggleFiltering () { - var filterDescription = document.querySelector('.filter-toggle') - var shouldPreserveSelection = !filterDescription.classList.contains('hidden') - - filterDescription.classList.toggle('hidden') - var selected = document.querySelectorAll('.filter-by-status input[type=checkbox]:checked') - var filterButton = document.querySelector('.filter-button') - - if (shouldPreserveSelection) { - filterSelection = [].map.call(selected, function (checkbox) { return checkbox.id }) - ;[].forEach.call(selected, function (checkbox) { checkbox.checked = false }) - - filterButton.setAttribute('aria-pressed', 'false') - } else { // restore it - filterSelection.forEach(function (id) { - var checkbox = document.getElementById(id) - checkbox.checked = true - }) - - filterButton.setAttribute('aria-pressed', 'true') - } - - document.querySelector('.expandable').classList.remove('expanded') - filterButton.classList.toggle('active') - - filterProposals() -} - -/** - * Expands or contracts the filter panel, which contains buttons that - * let users filter proposals based on their current stage in the - * Swift Evolution process. - */ -function toggleFilterPanel () { - var panel = document.querySelector('.expandable') - var button = document.querySelector('.toggle-filter-panel') - - panel.classList.toggle('expanded') - - if (panel.classList.contains('expanded')) { - button.setAttribute('aria-pressed', 'true') - } else { - button.setAttribute('aria-pressed', 'false') - } -} - -/** - * Applies both the status-based and text-input based filters to the proposals list. - */ -function filterProposals () { - var filterElement = document.querySelector('#search-filter') - var filter = filterElement.value - - var clearButton = document.querySelector('#clear-button') - if (filter.length === 0) { - clearButton.classList.add('hidden') - } else { - clearButton.classList.remove('hidden') - } - - var matchingSets = [proposals.concat()] - - // Comma-separated lists of proposal IDs are treated as an "or" search. - if (filter.match(/(SE-\d\d\d\d)($|((,SE-\d\d\d\d)+))/i)) { - var proposalIDs = filter.split(',').map(function (id) { - return id.toUpperCase() - }) - - matchingSets[0] = matchingSets[0].filter(function (proposal) { - return proposalIDs.indexOf(proposal.id) !== -1 - }) - } else if (filter.trim().length !== 0) { - // The search input treats words as order-independent. - matchingSets = filter.split(/\s/) - .filter(function (s) { return s.length > 0 }) - .map(function (part) { return _searchProposals(part) }) - } - - var intersection = matchingSets.reduce(function (intersection, candidates) { - return intersection.filter(function (alreadyIncluded) { return candidates.indexOf(alreadyIncluded) !== -1 }) - }, matchingSets[0] || []) - - _applyFilter(intersection) - _updateURIFragment() - - determineNumberOfProposals(intersection) - updateFilterStatus() -} - -/** - * Utility used by `filterProposals`. - * - * Picks out various fields in a proposal which users may want to key - * off of in their text-based filtering. - * - * @param {string} filterText - A raw word of text as entered by the user. - * @returns {Proposal[]} The proposals that match the entered text, taken from the global list. - */ -function _searchProposals (filterText) { - var filterExpression = filterText.toLowerCase() - - var searchableProperties = [ - ['id'], - ['title'], - ['reviewManager', 'name'], - ['status', 'state'], - ['status', 'version'], - ['authors', 'name'], - ['authors', 'link'], - ['implementation', 'account'], - ['implementation', 'repository'], - ['implementation', 'id'], - ['trackingBugs', 'link'], - ['trackingBugs', 'status'], - ['trackingBugs', 'id'], - ['trackingBugs', 'assignee'] - ] - - // reflect over the proposals and find ones with matching properties - var matchingProposals = proposals.filter(function (proposal) { - var match = false - searchableProperties.forEach(function (propertyList) { - var value = proposal - - propertyList.forEach(function (propertyName, index) { - if (!value) return - value = value[propertyName] - if (index < propertyList.length - 1) { - // For arrays, apply the property check to each child element. - // Note that this only looks to a depth of one property. - if (Array.isArray(value)) { - var matchCondition = value.some(function (element) { - return element[propertyList[index + 1]] && element[propertyList[index + 1]].toString().toLowerCase().indexOf(filterExpression) >= 0 - }) - - if (matchCondition) { - match = true - } - } else { - return - } - } else if (value && value.toString().toLowerCase().indexOf(filterExpression) >= 0) { - match = true - } - }) - }) - - return match - }) - - return matchingProposals -} - -/** - * Helper for `filterProposals` that actually makes the filter take effect. - * - * @param {Proposal[]} matchingProposals - The proposals that have passed the text filtering phase. - * @returns {Void} Toggles `display: hidden` to apply the filter. - */ -function _applyFilter (matchingProposals) { - // filter out proposals based on the grouping checkboxes - var allStateCheckboxes = document.querySelector('nav').querySelectorAll('.filter-by-status input:checked') - var selectedStates = [].map.call(allStateCheckboxes, function (checkbox) { return checkbox.value }) - - var selectedStateNames = [].map.call(allStateCheckboxes, function (checkbox) { return checkbox.nextElementSibling.innerText.trim() }) - updateFilterDescription(selectedStateNames) - - if (selectedStates.length) { - matchingProposals = matchingProposals - .filter(function (proposal) { - return selectedStates.some(function (state) { - return proposal.status.state.toLowerCase().indexOf(state.split('-')[0]) >= 0 - }) - }) - - // handle version-specific filtering options - if (selectedStates.some(function (state) { return state.match(/swift/i) })) { - matchingProposals = matchingProposals - .filter(function (proposal) { - return selectedStates.some(function (state) { - if (!(proposal.status.state === '.implemented')) return true // only filter among Implemented (N.N.N) - if (state === 'swift-swift-Next' && proposal.status.version === 'Next') return true // special case - - var version = state.split(/\D+/).filter(function (s) { return s.length }).join('.') - - if (!version.length) return false // it's not a state that represents a version number - if (proposal.status.version === version) return true - return false - }) - }) - } - } - - var filteredProposals = proposals.filter(function (proposal) { - return matchingProposals.indexOf(proposal) === -1 - }) - - matchingProposals.forEach(function (proposal) { - var matchingElements = [].concat.apply([], document.querySelectorAll('.' + proposal.id)) - matchingElements.forEach(function (element) { element.classList.remove('hidden') }) - }) - - filteredProposals.forEach(function (proposal) { - var filteredElements = [].concat.apply([], document.querySelectorAll('.' + proposal.id)) - filteredElements.forEach(function (element) { element.classList.add('hidden') }) - }) - - updateProposalsCount(matchingProposals.length) -} - -/** - * Parses a URI fragment and applies a search and filters to the page. - * - * Syntax (a query string within a fragment): - * fragment --> `#?` parameter-value-list - * parameter-value-list --> parameter-value | parameter-value-pair `&` parameter-value-list - * parameter-value-pair --> parameter `=` value - * parameter --> `proposal` | `status` | `version` | `search` - * value --> ** Any URL-encoded text. ** - * - * For example: - * /#?proposal:SE-0180,SE-0123 - * /#?status=rejected&version=3&search=access - * - * Four types of parameters are supported: - * - proposal: A comma-separated list of proposal IDs. Treated as an 'or' search. - * - filter: A comma-separated list of proposal statuses to apply as a filter. - * - version: A comma-separated list of Swift version numbers to apply as a filter. - * - search: Raw, URL-encoded text used to filter by individual term. - * - * @param {string} fragment - A URI fragment to use as the basis for a search. - */ -function _applyFragment (fragment) { - if (!fragment || fragment.substr(0, 2) !== '#?') return - fragment = fragment.substring(2) // remove the #? - - // use this literal's keys as the source of truth for key-value pairs in the fragment - var actions = { proposal: [], search: null, status: [], version: [] } - - // parse the fragment as a query string - Object.keys(actions).forEach(function (action) { - var pattern = new RegExp(action + '=([^=]+)(&|$)') - var values = fragment.match(pattern) - - if (values) { - var value = values[1] // 1st capture group from the RegExp - if (action === 'search') { - value = decodeURIComponent(value) - } else { - value = value.split(',') - } - - actions[action] = value - } - }) - - // perform key-specific parsing and checks - - if (actions.proposal.length) { - document.querySelector('#search-filter').value = actions.proposal.join(',') - } else if (actions.search) { - document.querySelector('#search-filter').value = actions.search - } - - if (actions.version.length) { - var versionSelections = actions.version.map(function (version) { - return document.querySelector('#filter-by-swift-' + _idSafeName(version)) - }).filter(function (version) { - return !!version - }) - - versionSelections.forEach(function (versionSelection) { - versionSelection.checked = true - }) - - if (versionSelections.length) { - document.querySelector( - '#filter-by-' + states['.implemented'].className - ).checked = true - } - } - - // track this state specifically for toggling the version panel - var implementedSelected = false - - // update the filter selections in the nav - if (actions.status.length) { - var statusSelections = actions.status.map(function (status) { - var stateName = Object.keys(states).filter(function (state) { - return states[state].className === status - })[0] - - if (!stateName) return // fragment contains a nonexistent state - var state = states[stateName] - - if (stateName === '.implemented') implementedSelected = true - - return document.querySelector('#filter-by-' + state.className) - }).filter(function (status) { - return !!status - }) - - statusSelections.forEach(function (statusSelection) { - statusSelection.checked = true - }) - } - - // the version panel needs to be activated if any are specified - if (actions.version.length || implementedSelected) { - ;['#version-options', '#version-options-label'].forEach(function (selector) { - document.querySelector('.filter-options') - .querySelector(selector).classList - .toggle('hidden') - }) - } - - // specifying any filter in the fragment should activate the filters in the UI - if (actions.version.length || actions.status.length) { - toggleFilterPanel() - toggleFiltering() - } - - filterProposals() -} - -/** - * Writes out the current search and filter settings to document.location - * via window.replaceState. - */ -function _updateURIFragment () { - var actions = { proposal: [], search: null, status: [], version: [] } - - var search = document.querySelector('#search-filter') - - if (search.value && search.value.match(/(SE-\d\d\d\d)($|((,SE-\d\d\d\d)+))/i)) { - actions.proposal = search.value.toUpperCase().split(',') - } else { - actions.search = search.value - } - - var selectedVersions = document.querySelectorAll('.filter-by-swift-version:checked') - var versions = [].map.call(selectedVersions, function (checkbox) { - return checkbox.value.split('swift-swift-')[1].split('-').join('.') - }) - - actions.version = versions - - var selectedStatuses = document.querySelectorAll('.filtered-by-status:checked') - var statuses = [].map.call(selectedStatuses, function (checkbox) { - var className = checkbox.value - - var correspondingStatus = Object.keys(states).filter(function (status) { - if (states[status].className === className) return true - return false - })[0] - - return states[correspondingStatus].className - }) - - // .implemented is redundant if any specific implementation versions are selected. - if (actions.version.length) { - statuses = statuses.filter(function (status) { - return status !== states['.implemented'].className - }) - } - - actions.status = statuses - - // build the actual fragment string. - var fragments = [] - if (actions.proposal.length) fragments.push('proposal=' + actions.proposal.join(',')) - if (actions.status.length) fragments.push('status=' + actions.status.join(',')) - if (actions.version.length) fragments.push('version=' + actions.version.join(',')) - - // encoding the search lets you search for `??` and other edge cases. - if (actions.search) fragments.push('search=' + encodeURIComponent(actions.search)) - - if (!fragments.length) { - window.history.replaceState(null, null, './') - return - } - - var fragment = '#?' + fragments.join('&') - - // avoid creating new history entries each time a search or filter updates - window.history.replaceState(null, null, fragment) -} - -/** Helper to give versions like 3.0.1 an okay ID to use in a DOM element. (swift-3-0-1) */ -function _idSafeName (name) { - return 'swift-' + name.replace(/\./g, '-') -} - -/** - * Changes the text after 'Filtered by: ' to reflect the current status filters. - * - * After FILTER_DESCRIPTION_LIMIT filters are explicitly named, start combining the descriptive text - * to just state the number of status filters taking effect, not what they are. - * - * @param {string[]} selectedStateNames - CSS class names corresponding to which statuses were selected. - * Populated from the global `stateNames` array. - */ -function updateFilterDescription (selectedStateNames) { - var FILTER_DESCRIPTION_LIMIT = 2 - var stateCount = selectedStateNames.length - - // Limit the length of filter text on small screens. - if (window.matchMedia('(max-width: 414px)').matches) { - FILTER_DESCRIPTION_LIMIT = 1 - } - - var container = document.querySelector('.toggle-filter-panel') - - // modify the state names to clump together Implemented with version names - var swiftVersionStates = selectedStateNames.filter(function (state) { return state.match(/swift/i) }) - - if (swiftVersionStates.length > 0 && swiftVersionStates.length <= FILTER_DESCRIPTION_LIMIT) { - selectedStateNames = selectedStateNames.filter(function (state) { return !state.match(/swift|implemented/i) }) - .concat('Implemented (' + swiftVersionStates.join(', ') + ')') - } - - if (selectedStateNames.length > FILTER_DESCRIPTION_LIMIT) { - container.innerText = stateCount + ' Filters' - } else if (selectedStateNames.length === 0) { - container.innerText = 'All Statuses' - } else { - selectedStateNames = selectedStateNames.map(cleanNumberFromState) - container.innerText = selectedStateNames.join(' or ') - } -} - -/** Updates the `${n} Proposals` display just above the proposals list. */ -function updateProposalsCount (count) { - var numberField = document.querySelector('#proposals-count-number') - numberField.innerText = (count.toString() + ' proposal' + (count !== 1 ? 's' : '')) -} - -function updateFilterStatus () { - var labels = [].concat.apply([], document.querySelectorAll('#filter-options label')) - labels.forEach(function (label) { - var count = states[label.getAttribute('data-state-key')].count - var cleanedLabel = cleanNumberFromState(label.innerText) - label.innerText = addNumberToState(cleanedLabel, count) - }) -} - -function cleanNumberFromState (state) { - return state.replace(/ *\([^)]*\) */g, '') -} - -function addNumberToState (state, count) { - return state + ' (' + count + ')' -} diff --git a/process.md b/process.md index e4a0815181..3408e2d567 100644 --- a/process.md +++ b/process.md @@ -1,10 +1,28 @@ # Swift Evolution Process -Swift is a powerful and intuitive programming language that is designed to make writing and maintaining correct programs easier. Swift is growing and evolving, guided by a community-driven process referred to as the Swift evolution process. This document outlines the Swift evolution process and how a feature grows from a rough idea into something that can improve the Swift development experience for millions of programmers. +Swift is a powerful and intuitive programming language that is designed to make writing and maintaining correct programs easier. Swift is growing and evolving, guided by a community-driven process referred to as the Swift evolution process, maintained by the [Language Steering Group][language-steering-group]. This document outlines the Swift evolution process and how a feature grows from a rough idea into something that can improve the Swift development experience for millions of programmers. ## Scope -The Swift evolution process covers all changes to the Swift language and the public interface of the Swift standard library, including new language features and APIs (no matter how small), changes to existing language features or APIs, removal of existing features, and so on. Smaller changes, such as bug fixes, optimizations, or diagnostic improvements can be contributed via the normal contribution process; see [Contributing to Swift](https://swift.org/community/#contributing). +The Swift evolution process covers all design changes, no matter how small, to the Swift language, its standard library, and the core tools necessary to build Swift programs. This includes additions, removals, and changes to: + +- the features of the Swift language, +- the public interface of the Swift standard library, +- the configuration of the Swift compiler, +- the core tools of the Swift package ecosystem, including the configuration of + the [Swift package manager](https://www.swift.org/package-manager/) and the + design of its manifest files, and +- the public interfaces of the following libraries: + - [Swift Testing](https://github.com/swiftlang/swift-testing) + - [XCTest](https://github.com/swiftlang/swift-corelibs-xctest) + +The design of other tools, such as IDEs, debuggers, and documentation generators, is not covered by the evolution process. The Core Team may create workgroups to guide and make recommendations about the development of these tools, but the output of those workgroups is not reviewed. + +The evolution process does not cover experimental features, which can be added, changed, or removed at any time. Implementors should take steps to prevent the accidental use of experimental features, such as by enabling them only under explicitly experimental options. Features should not be allowed to remain perpetually experimental; a feature with no clear path for development into an official feature should be removed. + +Changes such as bug fixes, optimizations, or diagnostic improvements can be contributed via the normal contribution process; see [Contributing to Swift](https://www.swift.org/contributing/). Some bug fixes are effectively substantial changes to the design, even if they're just making the implementation match the official documentation; whether such a change requires evolution review is up to the appropriate evolution workgroup. + +Which parts of the Swift project are covered by the evolution process is ultimately up to the judgment of the Core Team. ## Goals @@ -15,6 +33,33 @@ The Swift evolution process aims to leverage the collective ideas, insights, and There is a natural tension between these two goals. Open evolution processes are, by nature, chaotic. Yet, maintaining a coherent vision for something as complicated as a programming language requires some level of coordination. The Swift evolution process aims to strike a balance that best serves the Swift community as a whole. +## Community structure + +The [Core Team](https://www.swift.org/community/#core-team) is responsible for the strategic direction of Swift. The Core Team creates workgroups focused on specific parts of the project. When the Core Team gives a workgroup authority over part of the evolution of the project, that workgroup is called an evolution workgroup. Evolution workgroups manage the evolution process for proposals under their authority, working together with other workgroups as needed. + +Currently, there are three evolution workgroups: + +* The [Language Steering Group][language-steering-group] has authority over the evolution of the Swift language, its standard library, and any language configuration features of the Swift package manager. +* The [Platform Steering Group][platform-steering-group] has authority over the evolution of all other features of the Swift package manager and its manifest files. +* The [Testing Workgroup][testing-workgroup] has authority over the evolution of + the Swift Testing and Corelibs XCTest projects. + +The Core Team manages (or delegates) the evolution process for proposals outside these areas. The Core Team also retains the power to override the evolution decisions of workgroups when necessary. + +## Proposals, roadmaps, and visions + +There are three kinds of documents commonly used in the evolution process. + +* An evolution *proposal* describes a specific proposed change in detail. All evolution changes are advanced as proposals which will be discussed in the community and given a formal open review. + +* An evolution *roadmap* describes a concrete plan for how a complex change will be broken into separate proposals that can be individually pitched and reviewed. Considering large changes in small pieces allows the community to provide more focused feedback about each part of the change. A roadmap makes this organization easier for community members to understand. + + Roadmaps are planning documents that do not need to be reviewed. + +* An evolution *vision* describes a high-level design for a broad topic (for example, string processing or concurrency). A vision creates a baseline of understanding in the community for future conversations on that topic, setting goals and laying out a possible program of work. + + Visions must be approved by the appropriate evolution workgroup. This approval is an endorsement of the vision's basic ideas, but not of any of its concrete proposals, which must still be separately developed and reviewed. + ## Participation Everyone is welcome to propose, discuss, and review ideas to improve @@ -22,11 +67,6 @@ the Swift language and standard library in the [Evolution section of the Swift forums](https://forums.swift.org/c/evolution). Before posting a review, please see the section "What goes into a review?" below. -The Swift [core team](https://swift.org/community/#core-team) is -responsible for the strategic direction of Swift. Core team members -initiate, participate in, and manage the public review of proposals -and have the authority to accept or reject changes to Swift. - ## What goes into a review? The goal of the review process is to improve the proposal under review @@ -44,29 +84,89 @@ Please state explicitly whether you believe that the proposal should be accepted ## How to propose a change -* **Check prior proposals**: many ideas come up frequently, and may either be in active discussion on the forums, or may have been discussed already and have joined the [Commonly Rejected Proposals](commonly_proposed.md) list. Please [search the forums](https://forums.swift.org/search) for context before proposing something new. -* **Consider the goals of the upcoming Swift release**: Each major -Swift release is focused on a [specific set of goals](README.md) -described early in the release cycle. When proposing a change to -Swift, please consider how your proposal fits in with the larger goals -of the upcoming Swift release. Proposals that are clearly out of scope -for the upcoming Swift release will not be brought up for review. If you can't resist discussing a proposal that you know is out of scope, please include the tag `[Out of scope]` in the subject. -* **Socialize the idea**: propose a rough sketch of the idea in the ["pitches" section of the Swift forums](https://forums.swift.org/c/evolution/pitches), the problems it solves, what the solution looks like, etc., to gauge interest from the community. -* **Develop the proposal**: expand the rough sketch into a complete proposal, using the [proposal template](proposal-templates/0000-swift-template.md), and continue to refine the proposal on the forums. Prototyping an implementation and its uses along with the proposal is *required* because it helps ensure both technical feasibility of the proposal as well as validating that the proposal solves the problems it is meant to solve. -* **Request a review**: initiate a pull request to the [swift-evolution repository][swift-evolution-repo] to indicate to the core team that you would like the proposal to be reviewed. When the proposal is sufficiently detailed and clear, and addresses feedback from earlier discussions of the idea, the pull request will be accepted. The proposal will be assigned a proposal number as well as a core team member to manage the review. -* **Address feedback**: in general, and especially [during the review period][proposal-status], be responsive to questions and feedback about the proposal. +1. **Check prior proposals** + + Many ideas come up frequently, and may either be in active discussion on the forums, or may have been discussed already and have joined the [Commonly Rejected Proposals](commonly_proposed.md) list. Please [search the forums](https://forums.swift.org/search) for context before proposing something new. + +1. **Consider the goals of the upcoming Swift release** + + Each major Swift release is focused on a [specific set of goals](README.md) + described early in the release cycle. When proposing a change to + Swift, please consider how your proposal fits in with the larger goals + of the upcoming Swift release. Proposals that are clearly out of scope + for the upcoming Swift release will not be brought up for review. If you can't resist discussing a proposal that you know is out of scope, please include the tag `[Out of scope]` in the subject. + +1. **Socialize the idea** + + Propose a rough sketch of the idea in the ["pitches" section of the Swift forums](https://forums.swift.org/c/evolution/pitches), the problems it solves, what the solution looks like, etc., to gauge interest from the community. + +1. **Develop the proposal and implementation** + + 1. Expand the rough sketch into a formal proposal using the + [relevant proposal template](#proposal-templates). + 1. In the [swift-evolution repository][swift-evolution-repo], open a + [draft pull request][draft-pr] that adds your proposal to the appropriate + [proposal directory](#proposal-locations). + 1. Announce the pull request on the forums and edit the root post to link out to the pull request. + 1. Refine the formal proposal in the open as you receive further feedback on the forums or the pull request. + A ripe proposal is expected to address commentary from present and past + discussions of the idea. + + Meanwhile, start working on an implementation. + Prototyping an implementation and its uses *alongside* the formal proposal + is important because it helps to determine an adequate scope, ensure + technical feasibility, and validate that the proposal lives up to + its motivation. + + A pull request with a working implementation is *required* for the + proposal to be accepted for review. + Proposals that can ship as part of the [Standard Library Preview package][preview-package] + should be paired with a pull request against the [swift-evolution-staging repository][swift-evolution-staging]. + All other proposals should be paired with an implementation pull request + against the [main Swift repository](https://github.com/swiftlang/swift). + + The preview package can accept new types, new protocols, and extensions to + existing types and protocols that can be implemented without access to + standard library internals or other non-public features. + For more information about the kinds of changes that can be implemented in + the preview package, see [SE-0264](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0264-stdlib-preview-package.md). + +1. **Request a review** + + Once you have a working implementation and believe the proposal is sufficiently detailed and clear, mark the draft pull request in the [swift-evolution repository][swift-evolution-repo] as ready for review to indicate to the appropriate evolution workgroup that you would like the proposal to be reviewed. + +> [!IMPORTANT] +> In general, and especially [during the review period](#review-process), be responsive to questions and feedback about the proposal. + +[draft-pr]: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests#draft-pull-requests + +### Proposal templates + +When writing a formal proposal document, start with the most relevant template +given the primary subject of the proposal: -## Preparing an implementation +- Use the [Swift template][swift-template] for proposals concerning the Swift + language, compiler, or standard library. +- Use the [Swift Package Manager template][swiftpm-template] for proposals + related to SwiftPM, including the design of its package manifest files and + command-line tools. +- Use the [Swift Testing template][swift-testing-template] for proposals focused + on Swift Testing features or public interfaces. -When you are ready to request a review, a pull request with an implementation is required in addition to your proposal. Proposals that can ship as part of the [Standard Library Preview package][preview-package] should be paired with a pull request against the [swift-evolution-staging repository][swift-evolution-staging]. All other proposals should be paired with an implementation pull request against the [main Swift repository](https://github.com/apple/swift). +### Proposal locations -The preview package can accept new types, new protocols, and extensions to existing types and protocols that can be implemented without access to standard library internals or other non-public features. For more information about the kinds of changes that can be implemented in the preview package, see [SE-0264](https://github.com/apple/swift-evolution/blob/master/proposals/0264-stdlib-preview-package.md). +When opening a pull request to add a new proposal to the +[swift-evolution repository][swift-evolution-repo], place proposals which use +the [Swift][swift-template] or [Swift Package Manager][swiftpm-template] +templates in the top-level [proposals](/proposals) directory. Place proposals +which use the newer [Swift Testing template][swift-testing-template] in the +[proposals/testing](/proposals/testing) subdirectory. ## Review process The review process for a particular proposal begins when a member of -the core team accepts a pull request of a new or updated proposal into -the [swift-evolution repository][swift-evolution-repo]. That core team +the appropriate evolution workgroup accepts a pull request of a new or updated proposal into +the [swift-evolution repository][swift-evolution-repo]. That member becomes the *review manager* for the proposal. The proposal is assigned a proposal number (if it is a new proposal), and then enters the review queue. If your proposal's accompanying implementation takes the form of a package, the review manager will merge your pull request into a new branch in the [swift-evolution-staging repository][swift-evolution-staging]. @@ -82,38 +182,55 @@ reviews. To avoid delays, it is important that the proposal authors be available to answer questions, address feedback, and clarify their intent during the review period. -After the review has completed, the core team will make a decision on +After the review has completed, the managing evolution workgroup will make a decision on the proposal. The review manager is responsible for determining -consensus among the core team members, then reporting their decision +consensus among the workgroup members, then reporting their decision to the proposal authors and forums. The review manager will update the proposal's state in the [swift-evolution repository][swift-evolution-repo] to reflect that decision. ## Proposal states + +```mermaid +flowchart LR + %% + + %% Nodes: + 1{{"Awaiting
review"}} + 2{{"Scheduled
for review"}} + 3{"Active
review"} + 4["Returned
for revision"] + 5(["Withdrawn"]) + 6(["Rejected"]) + 7_8["Accepted
(with revisions)"] + 9[["Previewing"]] + 10(["Implemented"]) + + %% Links: + 1 ==> 3 ==> 7_8 ==> 10 + 1 -.-> 2 -.-> 3 -.-> 4 -.-> 5 & 1 + 3 -.-> 6 + 7_8 -.-> 9 -.-> 10 +``` + A given proposal can be in one of several states: * **Awaiting review**: The proposal is awaiting review. Once known, the dates for the actual review will be placed in the proposal document. When the review period begins, the review manager will update the state to *Active review*. -* **Scheduled for review (FULL_MONTH_NAME DAY...FULL_MONTH_NAME DAY)**: The public review of the proposal +* **Scheduled for review (...)**: The public review of the proposal in the [Swift forums][proposal-reviews] has been scheduled for the specified date range. -* **Active review (FULL_MONTH_NAME DAY...FULL_MONTH_NAME DAY)**: The proposal is undergoing public review +* **Active review (...)**: The proposal is undergoing public review in the [Swift forums][proposal-reviews]. The review will continue through the specified date range. * **Returned for revision**: The proposal has been returned from review for additional revision to the current draft. * **Withdrawn**: The proposal has been withdrawn by the original submitter. -* **Deferred**: Consideration of the proposal has been deferred - because it does not meet the [goals of the upcoming major Swift - release](README.md). Deferred proposals will be reconsidered when - scoping the next major Swift release. * **Rejected**: The proposal has been considered and rejected. -* **Accepted (YYYY-MM-DD)**: - The proposal has been accepted (on the specified date) and is either awaiting +* **Accepted**: The proposal has been accepted and is either awaiting implementation or is actively being implemented. -* **Accepted with revisions (YYYY-MM-DD)**: - The proposal has been accepted (on the specified date), +* **Accepted with revisions**: The proposal has been accepted, contingent upon the inclusion of one or more revisions. * **Previewing**: The proposal has been accepted and is available for preview in the [Standard Library Preview package][preview-package]. @@ -122,19 +239,29 @@ A given proposal can be in one of several states: If the proposal's implementation spans multiple version numbers, write the version number for which the implementation will be complete. -[swift-evolution-repo]: https://github.com/apple/swift-evolution "Swift evolution repository" -[swift-evolution-staging]: https://github.com/apple/swift-evolution-staging "Swift evolution staging repository" -[proposal-reviews]: https://forums.swift.org/c/evolution/proposal-reviews "'Proposal reviews' category of the Swift forums" -[proposal-status]: https://apple.github.io/swift-evolution/ +[swift-evolution-repo]: https://github.com/swiftlang/swift-evolution "Swift evolution repository" +[swift-evolution-staging]: https://github.com/swiftlang/swift-evolution-staging "Swift evolution staging repository" +[proposal-reviews]: https://forums.swift.org/c/evolution/proposal-reviews "'Proposal reviews' subcategory of the Swift forums" +[status-page]: https://www.swift.org/swift-evolution [preview-package]: https://github.com/apple/swift-standard-library-preview/ +[language-steering-group]: https://www.swift.org/language-steering-group +[platform-steering-group]: https://www.swift.org/platform-steering-group +[testing-workgroup]: https://www.swift.org/testing-workgroup "Testing Workgroup page on Swift.org" +[swift-template]: proposal-templates/0000-swift-template.md "Swift proposal template" +[swiftpm-template]: proposal-templates/0000-swiftpm-template.md "Swift Package Manager proposal template" +[swift-testing-template]: proposal-templates/0000-swift-testing-template.md "Swift Testing proposal template" ## Review announcement -When a proposal enters review, a new topic will be posted to the ["Proposal Reviews" section of the Swift forums][proposal-reviews] -using the following template: +When a proposal enters review, a new topic will be posted to the +["Proposal Reviews" subcategory of the Swift forums][proposal-reviews] using the +relevant announcement template below: --- +
+Swift language, compiler, and standard library + Hello Swift community, The review of "\<\>" begins now and runs through \<\>. The proposal is available here: Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager. When emailing the review manager directly, please keep the proposal link at the top of the message. +##### Trying it out + +If you'd like to try this proposal out, you can [download a toolchain supporting it here](). You will need to add `-enable-experimental-feature FLAGNAME` to your build flags. \<\\> + ##### What goes into a review? The goal of the review process is to improve the proposal under review @@ -162,7 +293,7 @@ answer in your review: More information about the Swift evolution process is available at -> +> Thank you, @@ -170,4 +301,72 @@ Thank you, Review Manager ---- +
+ +
+Swift Testing public interfaces and features + +Hello Swift community, + +The review of "\<\>" begins now and runs through \<\>. +The proposal is available here: + +> https://linkToProposal + +Reviews are an important part of the Swift evolution process. All review +feedback should be either on this forum thread or, if you would like to keep +your feedback private, directly to the review manager. When emailing the review +manager directly, please keep the proposal link at the top of the message. + +##### Trying it out + +To try this feature out, add a dependency to the `main` branch of +`swift-testing` to your package: + +```swift +dependencies: [ + ... + .package(url: "https://github.com/swiftlang/swift-testing.git", branch: "main"), +] +``` + +Then, add a target dependency to your test target: + +```swift +.testTarget( + ... + dependencies: [ + ... + .product(name: "Testing", package: "swift-testing"), + ] +``` + +Finally, import Swift Testing using `@_spi(Experimental) import Testing`. + +##### What goes into a review? + +The goal of the review process is to improve the proposal under review through +constructive criticism and, eventually, determine the direction of Swift. When +writing your review, here are some questions you might want to answer in your +review: + +* What is your evaluation of the proposal? +* Is the problem being addressed significant enough to warrant a change to Swift + Testing? +* Does this proposal fit well with the feel and direction of Swift Testing? +* If you have used other languages or libraries with a similar feature, how do + you feel that this proposal compares to those? +* How much effort did you put into your review? A glance, a quick reading, or an + in-depth study? + +More information about the Swift evolution process is available at + +> https://github.com/swiftlang/swift-evolution/blob/main/process.md + +Thank you, + +-\<\> + +Review Manager + +
diff --git a/proposal-templates/0000-swift-template.md b/proposal-templates/0000-swift-template.md index 67245d08f3..c73d88ff67 100644 --- a/proposal-templates/0000-swift-template.md +++ b/proposal-templates/0000-swift-template.md @@ -3,15 +3,91 @@ * Proposal: [SE-NNNN](NNNN-filename.md) * Authors: [Author 1](https://github.com/swiftdev), [Author 2](https://github.com/swiftdev) * Review Manager: TBD -* Status: **Awaiting implementation** +* Status: **Awaiting implementation** or **Awaiting review** +* Vision: *if applicable* [Vision Name](https://github.com/swiftlang/swift-evolution/visions/NNNNN.md) +* Roadmap: *if applicable* [Roadmap Name](https://forums.swift.org/...) +* Bug: *if applicable* [swiftlang/swift#NNNNN](https://github.com/swiftlang/swift/issues/NNNNN) +* Implementation: [swiftlang/swift#NNNNN](https://github.com/swiftlang/swift/pull/NNNNN) or [swiftlang/swift-evolution-staging#NNNNN](https://github.com/swiftlang/swift-evolution-staging/pull/NNNNN) +* Upcoming Feature Flag: *if applicable* `MyFeatureName` +* Previous Proposal: *if applicable* [SE-XXXX](XXXX-filename.md) +* Previous Revision: *if applicable* [1](https://github.com/swiftlang/swift-evolution/blob/...commit-ID.../proposals/NNNN-filename.md) +* Review: ([pitch](https://forums.swift.org/...)) -*During the review process, add the following fields as needed:* +When filling out this template, you should delete or replace all of +the text except for the section headers and the header fields above. +For example, you should delete everything from this paragraph down to +the Introduction section below. -* Implementation: [apple/swift#NNNNN](https://github.com/apple/swift/pull/NNNNN) or [apple/swift-evolution-staging#NNNNN](https://github.com/apple/swift-evolution-staging/pull/NNNNN) -* Decision Notes: [Rationale](https://forums.swift.org/), [Additional Commentary](https://forums.swift.org/) -* Bugs: [SR-NNNN](https://bugs.swift.org/browse/SR-NNNN), [SR-MMMM](https://bugs.swift.org/browse/SR-MMMM) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/...commit-ID.../proposals/NNNN-filename.md) -* Previous Proposal: [SE-XXXX](XXXX-filename.md) +As a proposal author, you should fill out all of the header fields +except `Review Manager`. The review manager will set that field and +change several others as part of initiating the review. Delete any +header fields marked *if applicable* that are not applicable to your +proposal. + +When sharing a link to the proposal while it is still a PR, be sure +to share a live link to the proposal, not an exact commit, so that +readers will always see the latest version when you make changes. +On GitHub, you can find this link by browsing the PR branch: from the +PR page, click the "username wants to merge ... from username:my-branch-name" +link and find the proposal file in that branch. + +`Status` should reflect the current implementation status while the +proposal is still a PR. The proposal cannot be reviewed until an +implementation is available, but early readers should see the correct +status. + +`Vision` should link to the [vision document](https://forums.swift.org/t/the-role-of-vision-documents-in-swift-evolution/62101) +for this proposal, if it is part of a vision. Most proposals are not +part of a vision. If a vision has been written but not yet accepted, +link to the discussion thread for the vision. + +`Roadmap` should link to the discussion thread for the roadmap for +this proposal, if applicable. When a complex feature is broken down +into several closely-related proposals to make evolution review easier +and more focused, it's helpful to make a forum post explaining what's +going on and detailing how the proposals are expected to be submitted +to review. That post is called a "roadmap". Most proposals don't need +roadmaps, but if this proposal was part of one, this field should link +to it. + +`Bug` should be used when this proposal is fixing a bug with significant +discussion in the bug report. It is not necessary to link bugs that do +not contain significant discussion or that merely duplicate discussion +linked somewhere else. Do not link bugs from private bug trackers. + +`Implementation` should link to the PR(s) implementing the feature. +If the proposal has not been implemented yet, or if it simply codifies +existing behavior, just say that. If the implementation has already +been committed to the main branch (as an experimental feature), say +that and specify the experimental feature flag. If the implementation +is spread across multiple PRs, just link to the most important ones. + +`Upcoming Feature Flag` should be the feature name used to identify this +feature under [SE-0362](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md#proposals-define-their-own-feature-identifier). +Not all proposals need an upcoming feature flag. You should think about +whether one would be useful for your proposal as part of filling this +field out. + +`Previous Proposal` should be used when there is a specific line of +succession between this proposal and another proposal. For example, +this proposal might have been removed from a previous proposal so +that it can be reviewed separately, or this proposal might supersede +a previous proposal in some way that was felt to exceed the scope of +a "revision". Include text briefly explaining the relationship, +such as "Supersedes SE-1234" or "Extracted from SE-01234". If possible, +link to a post explaining the relationship, such as a review decision +that asked for part of the proposal to be split off. Otherwise, you +can just link to the previous proposal. + +`Previous Revision` should be added after a major substantive revision +of a proposal that has undergone review. It links to the previously +reviewed revision. It is not necessary to add or update this field +after minor editorial changes. + +`Review` is a history of all discussion threads about this proposal, +in chronological order. Use these standardized link names: `pitch` +`review` `revision` `acceptance` `rejection`. If there are multiple +such threads, spell the ordinal out: `first pitch` `second review` etc. ## Introduction @@ -19,8 +95,6 @@ A short description of what the feature is. Try to keep it to a single-paragraph "elevator pitch" so the reader understands what problem this proposal is addressing. -Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/) - ## Motivation Describe the problems that this proposal seeks to address. If the @@ -36,6 +110,10 @@ Describe your solution to the problem. Provide examples and describe how they work. Show how your solution is better than current workarounds: is it cleaner, safer, or more efficient? +This section doesn't have to be comprehensive. Focus on the most +important parts of the proposal and make arguments about why the +proposal is better than the status quo. + ## Detailed design Describe the design of the solution in detail. If it involves new @@ -47,58 +125,162 @@ reasonably implement the feature. ## Source compatibility -Relative to the Swift 3 evolution process, the source compatibility -requirements for Swift 4 are *much* more stringent: we should only -break source compatibility if the Swift 3 constructs were actively -harmful in some way, the volume of affected Swift 3 code is relatively -small, and we can provide source compatibility (in Swift 3 -compatibility mode) and migration. - -Will existing correct Swift 3 or Swift 4 applications stop compiling -due to this change? Will applications still compile but produce -different behavior than they used to? If "yes" to either of these, is -it possible for the Swift 4 compiler to accept the old syntax in its -Swift 3 compatibility mode? Is it possible to automatically migrate -from the old syntax to the new syntax? Can Swift applications be -written in a common subset that works both with Swift 3 and Swift 4 to -aid in migration? - -## Effect on ABI stability - -Does the proposal change the ABI of existing language features? The -ABI comprises all aspects of the code generation model and interaction -with the Swift runtime, including such things as calling conventions, -the layout of data types, and the behavior of dynamic features in the -language (reflection, dynamic dispatch, dynamic casting via `as?`, -etc.). Purely syntactic changes rarely change existing ABI. Additive -features may extend the ABI but, unless they extend some fundamental -runtime behavior (such as the aforementioned dynamic features), they -won't change the existing ABI. - -Features that don't change the existing ABI are considered out of -scope for [Swift 4 stage 1](README.md). However, additive features -that would reshape the standard library in a way that changes its ABI, -such as [where clauses for associated -types](https://github.com/apple/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md), -can be in scope. If this proposal could be used to improve the -standard library in ways that would affect its ABI, describe them -here. - -## Effect on API resilience - -API resilience describes the changes one can make to a public API -without breaking its ABI. Does this proposal introduce features that -would become part of a public API? If so, what kinds of changes can be -made without breaking ABI? Can this feature be added/removed without -breaking ABI? For more information about the resilience model, see the -[library evolution -document](https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst) -in the Swift repository. +Describe the impact of this proposal on source compatibility. As a +general rule, all else being equal, Swift code that worked in previous +releases of the tools should work in new releases. That means both that +it should continue to build and that it should continue to behave +dynamically the same as it did before. Changes that cannot satisfy +this must be opt-in, generally by requiring a new language mode. + +This is not an absolute guarantee, and the Language Workgroup will +consider intentional compatibility breaks if their negative impact +can be shown to be small and the current behavior is causing +substantial problems in practice. + +For proposals that affect parsing, consider whether existing valid +code might parse differently under the proposal. Does the proposal +reserve new keywords that can no longer be used as identifiers? + +For proposals that affect type checking, consider whether existing valid +code might type-check differently under the proposal. Does it add new +conversions that might make more overload candidates viable? Does it +change how names are looked up in existing code? Does it make +type-checking more expensive in ways that might run into implementation +limits more often? + +For proposals that affect the standard library, consider the impact on +existing clients. If clients provide a similar API, will type-checking +find the right one? If the feature overloads an existing API, is it +problematic that existing users of that API might start resolving to +the new API? + +## ABI compatibility + +Describe the impact on ABI compatibility. As a general rule, the ABI +of existing code must not change between tools releases or language +modes. This rule does not apply as often as source compatibility, but +it is much stricter, and the Language Workgroup generally cannot allow +exceptions. + +The ABI encompasses all aspects of how code is generated for the +language, how that code interacts with other code that has been +compiled separately, and how that code interacts with the Swift +runtime library. Most ABI changes center around interactions with +specific declarations. Proposals that do not affect how code is +generated to interact with an external declaration usually do not +have ABI impact. + +For proposals that affect general code generation rules, consider +the impact on code that's already been compiled. Does the proposal +affect declarations that haven't explicitly adopted it, and if so, +does it change ABI details such as symbol names or conventions +around their use? Will existing code change its dynamic behavior +when running against a new version of the language runtime or +standard library? Conversely, will code compiled in the new way +continue to run on old versions of the language runtime or standard +library? + +For proposals that affect the standard library, consider the impact +on any existing declarations. As above, does the proposal change symbol +names, conventions, or dynamic behavior? Will newly-compiled code work +on old library versions, and will new library versions work with +previously-compiled code? + +This section will often end up very short. A proposal that just +adds a new standard library feature, for example, will usually +say either "This proposal is purely an extension of the ABI of the +standard library and does not change any existing features" or +"This proposal is purely an extension of the standard library which +can be implemented without any ABI support" (whichever applies). +Nonetheless, it is important to demonstrate that you've considered +the ABI implications. + +If the design of the feature was significantly constrained by +the need to maintain ABI compatibility, this section is a reasonable +place to discuss that. + +## Implications on adoption + +The compatibility sections above are focused on the direct impact +of the proposal on existing code. In this section, describe issues +that intentional adopters of the proposal should be aware of. + +For proposals that add features to the language or standard library, +consider whether the features require ABI support. Will adopters need +a new version of the library or language runtime? Be conservative: if +you're hoping to support back-deployment, but you can't guarantee it +at the time of review, just say that the feature requires a new +version. + +Consider also the impact on library adopters of those features. Can +adopting this feature in a library break source or ABI compatibility +for users of the library? If a library adopts the feature, can it +be *un*-adopted later without breaking source or ABI compatibility? +Will package authors be able to selectively adopt this feature depending +on the tools version available, or will it require bumping the minimum +tools version required by the package? + +If there are no concerns to raise in this section, leave it in with +text like "This feature can be freely adopted and un-adopted in source +code with no deployment constraints and without affecting source or ABI +compatibility." + +## Future directions + +Describe any interesting proposals that could build on this proposal +in the future. This is especially important when these future +directions inform the design of the proposal, for example by making +sure an attribute encodes enough information to be used for other +purposes. + +The rest of the proposal should generally not talk about future +directions except by referring to this section. It is important +not to confuse reviewers about what is covered by this specific +proposal. If there's a larger vision that needs to be explained +in order to understand this proposal, consider starting a discussion +thread on the forums to capture your broader thoughts. + +Avoid making affirmative statements in this section, such as "we +will" or even "we should". Describe the proposals neutrally as +possibilities to be considered in the future. + +Consider whether any of these future directions should really just +be part of the current proposal. It's important to make focused, +self-contained proposals that can be incrementally implemented and +reviewed, but it's also good when proposals feel "complete" rather +than leaving significant gaps in their design. For example, when +[SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md) +introduced the `@inlinable` attribute, it also included the +`@usableFromInline` attribute so that declarations used in inlinable +functions didn't have to be `public`. This was a relatively small +addition to the proposal which avoided creating a serious usability +problem for many adopters of `@inlinable`. ## Alternatives considered -Describe alternative approaches to addressing the same problem, and -why you chose this approach instead. +Describe alternative approaches to addressing the same problem. +This is an important part of most proposal documents. Reviewers +are often familiar with other approaches prior to review and may +have reasons to prefer them. This section is your first opportunity +to try to convince them that your approach is the right one, and +even if you don't fully succeed, you can help set the terms of the +conversation and make the review a much more productive exchange +of ideas. + +You should be fair about other proposals, but you do not have to +be neutral; after all, you are specifically proposing something +else. Describe any advantages these alternatives might have, but +also be sure to explain the disadvantages that led you to prefer +the approach in this proposal. + +You should update this section during the pitch phase to discuss +any particularly interesting alternatives raised by the community. +You do not need to list every idea raised during the pitch, just +the ones you think raise points that are worth discussing. Of course, +if you decide the alternative is more compelling than what's in +the current proposal, you should change the main proposal; be sure +to then discuss your previous proposal in this section and explain +why the new idea is better. ## Acknowledgments @@ -106,3 +288,6 @@ If significant changes or improvements suggested by members of the community were incorporated into the proposal as it developed, take a moment here to thank them for their contributions. Swift evolution is a collaborative process, and everyone's input should receive recognition! + +Generally, you should not acknowledge anyone who is listed as a +co-author or as the review manager. diff --git a/proposal-templates/0000-swift-testing-template.md b/proposal-templates/0000-swift-testing-template.md new file mode 100644 index 0000000000..642ac7bd53 --- /dev/null +++ b/proposal-templates/0000-swift-testing-template.md @@ -0,0 +1,188 @@ +# Swift Testing Feature name + +* Proposal: [ST-NNNN](NNNN-filename.md) +* Authors: [Author 1](https://github.com/author1), [Author 2](https://github.com/author2) +* Review Manager: TBD +* Status: **Awaiting implementation** or **Awaiting review** +* Bug: _if applicable_ [swiftlang/swift-testing#NNNNN](https://github.com/swiftlang/swift-testing/issues/NNNNN) +* Implementation: [swiftlang/swift-testing#NNNNN](https://github.com/swiftlang/swift-testing/pull/NNNNN) +* Previous Proposal: _if applicable_ [ST-XXXX](XXXX-filename.md) +* Previous Revision: _if applicable_ [1](https://github.com/swiftlang/swift-evolution/blob/...commit-ID.../proposals/testing/NNNN-filename.md) +* Review: ([pitch](https://forums.swift.org/...)) + +When filling out this template, you should delete or replace all of the text +except for the section headers and the header fields above. For example, you +should delete everything from this paragraph down to the Introduction section +below. + +As a proposal author, you should fill out all of the header fields except +`Review Manager`. The review manager will set that field and change several +others as part of initiating the review. Delete any header fields marked _if +applicable_ that are not applicable to your proposal. + +When sharing a link to the proposal while it is still a PR, be sure to share a +live link to the proposal, not an exact commit, so that readers will always see +the latest version when you make changes. On GitHub, you can find this link by +browsing the PR branch: from the PR page, click the "username wants to merge ... +from username:my-branch-name" link and find the proposal file in that branch. + +`Status` should reflect the current implementation status while the proposal is +still a PR. The proposal cannot be reviewed until an implementation is available, +but early readers should see the correct status. + +`Bug` should be used when this proposal is fixing a bug with significant +discussion in the bug report. It is not necessary to link bugs that do not +contain significant discussion or that merely duplicate discussion linked +somewhere else. Do not link bugs from private bug trackers. + +`Implementation` should link to the PR(s) implementing the feature. If the +proposal has not been implemented yet, or if it simply codifies existing +behavior, just say that. If the implementation has already been committed to the +main branch (as an experimental feature or SPI), mention that. If the +implementation is spread across multiple PRs, just link to the most important +ones. + +`Previous Proposal` should be used when there is a specific line of succession +between this proposal and another proposal. For example, this proposal might +have been removed from a previous proposal so that it can be reviewed separately, +or this proposal might supersede a previous proposal in some way that was felt +to exceed the scope of a "revision". Include text briefly explaining the +relationship, such as "Supersedes ST-1234" or "Extracted from ST-01234". If +possible, link to a post explaining the relationship, such as a review decision +that asked for part of the proposal to be split off. Otherwise, you can just +link to the previous proposal. + +`Previous Revision` should be added after a major substantive revision of a +proposal that has undergone review. It links to the previously reviewed revision. +It is not necessary to add or update this field after minor editorial changes. + +`Review` is a history of all discussion threads about this proposal, in +chronological order. Use these standardized link names: `pitch` `review` +`revision` `acceptance` `rejection`. If there are multiple such threads, spell +the ordinal out: `first pitch` `second review` etc. + +## Introduction + +A short description of what the feature is. Try to keep it to a single-paragraph +"elevator pitch" so the reader understands what problem this proposal is +addressing. + +## Motivation + +Describe the problems that this proposal seeks to address. If the problem is +that some common pattern is currently hard to express, show how one can +currently get a similar effect and describe its drawbacks. If it's completely +new functionality that cannot be emulated, motivate why this new functionality +would help Swift developers test their code more effectively. + +## Proposed solution + +Describe your solution to the problem. Provide examples and describe how they +work. Show how your solution is better than current workarounds: is it cleaner, +safer, or more efficient? + +This section doesn't have to be comprehensive. Focus on the most important parts +of the proposal and make arguments about why the proposal is better than the +status quo. + +## Detailed design + +Describe the design of the solution in detail. If it includes new API, show the +full API and its documentation comments detailing what it does. If it involves +new macro logic, describe the behavior changes and include a succinct example of +the additions or modifications to the macro expansion code. The detail in this +section should be sufficient for someone who is *not* one of the authors to be +able to reasonably implement the feature. + +## Source compatibility + +Describe the impact of this proposal on source compatibility. As a general rule, +all else being equal, test code that worked in previous releases of the testing +library should work in new releases. That means both that it should continue to +build and that it should continue to behave dynamically the same as it did +before. + +This is not an absolute guarantee, and the testing library administrators will +consider intentional compatibility breaks if their negative impact can be shown +to be small and the current behavior is causing substantial problems in practice. + +For proposals that affect testing library API, consider the impact on existing +clients. If clients provide a similar API, will type-checking find the right one? +If the feature overloads an existing API, is it problematic that existing users +of that API might start resolving to the new API? + +## Integration with supporting tools + +In this section, describe how this proposal affects tools which integrate with +the testing library. Some features depend on supporting tools gaining awareness +of the new feature for users to realize new benefits. Other features do not +strictly require integration but bring improvement opportunities which are worth +considering. Use this section to discuss any impact on tools. + +This section does need not to include details of how this proposal may be +integrated with _specific_ tools, but it should consider the general ways that +tools might support this feature and note any accompanying SPI intended for +tools which are included in the implementation. Note that tools may evolve +independently and have differing release schedules than the testing library, so +special care should be taken to ensure compatibility across versions according +to the needs of each tool. + +## Future directions + +Describe any interesting proposals that could build on this proposal in the +future. This is especially important when these future directions inform the +design of the proposal, for example by making sure an interface meant for tools +integration can be extended to include additional information. + +The rest of the proposal should generally not talk about future directions +except by referring to this section. It is important not to confuse reviewers +about what is covered by this specific proposal. If there's a larger vision that +needs to be explained in order to understand this proposal, consider starting a +discussion thread on the forums to capture your broader thoughts. + +Avoid making affirmative statements in this section, such as "we will" or even +"we should". Describe the proposals neutrally as possibilities to be considered +in the future. + +Consider whether any of these future directions should really just be part of +the current proposal. It's important to make focused, self-contained proposals +that can be incrementally implemented and reviewed, but it's also good when +proposals feel "complete" rather than leaving significant gaps in their design. +An an example from the Swift project, when +[SE-0193](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0193-cross-module-inlining-and-specialization.md) +introduced the `@inlinable` attribute, it also included the `@usableFromInline` +attribute so that declarations used in inlinable functions didn't have to be +`public`. This was a relatively small addition to the proposal which avoided +creating a serious usability problem for many adopters of `@inlinable`. + +## Alternatives considered + +Describe alternative approaches to addressing the same problem. This is an +important part of most proposal documents. Reviewers are often familiar with +other approaches prior to review and may have reasons to prefer them. This +section is your first opportunity to try to convince them that your approach is +the right one, and even if you don't fully succeed, you can help set the terms +of the conversation and make the review a much more productive exchange of ideas. + +You should be fair about other proposals, but you do not have to be neutral; +after all, you are specifically proposing something else. Describe any +advantages these alternatives might have, but also be sure to explain the +disadvantages that led you to prefer the approach in this proposal. + +You should update this section during the pitch phase to discuss any +particularly interesting alternatives raised by the community. You do not need +to list every idea raised during the pitch, just the ones you think raise points +that are worth discussing. Of course, if you decide the alternative is more +compelling than what's in the current proposal, you should change the main +proposal; be sure to then discuss your previous proposal in this section and +explain why the new idea is better. + +## Acknowledgments + +If significant changes or improvements suggested by members of the community +were incorporated into the proposal as it developed, take a moment here to thank +them for their contributions. This is a collaborative process, and everyone's +input should receive recognition! + +Generally, you should not acknowledge anyone who is listed as a co-author or as +the review manager. diff --git a/proposal-templates/0000-swiftpm-template.md b/proposal-templates/0000-swiftpm-template.md index 57fd0b27cf..bd65132b9e 100644 --- a/proposal-templates/0000-swiftpm-template.md +++ b/proposal-templates/0000-swiftpm-template.md @@ -7,10 +7,10 @@ *During the review process, add the following fields as needed:* -* Implementation: [apple/swift-package-manager#NNNNN](https://github.com/apple/swift-package-manager/pull/NNNNN) +* Implementation: [swiftlang/swift-package-manager#NNNNN](https://github.com/swiftlang/swift-package-manager/pull/NNNNN) * Decision Notes: [Rationale](https://forums.swift.org/), [Additional Commentary](https://forums.swift.org/) * Bugs: [SR-NNNN](https://bugs.swift.org/browse/SR-NNNN), [SR-MMMM](https://bugs.swift.org/browse/SR-MMMM) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/...commit-ID.../proposals/NNNN-filename.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/...commit-ID.../proposals/NNNN-filename.md) * Previous Proposal: [SE-XXXX](XXXX-filename.md) ## Introduction diff --git a/proposals/0002-remove-currying.md b/proposals/0002-remove-currying.md index 490a90e235..9761465a5c 100644 --- a/proposals/0002-remove-currying.md +++ b/proposals/0002-remove-currying.md @@ -2,7 +2,7 @@ * Proposal: [SE-0002](0002-remove-currying.md) * Author: [Joe Groff](https://github.com/jckarter) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Implementation: [apple/swift@983a674](https://github.com/apple/swift/commit/983a674e0ca35a85532d70a3eb61e71a6d024108) ## Introduction diff --git a/proposals/0003-remove-var-parameters.md b/proposals/0003-remove-var-parameters.md index a36158f05b..a78dc80b0c 100644 --- a/proposals/0003-remove-var-parameters.md +++ b/proposals/0003-remove-var-parameters.md @@ -1,9 +1,9 @@ # Removing `var` from Function Parameters * Proposal: [SE-0003](0003-remove-var-parameters.md) -* Author: [David Farler](https://github.com/bitjammer) +* Author: [Ashley Garland](https://github.com/bitjammer) * Review Manager: [Joe Pamer](https://github.com/jopamer) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/se-0003-removing-var-from-function-parameters-and-pattern-matching/1230) * Implementation: [apple/swift@8a5ed40](https://github.com/apple/swift/commit/8a5ed405bf1f92ec3fc97fa46e52528d2e8d67d9) @@ -97,7 +97,7 @@ this language change. This proposal originally included removal of `var` bindings for all refutable patterns as well as function parameters. -[Original SE-0003 Proposal](https://github.com/apple/swift-evolution/blob/8cd734260bc60d6d49dbfb48de5632e63bf200cc/proposals/0003-remove-var-parameters-patterns.md) +[Original SE-0003 Proposal](https://github.com/swiftlang/swift-evolution/blob/8cd734260bc60d6d49dbfb48de5632e63bf200cc/proposals/0003-remove-var-parameters-patterns.md) Removal of `var` from refutable patterns was reconsidered due to the burden it placed on valid mutation patterns already in use in Swift 2 diff --git a/proposals/0004-remove-pre-post-inc-decrement.md b/proposals/0004-remove-pre-post-inc-decrement.md index 56386e0d10..c33507a664 100644 --- a/proposals/0004-remove-pre-post-inc-decrement.md +++ b/proposals/0004-remove-pre-post-inc-decrement.md @@ -2,7 +2,7 @@ * Proposal: [SE-0004](0004-remove-pre-post-inc-decrement.md) * Author: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Implementation: [apple/swift@8e12008](https://github.com/apple/swift/commit/8e12008d2b34a605f8766310f53d5668f3d50955) ## Introduction diff --git a/proposals/0005-objective-c-name-translation.md b/proposals/0005-objective-c-name-translation.md index 0ca0809d7a..9a494db4c3 100644 --- a/proposals/0005-objective-c-name-translation.md +++ b/proposals/0005-objective-c-name-translation.md @@ -3,7 +3,7 @@ * Proposal: [SE-0005](0005-objective-c-name-translation.md) * Authors: [Doug Gregor](https://github.com/DougGregor), [Dave Abrahams](https://github.com/dabrahams) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modification-se-0005-better-translation-of-objective-c-apis-into-swift/1668) ## Reviewer notes diff --git a/proposals/0006-apply-api-guidelines-to-the-standard-library.md b/proposals/0006-apply-api-guidelines-to-the-standard-library.md index 86ef04eb88..76349a2287 100644 --- a/proposals/0006-apply-api-guidelines-to-the-standard-library.md +++ b/proposals/0006-apply-api-guidelines-to-the-standard-library.md @@ -3,7 +3,7 @@ * Proposal: [SE-0006](0006-apply-api-guidelines-to-the-standard-library.md) * Authors: [Dave Abrahams](https://github.com/dabrahams), [Dmitri Gribenko](https://github.com/gribozavr), [Maxim Moiseev](https://github.com/moiseev) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0006-apply-api-guidelines-to-the-standard-library/1667) ## Reviewer notes @@ -1144,7 +1144,7 @@ public struct OpaquePointer : ... { - allowedCharacters: NSCharacterSet - ) -> String? + public func addingPercentEncoding( -+ withAllowedCharaters allowedCharacters: NSCharacterSet ++ withAllowedCharacters allowedCharacters: NSCharacterSet + ) -> String? - public func stringByAddingPercentEscapesUsingEncoding( diff --git a/proposals/0007-remove-c-style-for-loops.md b/proposals/0007-remove-c-style-for-loops.md index 77149118b9..1bef3ab55a 100644 --- a/proposals/0007-remove-c-style-for-loops.md +++ b/proposals/0007-remove-c-style-for-loops.md @@ -3,7 +3,7 @@ * Proposal: [SE-0007](0007-remove-c-style-for-loops.md) * Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0007-remove-c-style-for-loops-with-conditions-and-incrementers/512) * Bugs: [SR-226](https://bugs.swift.org/browse/SR-226), [SR-227](https://bugs.swift.org/browse/SR-227) diff --git a/proposals/0008-lazy-flatmap-for-optionals.md b/proposals/0008-lazy-flatmap-for-optionals.md index 90e7e9e3bc..2abda3ceee 100644 --- a/proposals/0008-lazy-flatmap-for-optionals.md +++ b/proposals/0008-lazy-flatmap-for-optionals.md @@ -3,7 +3,7 @@ * Proposal: [SE-0008](0008-lazy-flatmap-for-optionals.md) * Author: [Oisin Kidney](https://github.com/oisdk) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0008-add-a-lazy-flatmap-for-sequences-of-optionals/748) * Bug: [SR-361](https://bugs.swift.org/browse/SR-361) diff --git a/proposals/0013-remove-partial-application-super.md b/proposals/0013-remove-partial-application-super.md index ae3132c407..40cfaa459c 100644 --- a/proposals/0013-remove-partial-application-super.md +++ b/proposals/0013-remove-partial-application-super.md @@ -1,7 +1,7 @@ # Remove Partial Application of Non-Final Super Methods (Swift 2.2) * Proposal: [SE-0013](0013-remove-partial-application-super.md) -* Author: [David Farler](https://github.com/bitjammer) +* Author: [Ashley Garland](https://github.com/bitjammer) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0013-remove-partial-application-of-non-final-super-methods/1157) diff --git a/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md b/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md index 22518458dc..ed282b7e13 100644 --- a/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md +++ b/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md @@ -3,10 +3,10 @@ * Proposal: [SE-0016](0016-initializers-for-converting-unsafe-pointers-to-ints.md) * Author: [Michael Buckley](https://github.com/MichaelBuckley) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0016-adding-initializers-to-int-and-uint-to-convert-from-unsafepointer-and-unsafemutablepointer/2005) * Bug: [SR-1115](https://bugs.swift.org/browse/SR-1115) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/ae2d7c24fff7cbdff754d9a4339e4fb02df5c690/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/ae2d7c24fff7cbdff754d9a4339e4fb02df5c690/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md) ## Introduction diff --git a/proposals/0017-convert-unmanaged-to-use-unsafepointer.md b/proposals/0017-convert-unmanaged-to-use-unsafepointer.md index 7799dec5b8..cc2bb68860 100644 --- a/proposals/0017-convert-unmanaged-to-use-unsafepointer.md +++ b/proposals/0017-convert-unmanaged-to-use-unsafepointer.md @@ -3,7 +3,7 @@ * Proposal: [SE-0017](0017-convert-unmanaged-to-use-unsafepointer.md) * Author: [Jacob Bandes-Storch](https://github.com/jtbandes) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0017-change-unmanaged-to-use-unsafepointer/2461) * Bug: [SR-1485](https://bugs.swift.org/browse/SR-1485) diff --git a/proposals/0018-flexible-memberwise-initialization.md b/proposals/0018-flexible-memberwise-initialization.md index 42432c781a..2bddbf171c 100644 --- a/proposals/0018-flexible-memberwise-initialization.md +++ b/proposals/0018-flexible-memberwise-initialization.md @@ -3,15 +3,13 @@ * Proposal: [SE-0018](0018-flexible-memberwise-initialization.md) * Author: [Matthew Johnson](https://github.com/anandabits) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Deferred** -* Decision Notes: [Rationale](https://forums.swift.org/t/review-se-0018-flexible-memberwise-initialization/939/22) +* Status: **Returned for revision** +* Review: ([pitch](https://forums.swift.org/t/proposal-draft-flexible-memberwise-initialization/698)) ([review](https://forums.swift.org/t/review-se-0018-flexible-memberwise-initialization/939)) ([deferral](https://forums.swift.org/t/review-se-0018-flexible-memberwise-initialization/939/22)) ([return for revision](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction The Swift compiler is currently able to generate a memberwise initializer for use in some circumstances, however there are currently many limitations to this. This proposal builds on the idea of a compiler generated memberwise initializer, making the capability available to any initializer that opts in. -Swift-evolution thread: [Proposal Draft: flexible memberwise initialization](https://forums.swift.org/t/proposal-draft-flexible-memberwise-initialization/698) - ## Motivation When designing initializers for a type we are currently faced with the unfortunate fact that the more flexibility we wish to offer users the more boilerplate we are required to write and maintain. We usually end up with more boilerplate and less flexibility than desired. There have been various strategies employed to mitigate this problem, including: @@ -22,7 +20,7 @@ When designing initializers for a type we are currently faced with the unfortuna Underlying this problem is the fact that initialization scales with M x N complexity (M members, N initializers). We need as much help from the compiler as we can get! -Flexible and concise initialization for both type authors and consumers will encourages using immutability where possible and removes the need for boilerplate from the concerns one must consider when designing the intializers for a type. +Flexible and concise initialization for both type authors and consumers will encourages using immutability where possible and removes the need for boilerplate from the concerns one must consider when designing the initializers for a type. Quoting [Chris Lattner](https://forums.swift.org/t/proposal-helpers-for-initializing-properties-of-same-name-as-parameters/129/8): @@ -33,9 +31,9 @@ Quoting [Chris Lattner](https://forums.swift.org/t/proposal-helpers-for-initiali 4) var properties with default initializers should have their parameter to the synthesized initializer defaulted. 5) lazy properties with memberwise initializers have problems (the memberwise init eagerly touches it). -Add to the list “all or nothing”. The compiler generates the entire initializer and does not help to eliminate boilerplate for any other initializers where it may be desirable to use memberwise intialization for a subset of members and initialize others manually. +Add to the list “all or nothing”. The compiler generates the entire initializer and does not help to eliminate boilerplate for any other initializers where it may be desirable to use memberwise initialization for a subset of members and initialize others manually. -It is common to have a type with a number of public members that are intended to be configured by clients, but also with some private state comprising implementation details of the type. This is especially prevalent in UI code which may expose many properties for configuring visual appearance, etc. Flexibile memberwise initialization can provide great benefit in these use cases, but it immediately becomes useless if it is "all or nothing". +It is common to have a type with a number of public members that are intended to be configured by clients, but also with some private state comprising implementation details of the type. This is especially prevalent in UI code which may expose many properties for configuring visual appearance, etc. Flexible memberwise initialization can provide great benefit in these use cases, but it immediately becomes useless if it is "all or nothing". We need a flexible solution that can synthesize memberwise initialization for some members while allowing the type author full control over initialization of implementation details. @@ -49,7 +47,7 @@ The two approaches are not mutually exclusive: it is possible to use the *automa The *automatic* model of the current proposal determines the set of properties that receive memberwise initialization parameters by considering *only* the initializer declaration and the declarations for all properties that are *at least* as visible as the initializer (including any behaviors attached to the properties). The rules are as follows: -1. The access level of the property is *at least* as visible as the memberwise initializer. The visiblity of the **setter** is used for `var` properties. +1. The access level of the property is *at least* as visible as the memberwise initializer. The visibility of the **setter** is used for `var` properties. 2. They do not have a behavior which prohibits memberwise initialization (e.g. the 'lazy' behavior). 3. If the property is a `let` property it *may not* have an initial value. @@ -194,7 +192,7 @@ Throughout this design the term **memberwise initialization parameter** is used 1. Determine the set of properties eligible for memberwise initialization synthesis. Properties are eligible for memberwise initialization synthesis if: - 1. The access level of the property is *at least* as visible as the memberwise initializer. The visiblity of the **setter** is used for `var` properties. + 1. The access level of the property is *at least* as visible as the memberwise initializer. The visibility of the **setter** is used for `var` properties. 2. They do not have a behavior which prohibits memberwise initialization. 3. If the property is a `let` property it *may not* have an initial value. @@ -218,7 +216,7 @@ This proposal will also support generating an *implicit* memberwise initializer 2. The type is: 1. a struct 2. a root class - 3. a class whose superclass has a designated intializer requiring no arguments + 3. a class whose superclass has a designated initializer requiring no arguments The implicitly generated memberwise initializer will have the highest access level possible while still allowing all stored properties to be eligible for memberwise parameter synthesis, but will have at most `internal` visibility. Currently this means its visibility will be `internal` when all stored properties of the type have setters with *at least* `internal` visibility, and `private` otherwise (when one or more stored properties are `private` or `private(set)`). @@ -230,7 +228,7 @@ The changes described in this proposal are *almost* entirely additive. The only 1. If the implicitly synthesized memberwise initializer was only used *within* the same source file no change is necessary. An implicit `private` memberwise initializer will still be synthesized by the compiler. 2. A mechanical migration could generate the explicit code necessary to declare the previously implicit initializer. This would be an `internal` memberwise initializer with *explicit* parameters used to manually initialize the stored properties with `private` setters. -3. If the "Access control for init" enhancement were accepted the `private` members could have their access control modified to `private internal(init)` which would allow the implicit memberwise intializer to continue to have `internal` visibility as all stored properties would be eligible for parameter synthesis by an `internal` memberwise initializer. +3. If the "Access control for init" enhancement were accepted the `private` members could have their access control modified to `private internal(init)` which would allow the implicit memberwise initializer to continue to have `internal` visibility as all stored properties would be eligible for parameter synthesis by an `internal` memberwise initializer. The only other impact on existing code is that memberwise parameters corresponding to `var` properties with initial values will now have default values. This will be a change in the behavior of the implicit memberwise initializer but will not break any code. The change will simply allow new code to use that initializer without providing an argument for such parameters. @@ -295,7 +293,7 @@ The rules of the current proposal are designed to synthesize memberwise paramete Introducing a `memberwise` declaration modifier for properties would allow programmers to specify exactly which properties should participate in memberwise initialization synthesis. It allows full control and has the clarity afforded by being explicit. -Specifc use cases this feature would support include allowing `private` properties to receive synthesized memberwise parameters in a `public` initializer, or allow `public` properties to be omitted from parameter synthesis. +Specific use cases this feature would support include allowing `private` properties to receive synthesized memberwise parameters in a `public` initializer, or allow `public` properties to be omitted from parameter synthesis. An example of this @@ -350,7 +348,7 @@ struct S { If this enhancement were submitted the first property eligibility rule would be updates as follows: -1. Their **init** access level is *at least* as visible as the memberwise initializer. If the property does not have an **init** acccess level, the access level of its **setter** must be *at least* as visible as the memberwise initializer. +1. Their **init** access level is *at least* as visible as the memberwise initializer. If the property does not have an **init** access level, the access level of its **setter** must be *at least* as visible as the memberwise initializer. ### @nomemberwise @@ -402,7 +400,7 @@ struct S { init(s: String) { /* synthesized */ self.s = s - // body of the user's intializer remains + // body of the user's initializer remains i = 42 } } @@ -410,7 +408,7 @@ struct S { ### Memberwise initializer chaining / parameter forwarding -Ideally it would be possible to define convenience and delegating initializers without requiring them to manually declare parameters and pass arguments to the designated initializer for memberwise intialized properties. It would also be ideal if designated initializers also did not have to the same for memberwise intialization parmaeters of super. +Ideally it would be possible to define convenience and delegating initializers without requiring them to manually declare parameters and pass arguments to the designated initializer for memberwise initialized properties. It would also be ideal if designated initializers also did not have to the same for memberwise initialization parameters of super. A general solution for parameter forwarding would solve this problem. A future parameter forwarding proposal to support this use case and others is likely to be pursued. @@ -437,7 +435,7 @@ Obviously supporting memberwise initialization with Cocoa classes would require This is a reasonable option and and I expect a healthy debate about which default is better. The decision to adopt the *automatic* model by default was made for several reasons: 1. The memberwise initializer for structs does not currently require an annotation for properties to opt-in. Requiring an annotation for a mechanism designed to supersede that mechanism may be viewed as boilerplate. -2. Stored properties with public visibility are often intialized directly with a value provided by the caller. +2. Stored properties with public visibility are often initialized directly with a value provided by the caller. 3. Stored properties with **less visibility** than a memberwise initializer are not eligible for memberwise initialization. No annotation is required to indicate that and it is usually not desired. 4. The *automatic* model cannot exist unless it is the default. The *opt-in* model can exist alongside the *automatic* model and itself be opted-into simply by specifying the `memberwise` declaration modifier on one or more properties. @@ -464,7 +462,7 @@ Reasons to limit memberwise parameter synthesis to members which are *at least* 5. If a proposal for `@nomemberwise` is put forward and adopted that would allow us to prevent synthesis of parameters for members as desired. Unfortunately `@nomemberwise` would need to be used much more heavily than it otherwise would (i.e. to prevent synthesis of memberwise parameters for more-private members). It would be better if `@nomemberwise` was not necessary most of the time. 6. If callers must be able to provide memberwise arguments for more-private members directly it is still possible to allow that while taking advantage of memberwise initialization for same-or-less-private members. You just need to declare a `memberwise init` with explicitly declared parameters for the more-private members and initialize them manually in the body. If the "Access control for init" enhancement is accepted another option would be upgrading the visibility of `init` for the more-private member while retaining its access level for the getter and setter. Requiring the programmer to explicitly expose a more-private member either via `init` access control or by writing code that it directly is arguably a very good thing. -Reasons we might want to allow memberwise parameter synthesis for members with lower visiblity than the initializer: +Reasons we might want to allow memberwise parameter synthesis for members with lower visibility than the initializer: 1. Not doing so puts tension between access control for stored properties and memberwise inits. You have to choose between narrower access control or getting the benefit of a memberwise init. Another way to say it: this design means that narrow access control leads to boilerplate. @@ -472,7 +470,7 @@ NOTE: The tension mentioned here is lessened by #6 above: memberwise initializat ### Require initializers to explicitly specify memberwise initialization parameters -The thread "[helpers for initializing properties of the same name as parameters](https://forums.swift.org/t/proposal-helpers-for-initializing-properties-of-same-name-as-parameters/129/3)" discussed an idea for synthesizing property initialization in the body of the initializer while requiring the parameters to be declard explicitly. +The thread "[helpers for initializing properties of the same name as parameters](https://forums.swift.org/t/proposal-helpers-for-initializing-properties-of-same-name-as-parameters/129/3)" discussed an idea for synthesizing property initialization in the body of the initializer while requiring the parameters to be declared explicitly. ```swift struct Foo { @@ -496,7 +494,7 @@ Under the current proposal full control is still available. It requires initial I believe the `memberwise` declaration modifier on the initializer and the placeholder in the parameter list make it clear that the compiler will synthesize additional parameters. Furthermore, IDEs and generated documentation will contain the full, synthesized signature of the initializer. -Finally, this idea is not mutually exclusive with the current proposal. It could even work in the declaration of a memberwise initializer, so long the corresponding property was made ineligible for memberwise intialization synthesis. +Finally, this idea is not mutually exclusive with the current proposal. It could even work in the declaration of a memberwise initializer, so long the corresponding property was made ineligible for memberwise initialization synthesis. ### Adopt "type parameter list" syntax like Kotlin and Scala @@ -542,7 +540,7 @@ Responses to these points follow: 1. If the expansion of this syntax does not supply initial values to the synthesized properties and only uses the default value for parameters of the synthesized initializer this is true. The downside of doing this is that `var` properties no longer have an initial value which may be desirable if you write additional initializers for the type. I believe we should continue the discussion about default values for `let` properties. Ideally we can find an acceptable solution that will work with the current proposal, as well as any additional syntactic sugar we add in the future. -2. I don't believe allowing parameter labels for memberwise initialization parameters is a good idea. Callers are directly initializing a property and are best served by a label that matches the name of the property. If you really need to provide a different name you can still do so by writing your initializer manually. With future enhancements to the current proposal you may be able to use memberwise intialization for properties that do not require a custom label while manually initialzing properties that do need one. +2. I don't believe allowing parameter labels for memberwise initialization parameters is a good idea. Callers are directly initializing a property and are best served by a label that matches the name of the property. If you really need to provide a different name you can still do so by writing your initializer manually. With future enhancements to the current proposal you may be able to use memberwise initialization for properties that do not require a custom label while manually initializing properties that do need one. 3. The Scala / Kotlin syntax is indeed more concise in some cases, but not in all cases. Under this proposal the example given above is actually more concise than it is with that syntax: ```swift diff --git a/proposals/0019-package-manager-testing.md b/proposals/0019-package-manager-testing.md index 6502f760e4..252c9bdaa0 100644 --- a/proposals/0019-package-manager-testing.md +++ b/proposals/0019-package-manager-testing.md @@ -3,7 +3,7 @@ * Proposal: [SE-0019](0019-package-manager-testing.md) * Authors: [Max Howell](https://github.com/mxcl), [Daniel Dunbar](https://github.com/ddunbar), [Mattt Thompson](https://github.com/mattt) * Review Manager: [Rick Ballard](https://github.com/rballard) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0019-swift-testing-package-manager/1155) * Bug: [SR-592](https://bugs.swift.org/browse/SR-592) diff --git a/proposals/0020-if-swift-version.md b/proposals/0020-if-swift-version.md index ec90962201..a59e6ad4dc 100644 --- a/proposals/0020-if-swift-version.md +++ b/proposals/0020-if-swift-version.md @@ -1,7 +1,7 @@ # Swift Language Version Build Configuration * Proposal: [SE-0020](0020-if-swift-version.md) -* Author: [David Farler](https://github.com/bitjammer) +* Author: [Ashley Garland](https://github.com/bitjammer) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Implemented (Swift 2.2)** * Implementation: [apple/swift@c32fb8e](https://github.com/apple/swift/commit/c32fb8e7b9a67907e8b6580a46717c6a345ec7c6) diff --git a/proposals/0023-api-guidelines.md b/proposals/0023-api-guidelines.md index e131d88831..e8e5c3fe02 100644 --- a/proposals/0023-api-guidelines.md +++ b/proposals/0023-api-guidelines.md @@ -1,9 +1,9 @@ # API Design Guidelines * Proposal: [SE-0023](0023-api-guidelines.md) -* Authors: [Dave Abrahams](https://github.com/dabrahams), [Doug Gregor](https://github.com/DougGregor), [Dmitri Gribenko](https://github.com/gribozavr), [Ted Kremenek](https://github.com/tkremenek), [Chris Lattner](http://github.com/lattner), Alex Migicovsky, [Max Moiseev](https://github.com/moiseev), Ali Ozer, [Tony Parker](https://github.com/parkera) +* Authors: [Dave Abrahams](https://github.com/dabrahams), [Doug Gregor](https://github.com/DougGregor), [Dmitri Gribenko](https://github.com/gribozavr), [Ted Kremenek](https://github.com/tkremenek), [Chris Lattner](https://github.com/lattner), Alex Migicovsky, [Max Moiseev](https://github.com/moiseev), Ali Ozer, [Tony Parker](https://github.com/parkera) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0023-api-design-guidelines/1666) ## Reviewer notes diff --git a/proposals/0025-scoped-access-level.md b/proposals/0025-scoped-access-level.md index 310956cd11..470666dd8b 100644 --- a/proposals/0025-scoped-access-level.md +++ b/proposals/0025-scoped-access-level.md @@ -2,11 +2,11 @@ * Proposal: [SE-0025](0025-scoped-access-level.md) * Author: Ilya Belenkiy -* Status: **Implemented (Swift 3)** -* Review Manager: [Doug Gregor](http://github.com/DougGregor) +* Status: **Implemented (Swift 3.0)** +* Review Manager: [Doug Gregor](https://github.com/DougGregor) * Decision Notes: [Rationale](https://forums.swift.org/t/se-0025-scoped-access-level-next-steps/1797/131) * Bug: [SR-1275](https://bugs.swift.org/browse/SR-1275) -* Previous revision: [1](https://github.com/apple/swift-evolution/blob/e4328889a9643100177aef19f6f428855c5d0cf2/proposals/0025-scoped-access-level.md) +* Previous revision: [1](https://github.com/swiftlang/swift-evolution/blob/e4328889a9643100177aef19f6f428855c5d0cf2/proposals/0025-scoped-access-level.md) ## Introduction diff --git a/proposals/0026-abstract-classes-and-methods.md b/proposals/0026-abstract-classes-and-methods.md index d042e646c1..24b89a5eb9 100644 --- a/proposals/0026-abstract-classes-and-methods.md +++ b/proposals/0026-abstract-classes-and-methods.md @@ -3,8 +3,8 @@ * Proposal: [SE-0026](0026-abstract-classes-and-methods.md) * Author: David Scrève * Review Manager: [Joe Groff](https://github.com/jckarter/) -* Status: **Deferred** -* Decision Notes: [Rationale](https://forums.swift.org/t/deferred-se-0026-abstract-classes-and-methods/1705) +* Status: **Rejected** +* Review: ([pitch](https://forums.swift.org/t/proposal-draff-abstract-classes-and-methods/965)) ([review](https://forums.swift.org/t/review-se-0026-abstract-classes-and-methods/1580)) ([deferral](https://forums.swift.org/t/deferred-se-0026-abstract-classes-and-methods/1705)) ([rejection](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction @@ -14,11 +14,9 @@ they cannot have attributes as classes have. A partial class combines the behavior of a class with the requirement of implementing methods in inherited class like protocols. -[Swift-Evolution Discussion](https://forums.swift.org/t/proposal-draff-abstract-classes-and-methods/965) - ## Motivation -like pure virtual methods in C++ and abtract classes in Java and C#, frameworks development +like pure virtual methods in C++ and abstract classes in Java and C#, frameworks development sometimes required abstract classes facility. An abstract class is like a regular class, but some methods/properties are not implemented and must be implemented in one of inherited classes. @@ -99,7 +97,7 @@ class MyRestServiceClient : RESTClient { ``` ## Detailed design -An abstract class cannot be instanciated. +An abstract class cannot be instantiated. Abstract method/property cannot have implementation. diff --git a/proposals/0028-modernizing-debug-identifiers.md b/proposals/0028-modernizing-debug-identifiers.md index 6787f7208f..9e51fb46f9 100644 --- a/proposals/0028-modernizing-debug-identifiers.md +++ b/proposals/0028-modernizing-debug-identifiers.md @@ -1,7 +1,7 @@ # Modernizing Swift's Debugging Identifiers * Proposal: [SE-0028](0028-modernizing-debug-identifiers.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Implemented (Swift 2.2)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0028-modernizing-swifts-debugging-identifiers-line-etc/1303) diff --git a/proposals/0029-remove-implicit-tuple-splat.md b/proposals/0029-remove-implicit-tuple-splat.md index 9b4c779be6..e16653aef4 100644 --- a/proposals/0029-remove-implicit-tuple-splat.md +++ b/proposals/0029-remove-implicit-tuple-splat.md @@ -1,9 +1,9 @@ # Remove implicit tuple splat behavior from function applications * Proposal: [SE-0029](0029-remove-implicit-tuple-splat.md) -* Author: [Chris Lattner](http://github.com/lattner) -* Review Manager: [Joe Groff](http://github.com/jckarter) -* Status: **Implemented (Swift 3)** +* Author: [Chris Lattner](https://github.com/lattner) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0029-remove-implicit-tuple-splat-behavior-from-function-applications/1380) * Implementation: [apple/swift@8e12008](https://github.com/apple/swift/commit/8e12008d2b34a605f8766310f53d5668f3d50955) diff --git a/proposals/0031-adjusting-inout-declarations.md b/proposals/0031-adjusting-inout-declarations.md index c4acd2d593..a49eadb4eb 100644 --- a/proposals/0031-adjusting-inout-declarations.md +++ b/proposals/0031-adjusting-inout-declarations.md @@ -1,9 +1,9 @@ # Adjusting `inout` Declarations for Type Decoration * Proposal: [SE-0031](0031-adjusting-inout-declarations.md) -* Authors: [Joe Groff](https://github.com/jckarter), [Erica Sadun](http://github.com/erica) +* Authors: [Joe Groff](https://github.com/jckarter), [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0031-adjusting-inout-declarations-for-type-decoration/1478) * Implementation: [apple/swift#1333](https://github.com/apple/swift/pull/1333) diff --git a/proposals/0032-sequencetype-find.md b/proposals/0032-sequencetype-find.md index 9dbd6358a3..7dd6b27397 100644 --- a/proposals/0032-sequencetype-find.md +++ b/proposals/0032-sequencetype-find.md @@ -3,10 +3,10 @@ * Proposal: [SE-0032](0032-sequencetype-find.md) * Author: [Lily Ballard](https://github.com/lilyball) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0032-add-find-method-to-sequence/2462) * Bug: [SR-1519](https://bugs.swift.org/browse/SR-1519) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0032-sequencetype-find.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0032-sequencetype-find.md) ## Introduction diff --git a/proposals/0033-import-objc-constants.md b/proposals/0033-import-objc-constants.md index 744974b40e..06e12fcb71 100644 --- a/proposals/0033-import-objc-constants.md +++ b/proposals/0033-import-objc-constants.md @@ -3,7 +3,7 @@ * Proposal: [SE-0033](0033-import-objc-constants.md) * Author: [Jeff Kelley](https://github.com/SlaunchaMan) * Review Manager: [John McCall](https://github.com/rjmccall) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0033-import-objective-c-constants-as-swift-types/1706) ## Introduction diff --git a/proposals/0034-disambiguating-line.md b/proposals/0034-disambiguating-line.md index 97b2164fe2..61b51ffe47 100644 --- a/proposals/0034-disambiguating-line.md +++ b/proposals/0034-disambiguating-line.md @@ -1,9 +1,9 @@ # Disambiguating Line Control Statements from Debugging Identifiers * Proposal: [SE-0034](0034-disambiguating-line.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0034-disambiguating-line-control-statements-from-debugging-identifiers/1614) * Bug: [SR-840](https://bugs.swift.org/browse/SR-840) diff --git a/proposals/0035-limit-inout-capture.md b/proposals/0035-limit-inout-capture.md index 466df50182..4a44fd09ff 100644 --- a/proposals/0035-limit-inout-capture.md +++ b/proposals/0035-limit-inout-capture.md @@ -3,7 +3,7 @@ * Proposal: [SE-0035](0035-limit-inout-capture.md) * Author: [Joe Groff](https://github.com/jckarter) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0035-limiting-inout-capture-to-noescape-contexts/1544) * Bug: [SR-807](https://bugs.swift.org/browse/SR-807) diff --git a/proposals/0036-enum-dot.md b/proposals/0036-enum-dot.md index 8e7f6cfaa0..c9cd52fbcd 100644 --- a/proposals/0036-enum-dot.md +++ b/proposals/0036-enum-dot.md @@ -1,9 +1,9 @@ # Requiring Leading Dot Prefixes for Enum Instance Member Implementations * Proposal: [SE-0036](0036-enum-dot.md) -* Authors: [Erica Sadun](http://github.com/erica), [Chris Lattner](https://github.com/lattner) +* Authors: [Erica Sadun](https://github.com/erica), [Chris Lattner](https://github.com/lattner) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0036-requiring-leading-dot-prefixes-for-enum-instance-member-implementations/2196) * Bug: [SR-1236](https://bugs.swift.org/browse/SR-1236) diff --git a/proposals/0037-clarify-comments-and-operators.md b/proposals/0037-clarify-comments-and-operators.md index 293fd83dc6..9866d04c29 100644 --- a/proposals/0037-clarify-comments-and-operators.md +++ b/proposals/0037-clarify-comments-and-operators.md @@ -2,8 +2,8 @@ * Proposal: [SE-0037](0037-clarify-comments-and-operators.md) * Author: [Jesse Rusak](https://github.com/jder) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0037-clarify-interaction-between-comments-operators/1833) * Bug: [SR-960](https://bugs.swift.org/browse/SR-960) diff --git a/proposals/0038-swiftpm-c-language-targets.md b/proposals/0038-swiftpm-c-language-targets.md index 9ef4f20715..d86482524e 100644 --- a/proposals/0038-swiftpm-c-language-targets.md +++ b/proposals/0038-swiftpm-c-language-targets.md @@ -3,7 +3,7 @@ * Proposal: [SE-0038](0038-swiftpm-c-language-targets.md) * Author: [Daniel Dunbar](https://github.com/ddunbar) * Review Manager: [Rick Ballard](https://github.com/rballard) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0038-package-manager-c-language-target-support/1569) * Bug: [SR-821](https://bugs.swift.org/browse/SR-821) diff --git a/proposals/0039-playgroundliterals.md b/proposals/0039-playgroundliterals.md index 42ccdab79d..c03981cf64 100644 --- a/proposals/0039-playgroundliterals.md +++ b/proposals/0039-playgroundliterals.md @@ -1,9 +1,9 @@ # Modernizing Playground Literals * Proposal: [SE-0039](0039-playgroundliterals.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0039-modernizing-playground-literals/1746) * Bug: [SR-917](https://bugs.swift.org/browse/SR-917) diff --git a/proposals/0040-attributecolons.md b/proposals/0040-attributecolons.md index 18f4ced423..17c49bbd26 100644 --- a/proposals/0040-attributecolons.md +++ b/proposals/0040-attributecolons.md @@ -1,9 +1,9 @@ # Replacing Equal Signs with Colons For Attribute Arguments * Proposal: [SE-0040](0040-attributecolons.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0040-replacing-equal-signs-with-colons-for-attribute-arguments/1719) * Implementation: [apple/swift#1537](https://github.com/apple/swift/pull/1537) diff --git a/proposals/0041-conversion-protocol-conventions.md b/proposals/0041-conversion-protocol-conventions.md index e3af4e753f..9b61c3cc3b 100644 --- a/proposals/0041-conversion-protocol-conventions.md +++ b/proposals/0041-conversion-protocol-conventions.md @@ -1,8 +1,8 @@ # Updating Protocol Naming Conventions for Conversions * Proposal: [SE-0041](0041-conversion-protocol-conventions.md) -* Authors: [Matthew Johnson](https://github.com/anandabits), [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Authors: [Matthew Johnson](https://github.com/anandabits), [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0041-updating-protocol-naming-conventions-for-conversions/2684) diff --git a/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md b/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md index 048a2c520a..abc9390c74 100644 --- a/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md +++ b/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md @@ -3,7 +3,7 @@ * Proposal: [SE-0043](0043-declare-variables-in-case-labels-with-multiple-patterns.md) * Author: [Andrew Bennett](https://github.com/therealbnut) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0043-declare-variables-in-case-labels-with-multiple-patterns/1925) * Implementation: [apple/swift#1383](https://github.com/apple/swift/pull/1383) diff --git a/proposals/0044-import-as-member.md b/proposals/0044-import-as-member.md index 8ea02ed09e..a978919016 100644 --- a/proposals/0044-import-as-member.md +++ b/proposals/0044-import-as-member.md @@ -2,7 +2,7 @@ * Proposal: [SE-0044](0044-import-as-member.md) * Author: [Michael Ilseman](https://github.com/milseman) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0044-import-as-member/1929) * Bug: [SR-1053](https://bugs.swift.org/browse/SR-1053) diff --git a/proposals/0045-scan-takewhile-dropwhile.md b/proposals/0045-scan-takewhile-dropwhile.md index 143395fbdf..c4f3a751c0 100644 --- a/proposals/0045-scan-takewhile-dropwhile.md +++ b/proposals/0045-scan-takewhile-dropwhile.md @@ -2,11 +2,11 @@ * Proposal: [SE-0045](0045-scan-takewhile-dropwhile.md) * Author: [Lily Ballard](https://github.com/lilyball) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Implemented (Swift 3.1)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0045-add-scan-prefix-while-drop-while-and-unfold-to-the-stdlib/2466) * Bug: [SR-1516](https://bugs.swift.org/browse/SR-1516) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/b39d653f7e3d5e982b562664343f26c826652291/proposals/0045-scan-takewhile-dropwhile.md), [2](https://github.com/apple/swift-evolution/blob/baec22a8a5ddaa0407086380da32b5cad2144800/proposals/0045-scan-takewhile-dropwhile.md), [3](https://github.com/apple/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0045-scan-takewhile-dropwhile.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/b39d653f7e3d5e982b562664343f26c826652291/proposals/0045-scan-takewhile-dropwhile.md), [2](https://github.com/swiftlang/swift-evolution/blob/baec22a8a5ddaa0407086380da32b5cad2144800/proposals/0045-scan-takewhile-dropwhile.md), [3](https://github.com/swiftlang/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0045-scan-takewhile-dropwhile.md) ## Introduction diff --git a/proposals/0046-first-label.md b/proposals/0046-first-label.md index 30b54b1705..6b401da8aa 100644 --- a/proposals/0046-first-label.md +++ b/proposals/0046-first-label.md @@ -1,9 +1,9 @@ # Establish consistent label behavior across all parameters including first labels * Proposal: [SE-0046](0046-first-label.md) -* Authors: [Jake Carter](https://github.com/JakeCarter), [Erica Sadun](http://github.com/erica) +* Authors: [Jake Carter](https://github.com/JakeCarter), [Erica Sadun](https://github.com/erica) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0046-establish-consistent-label-behavior-across-all-parameters-including-first-labels/1834) * Bug: [SR-961](https://bugs.swift.org/browse/SR-961) diff --git a/proposals/0047-nonvoid-warn.md b/proposals/0047-nonvoid-warn.md index 8026a5e3ee..ab940ba456 100644 --- a/proposals/0047-nonvoid-warn.md +++ b/proposals/0047-nonvoid-warn.md @@ -1,9 +1,9 @@ # Defaulting non-Void functions so they warn on unused results * Proposal: [SE-0047](0047-nonvoid-warn.md) -* Authors: [Erica Sadun](http://github.com/erica), [Adrian Kashivskyy](https://github.com/akashivskyy) +* Authors: [Erica Sadun](https://github.com/erica), [Adrian Kashivskyy](https://github.com/akashivskyy) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0047-defaulting-non-void-functions-so-they-warn-on-unused-results/1927) * Bug: [SR-1052](https://bugs.swift.org/browse/SR-1052) diff --git a/proposals/0048-generic-typealias.md b/proposals/0048-generic-typealias.md index 214861b5ec..4f6efecb5a 100644 --- a/proposals/0048-generic-typealias.md +++ b/proposals/0048-generic-typealias.md @@ -3,7 +3,7 @@ * Proposal: [SE-0048](0048-generic-typealias.md) * Author: [Chris Lattner](https://github.com/lattner) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0048-generic-type-aliases/2192) diff --git a/proposals/0049-noescape-autoclosure-type-attrs.md b/proposals/0049-noescape-autoclosure-type-attrs.md index 6b6e5eb406..a6572f2c82 100644 --- a/proposals/0049-noescape-autoclosure-type-attrs.md +++ b/proposals/0049-noescape-autoclosure-type-attrs.md @@ -3,7 +3,7 @@ * Proposal: [SE-0049](0049-noescape-autoclosure-type-attrs.md) * Author: [Chris Lattner](https://github.com/lattner) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0049-move-noescape-and-autoclosure-to-be-type-attributes/2194) * Bug: [SR-1235](https://bugs.swift.org/browse/SR-1235) diff --git a/proposals/0050-floating-point-stride.md b/proposals/0050-floating-point-stride.md index d808a91634..63031b0f67 100644 --- a/proposals/0050-floating-point-stride.md +++ b/proposals/0050-floating-point-stride.md @@ -1,8 +1,8 @@ # Decoupling Floating Point Strides from Generic Implementations * Proposal: [SE-0050](0050-floating-point-stride.md) -* Authors: [Erica Sadun](http://github.com/erica), [Xiaodi Wu](http://github.com/xwu) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Authors: [Erica Sadun](https://github.com/erica), [Xiaodi Wu](https://github.com/xwu) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Withdrawn** * Decision Notes: [Rationale](https://forums.swift.org/t/returned-for-revision-se-0050-decoupling-floating-point-strides-from-generic-implementations/2823) diff --git a/proposals/0051-stride-semantics.md b/proposals/0051-stride-semantics.md index bb96a09150..aabe1b2cad 100644 --- a/proposals/0051-stride-semantics.md +++ b/proposals/0051-stride-semantics.md @@ -1,7 +1,7 @@ # Conventionalizing `stride` semantics * Proposal: [SE-0051](0051-stride-semantics.md) -* Author: [Erica Sadun](http://github.com/erica) +* Author: [Erica Sadun](https://github.com/erica) * Review Manager: N/A * Status: **Withdrawn** diff --git a/proposals/0052-iterator-post-nil-guarantee.md b/proposals/0052-iterator-post-nil-guarantee.md index 28d3219020..7b9c287fe7 100644 --- a/proposals/0052-iterator-post-nil-guarantee.md +++ b/proposals/0052-iterator-post-nil-guarantee.md @@ -3,7 +3,7 @@ * Proposal: [SE-0052](0052-iterator-post-nil-guarantee.md) * Author: [Patrick Pijnappel](https://github.com/PatrickPijnappel) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0052-change-iteratortype-post-nil-guarantee/2463) * Implementation: [apple/swift#1702](https://github.com/apple/swift/pull/1702) diff --git a/proposals/0053-remove-let-from-function-parameters.md b/proposals/0053-remove-let-from-function-parameters.md index 8d2fcd1727..c3b22b4907 100644 --- a/proposals/0053-remove-let-from-function-parameters.md +++ b/proposals/0053-remove-let-from-function-parameters.md @@ -3,7 +3,7 @@ * Proposal: [SE-0053](0053-remove-let-from-function-parameters.md) * Author: [Nicholas Maccharoli](https://github.com/nirma) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0053-remove-explicit-use-of-let-from-function-parameters/1966) * Implementation: [apple/swift#1812](https://github.com/apple/swift/pull/1812) diff --git a/proposals/0054-abolish-iuo.md b/proposals/0054-abolish-iuo.md index 60a562f10e..f8f12c9fe2 100644 --- a/proposals/0054-abolish-iuo.md +++ b/proposals/0054-abolish-iuo.md @@ -1,7 +1,7 @@ # Abolish `ImplicitlyUnwrappedOptional` type * Proposal: [SE-0054](0054-abolish-iuo.md) -* Author: [Chris Willmore](http://github.com/cwillmor) +* Author: [Chris Willmore](https://github.com/cwillmor) * Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Implemented (Swift 4.2)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-pending-implementation-se-0054-abolish-implicitlyunwrappedoptional-type/2009) diff --git a/proposals/0055-optional-unsafe-pointers.md b/proposals/0055-optional-unsafe-pointers.md index cc8e645342..6c69e20408 100644 --- a/proposals/0055-optional-unsafe-pointers.md +++ b/proposals/0055-optional-unsafe-pointers.md @@ -3,7 +3,7 @@ * Proposal: [SE-0055](0055-optional-unsafe-pointers.md) * Author: [Jordan Rose](https://github.com/jrose-apple) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0055-make-unsafe-pointer-nullability-explicit-using-optional/2012) * Implementation: [apple/swift#1878](https://github.com/apple/swift/pull/1878) diff --git a/proposals/0057-importing-objc-generics.md b/proposals/0057-importing-objc-generics.md index 3a5edd1577..7faf19fa4b 100644 --- a/proposals/0057-importing-objc-generics.md +++ b/proposals/0057-importing-objc-generics.md @@ -3,9 +3,9 @@ * Proposal: [SE-0057](0057-importing-objc-generics.md) * Author: [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0057-importing-objective-c-lightweight-generics/2185) -* Previous Revision: [Originally Accepted Proposal](https://github.com/apple/swift-evolution/blob/3abbed3edd12dd21061181993df7952665d660dd/proposals/0057-importing-objc-generics.md) +* Previous Revision: [Originally Accepted Proposal](https://github.com/swiftlang/swift-evolution/blob/3abbed3edd12dd21061181993df7952665d660dd/proposals/0057-importing-objc-generics.md) ## Introduction @@ -190,7 +190,7 @@ classes without fundamentally changing the Swift model. ## Revision history -The [originally accepted proposal](https://github.com/apple/swift-evolution/blob/3abbed3edd12dd21061181993df7952665d660dd/proposals/0057-importing-objc-generics.md) +The [originally accepted proposal](https://github.com/swiftlang/swift-evolution/blob/3abbed3edd12dd21061181993df7952665d660dd/proposals/0057-importing-objc-generics.md) included a mechanism by which Objective-C generic classes could implement an informal protocol to provide reified generic arguments to Swift clients: diff --git a/proposals/0058-objectivecbridgeable.md b/proposals/0058-objectivecbridgeable.md index 8ae8771038..478a23adf4 100644 --- a/proposals/0058-objectivecbridgeable.md +++ b/proposals/0058-objectivecbridgeable.md @@ -3,16 +3,13 @@ * Proposal: [SE-0058](0058-objectivecbridgeable.md) * Authors: [Russ Bishop](https://github.com/russbishop), [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Joe Groff](https://github.com/jckarter) -* Status: **Deferred** -* Decision Notes: [Rationale](https://forums.swift.org/t/deferred-se-0058-allow-swift-types-to-provide-custom-objective-c-representations/2167) +* Status: **Rejected** +* Review: ([pitch](https://forums.swift.org/t/idea-objectivecbridgeable/1559)) ([review](https://forums.swift.org/t/review-se-0058-allow-swift-types-to-provide-custom-objective-c-representations/2054)) ([deferral](https://forums.swift.org/t/deferred-se-0058-allow-swift-types-to-provide-custom-objective-c-representations/2167)) ([rejection](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction Provide an `ObjectiveCBridgeable` protocol that allows a Swift type to control how it is represented in Objective-C by converting into and back from an entirely separate `@objc` type. This frees library authors to create truly native Swift APIs while still supporting Objective-C. -Swift-evolution thread: [\[Idea\] ObjectiveCBridgeable](https://forums.swift.org/t/idea-objectivecbridgeable/1559) - - ## Motivation There is currently no good way to define a Swift-y API that makes use of generics, enums with associated values, structs, protocols with associated types, and other Swift features while still exposing that API to Objective-C. @@ -80,7 +77,7 @@ public protocol ObjectiveCBridgeable { /// Objective-C thunk or when calling Objective-C code. /// /// - note: This initializer should eagerly perform the - /// conversion without defering any work for later, + /// conversion without deferring any work for later, /// returning `nil` if the conversion fails. init?(bridgedFromObjectiveC: ObjectiveCType) @@ -147,7 +144,7 @@ The compiler generates automatic thunks only when there is no ambiguity, while e 3. Bridged collection types will still observe the protocol conformance if cast to a Swift type (eg: `NSArray as? [Int]` will call the `ObjectiveCBridgeable` implementation on `Array`, which itself will call the implementation on `Int` for the elements) 2. A Swift type may bridge to an Objective-C base class then provide different subclass instances at runtime, but no other Swift type may bridge to that base class or any of its subclasses. 1. The compiler should emit a diagnostic when it detects two Swift types attempting to bridge to the same `ObjectiveCType`. -3. An exception to these rules exists for trivially convertable built-in types like `NSInteger` <--> `Int` when specified outside of a bridged collection type. In those cases the compiler will continue the existing behavior, bypassing the `ObjectiveCBridgeable` protocol. The effect is that types like `Int` will not bridge to `NSNumber` unless contained inside a collection type (see `BuiltInBridgeable below`). +3. An exception to these rules exists for trivially convertible built-in types like `NSInteger` <--> `Int` when specified outside of a bridged collection type. In those cases the compiler will continue the existing behavior, bypassing the `ObjectiveCBridgeable` protocol. The effect is that types like `Int` will not bridge to `NSNumber` unless contained inside a collection type (see `BuiltInBridgeable below`). ### Resiliance diff --git a/proposals/0059-updated-set-apis.md b/proposals/0059-updated-set-apis.md index e38ab57229..1d69edb0c4 100644 --- a/proposals/0059-updated-set-apis.md +++ b/proposals/0059-updated-set-apis.md @@ -3,7 +3,7 @@ * Proposal: [SE-0059](0059-updated-set-apis.md) * Author: [Dave Abrahams](https://github.com/dabrahams) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0059-update-api-naming-guidelines-and-rewrite-set-apis-accordingly/2251) ## Introduction diff --git a/proposals/0060-defaulted-parameter-order.md b/proposals/0060-defaulted-parameter-order.md index 70e3fa69b6..f6cd8d01f8 100644 --- a/proposals/0060-defaulted-parameter-order.md +++ b/proposals/0060-defaulted-parameter-order.md @@ -2,8 +2,8 @@ * Proposal: [SE-0060](0060-defaulted-parameter-order.md) * Author: [Joe Groff](https://github.com/jckarter) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0060-enforcing-order-of-defaulted-parameters/2573) * Bug: [SR-1489](https://bugs.swift.org/browse/SR-1489) diff --git a/proposals/0061-autoreleasepool-signature.md b/proposals/0061-autoreleasepool-signature.md index 27c5d2346d..e9dc5a961c 100644 --- a/proposals/0061-autoreleasepool-signature.md +++ b/proposals/0061-autoreleasepool-signature.md @@ -2,8 +2,8 @@ * Proposal: [SE-0061](0061-autoreleasepool-signature.md) * Author: [Timothy J. Wood](https://github.com/tjw) -* Review Manager: [Dave Abrahams](http://github.com/dabrahams) -* Status: **Implemented (Swift 3)** +* Review Manager: [Dave Abrahams](https://github.com/dabrahams) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0061-add-generic-result-and-error-handling-to-autoreleasepool/2425) * Bugs: [SR-842](https://bugs.swift.org/browse/SR-842), [SR-1394](https://bugs.swift.org/browse/SR-1394) diff --git a/proposals/0062-objc-keypaths.md b/proposals/0062-objc-keypaths.md index 952c2c78a3..54fa946647 100644 --- a/proposals/0062-objc-keypaths.md +++ b/proposals/0062-objc-keypaths.md @@ -3,7 +3,7 @@ * Proposal: [SE-0062](0062-objc-keypaths.md) * Author: [David Hart](https://github.com/hartbit) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0062-referencing-objective-c-key-paths/2198) * Bug: [SR-1237](https://bugs.swift.org/browse/SR-1237) diff --git a/proposals/0063-swiftpm-system-module-search-paths.md b/proposals/0063-swiftpm-system-module-search-paths.md index 92aa798404..cb11562abf 100644 --- a/proposals/0063-swiftpm-system-module-search-paths.md +++ b/proposals/0063-swiftpm-system-module-search-paths.md @@ -3,7 +3,7 @@ * Proposal: [SE-0063](0063-swiftpm-system-module-search-paths.md) * Author: [Max Howell](https://github.com/mxcl) * Review Manager: [Anders Bertelrud](https://github.com/abertelrud) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0063-swiftpm-system-module-search-paths/2218) * Implementation: [apple/swift-package-manager#257](https://github.com/apple/swift-package-manager/pull/257) diff --git a/proposals/0064-property-selectors.md b/proposals/0064-property-selectors.md index 92d670a3c0..e3e7b61e35 100644 --- a/proposals/0064-property-selectors.md +++ b/proposals/0064-property-selectors.md @@ -3,7 +3,7 @@ * Proposal: [SE-0064](0064-property-selectors.md) * Author: [David Hart](https://github.com/hartbit) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0064-referencing-the-objective-c-selector-of-property-getters-and-setters/2199) * Bug: [SR-1239](https://bugs.swift.org/browse/SR-1239) diff --git a/proposals/0065-collections-move-indices.md b/proposals/0065-collections-move-indices.md index 68e3c89589..2353847ca5 100644 --- a/proposals/0065-collections-move-indices.md +++ b/proposals/0065-collections-move-indices.md @@ -3,10 +3,10 @@ * Proposal: [SE-0065](0065-collections-move-indices.md) * Authors: [Dmitri Gribenko](https://github.com/gribozavr), [Dave Abrahams](https://github.com/dabrahams), [Maxim Moiseev](https://github.com/moiseev) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0065-a-new-model-for-collections/2371), [Swift-evolution thread](https://forums.swift.org/t/rfc-new-collections-model-collections-advance-indices/1643) * Implementation: [apple/swift#2108](https://github.com/apple/swift/pull/2108) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/21fac2e8034e79e4f44c1c8799808fc8cba83395/proposals/0065-collections-move-indices.md), [2](https://github.com/apple/swift-evolution/blob/1a821cf7ccbdf1d7566e9ce2e991bdd835ba3b7d/proposals/0065-collections-move-indices.md), [3](https://github.com/apple/swift-evolution/blob/d44c3e7c189ba39ddf8a914ae8b78b71f88fdcdf/proposals/0065-collections-move-indices.md), [4](https://github.com/apple/swift-evolution/blob/57639040dc08d2f0b16d9bda527db069589b58d1/proposals/0065-collections-move-indices.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/21fac2e8034e79e4f44c1c8799808fc8cba83395/proposals/0065-collections-move-indices.md), [2](https://github.com/swiftlang/swift-evolution/blob/1a821cf7ccbdf1d7566e9ce2e991bdd835ba3b7d/proposals/0065-collections-move-indices.md), [3](https://github.com/swiftlang/swift-evolution/blob/d44c3e7c189ba39ddf8a914ae8b78b71f88fdcdf/proposals/0065-collections-move-indices.md), [4](https://github.com/swiftlang/swift-evolution/blob/57639040dc08d2f0b16d9bda527db069589b58d1/proposals/0065-collections-move-indices.md) ## Summary diff --git a/proposals/0066-standardize-function-type-syntax.md b/proposals/0066-standardize-function-type-syntax.md index 0e65fa04ff..e5cd1c72a7 100644 --- a/proposals/0066-standardize-function-type-syntax.md +++ b/proposals/0066-standardize-function-type-syntax.md @@ -3,7 +3,7 @@ * Proposal: [SE-0066](0066-standardize-function-type-syntax.md) * Author: [Chris Lattner](https://github.com/lattner) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0066-standardize-function-type-argument-syntax-to-require-parentheses/2488) * Implementation: [apple/swift@3d2b5bc](https://github.com/apple/swift/commit/3d2b5bcc5350e1dea2ed8a0a95cd12ff5c760f24) diff --git a/proposals/0067-floating-point-protocols.md b/proposals/0067-floating-point-protocols.md index ec72adab34..b93f354014 100644 --- a/proposals/0067-floating-point-protocols.md +++ b/proposals/0067-floating-point-protocols.md @@ -3,10 +3,10 @@ * Proposal: [SE-0067](0067-floating-point-protocols.md) * Author: [Stephen Canon](https://github.com/stephentyrone) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0067-enhanced-floating-point-protocols/2420) * Implementation: [apple/swift#2453](https://github.com/apple/swift/pull/2453) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/fb1368a6a5474f57aa8f1846b5355d18753098f3/proposals/0067-floating-point-protocols.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/fb1368a6a5474f57aa8f1846b5355d18753098f3/proposals/0067-floating-point-protocols.md) ## Introduction diff --git a/proposals/0068-universal-self.md b/proposals/0068-universal-self.md index e6dda07d6b..f129386def 100644 --- a/proposals/0068-universal-self.md +++ b/proposals/0068-universal-self.md @@ -7,7 +7,7 @@ * Implementation: [apple/swift#22863](https://github.com/apple/swift/pull/22863) * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modification-se-0068-expanding-swift-self-to-class-members-and-value-types/2373) * Bug: [SR-1340](https://bugs.swift.org/browse/SR-1340) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/bcd77b028cb2fc9f07472532b120e927c7e48b34/proposals/0068-universal-self.md), [2](https://github.com/apple/swift-evolution/blob/13d9771e86c5639b8320f05e5daa31a62bac0f07/proposals/0068-universal-self.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/bcd77b028cb2fc9f07472532b120e927c7e48b34/proposals/0068-universal-self.md), [2](https://github.com/swiftlang/swift-evolution/blob/13d9771e86c5639b8320f05e5daa31a62bac0f07/proposals/0068-universal-self.md) ## Introduction diff --git a/proposals/0069-swift-mutability-for-foundation.md b/proposals/0069-swift-mutability-for-foundation.md index c7d6769603..e16169cf97 100644 --- a/proposals/0069-swift-mutability-for-foundation.md +++ b/proposals/0069-swift-mutability-for-foundation.md @@ -3,7 +3,7 @@ * Proposal: [SE-0069](0069-swift-mutability-for-foundation.md) * Author: [Tony Parker](https://github.com/parkera) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0069-mutability-and-foundation-value-types/2460) ## Introduction diff --git a/proposals/0070-optional-requirements.md b/proposals/0070-optional-requirements.md index 443e8258b9..0fa44d5c14 100644 --- a/proposals/0070-optional-requirements.md +++ b/proposals/0070-optional-requirements.md @@ -2,8 +2,8 @@ * Proposal: [SE-0070](0070-optional-requirements.md) * Author: [Doug Gregor](https://github.com/DougGregor) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0070-make-optional-requirements-objective-c-only/2426) * Bug: [SR-1395](https://bugs.swift.org/browse/SR-1395) diff --git a/proposals/0071-member-keywords.md b/proposals/0071-member-keywords.md index 8e10769fd5..3ac330d7b0 100644 --- a/proposals/0071-member-keywords.md +++ b/proposals/0071-member-keywords.md @@ -3,7 +3,7 @@ * Proposal: [SE-0071](0071-member-keywords.md) * Author: [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0071-allow-most-keywords-in-member-references/2421) ## Introduction diff --git a/proposals/0072-eliminate-implicit-bridging-conversions.md b/proposals/0072-eliminate-implicit-bridging-conversions.md index 76f2f2d266..47910a312a 100644 --- a/proposals/0072-eliminate-implicit-bridging-conversions.md +++ b/proposals/0072-eliminate-implicit-bridging-conversions.md @@ -3,7 +3,7 @@ * Proposal: [SE-0072](0072-eliminate-implicit-bridging-conversions.md) * Author: [Joe Pamer](https://github.com/jopamer) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0072-fully-eliminate-implicit-bridging-conversions-from-swift/2487) * Implementation: [apple/swift#2419](https://github.com/apple/swift/pull/2419) diff --git a/proposals/0073-noescape-once.md b/proposals/0073-noescape-once.md index 9d1e03ea69..793f313095 100644 --- a/proposals/0073-noescape-once.md +++ b/proposals/0073-noescape-once.md @@ -2,7 +2,7 @@ * Proposal: [SE-0073](0073-noescape-once.md) * Authors: [Félix Cloutier](https://github.com/zneak), [Gwendal Roué](https://github.com/groue) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0073-marking-closures-as-executing-exactly-once/2575) @@ -128,7 +128,7 @@ expected. ## Not requiring exactly one execution Assuming that the main goal of this proposal is to relax initialization -requirements, a unique invocation of the closure is not stricly required. +requirements, a unique invocation of the closure is not strictly required. However the requirement of unique invocation makes the proposal simpler to understand. diff --git a/proposals/0074-binary-search.md b/proposals/0074-binary-search.md index e8d135e456..356615ad03 100644 --- a/proposals/0074-binary-search.md +++ b/proposals/0074-binary-search.md @@ -2,7 +2,7 @@ * Proposal: [SE-0074](0074-binary-search.md) * Authors: [Lorenzo Racca](https://github.com/lorenzoracca), [Jeff Hajewski](https://github.com/j-haj), [Nate Cook](https://github.com/natecook1000) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0074-implementation-of-binary-search-functions/2576) diff --git a/proposals/0075-import-test.md b/proposals/0075-import-test.md index 2ea41c3a8c..7844243832 100644 --- a/proposals/0075-import-test.md +++ b/proposals/0075-import-test.md @@ -1,8 +1,8 @@ # Adding a Build Configuration Import Test * Proposal: [SE-0075](0075-import-test.md) -* Author: [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Author: [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Implemented (Swift 4.1)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0075-adding-a-build-configuration-import-test/2683) * Bug: [SR-1560](https://bugs.swift.org/browse/SR-1560) @@ -33,7 +33,7 @@ Swift's existing set of build configurations specify platform differences, not m #endif ``` -Guarding code with operating system tests can be less future-proofed than testing for module support. Excluding OS X to use UIColor creates code that might eventually find its way to a Linux plaform. Targeting Apple platforms by inverting a test for Linux essentially broke after the introduction of `Windows` and `FreeBSD` build configurations: +Guarding code with operating system tests can be less future-proofed than testing for module support. Excluding OS X to use UIColor creates code that might eventually find its way to a Linux platform. Targeting Apple platforms by inverting a test for Linux essentially broke after the introduction of `Windows` and `FreeBSD` build configurations: ```swift // Exclusive os tests are brittle diff --git a/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md b/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md index d46f18ddd6..a84194e2f9 100644 --- a/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md +++ b/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md @@ -2,8 +2,8 @@ * Proposal: [SE-0076](0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md) * Author: [Janosch Hildebrand](https://github.com/Jnosh) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0076-add-overrides-taking-an-unsafepointer-source-to-non-destructive-copying-methods-on-unsafemutablepointer/2577) * Bug: [SR-1490](https://bugs.swift.org/browse/SR-1490) diff --git a/proposals/0077-operator-precedence.md b/proposals/0077-operator-precedence.md index 9274ca6a46..aaecb84955 100644 --- a/proposals/0077-operator-precedence.md +++ b/proposals/0077-operator-precedence.md @@ -2,14 +2,14 @@ * Proposal: [SE-0077](0077-operator-precedence.md) * Author: [Anton Zhilin](https://github.com/Anton3) -* Review Manager: [Joe Groff](http://github.com/jckarter) -* Status: **Implemented (Swift 3)** +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0077-v2-improved-operator-declarations/3321) **Revision history** -- **[v1](https://github.com/apple/swift-evolution/blob/40c2acad241106e1cfe697d0f75e1855dc9e96d5/proposals/0077-operator-precedence.md)** Initial version -- **[v2](https://github.com/apple/swift-evolution/blob/1f3ae8bfecb2ba70d30767607f0bd3279feeec90/proposals/0077-operator-precedence.md)** After the first review +- **[v1](https://github.com/swiftlang/swift-evolution/blob/40c2acad241106e1cfe697d0f75e1855dc9e96d5/proposals/0077-operator-precedence.md)** Initial version +- **[v2](https://github.com/swiftlang/swift-evolution/blob/1f3ae8bfecb2ba70d30767607f0bd3279feeec90/proposals/0077-operator-precedence.md)** After the first review - **v3** After the second review ## Introduction diff --git a/proposals/0078-rotate-algorithm.md b/proposals/0078-rotate-algorithm.md index ddec37005a..23a82ae469 100644 --- a/proposals/0078-rotate-algorithm.md +++ b/proposals/0078-rotate-algorithm.md @@ -2,10 +2,10 @@ * Proposal: [SE-0078](0078-rotate-algorithm.md) * Authors: [Nate Cook](https://github.com/natecook1000), [Sergey Bolshedvorsky](https://github.com/bolshedvorsky) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Deferred** -* Decision Notes: [Rationale](https://forums.swift.org/t/deferred-se-0078-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/2744) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/f5936651da1a08e2335a4991831db61da29aba15/proposals/0078-rotate-algorithm.md), [2](https://github.com/apple/swift-evolution/blob/8d45024ed7baacce94e22080d74f136bebc5c075/proposals/0078-rotate-algorithm.md) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Returned for revision** +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/f5936651da1a08e2335a4991831db61da29aba15/proposals/0078-rotate-algorithm.md), [2](https://github.com/swiftlang/swift-evolution/blob/8d45024ed7baacce94e22080d74f136bebc5c075/proposals/0078-rotate-algorithm.md) +* Review: ([pitch](https://forums.swift.org/t/proposal-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/491)) ([review](https://forums.swift.org/t/review-se-0078-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/2440)) ([return for revision](https://forums.swift.org/t/review-se-0078-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/2440/3)) ([immediate deferral](https://forums.swift.org/t/deferred-se-0078-implement-a-rotate-algorithm-equivalent-to-std-rotate-in-c/2744)) ([return for revision #2](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction diff --git a/proposals/0080-failable-numeric-initializers.md b/proposals/0080-failable-numeric-initializers.md index fcc402eb2f..f03aeec441 100644 --- a/proposals/0080-failable-numeric-initializers.md +++ b/proposals/0080-failable-numeric-initializers.md @@ -2,7 +2,7 @@ * Proposal: [SE-0080](0080-failable-numeric-initializers.md) * Author: [Matthew Johnson](https://github.com/anandabits) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Implemented (Swift 3.1)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0080-failable-numeric-conversion-initializers/2578) * Bug: [SR-1491](https://bugs.swift.org/browse/SR-1491) diff --git a/proposals/0081-move-where-expression.md b/proposals/0081-move-where-expression.md index e8ada12953..700590d4cb 100644 --- a/proposals/0081-move-where-expression.md +++ b/proposals/0081-move-where-expression.md @@ -2,8 +2,8 @@ * Proposal: [SE-0081](0081-move-where-expression.md) * Authors: [David Hart](https://github.com/hartbit), [Robert Widmann](https://github.com/CodaFi), [Pyry Jahkola](https://github.com/pyrtsa) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0081-move-where-clause-to-end-of-declaration/2685) * Bug: [SR-1561](https://bugs.swift.org/browse/SR-1561) diff --git a/proposals/0082-swiftpm-package-edit.md b/proposals/0082-swiftpm-package-edit.md index ff458171b2..5e3e24bfed 100644 --- a/proposals/0082-swiftpm-package-edit.md +++ b/proposals/0082-swiftpm-package-edit.md @@ -93,7 +93,7 @@ This solution is intended to directly address the desired behaviors of the package manager: * By hiding the sources by default, we minimize the distractions in the common - case where a user is programming against a known, well-establised, library + case where a user is programming against a known, well-established, library they do not need to modify. * By adding a new, explicit workflow for switching to an "editable" package, we diff --git a/proposals/0083-remove-bridging-from-dynamic-casts.md b/proposals/0083-remove-bridging-from-dynamic-casts.md index 8e6e110b03..eb64f9e2b1 100644 --- a/proposals/0083-remove-bridging-from-dynamic-casts.md +++ b/proposals/0083-remove-bridging-from-dynamic-casts.md @@ -2,9 +2,9 @@ * Proposal: [SE-0083](0083-remove-bridging-from-dynamic-casts.md) * Author: [Joe Groff](https://github.com/jckarter) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Deferred** -* Decision Notes: [Rationale](https://forums.swift.org/t/deferred-to-later-in-swift-3-se-0083-remove-bridging-conversion-behavior-from-dynamic-casts/2780) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Rejected** +* Review: ([pitch](https://forums.swift.org/t/pitch-reducing-the-bridging-magic-in-dynamic-casts/2398)) ([review](https://forums.swift.org/t/review-se-0083-remove-bridging-conversion-behavior-from-dynamic-casts/2544)) ([deferral](https://forums.swift.org/t/deferred-to-later-in-swift-3-se-0083-remove-bridging-conversion-behavior-from-dynamic-casts/2780)) ([rejection](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction @@ -16,8 +16,6 @@ easier to understand. To replace this functionality, initializers should be added to bridged types, providing an interface for these conversions that's more consistent with the conventions of the standard library. -Swift-evolution thread: [Reducing the bridging magic in dynamic casts](https://forums.swift.org/t/pitch-reducing-the-bridging-magic-in-dynamic-casts/2398) - ## Motivation When we introduced Swift, we wanted to provide value types for common diff --git a/proposals/0084-trailing-commas.md b/proposals/0084-trailing-commas.md index f651d39336..44b0d5d01b 100644 --- a/proposals/0084-trailing-commas.md +++ b/proposals/0084-trailing-commas.md @@ -1,8 +1,8 @@ # Allow trailing commas in parameter lists and tuples * Proposal: [SE-0084](0084-trailing-commas.md) -* Authors: [Grant Paul](https://github.com/grp), [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Authors: [Grant Paul](https://github.com/grp), [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0084-allow-trailing-commas-in-parameter-lists-and-tuples/2777) @@ -95,4 +95,4 @@ The acceptance of SE-0084 will not affect existing code. * Chris Lattner: A narrower way to solve the same problem would be to allow a comma before the `)`, but *only* when there is a newline between them. -* Vlad S suggests introducing "newlines as separators for any comma-separated list, not limited by funcs/typles but also array/dicts/generic type list etc." +* Vlad S suggests introducing "newlines as separators for any comma-separated list, not limited by funcs/tuples but also array/dicts/generic type list, etc." diff --git a/proposals/0085-package-manager-command-name.md b/proposals/0085-package-manager-command-name.md index 23bb18e9b6..44c299dbe5 100644 --- a/proposals/0085-package-manager-command-name.md +++ b/proposals/0085-package-manager-command-name.md @@ -1,9 +1,9 @@ # Package Manager Command Names * Proposal: [SE-0085](0085-package-manager-command-name.md) -* Authors: [Rick Ballard](https://github.com/rballard), [Daniel Dunbar](http://github.com/ddunbar) -* Review Manager: [Daniel Dunbar](http://github.com/ddunbar) -* Status: **Implemented (Swift 3)** +* Authors: [Rick Ballard](https://github.com/rballard), [Daniel Dunbar](https://github.com/ddunbar) +* Review Manager: [Daniel Dunbar](https://github.com/ddunbar) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/review-se-0085-package-manager-command-names/2530/6) * Implementation: [apple/swift-package-manager#364](https://github.com/apple/swift-package-manager/pull/364) diff --git a/proposals/0086-drop-foundation-ns.md b/proposals/0086-drop-foundation-ns.md index b8f1da7186..1fb18ea38a 100644 --- a/proposals/0086-drop-foundation-ns.md +++ b/proposals/0086-drop-foundation-ns.md @@ -3,7 +3,7 @@ * Proposal: [SE-0086](0086-drop-foundation-ns.md) * Authors: [Tony Parker](https://github.com/parkera), [Philippe Hausler](https://github.com/phausler) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0086-drop-ns-prefix-in-swift-foundation/3382) ##### Related radars or Swift bugs diff --git a/proposals/0087-lazy-attribute.md b/proposals/0087-lazy-attribute.md index d6de4a1042..161687367e 100644 --- a/proposals/0087-lazy-attribute.md +++ b/proposals/0087-lazy-attribute.md @@ -2,7 +2,7 @@ * Proposal: [SE-0087](0087-lazy-attribute.md) * Author: [Anton3](https://github.com/Anton3) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0087-rename-lazy-to-lazy/2778) diff --git a/proposals/0088-libdispatch-for-swift3.md b/proposals/0088-libdispatch-for-swift3.md index 0d1cd5c8e0..19bfd9d3bc 100644 --- a/proposals/0088-libdispatch-for-swift3.md +++ b/proposals/0088-libdispatch-for-swift3.md @@ -2,10 +2,10 @@ * Proposal: [SE-0088](0088-libdispatch-for-swift3.md) * Author: [Matt Wright](https://github.com/mwwa) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0088-modernize-libdispatch-for-swift-3-naming-conventions/2697) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/ef372026d5f7e46848eb2a64f292328028b667b9/proposals/0088-libdispatch-for-swift3.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/ef372026d5f7e46848eb2a64f292328028b667b9/proposals/0088-libdispatch-for-swift3.md) ## Introduction @@ -259,7 +259,7 @@ struct DispatchData : RandomAccessCollection, _ObjectiveCBridgeable { } ``` -This proposal will introduce new accessor methods to access the bytes in a Data object. Along with becoming iteratable, several methods will be introduced that replace the ```dispatch_data_create_map``` approach used in C: +This proposal will introduce new accessor methods to access the bytes in a Data object. Along with becoming iterable, several methods will be introduced that replace the ```dispatch_data_create_map``` approach used in C: ```swift struct DispatchData : RandomAccessCollection, _ObjectiveCBridgeable { diff --git a/proposals/0089-rename-string-reflection-init.md b/proposals/0089-rename-string-reflection-init.md index 5c627f9c20..e6808a3683 100644 --- a/proposals/0089-rename-string-reflection-init.md +++ b/proposals/0089-rename-string-reflection-init.md @@ -2,11 +2,11 @@ * Proposal: [SE-0089](0089-rename-string-reflection-init.md) * Authors: [Austin Zheng](https://github.com/austinzheng), [Becca Royal-Gordon](https://github.com/beccadax) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0089-renaming-string-init-t-t/3097) * Bug: [SR-1881](https://bugs.swift.org/browse/SR-1881) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/40aecf3647c19ae37730e39aa9e54b67fcc2be86/proposals/0089-rename-string-reflection-init.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/40aecf3647c19ae37730e39aa9e54b67fcc2be86/proposals/0089-rename-string-reflection-init.md) ## Introduction diff --git a/proposals/0090-remove-dot-self.md b/proposals/0090-remove-dot-self.md index 3bf2f5166f..c736c5069c 100644 --- a/proposals/0090-remove-dot-self.md +++ b/proposals/0090-remove-dot-self.md @@ -2,10 +2,9 @@ * Proposal: [SE-0090](0090-remove-dot-self.md) * Authors: [Joe Groff](https://github.com/jckarter), [Tanner Nelson](https://github.com/tannernelson) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Deferred** -* Decision Notes: [Rationale](https://forums.swift.org/t/deferred-se-0090-remove-self-and-freely-allow-type-references-in-expressions/2781) -* Revision: 2 +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Returned for revision** +* Review: ([pitch](https://forums.swift.org/t/making-self-after-type-optional/1737)) ([review](https://forums.swift.org/t/review-se-0090-remove-self-and-freely-allow-type-references-in-expressions/2664)) ([deferral](https://forums.swift.org/t/deferred-se-0090-remove-self-and-freely-allow-type-references-in-expressions/2781)) ([return for revision](https://forums.swift.org/t/returning-or-rejecting-all-the-deferred-evolution-proposals/60724)) ## Introduction @@ -15,8 +14,6 @@ for `T`, one must refer to the special member `T.self`. I propose allowing type references to appear freely in expressions and removing the `.self` member from the language. -Swift-evolution thread: [Making `.self` After `Type` Optional](https://forums.swift.org/t/making-self-after-type-optional/1737) - ## Motivation The constructor-or-member restriction on type references exists to provide diff --git a/proposals/0091-improving-operators-in-protocols.md b/proposals/0091-improving-operators-in-protocols.md index 563722a9f1..940e84b6c2 100644 --- a/proposals/0091-improving-operators-in-protocols.md +++ b/proposals/0091-improving-operators-in-protocols.md @@ -2,11 +2,11 @@ * Proposal: [SE-0091](0091-improving-operators-in-protocols.md) * Authors: [Tony Allevato](https://github.com/allevato), [Doug Gregor](https://github.com/DougGregor) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0091-improving-operator-requirements-in-protocols/3390) * Bug: [SR-2073](https://bugs.swift.org/browse/SR-2073) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/eaab20ed34df1dc8ba8aa07e49abc8c5fa216f3e/proposals/0091-improving-operators-in-protocols.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/eaab20ed34df1dc8ba8aa07e49abc8c5fa216f3e/proposals/0091-improving-operators-in-protocols.md) ## Introduction diff --git a/proposals/0092-typealiases-in-protocols.md b/proposals/0092-typealiases-in-protocols.md index 016c03f362..d6d3912fc7 100644 --- a/proposals/0092-typealiases-in-protocols.md +++ b/proposals/0092-typealiases-in-protocols.md @@ -2,8 +2,8 @@ * Proposal: [SE-0092](0092-typealiases-in-protocols.md) * Authors: [David Hart](https://github.com/hartbit), [Doug Gregor](https://github.com/DougGregor) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0092-typealiases-in-protocols-and-protocol-extensions/2639) * Bug: [SR-1539](https://bugs.swift.org/browse/SR-1539) diff --git a/proposals/0093-slice-base.md b/proposals/0093-slice-base.md index 6d9035777c..361c42c159 100644 --- a/proposals/0093-slice-base.md +++ b/proposals/0093-slice-base.md @@ -3,7 +3,7 @@ * Proposal: [SE-0093](0093-slice-base.md) * Author: [Max Moiseev](https://github.com/moiseev) * Review Manager: [Dave Abrahams](https://github.com/dabrahams) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/review-se-0093-adding-a-public-base-property-to-slices/2695/4) * Implementation: [apple/swift#2929](https://github.com/apple/swift/pull/2929) diff --git a/proposals/0094-sequence-function.md b/proposals/0094-sequence-function.md index 5070e09c64..f96272e1da 100644 --- a/proposals/0094-sequence-function.md +++ b/proposals/0094-sequence-function.md @@ -1,12 +1,12 @@ # Add sequence(first:next:) and sequence(state:next:) to the stdlib * Proposal: [SE-0094](0094-sequence-function.md) -* Authors: [Lily Ballard](https://github.com/lilyball), [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Authors: [Lily Ballard](https://github.com/lilyball), [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0094-add-sequence-initial-next-and-sequence-state-next-to-the-stdlib/2775) * Bug: [SR-1622](https://bugs.swift.org/browse/SR-1622) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/7d220a152a681e28761493c7d9781dd867a04cf7/proposals/0094-sequence-function.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/7d220a152a681e28761493c7d9781dd867a04cf7/proposals/0094-sequence-function.md) * Previous Proposal: [SE-0045](0045-scan-takewhile-dropwhile.md) ## Introduction @@ -23,7 +23,7 @@ Swift-evolution thread: ## Motivation -[SE-0045](0045-scan-takewhile-dropwhile.md), originally proposed `iterate(_:apply:)` (see [SE-0045r1](https://github.com/apple/swift-evolution/blob/dd0a39dd051b11e4460accad5af0e74223533e95/proposals/0045-scan-takewhile-dropwhile.md)), a method that +[SE-0045](0045-scan-takewhile-dropwhile.md), originally proposed `iterate(_:apply:)` (see [SE-0045r1](https://github.com/swiftlang/swift-evolution/blob/dd0a39dd051b11e4460accad5af0e74223533e95/proposals/0045-scan-takewhile-dropwhile.md)), a method that was subsequently changed to `unfold(_:applying:)`. The proposal was accepted with modifications. The core team rejected `unfold` based on its naming. As its core utility remains unquestionably high, this proposal re-introduces the method with better, more Swift-appropriate naming. @@ -50,8 +50,8 @@ See also: * [SE-0007 Remove C-style For Loops](0007-remove-c-style-for-loops.md), * [SE-0045](0045-scan-takewhile-dropwhile.md), -* [SE-0045r1](https://github.com/apple/swift-evolution/blob/b39d653f7e3d5e982b562664343f26c826652291/proposals/0045-scan-takewhile-dropwhile.md), -* [SE-0045r3](https://github.com/apple/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0045-scan-takewhile-dropwhile.md) +* [SE-0045r1](https://github.com/swiftlang/swift-evolution/blob/b39d653f7e3d5e982b562664343f26c826652291/proposals/0045-scan-takewhile-dropwhile.md), +* [SE-0045r3](https://github.com/swiftlang/swift-evolution/blob/d709546002e1636a10350d14da84eb9e554c3aac/proposals/0045-scan-takewhile-dropwhile.md) ## Detailed design diff --git a/proposals/0095-any-as-existential.md b/proposals/0095-any-as-existential.md index 1f74a7e05f..d8379f45f6 100644 --- a/proposals/0095-any-as-existential.md +++ b/proposals/0095-any-as-existential.md @@ -2,11 +2,11 @@ * Proposal: [SE-0095](0095-any-as-existential.md) * Authors: [Adrian Zubarev](https://github.com/DevAndArtist), [Austin Zheng](https://github.com/austinzheng) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0095-replace-protocol-p1-p2-syntax-with-p1-p2-syntax/3198) * Bug: [SR-1938](https://bugs.swift.org/browse/SR-1938) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/a4356fee94c06181715fad83aa61e923eb73f8ec/proposals/0095-any-as-existential.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/a4356fee94c06181715fad83aa61e923eb73f8ec/proposals/0095-any-as-existential.md) ## Introduction diff --git a/proposals/0096-dynamictype.md b/proposals/0096-dynamictype.md index a2daa67db9..bc7a45ba8d 100644 --- a/proposals/0096-dynamictype.md +++ b/proposals/0096-dynamictype.md @@ -2,8 +2,8 @@ * Proposal: [SE-0096](0096-dynamictype.md) * Author: [Erica Sadun](https://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0098-converting-dynamictype-from-a-property-to-an-operator/2853) * Bug: [SR-2218](https://bugs.swift.org/browse/SR-2218) diff --git a/proposals/0097-negative-attributes.md b/proposals/0097-negative-attributes.md index 6a4d3c19cd..6a902d535a 100644 --- a/proposals/0097-negative-attributes.md +++ b/proposals/0097-negative-attributes.md @@ -2,7 +2,7 @@ * Proposal: [SE-0097](0097-negative-attributes.md) * Author: [Erica Sadun](https://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0097-normalizing-naming-for-negative-attributes/2854) diff --git a/proposals/0098-didset-capitalization.md b/proposals/0098-didset-capitalization.md index d1a3cfff8e..52e9279924 100644 --- a/proposals/0098-didset-capitalization.md +++ b/proposals/0098-didset-capitalization.md @@ -2,7 +2,7 @@ * Proposal: [SE-0098](0098-didset-capitalization.md) * Author: [Erica Sadun](https://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0098-lowercase-didset-and-willset-for-more-consistent-keyword-casing/2852) diff --git a/proposals/0099-conditionclauses.md b/proposals/0099-conditionclauses.md index dd12a4e19e..95ca694357 100644 --- a/proposals/0099-conditionclauses.md +++ b/proposals/0099-conditionclauses.md @@ -3,13 +3,13 @@ * Proposal: [SE-0099](0099-conditionclauses.md) * Authors: [Erica Sadun](https://github.com/erica), [Chris Lattner](https://github.com/lattner) * Review Manager: [Joe Groff](https://github.com/jckarter) -* Status: **Implemented (Swift 3)** -* Decision Notes: [Rationale](#rationale) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/83053c5f5395987caf2ecb3830a5cd8dc6213237/proposals/0099-conditionclauses.md) +* Status: **Implemented (Swift 3.0)** +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/83053c5f5395987caf2ecb3830a5cd8dc6213237/proposals/0099-conditionclauses.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-making-where-and-interchangeable-in-guard-conditions/2702)), ([review](https://forums.swift.org/t/review-se-0099-restructuring-condition-clauses/2808)), ([acceptance](https://forums.swift.org/t/accepted-with-revision-se-0099-restructuring-condition-clauses/2921)) ## Introduction -Swift condition clauses appear in `guard`, `if`, and `while` statements. This proposal re-architects the condition grammar to enable an arbitrary mix of Boolean expressions, `let` conditions (which test and unwrap optionals), general `case` clauses for arbitrary pattern matching, and availability tests. It removes `where` clauses from optional binding conditions and case conditions, and eliminates gramatical ambiguity by using commas for separation between clauses instead of using them both to separate clauses and terms within each clause. These modifications streamline Swift's syntax and alleviate the situation where many Swift developers don't know they can use arbitrary Boolean conditions after a value binding. +Swift condition clauses appear in `guard`, `if`, and `while` statements. This proposal re-architects the condition grammar to enable an arbitrary mix of Boolean expressions, `let` conditions (which test and unwrap optionals), general `case` clauses for arbitrary pattern matching, and availability tests. It removes `where` clauses from optional binding conditions and case conditions, and eliminates grammatical ambiguity by using commas for separation between clauses instead of using them both to separate clauses and terms within each clause. These modifications streamline Swift's syntax and alleviate the situation where many Swift developers don't know they can use arbitrary Boolean conditions after a value binding. Swift-evolution thread: [\[Pitch\] making where and , interchangeable in guard conditions](https://forums.swift.org/t/pitch-making-where-and-interchangeable-in-guard-conditions/2702) diff --git a/proposals/0101-standardizing-sizeof-naming.md b/proposals/0101-standardizing-sizeof-naming.md index 280e7d45bd..ec656c098b 100644 --- a/proposals/0101-standardizing-sizeof-naming.md +++ b/proposals/0101-standardizing-sizeof-naming.md @@ -1,9 +1,9 @@ # Reconfiguring `sizeof` and related functions into a unified `MemoryLayout` struct * Proposal: [SE-0101](0101-standardizing-sizeof-naming.md) -* Authors: [Erica Sadun](http://github.com/erica), [Dave Abrahams](https://github.com/dabrahams) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Authors: [Erica Sadun](https://github.com/erica), [Dave Abrahams](https://github.com/dabrahams) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0101-reconfiguring-sizeof-and-related-functions-into-a-unified-memorylayout-struct/3477) ## Introduction @@ -13,7 +13,7 @@ This proposal addresses `sizeof`, `sizeofValue`, `strideof`, `strideofValue`, `a Review 1: * [Swift Evolution Review Thread](https://forums.swift.org/t/review-se-0101-rename-sizeof-and-related-functions-to-comply-with-api-guidelines/3060) -* [Original Proposal](https://github.com/apple/swift-evolution/blob/26e1e5b546b13fb66ee8877ad7018a7856e467ca/proposals/0101-standardizing-sizeof-naming.md) +* [Original Proposal](https://github.com/swiftlang/swift-evolution/blob/26e1e5b546b13fb66ee8877ad7018a7856e467ca/proposals/0101-standardizing-sizeof-naming.md) Prior Discussions: diff --git a/proposals/0102-noreturn-bottom-type.md b/proposals/0102-noreturn-bottom-type.md index b73de7bfbb..4a91344483 100644 --- a/proposals/0102-noreturn-bottom-type.md +++ b/proposals/0102-noreturn-bottom-type.md @@ -2,8 +2,8 @@ * Proposal: [SE-0102](0102-noreturn-bottom-type.md) * Author: [Joe Groff](https://github.com/jckarter) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0102-remove-noreturn-attribute-and-introduce-an-empty-never-type/3213) * Bug: [SR-1953](https://bugs.swift.org/browse/SR-1953) diff --git a/proposals/0103-make-noescape-default.md b/proposals/0103-make-noescape-default.md index e6e9f634cf..1948030cd1 100644 --- a/proposals/0103-make-noescape-default.md +++ b/proposals/0103-make-noescape-default.md @@ -2,11 +2,11 @@ * Proposal: [SE-0103](0103-make-noescape-default.md) * Author: [Trent Nadeau](https://github.com/tanadeau) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0103-make-non-escaping-closures-the-default/3212) * Bug: [SR-1952](https://bugs.swift.org/browse/SR-1952) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/833afd64b5d24a777fe2c42800d4b4dcd52bb487/proposals/0103-make-noescape-default.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/833afd64b5d24a777fe2c42800d4b4dcd52bb487/proposals/0103-make-noescape-default.md) ## Introduction diff --git a/proposals/0104-improved-integers.md b/proposals/0104-improved-integers.md index c8253fee25..5dbf33b262 100644 --- a/proposals/0104-improved-integers.md +++ b/proposals/0104-improved-integers.md @@ -3,12 +3,12 @@ * Proposal: [SE-0104](0104-improved-integers.md) * Authors: [Dave Abrahams](https://github.com/dabrahams), [Maxim Moiseev](https://github.com/moiseev) * Review Manager: [Ben Cohen](https://github.com/airspeedswift) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Bug: [SR-3196](https://bugs.swift.org/browse/SR-3196) * Previous Revisions: - [1](https://github.com/apple/swift-evolution/blob/0440700fc555a6c72abb4af807c8b79fb1bec592/proposals/0104-improved-integers.md), - [2](https://github.com/apple/swift-evolution/blob/957ab545e05adb94507792e7871b38e34b56a0a5/proposals/0104-improved-integers.md), - [3](https://github.com/apple/swift-evolution/blob/80f57a6b7645126fe0220dcb91c19565e447d5d8/proposals/0104-improved-integers.md) + [1](https://github.com/swiftlang/swift-evolution/blob/0440700fc555a6c72abb4af807c8b79fb1bec592/proposals/0104-improved-integers.md), + [2](https://github.com/swiftlang/swift-evolution/blob/957ab545e05adb94507792e7871b38e34b56a0a5/proposals/0104-improved-integers.md), + [3](https://github.com/swiftlang/swift-evolution/blob/80f57a6b7645126fe0220dcb91c19565e447d5d8/proposals/0104-improved-integers.md) * Discussion on swift-evolution: [here](https://forums.swift.org/t/protocol-oriented-integers-take-2/4884). * Decision notes: [Rationale](https://forums.swift.org/t/accepted-se-0104-protocol-oriented-integers/5346) diff --git a/proposals/0105-remove-where-from-forin-loops.md b/proposals/0105-remove-where-from-forin-loops.md index a3f29187a6..3df0c2d63e 100644 --- a/proposals/0105-remove-where-from-forin-loops.md +++ b/proposals/0105-remove-where-from-forin-loops.md @@ -1,8 +1,8 @@ # Removing Where Clauses from For-In Loops * Proposal: [SE-0105](0105-remove-where-from-forin-loops.md) -* Author: [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Author: [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0105-removing-where-clauses-from-for-in-loops/3205) diff --git a/proposals/0106-rename-osx-to-macos.md b/proposals/0106-rename-osx-to-macos.md index c04de3c4c1..e5f4714024 100644 --- a/proposals/0106-rename-osx-to-macos.md +++ b/proposals/0106-rename-osx-to-macos.md @@ -1,9 +1,9 @@ # Add a `macOS` Alias for the `OSX` Platform Configuration Test * Proposal: [SE-0106](0106-rename-osx-to-macos.md) -* Author: [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Author: [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0106-add-a-macos-alias-for-the-osx-platform-configuration-test/3176) * Bugs: [SR-1823](https://bugs.swift.org/browse/SR-1823), [SR-1887](https://bugs.swift.org/browse/SR-1887) @@ -73,7 +73,7 @@ This proposal is purely additive. It will not affect existing code other than ad Instead of retaining and aliasing `os(OSX)`, it can be fully replaced by `os(macOS)`. This mirrors the situation with the phoneOS to iOS rename and would require a migration assistant to fixit old-style use. -Charlie Monroe points out: "Since Swift 3.0 is a code-breaking change my guess is that there is no burden if the Xcode migration assistent automatically changes all `#if os(OSX)` to `#if os(macOS)`, thus deprecating the term OSX, not burdening the developer at all. If iOS was renamed to phoneOS and kept versioning, you'd still expect `#if os(iOS)` to be matched when targeting phoneOS and vice-versa." +Charlie Monroe points out: "Since Swift 3.0 is a code-breaking change my guess is that there is no burden if the Xcode migration assistant automatically changes all `#if os(OSX)` to `#if os(macOS)`, thus deprecating the term OSX, not burdening the developer at all. If iOS was renamed to phoneOS and kept versioning, you'd still expect `#if os(iOS)` to be matched when targeting phoneOS and vice-versa." ## Unaddressed Issues diff --git a/proposals/0107-unsaferawpointer.md b/proposals/0107-unsaferawpointer.md index e70f518f54..6acefab32f 100644 --- a/proposals/0107-unsaferawpointer.md +++ b/proposals/0107-unsaferawpointer.md @@ -2,8 +2,8 @@ * Proposal: [SE-0107](0107-unsaferawpointer.md) * Author: [Andrew Trick](https://github.com/atrick) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0107-unsaferawpointer-api/3389) For detailed instructions on how to migrate your code to this new @@ -160,7 +160,7 @@ value argument could result in miscompilation if the inferred type ever deviates from the user's original expectations. The type parameter also importantly conveys that the raw memory becomes accessible via a pointer to that type at the point of the call. The -type should be explicitly spelled at this point because accesing the +type should be explicitly spelled at this point because accessing the memory via a typed pointer of an unrelated type could also result in miscompilation. diff --git a/proposals/0108-remove-assoctype-inference.md b/proposals/0108-remove-assoctype-inference.md index 082ab85073..b1373d8a75 100644 --- a/proposals/0108-remove-assoctype-inference.md +++ b/proposals/0108-remove-assoctype-inference.md @@ -2,7 +2,7 @@ * Proposal: [SE-0108](0108-remove-assoctype-inference.md) * Authors: [Douglas Gregor](https://github.com/DougGregor), Austin Zheng -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0108-remove-associated-type-inference/3304) @@ -165,7 +165,7 @@ There are some advantages to this approach. Brevity is slightly improved. A type As well, Dave Abrahams expresses a [potential issue](https://forums.swift.org/t/pitch-remove-type-inference-for-associated-types/3135/17): -> Finally, I am very concerned that there are protocols such as `Collection`, with many inferrable associated types, and that conforming to these protocols could become *much* uglier. +> Finally, I am very concerned that there are protocols such as `Collection`, with many inferable associated types, and that conforming to these protocols could become *much* uglier. As with many proposals, there is a tradeoff between the status quo and the proposed behavior. As *Completing Generics* puts it, @@ -173,4 +173,4 @@ As with many proposals, there is a tradeoff between the status quo and the propo ### Require explicit declaration using `associatedtype` -An [earlier draft of this proposal](https://github.com/apple/swift-evolution/blob/18a1781d930034583ffc0325a180099f15fbb834/proposals/XXXX-remove-assoctype-inference.md) detailed a design in which types would explicitly bind their associated types using an `associatedtype` declaration. It is presented as an alternative for consideration. +An [earlier draft of this proposal](https://github.com/swiftlang/swift-evolution/blob/18a1781d930034583ffc0325a180099f15fbb834/proposals/XXXX-remove-assoctype-inference.md) detailed a design in which types would explicitly bind their associated types using an `associatedtype` declaration. It is presented as an alternative for consideration. diff --git a/proposals/0109-remove-boolean.md b/proposals/0109-remove-boolean.md index 8534b28354..89a62aabd3 100644 --- a/proposals/0109-remove-boolean.md +++ b/proposals/0109-remove-boolean.md @@ -2,8 +2,8 @@ * Proposal: [SE-0109](0109-remove-boolean.md) * Authors: [Anton Zhilin](https://github.com/Anton3), [Chris Lattner](https://github.com/lattner) -* Review Manager: [Doug Gregor](http://github.com/DougGregor) -* Status: **Implemented (Swift 3)** +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0109-remove-the-boolean-protoco/3380) * Implementation: [apple/swift@76cf339](https://github.com/apple/swift/commit/76cf339694a41293dbbec9672b6df87a864087f2), [apple/swift@af30ae3](https://github.com/apple/swift/commit/af30ae32226813ec14c2bef80cb090d3e6c586fb) diff --git a/proposals/0110-distingish-single-tuple-arg.md b/proposals/0110-distinguish-single-tuple-arg.md similarity index 83% rename from proposals/0110-distingish-single-tuple-arg.md rename to proposals/0110-distinguish-single-tuple-arg.md index ed7c4b9bb2..be56d47d1c 100644 --- a/proposals/0110-distingish-single-tuple-arg.md +++ b/proposals/0110-distinguish-single-tuple-arg.md @@ -1,12 +1,12 @@ # Distinguish between single-tuple and multiple-argument function types -* Proposal: [SE-0110](0110-distingish-single-tuple-arg.md) +* Proposal: [SE-0110](0110-distinguish-single-tuple-arg.md) * Authors: Vladimir S., [Austin Zheng](https://github.com/austinzheng) * Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Implemented** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0110-distinguish-between-single-tuple-and-multiple-argument-function-types/3305), [Additional Commentary](https://forums.swift.org/t/core-team-addressing-the-se-0110-usability-regression-in-swift-4/6147) * Bug: [SR-2008](https://bugs.swift.org/browse/SR-2008) -* Previous Revision: [Originally Accepted Proposal](https://github.com/apple/swift-evolution/blob/9e44932452e1daead98f2bc2e58711eb489e9751/proposals/0110-distingish-single-tuple-arg.md) +* Previous Revision: [Originally Accepted Proposal](https://github.com/swiftlang/swift-evolution/blob/9e44932452e1daead98f2bc2e58711eb489e9751/proposals/0110-distingish-single-tuple-arg.md) ## Introduction @@ -67,4 +67,4 @@ Don't make this change. ## Revision history -The [original proposal as reviewed](https://github.com/apple/swift-evolution/blob/9e44932452e1daead98f2bc2e58711eb489e9751/proposals/0110-distingish-single-tuple-arg.md) did not include the special-case conversion from `(T, U, ...) -> V` to `((T, U, ...)) -> V` for function arguments. In response to community feedback, [this conversion was added](https://forums.swift.org/t/core-team-addressing-the-se-0110-usability-regression-in-swift-4/6147) as part of the Core Team's acceptance of the proposal. +The [original proposal as reviewed](https://github.com/swiftlang/swift-evolution/blob/9e44932452e1daead98f2bc2e58711eb489e9751/proposals/0110-distingish-single-tuple-arg.md) did not include the special-case conversion from `(T, U, ...) -> V` to `((T, U, ...)) -> V` for function arguments. In response to community feedback, [this conversion was added](https://forums.swift.org/t/core-team-addressing-the-se-0110-usability-regression-in-swift-4/6147) as part of the Core Team's acceptance of the proposal. diff --git a/proposals/0111-remove-arg-label-type-significance.md b/proposals/0111-remove-arg-label-type-significance.md index 29ba3c20b6..0c6db853bd 100644 --- a/proposals/0111-remove-arg-label-type-significance.md +++ b/proposals/0111-remove-arg-label-type-significance.md @@ -2,8 +2,8 @@ * Proposal: [SE-0111](0111-remove-arg-label-type-significance.md) * Author: Austin Zheng -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0111-remove-type-system-significance-of-function-argument-labels/3306), [Additional Commentary](https://forums.swift.org/t/update-commentary-se-0111-remove-type-system-significance-of-function-argument-labels/3391) * Bug: [SR-2009](https://bugs.swift.org/browse/SR-2009) diff --git a/proposals/0112-nserror-bridging.md b/proposals/0112-nserror-bridging.md index ac806b8268..da41dc387f 100644 --- a/proposals/0112-nserror-bridging.md +++ b/proposals/0112-nserror-bridging.md @@ -2,8 +2,8 @@ * Proposal: [SE-0112](0112-nserror-bridging.md) * Authors: [Doug Gregor](https://github.com/DougGregor), [Charles Srstka](https://github.com/CharlesJS) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0112-improved-nserror-bridging/3362) ## Introduction @@ -40,7 +40,7 @@ Swift-evolution thread: [Charles Srstka's pitch for Consistent bridging for NSErrors at the language boundary](https://forums.swift.org/t/pitch-consistent-bridging-for-nserrors-at-the-language-boundary/2482), which discussed Charles' [original -proposal](https://github.com/apple/swift-evolution/pull/331) that +proposal](https://github.com/swiftlang/swift-evolution/pull/331) that addressed these issues by providing ``NSError`` to ``ErrorProtocol`` bridging and exposing the domain, code, and user-info dictionary for all errors. This proposal expands upon that work, but without directly diff --git a/proposals/0113-rounding-functions-on-floatingpoint.md b/proposals/0113-rounding-functions-on-floatingpoint.md index 833eac6849..d3d78f5036 100644 --- a/proposals/0113-rounding-functions-on-floatingpoint.md +++ b/proposals/0113-rounding-functions-on-floatingpoint.md @@ -2,8 +2,8 @@ * Proposal: [SE-0113](0113-rounding-functions-on-floatingpoint.md) * Author: [Karl Wagner](https://github.com/karwa) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0113-add-integral-rounding-functions-to-floatingpoint/3308) * Bug: [SR-2010](https://bugs.swift.org/browse/SR-2010) diff --git a/proposals/0114-buffer-naming.md b/proposals/0114-buffer-naming.md index 3997103db4..1be984c287 100644 --- a/proposals/0114-buffer-naming.md +++ b/proposals/0114-buffer-naming.md @@ -1,9 +1,9 @@ # Updating Buffer "Value" Names to "Header" Names * Proposal: [SE-0114](0114-buffer-naming.md) -* Author: [Erica Sadun](http://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Author: [Erica Sadun](https://github.com/erica) +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0114-updating-buffer-value-names-to-header-names/3359) * Implementation: [apple/swift#3374](https://github.com/apple/swift/pull/3374) diff --git a/proposals/0115-literal-syntax-protocols.md b/proposals/0115-literal-syntax-protocols.md index 12e8c096b6..c5353edbdf 100644 --- a/proposals/0115-literal-syntax-protocols.md +++ b/proposals/0115-literal-syntax-protocols.md @@ -2,8 +2,8 @@ * Proposal: [SE-0115](0115-literal-syntax-protocols.md) * Author: [Matthew Johnson](https://github.com/anandabits) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0115-rename-literal-syntax-protocols/3358) * Bug: [SR-2054](https://bugs.swift.org/browse/SR-2054) @@ -29,7 +29,7 @@ Further, the standard library team has observed: > (e.g., Int or String), and as far as the user at the call site is concerned, > there is no visible conversion (even if one is happening behind the scenes). -[An earlier proposal](https://github.com/apple/swift-evolution/blob/master/proposals/0041-conversion-protocol-conventions.md) was intended to address the first problem by introducing strong naming conventions for three kinds of conversion protocols (*from*, *to*, and *bidirectional*). The review highlighted the difficulty in establishing conventions that everyone is happy with. This proposal takes a different approach to solving the problem that originally inspired that proposal while also solving the awkwardness of the current names described by the standard library team. +[An earlier proposal](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0041-conversion-protocol-conventions.md) was intended to address the first problem by introducing strong naming conventions for three kinds of conversion protocols (*from*, *to*, and *bidirectional*). The review highlighted the difficulty in establishing conventions that everyone is happy with. This proposal takes a different approach to solving the problem that originally inspired that proposal while also solving the awkwardness of the current names described by the standard library team. ## Proposed solution diff --git a/proposals/0116-id-as-any.md b/proposals/0116-id-as-any.md index 6168ac6b3c..f1cb4d528e 100644 --- a/proposals/0116-id-as-any.md +++ b/proposals/0116-id-as-any.md @@ -2,10 +2,10 @@ * Proposal: [SE-0116](0116-id-as-any.md) * Author: [Joe Groff](https://github.com/jckarter) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0116-import-objective-c-id-as-swift-any-type/3476) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/b9a0ab5f7db4d3806c7941a07acedc5f0fe36e55/proposals/0116-id-as-any.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/b9a0ab5f7db4d3806c7941a07acedc5f0fe36e55/proposals/0116-id-as-any.md) ## Introduction diff --git a/proposals/0117-non-public-subclassable-by-default.md b/proposals/0117-non-public-subclassable-by-default.md index a47ba3df15..50e2d07743 100644 --- a/proposals/0117-non-public-subclassable-by-default.md +++ b/proposals/0117-non-public-subclassable-by-default.md @@ -2,11 +2,11 @@ * Proposal: [SE-0117](0117-non-public-subclassable-by-default.md) * Authors: [Javier Soto](https://github.com/JaviSoto), [John McCall](https://github.com/rjmccall) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0117-allow-distinguishing-between-public-access-and-public-overridability/3578) * Implementation: [apple/swift#3882](https://github.com/apple/swift/pull/3882) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/367086f18a5deaf8f9dfbe3f5a4846ef19addf38/proposals/0117-non-public-subclassable-by-default.md), [2](https://github.com/apple/swift-evolution/blob/2989538daa1640cfa6a56f80b5c7599967af0905/proposals/0117-non-public-subclassable-by-default.md), [3](https://github.com/apple/swift-evolution/blob/15c18d24adb7e701ae831b643e0803f1b6e601d9/proposals/0117-non-public-subclassable-by-default.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/367086f18a5deaf8f9dfbe3f5a4846ef19addf38/proposals/0117-non-public-subclassable-by-default.md), [2](https://github.com/swiftlang/swift-evolution/blob/2989538daa1640cfa6a56f80b5c7599967af0905/proposals/0117-non-public-subclassable-by-default.md), [3](https://github.com/swiftlang/swift-evolution/blob/15c18d24adb7e701ae831b643e0803f1b6e601d9/proposals/0117-non-public-subclassable-by-default.md) ## Introduction diff --git a/proposals/0118-closure-parameter-names-and-labels.md b/proposals/0118-closure-parameter-names-and-labels.md index 66f8d57dbb..2f89190564 100644 --- a/proposals/0118-closure-parameter-names-and-labels.md +++ b/proposals/0118-closure-parameter-names-and-labels.md @@ -2,14 +2,14 @@ * Proposal: [SE-0118](0118-closure-parameter-names-and-labels.md) * Authors: [Dave Abrahams](https://github.com/dabrahams), [Dmitri Gribenko](https://github.com/gribozavr), [Maxim Moiseev](https://github.com/moiseev) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0118-closure-parameter-names-and-labels/3387) * Bug: [SR-2072](https://bugs.swift.org/browse/SR-2072) ## Revision History -- [v1](https://github.com/apple/swift-evolution/blob/ae4a55ab217cc9755004cbf2b29db24e28645d15/proposals/0118-closure-parameter-names-and-labels.md) (as proposed) +- [v1](https://github.com/swiftlang/swift-evolution/blob/ae4a55ab217cc9755004cbf2b29db24e28645d15/proposals/0118-closure-parameter-names-and-labels.md) (as proposed) - v2: fixed spelling of identifiers containing `Utf8` to read `UTF8` per convention. ## Introduction diff --git a/proposals/0119-extensions-access-modifiers.md b/proposals/0119-extensions-access-modifiers.md index a5a63aa74f..9112147c9a 100644 --- a/proposals/0119-extensions-access-modifiers.md +++ b/proposals/0119-extensions-access-modifiers.md @@ -2,21 +2,21 @@ * Proposal: [SE-0119](0119-extensions-access-modifiers.md) * Author: [Adrian Zubarev](https://github.com/DevAndArtist) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0119-remove-access-modifiers-from-extensions/3493) ## Introduction -

One great goal for Swift 3 is to sort out any source breaking language changes. This proposal aims to fix access modifier inconsistency on extensions compared to other scope declarations types.

+One great goal for Swift 3 is to sort out any source breaking language changes. This proposal aims to fix access modifier inconsistency on extensions compared to other scope declarations types. Swift-evolution thread: [\[Proposal\] Revising access modifiers on extensions](https://forums.swift.org/t/proposal-revising-access-modifiers-on-extensions/3138) ## Motivation -

The access control of classes, enums and structs in Swift is very easy to learn and memorize. It also disallows to suppress the access modifier of implemented conformance members to lower access modifier if the host type has an access modifier of higher or equal level.

+The access control of classes, enums and structs in Swift is very easy to learn and memorize. It also disallows to suppress the access modifier of implemented conformance members to lower access modifier if the host type has an access modifier of higher or equal level. -
`public` > `internal` > `fileprivate` >= `private`
+`public` > `internal` > `fileprivate` >= `private` ```swift public class A { @@ -56,7 +56,7 @@ This simple access control model also allows us to nest types inside each other *Extensions* however behave differently when it comes to their access control: -* The *access modifier* of an *extension* sets the default modifier of its members which do not have their own localy defined modifier. +* The *access modifier* of an *extension* sets the default modifier of its members which do not have their own locally defined modifier. ```swift public struct D {} @@ -234,7 +234,7 @@ I propose to revise the access control on extensions by removing access modifier fileprivate group { - // Every group memebr is `fileprivate` + // Every group member is `fileprivate` func member1() {} func member2() {} func member3() {} @@ -286,7 +286,7 @@ Iff the *access-level-modifier* is not present, the access modifier on extension ```diff - extension SomeType : SomeProtocol { + public extension SomeType : SomeProtocol { - public func someMemeber() + public func someMember() } ``` diff --git a/proposals/0120-revise-partition-method.md b/proposals/0120-revise-partition-method.md index 275f6a7497..a222aaaa45 100644 --- a/proposals/0120-revise-partition-method.md +++ b/proposals/0120-revise-partition-method.md @@ -2,11 +2,11 @@ * Proposal: [SE-0120](0120-revise-partition-method.md) * Authors: [Lorenzo Racca](https://github.com/lorenzoracca), [Jeff Hajewski](https://github.com/j-haj), [Nate Cook](https://github.com/natecook1000) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0120-revise-partition-method-signature/3475) * Bug: [SR-1965](https://bugs.swift.org/browse/SR-1965) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/1dcfd35856a6f9c86af2cf7c94a9ab76411739e3/proposals/0120-revise-partition-method.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/1dcfd35856a6f9c86af2cf7c94a9ab76411739e3/proposals/0120-revise-partition-method.md) ## Introduction diff --git a/proposals/0121-remove-optional-comparison-operators.md b/proposals/0121-remove-optional-comparison-operators.md index 34ce813165..45073740df 100644 --- a/proposals/0121-remove-optional-comparison-operators.md +++ b/proposals/0121-remove-optional-comparison-operators.md @@ -2,8 +2,8 @@ * Proposal: [SE-0121](0121-remove-optional-comparison-operators.md) * Author: [Jacob Bandes-Storch](https://github.com/jtbandes) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0121-remove-optional-comparison-operators/3478) * Implementation: [apple/swift#3637](https://github.com/apple/swift/pull/3637) diff --git a/proposals/0122-use-colons-for-subscript-type-declarations.md b/proposals/0122-use-colons-for-subscript-type-declarations.md index d21669c174..cfabbd5dbd 100644 --- a/proposals/0122-use-colons-for-subscript-type-declarations.md +++ b/proposals/0122-use-colons-for-subscript-type-declarations.md @@ -2,7 +2,7 @@ * Proposal: [SE-0122](0122-use-colons-for-subscript-type-declarations.md) * Author: [James Froggatt](https://github.com/MutatingFunk) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0122-use-colons-for-subscript-declarations/3545) diff --git a/proposals/0123-disallow-value-to-optional-coercion-in-operator-arguments.md b/proposals/0123-disallow-value-to-optional-coercion-in-operator-arguments.md index 39dcf33d77..29ecfb3225 100644 --- a/proposals/0123-disallow-value-to-optional-coercion-in-operator-arguments.md +++ b/proposals/0123-disallow-value-to-optional-coercion-in-operator-arguments.md @@ -2,7 +2,7 @@ * Proposal: [SE-0123](0123-disallow-value-to-optional-coercion-in-operator-arguments.md) * Authors: [Mark Lacey](https://github.com/rudkx), [Doug Gregor](https://github.com/DougGregor), [Jacob Bandes-Storch](https://github.com/jtbandes) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0123-disallow-coercion-to-optionals-in-operator-arguments/3479) diff --git a/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md b/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md index 29ee149442..892a806a96 100644 --- a/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md +++ b/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md @@ -2,8 +2,8 @@ * Proposal: [SE-0124](0124-bitpattern-label-for-int-initializer-objectidentfier.md) * Author: [Arnold Schwaighofer](https://github.com/aschwaighofer) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0124-int-init-objectidentifier-and-uint-init-objectidentifier-should-have-a-bitpattern-label/3474) * Bug: [SR-2064](https://bugs.swift.org/browse/SR-2064) diff --git a/proposals/0125-remove-nonobjectivecbase.md b/proposals/0125-remove-nonobjectivecbase.md index 5e51de4b5f..d9b887fed1 100644 --- a/proposals/0125-remove-nonobjectivecbase.md +++ b/proposals/0125-remove-nonobjectivecbase.md @@ -2,8 +2,8 @@ * Proposal: [SE-0125](0125-remove-nonobjectivecbase.md) * Author: [Arnold Schwaighofer](https://github.com/aschwaighofer) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0125-remove-nonobjectivecbase-and-isuniquelyreferenced/3548) * Bug: [SR-1962](http://bugs.swift.org/browse/SR-1962) diff --git a/proposals/0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md b/proposals/0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md index 3fc1d08d20..6901aa0b6e 100644 --- a/proposals/0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md +++ b/proposals/0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md @@ -2,7 +2,7 @@ * Proposal: [SE-0126](0126-refactor-metatypes-repurpose-t-dot-self-and-mirror.md) * Authors: [Adrian Zubarev](https://github.com/DevAndArtist), [Anton Zhilin](https://github.com/Anton3) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Withdrawn** * Decision Notes: [Rationale](https://forums.swift.org/t/withdrawn-for-revision-se-0126-refactor-metatypes-repurpose-t-self-and-mirror/3499) diff --git a/proposals/0127-cleaning-up-stdlib-ptr-buffer.md b/proposals/0127-cleaning-up-stdlib-ptr-buffer.md index 1965632186..f00bb6631a 100644 --- a/proposals/0127-cleaning-up-stdlib-ptr-buffer.md +++ b/proposals/0127-cleaning-up-stdlib-ptr-buffer.md @@ -2,8 +2,8 @@ * Proposal: [SE-0127](0127-cleaning-up-stdlib-ptr-buffer.md) * Author: [Charlie Monroe](https://github.com/charlieMonroe) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0127-cleaning-up-stdlib-pointer-and-buffer-routines/3549) * Bugs: [SR-1937](https://bugs.swift.org/browse/SR-1937), [SR-1955](https://bugs.swift.org/browse/SR-1955), diff --git a/proposals/0128-unicodescalar-failable-initializer.md b/proposals/0128-unicodescalar-failable-initializer.md index 39472db3b5..29e75877b8 100644 --- a/proposals/0128-unicodescalar-failable-initializer.md +++ b/proposals/0128-unicodescalar-failable-initializer.md @@ -2,8 +2,8 @@ * Proposal: [SE-0128](0128-unicodescalar-failable-initializer.md) * Author: [Xin Tong](https://github.com/trentxintong) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0128-change-failable-unicodescalar-initializers-to-failable/3546) * Implementation: [apple/swift#3662](https://github.com/apple/swift/pull/3662) diff --git a/proposals/0129-package-manager-test-naming-conventions.md b/proposals/0129-package-manager-test-naming-conventions.md index 81c555605c..f3e218b70c 100644 --- a/proposals/0129-package-manager-test-naming-conventions.md +++ b/proposals/0129-package-manager-test-naming-conventions.md @@ -3,7 +3,7 @@ * Proposal: [SE-0129](0129-package-manager-test-naming-conventions.md) * Author: [Anders Bertelrud](https://github.com/abertelrud) * Review Manager: [Daniel Dunbar](https://github.com/ddunbar) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0129-package-manager-test-naming-conventions/3574) ## Introduction diff --git a/proposals/0130-string-initializers-cleanup.md b/proposals/0130-string-initializers-cleanup.md index 7cbd97d8de..05a3e7f821 100644 --- a/proposals/0130-string-initializers-cleanup.md +++ b/proposals/0130-string-initializers-cleanup.md @@ -2,8 +2,8 @@ * Proposal: [SE-0130](0130-string-initializers-cleanup.md) * Author: Roman Levenstein -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0130-replace-repeating-character-and-unicodescalar-forms-of-string-init/3547) ## Introduction diff --git a/proposals/0131-anyhashable.md b/proposals/0131-anyhashable.md index 4a4621cfd6..aa7547d367 100644 --- a/proposals/0131-anyhashable.md +++ b/proposals/0131-anyhashable.md @@ -2,8 +2,8 @@ * Proposal: [SE-0131](0131-anyhashable.md) * Author: [Dmitri Gribenko](https://github.com/gribozavr) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0131-add-anyhashable-to-the-standard-library/3553) ## Introduction diff --git a/proposals/0132-sequence-end-ops.md b/proposals/0132-sequence-end-ops.md index f61db449e4..a9136f73e5 100644 --- a/proposals/0132-sequence-end-ops.md +++ b/proposals/0132-sequence-end-ops.md @@ -2,7 +2,7 @@ * Proposal: [SE-0132](0132-sequence-end-ops.md) * Authors: [Becca Royal-Gordon](https://github.com/beccadax), [Dave Abrahams](https://github.com/dabrahams) -* Review Manager: [Chris Lattner](http://github.com/lattner) +* Review Manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/deferred-se-0132-rationalizing-sequence-end-operation-names/3577) @@ -152,7 +152,7 @@ the direction, but do not indicate the direction in their names: Adding a direction to these APIs would make their behavior clearer and permit us to offer opposite-end equivalents in the future. (Unmerged -[swift-evolution pull request 329](https://github.com/apple/swift-evolution/pull/329) +[swift-evolution pull request 329](https://github.com/swiftlang/swift-evolution/pull/329) would add `lastIndex` methods.) ### Operations taking an index are really slicing diff --git a/proposals/0133-rename-flatten-to-joined.md b/proposals/0133-rename-flatten-to-joined.md index 600065785f..4408b1a775 100644 --- a/proposals/0133-rename-flatten-to-joined.md +++ b/proposals/0133-rename-flatten-to-joined.md @@ -2,8 +2,8 @@ * Proposal: [SE-0133](0133-rename-flatten-to-joined.md) * Author: [Jacob Bandes-Storch](https://github.com/jtbandes) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0133-rename-flatten-to-joined/3575) * Implementation: [apple/swift#3809](https://github.com/apple/swift/pull/3809), [apple/swift#3838](https://github.com/apple/swift/pull/3838), diff --git a/proposals/0134-rename-string-properties.md b/proposals/0134-rename-string-properties.md index a406bef70c..6d9abf3cd9 100644 --- a/proposals/0134-rename-string-properties.md +++ b/proposals/0134-rename-string-properties.md @@ -2,11 +2,11 @@ * Proposal: [SE-0134](0134-rename-string-properties.md) * Authors: [Xiaodi Wu](https://github.com/xwu), [Erica Sadun](https://github.com/erica) -* Review Manager: [Chris Lattner](http://github.com/lattner) -* Status: **Implemented (Swift 3)** +* Review Manager: [Chris Lattner](https://github.com/lattner) +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0134-rename-two-utf8-related-properties-on-string/3576) * Implementation: [apple/swift#3816](https://github.com/apple/swift/pull/3816) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/aea8b836d21051076663c5692ec1d09bb3222527/proposals/0134-rename-string-properties.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/aea8b836d21051076663c5692ec1d09bb3222527/proposals/0134-rename-string-properties.md) ## Introduction diff --git a/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md b/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md index 78b13b2d3b..ef8c7fbad4 100644 --- a/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md +++ b/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md @@ -3,7 +3,7 @@ * Proposal: [SE-0135](0135-package-manager-support-for-differentiating-packages-by-swift-version.md) * Author: [Anders Bertelrud](https://github.com/abertelrud) * Review Manager: [Daniel Dunbar](https://github.com/ddunbar) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0135-package-manager-support-for-differentiating-packages-by-swift-version/3687) ## Introduction diff --git a/proposals/0136-memory-layout-of-values.md b/proposals/0136-memory-layout-of-values.md index fa1d39cd2d..9705baaa85 100644 --- a/proposals/0136-memory-layout-of-values.md +++ b/proposals/0136-memory-layout-of-values.md @@ -3,7 +3,7 @@ * Proposal: [SE-0136](0136-memory-layout-of-values.md) * Author: [Xiaodi Wu](https://github.com/xwu) * Review Manager: [Dave Abrahams](https://github.com/dabrahams) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0136-memory-layout-of-values/3760) * Implementation: [apple/swift#4041](https://github.com/apple/swift/pull/4041) diff --git a/proposals/0137-avoiding-lock-in.md b/proposals/0137-avoiding-lock-in.md index 23b7fe3325..43136b648a 100644 --- a/proposals/0137-avoiding-lock-in.md +++ b/proposals/0137-avoiding-lock-in.md @@ -3,7 +3,7 @@ * Proposal: [SE-0137](0137-avoiding-lock-in.md) * Authors: [Dave Abrahams](https://github.com/dabrahams), [Dmitri Gribenko](https://github.com/gribozavr) * Review Manager: [John McCall](https://github.com/rjmccall) -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0137-avoiding-lock-in-to-legacy-protocol-designs/3781) ## Introduction diff --git a/proposals/0138-unsaferawbufferpointer.md b/proposals/0138-unsaferawbufferpointer.md index 74f3c7b0a5..b425d8356a 100644 --- a/proposals/0138-unsaferawbufferpointer.md +++ b/proposals/0138-unsaferawbufferpointer.md @@ -32,7 +32,7 @@ for binding memory to a type for subsequent normal typed access. However, migration is not always straightforward because SE-0107 provided only minimal support for raw pointers. Extending raw pointer support to the `UnsafeBufferPointer` type will fill in this -funcionality gap. This is especially important for code that currently +functionality gap. This is especially important for code that currently views "raw" bytes of memory as `UnsafeBufferPointer`. Converting between `UInt8` and the client's element type at every API transition is difficult to do @@ -69,7 +69,7 @@ is natural for the same type that encapsulates a raw pointer and length to also allow clients to view that memory as raw bytes without the need to explicitly bind the memory type each time memory is accessed. This would also improve performance in some cases that I've -encoutered by avoiding array copies. Let's call this new type +encountered by avoiding array copies. Let's call this new type `Unsafe[Mutable]RawBufferPointer`. Any array could be viewed as `UnsafeRawBufferPointer`, and that raw @@ -488,7 +488,7 @@ collection of bytes, so there's no loss in functionality: ```swift public final class BufferedOutputByteStream: OutputByteStream { // FIXME: For inmemory implementation we should be share this buffer with OutputByteStream. - // One way to do this is by allowing OuputByteStream to install external buffers. + // One way to do this is by allowing OutputByteStream to install external buffers. private var contents = [UInt8]() override final func writeImpl(_ bytes: UnsafeRawBufferPointer) { diff --git a/proposals/0140-bridge-optional-to-nsnull.md b/proposals/0140-bridge-optional-to-nsnull.md index baaa3ca5b4..e5eca4d8a5 100644 --- a/proposals/0140-bridge-optional-to-nsnull.md +++ b/proposals/0140-bridge-optional-to-nsnull.md @@ -195,9 +195,11 @@ error, so should fail early at runtime: This point of view is understandable, but is inconsistent with how Swift itself dynamically treats Optionals inside Anys: +```swift let a: Int? = 3 let b = a as Any let c = a as! Int // Casts '3' out of the Optional as a non-optional Int +``` And while it's true that Cocoa uses `NSNull` sparingly, it *is* the standard sentinel used in the few places where a null-like object is expected, such as diff --git a/proposals/0142-associated-types-constraints.md b/proposals/0142-associated-types-constraints.md index 5a61e84ae2..2c561ef782 100644 --- a/proposals/0142-associated-types-constraints.md +++ b/proposals/0142-associated-types-constraints.md @@ -3,7 +3,7 @@ * Proposal: [SE-0142](0142-associated-types-constraints.md) * Authors: [David Hart](https://github.com/hartbit), [Jacob Bandes-Storch](https://github.com/jtbandes), [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0142-permit-where-clauses-to-constrain-associated-types/4191) * Bugs: [SR-4506](https://bugs.swift.org/browse/SR-4506) @@ -98,7 +98,7 @@ protocol Collection : Sequence { But as Douglas notes himself, that syntax is ambiguous since we adopted the generic `where` clause at the end of declarations of the following proposal: -[SE-0081: Move where clause to end of declaration](https://github.com/apple/swift-evolution/blob/master/proposals/0081-move-where-expression.md). For those reasons, it might be wiser not to introduce the shorthand syntax. +[SE-0081: Move where clause to end of declaration](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0081-move-where-expression.md). For those reasons, it might be wiser not to introduce the shorthand syntax. ## Acknowledgements diff --git a/proposals/0143-conditional-conformances.md b/proposals/0143-conditional-conformances.md index 732b27e72f..f6cfaf5a66 100644 --- a/proposals/0143-conditional-conformances.md +++ b/proposals/0143-conditional-conformances.md @@ -5,7 +5,7 @@ * Review Manager: [Joe Groff](https://github.com/jckarter) * Status: **Implemented (Swift 4.2)** * Decision Notes: [Review extended](https://forums.swift.org/t/review-se-0143-conditional-conformances/4130/10), [Rationale](https://forums.swift.org/t/accepted-se-0143-conditional-conformances/4537) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/91725ee83fa34c81942a634dcdfa9d2441fbd853/proposals/0143-conditional-conformances.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/91725ee83fa34c81942a634dcdfa9d2441fbd853/proposals/0143-conditional-conformances.md) ## Introduction diff --git a/proposals/0144-allow-single-dollar-sign-as-valid-identifier.md b/proposals/0144-allow-single-dollar-sign-as-valid-identifier.md index b12e2dc645..3b5d3a9ea1 100644 --- a/proposals/0144-allow-single-dollar-sign-as-valid-identifier.md +++ b/proposals/0144-allow-single-dollar-sign-as-valid-identifier.md @@ -2,7 +2,7 @@ * Proposal: [SE-0144](0144-allow-single-dollar-sign-as-valid-identifier.md) * Author: [Ankur Patel](https://github.com/ankurp) -* Review manager: [Chris Lattner](http://github.com/lattner) +* Review manager: [Chris Lattner](https://github.com/lattner) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0144-allow-single-dollar-sign-as-a-valid-identifier/4340) diff --git a/proposals/0145-package-manager-version-pinning.md b/proposals/0145-package-manager-version-pinning.md index f29d0c195f..6900359fa2 100644 --- a/proposals/0145-package-manager-version-pinning.md +++ b/proposals/0145-package-manager-version-pinning.md @@ -5,7 +5,7 @@ * Review Manager: [Anders Bertelrud](https://github.com/abertelrud) * Status: **Implemented (Swift 3.1)** * Decision Notes: [Rationale](https://forums.swift.org/t/swift-evolution-accepted-se-0145-package-manager-version-pinning-revised/4653) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/91725ee83fa34c81942a634dcdfa9d2441fbd853/proposals/0145-package-manager-version-pinning.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/91725ee83fa34c81942a634dcdfa9d2441fbd853/proposals/0145-package-manager-version-pinning.md) * Previous Discussion: [Email Thread](https://forums.swift.org/t/review-se-0145-package-manager-version-pinning/4405/15) ## Introduction @@ -366,7 +366,7 @@ specification in the manifest (which is the "requirement"). The meaning of pin connotes this transient relationship between the pin action and the underlying dependency. -In constrast, not only does lock have the wrong connotation, but it also is a +In contrast, not only does lock have the wrong connotation, but it also is a heavily overloaded word which can lead to confusion. For example, if the package manager used POSIX file locking to prevent concurrent manipulation of packages (a feature we intend to implement), and we also referred to the pinning files as diff --git a/proposals/0146-package-manager-product-definitions.md b/proposals/0146-package-manager-product-definitions.md index 4755e6fb7f..ce87eeb8fe 100644 --- a/proposals/0146-package-manager-product-definitions.md +++ b/proposals/0146-package-manager-product-definitions.md @@ -3,7 +3,7 @@ * Proposal: [SE-0146](0146-package-manager-product-definitions.md) * Author: [Anders Bertelrud](https://github.com/abertelrud) * Review manager: Daniel Dunbar -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/review-se-0146-package-manager-product-definitions/4540/2) * Bug: [SR-3606](https://bugs.swift.org/browse/SR-3606) diff --git a/proposals/0147-move-unsafe-initialize-from.md b/proposals/0147-move-unsafe-initialize-from.md index b40479613e..7964d5d408 100644 --- a/proposals/0147-move-unsafe-initialize-from.md +++ b/proposals/0147-move-unsafe-initialize-from.md @@ -54,7 +54,7 @@ Therefore: - Over-allocating the destination buffer relative to `underestimatedCount` is valid and simply results in sequence underflow with potentially uninitialized buffer memory (a likely case with arrays that reserve more than they need). - The source sequence's actual count may exceed both `underestimatedCount` and the destination buffer size, resulting in sequence overflow. This is also valid and handled by returning an iterator to the uncopied elements as an overflow sequence. -A matching change should also be made to `UnsafeRawBufferPointer.initializeMemory(from:)`. The one difference is that for convenience this should return an `UnsafeMutableBufferPointer` of the (typed) intialized elements instead of an index into the raw buffer. +A matching change should also be made to `UnsafeRawBufferPointer.initializeMemory(from:)`. The one difference is that for convenience this should return an `UnsafeMutableBufferPointer` of the (typed) initialized elements instead of an index into the raw buffer. ## Detailed design diff --git a/proposals/0148-generic-subscripts.md b/proposals/0148-generic-subscripts.md index b692a50f8f..1eb2e0dae8 100644 --- a/proposals/0148-generic-subscripts.md +++ b/proposals/0148-generic-subscripts.md @@ -3,7 +3,7 @@ * Proposal: [SE-0148](0148-generic-subscripts.md) * Author: [Chris Eidhof](https://github.com/chriseidhof) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0148-generic-subscripts/5017) * Bug: [SR-115](https://bugs.swift.org/browse/SR-115) @@ -38,7 +38,7 @@ Currently, subscripts can't be generic. This is limiting in a number of ways: - Some subscripts are very specific and could be made more generic. - Some generic methods would feel more natural as a subscript, but currently can't be. This also makes it impossible to use them as lvalues. -This feature is also mentioned in the generics manifesto under [generic subscripts](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#generic-subscripts). The [Rationalizing Sequence end-operation names](https://github.com/apple/swift-evolution/blob/master/proposals/0132-sequence-end-ops.md) proposal could greatly benefit from this, as well as the ideas in the [String Manifesto](https://github.com/apple/swift/blob/master/docs/StringManifesto.md). +This feature is also mentioned in the generics manifesto under [generic subscripts](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#generic-subscripts). The [Rationalizing Sequence end-operation names](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0132-sequence-end-ops.md) proposal could greatly benefit from this, as well as the ideas in the [String Manifesto](https://github.com/apple/swift/blob/master/docs/StringManifesto.md). ## Proposed solution diff --git a/proposals/0149-package-manager-top-of-tree.md b/proposals/0149-package-manager-top-of-tree.md index e224afea7b..8b30f65cde 100644 --- a/proposals/0149-package-manager-top-of-tree.md +++ b/proposals/0149-package-manager-top-of-tree.md @@ -3,7 +3,7 @@ * Proposal: [SE-0149](0149-package-manager-top-of-tree.md) * Author: [Boris Bügling](https://github.com/neonichu) * Review Manager: [Daniel Dunbar](https://github.com/ddunbar) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0149-package-manager-support-for-top-of-tree-development/5072) * Bug: [SR-3709](https://bugs.swift.org/browse/SR-3709) diff --git a/proposals/0150-package-manager-branch-support.md b/proposals/0150-package-manager-branch-support.md index 316ae8bddc..6fa969c889 100644 --- a/proposals/0150-package-manager-branch-support.md +++ b/proposals/0150-package-manager-branch-support.md @@ -3,7 +3,7 @@ * Proposal: [SE-0150](0150-package-manager-branch-support.md) * Author: [Boris Bügling](https://github.com/neonichu) * Review Manager: [Daniel Dunbar](https://github.com/ddunbar) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0150-package-manager-support-for-branches/5074) * Bug: [SR-666](https://bugs.swift.org/browse/SR-666) diff --git a/proposals/0151-package-manager-swift-language-compatibility-version.md b/proposals/0151-package-manager-swift-language-compatibility-version.md index e94c203f6a..e9421cf04b 100644 --- a/proposals/0151-package-manager-swift-language-compatibility-version.md +++ b/proposals/0151-package-manager-swift-language-compatibility-version.md @@ -1,7 +1,7 @@ # Package Manager Swift Language Compatibility Version * Proposal: [SE-0151](0151-package-manager-swift-language-compatibility-version.md) -* Authors: [Daniel Dunbar](https://github.com/ddunbar), [Rick Ballard](http://github.com/rballard) +* Authors: [Daniel Dunbar](https://github.com/ddunbar), [Rick Ballard](https://github.com/rballard) * Review Manager: [Anders Bertelrud](https://github.com/abertelrud) * Status: **Implemented (Swift 3.1)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0151-package-manager-swift-language-compatibility-version/5183) diff --git a/proposals/0154-dictionary-key-and-value-collections.md b/proposals/0154-dictionary-key-and-value-collections.md index f0dd0d9b95..f5ad8d6d71 100644 --- a/proposals/0154-dictionary-key-and-value-collections.md +++ b/proposals/0154-dictionary-key-and-value-collections.md @@ -3,7 +3,7 @@ * Proposal: [SE-0154](0154-dictionary-key-and-value-collections.md) * Author: [Nate Cook](https://github.com/natecook1000) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0154-provide-custom-collections-for-dictionary-keys-and-values/5322) ## Introduction diff --git a/proposals/0155-normalize-enum-case-representation.md b/proposals/0155-normalize-enum-case-representation.md index faf802867a..8e0474d166 100644 --- a/proposals/0155-normalize-enum-case-representation.md +++ b/proposals/0155-normalize-enum-case-representation.md @@ -3,7 +3,7 @@ * Proposal: [SE-0155][] * Authors: [Daniel Duan][], [Joe Groff][] * Review Manager: [John McCall][] -* Status: **Implemented (Swift 3)** +* Status: **Implemented (Swift 3.0)** * Decision Notes: [Rationale][] * Previous Revision: [1][Revision 1], [Originally Accepted Proposal][], [Expired Proposal][] * Bugs: [SR-4691](https://bugs.swift.org/browse/SR-4691), [SR-12206](https://bugs.swift.org/browse/SR-12206), [SR-12229](https://bugs.swift.org/browse/SR-12229) @@ -257,7 +257,7 @@ func switchFoo(x: Foo) { ``` However, it was decided in review that this was still too restrictive and -source-breaking, and so the core team [accepted the proposal][Rationale] with the modification that pattern matches only had to match the case declaration in arity, and case labels could be either provided or elided in their entirety, unless there was an ambiguity. Even then, as of Swift 5.2, this part of the proposal has not been implemented, and it would be a source breaking change to do so. Therefore, the "Pattern Consistency" section of the original proposal has been removed, and replaced with a ["Disambiguating pattern matches" section](https://github.com/apple/swift-evolution/blob/aecced4919ab297f343dafd7235d392d8b859839/proposals/0155-normalize-enum-case-representation.md), which provided a minimal disambiguation rule for pattern matching cases that share a +source-breaking, and so the core team [accepted the proposal][Rationale] with the modification that pattern matches only had to match the case declaration in arity, and case labels could be either provided or elided in their entirety, unless there was an ambiguity. Even then, as of Swift 5.2, this part of the proposal has not been implemented, and it would be a source breaking change to do so. Therefore, the "Pattern Consistency" section of the original proposal has been removed, and replaced with a ["Disambiguating pattern matches" section](https://github.com/swiftlang/swift-evolution/blob/aecced4919ab297f343dafd7235d392d8b859839/proposals/0155-normalize-enum-case-representation.md), which provided a minimal disambiguation rule for pattern matching cases that share a base name. This new design still had not been implemented at the time the [core team adopted a new expiration policy for unimplemented proposals](https://forums.swift.org/t/addressing-unimplemented-evolution-proposals/40322), so it has expired. [SE-0155]: 0155-normalize-enum-case-representation.md @@ -266,8 +266,8 @@ base name. This new design still had not been implemented at the time the [core [Joe Groff]: https://github.com/jckarter [John McCall]: https://github.com/rjmccall [TJs comment]: https://forums.swift.org/t/draft-compound-names-for-enum-cases/4933/33 -[Revision 1]: https://github.com/apple/swift-evolution/blob/43ca098355762014f53e1b54e02d2f6a01253385/proposals/0155-normalize-enum-case-representation.md +[Revision 1]: https://github.com/swiftlang/swift-evolution/blob/43ca098355762014f53e1b54e02d2f6a01253385/proposals/0155-normalize-enum-case-representation.md [Normalize Enum Case Representation (rev. 2)]: https://forums.swift.org/t/normalize-enum-case-representation-rev-2/5395 -[Originally Accepted Proposal]: https://github.com/apple/swift-evolution/blob/4cbb1f1fa836496d4bfba95c4b78a9754690956d/proposals/0155-normalize-enum-case-representation.md -[Expired Proposal]: https://github.com/apple/swift-evolution/blob/aecced4919ab297f343dafd7235d392d8b859839/proposals/0155-normalize-enum-case-representation.md +[Originally Accepted Proposal]: https://github.com/swiftlang/swift-evolution/blob/4cbb1f1fa836496d4bfba95c4b78a9754690956d/proposals/0155-normalize-enum-case-representation.md +[Expired Proposal]: https://github.com/swiftlang/swift-evolution/blob/aecced4919ab297f343dafd7235d392d8b859839/proposals/0155-normalize-enum-case-representation.md [Rationale]: https://forums.swift.org/t/accepted-se-0155-normalize-enum-case-representation/5732 diff --git a/proposals/0156-subclass-existentials.md b/proposals/0156-subclass-existentials.md index e0a66e1117..a3107aef79 100644 --- a/proposals/0156-subclass-existentials.md +++ b/proposals/0156-subclass-existentials.md @@ -1,9 +1,9 @@ # Class and Subtype existentials * Proposal: [SE-0156](0156-subclass-existentials.md) -* Authors: [David Hart](http://github.com/hartbit), [Austin Zheng](http://github.com/austinzheng) +* Authors: [David Hart](https://github.com/hartbit), [Austin Zheng](https://github.com/austinzheng) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0156-class-and-subtype-existentials/5477) * Bug: [SR-4296](https://bugs.swift.org/browse/SR-4296) @@ -190,7 +190,7 @@ let myViewController = MyViewController() myViewController.setup(UIViewController()) ``` -The previous code continues to compile but still crashs if the Objective-C code calls a method of `UITableViewDataSource` or `UITableViewDelegate`. But if this proposal is accepted and implemented as-is, the Objective-C code will be imported in Swift 4 mode as: +The previous code continues to compile but still crashes if the Objective-C code calls a method of `UITableViewDataSource` or `UITableViewDelegate`. But if this proposal is accepted and implemented as-is, the Objective-C code will be imported in Swift 4 mode as: ```swift class MyViewController { @@ -205,7 +205,7 @@ That would then cause the Swift code run in version 4 mode to fail to compile wi An alternative solution to the `class`/`AnyObject` duplication was to keep both, redefine `AnyObject` as `typealias AnyObject = class` and favor the latter when used as a type name. The [reviewed version of the -proposal](https://github.com/apple/swift-evolution/blob/78da25ec4acdc49ad9b68fb58300e49c33bc6355/proposals/0156-subclass-existentials.md) +proposal](https://github.com/swiftlang/swift-evolution/blob/78da25ec4acdc49ad9b68fb58300e49c33bc6355/proposals/0156-subclass-existentials.md) included rules that required the class type (or `AnyObject`) to be first within the protocol composition, e.g., `AnyObject & Protocol1` was well-formed but `Protocol1 & AnyObject` would produce a compiler @@ -214,4 +214,4 @@ rules; see the decision notes at the top for more information. ## Acknowledgements -Thanks to [Austin Zheng](http://github.com/austinzheng) and [Matthew Johnson](https://github.com/anandabits) who brought a lot of attention to existentials in this mailing-list and from whom most of the ideas in the proposal come from. +Thanks to [Austin Zheng](https://github.com/austinzheng) and [Matthew Johnson](https://github.com/anandabits) who brought a lot of attention to existentials in this mailing-list and from whom most of the ideas in the proposal come from. diff --git a/proposals/0157-recursive-protocol-constraints.md b/proposals/0157-recursive-protocol-constraints.md index cfb14c042a..81c1328494 100644 --- a/proposals/0157-recursive-protocol-constraints.md +++ b/proposals/0157-recursive-protocol-constraints.md @@ -93,7 +93,7 @@ Implementation details regarding the compiler changes necessary to implement the found in [this document](https://gist.github.com/DougGregor/e7c4e7bb4465d6f5fa2b59be72dbdba6). The second part of the solution involves updating the standard library to take advantage of the removal of this -restriction. Such changes are made with [SE-0142](https://github.com/apple/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md) +restriction. Such changes are made with [SE-0142](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md) in mind, and incorporate both recursive constraints and `where` clauses. The changes necessary for this are described in the _Detailed Design_ section below. diff --git a/proposals/0158-package-manager-manifest-api-redesign.md b/proposals/0158-package-manager-manifest-api-redesign.md index 0ef23dfd0a..adbe933201 100644 --- a/proposals/0158-package-manager-manifest-api-redesign.md +++ b/proposals/0158-package-manager-manifest-api-redesign.md @@ -3,7 +3,7 @@ * Proposal: [SE-0158](0158-package-manager-manifest-api-redesign.md) * Author: [Ankit Aggarwal](https://github.com/aciidb0mb3r) * Review Manager: [Rick Ballard](https://github.com/rballard) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Bug: [SR-3949](https://bugs.swift.org/browse/SR-3949) * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0158-package-manager-manifest-api-redesign/5468) @@ -160,7 +160,7 @@ access modifier is `public` for all APIs unless specified. "Foo")` instead of `.product(name: "Foo", package: nil)`. If - [SE-0155](https://github.com/apple/swift-evolution/blob/master/proposals/0155-normalize-enum-case-representation.md) + [SE-0155](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0155-normalize-enum-case-representation.md) is accepted, we can directly add a default value. Otherwise, we will use a static factory method to provide default value for `package`. @@ -456,7 +456,7 @@ access modifier is `public` for all APIs unless specified. .package(url: "/SwiftyJSON", "1.2.3"..."1.2.8"), ``` * As a slight modification to the - [branch proposal](https://github.com/apple/swift-evolution/blob/master/proposals/0150-package-manager-branch-support.md), + [branch proposal](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0150-package-manager-branch-support.md), we will add cases for specifying a branch or revision, rather than adding factory methods for them: @@ -594,7 +594,7 @@ let package = Package( The above changes will be implemented only in the new Package Description v4 library. The v4 runtime library will release with Swift 4 and packages will be able to opt-in into it as described by -[SE-0152](https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md). +[SE-0152](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md). There will be no automatic migration feature for updating the manifests from v3 to v4. To indicate the replacements of old APIs, we will annotate them using @@ -614,7 +614,7 @@ the v3 manifest API, they will build as expected. A package which needs to support both Swift 3 and Swift 4 tools will need to stay on the v3 manifest API and support the Swift 3 language version for its sources, using the API described in the proposal -[SE-0151](https://github.com/apple/swift-evolution/blob/master/proposals/0151-package-manager-swift-language-compatibility-version.md). +[SE-0151](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0151-package-manager-swift-language-compatibility-version.md). An existing package which wants to use the new v4 manifest APIs will need to bump its minimum tools version to 4.0 or later using the command `$ swift package tools-version diff --git a/proposals/0159-fix-private-access-levels.md b/proposals/0159-fix-private-access-levels.md index 75aaecac6d..45660527a8 100644 --- a/proposals/0159-fix-private-access-levels.md +++ b/proposals/0159-fix-private-access-levels.md @@ -1,14 +1,14 @@ # Fix Private Access Levels * Proposal: [SE-0159](0159-fix-private-access-levels.md) -* Author: [David Hart](http://github.com/hartbit) +* Author: [David Hart](https://github.com/hartbit) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Rejected** * Decision Notes: [Rationale](https://forums.swift.org/t/rejected-se-0159-fix-private-access-levels/5576) ## Introduction -This proposal presents the problems that came with the the access level modifications in [SE-0025](https://github.com/apple/swift-evolution/blob/master/proposals/0025-scoped-access-level.md) and proposes reverting to Swift 2 behaviour. +This proposal presents the problems that came with the the access level modifications in [SE-0025](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0025-scoped-access-level.md) and proposes reverting to Swift 2 behaviour. ## Motivation diff --git a/proposals/0160-objc-inference.md b/proposals/0160-objc-inference.md index 7cb90a51c8..8c595e5318 100644 --- a/proposals/0160-objc-inference.md +++ b/proposals/0160-objc-inference.md @@ -3,9 +3,9 @@ * Proposal: [SE-0160](0160-objc-inference.md) * Author: [Doug Gregor](https://github.com/DougGregor) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0160-limiting-objc-inference/5621) -* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/0389b1f49fc55b1a898701c549ce89738307b9fc/proposals/0160-objc-inference.md) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/0389b1f49fc55b1a898701c549ce89738307b9fc/proposals/0160-objc-inference.md) * Implementation: [apple/swift#8379](https://github.com/apple/swift/pull/8379) * Bug: [SR-4481](https://bugs.swift.org/browse/SR-4481) @@ -176,7 +176,7 @@ become well-formed, and the method `bar()` will continue to work as it does today through the Objective-C runtime. Indeed, this change is the right way forward even if Swift never supports `dynamic` in its own runtime, following the precedent of -[SE-0070](https://github.com/apple/swift-evolution/blob/master/proposals/0070-optional-requirements.md), +[SE-0070](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0070-optional-requirements.md), which required the Objective-C-only protocol feature "optional requirements" to be explicitly marked with `@objc`. @@ -606,6 +606,6 @@ relationship to this proposal. # Revision history -[Version 1](https://github.com/apple/swift-evolution/blob/0389b1f49fc55b1a898701c549ce89738307b9fc/proposals/0160-objc-inference.md) +[Version 1](https://github.com/swiftlang/swift-evolution/blob/0389b1f49fc55b1a898701c549ce89738307b9fc/proposals/0160-objc-inference.md) of this proposal did not include the use of `@objcMembers` on classes or the use of `@objc`/`@nonobjc` on extensions to mass-annotate. diff --git a/proposals/0161-key-paths.md b/proposals/0161-key-paths.md index 608127f527..b945854b41 100644 --- a/proposals/0161-key-paths.md +++ b/proposals/0161-key-paths.md @@ -3,9 +3,9 @@ * Proposal: [SE-0161](0161-key-paths.md) * Authors: [David Smith](https://github.com/Catfish-Man), [Michael LeHew](https://github.com/mlehew), [Joe Groff](https://github.com/jckarter) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0161-smart-keypaths-better-key-value-coding-for-swift/5690) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/55e61f459632eca2face40e571a517919f846cfb/proposals/0161-key-paths.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/55e61f459632eca2face40e571a517919f846cfb/proposals/0161-key-paths.md) ## Introduction We propose a family of concrete _Key Path_ types that represent uninvoked references to properties that can be composed to form paths through many values and directly get/set their underlying values. @@ -182,9 +182,9 @@ We also explored many different spellings, each with different strengths. We hav | Case | `#keyPath` | Function Type Reference | Escape | | --- | --- | --- | --- | | Fully qualified | `#keyPath(Person, .friends[0].name)` | `Person.friends[0].name` | `\Person.friends[0].name` | -| Type Inferred| `#keyPath(.friends[0].name)` |`Person.friends[0].name` | `\.friends[0].name` | +| Type Inferred | `#keyPath(.friends[0].name)` |`Person.friends[0].name` | `\.friends[0].name` | -While the crispness of the function-type-reference is appealing, it becomes ambigious when working with type properties. The escape-sigil variant avoids this, and remains quite readable. +While the crispness of the function-type-reference is appealing, it becomes ambiguous when working with type properties. The escape-sigil variant avoids this, and remains quite readable. #### Why `\`? During review many different sigils were considered: @@ -199,4 +199,3 @@ During review many different sigils were considered: #### Function Type References We think the disambiguating benefits of the escape-sigil would greatly benefit function type references, but such considerations are outside the scope of this proposal. - diff --git a/proposals/0162-package-manager-custom-target-layouts.md b/proposals/0162-package-manager-custom-target-layouts.md index f19ee92f50..78d428e350 100644 --- a/proposals/0162-package-manager-custom-target-layouts.md +++ b/proposals/0162-package-manager-custom-target-layouts.md @@ -3,7 +3,7 @@ * Proposal: [SE-0162](0162-package-manager-custom-target-layouts.md) * Author: [Ankit Aggarwal](https://github.com/aciidb0mb3r) * Review Manager: [Rick Ballard](https://github.com/rballard) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0162-package-manager-custom-target-layouts/5647) * Bug: [SR-29](https://bugs.swift.org/browse/SR-29) @@ -91,7 +91,7 @@ remember. consider upgrading this to its own type to allow per-file build settings. The new type would conform to `CustomStringConvertible`, so existing declarations would continue to work (except where the strings were - constructed programatically). + constructed programmatically). * `exclude`: This property can be used to exclude certain files and directories from being picked up as sources. Exclude paths are relative diff --git a/proposals/0163-string-revision-1.md b/proposals/0163-string-revision-1.md index 23cff3b394..f584cc67f7 100644 --- a/proposals/0163-string-revision-1.md +++ b/proposals/0163-string-revision-1.md @@ -1,11 +1,11 @@ # String Revision: Collection Conformance, C Interop, Transcoding * Proposal: [SE-0163](0163-string-revision-1.md) -* Authors: [Ben Cohen](https://github.com/airspeedswift), [Dave Abrahams](http://github.com/dabrahams/) +* Authors: [Ben Cohen](https://github.com/airspeedswift), [Dave Abrahams](https://github.com/dabrahams/) * Review Manager: [John McCall](https://github.com/rjmccall) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Revision: 2 -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/7513547ddac66b06770a1fd620aad915d75987ff/proposals/0163-string-revision-1.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/7513547ddac66b06770a1fd620aad915d75987ff/proposals/0163-string-revision-1.md) * Decision Notes: [Rationale #1](https://forums.swift.org/t/accepted-se-0163-string-revision-collection-conformance-c-interop-transcoding/5716/2), [Rationale #2](https://forums.swift.org/t/accepted-se-0163-string-revision-collection-conformance-c-interop-transcoding/5952) ## Introduction diff --git a/proposals/0164-remove-final-support-in-protocol-extensions.md b/proposals/0164-remove-final-support-in-protocol-extensions.md index 5ae21b0f81..5e807e4bd3 100644 --- a/proposals/0164-remove-final-support-in-protocol-extensions.md +++ b/proposals/0164-remove-final-support-in-protocol-extensions.md @@ -3,7 +3,7 @@ * Proposal: [SE-0164](0164-remove-final-support-in-protocol-extensions.md) * Author: [Brian King](https://github.com/KingOfBrian) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0164-remove-final-support-in-protocol-extensions/5687) * Bug: [SR-1762](https://bugs.swift.org/browse/SR-1762) diff --git a/proposals/0165-dict.md b/proposals/0165-dict.md index 0c96c4a61f..73897eccda 100644 --- a/proposals/0165-dict.md +++ b/proposals/0165-dict.md @@ -1,10 +1,10 @@ # Dictionary & Set Enhancements -- Proposal: [SE-0165](0165-dict.md) -- Author: [Nate Cook](https://github.com/natecook1000) -- Review Manager: [Ben Cohen](https://github.com/airspeedswift) -- Status: **Implemented (Swift 4)** -- Decision Notes: [Rationale][rationale] +* Proposal: [SE-0165](0165-dict.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 4.0)** +* Decision Notes: [Rationale][rationale] ## Introduction @@ -29,7 +29,7 @@ The `Dictionary` type should allow initialization from a sequence of `(Key, Valu - [First message of discussion thread](https://forums.swift.org/t/map-like-operation-that-returns-a-dictionary/999) - [Initial proposal draft](https://forums.swift.org/t/map-like-operation-that-returns-a-dictionary/999/18) -- [Prior standalone proposal (SE-100)](https://github.com/apple/swift-evolution/blob/master/proposals/0100-add-sequence-based-init-and-merge-to-dictionary.md) +- [Prior standalone proposal (SE-100)](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0100-add-sequence-based-init-and-merge-to-dictionary.md) `Array` and `Set` both have initializers that create a new instance from a sequence of elements. The `Array` initializer is useful for converting other sequences and collections to the "standard" collection type, while the `Set` initializer is essential for recovering set operations after performing any functional operations on a set. For example, filtering a set produces a collection without any set operations available. diff --git a/proposals/0166-swift-archival-serialization.md b/proposals/0166-swift-archival-serialization.md index 00f5815005..53a9a69229 100644 --- a/proposals/0166-swift-archival-serialization.md +++ b/proposals/0166-swift-archival-serialization.md @@ -3,7 +3,7 @@ * Proposal: [SE-0166](0166-swift-archival-serialization.md) * Authors: [Itai Ferber](https://github.com/itaiferber), [Michael LeHew](https://github.com/mlehew), [Tony Parker](https://github.com/parkera) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0166-swift-archival-serialization/5780) * Implementation: [apple/swift#9004](https://github.com/apple/swift/pull/9004) diff --git a/proposals/0167-swift-encoders.md b/proposals/0167-swift-encoders.md index dce5f312e8..223aa7f390 100644 --- a/proposals/0167-swift-encoders.md +++ b/proposals/0167-swift-encoders.md @@ -3,7 +3,7 @@ * Proposal: [SE-0167](0167-swift-encoders.md) * Authors: [Itai Ferber](https://github.com/itaiferber), [Michael LeHew](https://github.com/mlehew), [Tony Parker](https://github.com/parkera) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0167-swift-encoders/5781) * Implementation: [apple/swift#9005](https://github.com/apple/swift/pull/9005) diff --git a/proposals/0168-multi-line-string-literals.md b/proposals/0168-multi-line-string-literals.md index af6b601239..bf8baa4a2a 100644 --- a/proposals/0168-multi-line-string-literals.md +++ b/proposals/0168-multi-line-string-literals.md @@ -3,7 +3,7 @@ * Proposal: [SE-0168](0168-multi-line-string-literals.md) * Authors: [John Holdsworth](https://github.com/johnno1962), [Becca Royal-Gordon](https://github.com/beccadax), [Tyler Cloutier](https://github.com/TheArtOfEngineering) * Review Manager: [Joe Groff](https://github.com/jckarter) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Implementation: [apple/swift#8813](https://github.com/apple/swift/pull/8813) * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0168-multi-line-string-literals/5715) * Bugs: [SR-170](https://bugs.swift.org/browse/SR-170), [SR-4701](https://bugs.swift.org/browse/SR-4701), [SR-4708](https://bugs.swift.org/browse/SR-4708), [SR-4874](https://bugs.swift.org/browse/SR-4874) diff --git a/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md b/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md index 2705eceb08..043f58d5b6 100644 --- a/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md +++ b/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md @@ -1,9 +1,9 @@ # Improve Interaction Between `private` Declarations and Extensions * Proposal: [SE-0169](0169-improve-interaction-between-private-declarations-and-extensions.md) -* Authors: [David Hart](http://github.com/hartbit), [Chris Lattner](https://github.com/lattner) +* Authors: [David Hart](https://github.com/hartbit), [Chris Lattner](https://github.com/lattner) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0169-improve-interaction-between-private-declarations-and-extensions/5692) * Previous Revision: [1][Revision 1] * Bug: [SR-4616](https://bugs.swift.org/browse/SR-4616) @@ -156,6 +156,6 @@ but broader in other ways (allow access across files) would lead to a more confusing and fractured model. -[Revision 1]: https://github.com/apple/swift-evolution/blob/e0e04f785dbf5bff138b75e9c47bf94e7db28447/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md +[Revision 1]: https://github.com/swiftlang/swift-evolution/blob/e0e04f785dbf5bff138b75e9c47bf94e7db28447/proposals/0169-improve-interaction-between-private-declarations-and-extensions.md diff --git a/proposals/0170-nsnumber_bridge.md b/proposals/0170-nsnumber_bridge.md index 4a426759ca..3dff41e9b7 100644 --- a/proposals/0170-nsnumber_bridge.md +++ b/proposals/0170-nsnumber_bridge.md @@ -3,7 +3,7 @@ * Proposal: [SE-0170](0170-nsnumber_bridge.md) * Author: [Philippe Hausler](https://github.com/phausler) * Review Manager: [Ben Cohen](https://github.com/airspeedswift) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0170-nsnumber-bridging-and-numeric-types/5801) ##### Revision history diff --git a/proposals/0171-reduce-with-inout.md b/proposals/0171-reduce-with-inout.md index dad7187fa6..d1ab3d448c 100644 --- a/proposals/0171-reduce-with-inout.md +++ b/proposals/0171-reduce-with-inout.md @@ -3,7 +3,7 @@ * Proposal: [SE-0171](0171-reduce-with-inout.md) * Author: [Chris Eidhof](https://github.com/chriseidhof) * Review Manager: [Ben Cohen](https://github.com/airspeedswift) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0171-reduce-with-inout/5769) ## Introduction diff --git a/proposals/0172-one-sided-ranges.md b/proposals/0172-one-sided-ranges.md index 00522d4bfc..0864e86a10 100644 --- a/proposals/0172-one-sided-ranges.md +++ b/proposals/0172-one-sided-ranges.md @@ -3,7 +3,7 @@ * Proposal: [SE-0172](0172-one-sided-ranges.md) * Authors: [Ben Cohen](https://github.com/airspeedswift), [Dave Abrahams](https://github.com/dabrahams), [Becca Royal-Gordon](https://github.com/beccadax) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0172-one-sided-ranges/5768) ## Introduction @@ -64,7 +64,7 @@ variants of `Sequence.enumerated()` when you either want them non-zero-based i.e. `zip(1..., greeting)`, or want to flip the order i.e. `zip(greeting, 0...)`. -This syntax would supercede the existing `prefix` and `suffix` operations that +This syntax would supersede the existing `prefix` and `suffix` operations that take indices, which will be deprecated in a later release. Note that the versions that take distances are not covered by this proposal, and would remain. @@ -171,7 +171,7 @@ extension MutableCollection { where R.Bound == Index { get set } } -extension RangeReplaceableColleciton { +extension RangeReplaceableCollection { public mutating func replaceSubrange( _ subrange: ${Range}, with newElements: C ) where C.Iterator.Element == Iterator.Element, R.Bound == Index diff --git a/proposals/0173-swap-indices.md b/proposals/0173-swap-indices.md index 6c6b6c080a..26055065f5 100644 --- a/proposals/0173-swap-indices.md +++ b/proposals/0173-swap-indices.md @@ -3,7 +3,7 @@ * Proposal: [SE-0173](0173-swap-indices.md) * Author: [Ben Cohen](https://github.com/airspeedswift) * Review Manager: [Ted Kremenek](https://github.com/tkremenek) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0173-add-mutablecollection-swapat/5811) * Implementation: [apple/swift#9119](https://github.com/apple/swift/pull/9119) @@ -82,7 +82,7 @@ protocol MutableCollection { The current `swap` is required to `fatalError` on attempts to swap an element with itself for implementation reasons. This pushes the burden to check this first onto the caller. While swapping an element with itself is often a logic -errror (for example, in a `sort` algorithm where you have a fenceposts bug), it +error (for example, in a `sort` algorithm where you have a fenceposts bug), it is occasionally a valid situation (for example, it can occur easily in an implementation of `shuffle`). This implementation removes the precondition. diff --git a/proposals/0174-filter-range-replaceable.md b/proposals/0174-filter-range-replaceable.md index 1edd946141..ead1768f74 100644 --- a/proposals/0174-filter-range-replaceable.md +++ b/proposals/0174-filter-range-replaceable.md @@ -15,7 +15,7 @@ to return the same type as the filtered collection. ## Motivation The recently accepted -[SE-165](https://github.com/apple/swift-evolution/blob/master/proposals/0165-dict.md) +[SE-165](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0165-dict.md) introduced a version of `filter` on `Dictionary` that returned a `Dictionary`. This had both performance and usability benefits: in most cases, a `Dictionary` is what the user wanted from the filter, and creating one @@ -34,7 +34,7 @@ filter it, you will still get an `Array`. An implementation of `filter` on `RangeReplaceableCollection` will be provided, using `init()` and `append(_:)`, so all range-replaceable collections will -have a `filter` method returning of `Self`. Per [SE-163](https://github.com/apple/swift-evolution/blob/master/proposals/0163-string-revision-1.md), +have a `filter` method returning of `Self`. Per [SE-163](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0163-string-revision-1.md), this will include `String`. Note, many sequences (for example, strides or ranges), cannot represent a @@ -70,7 +70,7 @@ They may be be relying on an array being returned (albeit often in order to then transform it back into the original type), but this version will still be available (via the extension on `Sequence`) and will be called if forced through type context. The only code that will break is if this operation spans -multple lines: +multiple lines: ```swift // filtered used to be [Character], now String diff --git a/proposals/0175-package-manager-revised-dependency-resolution.md b/proposals/0175-package-manager-revised-dependency-resolution.md index 8f236f3606..9dbc7e116f 100644 --- a/proposals/0175-package-manager-revised-dependency-resolution.md +++ b/proposals/0175-package-manager-revised-dependency-resolution.md @@ -3,14 +3,14 @@ * Proposal: [SE-0175](0175-package-manager-revised-dependency-resolution.md) * Author: [Rick Ballard](https://github.com/rballard) * Review Manager: [Ankit Aggarwal](https://github.com/aciidb0mb3r) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0175-package-manager-revised-dependency-resolution/5896) ## Introduction This proposal makes the package manager's dependency resolution behavior clearer and more intuitive. It removes the pinning commands (`swift package pin` & `swift package unpin`), replaces the `swift package fetch` command with a new `swift package resolve` command with improved behavior, and replaces the optional `Package.pins` file with a `Package.resolved` file which is always created during dependency resolution. ## Motivation -When [SE-0145 Package Manager Version Pinning](https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md) was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like `swift package update`, which has a `--repin` flag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing. +When [SE-0145 Package Manager Version Pinning](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md) was proposed, it was observed that the proposal was overly complex. In particular, it introduced a configuration option allowing some packages to have autopinning on (the default), while others turned it off; this option affected the behavior of other commands (like `swift package update`, which has a `--repin` flag that does nothing for packages that use autopinning). This configuration option has proved to be unnecessarily confusing. In the existing design, when autopinning is on (which is true by default) the `swift package pin` command can't be used to pin packages at specific revisions while allowing other packages to be updated. In particular, if you edit your package's version requirements in the `Package.swift` manifest, there is no way to resolve your package graph to conform to those new requirements without automatically repinning all packages to the latest allowable versions. Thus, specific, intentional pins can not be preserved without turning off autopinning. @@ -52,7 +52,7 @@ If a dependency is in edit mode, it is allowed to have a different version check We considered repurposing the existing `fetch` command for this new behavior, instead of renaming the command to `resolve`. However, the name `fetch` is defined by `git` to mean getting the latest content for a repository over the network. Since this package manager command does not always actually fetch new content from the network, it is confusing to use the name `fetch`. In the future, we may offer additional control over when dependency resolution is allowed to perform network access, and we will likely use the word `fetch` in flag names that control that behavior. -We considered continuing to write out the `Package.pins` file for packages whose [Swift tools version](https://github.com/apple/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md) was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools. +We considered continuing to write out the `Package.pins` file for packages whose [Swift tools version](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0152-package-manager-tools-version.md) was less than 4.0, for maximal compatibility with the Swift 3.1 tools. However, as the old pinning behavior was a workflow feature and not a fundamental piece of package compatibility, we do not consider it necessary to support in the 4.0 tools. We considered keeping the `pin` and `unpin` commands, with the new behavior as discussed briefly in this proposal. While we think we may wish to bring this feature back in the future, we do not consider it critical for this release; the workflow it supports (updating all packages except a handful which have been pinned) is not something most users will need, and there are workarounds (e.g. specify an explicit dependency in the `Package.swift` manifest). diff --git a/proposals/0176-enforce-exclusive-access-to-memory.md b/proposals/0176-enforce-exclusive-access-to-memory.md index 70679c3a18..f11512ff96 100644 --- a/proposals/0176-enforce-exclusive-access-to-memory.md +++ b/proposals/0176-enforce-exclusive-access-to-memory.md @@ -3,8 +3,8 @@ * Proposal: [SE-0176](0176-enforce-exclusive-access-to-memory.md) * Author: [John McCall](https://github.com/rjmccall) * Review Manager: [Ben Cohen](https://github.com/airspeedswift) -* Status: **Implemented (Swift 4)** -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/7e6816c22a29b0ba9bdf63ff92b380f9e963860a/proposals/0176-enforce-exclusive-access-to-memory.md) +* Status: **Implemented (Swift 4.0)** +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/7e6816c22a29b0ba9bdf63ff92b380f9e963860a/proposals/0176-enforce-exclusive-access-to-memory.md) * Previous Discussion: [Email Thread](https://forums.swift.org/t/review-se-0176-enforce-exclusive-access-to-memory/5836) ## Introduction @@ -479,7 +479,7 @@ In the short term, these problems can be worked around with ``withUnsafeMutableBufferPointer``. We do know that swapping two array elements will be problematic, -and accordingly we are (separately proposing)[https://github.com/apple/swift-evolution/blob/master/proposals/0173-swap-indices.md] to add a +and accordingly we are (separately proposing)[https://github.com/swiftlang/swift-evolution/blob/master/proposals/0173-swap-indices.md] to add a ``swapAt`` method to ``MutableCollection`` that takes two indices rather than two ``inout`` arguments. The Swift 3 compatibility mode should recognize the swap-of-elements pattern and automatically @@ -740,7 +740,7 @@ automatically to avoid source-compatibility problems. ## Effect on ABI stability and resilience -In order to gain the performance and language-desing benefits of +In order to gain the performance and language-design benefits of exclusivity, we must be able to assume that it is followed faithfully in various places throughout the ABI. Therefore, exclusivity must be enforced before we commit to a stable ABI, or else we'll be stuck with diff --git a/proposals/0178-character-unicode-view.md b/proposals/0178-character-unicode-view.md index 2174ddb2b3..64264a6e97 100644 --- a/proposals/0178-character-unicode-view.md +++ b/proposals/0178-character-unicode-view.md @@ -3,7 +3,7 @@ * Proposal: [SE-0178](0178-character-unicode-view.md) * Author: [Ben Cohen](https://github.com/airspeedswift) * Review Manager: [Ted Kremenek](https://github.com/tkremenek) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0178-add-unicodescalars-property-to-character/5941) * Implementation: [apple/swift#9675](https://github.com/apple/swift/pull/9675) diff --git a/proposals/0179-swift-run-command.md b/proposals/0179-swift-run-command.md index 9f50affd69..f43b30b473 100644 --- a/proposals/0179-swift-run-command.md +++ b/proposals/0179-swift-run-command.md @@ -1,9 +1,9 @@ # Swift `run` Command * Proposal: [SE-0179](0179-swift-run-command.md) -* Author: [David Hart](http://github.com/hartbit/) +* Author: [David Hart](https://github.com/hartbit/) * Review Manager: [Daniel Dunbar](https://github.com/ddunbar) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0179-swift-run-command/6031) * Implementation: [apple/swift-package-manager#1187](https://github.com/apple/swift-package-manager/pull/1187) diff --git a/proposals/0180-string-index-overhaul.md b/proposals/0180-string-index-overhaul.md index be0025edf0..abe8de6074 100644 --- a/proposals/0180-string-index-overhaul.md +++ b/proposals/0180-string-index-overhaul.md @@ -3,10 +3,10 @@ * Proposal: [SE-0180](0180-string-index-overhaul.md) * Author: [Dave Abrahams](https://github.com/dabrahams) * Review Manager: [Ted Kremenek](https://github.com/tkremenek) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0180-string-index-overhaul/6286) * Implementation: [apple/swift#9806](https://github.com/apple/swift/pull/9806) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/72b8d90becd60b7cc7695607ae908ef251f1e966/proposals/0180-string-index-overhaul.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/72b8d90becd60b7cc7695607ae908ef251f1e966/proposals/0180-string-index-overhaul.md) ## Introduction @@ -129,7 +129,7 @@ let tagEnd = html.utf16[tagStart...].index(of: close) let tag = html[tagStart...tagEnd] ``` -A property and an intializer will be added to `String.Index`, exposing +A property and an initializer will be added to `String.Index`, exposing the offset of the index in code units (currently only UTF-16) from the beginning of the string: diff --git a/proposals/0181-package-manager-cpp-language-version.md b/proposals/0181-package-manager-cpp-language-version.md index 2c1bf3a9ff..5fd52c8f02 100644 --- a/proposals/0181-package-manager-cpp-language-version.md +++ b/proposals/0181-package-manager-cpp-language-version.md @@ -3,7 +3,7 @@ * Proposal: [SE-0181](0181-package-manager-cpp-language-version.md) * Author: [Ankit Aggarwal](https://github.com/aciidb0mb3r) * Review Manager: [Daniel Dunbar](https://github.com/ddunbar) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-revision-se-0181-package-manager-c-c-language-standard-support/6353) * Implementation: [apple/swift-package-manager#1264](https://github.com/apple/swift-package-manager/pull/1264) diff --git a/proposals/0182-newline-escape-in-strings.md b/proposals/0182-newline-escape-in-strings.md index 285999e649..6f3cfb9fc9 100644 --- a/proposals/0182-newline-escape-in-strings.md +++ b/proposals/0182-newline-escape-in-strings.md @@ -3,7 +3,7 @@ * Proposal: [SE-0182](0182-newline-escape-in-strings.md) * Authors: [John Holdsworth](https://github.com/johnno1962), [David Hart](https://github.com/hartbit), [Adrian Zubarev](https://github.com/DevAndArtist) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Implementation: [apple/swift#11080](https://github.com/apple/swift/pull/11080) * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0182-string-newline-escaping/6355) * Previous Proposal: [SE-0168](0168-multi-line-string-literals.md) diff --git a/proposals/0183-substring-affordances.md b/proposals/0183-substring-affordances.md index 521a82f3ae..eb83064787 100644 --- a/proposals/0183-substring-affordances.md +++ b/proposals/0183-substring-affordances.md @@ -3,7 +3,7 @@ * Proposal: [SE-0183](0183-substring-affordances.md) * Author: [Ben Cohen](https://github.com/airspeedswift) * Review Manager: [Chris Lattner](https://github.com/lattner) -* Status: **Implemented (Swift 4)** +* Status: **Implemented (Swift 4.0)** * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0183-substring-performance-affordances/6393) * Bug: [SR-4933](https://bugs.swift.org/browse/SR-4933) @@ -74,7 +74,7 @@ case of `filter`). ## Effect on ABI stability -The switch from conrete to generic types needs to be made before ABI stability. +The switch from concrete to generic types needs to be made before ABI stability. ## Alternatives considered diff --git a/proposals/0184-unsafe-pointers-add-missing.md b/proposals/0184-unsafe-pointers-add-missing.md index d108e06646..a8cff99c6b 100644 --- a/proposals/0184-unsafe-pointers-add-missing.md +++ b/proposals/0184-unsafe-pointers-add-missing.md @@ -1,7 +1,7 @@ # Unsafe[Mutable][Raw][Buffer]Pointer: add missing methods, adjust existing labels for clarity, and remove deallocation size * Proposal: [SE-0184](0184-unsafe-pointers-add-missing.md) -* Author: [Kelvin Ma (“Taylor Swift”)](https://github.com/kelvin13) +* Author: [Diana Ma (“Taylor Swift”)](https://github.com/tayloraswift) * Review Manager: [Doug Gregor](https://github.com/DougGregor) * Status: **Implemented (Swift 4.1)** * Implementation: [apple/swift#12200](https://github.com/apple/swift/pull/12200) @@ -10,13 +10,13 @@ ## Introduction -*This document is a spin-off from a much larger [original proposal](https://github.com/kelvin13/swift-evolution/blob/e888af466c9993de977f6999a131eadd33291b06/proposals/0184-unsafe-pointers-add-missing.md), which covers only those aspects of SE-0184 which do not deal with partial buffer memory state. Designing the partial buffer memory state API clearly requires more work, and has been left out of the scope of this document.* +*This document is a spin-off from a much larger [original proposal](https://github.com/tayloraswift/swift-evolution/blob/e888af466c9993de977f6999a131eadd33291b06/proposals/0184-unsafe-pointers-add-missing.md), which covers only those aspects of SE-0184 which do not deal with partial buffer memory state. Designing the partial buffer memory state API clearly requires more work, and has been left out of the scope of this document.* Swift’s pointer types are an important interface for low-level memory manipulation, but the current API design is not very consistent, complete, or convenient. In some places, poor naming choices and overengineered function signatures compromise memory safety by leading users to believe that they have allocated or freed memory when in fact, they have not. This proposal seeks to improve the Swift pointer API by ironing out naming inconsistencies, adding missing methods, and reducing excessive verbosity, offering a more convenient, more sensible, and less bug-prone API. Swift-evolution threads: [Pitch: Improved Swift pointers](https://forums.swift.org/t/pitch-improved-swift-pointers/6318), [Pitch: More Improved Swift pointers](https://forums.swift.org/t/pitch-improved-swift-pointers/6318) -Implementation branch: [`kelvin13:se-0184a`](https://github.com/kelvin13/swift/tree/se-0184a) +Implementation branch: [`tayloraswift:se-0184a`](https://github.com/tayloraswift/swift/tree/se-0184a) ## Background diff --git a/proposals/0187-introduce-filtermap.md b/proposals/0187-introduce-filtermap.md index 58fc221b71..a03909edc1 100644 --- a/proposals/0187-introduce-filtermap.md +++ b/proposals/0187-introduce-filtermap.md @@ -10,7 +10,7 @@ [Review #1](https://forums.swift.org/t/review-se-0187-introduce-sequence-filtermap/6977), [Review #2](https://forums.swift.org/t/accepted-and-focused-re-review-se-0187-introduce-sequence-filtermap/7076), [Rationale](https://forums.swift.org/t/accepted-with-revisions-se-0187-introduce-sequence-filtermap/7290) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/2d24b0ce9f138858b8341467170d6d8ba973827f/proposals/0187-introduce-filtermap.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/2d24b0ce9f138858b8341467170d6d8ba973827f/proposals/0187-introduce-filtermap.md) ## Introduction diff --git a/proposals/0191-eliminate-indexdistance.md b/proposals/0191-eliminate-indexdistance.md index 917cbf0340..73a23ed9d8 100644 --- a/proposals/0191-eliminate-indexdistance.md +++ b/proposals/0191-eliminate-indexdistance.md @@ -17,8 +17,8 @@ Eliminate the associated type `IndexDistance` from `Collection`, and modify all an `Int`, generic algorithms on `Collection` need to either constrain `IndexDistance == Int` or write their algorithm to be generic over any `SignedInteger`. Swift 4.0 introduced the ability to constrain associated types with `where` clauses -([SE-142](https://github.com/apple/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md)) and will soon allow protocol constraints -to be recursive ([SE-157](https://github.com/apple/swift-evolution/blob/master/proposals/0157-recursive-protocol-constraints.md)). With these features, +([SE-142](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md)) and will soon allow protocol constraints +to be recursive ([SE-157](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0157-recursive-protocol-constraints.md)). With these features, writing generic algorithms against `Collection` is finally a realistic tool for intermediate Swift programmers. You no longer need to know to constrain `SubSequence.Element == Element` or `SubSequence: Collection`, missing constraints that previously led to inexplicable error messages. diff --git a/proposals/0192-non-exhaustive-enums.md b/proposals/0192-non-exhaustive-enums.md index e9ba9a042c..534cef4c91 100644 --- a/proposals/0192-non-exhaustive-enums.md +++ b/proposals/0192-non-exhaustive-enums.md @@ -3,9 +3,9 @@ * Proposal: [SE-0192](0192-non-exhaustive-enums.md) * Author: [Jordan Rose](https://github.com/jrose-apple) * Review Manager: [Ted Kremenek](https://github.com/tkremenek) -* Status: **Implemented (Swift 5)** +* Status: **Implemented (Swift 5.0)** * Implementation: [apple/swift#14945](https://github.com/apple/swift/pull/14945) -* Previous revision: [1](https://github.com/apple/swift-evolution/blob/a773d07ff4beab8b7855adf0ac56d1e13bb7b44c/proposals/0192-non-exhaustive-enums.md), [2 (informal)](https://github.com/jrose-apple/swift-evolution/blob/57dfa2408fe210ed1d5a1251f331045b988ee2f0/proposals/0192-non-exhaustive-enums.md), [3](https://github.com/apple/swift-evolution/blob/af284b519443d3d985f77cc366005ea908e2af59/proposals/0192-non-exhaustive-enums.md) +* Previous revision: [1](https://github.com/swiftlang/swift-evolution/blob/a773d07ff4beab8b7855adf0ac56d1e13bb7b44c/proposals/0192-non-exhaustive-enums.md), [2 (informal)](https://github.com/jrose-swiftlang/swift-evolution/blob/57dfa2408fe210ed1d5a1251f331045b988ee2f0/proposals/0192-non-exhaustive-enums.md), [3](https://github.com/swiftlang/swift-evolution/blob/af284b519443d3d985f77cc366005ea908e2af59/proposals/0192-non-exhaustive-enums.md) * Pre-review discussion: [Enums and Source Compatibility](https://forums.swift.org/t/enums-and-source-compatibility/6460), with additional [orphaned thread](https://forums.swift.org/t/enums-and-source-compatibility/6651) * Review discussion: [Review author summarizes some feedback from review discussion and proposes alternatives](https://forums.swift.org/t/se-0192-non-exhaustive-enums/7291/26), [full discussion thread](https://forums.swift.org/t/se-0192-non-exhaustive-enums/7291/337), plus [Handling unknown cases in enums](https://forums.swift.org/t/handling-unknown-cases-in-enums-re-se-0192/7388/) * Decision Notes: [Rationale](https://forums.swift.org/t/se-0192-non-exhaustive-enums-review-2/11043/62) @@ -561,7 +561,7 @@ The core team decided that this feature was not worth the disruption and long-te The [initial version][] of this proposal did not include `@unknown`, and required people to use a normal `default` to handle cases added in the future instead. However, many people were unhappy with the loss of exhaustivity checking for `switch` statements, both for enums in libraries distributed as source and enums imported from Apple's SDKs. While this is an additive feature that does not affect ABI, it seems to be one that the community considers a necessary part of a language model that provides non-frozen enums. - [initial version]: https://github.com/apple/swift-evolution/blob/a773d07ff4beab8b7855adf0ac56d1e13bb7b44c/proposals/0192-non-exhaustive-enums.md + [initial version]: https://github.com/swiftlang/swift-evolution/blob/a773d07ff4beab8b7855adf0ac56d1e13bb7b44c/proposals/0192-non-exhaustive-enums.md ### Mixing `@unknown` with other catch-all cases diff --git a/proposals/0193-cross-module-inlining-and-specialization.md b/proposals/0193-cross-module-inlining-and-specialization.md index 7daee5b463..bbdcc7305a 100644 --- a/proposals/0193-cross-module-inlining-and-specialization.md +++ b/proposals/0193-cross-module-inlining-and-specialization.md @@ -75,7 +75,7 @@ The attribute cannot be applied to local declarations, that is, declarations nes When applied to a subscript or computed property, the attribute applies to both the getter and setter. -Note that only delegating initializers (those that assign to `self` or call another initializer via `self.init`) can be inlinable. Root initializers which initialize the stored properties of a struct or class directly cannot be inlinable. For motivation, see [SE-0189 Restrict Cross-module Struct Initializers](https://github.com/apple/swift-evolution/blob/master/proposals/0189-restrict-cross-module-struct-initializers.md). +Note that only delegating initializers (those that assign to `self` or call another initializer via `self.init`) can be inlinable. Root initializers which initialize the stored properties of a struct or class directly cannot be inlinable. For motivation, see [SE-0189 Restrict Cross-module Struct Initializers](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0189-restrict-cross-module-struct-initializers.md). ### Inlinable contexts @@ -171,7 +171,7 @@ The closest analogue in C to `@usableFromInline` is a non-`static` function that ## Alternatives considered -One possible alterative would be to add a new compiler mode where _all_ declarations become implicitly `@inlinable`, and _all_ private and internal declarations become `@usableFromInline`. +One possible alternative would be to add a new compiler mode where _all_ declarations become implicitly `@inlinable`, and _all_ private and internal declarations become `@usableFromInline`. However, such a compilation mode would not solve the problem of delivering a stable ABI and standard library which can be deployed separately from user code. We _don't want_ all declaration bodies in the standard library to be available to the optimizer when building user code. diff --git a/proposals/0195-dynamic-member-lookup.md b/proposals/0195-dynamic-member-lookup.md index 55b18c7acb..3989a15b50 100644 --- a/proposals/0195-dynamic-member-lookup.md +++ b/proposals/0195-dynamic-member-lookup.md @@ -4,7 +4,7 @@ * Author: [Chris Lattner](https://github.com/lattner) * Review Manager: [Ted Kremenek](https://github.com/tkremenek) * Implementation: [apple/swift#14546](https://github.com/apple/swift/pull/14546) -* [Previous Revision #1](https://github.com/apple/swift-evolution/commit/59c7455170c231f3df9ab0ba923262e126afaa06#diff-b3460d13f154c3d6b1d8396e4159a1d2) +* [Previous Revision #1](https://github.com/swiftlang/swift-evolution/commit/59c7455170c231f3df9ab0ba923262e126afaa06#diff-b3460d13f154c3d6b1d8396e4159a1d2) * Status: **Implemented (Swift 4.2)** * Decision Notes: [Review extended](https://forums.swift.org/t/se-0195-introduce-user-defined-dynamic-member-lookup-types/8658/126), [Rationale](https://forums.swift.org/t/se-0195-introduce-user-defined-dynamic-member-lookup-types/8658/160) diff --git a/proposals/0196-diagnostic-directives.md b/proposals/0196-diagnostic-directives.md index 9d6a940864..2b69a98991 100644 --- a/proposals/0196-diagnostic-directives.md +++ b/proposals/0196-diagnostic-directives.md @@ -4,7 +4,7 @@ * Author: [Harlan Haskins](https://github.com/harlanhaskins) * Review Manager: [Ted Kremenek](https://github.com/tkremenek) * Implementation: [apple/swift#14048](https://github.com/apple/swift/pull/14048) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/ab0c22a2340be9bfcb82e6f237752b4d959a93b7/proposals/0196-diagnostic-directives.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/ab0c22a2340be9bfcb82e6f237752b4d959a93b7/proposals/0196-diagnostic-directives.md) * Status: **Implemented (Swift 4.2)** ## Introduction @@ -152,10 +152,10 @@ to this proposal, and both could be addressed in future proposals. # Rationale On February 1, 2018 the Core Team decided to **accept** this proposal with -slight revision over the [original proposal](https://github.com/apple/swift-evolution/blob/ab0c22a2340be9bfcb82e6f237752b4d959a93b7/proposals/0196-diagnostic-directives.md). +slight revision over the [original proposal](https://github.com/swiftlang/swift-evolution/blob/ab0c22a2340be9bfcb82e6f237752b4d959a93b7/proposals/0196-diagnostic-directives.md). The only revision over the original proposal is to change the syntax to use -`#warning()` instead of `#warning `. This fits well with +`#warning()` instead of `#warning `. This fits well with most of Swift's existing compiler directives, and was strongly supported in the [review discussion](https://forums.swift.org/t/se-0196-compiler-diagnostic-directives/8734). diff --git a/proposals/0197-remove-where.md b/proposals/0197-remove-where.md index e74c332e92..514aabc2cf 100644 --- a/proposals/0197-remove-where.md +++ b/proposals/0197-remove-where.md @@ -6,7 +6,7 @@ * Status: **Implemented (Swift 4.2)** * Implementation: [apple/swift#11576](https://github.com/apple/swift/pull/11576) * Review: [Thread](https://forums.swift.org/t/se-0197-add-in-place-remove-where/8872) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/feec7890d6c193e9260ac9905456f25ef5656acd/proposals/0197-remove-where.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/feec7890d6c193e9260ac9905456f25ef5656acd/proposals/0197-remove-where.md) ## Introduction diff --git a/proposals/0198-playground-quicklook-api-revamp.md b/proposals/0198-playground-quicklook-api-revamp.md index 59b4d0722f..7b6608d542 100644 --- a/proposals/0198-playground-quicklook-api-revamp.md +++ b/proposals/0198-playground-quicklook-api-revamp.md @@ -261,7 +261,7 @@ support the replacement of `CustomPlaygroundQuickLookable` with `CustomPlaygroundDisplayConvertible`. Instead, we intend for Swift 4.1 to be a deprecation period for these APIs, allowing any code bases which implement `CustomPlaygroundQuickLookable` to manually switch to the new protocol. While -this migration may not be trivial programatically, it should -- in most cases -- +this migration may not be trivial programmatically, it should -- in most cases -- be fairly trivial for someone to hand-migrate to `CustomPlaygroundDisplayConvertible`. During the deprecation period, the PlaygroundLogger framework will continue to honor implementations of diff --git a/proposals/0199-bool-toggle.md b/proposals/0199-bool-toggle.md index 819153e336..b9a4c22384 100644 --- a/proposals/0199-bool-toggle.md +++ b/proposals/0199-bool-toggle.md @@ -1,7 +1,7 @@ # Adding `toggle` to `Bool` * Proposal: [SE-0199](0199-bool-toggle.md) -* Author: [Chris Eidhof](http://chris.eidhof.nl) +* Author: [Chris Eidhof](https://github.com/chriseidhof) * Review Manager: [Ben Cohen](https://github.com/airspeedswift/) * Status: **Implemented (Swift 4.2)** * Decision notes: [Rationale](https://forums.swift.org/t/accepted-se-199-add-toggle-to-bool/10681) diff --git a/proposals/0200-raw-string-escaping.md b/proposals/0200-raw-string-escaping.md index 1032224a4e..b08508ca0c 100644 --- a/proposals/0200-raw-string-escaping.md +++ b/proposals/0200-raw-string-escaping.md @@ -3,8 +3,8 @@ * Proposal: [SE-0200](0200-raw-string-escaping.md) * Authors: [John Holdsworth](https://github.com/johnno1962), [Becca Royal-Gordon](https://github.com/beccadax), [Erica Sadun](https://github.com/erica) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/102b2f2770f0dab29f254a254063847388647a4a/proposals/0200-raw-string-escaping.md) -* Status: **Implemented (Swift 5)** +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/102b2f2770f0dab29f254a254063847388647a4a/proposals/0200-raw-string-escaping.md) +* Status: **Implemented (Swift 5.0)** * Implementation: [apple/swift#17668](https://github.com/apple/swift/pull/17668) * Bugs: [SR-6362](https://bugs.swift.org/browse/SR-6362) * Review: [Discussion thread](https://forums.swift.org/t/se-0200-enhancing-string-literals-delimiters-to-support-raw-text/15420), [Announcement thread](https://forums.swift.org/t/accepted-se-0200-enhancing-string-literals-delimiters-to-support-raw-text/15822/2) @@ -150,9 +150,9 @@ Removing escaped snippets to external files makes code review harder. Escaping ( ## Initial Proposal -"Raw-mode" strings were first discussed during the [SE-0168 Multi-Line String literals](https://github.com/apple/swift-evolution/blob/master/proposals/0168-multi-line-string-literals.md) review and postponed for later consideration. This proposal focuses on raw strings to allow the entry of single and multi-line string literals. +"Raw-mode" strings were first discussed during the [SE-0168 Multi-Line String literals](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0168-multi-line-string-literals.md) review and postponed for later consideration. This proposal focuses on raw strings to allow the entry of single and multi-line string literals. -The first iteration of [SE-0200](https://github.com/apple/swift-evolution/blob/102b2f2770f0dab29f254a254063847388647a4a/proposals/0200-raw-string-escaping.md) proposed adopting Python's model, using `r"...raw string..."`. The proposal was returned for revision with the [following feedback](https://forums.swift.org/t/returned-for-revision-se-0200-raw-mode-string-literals/11630): +The first iteration of [SE-0200](https://github.com/swiftlang/swift-evolution/blob/102b2f2770f0dab29f254a254063847388647a4a/proposals/0200-raw-string-escaping.md) proposed adopting Python's model, using `r"...raw string..."`. The proposal was returned for revision with the [following feedback](https://forums.swift.org/t/returned-for-revision-se-0200-raw-mode-string-literals/11630): > During the review discussion, a few issues surfaced with the proposal, including: > @@ -361,7 +361,7 @@ The same behavior is extended to multi-line strings: """# ``` -New line escaping works as per [SE-182](https://github.com/apple/swift-evolution/blob/master/proposals/0182-newline-escape-in-strings.md): +New line escaping works as per [SE-182](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0182-newline-escape-in-strings.md): ``` #""" diff --git a/proposals/0202-random-unification.md b/proposals/0202-random-unification.md index 81744c6ea4..48eeb31a90 100644 --- a/proposals/0202-random-unification.md +++ b/proposals/0202-random-unification.md @@ -2,7 +2,7 @@ * Proposal: [SE-0202](0202-random-unification.md) * Author: [Alejandro Alonso](https://github.com/Azoy) -* Review Manager: [Ben Cohen](http://github.com/AirspeedSwift/) +* Review Manager: [Ben Cohen](https://github.com/AirspeedSwift/) * Status: **Implemented (Swift 4.2)** * Implementation: [apple/swift#12772](https://github.com/apple/swift/pull/12772) * Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-020-random-unification/12040) diff --git a/proposals/0204-add-last-methods.md b/proposals/0204-add-last-methods.md index 559e60c55a..8726f9cadb 100644 --- a/proposals/0204-add-last-methods.md +++ b/proposals/0204-add-last-methods.md @@ -118,7 +118,7 @@ This change does not affect ABI stability or API resilience beyond the addition - Another previous proposal included renaming `index(of:)` and `index(where:)` to `firstIndex(of:)` and `firstIndex(where:)`, respectively. A proposal after *that one* removed that source-breaking change. *This* version of the proposal adds the source-breaking change back in again. -- An [alternative approach](https://github.com/apple/swift-evolution/pull/773#issuecomment-351148673) is to add a defaulted `options` parameter to the existing searching methods, like so: +- An [alternative approach](https://github.com/swiftlang/swift-evolution/pull/773#issuecomment-351148673) is to add a defaulted `options` parameter to the existing searching methods, like so: ```swift let a = [20, 30, 10, 40, 20, 30, 10, 40, 20] diff --git a/proposals/0206-hashable-enhancements.md b/proposals/0206-hashable-enhancements.md index ce6a0d5106..289cb6d985 100644 --- a/proposals/0206-hashable-enhancements.md +++ b/proposals/0206-hashable-enhancements.md @@ -9,7 +9,7 @@ [apple/swift#14913](https://github.com/apple/swift/pull/14913) (standard library, underscored),
[apple/swift#16009](https://github.com/apple/swift/pull/16009) (`Hasher` interface),
[apple/swift#16073](https://github.com/apple/swift/pull/16073) (automatic synthesis, de-underscoring)
-* Previous Revision: [1](https://github.com/apple/swift-evolution/blob/f5a020ec79cdb64fc8700af91b1a1ece2d2fb141/proposals/0206-hashable-enhancements.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/f5a020ec79cdb64fc8700af91b1a1ece2d2fb141/proposals/0206-hashable-enhancements.md) ### Stored Property Isolation -The stored properties of classes, structs, and enums are currently permitted to have global-actor isolation applied to them. But, this creates a problems for both initialization and deinitialization. For example, when users specify a default value for the stored property, those default values are evaluated by the non-delegating initializer of a nominal type: +The classes, structs, and enums are currently permitted to have global-actor isolation independently applied to each of their stored properties. When programmers specify a default value for a stored property, those default values are computed by each of the non-delegating initializers of the type. The problem is that the expressions for those default values are treated as though they are running on that global-actor's executor. So, it is possible to create impossible constraints on those initializers: ```swift @MainActor func getStatus() -> Int { /* ... */ } @@ -172,29 +192,16 @@ class Process { @PIDActor var pid: ProcessID = genPID() init() {} // Problem: what is the isolation of this init? - - init() async {} // Problem: no `await` is written to acknowledge - // that to initialize `status` and `pid`, an - // async call would be required. - - deinit { - // Problem: how do we release the resources contained - // in our global-actor isolated stored properties from - // a deinit, which can never be actor-isolated? - } } ``` -In the example above, because `status` and `pid` are isolated to two different global-actors, there's no single actor-isolation that can be specified for the synchronous `init`. -In fact, all non-delegating initializers would need to have the same isolation as all stored properties. -For the asynchronous `init`, the fact that a suspension may occur is not explicit in the program, because no `await` is needed on the right-hand side expression of the property declaration's assignment. -Finally, even if the isolation of the initializers and stored properties matched, the deinit still can _never_ access the stored properties in order to invoke clean-ups routines, without using unsafe lifetime extensions of the actor from the `deinit`. +The example above is accepted in Swift 5.5, but is impossible to implement. Because `status` and `pid` are isolated to two different global-actors, there is no single actor-isolation that can be specified for the non-async `init`. In fact, it's not possible to perform the appropriate actor hops within the non-async initializer. As a result, `getStatus` and `genPID` are being called without hopping to the appropriate executor. ### Initializer Delegation -All nominal types in Swift, except actors, explicitly support initializer delegation, which is when one initializer calls another one to perform initialization. +All nominal types in Swift support initializer delegation, which is when one initializer calls another one to perform the rest of the initialization. For classes, initializer [delegation rules](https://docs.swift.org/swift-book/LanguageGuide/Initialization.html#ID216) are complex due to the presence of inheritance. -So, classes have a required and explicit `convenience` modifier to make, for example, a distinction between initializers that *must* delegate and those that do not. +So, classes have a required and explicit `convenience` modifier to make a distinction between initializers that delegate. In contrast, value types do *not* support inheritance, so [the rules](https://docs.swift.org/swift-book/LanguageGuide/Initialization.html#ID215) are much simpler: any `init` can delegate, but if it does, then it must delegate or assign to `self` in all cases: ```swift @@ -211,334 +218,674 @@ struct S { } ``` -Actors, which are reference types (like a classes), do not support inheritance. But, currently they must use the `convenience` modifier on an initializer to perform any delegation. Is this modifier still needed? +Unlike classes, actors do not support inheritance. But, the proposal for actors did not specify whether `convenience` is required or not in order to have a delegating initializer. Yet, Swift 5.5 requires the use of a `convenience` modifier to mark actor initializers that perform delegation. + + - -## Proposed solution -The previous sections described problems with the current state of actor initialization and deinitialization, as listed in the introduction of the Motivation section. -The remainder of this section details the proposed solution to those problems. -### Problem 1: Initializer Data Races -This proposal aims to eliminate data races through the selective application of a usage restriction on `self` in an actor's initializer. -For this discussion, an _escaping use of `self`_ means that a copy of `self` is exposed outside of the actor's initializer, before the initializer has finished. -By rejecting programs with escaping uses of `self`, there is no way to construct the data race described earlier. -> **NOTE:** Preventing `self` from escaping the `init` directly resolves the data race, because it forces the unique reference `self` to stay on the current thread until the completion of the `init`. -Specifically, the only way to create a race is for there to be at least two copies of the reference `self`. -Since a secondary thread can only gain access to a copy of `self` by having it "escape" the `init`, preventing the escape closes the possibility of a race. -An actor's initializer that obeys the escaping-use restriction means that the following are rejected throughout the entire initializer: -- Capturing `self` in a closure. -- Calling a method or computed property on `self`. -- Passing `self` as any kind of argument, whether by-value, `autoclosure`, or `inout`. -The escaping-use restriction is not a new concept in Swift: for all nominal types, a very similar kind of restriction is applied to `self` until it becomes fully-initialized. -#### Applying the Escaping-use Restriction -If an actor's non-delegating initializer is synchronous or isolated to a global-actor, then it must obey the escaping-use restriction. -This leaves only the instance-isolated `async` actor initializer, and all delegating initializers, as being free from this new restriction. -For a synchronous initializer, we cannot reserve the actor's executor by hopping to it from a synchronous context. -Thus, the need for the restriction is clear: the only way to prevent simultaneous access to the actor's state is to prevent another thread from getting a copy of `self`. -In contrast, an instance-isolated `async` initializer _will_ perform that hop immediately after `self` is fully-initialized in the `init`, so no restriction is applied. -For a global-actor isolated initializer, the need for the escaping-use restriction is a bit more subtle. -In Swift's type system, a declaration cannot be isolated to _two_ actors at the same time. -Because the programmer has to opt-in to global-actor isolation, it takes precedence when appearing on the `init` of an actor type and will be respected. -In such cases, protection for the `self` actor instance, after it is fully-initialized, is provided by the escaping-use restriction. -This means that, within an `init` isolated to some global-actor `A`, the stored properties of `self` belonging to a different actor `B` can be accessed *without* synchronization. -Thus, the `ConnectionManager` example from earlier will work as-is, because only stored properties of the actor-instance `self` are accessed. -### Problem 2: Stored Property Isolation -Actor-isolation on a stored property only prevents concurrent access to the storage for the value, and not subsequent accesses. -For example, if `pid` is an actor-isolated stored property, then the access `p.pid.reset()` only protects the access of `pid` from `p`, and not the call to `reset` afterwards. -Thus, for value types (enums and structs), global-actor isolation on stored properties serves virtually no use: mutations of stored properties in value types can never race (due to copy-on-write semantics). -The [global actors](0316-global-actors.md) proposal explicitly excludes actor types from having stored properties that are global-actor isolated. -The only nominal type left in Swift to consider are classes. For a class, the benefit of global-actor isolated stored properties is to prevent races during an access. But, because a `deinit` cannot be made `async`, and it is undefined behavior for a class value's lifetime to extend beyond the invocation of a `deinit`, there would be no way to access the stored property during a `deinit`. -In summary, the most straightforward solution to the problems described earlier is: global-actor isolation should not apply to the stored properties appearing within _any_ nominal type. -### Problem 3: Initializer Delegation +## Proposed functionality -Next, one of the key downsides of the escaping-use restriction is that it becomes impossible to invoke a method in the time *after* `self` is fully-initialized, but *before* a non-delegating `init` returns. -This pattern is important, for example, to organize set-up code that is needed both during initialization and the lifetime of the instance: +The previous sections briefly described some problems with the current state of initialization and deinitialization in Swift. +The remainder of this section aims to fix those problems while defining how actor and global-actor isolated type (GAIT) initializers and deinitializers differ from those belonging to an ordinary class. While doing so, this proposal will highlight how the problems above are resolved. + +### Non-delegating Initializers + +A non-delegating initializer of an actor or a global-actor isolated type (GAIT) is required to initialize all of the stored properties of that type. + +#### Flow-sensitive Actor Isolation + +The focus of this section is on non-delegating initializers for `actor` types, not GAITs. +In Swift 5.5, an actor's initializer that obeys the _escaping-use restriction_ means that the following are rejected throughout the entire initializer: + +- Capturing `self` in a closure. +- Calling a method or computed property on `self`. +- Passing `self` as any kind of argument, whether by-value, `autoclosure`, or `inout`. + +But, those rules are an over-approximation of the restrictions needed to prevent the races described earlier. This proposal removes the escaping-use restriction for initializers. Instead, we propose a simpler set of rules. First we define two categories of initializers, distinguished by their isolation: + +- An initializer has a `nonisolated self` reference if it is: + - non-async + - or global-actor isolated + - or `nonisolated` +- Asynchronous actor initializers have an `isolated self` reference. + +The remainder of this section discusses how these two classes of initializers work. + +##### Initializers with `isolated self` + +For an asynchronous initializer, a hop to the actor's executor will be performed immediately after `self` becomes fully-initialized, in order to ascribe the isolation to `self`. Choosing this location for performing the executor hop preserves the concept of `self` being isolated throughout the entire async initializer. That is, before any escaping uses of `self` can happen in an initializer, the executor hop has been performed. + +It's important to recognize that an executor hop is a suspension point. There are many possible points in an initializer where these suspensions can happen, since there are multiple places where a store to `self` cause it to become initialized. Consider this example of `Bob`: ```swift -actor A { - var friends: [A] +actor Bob { + var x: Int + var y: Int = 2 + func f() {} + init(_ cond: Bool) async { + if cond { + self.x = 1 // initializing store + } + self.x = 2 // initializing store - init(withFriends fs: [A]) { - friends = fs - self.notifyAll() // ❌ disallowed by escaping-use restriction. + f() // this is ok, since we're on the executor here. } +} +``` - @MainActor - init() { - friends = ... - self.notifyAll() // ❌ disallowed by escaping-use restriction. +The problem with trying to explicitly mark the suspension points in `Bob.init` is that they are not easy for programmers to track, nor are they consistent enough to stay the same under simple refactorings. Adding or removing a default value for a stored property, or changing the number of stored properties, can greatly influence where the hops may occur. Consider this slightly modified example from before: + +```swift +actor EvolvedBob { + var x: Int + var y: Int + func f() {} + init(_ cond: Bool) async { + if cond { + self.x = 1 + } + self.x = 2 + self.y = 2 // initializing store + + f() // this is ok, since we're on the executor here. } +} +``` + +Relative to `Bob`, the only change made to `EvolvedBob` is that its default value for `y` was converted into an unconditional store in the body of the initializer. From an observational point of view, `Bob.init` and `EvolvedBob.init` are identical. But from an implementation perspective, the suspension points for performing an executor hop differ dramatically. If those points required some sort of annotation in Swift, such as with `await`, then the reason why those suspension points moved is hard to explain to programmers. - func verify() { ... } - func notifyAll() { ... } +In summary, we propose to _implicitly_ perform suspensions to hop to the actors executor once `self` is initialized, instead of having programmers mark those points explicitly, for the following reasons: + +- The finding and continually updating the suspension points is annoying for programmers. +- The reason _why_ some simple stores to a property can trigger a suspension is an implementation detail that is hard to explain to programmers. +- The benefits of marking these suspensions is very low. The reference to `self` is known to be unique by the time the suspension will happen, so it is impossible to create an [actor reentrancy](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md#actor-reentrancy) situation. +- There is [already precedent](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0317-async-let.md#requiring-an-awaiton-any-execution-path-that-waits-for-an-async-let) in the language for performing implicit suspensions, namely for `async let`, when the benefits outweigh the negatives. + +The net effect of these implicit executor-hops is that, for programmers, an `async` initializer does not appear to have any additional rules added to it! That is, programmers can simply view the initializer as being isolated throughout, like any ordinary `async` method would be! The flow-sensitive points where the hop is inserted into the initializer can be safely ignored as an implementation detail for all but the most rare situations. For example: + +```swift +actor OddActor { + var x: Int + init() async { + let name = Thread.current.name + self.x = 0 // initializing store + assert(name == Thread.current.name) // may fail + } } ``` -Another important observation is that an isolated initializer that performs delegation is not particularly useful. -A delegating initializer that is synchronous would still need to obey the escaping-use restriction, but now they also must first call some other `init` on all paths. -But, _because_ an `init` must be called first on all paths of a delegating `init`, such an initializer has an explicit point where `self` is fully-initialized. -This provides an excellent opportunity to perform _follow-up work_, after `self` is fully-initialized, but before completely returning from initialization. -To do the follow-up work in a delegating init, we must be in a context that is not isolated to the actor instance, because the initialized instance's executor starts in an unreserved state. -In addition, because _all_ initializers are viewed as `nonisolated` from the outside, an entire body of the delegating initializer can be cleanly treated as `nonisolated`! +Note that the callers of `OddActor.init` cannot assume that the callee hasn't performed a suspension, just as with any `async` method, because an `await` is required to enter the initializer. Thus, this ability to observe an unmarked suspension is extremely limited. + +**In-depth discussions** -For ABI compatibility reasons with Swift 5.5, and to make the implicit `nonisolated` semantics clear, this proposal keeps the `convenience` modifier for actor initializers, as a way to mark initializers that _must_ delegate. -If a programmer marks a convenience initializer with `nonisolated`, a warning will be emitted that says it is a redundant modifier, since `convenience` implies `nonisolated`. -Global-actor isolation of a `convenience` init is allowed, and will override the implicit `nonisolated` behavior. -Rewriting the above with this new rule would look like this: +The remainder of this subsection covers some technical details that are not required to understand this proposal and may be safely skipped. + +**Compiler Implementation Notes:** Identifying the assignment that fully-initializes `self` _does_ require a non-trivial data-flow analysis. Such an analysis is not feasible to do early in the compiler, during type checking. Does acceptance of this proposal mean that the actor-isolation checker, which is run as part of type-checking, will require additional analysis or significant changes? Nope! We can rely on existing restrictions on uses of `self`, prior to initialization, to exclude all places where `self` could be considered only `nonisolated`: ```swift -// NOTE: Task.detached is _not_ an exact substitute for this. -// It is expected that Custom Executors will provide a capability -// that implements this function, which atomically enqueues a paused task -// on the target actor before returning. -func spawnAndEnqueueTask(_ a: A, _ f: () -> Void) { ... } +func isolatedFunc(_ a: isolated Alice) {} -actor A { - var friends: [A] +actor Alice { + var x: Int + var y: Task - private init(with fs: [A]) { - friends = fs - } + nonisolated func nonisolatedMethod() {} + func isolatedMethod() {} - // Version 1: synchronous delegating initializer - convenience init() { - self.init(with: ...) - // ✅ self can be captured by closure, or passed as argument - spawnAndEnqueueTask(self) { - await self.notifyAll() + init() async { + self.x = self.nonisolatedMethod() // error: illegal use of `self` before initialization. + self.y = Task { self.isolatedMethod() } // error: illegal capture of `self` before initialization + Task { + self.isolatedMethod() // no await needed, since `self` is isolated. } + self.isolatedMethod() // OK + isolatedFunc(self) // OK } +} +``` - // Version 2: asynchronous delegating initializer - convenience init(withFakeFriends f: Double) async { - if f < 0 { - self.init() - } else { - self.init(with: manufacturedFriends(count: Int(f))) - await self.notifyAll() +This means that the actor-isolation checker, run prior to converting the program to SIL, can uniformly view the parameter `self` as having type `isolated Self` for the async initializer above. Later in SIL, the defined-before-use verification (i.e., "definite initialization") will find and emit the errors above. As a bonus, that same analysis can be leveraged to find the initializing assignment and introduce the suspension to hop to the actor's executor. + +**Data-race Safety:** In terms of correctness, the proposed `isolated self` initializers are race-free because a hop to the actor's executor happens immediately after the initializing store to `self`, but before the next statement begins executing. Gaining access to the executor at this exact point prevents races, because escaping `self` to another task is only possible _after_ that point. In the `Alice` example above, we can see this in action, where the rejected assignment to `self.y` is due to an illegal capture of `self`. + +**Only one suspension is performed:** It is possible to construct an initializer with control-flow that crosses an implicit suspension points multiple times, as seen in `Bob` above and loops such as: + +```swift +actor LoopyBob { + var x: Int + init(_ counter: Int) async { + var i = 0 + repeat { + self.x = 0 // initializing store + i += 1 + } while i < counter + } +} +``` + +Once gaining access to an executor by crossing the first suspension point, crossing another suspension point does not change the executor, nor will that actually perform a suspension. Avoiding these unnecessary executor hops is an optimization that is done throughout Swift (e.g., self-recursive `async` and `isolated` functions). + + + +##### Initializers with `nonisolated self` + +The category of actor initializers that have a `nonisolated self` contain those which are non-async, or have an isolation that differs from being isolated to `self`. Unlike its methods, an actor's non-async initializer does _not_ require an `await` to be invoked, because there is no actor-instance to synchronize with. In addition, an initializer with a `nonisolated self` can access the instance's stored properties without synchronization, when it is safe to do so. + +Accesses to the stored properties of `self` is required to bootstrap an instance of an actor. Such accesses are considered to be a weaker form of isolation that relies on having exclusive access to the reference `self`. If `self` escapes the initializer, such uniqueness can no-longer be guaranteed without time-consuming analysis. Thus, the isolation of `self` decays (or changes) to `nonisolated` during any use of `self` that is not a direct stored-property access. That change happens once on a given control-flow path and persists through the end of the initializer. Here are some example uses of `self` within an initializer that cause it to decay to a `nonisolated` reference: + +1. Passing `self` as an argument in any procedure call. This includes: + - Invoking a method of `self`. + - Accessing a computed property of `self`, including ones using a property wrapper. + - Triggering an observed property (i.e., one with a `didSet` and/or `willSet`). +2. Capturing `self` in a closure (or autoclosure). +3. Storing `self` to memory. + +Consider the following example that helps demonstrate how this isolation decay works: + +```swift +class NotSendableString { /* ... */ } +class Address: Sendable { /* ... */ } +func greetCharlie(_ charlie: Charlie) {} + +actor Charlie { + var score: Int + let fixedNonSendable: NotSendableString + let fixedSendable: Address + var me: Self? = nil + + func incrementScore() { self.score += 1 } + nonisolated func nonisolatedMethod() {} + + init(_ initialScore: Int) { + self.score = initialScore + self.fixedNonSendable = NotSendableString("Charlie") + self.fixedSendable = NotSendableString("123 Main St.") + + if score > 50 { + nonisolatedMethod() // ✅ a nonisolated use of `self` + greetCharlie(self) // ✅ a nonisolated use of `self` + self.me = self // ✅ a nonisolated use of `self` + } else if score < 50 { + score = 50 } - await self.verify() + + assert(score >= 50) // ❌ error: cannot access mutable isolated storage after `nonisolated` use of `self` + + _ = self.fixedNonSendable // ❌ error: cannot access non-Sendable property after `nonisolated` use of `self` + _ = self.fixedSendable + + Task { await self.incrementScore() } // ✅ a nonisolated use of `self` } +} +``` + +The central piece of this example is the `if-else` statement chain, which introduces multiple control-flow paths in the initializer. In the body of one of the first conditional block, several different `nonisolated` uses of `self` appear. In the other conditional cases (the `else-if`'s block and the implicitly empty `else`), it is still OK for reads and writes of `score` to appear. But, once control-flow meets-up after the `if-else` statement at the `assert`, `self` is considered `nonisolated` because one of the blocks that can reach that point introduces non-isolation. + +As a consequence, the only stored properties that are accessible after `self` becomes `nonisolated` are let-bound properties whose type is `Sendable`. +The diagnostics emitted for illegal accesses to other stored properties will point to one of the earlier uses of `self` that caused the isolation to change. The sense of "earlier" here is in terms of control-flow and not in terms of where the statements appear in the program. To see how this can happen in practice, consider this alternative definition of `Charlie.init` that uses `defer`: - // Version 3: global-actor isolated inits can also be delegating. - @MainActor - convenience init(alt: Void) async { - self.init(with: ...) - await self.notifyAll() +```swift +init(hasADefer: Void) { + self.score = 0 + defer { + print(self.score) // ❌ error: cannot access mutable isolated storage after `nonisolated` use of `self` } + Task { await self.incrementScore() } // note: a nonisolated use of `self` +} +``` + +Here, we defer the printing of `self.score` until the end of the initializer. But, because `self` is captured in a closure before the `defer` is executed, that read of `self.score` is not always safe from data-races, so it is flagged as an error. Another scenario where an illegal property access can visually precede the decaying use is for loops: - init(bad1: Void) { - self.init() // ❌ error: only convenience initializers can delegate +```swift +init(hasALoop: Void) { + self.score = 0 + for i in 0..<10 { + self.score += i // error: cannot access mutable isolated storage after `nonisolated` use of `self` + greetCharlie(self) // note: a nonisolated use of `self` } +} +``` + +In this for-loop example, we must still flag the mutation of `self.score` in a loop as an error, because it is only safe on the first loop iteration. On subsequent loop iterations, it will not be safe because `self` may be concurrently accessed after being escaped in a procedure call. + +**Other Examples** + +Other than non-async inits, a global-actor isolated initializer or one that is marked with `nonisolated` will have a `nonisolated self`. Consider this example of such an initializer: + +```swift +func printStatus(_ s: Status) { /* ... */} - nonisolated init(bad2: Void) { - self.init() // ❌ error: only convenience initializers can delegate +actor Status { + var valid: Bool + + // an isolated method + func exchange(with new: Bool) { + let old = valid + valid = new + return old } - // warning: nonisolated on a synchronous non-delegating initializer is redundant - nonisolated init(bad3: Void) { - self.friends = [] - self.notifyAll() // ❌ disallowed by escaping-use restriction. + // an isolated method + func isValid() { return self.valid } + + // A `nonisolated self` initializer that calls isolated methods with `await`. + @MainActor init(_ val: Bool) async { + self.valid = val + + let old = await self.exchange(with: false) // note: a non-isolated use + assert(old == val) + + _ = self.valid // ❌ error: cannot access mutable isolated storage after non-isolated use of `self` + + let isValid = await self.isValid() // ✅ OK + + assert(isValid == false) } +} +``` + +Notice that calling an isolated method from an initializer with a `nonisolated self` is permitted, provided that you can `await` the call. That call is considered a nonisolated use, i.e., it's the first use of `self` other than to access a stored property. Afterwards, access to most stored properties within the `init` is lost, just like for the non-async case. Because this initializer is `async`, it could technically `await` to read the `Sendable` value of `self.valid`. But, we have chosen to forbid awaited access to stored properties in this situation. See the [discussion](#permitting-await-for-property-access-in-nonisolated-self-initializers) in the Alternatives Considerred section for more details. - nonisolated init(ok: Void) async { - self.friends = [] - self.notifyAll() // ❌ disallowed by escaping-use restriction. + +**In-depth discussions** + +The remainder of this subsection covers some technical details that are not required to understand this proposal and may be safely skipped. + +**Limitations of Static Analysis** +Not all loops iterate more than once, or even at all. The Swift compiler will be free to reject programs that may never exhibit a race dynamically, based on the static assumption that loops can iterate more than once and conditional blocks can be executed. To make this more concrete, consider these two silly loops: + +```swift +init(hasASillyLoop1: Void) { + self.score = 0 + while false { + self.score += i // error: cannot access isolated storage after `nonisolated` use of `self` + greetCharlie(self) // note: a nonisolated use of `self` } +} - func verify() { ... } - func notifyAll() { ... } +init(hasASillyLoop2: Void) { + self.score = 0 + repeat { + self.score += i // error: cannot access isolated storage after `nonisolated` use of `self` + greetCharlie(self) // note: a nonisolated use of `self` + } while false } ``` -An easy way to remember the rules around actor initializers is, if the initializer is just `async`, with no other actor isolation changes, then there is no escaping-use restriction. -Thus, if any one of the following apply to an initializer, it must obey the escaping-use restriction to maintain data-race safety for `self`: +In both loops above, it is clear to the programmer that no race will happen, because control-flow will not dynamically reach the statement incrementing `score` _after_ passing `self` in a procedure call. For these trivial examples, the compiler _may_ be able to prove that these loops do not execute more than once, but that is not guaranteed due to the [limitations of static analysis](https://en.wikipedia.org/wiki/Halting_problem). -1. not `async` -2. `nonisolated` -3. global-actor isolated +**Data-race Safety** -### Summary +In effect, the concept of isolation decay prevents data-races by disallowing access to stored properties once the compiler can no-longer prove that the reference to `self` will not be concurrently accessed. For efficiency reasons, the compiler might not perform interprocedural analysis to prove that passing `self` to another function is safe from concurrent access by another task. Interprocedural analysis is inherently limited due to the nature of modules in Swift (i.e., separate compilation). Immediately after `self` has escaped the initializer, the treatment of `self` in the initializer changes to match the unacquired status of the actor's executor. -The following table summarizes the capabilities and requirements of actor initializers in this proposal: +#### Global-actor isolated types -| Initializer Kind / Rules | Has escaping-use restriction | Delegation | -|---------------------------|-------------------------------|-------------| -| *Not* isolated to `self` | Yes | No | -| Isolated to `self` + synchronous | Yes | No | -| Isolated to `self` + `async` | No | No | -| `convenience` + anything | No | Yes (required) | +A non-isolated initializer of a global-actor isolated type (GAIT) is in the same situation as a non-async actor initializer, in that it must bootstrap the instance without the executor's protection. Thus, we can construct a data-race just like before: -## Source compatibility +```swift +@MainActor +class RequiresFlowIsolation + where T: Sendable, T: Equatable { -The following are known source compatibility breaks with this proposal: + var item: T -1. The escaping-use restriction. -2. `nonisolated` is ignored for `async` inits. -3. Global-actor isolation on stored properties of a nominal type. + func mutateItem() { /* ... */ } + + nonisolated init(with t: T) { + self.item = t + Task { await self.mutateItem() } + self.item = t // 💥 races with the task! + } +} +``` -**Breakage 1** +To solve this race, we propose to apply flow-sensitive actor isolation to the initializers of GAITs that are marked as non-isolated. -There is no simple way to automatically migrate applications that use `self` in an escaping manner within an actor initializer. -At its core, the simplest migration path is to mark the initializer `async`, but that would introduce `async` requirements on callers. For example, in this code: +For isolated initializers, GAITs have the ability to gain actor-isolation prior to calling the initializer itself. That's because its executor is a static instance, existing prior to even allocating uninitialized memory for a GAIT instance. Thus, all isolated initializers of a GAIT require callers to `await`, which will gain access to the right executor before starting initialization. That executor is held until the initializer returns. Thus for isolated initializers of GAITs, there is no danger of race among the isolated stored properties: ```swift -actor C { - init() { - self.f() // ❌ now rejected by this proposal - } +@MainActor +class ProtectedByExecutor { + var item: T - func f() { /* ... */} + func mutateItem() { /* ... */ } + + init(with t: T) { + self.item = t + Task { self.mutateItem() } // ✅ we're on the executor when creating this task. + assert(self.item == t) // ✅ always true, since we hold the executor here. + } } +``` + +GAITs that have `nonisolated` stored properties rely on Swift's existing `Sendable` restrictions to help prevent data races. + + +### Delegating Initializers + +This section defines the syntactic form and rules about delegating initializers for `actor` types and global-actor isolated types (GAITs). + +#### Syntactic Form + +While `actor`s are a reference type, their delegating initializers will follow the same basic rules that exist for value types, namely: + +1. If an initializer body contains a call to some `self.init`, then it's a delegating initializer. No `convenience` keyword is required. +2. For delegating initializers, `self.init` must always be called on all paths, before `self` can be used. -func user() { - let c = C() +The reason for this difference between `actor` and `class` types is that `actor`s do not support inheritance, so they can shed the complexity of `class` initializer delegation. GAITs use the same syntactic form as ordinary classes to define delegating initializers. + +#### Isolation + +Much like their non-delegating counterparts, an actor's delegating initializer either has an `isolated self` or a `nonisolated self` reference. The decision procedure for categorizing these initializers are exactly the same: non-async delegating initializers have a `nonisolated self`, *etc*. + +But, the delegating initializers of an actor have simpler rules about what can appear in their body, because they are not required to initialize the instance's stored properties. Thus, instead of using flow-sensitive actor isolation, delegating initializers have a uniform isolation for `self`, much like an ordinary function. + +### Sendability + +When passing values to any of an `actor`'s initializers, from outside of that actor, those values must be `Sendable`. +Thus, during the initialization of a new instance, the actor's "boundary" in terms of Sendability begins at the initial call-site to one of its initializers. +This rule forces programmers to correctly deal with `Sendable` values when creating a new actor instance. Fundamentally, programmers will have only two options for initializing a non-`Sendable` stored property of an actor: + +```swift +class NotSendableType { /* ... */ } +struct Piece: Sendable { /* ... */ } + +actor Greg { + var ns: NotSendableType + + // Option 1: an initializer that can be called from anywhere, + // because its arguments are Sendable. + init(fromPieces ps: (Piece, Piece)) { + self.ns = NotSendableType(ps) + } + + // Option 2: an initializer that can only be delegated to, + // because its arguments are not Sendable. + init(with ns: NotSendableType) { + self.init(fromPieces: ns.getPieces()) + } } ``` -we cannot introduce an `async` version of `init()`, whether it is delegating or not, because the `async` must be propagated to all callers, breaking the API. -Fortunately, Swift concurrency has only been available for a few months, as of September 2021. +As shown in the example above, you _can_ construct an actor that has a non-`Sendable` stored property. But, you should create a new instance of that type from `Sendable` pieces of data in order to store it in the actor instance. Once inside an actor's initializer, non-Sendable values can be freely passed when delegating to another initializer, or calling its methods, *etc*. The following example illustrates this rule: + +```swift +class NotSendableType { /* ... */ } +struct Piece: Sendable { /* ... */ } + +actor Gene { + var ns: NotSendableType + + init(_ ns: NotSendableType) { + self.ns = ns + } + + init(with ns: NotSendableType) async { + self.init(ns) // ✅ non-Sendable is OK during initializer delegation... + someMethod(ns) // ✅ and when calling a method from an initializer, etc. + } + + init(fromPieces ps: (Piece, Piece)) async { + let ns = NotSendableType(ps) + await self.init(with: ns) // ✅ non-Sendable is OK during initializer delegation + } + + func someMethod(_: NotSendableType) { /* ... */ } +} + +func someFunc() async { + let ns = NotSendableType() -To resolve this source incompatibility issue without too much code churn, it is proposed that the escaping-use restriction turns into an error in Swift 6 and later. For earlier versions that support concurrency, only a warning is emitted by the compiler. + _ = Gene(ns) // ❌ error: cannot pass non-Sendable value across actor boundary + _ = await Gene(with: ns) // ❌ error: cannot pass non-Sendable value across actor boundary + _ = await Gene(fromPieces: ns.getPieces()) // ✅ OK because (Piece, Piece) is Sendable -**Breakage 2** + _ = await SomeGAIT(isolated: ns) // ❌ error: cannot pass non-Sendable value across actor boundary + _ = await SomeGAIT(secondNonIso: ns) // ❌ error: cannot pass non-Sendable value across actor boundary +} +``` + +For a global-actor isolated type (GAIT), the same rule applies to its `nonisolated` initializers. Thus, upon entering such an initializer from outside of the actor, the values must be `Sendable`. The differences from an `actor` are that: + +1. The caller of the first initializer may already be isolated to the global-actor, so there is no `Sendable` barrier (as usual). +2. When delegating from a `nonisolated` initializer to one that is isolated to the global actor, the value must be `Sendable`. -In Swift 5.5, if a programmer requests that an `async` initializer be `nonisolated`, the escaping-use restriction is not applied, because isolation to `self` is applied regardless. For example, in this code: +The second difference only manifests when a `nonisolated` and `async` initializer delegates to an isolated initializer of the GAIT: ```swift -actor MyActor { - var x: Int +@MainActor +class SomeGAIT { + var ns: NotSendableType - nonisolated init(a: Int) async { - self.x = a - self.f() // permitted in Swift 5.5 - assert(self.x == a) // guaranteed to always be true + init(isolated ns: NotSendableType) { + self.ns = ns } - func f() { - // create a task to try racing with init(a:) - Task.detached { await self.mutate() } + nonisolated init(firstNonIso ns: NotSendableType) async { + await self.init(isolated: ns) // ❌ error: cannot pass non-Sendable value across actor boundary } - func mutate() { self.x += 1 } + nonisolated init(secondNonIso ns: NotSendableType) async { + await self.init(firstNonIso: ns) // ✅ + } } ``` -the `nonisolated` is simply ignored, and isolation is enforced with a hop-to-executor anyway. -Fixing this bug to match the proposal is very simple: remove the `nonisolated`. -Callers of the `init` will not be affected, since no synchronization is needed to enter the `init`, regardless of its isolation. -The compiler will be augmented with a fix-it in this scenario to make upgrading easy. +The barrier in the example above can be resolved by removing the `nonisolated` attribute, so that the initializer has a matching isolation. + +### Deinitializers + +In Swift 5.5, two different kinds of data races with an actor or global-actor isolated type (GAIT) can be created within a `deinit`, as shown in an earlier section. The first one involves a reference to `self` being shared with another task, and the second one with actors having shared executors. + +To solve the first kind of race, we propose having the same flow-sensitive actor isolation rules discussed earlier for a `nonisolated self` apply to an actor's `deinit`. A `deinit` falls under the `nonisolated self` category, because it is effectively a non-async, non-delegating initializer whose purpose is to clean-up or tear-down, instead of bootstrap. In particular, a `deinit` starts with a unique reference to `self`, so the rules for decaying to a `nonisolated self` match up perfectly. This solution will apply to the `deinit` of both actor types and GAITs. -**Breakage 3** -The removal of global-actor isolation on stored properties imposes some source incompatibility. -For structs and enums, removal of a now invalid global-actor isolation on a stored property -without a property initializer is not a source break, as it would only generate -warnings that an `await` is now unnecessary: +To solve the second race, we propose that a `deinit` can only access the stored properties of `self` that are `Sendable`. This means that, even when `self` is a unique reference and has not decayed to being `nonisolated`, only the `Sendable` stored properties of an actor or GAIT can be accessed. This restriction is not needed for an `init`, because the initializer has known call-sites that are checked for isolation and `Sendable` arguments. The lack of knowledge about when and where a `deinit` will be invoked is why `deinit`s must carry this extra burden. In effect, non-`Sendable` actor-isolated state can only be deinitialized by an actor by invoking that state's `deinit`. + +Here is an example to help illustrate the new rules for `deinit`: ```swift -struct S { - var counter: Int // suppose a fix-it removed @MainActor from this. +actor A { + let immutableSendable = SendableType() + var mutableSendable = SendableType() + let nonSendable = NonSendableType() + + init() { + _ = self.immutableSendable // ✅ ok + _ = self.mutableSendable // ✅ ok + _ = self.nonSendable // ✅ ok + + f(self) // trigger a decay to `nonisolated self` - func f() async { - _ = await self.counter // warning: no 'async' operations occur within 'await' expression + _ = self.immutableSendable // ✅ ok + _ = self.mutableSendable // ❌ error: must be immutable + _ = self.nonSendable // ❌ error: must be sendable + } + + + deinit { + _ = self.immutableSendable // ✅ ok + _ = self.mutableSendable // ✅ ok + _ = self.nonSendable // ❌ error: must be sendable + + f(self) // trigger a decay to `nonisolated self` + + _ = self.immutableSendable // ✅ ok + _ = self.mutableSendable // ❌ error: must be immutable + _ = self.nonSendable // ❌ error: must be sendable } } ``` -The behavior of the program changes only in a positive way: a superfluous synchronization is removed. -If the property's initializer requires global-actor isolation to evaluate, then the -programmer will need to move that expression into the type's initializer: +In the above, the only difference between the `init` and the `deinit` is that the `deinit` can only access `Sendable` properties, whereas the `init` can access non-`Sendable` properties prior to the isolation decay. + + +### Global-actor isolation and instance members + +**Note:** The isolation rules in this section for stored property initial values was never implemented because it was too onerous in existing code patterns that make use of `@MainActor`-isolated types. These rules have been subsumed by [SE-0411: Isolated default values](/proposals/0411-isolated-default-values.md). + +The main problem with global-actor isolation on the stored properties of a type is that, if the property is isolated to a global actor, then its default-value expression is also isolated to that actor. Since global-actor isolation can be applied independently to each stored property, an impossible isolation requirement can be constructed. The isolation needed for a type's non-delegating *and* non-async initializers would be the union of all isolation applied to its stored properties that have a default value. That's because a non-async initializer cannot hop to any executor, and a function cannot be isolated to two global actors. Currently, Swift 5.5 accepts programs with these impossible requirements. + +To fix this problem, we propose to remove any isolation applied to the default-value expressions of stored properties that are a member of a nominal type. Instead, those expressions will be treated by the type system as being `nonisolated`. If isolation is required to initialize those properties, then an `init` can always be defined and given the appropriate isolation. + +For global or static stored properties, the isolation of the default-value expression will continue to match the isolation applied to the property. This isolation is needed to support declarations such as: ```swift -@MainActor func getNumber() -> Int { 4 } +@MainActor +var x = 20 -struct S { - // 'await' operation cannot occur in a property initializer - var counter: Int /* = await getNumber() */ +@MainActor +var y = x + 2 +``` - init() async { - counter = await getNumber() // OK + +#### Removing Redundant Isolation + +Global-actor isolation on a stored property provides safe concurrent access to the storage occupied by that stored property in the type's instances. +For example, if `pid` is an actor-isolated stored property (i.e., one without an observer or property wrapper), then the access `p.pid.reset()` only protects the memory read of `pid` from `p`, and not the call to `reset` afterwards. Thus, for value types (enums and structs), global-actor isolation on those stored properties fundamentally serves no use: mutations of the storage occupied by the stored property in a value type are concurrency-safe by default, because mutable variables cannot be shared between tasks. For example, it is error when trying to capture a mutable var in a Sendable closure: + +```swift +@MainActor +struct StatTracker { + var count = 0 + + mutating func update() { + count += 1 } } + +var st = StatTracker() +Task { await st.update() } // error: mutation of captured var 'st' in concurrently-executing code ``` -This, combined with the rule change for classes, where the synchronization is not superfluous, means that some minor source fixes will be required. A warning about this change will be emitted in when the compiler is operating in Swift 5 mode, because it will become an error in Swift 6. +As a result, there is no way to concurrently mutate the memory of a struct, regardless of whether the stored properties of the struct are isolated to a global actor. Whether the instance can be shared only depends on whether it's var-bound or not, and the only kind of sharing permitted is via copying. Any mutations of reference types stored _within_ the struct require the usual actor-isolation applied to that reference type itself. In other words, applying global-actor isolation to a stored property containing a class type does _not_ protect the members of that class instance from concurrent access. +So, we propose to remove the requirement that access to those properties are protected by isolation. That is, accessing those stored properties do not require an `await`. -## Alternatives considered +The [global actors](0316-global-actors.md) proposal explicitly excludes actor types from having stored properties that are global-actor isolated. But in Swift 5.5, that is not enforced by the compiler. We feel that the rule should be enforced, i.e., the storage of an actor should uniformly be isolated to the actor instance. One benefit of this rule is that it reduces the possibility of [false sharing](https://en.wikipedia.org/wiki/False_sharing) among threads. Specifically, only one thread will have write access the memory occupied by an actor instance at any given time. -This section explains alternate approaches that were ultimately not chosen for this proposal. -### Deinitializers +## Source compatibility +There are some changes in this proposal that are backwards compatible or easy to migrate: + +- The set of `init` declarations accepted by the compiler in Swift 5.5 (without emitted warnings) is a strict subset of the ones that will be permitted if this proposal is accepted, i.e., flow-sensitive isolation broadens the set of permitted programs. +- Appearances of `convenience` on an actor's initializer can be ignored and/or have a fix-it emitted. +- Appearances of superfluous global-actor isolation annotations on ordinary stored properties (say, in value types) can be ignored and/or have a fix-it emitted. + +But, there are others which will cause a non-trivial source break to patch holes in the concurrency model of Swift 5.5, for example: + +- The set of `deinit`s accepted by the compiler for actors and GAITs will be narrowed. +- GAITs will have data-race protections applied to their non-isolated `init`s, which slightly narrows the set of acceptable `init` declarations. +- Global-actor isolation on stored-property members of an actor type are prohibited. +- Stored-property members that are still permitted to have actor isolation applied to them will have a `nonisolated` default-value expression. -One workaround for the lack of ability to synchronize with an actor's executor prior to destruction is to wrap the body of the `deinit` in a task. -If this task wrapping is done implicitly, then it breaks the expectation within Swift that all tasks are explicitly created by the programmer. -If the programmer decides to go the route of explicitly spawning a new task upon `deinit`, that decision is better left to the programmer. -It is important to keep in mind that it is undefined behavior in Swift for a reference to `self` to escape a `deinit`, such as through task creation. -Nevertheless, a program that does extend the lifetime of `self` in a `deinit` is not currently rejected by the compiler; and will not be if this proposal is accepted. +Note that these changes to GAITs will only apply to classes defined in Swift. Classes imported from Objective-C with MainActor-isolation applied will be assumed to not have data races. -### Flow-sensitive actor isolation -The solution in this proposal focuses on having an _explicit_ point at which an actor's `self` transitions to becoming fully-initialized, by leaning on delegating initializers. +## Alternatives considered + +This section explains alternate approaches that were ultimately not chosen for this proposal. + +### Introducing `nonisolation` after `self` is fully-initialized -If actor-isolation were formulated to change implicitly, after the point at which `self` becomes initialized in an actor, we could combine some of the capabilities of delegating and non-delegating inits. -In particular, accesses to stored properties in an initializer would be conditionally asynchronous, at multiple control-flow sensitive points: +It is tempting to say that, to avoid introducing another concept into the language, `nonisolation` should begin at the point where `self` becomes fully-initialized. But, because control-flow can cross from a scope where `self` is fully-initialized, to another scope where `self` _might_ be fully-initialized, this rule is not enough to determine whether an initializer has a race. Here are two examples of initializers where this simplistic rule breaks down: ```swift -actor A { +actor CounterExampleActor { var x: Int - var y: Int + + func mutate() { self.x += 1 } + + nonisolated func f() { + Task { await self.mutate() } + } - init(with z: Int) { - self.y = z - guard z > 0 else { - self.x = -1 - // `self` fully initialized here - print(self.x) // ❌ error: must 'await' access to 'x' - return + init(ex1 cond: Bool) { + if cond { + self.x = 0 + f() } - self.x = self.y - // `self` fully initialized here - _ = self.y // ❌ error: must await access to 'y' + self.x = 1 // if cond is true, this might race! + } + + init(ex2 max: Int) { + var i = 0 + repeat { + self.x = i // after first loop iteration, this might race! + f() + i += 1 + } while i < max } } ``` -This approach was not pursued for a two reasons. -First, it is likely to be confusing to users if the body of an initializer can change its isolation part-way through, at invisible points. -Second, the existing implementation of the compiler is not designed to handle conditional async-ness. -In order to translate the program from an AST to the SIL representation, we need to decide whether an expression is async. -But, the existing control-flow analysis, to determine where `self` becomes fully-initialized, must be run on the SIL representation of the program. -Performing control-flow analysis on an AST representation would be painful and become a maintenance burden. -SIL is a normalized representation that is specifically designed to support such analyses. +In Swift, `self` can be freely used, _immediately_ after becoming fully-initialized. Thus, if we tie `nonisolation` to whether `self` is fully-initialized _at each use_, both initializers above should be accepted, even though they permit data races: `f` can escape `self` into a task that mutates the actor, yet the initializer will continue after returning from `f` with unsynchronized access to its stored properties. + +With the flow-sensitive isolation rules in this proposal, both property accesses above that can race are rejected because of a flow-isolation error. The source of `nonisolation` would be identified as the calls to `f()`, so that programmers can correct their code. + +Now, consider what would happen if the calls to `f` above were removed. With the proposed isolation rules, the programs would now be accepted because they are safe: there is no source of `nonisolation`. If we had said that `nonisolation` _always_ starts immediately after `self` is fully-initialized, and _persists until the end of the initializer_, then even without the calls to `f`, the initializers above would be would be needlessly rejected. + + +### Permitting `await` for property access in `nonisolated self` initializers + +In an `nonisolated self` initializer, we reject stored property accesses after the first non-isolated use. For a non-async initializer, there is no alternative to rejecting the program, since one cannot hop to the actor's executor in that context. But an `async` initializer that is not isolated to `self` _could_ perform that hop: + +```swift +actor AwkwardActor { + var x: SomeClass + nonisolated func f() { /* ... */ } + + nonisolated init() async { + self.x = SomeClass() + let a = self.x + f() + let b = await self.x // SomeClass would need to be Sendable for this access. + print(a + b) + } +} +``` + +From an implementation perspective, it _is_ feasible to support the program above, where property accesses can become `async` expressions based on flow-sensitive isolation. But, this proposal takes the subjective position that such code should be rejected. + +The expressiveness gained by supporting such a flow-sensitive `async` property access is not worth the confusion they might create. For programmers who simply _read_ this valid code in a project, the `await` might look unnecessary and challenge their understanding of isolation applying to entire functions. But, this specific kind of `nonisolated self` _and_ `async` initializer would be the only place where one could demonstrate to _readers_ that isolation can change mid-function in valid Swift code. + +The ability to observe an isolation-change mid-function in _valid_ Swift code is the reason for rejecting the program above. This proposal says that, for a non-async and `nonisolated self` initializer, some property accesses are _rejected_ for violations of the same conceptual isolation-change. The valid formulation of those kinds of initializers have no observable isolation change, so casual readers notice nothing unusual. Only when modifying that code does the isolation-decay concept become relevant. But, isolation "decay" is just a tool used to explain the concept in this porposal. Programmers only need to keep in mind that accesses to stored properties are lost after you escape `self` in the initializer. + + +### Async Actor Deinitializers + +One idea for working around the inability to synchronize from a `deinit` with the actor or GAIT's executor prior to destruction is to wrap the body of the `deinit` in a task. This would effectively allow the non-async `deinit` to act as though it were `async` in its body. There is no other way to define an asynchronous `deinit`, since the callers of a deinit are never guaranteed to be in an asynchronous context. -### Removing the need for `convenience` +The primary danger here is that it is currently undefined behavior in Swift for a reference to `self` to escape a `deinit` and persist after the `deinit` has completed, which must be possible if the `deinit` were asynchronous. The only other option would be to have `deinit` be blocking, but Swift concurrency is designed to avoid blocking. -The removal of `convenience` to distinguish delegating initializers *will* create an ABI break. -Currently, the addition or removal of `convenience` on an actor initializer is an ABI-breaking change, as it is with classes, because the emitted symbols and/or name mangling will change. +### Requiring Sendable arguments only for delegating initializers -If we were to disallow `nonisolated`, non-delegating initializers, we could enforce the rule that `nonisolated` means that it must delegate. -But, such semantics would not align with global-actor isolation, which is conceptually the same as `nonisolated` with respect to an initializer: not being isolated to `self`. -In addition, any Swift 5.5 code with `nonisolated` or equivalent on an actor initializer would become ABI and source incompatible with Swift 6. +Delegating initializers are conceptually a good place to construct a fresh +instance of a non-Sendable value to pass-along to the actor during initialization. +This could only work by saying that only `nonisolated self` delegating initializers +can accept a non-Sendable value from any context. But also, an initializer's +delegation status must now be published in the interface of a type, i.e., +some annotation like `convenience` is required. Eliminating the need for +`convenience` was chosen over non-Sendable values for a specific kind of delegating +initializer for a few reasons: -Thus, is not ultimately worthwhile to try to eliminate `convenience`, since it does provide some benefit: marking initializers that _must_ delegate. -While a `nonisolated` synchronous initializer is mostly useless, the compiler can simple tell programmers to remove the `nonisolated`, because it is meaningless in that case. -Note that `nonisolated` _does_ provide utility for an `async` initializer, since it means that no implicit executor synchronization is performed, while allowing other `async` calls to happen within the initializer. +1. The rules for Sendable values and initializers would become complex, being dependent on three factors: delegation status, isolation of `self`, and the caller's context. The usual Sendable rules only depend on two. +2. Requiring `convenience` for just one narrow use-case is not worth it. +3. Static functions, using a factory pattern, can replace the need for initializers with non-Sendable arguments callable from anywhere. ## Effect on ABI stability @@ -546,8 +893,8 @@ This proposal does not affect ABI stability. ## Effect on API resilience -Any changes to the isolation of a declaration continues to be an [ABI-breaking change](0306-actors.md#effect-on-api-resilience), but a change in what is allowed in the _implementation_ of, say, a `nonisolated` member will not affect API resilience. +This proposal does not affect API resilience. ## Acknowledgments -Thank you to the members of the Swift Forums for their discussions about this topic, which helped shape this proposal. In particular, we would like to thank anyone who participated in [this thread](https://forums.swift.org/t/on-actor-initializers/49001). +Thank you to the members of the Swift Forums for their time spent reading this proposal and its prior versions and providing comments. diff --git a/proposals/0328-structural-opaque-result-types.md b/proposals/0328-structural-opaque-result-types.md index 1609d0820a..ab57b9a362 100644 --- a/proposals/0328-structural-opaque-result-types.md +++ b/proposals/0328-structural-opaque-result-types.md @@ -3,13 +3,14 @@ * Proposal: [SE-0328](0328-structural-opaque-result-types.md) * Authors: [Benjamin Driscoll](https://github.com/willtunnels), [Holly Borla](https://github.com/hborla) * Review Manager: [Ben Cohen](https://github.com/airspeedswift) -* Status: **Implemented (Swift 5.6)** -* Decision Notes: [Accepted with modifications](https://forums.swift.org/t/accepted-with-modifications-se-0328-structural-opaque-result-type/53789) -* Implementation: [apple/swift#38392](https://github.com/apple/swift/pull/38392), [apple/swift#40361](https://github.com/apple/swift/pull/40361) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-with-modifications-se-0328-structural-opaque-result-type/53789) +* Implementation: [apple/swift#38392](https://github.com/apple/swift/pull/38392) +* Toolchain: Any recent [nightly main snapshot](https://swift.org/download/#snapshots) ## Introduction -An [opaque result type](https://github.com/apple/swift-evolution/blob/main/proposals/0244-opaque-result-types.md) may be used as the result type of a function, the type of a variable, or the result type of a subscript. In all cases, the opaque result type must be the entire type. This proposal recommends lifting that restriction and allowing opaque result types in "structural" positions. +An [opaque result type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md) may be used as the result type of a function, the type of a variable, or the result type of a subscript. In all cases, the opaque result type must be the entire type. This proposal recommends lifting that restriction and allowing opaque result types in "structural" positions. Swift-evolution thread: [Structural opaque result types](https://forums.swift.org/t/structural-opaque-result-types/50998) @@ -47,7 +48,13 @@ The `some` keyword binds more loosely than `?` or `!`. An optional of an opaque ### Higher order functions -If the result type of a function, the type of a variable, or the result type of a subscript is a function type, that function type can contain arbitrary structural opaque result types. For example, `func f() -> () -> some P` and `func g() -> (some P) -> ()` are valid function definitions. +If the result type of a function, the type of a variable, or the result type of a subscript is a function type, that function type can only contain structural opaque types in return position. For example, `func f() -> () -> some P` is valid, and `func g() -> (some P) -> ()` produces an error: + +```swift +protocol P {} + +func g() -> (some P) -> () { ... } // error: 'some' cannot appear in parameter position in result type '(some P) -> ()' +``` ### Constraint inference @@ -85,7 +92,7 @@ func f(_ t: T) -> H { /* ... */ } This change is purely additive so has no source compatibility consequences. -As discussed in [SE-0244](https://github.com/apple/swift-evolution/blob/main/proposals/0244-opaque-result-types.md#source-compatibility): +As discussed in [SE-0244](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md#source-compatibility): > If opaque result types are retroactively adopted in a library, it would initially break source compatibility [...] but could provide longer-term benefits for both source and ABI stability because fewer details would be exposed to clients. There are some mitigations for source compatibility, e.g., a longer deprecation cycle for the types or overloading the old signature (that returns the named types) with the new signature (that returns an opaque result type). @@ -95,13 +102,13 @@ This change is purely additive so has no ABI stability consequences. ## Effect on API resilience -This change is purely additive so has no API resilience consequences. Adopting opaque types in structural positions in a resilient library has the same implications as top-level opaque result types. From [SE-0244](https://github.com/apple/swift-evolution/blob/main/proposals/0244-opaque-result-types.md#effect-on-api-resilience): +This change is purely additive so has no API resilience consequences. Adopting opaque types in structural positions in a resilient library has the same implications as top-level opaque result types. From [SE-0244](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md#effect-on-api-resilience): > Opaque result types are part of the result type of a function/type of a variable/element type of a subscript. The requirements that describe the opaque result type cannot change without breaking the API/ABI. However, the underlying concrete type can change from one version to the next without breaking ABI, because that type is not known to clients of the API. ## Rust's `impl Trait` -As discussed in [SE-0244](https://github.com/apple/swift-evolution/blob/main/proposals/0244-opaque-result-types.md#rusts-impl-trait), Swift's opaque result types were inspired by `impl Trait` in Rust, which is described in [RFC-1522](https://github.com/rust-lang/rfcs/blob/master/text/1522-conservative-impl-trait.md) and extended in [RFC-1951](https://github.com/rust-lang/rfcs/blob/master/text/1951-expand-impl-trait.md). +As discussed in [SE-0244](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md#rusts-impl-trait), Swift's opaque result types were inspired by `impl Trait` in Rust, which is described in [RFC-1522](https://github.com/rust-lang/rfcs/blob/master/text/1522-conservative-impl-trait.md) and extended in [RFC-1951](https://github.com/rust-lang/rfcs/blob/master/text/1951-expand-impl-trait.md). Though SE-0244 lists several differences between `some` and `impl Trait`, one difference it does not make explicit is that `impl Trait` is allowed in structural positions, in similar to the manner to that suggested by this proposal. One difference between this proposal and `impl Trait` is that `impl Trait` may not appear in the return type of closure traits or function pointers. @@ -117,13 +124,9 @@ Furthermore, since `P?` is never a correct constraint, it would be possible to ( ### Higher order functions -Consider the function `func f() -> (some P) -> ()`. The closure value produced by calling `f` has type `(some P) -> ()`, meaning it takes an opaque result type as an argument. That argument has some concrete type, `T`, determined by the body of the closure. Assuming no special structure on `P`, such as `ExpressibleByIntegerLiteral`, the user cannot call the closure. If they were able to, then they would be depending at the source level on the concrete type of `T` to remain fixed, which is one of the things opaque result types are designed to prevent. - -We could try to stop the user from defining a higher order function which returns a function any of whose arguments are opaque result types. However, is already possible to define uncallable functions in Swift, for instance a function taking an uninhabited protocol composition, and, in fact, the returned closure might be callable, something it is non-trivial to determine in general. - -Another reason one might want to disallow returning functions that take opaque result types as arguments is that top-level functions can never have opaque result types as arguments. +Consider the function `func f() -> (some P) -> ()`. If this were a valid structural opaque result type, the closure value produced by calling `f` has type `(some P) -> ()`, meaning it takes an opaque result type as an argument. That argument has some concrete type, `T`, determined by the body of the closure. Assuming no special structure on `P`, such as `ExpressibleByIntegerLiteral`, the user cannot call the closure. If they were able to, then they would be depending at the source level on the concrete type of `T` to remain fixed, which is one of the things opaque result types are designed to prevent. -This decision should be considered in the context of generalized `some` syntax, which we are likely to implement in the future. Which approach to higher order functions is more consistent with this generalized syntax is debatable. +Another reason to disallow returning functions that take opaque result types is that [SE-0341: Opaque Parameter Declarations](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md) proposes a different meaning for `some` in parameter position in function declarations, which would cause confusion if opaque parameter types mean something different within a function type. ### Constraint inference diff --git a/proposals/0329-clock-instant-date-duration.md b/proposals/0329-clock-instant-date-duration.md deleted file mode 100644 index 337960c396..0000000000 --- a/proposals/0329-clock-instant-date-duration.md +++ /dev/null @@ -1,526 +0,0 @@ -# Clock, Instant, Date, and Duration - -* Proposal: [SE-0329](0329-clock-instant-date-duration.md) -* Author(s): [Philippe Hausler](https://github.com/phausler) -* Review Manager: [John McCall](https://github.com/rjmccall) -* Status: **Returned for revision** -* Implementation: [apple/swift#39753](https://github.com/apple/swift/pull/39753) -* Review: ([first review](https://forums.swift.org/t/se-0329-clock-instant-date-and-duration/53309)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0329-clock-instant-date-and-duration/53635)) - -## Revision history -* **v1** Initial pre-pitch -* **v1.1** Refinement to clock, deadline and duration types - * Expanded to include a Deadline type - * Posed a Clock defined type protocol grouping instead of Duration based protocol grouping (Clock -> Deadline -> Duration rather than Duration + Clock) -* **v1.2** - * Removed the DurationProtocol concept to aide ease of use and simplify implementations - * Introduced WallClock.Duration as the lowered Date type -* **v1.3** - * Rename Deadline to Instant since that makes a bit more sense generally (especially for Date) - * Add the requirement of a referencePoint to ClockProtocol -* **v1.4** - * Move the concept of `now` to the protocol requirement for clocks as an instance method - * Move the `duration(from:to:)` to an instance method on `InstantProtocol` - * Add a number of really useful operators - * Concrete clock types now have `.now` on their `Instant` types - * Added an example ManualClock -* **v1.4.1** - * Clarify the concrete clock types to show their conformances -* **v1.4.2** - * Move the measurement function to clock itself to prevent conflicts with existing APIs -* **v1.4.3** - * Re-added hours and minutes construction - * added a base requirement for `ClockProtocol` to require a `minimumResolution` -* **v1.4.4** - * Rename `ClockProtocol` to `Clock` to better adhere to naming guidelines - * Adjusted measurement to have both a required method (plus default implementation) as well as a free floating function for standardized measurement. - -## Introduction - -The concepts of time can be broken down into three distinct parts: - -1. An item to provide a concept of now plus a way to wake up after a given point in time -2. A concept of a point in time -3. A concept of a measurement in time. - -These three items are respectively a **clock**, an **instant** and a **duration**. The measurement of time can be used for many types of APIs, all the way from the high levels of a concept of a timeout on a network connection, to the amount of time to sleep a task. Currently, the APIs that take measurement of time types take `NSTimeInterval` (aka `TimeInterval`), `DispatchTimeInterval`, and even types like `timespec`. - -## Motivation - -To define a standard way of interacting with time, we need to ensure that in the cases where it is important to limit clock measurement to a specific concept, that ability is preserved. For example, if an API can only accept realtime deadlines as instants, that API cannot be passed to a monotonic instant. This specificity needs to be balanced with the ergonomics of being able to use high-level APIs with little encumbrance of needing to know exactly the time type that is needed; in UI, it might not be welcoming to new developers learning Swift to force them to understand the differential between the myriad of clock concepts available for the operating system. Likewise, any implementation must be robust and performant enough to support multiple operating system back ends (Linux, Darwin, Windows, etc.), but also be easy enough to get right for the common use cases. Practically speaking, durations should be a progressive disclosure to instants and clocks. - -From a performance standpoint, a distinct requirement is that any duration type (or clock type) must be reasonably performant enough to do tasks like measuring the execution performance of a function, without incurring a large overhead to the execution of the measurement. This means that any type that is expressing a duration should be small, and likely backed by some sort of (or group of) PoD type(s). - -Time itself is always measured in a manner that is in reference to a certain frame of analysis. For example, uptime is measured in relative perspective to how long the machine has been booted, whereas wall clock measurements are sourced from a network transaction to update time as a reference to coordinated universal time (UTC). Any instants expressed in terms of boot time versus UTC wall clock time can only be converted in a potentially lossy manner. Wall clock times can always be safely transmitted from one machine to another since the frame of reference is shared, whereas boot time on the other hand is meaningless when transmitted from two machines, but quite meaningful when transmitted from process to process on the same machine. - -As it stands today, there are a number of APIs and types to represent clocks, instants, and durations. Foundation, for example, defines instant as `Date`, which is constructed from a wall clock reference point, and `TimeInterval` which is defined as a `Double` representing the number of seconds between two points in time. Dispatch defines `DispatchTime`, `DispatchWallTime`, and `DispatchTimeInterval`; these, respectively, work in relation to a reference of uptime, a wall clock time, and a value of seconds/milliseconds/microseconds/nanoseconds. These obviously are not the only definitions, but when dealing with concurrency, a uniform accessor to all of these concepts is helpful to build the primitives needed for sleep and other temporal concepts. - -## Definitions - -Time is relative, temporal types doubly so. In this document, there will be some discussion with regards to the categorization of temporal types that readers should be distinctly aware of. - -**Absolute Time:** Time that always increments, but suspends while the machine is asleep. The reference point at which this starts is relative to the boot time of the machine so no-two machines would be expected to have the same uptime values. - -**Calendar:** A human locale based system in which to measure time. - -**Clock:** The mechanism in which to measure time, and understand how that time flows. - -**Continuous Time:** Time that always increments but does not stop incrementing while the system is asleep. This is useful to consider as a stopwatch style time; the reference point at which this starts and are most definitely different per machine. - -**Date:** A Date value encapsulates a single point in time, independent of any particular calendrical system or time zone. Date values represent a time interval relative to an absolute reference date. - -**Deadline:** In common parlance, it is a limit defined as an instant in time: a narrow field of time by which an objective must be accomplished. - -**Duration:** A measurement of how much time has elapsed between two deadlines or reference points. - -**Instant:** A precise moment in time. - -**Monotonic Time:** Darwin and BSD define this as continuous time. Linux, however, defines this as a time that always increments, but does stop incrementing while the system is asleep. - -**Network Update Time:** A value of wall clock time that is transmitted via ntp; used to synchronize the wall clocks of machines connected to a network. - -**Temporal:** Related to the concept of time. - -**Time Zone:** An arbitrary political defined system in which to normalize time in a quasi-geospatial delineation intended to keep the apex of the solar day around 12:00. - -**Uptime:** Darwin and BSD define this as absolute time. Linux, however, defines this as time that does not suspend while asleep but is relative to the boot. - -**Wall Clock Time:** Time like reading from a clock. This may be adjusted forwards or backwards for numerous reasons; in this context, it is time that is not specific to a time zone or locale, but measured from an absolute reference date. Network updates may adjust the drift on the clock either backwards or forwards depending on the relativistic drift, clock skew from inaccuracies with the processor, or from hardware power characteristics. - -Since there are platform differences in the definition of monotonic time and uptime, for the rest of this proposal it will be in terms of the definition on Darwin and BSD that are referencing monotonic and uptime. - -## Detailed Design - -### Prior Art - -There are a number of cases where these types end up being conflated with calendrical math. It is reasonable to say that the requirements for calendrical math have a distinct requirement of understanding of locales and time zones, and are clearly out of scope of any duration or clock types that might be introduced. That is distinct responsibilities of `Calendar` and `DateComponents`. - -#### Go -https://pkg.go.dev/time -https://golang.org/src/time/time.go - -Go stores time as a structure of a wall clock reference point (uint64), an 'ext' additional nanoseconds field (int64), and a location (pointer). -Go stores duration as an alias to int64 (nanoseconds). - -There is no control over the reference points in Go to specify a given clock; either monotonic or wall clock. The base implementation attempts to encapsulate both monotonic and wall clock values together in Go. For common use case this likely has little to no impact, however it lacks the specificity needed to identify a progressive disclosure of use. - -#### Rust -https://doc.rust-lang.org/stable/std/time/struct.Duration.html - -Rust stores duration as a u64 seconds and a u32 nanoseconds. -The measurement of time in Rust uses Instant, which seems to use a monotonic clock for most platforms. - -#### Kotlin -https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.time/-duration/ - -Kotlin stores Duration as a Long plus a unit discriminator comprised of either milliseconds or nanoseconds. Kotlin's measurement functions do not return duration (yet?) but instead rely on conversion functions from Long values in milliseconds etc and those currently measurement functions use system uptime to determine reference points. - -#### Swift -So given all of that, Swift can take this to another level of both accuracy of intent and ease of use beyond any of the other examples given. Following in the themes of other Swift APIs, we can embrace the concept of progressive disclosure and leverage the existing frameworks that define time concepts. - -The given requirements are that we must have a way of expressing the frame of reference of time. This needs to be able to express a concept of now, and a concept of waking up after a given instant has passed. Instants must be able to be compared among each other but are specific to the clock they were obtained. Instants also must be able to be advanced by a given duration or a distance between two instants must be able to emit a duration. Durations must be comparable and also must have some intrinsic unit of time that can suffice for broad application. - -It is worth noting that any extensions to Foundation, Dispatch, or other frameworks beyond the Swift standard library and concurrency library are not within the scope of this proposal and are under the prevue of those teams. This may or may not include additional Clock adoptions, additional functions that take the new types and changes in deprecations. Any examples here are listed as illustrations of potential use cases and not to be considered as part of this proposal. - -##### Clock - -The base protocol for defining a clock requires two primitives; a way to wake up after a given instant, and a way to produce a concept of now. Clocks can also be defined in terms of a potential resolution of access; some clocks may offer resolution at the nanosecond scale, other clocks may offer only microsecond scale. Any values of elapsed time may be considered to be 0 if they are below the minimum resolution. - -```swift -public protocol Clock: Sendable { - associatedtype Instant: InstantProtocol - - var now: Instant { get } - - func sleep(until deadline: Instant) async throws - - var minimumResolution: Duration { get } - - func measure(_ work: () async throws -> Void) reasync rethrows -> Duration -} -``` - -This means that given an instant, it is intrinsically linked to the clock; e.g., a monotonic instant is not meaningfully comparable to a wall clock instant. However, as an ease of use concession, the durations between two instants can be compared. However, doing this across clocks is considered a programmer error, unless handled very carefully. By making the protocol hierarchy just clocks and instants, it means that we can easily express a compact form of a duration that is usable in all cases; particularly for APIs that might adopt Duration as a replacement to an existing type. - -The clock minimum resolution will have a default implementation that returns `.nanosecond(1)`. - -Clocks can then be used to measure a given amount of work. This means that clock should have the extensions to allow for the affordance of measuring workloads for metrics, but also measure them for performance benchmarks. In addition to the per clock definitions of measuring a base measurement function using the monotonic clock will also be added. - -```swift -public func measure(_ work: () async throws -> Void) reasync rethrows -> Duration -``` - -This means that making benchmarks is quite easy to do: - -```swift -let elapsed = measure { - someWorkToBenchmark() -} -``` - -For example, we can adapt existing DispatchQueue API to take an instant as a deadline given a specific clock, or allow for generalized clocks. This allows for fine grained execution with exactly how the developer intends to have it work. - -```swift -extension DispatchQueue { - func asyncAfter(deadline: UptimeClock.Instant, qos: DispatchQoS = .unspecified, flags: DispatchWorkItemFlags = [], execute work: @escaping () -> Void) - func asyncAfter(deadline: C.Instant, clock: C, qos: DispatchQoS = .unspecified, flags: DispatchWorkItemFlags = [], execute work: @escaping () -> Void) -} -``` - -With additions as such, developers can interact similarly to the existing API set, but utilize the new generalized clock concepts. This allows for future expansion of clock concepts by the teams in which it is meaningful without needing to plumb through concepts into Dispatch's implementation. - -```swift -DispatchQueue.main.asyncAfter(deadline: .now.advanced(by: .seconds(3)) { - doSomethingAfterThreeSecondsOfUptime() -} -DispatchQueue.main.asyncAfter(deadline: .now.advanced(by: .seconds(3), clock: .wall) { - doSomethingAfterThreeSecondsOfWallClock() -} -``` - -By providing the clock type, developers are empowered to make better choices for exactly the concept of time they want to utilize, but also allowed progressive disclosure to powerful tools to express that time. - -##### Instant - -As previously stated, instants need to be compared, and might be stored as a key, but only need to define a concept of now, and a way to advance them given a duration. By utilizing a protocol to define an instant, it provides a mechanism in which to use the right storage for the type, but also be type safe with regards to the clock they are intended for. The primary reasoning that instants are useful is that they can be composed. - -Given a function with a deadline as an instant, if it calls another function that takes a deadline as an instant, the original can just be passed without mutation to the next function. That means that the instant in which that deadline elapses does not have interference with the pre-existing calls or execution time in-between functions. One common example of this is the timeout associated with url requests; a timeout does not fully encapsulate how the execution deadline occurs; there is a deadline to meet for the connection to be established, data to be sent, and a response to be received; a timeout spanning all of those must then have measurement to account for each step, whereas a deadline is static throughout. - - -```swift -public protocol InstantProtocol: Comparable, Hashable, Sendable { - func advanced(by duration: Duration) -> Self - func duration(to other: Self) -> Duration -} - -extension InstantProtocol { - public static func + (_ lhs: Self, _ rhs: Duration) -> Self - public static func - (_ lhs: Self, _ rhs: Duration) -> Self - - public static func += (_ lhs: inout Self, _ rhs: Duration) - public static func -= (_ lhs: inout Self, _ rhs: Duration) - - public static func - (_ lhs: Self, _ rhs: Self) -> Duration -} -``` - -`InstantProtocol` in addition to the `advance(by:)` and `duration(to:)` methods also has operators to add and subtract durations. However, it does not adhere to `AdditiveArithemtic` since that requires same type addition as well as a "zero"; of which neither make sense generally for defining instants. - -This can be used to adapt existing behaviors like `URLRequest` timeout. Which then becomes more composable with other instant concepts than the existing timeout APIs. - -```swift -extension URLRequest { - public init(url: URL, cachePolicy: CachePolicy = .useProtocolCachePolicy, deadline: MonotonicClock.Instant) -} -``` - -This will be expanded upon further, but `RunLoop` will be modified to now take a type that is an `InstantProtocol` conforming type. - -```swift -RunLoop.main.run(until: .now.advanced(by: .seconds(3))) -``` - -##### Duration - -It is reasonable to consider that each clock's instant has it's own "unit" of time measurement. However, that complicates the adoption story and proliferates a practically identical type to solely prevent one potential minor mistake of comparing the duration from the difference of instants from two different clocks. Duration itself should be trivial to express, non-lossy storage, which avoids mathematical ambiguity. On one end of the spectrum is to make isolate monotonic durations different from wall clock durations, on the other is say everything is just a Double. Both have advantages, but both have distinct disadvantages. Making duration a structure that is trivial allows a happy middle ground, but also allows for the potential of incremental adoption. - -Meaningful durations can always be expressed in terms of nanoseconds plus a number of seconds, either a duration before a reference point or after. They can be constructed from meaningful human measured (or machine measured precision) but should not account for any calendrical calculations (e.g., a measure of days, months or years distinctly need a calendar to be meaningful). Durations should able to be serialized, compared, and stored as keys, but also should be able to be added and subtracted (and zero is meaningful). They are distinctly NOT `Numeric` due to the aforementioned issue with regards to multiplying two `TimeInterval` variables. That being said, there is utility for ad-hoc division and multiplication to calculate back-offs. - -The `Duration` must be able to account for high scale resolution of calculation; the storage will under the hood ensure proper rounding for division (by likely storing higher precision than exposed) and enough range to span the full range of potential reasonable instants. This means that spanning the full range of +/- thousands of years at a non lossy scale can be accomplished by storing the seconds and nanoseconds. - -```swift -public struct Duration: Sendable { - public var seconds: Int64 { get } - public var nanoseconds: Int64 { get } -} - -extension Duration { - public static func hours(_ seconds: T) -> Duration - public static func hours(_ seconds: Double) -> Duration - public static func minutes(_ seconds: T) -> Duration - public static func minutes(_ seconds: Double) -> Duration - public static func seconds(_ seconds: T) -> Duration - public static func seconds(_ seconds: Double) -> Duration - public static func milliseconds(_ milliseconds: T) -> Duration - public static func milliseconds(_ milliseconds: Double) -> Duration - public static func microseconds(_ microseconds: T) -> Duration - public static func microseconds(_ microseconds: Double) -> Duration - public static func nanoseconds(_ value: T) -> Duration -} - -extension Duration: Codable { } -extension Duration: Hashable { } -extension Duration: Equatable { } -extension Duration: Comparable { } -extension Duration: AdditiveArithmetic { } - -extension Duration { - public static func / (_ lhs: Duration, _ rhs: Double) -> Duration - public static func /= (_ lhs: inout Duration, _ rhs: Double) - public static func / (_ lhs: Duration, _ rhs: T) -> Duration - public static func /= (_ lhs: inout Duration, _ rhs: T) - public static func / (_ lhs: Duration, _ rhs: Duration) -> Double - public static func * (_ lhs: Duration, _ rhs: Double) -> Duration - public static func *= (_ lhs: inout Duration, _ rhs: Double) - public static func * (_ lhs: Duration, _ rhs: T) -> Duration - public static func *= (_ lhs: inout Duration, _ rhs: T) -} -``` - -##### Date - -When speaking of temporal types, `Date` has served a distinct and special place in the core of Swift in some really prominent places. A `Date` value encapsulates a single point in time, independent of any particular calendrical system or time zone. `Date` values represent a time interval relative to an absolute reference date. It could easily be considered the canonical representation of a wall clock reference point and is quite suited as a concept to be used as a deadline for wall clock based calculations. In short, as part of this proposal, we intend to give `Date` a new home and move it from Foundation to the standard library. Now this will not include all of the API associated with `Date`, but instead a distinct subset of the API surface area about `Date` that is relevant to representing wall clock time reference points. - -```swift -@available(macOS 10.9, iOS 7.0, tvOS 9.0, watchOS 2.0, macCatalyst 13.0, *) -@_originallyDefinedIn(module: "Foundation", macOS /*TBD*/, iOS /*TBD*/, tvOS /*TBD*/, watchOS /*TBD*/, macCatalyst /*TBD*/) -public struct Date { - public init(converting monotonicInstant: MonotonicClock.Instant) - public init(converting uptimeInstant: UptimeClock.Instant) - - @available(macOS 12, iOS 15, tvOS 15, watchOS 8, *) - public static var now : Date { get } -} - -extension Date: InstantProtocol { - public func advanced(by duration: Duration) -> Date - public func duration(to other: Date) -> Duration -} - -extension Date: Codable { } -extension Date: Hashable { } -extension Date: Equatable { } -``` - -As a _potential_ implementation detail; `Date` currently stores its value as a `Double` of seconds from Jan 1 2001 UTC. This causes floating point drift when the value is further out from that point in time, since we are taking the leap to move `Date` down the stack from Foundation to the standard library this seems like perfect opportunity to address this issue with a more robust storage solution. Instead of storing as a 64 bit `Double` value, it will now be stored as a `Int64` for seconds, and a `UInt32` for nanoseconds normalized where the nanoseconds storage will be no more than 1,000,000,000 nanoseconds (which is 29 bits) and a full range of seconds. This means that the storage size of `Date` will increase from 64 bits to 96 bits, with the benefit that the range of expressible dates will be +/-9,223,372,036,854,775,807.999999999 seconds around Jan 1 1970; which is full nanosecond resolution of a range of 585 billion years +/- a few months worth of leap year days and such - we feel that this range is suitable for any software and can be revisited in a few hundred billion years when it becomes an issue. - -To give clarity on the real world impact of changing the storage size of `Date`; Xcode (it was a handy target for me to test) in a reasonably real world scenario created over 10,000 `NSDate` objects and around 3,000 of which were still resident at a quiescence point. Xcode reflects a decently large scale application and the translation from `NSDate` to `Date` does not 100% apply here but it gives a metric for what type of impact that might have in an extreme case; approximately 12kB more memory usage - comparatively to the total memory used this seems quite small, so the system impact should be relatively negligible. - -Readers may have noticed that `Date` remains `Codable` at the standard library layer but gains a new storage mechanism. The coding format will remain the same. Since that represents a serialization mechanism that is written to disk and is therefore permanent for document formats. We do not intend for `Date` to break existing document formats and all current serialization will both emit and decode as it would for double values relative to Jan 1 2001 UTC as well as the `DateEncodingStrategy` for JSONSerialization. This does mean that when encoding and decoding `Date` values it may loose small portions of precision, however this is acceptable losses since any format stored as such inherently takes some amount of time to either transmit or write to disk; any sub-second (near nanosecond) precision that may be lost will be vastly out weighed from the write and read times. - -The storage change is not a hard requirement; and may be a point in which we might decide is not worth taking. - -All remaining APIs on Date will exist still at the Foundation layer for compatibility with existing software. - -To be clear, we are not suggesting that Calendar, Locale, or TimeZone be moved down; those transitions are distinctly out of scope of this proposal and are not a goal. - -##### WallClock - -Wall clocks are useful since they represent a transmittable form of time. Instants can be serialized and sent from one machine to another and the values are meaningful in a foreign context. That transmission can be immediately useful when dealing with concepts like distributed actors; where an actor may be hosted on a remote machine and a deadline for work is sent across from one domain to another. The `WallClock` type will use `Date` as its `Instant` type and provide an extension to access the clock instance as the inferred base type property. - -```swift -public struct WallClock { - public init() - - public static var now: Date { get } -} - -extension WallClock: Clock { - public typealias Instant = Date - - public var now: Date { get } - public func sleep(until deadline: Date) async throws -} - -extension Clock where Self == WallClock { - public static var wall: WallClock { get } -} -``` - -##### MonotonicClock - -When instants are for local processing only and need to be high resolution without the encumbrance of suspension while the machine is asleep `MonotonicClock` is the tool for the job. The `MonotonicClock.Instant` type can be initialized with a wall clock instant if that value can be expressed in terms of a relative point to now; knowing the delta between the current time and the specified wall clock instant a conversion to the current monotonic reference point can be made such that conversion (if possible) represents what the value would be in terms of the monotonic clock. Much like the wall clock version the monotonic clock also offers an extension to access the clock instance as the inferred base type property. - -```swift -public struct MonotonicClock { - public init() - - public static var now: Instant { get } -} - -extension MonotonicClock: Clock { - public struct Instant { - public init?(converting wallClockInstant: WallClock.Instant) - - public static var now: MonotonicClock.Instant { get } - } - - public var now: Instant { get } - public func sleep(until deadline: Instant) async throws -} - -extension MonotonicClock.Instant: InstantProtocol { - func advanced(by duration: Duration) -> MonotonicClock.Instant - func duration(to other: MonotonicClock.Instant) -> Duration -} - -extension Clock where Self == MonotonicClock { - public static var monotonic: MonotonicClock { get } -} -``` - -##### UptimeClock - -Where local process scoped or cross machine scoped instants are not suitable uptime serves the purpose of a clock that does not increment while the machine is asleep but is a time that is referenced to the boot time of the machine, this allows for the affordance of cross process communication in the scope of that machine. Similar to the other clocks there is an extension to access the clock instance as the inferred base type property. - -```swift -public struct UptimeClock: Clock { - public init() - - public static var now: Instant { get } -} - -extension UptimeClock: Clock { - public struct Instant { - public init?(converting wallClockInstant: WallClock.Instant) - public static var now: UptimeClock.Instant { get } - } - - public var now: Instant { get } - public func sleep(until deadline: Instant) async throws -} - -extension UptimeClock.Instant: InstantProtocol { - func advanced(by duration: Duration) -> UptimeClock.Instant - func duration(to other: UptimeClock.Instant) -> Duration -} - -extension Clock where Self == UptimeClock { - public static var uptime: UptimeClock { get } -} -``` - -##### Example Custom Clock - -One example for adopting `Clock` is a manual clock. This could be a useful item for testing (but not currently part of this proposal as an API to add). It allows for the manual advancement of time in a deterministic manner. The general intent is to allow the manual clock type to be advanced from one thread and the sleep function can then be used to act as if it was a standard clock in generic APIs. - -```swift -public final class ManualClock: Clock, @unchecked Sendable { - public struct Instant: InstantProtocol { - var offset: Duration = .zero - - public func advanced(by duration: Duration) -> ManualClock.Instant { - Instant(offset: offset + duration) - } - - public func duration(to other: ManualClock.Instant) -> Duration { - other.offset - offset - } - - public static func < (_ lhs: ManualClock.Instant, _ rhs: ManualClock.Instant) -> Bool { - lhs.offset < rhs.offset - } - } - - struct WakeUp { - var when: Instant - var continuation: UnsafeContinuation - } - - public private(set) var now = Instant() - - // General storage for the sleep points we want to wake-up for - // this could be optimized to be a more efficient data structure - // as well as enforced for generation stability for ordering - var wakeUps = [WakeUp]() - - // adjusting now or the wake-ups can be done from different threads/tasks - // so they need to be treated as critical mutations - let lock = os_unfair_lock_t.allocate(capacity: 1) - - deinit { - lock.deallocate() - } - - public func sleep(until deadline: Instant) async throws { - // Enqueue a pending wake-up into the list such that when - return await withUnsafeContinuation { - if deadline <= now { - $0.resume() - } else { - os_unfair_lock_lock(lock) - wakeUps.append(WakeUp(when: deadline, continuation: $0)) - os_unfair_lock_unlock(lock) - } - } - } - - public func advance(by amount: Duration) { - // step the now forward and gather all of the pending - // wake-ups that are in need of execution - os_unfair_lock_lock(lock) - now += amount - var toService = [WakeUp]() - for index in (0..<(wakeUps.count)).reversed() { - let wakeUp = wakeUps[index] - if wakeUp.when <= now { - toService.insert(wakeUp, at: 0) - wakeUps.remove(at: index) - } - } - os_unfair_lock_unlock(lock) - - // make sure to service them outside of the lock - toService.sort { lhs, rhs -> Bool in - lhs.when < rhs.when - } - for item in toService { - item.continuation.resume() - } - } -} -``` - -## Impact on Existing Code - -### Existing APIs - -Task will have a more distinct `sleep` function where a clock can be specified. - -```swift -extension Task where Success == Never, Failure == Never { - public static func sleep(until deadline: C.Instant, clock: C) async throws -} -``` - -Or, in the case where an ease of use is preferred over a raw nanoseconds; we will add a connivence API exposing a monotonic duration to sleep for. - -```swift -extension Task where Success == Never, Failure == Never { - public static func sleep(for duration: Duration) async throws -} -``` - -The `DispatchQueue` implementation can support three types of fundamental clock types; monotonic, wall, and uptime. This might be able to be expressed as overloads to the instant types and avoid ambiguity by specifying a clock. - -```swift -extension DispatchQueue { - public func asyncAfter(deadline: MonotonicClock.Instant, qos: DispatchQoS = .unspecified, flags: DispatchWorkItemFlags = [], execute work: @escaping @convention(block) () -> Void) - public func asyncAfter(deadline: WallClock.Instant, qos: DispatchQoS = .unspecified, flags: DispatchWorkItemFlags = [], execute work: @escaping @convention(block) () -> Void) - public func asyncAfter(deadline: UptimeClock.Instant, qos: DispatchQoS = .unspecified, flags: DispatchWorkItemFlags = [], execute work: @escaping @convention(block) () -> Void) -} -``` - -### Existing Application Code - -This proposal is purely additive and has no direct impact to existing application code. - -## Impact on ABI - -The proposed implementation will introduce two runtime functions; a way of obtaining time and a way of sleeping given a standard clock. Date when lowered will not immediately be frozen, however as a future direction we plan on making the type frozen for performance reasons; (it is mentioned in the evolution manifesto)[https://github.com/apple/swift/blob/main/docs/LibraryEvolutionManifesto.md#limitations] that a concept of a frozen as of a certain version will be a future direction available to evolution. We feel that this is an appropriate use case and should be revisited when that is available. - -## Alternatives Considered - -It has been considered to move Date down into the standard library to encompass a wall + monotonic concept like Go, but this was not viewed as extensible enough to capture all potential clock sources. Additionally this approach breaks the concept of comparability which is a key requirement for any `InstantProtocol`. - -It has been considered to leave the Duration type to be a structure and shared among all clocks. This exposes the potential error in which two durations could be interchanged that are measuring two different things. From an opinionated type system perspective a `MonotonicClock.Duration` measures monotonic seconds and a `WallClock.Duration` measures wall clock seconds which are two different unit systems. This point is debatable and can be changed with the caveat that developers may write inappropriate code. - -It has been considered to attempt to make Duration into a protocol form to restrict the concepts of measurement to only be compared in the clock scope they were defined by but that proves to be quite cumbersome for implementations and dramatically reduces the ease of use for APIs that might want to use interval types. - -A concrete type expressing Deadlines could be introduced however adding that defeats the progressive disclosure of the existing types and poses a compatibility problem with existing APIs. Effectively it would make functions that currently take Date instead need to take `Deadline` which seems anti-thematic to tight integration with existing APIs. - -It has been considered that Date should account for leap seconds, however after much consideration this is viewed as perhaps too disruptive to the storage and serialization of Date and that it is better suited to leverage calendrical calculations such as those in Foundation to account for leap seconds since that falls in line more so with DateComponents than Date. - -As an alternative to moving Date to the standard library we have considered other strategies such as introducing a new type with a name that connotes better the concept of a date being defined as a point in time devoid of a calendar. However this ends up having a multi-level fall out. The large swath of APIs that are both in existing Swift exports of frameworks and SDKs that exist today have a use cases that use the same definition - that means that introducing a new type that serves the same purpose then becomes a quandary of "which type should I use". As a point of reference the SDK for macOS has approximately 1000 occurrences of the type NSDate which is bridged to Date, we feel that changing this would lead to needless churn for developers for no clear advantage especially since we can reform the storage to be more accurate. This is a topic that has been debated at length and we feel that the overall benefits for the reduction of churn at the magnitude of impact out weigh the marginal gains of a different name that may carry the same level of nomenclature ambiguity. - -It was considered to make Foundation.TimeInterval to be implicitly converted to Duration; however it was determined that this is better suited as a potential future direction for importer adjustments instead since that type would potentially cause systemic problems with overloads based on the aliasing behavior of TimeInterval. diff --git a/proposals/0329-clock-instant-duration.md b/proposals/0329-clock-instant-duration.md new file mode 100644 index 0000000000..743e21a61d --- /dev/null +++ b/proposals/0329-clock-instant-duration.md @@ -0,0 +1,564 @@ +# Clock, Instant, and Duration + +* Proposal: [SE-0329](0329-clock-instant-duration.md) +* Author: [Philippe Hausler](https://github.com/phausler) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#40609](https://github.com/apple/swift/pull/40609) +* Review: ([first review](https://forums.swift.org/t/se-0329-clock-instant-date-and-duration/53309)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0329-clock-instant-date-and-duration/53635)) ([second review](https://forums.swift.org/t/se-0329-second-review-clock-instant-and-duration/54509)) ([third review](https://forums.swift.org/t/se-0329-third-review-clock-instant-and-duration/54727)) ([acceptance](https://forums.swift.org/t/accepted-se-0329-clock-instant-and-duration/55324)) + +
+Revision history + +* **v1** Initial pre-pitch +* **v1.1** Refinement to clock, deadline and duration types + * Expanded to include a Deadline type + * Posed a Clock defined type protocol grouping instead of Duration based protocol grouping (Clock -> Deadline -> Duration rather than Duration + Clock) +* **v1.2** + * Removed the DurationProtocol concept to aide ease of use and simplify implementations + * Introduced WallClock.Duration as the lowered Date type +* **v1.3** + * Rename Deadline to Instant since that makes a bit more sense generally (especially for Date) + * Add the requirement of a referencePoint to ClockProtocol +* **v1.4** + * Move the concept of `now` to the protocol requirement for clocks as an instance method + * Move the `duration(from:to:)` to an instance method on `InstantProtocol` + * Add a number of really useful operators + * Concrete clock types now have `.now` on their `Instant` types + * Added an example ManualClock +* **v1.4.1** + * Clarify the concrete clock types to show their conformances +* **v1.4.2** + * Move the measurement function to clock itself to prevent conflicts with existing APIs +* **v1.4.3** + * Re-added hours and minutes construction + * added a base requirement for `ClockProtocol` to require a `minimumResolution` +* **v1.4.4** + * Rename `ClockProtocol` to `Clock` to better adhere to naming guidelines + * Adjusted measurement to have both a required method (plus default implementation) as well as a free floating function for standardized measurement. +* **v2.0** + * Remove `Date` lowering + * Remove `WallClock` + * Add tolerances + * Remove `.hours` and `.minutes` + * Proposal reorganization + * Added `DurationProtocol` and per instant interval association + * Rename Monotonic and Uptime clocks to Continuous and Suspending to avoid platform ambiguity and perhaps add more clarity of uses. +* **v2.1** + * Refined `DurationProtocol` to only use `Int` instead of `BinaryInteger` for arithmetic. + * Added some additional alternatives considered for `DurationProtocol` and naming associated with it. + * Renamed the associated type on `InstantProtocol` to be `Duration`. + * Added back in task based sleep methods. Added a shorthand for sleeping tasks given a Duration. +* **v3.0** + * Moved `measure` into a category from a protocol requirement + * Renamed the `nanoseconds` and `seconds` property of `Duration` to `nanosecondsPortion` and `secondsPortion` to indicate their fractional composition to types like `timespec` +* **v3.1** + * Adjust the portion accessors to one singular `components` based accessor and add an initializer for raw value construction from components. +* **v3.2** + * Add `Duration` as an associated type requirement of `Clock`, so that it can be marked as the primary associated type. + +
+ +## Introduction + +The concepts of time can be broken down into three distinct parts: + +1. An item to provide a concept of now plus a way to wake up after a given point in time +2. A concept of a point in time +3. A concept of a measurement of elapsed time. + +These three items are respectively a **clock**, an **instant** and a **duration**. The measurement of time can be used for many types of APIs, all the way from the high levels of a concept of a timeout on a network connection, to the amount of time to sleep a task. Currently, the APIs that take measurement of time types take `NSTimeInterval` (aka `TimeInterval`), `DispatchTimeInterval`, and even types like `timespec`. + +## Motivation + +To define a standard way of interacting with time, we need to ensure that in the cases where it is important to limit clock measurement to a specific concept, that ability is preserved. For example, if an API can only accept realtime deadlines as instants, that API cannot be passed to a monotonic instant. This specificity needs to be balanced with the ergonomics of being able to use high-level APIs with little encumbrance of needing to know exactly the time type that is needed; in UI, it might not be welcoming to new developers learning Swift to force them to understand the differential between the myriad of clock concepts available for the operating system. Likewise, any implementation must be robust and performant enough to support multiple operating system back ends (Linux, Darwin, Windows, etc.), but also be easy enough to get right for the common use cases. Practically speaking, durations should be a progressive disclosure to instants and clocks. + +From a performance standpoint, a distinct requirement is that any duration type (or clock type) must be reasonably performant enough to do tasks like measuring the execution performance of a function, without incurring a large overhead to the execution of the measurement. This means that any type that is expressing a duration should be small, and likely backed by some sort of (or group of) PoD type(s). + +Time itself is always measured in a manner that is in reference to a certain frame of analysis. For example, uptime is measured in relative perspective to how long the machine has been booted, whereas other clocks may be relative to a specific epoch. Any instants expressed in terms of a specific reference point may be converted in potentially a lossy manner whereas others may not be convertible at all; so these conversions cannot be uniformly expressed as a general protocol requirement. + +The primary motivation for clocks is to offer a way to schedule work to be done at a later time. Instants are intended to serve a temporal reference point for that scheduling. Durations are specifically designed to be a high precision integral time representing an elapsed duration between two points in time. + +As it stands today, there are a number of APIs and types to represent clocks, instants, and durations. Foundation, for example, defines instant as `Date`, which is constructed from a UTC reference point using an epoch of Jan 1 2001, and `TimeInterval` which is defined as a `Double` representing the number of seconds between two points in time. Dispatch defines `DispatchTime`, `DispatchWallTime`, and `DispatchTimeInterval`; these, respectively, work in relation to a reference of uptime, a wall clock time, and a value of seconds/milliseconds/microseconds/nanoseconds. These obviously are not the only definitions, but when dealing with concurrency, a uniform accessor to all of these concepts is helpful to build the primitives needed for sleep and other temporal concepts. + +## Prior Art + +This proposal focuses on time as used for scheduling work in a process. The most useful clocks for this purpose are simple and local ones that calculate the time since the machine running the process was started. Time can also be expressed in human terms by using calendars, like "April 1, 2021" in the Gregorian calendar. To align with the different responsibilities of the standard library and Foundation, we aim to leave the definition of calendars and the math related to moving between dates in a calendar to Foundation's `Calendar`, `DateComponents`, `TimeZone` and `Date` types. + +For brevity three other languages were chosen to represent an analysis of how time is handled for other languages; Go, Rust, Kotlin. These by no means are the only examples in other languages. Python and C++ also have notable implementations that share some similarities with the proposed implementation. + +### Go +https://pkg.go.dev/time +https://golang.org/src/time/time.go + +Go stores time as a structure of a wall clock reference point (uint64), an 'ext' additional nanoseconds field (int64), and a location (pointer). +Go stores duration as an alias to int64 (nanoseconds). + +There is no control over the reference points in Go to specify a given clock; either monotonic or wall clock. The base implementation attempts to encapsulate both monotonic and wall clock values together in Go. For common use case this likely has little to no impact, however it lacks the specificity needed to identify a progressive disclosure of use. + +### Rust +https://doc.rust-lang.org/stable/std/time/struct.Duration.html + +Rust stores duration as a u64 seconds and a u32 nanoseconds. +The measurement of time in Rust uses Instant, which seems to use a monotonic clock for most platforms. + +### Kotlin +https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.time/-duration/ + +Kotlin stores Duration as a Long plus a unit discriminator comprised of either milliseconds or nanoseconds. Kotlin's measurement functions do not return duration (yet?) but instead rely on conversion functions from Long values in milliseconds etc and those currently measurement functions use system uptime to determine reference points. + +## Detailed Design + +Swift can take this to another level of both accuracy of intent and ease of use beyond any of the other examples given. Following in the themes of other Swift APIs, we can embrace the concept of progressive disclosure and leverage the existing frameworks that define time concepts. + +The given requirements are that we must have a way of expressing the frame of reference of time. This needs to be able to express a concept of now, and a concept of waking up after a given instant has passed. Instants must be able to be compared among each other but are specific to the clock they were obtained. Instants also must be able to be advanced by a given duration or a distance between two instants must be able to emit a duration. Durations must be comparable and also must have some intrinsic unit of time that can suffice for broad application. + +### Clock + +The base protocol for defining a clock requires two primitives; a way to wake up after a given instant, and a way to produce a concept of now. Clocks can also be defined in terms of a potential resolution of access; some clocks may offer resolution at the nanosecond scale, other clocks may offer only microsecond scale. Any values of elapsed time may be considered to be 0 if they are below the minimum resolution. + +```swift +public protocol Clock: Sendable { + associatedtype Duration: DurationProtocol + associatedtype Instant: InstantProtocol where Instant.Duration == Duration + + var now: Instant { get } + + func sleep(until deadline: Instant, tolerance: Instant.Duration?) async throws + + var minResolution: Instant.Duration { get } +} + +extension Clock { + func measure(_ work: () async throws -> Void) reasync rethrows -> Instant.Duration +} +``` + +This means that given an instant, it is intrinsically linked to the clock; e.g., a specific clock's instant is not meaningfully comparable to all other clock instants. However, as an ease of use concession, the duration between two instants can be compared. However, doing this across clocks is potentially considered a programmer error, unless handled very carefully. By making the protocol hierarchy just clocks and instants, it means that we can easily express a compact form of a duration that is usable in all cases; particularly for APIs that might adopt Duration as a replacement to an existing type. + +The clock minimum resolution will have a default implementation that returns `.nanosecond(1)`. This property serves to inform users of a clock the potential minimum granularity of what to invocations to now may return but also indicate the minimum variance between two instants that are significant. Practically speaking, this becomes relevant when measuring work - execution of a small work load may be executed in under the minimum resolution and not provide accurate information. + +Clocks can then be used to measure a given amount of work. This means that clock should have the extensions to allow for the affordance of measuring workloads for metrics, but also measure them for performance benchmarks. This means that making benchmarks is quite easy to do: + +```swift +let elapsed = someClock.measure { + someWorkToBenchmark() +} +``` + +The primary use for a clock beyond vending `now` is to wake up after a given deadline. This affords the possibility to schedule work to be done after that given instant. Wake-ups for scheduled work can incur power implications. Specifically waking up the CPU too often can cause undue power drain. By indicating a tolerance to the deadline it allows the underlying scheduling mechanisms from the kernel to potentially offer a slightly adjusted deadline to wake up by which means that work along with other work being scheduled can be grouped together for more power efficient execution. Not specifying a tolerance infers to the implementor of the clock that the tolerance is up to the implementation details of that clock to choose an appropriate value. The tolerance is a maximum duration after deadline by which the system may delay sleep by. + +``` +func delayedHello() async throws { + try await someClock.sleep(until: .now.advanced(by: .seconds(3)) + print("hello delayed world") +} +``` + +In the above example a clock is slept until 3 seconds from the instant it was called and then prints. The sleep function should throw if the task was cancelled while the sleep function is suspended. In this example the tolerance value is defaulted to nil by the clock and left as a "dealers choice" of how much tolerance may be applied to the deadline. + +### Instant + +As previously stated, instants need to be compared, and might be stored as a key, but only need to define a concept of now, and a way to advance them given a duration. By utilizing a protocol to define an instant, it provides a mechanism in which to use the right storage for the type, but also be type safe with regards to the clock they are intended for. + +The primary reasoning that instants are useful is that they can be composed. Given a function with a deadline as an instant, if it calls another function that takes a deadline as an instant, the original can just be passed without mutation to the next function. That means that the instant in which that deadline elapses does not have interference with the pre-existing calls or execution time in-between functions. One common example of this is the timeout associated with url requests; a timeout does not fully encapsulate how the execution deadline occurs; there is a deadline to meet for the connection to be established, data to be sent, and a response to be received; a timeout spanning all of those must then have measurement to account for each step, whereas a deadline is static throughout. + +```swift +public protocol InstantProtocol: Comparable, Hashable, Sendable { + associatedtype Duration: DurationProtocol + func advanced(by duration: Duration) -> Self + func duration(to other: Self) -> Duration +} + +extension InstantProtocol { + public static func + (_ lhs: Self, _ rhs: Duration) -> Self + public static func - (_ lhs: Self, _ rhs: Duration) -> Self + + public static func += (_ lhs: inout Self, _ rhs: Duration) + public static func -= (_ lhs: inout Self, _ rhs: Duration) + + public static func - (_ lhs: Self, _ rhs: Self) -> Duration +} +``` + +`InstantProtocol`, in addition to the `advance(by:)` and `duration(to:)` methods, has operators to add and subtract durations. However, it does not adhere to `AdditiveArithmetic`. That protocol would require adding two instant values together and defining a zero value (which comes from the clock, and cannot be statically know for all `InstantProtocol` types). Furthermore, InstantProtocol does not require `Strideable` because that requires the stride to be `SignedNumeric` which means that `Duration` would be required to be multiplied by another `Duration` which is inappropriate for two durations. + +If at such time that `Strideable` no longer requires `SignedNumeric` strides, or that `SignedNumeric` no longer requires the multiplication of self; this or adopting types should be considered for adjustment. + +### DurationProtocol + +Specific clocks may have concepts of durations that may express durations outside of temporal concepts. For example a clock tied to the GPU may express durations as a number of frames, whereas a manual clock may express them as steps. Most clocks however will express their duration type as a `Duration` represented by an integral measuring seconds/nanoseconds etc. We feel that it is not an incredibly common task to implement a clock and using the extended name of `Swift.Duration` is reasonable to expect and does not impact normal interactions with clocks. This duration has a few basic requirements; it must be comparable, and able to be added (similar to the concept previously stated with `InstantProtocol` they cannot be `Stridable` since it would mean that two `DurationProtocol` adopting types would then be allowed to be multiplied together). + +```swift +public protocol DurationProtocol: Comparable, AdditiveArithmetic, Sendable { + static func / (_ lhs: Self, _ rhs: Int) -> Self + static func /= (_ lhs: inout Self, _ rhs: Int) + static func * (_ lhs: Self, _ rhs: Int) -> Self + static func *= (_ lhs: inout Self, _ rhs: Int) + + static func / (_ lhs: Self, _ rhs: Self) -> Double +} +``` + +In order to ensure efficient calculations for durations there must be a few additional methods beyond just additive arithmetic that types conforming to `DurationProtocol` must implement - these are the division and multiplication by binary integers and a division creating a double value. This provides the most minimal set of functions to accomplish concepts like the scheduling of a timer, or back-off algorithms. This protocol definition is very close to a concept of `VectorSpace`; if at such time that a more refined protocol definition for a composition of `Comparable` and `AdditiveArithmetic` comes to be - this protocol should be considered as part of any potential improvement in that area. + +The naming of `DurationProtocol` was chosen because we feel that the canonical definition of durations is a temporal duration. All clocks being proposed here have an interval type of `Swift.Duration`; but other more specialized clocks may offer duration types that provide their own custom durations. + +### Duration + +Meaningful durations can always be expressed in terms of nanoseconds plus a number of seconds, either a duration before a reference point or after. They can be constructed from meaningful human measured (or machine measured precision) but should not account for any calendrical calculations (e.g., a measure of days, months or years distinctly need a calendar to be meaningful). Durations should able to be serialized, compared, and stored as keys, but also should be able to be added and subtracted (and zero is meaningful). They are distinctly NOT `Numeric` due to the aforementioned issue with regards to multiplying two `TimeInterval` variables. That being said, there is utility for ad-hoc division and multiplication to calculate back-offs. + +The `Duration` must be able to account for high scale resolution of calculation; the storage will under the hood ensure proper rounding for division (by likely storing higher precision than exposed) and enough range to span the full range of potential reasonable instants. This means that spanning the full range of +/- thousands of years at a non lossy scale can be accomplished by storing the seconds and nanoseconds. Not all systems will need that full range, however in order to properly represent nanosecond precision across the full range of times expressed in the operating systems that Swift works on a full 128 bit storage is needed to represent these values. That in turn necessitates exposing the conversion to existing types as breaking the duration into two components. These components of a duration are exposed for interoperability with existing APIs such as `timespec` as a seconds portion and an attoseconds portion (used to ensure full precision is not lost). If the Swift language gains a signed integer type that can support 128 bits of storage then `Duration` should be considered to replace the components accessor and initializer with a direct access and initialization to that stored attoseconds value. + +```swift +public struct Duration: Sendable { + public var components: (seconds: Int64, attoseconds: Int64) { get } + public init(secondsComponent: Int64, attosecondsComponent: Int64) +} + + +extension Duration { + public static func seconds(_ seconds: T) -> Duration + public static func seconds(_ seconds: Double) -> Duration + public static func milliseconds(_ milliseconds: T) -> Duration + public static func milliseconds(_ milliseconds: Double) -> Duration + public static func microseconds(_ microseconds: T) -> Duration + public static func microseconds(_ microseconds: Double) -> Duration + public static func nanoseconds(_ value: T) -> Duration +} + +extension Duration: Codable { } +extension Duration: Hashable { } +extension Duration: Equatable { } +extension Duration: Comparable { } +extension Duration: AdditiveArithmetic { } + +extension Duration { + public static func / (_ lhs: Duration, _ rhs: Double) -> Duration + public static func /= (_ lhs: inout Duration, _ rhs: Double) + public static func / (_ lhs: Duration, _ rhs: Int) -> Duration + public static func /= (_ lhs: inout Duration, _ rhs: Int) + public static func / (_ lhs: Duration, _ rhs: Duration) -> Double + public static func * (_ lhs: Duration, _ rhs: Double) -> Duration + public static func *= (_ lhs: inout Duration, _ rhs: Double) + public static func * (_ lhs: Duration, _ rhs: Int) -> Duration + public static func *= (_ lhs: inout Duration, _ rhs: Int) +} + +extension Duration: DurationProtocol { } +``` + +### ContinuousClock + +When instants are for local processing only and need to be high resolution without the encumbrance of suspension while the machine is asleep, `ContinuousClock` is the tool for the job. On Darwin platforms this refers to time derived from the monotonic clock, for linux platforms this is in reference to the uptime clock; being that those two are the closest in behavioral meaning. This clock also offers an extension to access the clock instance as the inferred base type property. + +```swift +public struct ContinuousClock { + public init() + + public static var now: Instant { get } +} + +extension ContinuousClock: Clock { + public struct Instant { + public static var now: ContinuousClock.Instant { get } + } + + public var now: Instant { get } + public var minimumResolution: Duration { get } + public func sleep(until deadline: Instant, tolerance: Duration? = nil) async throws +} + +extension ContinuousClock.Instant: InstantProtocol { + public func advanced(by duration: Duration) -> ContinuousClock.Instant + public func duration(to other: ContinuousClock.Instant) -> Duration +} + +extension Clock where Self == ContinuousClock { + public static var continuous: ContinuousClock { get } +} +``` + +### SuspendingClock + +Where local process scoped or cross machine scoped instants are not suitable: uptime serves the purpose of a clock that does not increment while the machine is asleep but is a time that is referenced to the boot time of the machine. This allows for the affordance of cross process communication in the scope of that machine. Similar to the other clocks there is an extension to access the clock instance as the inferred base type property. For Darwin based platforms this is derived from the uptime clock whereas for linux based platforms this is derived from the monotonic clock since those most closely represent the concept for not incrementing while the machine is asleep. + +```swift +public struct SuspendingClock { + public init() + + public static var now: Instant { get } +} + +extension SuspendingClock: Clock { + public struct Instant { + public static var now: SuspendingClock.Instant { get } + } + + public var minimumResolution: Duration { get } + public func sleep(until deadline: Instant, tolerance: Duration? = nil) async throws +} + +extension SuspendingClock.Instant: InstantProtocol { + public func advanced(by duration: Duration) -> SuspendingClock.Instant + public func duration(to other: SuspendingClock.Instant) -> Duration +} + +extension Clock where Self == SuspendingClock { + public static var suspending: SuspendingClock { get } +} +``` + +### Clocks Outside of the Standard Library + +In previous iterations of this proposal we offered a concept of a WallClock, however, after some compelling feedback we feel that this type may not be the most generally useful without the context of calendrical calculations. Since Foundation is the home of these types of calculations we feel that a clock based upon UTC more suitably belongs in that layer. This clock will adjust the fire time based upon the current UTC time; this means that that if a bit of work is scheduled by a specific time of day made by calculation via `Calendar` this clock can wake up from the sleep when the system time hits that deadline. + +Foundation will provide a type `UTCClock` that encompasses this behavior and use `Date` as the instant type. Additionally Foundation will provide conversions to and from `Date` to the other instant types in this proposal. + +```swift +public struct UTCClock { + public init() + + public static var now: Date { get } +} + +extension UTCClock: Clock { + public var minimumResolution: Duration { get } + public func sleep(until deadline: Date, tolerance: Duration? = nil) async throws +} + +extension Date { + public func leapSeconds(to other: Date) -> Duration + public init(_ instant: ContinuousClock.Instant) + public init(_ instant: SuspendingClock.Instant) +} + +extension ContinuousClock.Instant { + public init?(_ instant: Date) +} + +extension SuspendingClock.Instant { + public init?(_ instant: Date) +} + +extension Date: InstantProtocol { + public func advanced(by duration: Duration) -> Date + public func duration(to other: Date) -> Duration +} + +extension Clock where Self == UTCClock { + public static var utc: UTCClock { get } +} +``` + +The `UTCClock` will allow for a method in which to wake up after a deadline defined by a `Date`. The implementation of `Date` transacts upon the number of seconds since Jan 1 2001 as defined by the system clock so any network time (or manual) updates may shift that point of now either forward or backward depending on the skew the system clock may undergo. The value being stored is not dependent upon timezone, daylight savings, or calendrical representation but the current NTP updates do represent any applied leap seconds that may have occurred. In light of this particular edge case that previously was not exposed, `Date` will now offer a new method to determine the leap second duration that may have elapsed between a given data and another date. This provides a method in which to account for these leap seconds in a historical sense. Similar to timezone databases the leap seconds will be updated (if there is any additional planned leap seconds) along with software updates. + +Previous revisions of this proposal moved `Date` to the standard library along with a new wall clock that uses it. After feedback from the community, we now believe the utility of this clock is very specialized and more closely related to the calendar types in Foundation. Therefore, `Date` will remain in Foundation alongside them. + +`Date` is best used as the storage for point in time to be interpreted using a `Calendar`, `TimeZone`, and with formatting functions for display to people. A survey of the existing `Date` API in the macOS and iOS SDKs shows this to already be the case for the vast majority of properties and functions that use it. The discussion around the appropriateness of the `Date` name was mostly focused on its uses in *non*-calendrical contexts. We hope this combination of `Date` and `UTCClock` will help reinforce the relationship between those types and add clarity to when it should be used. + +This approach preserves compatibility with those APIs while still providing the capability to use `Date` for scheduling in the rare cases that it is needed. + +### Task + +The existing `Task` API has methods in which to sleep. These existing methods do not have any specified behavior of sleeping; however under the hood it uses a continuous clock on Darwin and a suspending clock on Linux. + +The existing API for sleeping will be deprecated, and the existing deprecation will be updated accordingly to point to the new APIs. + +```swift +extension Task { + @available(*, deprecated, renamed: "Task.sleep(for:)") + public static func sleep(_ duration: UInt64) async + + @available(*, deprecated, renamed: "Task.sleep(for:)") + public static func sleep(nanoseconds duration: UInt64) async throws + + public static func sleep(for: Duration) async throws + + public static func sleep(until deadline: C.Instant, tolerance: C.Instant.Duration? = nil, clock: C) async throws +} +``` + +### Example Custom Clock + +One example for adopting `Clock` is a manual clock. This could be a useful item for testing (but not currently part of this proposal as an API to add). It allows for the manual advancement of time in a deterministic manner. The general intent is to allow the manual clock type to be advanced from one thread and the sleep function can then be used to act as if it was a standard clock in generic APIs. + +```swift +public final class ManualClock: Clock, @unchecked Sendable { + public struct Instant: InstantProtocol { + var offset: Duration = .zero + + public func advanced(by duration: Duration) -> ManualClock.Instant { + Instant(offset: offset + duration) + } + + public func duration(to other: ManualClock.Instant) -> Duration { + other.offset - offset + } + + public static func < (_ lhs: ManualClock.Instant, _ rhs: ManualClock.Instant) -> Bool { + lhs.offset < rhs.offset + } + } + + struct WakeUp { + var when: Instant + var continuation: UnsafeContinuation + } + + public private(set) var now = Instant() + + // General storage for the sleep points we want to wake-up for + // this could be optimized to be a more efficient data structure + // as well as enforced for generation stability for ordering + var wakeUps = [WakeUp]() + + // adjusting now or the wake-ups can be done from different threads/tasks + // so they need to be treated as critical mutations + let lock = os_unfair_lock_t.allocate(capacity: 1) + + deinit { + lock.deallocate() + } + + public func sleep(until deadline: Instant, tolerance: Duration? = nil) async throws { + // Enqueue a pending wake-up into the list such that when + return await withUnsafeContinuation { + if deadline <= now { + $0.resume() + } else { + os_unfair_lock_lock(lock) + wakeUps.append(WakeUp(when: deadline, continuation: $0)) + os_unfair_lock_unlock(lock) + } + } + } + + public func advance(by amount: Duration) { + // step the now forward and gather all of the pending + // wake-ups that are in need of execution + os_unfair_lock_lock(lock) + now += amount + var toService = [WakeUp]() + for index in (0..<(wakeUps.count)).reversed() { + let wakeUp = wakeUps[index] + if wakeUp.when <= now { + toService.insert(wakeUp, at: 0) + wakeUps.remove(at: index) + } + } + os_unfair_lock_unlock(lock) + + // make sure to service them outside of the lock + toService.sort { lhs, rhs -> Bool in + lhs.when < rhs.when + } + for item in toService { + item.continuation.resume() + } + } +} +``` + +## Existing Application Code + +This proposal is purely additive and has no direct impact to existing application code. + +## Impact on ABI + +The proposed implementation will introduce three runtime functions; a way of obtaining time, a way of sleeping given a standard clock, and a way of obtaining the minimum resolution given a standard clock. + +## Alternatives Considered + +### Singular Instant Representation + +It was considered to have a singular type to represent monotonic, uptime, and wall clock instants similar to Go. However this approach causes a problem with comparability; an instant may be greater in one respect but less or equal in some other respect. In order to properly adhere to `Comparable` as a requisite to `InstantProtocol` we feel that combining the instants into one unified type is not ideal. + +### Inverted Protocol Hierarchy + +Another exploration was to have an inverted scheme of instant and clock however this means that the generic signatures of functions that use specific clocks or instants become much more difficult to write. + +### Lowering of Date/UTCClock + +Originally the proposal included a concept of lowering `Date` to the standard library in addition to altering its storage from `Double` to a `Duration`. There were strong objections on a few fronts with this move which ultimately had convincing merit. The primary objection was to the name `Date`; given that there was no additional contextual API within the standard library or concurrency library this meant that `Date` could easily get confused with the concept of a calendrical date (which that type definitively is not). Additionally it was rightfully brought up that `Date` is missing concepts of leap seconds (which has since been accepted and proposed as an alteration to Foundation) because we see the utility of that as an additional functionality to `Date`. + +Also in the original revisions of the proposal we had a concept of `WallClock`. After much discussion we feel that the name wall clock is misleading since the type really represents a clock based on UTC (once `Date` has a historical accounting of leap seconds). But furthermore, we feel that the general utility of scheduling via a UTC clock is not a common task and that a vast majority of clocks for scheduling are really things that transact either via a clock that time passes while the machine is asleep or a clock that time does not pass while the machine is asleep. That accounting means that we feel that the right home for `UTCClock` is in a higher level framework for that specialized task along side the calendrical calculation APIs; which is Foundation. + +### DurationProtocol Generalized Arithmetics and Protocol Definition + +It was considered to have a more general form of the arithmetics for `DurationProtocol`. This poses a potential pitfall for adopters that may inadvertently implement some truncation of values. Since most values passed around that are integral types are spelled as `Int` it means that this interface is better served as just using multiplication and division via `Int`. In that vein; it was also considered to use `Double` instead, this however does not work nicely for types that define durations like "steps" or "frames"; e.g. things that are not distinctly divisible beyond 1 unit. It is still under the domain of that `DurationProtocol` adopting type to define that behavior and how it rounds or asserts etc. + +Similarly to the arithmetics; it was also considered to have the associated type to `InstantProtocol` as just a glob of protocols `Comparable & AdditiveArithmetic & Sendable`, however this lacks the capability of fast-paths for things like back-offs (ala Zeno's algorithm) or debounce, or timer coalescing. Some of them could be re-written in terms of loops of addition, however it would likely result in hot-looping over missed intervals in some cases, or in others not even being able to implement them (e.g. division for back-offs). + +### Clock and Task Sleep Tolerance Optionality + +It was raised that the hint from IDEs such as Xcode for the `.none` autocomplete do exist and those nomenclatures are perhaps misleading for the `tolerance` parameter to the sleep functions. We agree that this is perhaps a less than ideal name to expose as an autocomplete, however it was decided that code using `.none` instead of not passing a parameter or passing `nil` is stylistically problematic and left-overs from earlier versions of swift. It was concluded that the solutions in this space should be applicable to any other method that has an optional parameter and not just `Clock` and `Task`; moreover it seems like this is perhaps a bug in Xcode's autocomplete than an issue with the API as proposed since the `ContinuousClock`, `SuspendingClock` and `UTCClock` being proposed are most meaningful of the lack of a parameter value than to introduce any sort of enumeration mirroring `Optional` without any sort of direct type passing capability. In short - a more general solution should be approached with this problem and the optional duration type should remain. + +### Alternative Names + +There have been a number of names that have been considered during this proposal (these are a few highlights): + +The protocol `Clock` has been considered to be named: +* `ClockProtocol` - The protocol suffix was considered superfluous and a violation of the naming guidelines. + +The protocol `InstantProtocol` has been considered to be named: +* `ReferencePoint` - This ended up being too vague and did not capture the concept of time +* `Deadline`/`DeadlineProtocol` - Not all instant types are actually deadlines, so the nomenclature became confusing. +* The associated type of `InstantProtocol.Duration` was considered for a few other names; `TimeSpan` and `Interval`. These names lack symmetry; `Clock` has an `Instant` which is an `InstantProtocol`, `InstantProtocol` has a `Duration` which is a `DurationProtocol`. + +The protocol `DurationProtocol` has been considered to be named: +* Not having it has been considered but ultimately rejected to ensure flexibility of the API for other clock types that transact in concept like "frames" or "steps". + +The clock `ContinuousClock` has been considered to be named: +* `MonotonicClock` - Unfortunately Darwin and Linux differ on the definition of monotonic. +* `UniformClock` - This does not disambiguate the behavioral difference between this clock and the `SuspendingClock` since both are uniform in their incrementing while the machine is not asleep. + +The clock `SuspendingClock` has been considered to be named: +* `UptimeClock` - Just as `MonotonicClock` has ambiguity with regards to Linux and Darwin behaviors. +* `AbsoluteClock` - Very vague when not immediately steeped in mach-isms. +* `ExecutionClock` - The name more infers the concept of `CLOCK_PROCESS_CPUTIME_ID` than `CLOCK_UPTIME_RAW` (on Darwin). +* `DiscontinuousClock` - Has its roots in the mathematical concept of discontinuous functions but perhaps is not immediately obvious that it is the clock that does not advance while the machine is asleep + +The type `Duration` has been considered to be named: +* `Interval` - This is quite ambiguous and could refer to numerous other concepts other than time. +* The `nanosecondsPortion` and `secondsPortion` were considered to be named `nanoseconds` and `seconds` however those names posed ambiguity of rounding; naming them with the term portion infers their fractional composition rather than just a rounded/truncated value. + +The type `Date` has been considered to be named: +* `Timestamp` - A decent alternative but still comes at a slight ambiguity with regards to being tied to a calendar. Also has string like connotations (with how it is used in logs) +* `Timepoint`/`TimePoint` - A reasonable alternative with less ambiguity but ultimately not compelling enough to churn thousands of APIs that already exist (just counting the ones included in the iOS and macOS SDKs, not to mention the other use sites that may exist). +* `WallClock.Instant`/`UTCClock.Instant` - This is a very wordy way of spelling the same idea as `Date` represents today. + +The `Task.sleep(for:tolerance:clock:)` API has been considered to be named: +* `Task.sleep(_:tolerance:clock:)` - even though this is still grammatically correct and omits potentially a needless word of "for", having this extra word still reads well but also offers a better fix-it for migration from deprecated APIs. That migration was considered worth it to keep the "for". + +## Appendix + +Time is relative, temporal types doubly so. In this document, there will be some discussion with regards to the categorization of temporal types that readers should be distinctly aware of. + +**Calendar:** A human locale based system in which to measure time. + +**Clock:** The mechanism in which to measure time, and understand how that time flows. + +**Continuous Time:** Time that always increments but does not stop incrementing while the system is asleep. This is useful to consider as a stopwatch style time; the reference point at which this starts and are most definitely different per machine. + +**Date:** A Date value encapsulates a single point in time, independent of any particular calendrical system or time zone. Date values represent a time interval relative to an absolute reference date. + +**Deadline:** In common parlance, it is a limit defined as an instant in time: a narrow field of time by which an objective must be accomplished. + +**Duration:** A measurement of how much time has elapsed between two deadlines or reference points. + +**Instant:** A precise moment in time. + +**Monotonic Time:** Darwin and BSD define this as continuous time. Linux, however, defines this as a time that always increments, but does stop incrementing while the system is asleep. + +**Network Update Time:** A value of wall clock time that is transmitted via ntp; used to synchronize the wall clocks of machines connected to a network. + +**Temporal:** Related to the concept of time. + +**Time Zone:** An arbitrary political defined system in which to normalize time in a quasi-geospatial delineation intended to keep the apex of the solar day around 12:00. + +**Tolerance:** The duration around a given point in time is accepted as accurate. + +**Uptime:** Darwin and BSD define this as absolute time suspending when asleep. Linux, however, defines this as time that does not suspend while asleep but is relative to the boot. + +**Wall Clock Time:** Time like reading from a clock. This may be adjusted forwards or backwards for numerous reasons; in this context, it is time that is not specific to a time zone or locale, but measured from an absolute reference date. Network updates may adjust the drift on the clock either backwards or forwards depending on the relativistic drift, clock skew from inaccuracies with the processor, or from hardware power characteristics. diff --git a/proposals/0331-remove-sendable-from-unsafepointer.md b/proposals/0331-remove-sendable-from-unsafepointer.md index e9d009a3a8..c392413e48 100644 --- a/proposals/0331-remove-sendable-from-unsafepointer.md +++ b/proposals/0331-remove-sendable-from-unsafepointer.md @@ -3,8 +3,8 @@ * Proposal: [SE-0331](0331-remove-sendable-from-unsafepointer.md) * Authors: [Andrew Trick](https://github.com/atrick) * Review Manager: [Doug Gregor](https://github.com/DougGregor) -* Status: **Active review (November 29...December 10, 2021)** - +* Status: **Implemented (Swift 5.6)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0331-remove-sendable-conformance-from-unsafe-pointer-types/53979) * Implementation: [apple/swift#39218](https://github.com/apple/swift/pull/39218) ## Introduction diff --git a/proposals/0332-swiftpm-command-plugins.md b/proposals/0332-swiftpm-command-plugins.md index 3be7c32e25..3e30dc8957 100644 --- a/proposals/0332-swiftpm-command-plugins.md +++ b/proposals/0332-swiftpm-command-plugins.md @@ -3,13 +3,14 @@ * Proposal: [SE-0332](0332-swiftpm-command-plugins.md) * Author: [Anders Bertelrud](https://github.com/abertelrud) * Review Manager: [Tom Doron](https://github.com/tomerd) -* Status: **Active review (November 29...December 10, 2021)** +* Status: **Implemented (Swift 5.6)** * Implementation: [apple/swift-package-manager#3855](https://github.com/apple/swift-package-manager/pull/3855) -* Pitch: [Forum discussion](https://forums.swift.org/t/pitch-package-manager-command-plugins/53172) +* Pitch: [Forum discussion](https://forums.swift.org/t/pitch-package-manager-command-plugins/) +* Review: [Forum discussion](https://forums.swift.org/t/se-0332-package-manager-command-plugins/) ## Introduction -SE-0303 introduced the ability to define *build tool plugins* in SwiftPM, allowing custom tools to be automatically invoked during a build. This proposal extends that plugin support to allow the definition of custom *command plugins* — plugins that users can invoke directly from the SwiftPM CLI, or from an IDE that supports Swift Packages, in order to perform custom actions on their packages. +[SE-0303](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md) introduced the ability to define *build tool plugins* in SwiftPM, allowing custom tools to be automatically invoked during a build. This proposal extends that plugin support to allow the definition of custom *command plugins* — plugins that users can invoke directly from the SwiftPM CLI, or from an IDE that supports Swift Packages, in order to perform custom actions on their packages. ## Motivation @@ -23,7 +24,7 @@ Separately to this proposal, it would also be useful to define custom actions th ## Proposed Solution -This proposal defines a new plugin capability called `command` that allows packages to augment the set of package-related commands availabile in the SwiftPM CLI and in IDEs that support packages. A command plugin specifies the semantic intent of the command — this might be one of the predefined intents such “documentation generation” or “source code formatting”, or it might be a custom intent with a specialized verb that can be passed to the `swift` `package` command. A command plugin can also specify any special permissions it needs (such as the permission to modify the files under the package directory). +This proposal defines a new plugin capability called `command` that allows packages to augment the set of package-related commands available in the SwiftPM CLI and in IDEs that support packages. A command plugin specifies the semantic intent of the command — this might be one of the predefined intents such “documentation generation” or “source code formatting”, or it might be a custom intent with a specialized verb that can be passed to the `swift` `package` command. A command plugin can also specify any special permissions it needs (such as the permission to modify the files under the package directory). The command's intent declaration provides a way of grouping command plugins by their functional categories, so that SwiftPM — or an IDE that supports SwiftPM packages — can show the commands that are available for a particular purpose. For example, this approach supports having different command plugins for generating documentation for a package, while still allowing those different commands to be grouped and discovered by intent. @@ -67,26 +68,26 @@ extension PluginCapability { The plugin specifies the intent of the command as either one of a set of predefined intents or as a custom intent with an custom verb and help description. -In this proposal, the intent is expressed as an enum provided by SwiftPM in `PackageDescription`: +In this proposal, the intent is expressed as an opaque struct with enum semantics in `PackageDescription`: ```swift -enum PluginCommandIntent { +public struct PluginCommandIntent { /// The intent of the command is to generate documentation, either by parsing the /// package contents directly or by using the build system support for generating /// symbol graphs. Invoked by a `generate-documentation` verb to `swift package`. - case documentationGeneration + public static func documentationGeneration() -> PluginCommandIntent /// The intent of the command is to modify the source code in the package based /// on a set of rules. Invoked by a `format-source-code` verb to `swift package`. - case sourceCodeFormatting + public static func sourceCodeFormatting() -> PluginCommandIntent /// An intent that doesn't fit into any of the other categories, with a custom /// verb through which it can be invoked. - case custom(verb: String, description: String) + public static func custom(verb: String, description: String) -> PluginCommandIntent } ``` -Future versions of SwiftPM will almost certainly add to this set of possible intents, using availability annotations gated on the tools version to conditionally make new enum cases available. +Future proposals will almost certainly add to this set of possible intents, using availability annotations gated on the tools version to conditionally make new types of intent available. If multiple command plugins in the dependency graph of a package specify the same intent, or specify a custom intent with the same verb, then the user will need to specify which plugin to invoke by qualifying the verb with the name of the plugin target followed by a `:` character, e.g. `MyPlugin:do-something`. Because plugin names are target names, they are already known to be unique within the package graph, so the combination of plugin name and verb is known to be unique. @@ -94,23 +95,21 @@ A command plugin can also specify the permissions it needs, which affect the way A command plugin that wants to modify the package source code (as for example a source code formatter might want to) needs to request the `writeToPackageDirectory` permission. This modifies the sandbox in which the plugin is invoked to let it write inside the package directory in the file system, after notifying the user about what is going to happen and getting approval in a way that is appropriate for the IDE in question. -The permissions needed by the command are expressed as an enum in `PackageDescription`: +The permissions needed by the command are expressed as an opaque static struct with enum semantics in `PackageDescription`: ```swift -enum PluginPermission { +public struct PluginPermission { /// The command plugin wants permission to modify the files under the package /// directory. The `reason` string is shown to the user at the time of request /// for approval, explaining why the plugin is requesting this access. - case writeToPackageDirectory(reason: String) - - /// It is likely that future proposals will want to provide some kind of network - /// access. In the interest of keeping this proposal bounded, we just note that - /// as a possible future need here but do not initially allow any network access. - - /// Any future enum cases should use @available() + public static func writeToPackageDirectory(reason: String) -> PluginPermission } ``` +Future proposals will almost certainly add to this set of possible permissions, using availability annotations gated on the tools version to conditionally make new types of permission available. + +In particular, it is likely that future proposals will want to provide a way for a plugin to ask for permission to access the network. In the interest of keeping this proposal bounded, we note that as a possible future need here, but do not initially allow any network access. + ### Plugin API This proposal extends the PackagePlugin API to: @@ -134,11 +133,6 @@ public protocol CommandPlugin: Plugin { /// directories, etc. context: PluginContext, - /// The targets to which the command should be applied. If the invoker of - /// the command has not specified particular targets, this will be a list - /// of all the targets in the package to which the command is applied. - targets: [Target], - /// Any literal arguments passed after the verb in the command invocation. arguments: [String], ) async throws @@ -149,7 +143,7 @@ public protocol CommandPlugin: Plugin { } ``` -This defines a basic entry point for a command plugin, passing it information about the context in which the plugin is invoked (including information about the package graph), the set of targets on which the command should operate, and the arguments passed by the user after the verb in the `swift` `package` invocation. +This defines a basic entry point for a command plugin, passing it information about the context in which the plugin is invoked (including information about the package graph) and the arguments passed by the user after the verb in the `swift` `package` invocation. The `context` parameter provides access to the package to which the user applies the plugin, including any dependencies, and it also provides access to a working directory that the plugin can use for any purposes, as well as a way to look up command line tools with a given name. This is the same as the support that is available to all plugins via SE-0325. @@ -159,6 +153,8 @@ Many command plugins will invoke tools using subprocesses in order to do the act Plugins can also use Foundation APIs for reading and writing files, encoding and decoding JSON, and other actions. +The arguments are a literal array of strings that the user specified when invoking the plugin. Plugins that operate on individual targets or products would typically support a `--target` or `--product` option that allows users to specify the names of targets or products to operate on in the package to which the plugin command is applied. + #### Accessing Package Manager Services In addition to invoking invoking tool executables and using Foundation APIs, command plugins can use the `packageManager` property to obtain more specialized information and to invoke certain SwiftPM services. This is a proxy to SwiftPM or to the IDE that is hosting the plugin, and provides access to some of its functionality. The set of services provided in this API is expected to grow over time, and would ideally, over time, comprise most of the SwiftPM functionality available in its CLI. @@ -332,11 +328,11 @@ public struct PackageManager { /// Represents the results of running a single test. public struct Test { public var name: String - public var outcome: Outcome + public var result: Result public var duration: Double - /// Represents the outcome of running a single test. - public enum Outcome { + /// Represents the result of running a single test. + public enum Result { case succeeded, skipped, failed } } @@ -390,7 +386,7 @@ public struct PackageManager { ### Permissions -Like other plugins, command plugins are run in a sandbox on platforms that support it. By default this sandbox does not allow the plugin to modify the file system (except in special temporary-files paths) and blocks any network access. +Like other plugins, command plugins are run in a sandbox on platforms that support it. By default this sandbox does not allow the plugin to modify the file system (except in special temporary-files paths) and it blocks any network access. Some commands, such as source code formatters, might need to modify the file system in order to be useful. Such plugins can specify the permissions they need, and this will: @@ -399,11 +395,15 @@ Some commands, such as source code formatters, might need to modify the file sys The exact form of the notification and approval will depend on the CLI or IDE that runs the plugin. SwiftPM’s CLI is expected to ask the user for permission using a console prompt (if connected to TTY), and to provide options for approving or rejecting the request when not connected to a TTY. +Note that this approval needs to be obtained before running the plugin, which is why it is declared in the package manifest. There is currently no provision for a plugin to ask for more permissions while it runs. + An IDE might present user interface affordances providing the notification and allowing the choice. In order to avoid having to request permission every time the plugin is invoked, some kind of caching of the response could be implemented. +SwiftPM or IDEs may also provide options to allow users to specify additional writable file system locations for the plugin, but that would not affect the API described in this proposal. + ### Invoking Command Plugins -In the SwiftPM CLI, command plugins provided by the package or its dependencies are available as verbs that can be specified in a `swift` `package` invocation. For example, if the root package defines a command plugin with a `do-something` verb — or if it has a dependency on a package that defines such a plugin — a user can run it using the invocation: +In the SwiftPM CLI, command plugins provided by a package or its direct dependencies are available as verbs that can be specified in a `swift` `package` invocation. For example, if the root package defines a command plugin with a `do-something` verb — or if it has a dependency on a package that defines such a plugin — a user can run it using the invocation: ```shell ❯ swift package do-something @@ -411,19 +411,13 @@ In the SwiftPM CLI, command plugins provided by the package or its dependencies This will invoke the plugin and only return when it completes. Since no other options were provided, this will pass all regular targets in the package to the plugin ("special" targets such as those that define plugins will be excluded). -To pass a subset of the targets to the plugin, one or more `--target` options can be used in the invocation: +Any parameters passed after the name of the plugin command are passed verbatim to the entry point of the plugin. For example, if a plugin accepts a `--target` option, a subset of the targets to operate on can be passed on the command line that invokes the plugin: ```shell -❯ swift package --target Foo --target Bar do-something +❯ swift package do-something --target Foo --target Bar --someOtherFlag ``` -This will pass the `Foo` and `Bar` targets to the plugin (assuming those are names of regular targets defined in the package — if they are not, an error is emitted). - -The user can also provide additional parameters that are passed directly to the plugin. In the following example, the plugin will receive the parameters `aParam` and `-aFlag`, in addition to the targets named `Foo` and `Bar`. - -```shell -❯ swift package --target Foo --target Bar do-something aParam -aFlag -``` +It is the responsibility of the plugin to interpret any command line arguments passed to it. Arguments are currently passed to the plugin exactly as they are written after the command’s verb. A future proposal could allow the plugin to define parameters (using SwiftArgumentParser) that SwiftPM could interpret and that would integrate better with SwiftPM’s own command line arguments. @@ -483,7 +477,7 @@ let package = Package( .plugin( name: "MyDocCPlugin", capability: .command( - intent: .documentationGeneration + intent: .documentationGeneration() ) ) ] @@ -500,31 +494,30 @@ import Foundation struct MyDocCPlugin: CommandPlugin { func performCommand( context: PluginContext, - targets: [Target], arguments: [String] ) async throws { // We'll be creating commands that invoke `docc`, so start by locating it. let doccTool = try context.tool(named: "docc") - + // Construct the path of the directory in which to emit documentation. let outputDir = context.pluginWorkDirectory.appending("Outputs") - - // Iterate over the targets we were given. - for target in targets { + + // Iterate over the targets in the package. + for target in context.package.targets { // Only consider those kinds of targets that can have source files. guard let target = target as? SourceModuleTarget else { continue } - + // Find the first DocC catalog in the target, if there is one (a more // robust example would handle the presence of multiple catalogs). let doccCatalog = target.sourceFiles.first { $0.path.extension == "docc" } - + // Ask SwiftPM to generate or update symbol graph files for the target. - let symbolGraphInfo = try packageManager.getSymbolGraph(for: target, + let symbolGraphInfo = try await packageManager.getSymbolGraph(for: target, options: .init( minimumAccessLevel: .public, includeSynthesized: false, includeSPI: false)) - + // Invoke `docc` with arguments and the optional catalog. let doccExec = URL(fileURLWithPath: doccTool.path.string) var doccArgs = ["convert"] @@ -541,9 +534,14 @@ struct MyDocCPlugin: CommandPlugin { let process = try Process.run(doccExec, arguments: doccArgs) process.waitUntilExit() - // The plugin should also report non-zero exit codes from `docc` here. - - print("Generated documentation at \(outputDir).") + // Check whether the subprocess invocation was successful. + if process.terminationReason == .exit && process.terminationStatus == 0 { + print("Generated documentation at \(outputDir).") + } + else { + let problem = "\(process.terminationReason):\(process.terminationStatus)" + Diagnostics.error("docc invocation failed: \(problem)") + } } } } @@ -574,8 +572,6 @@ Users can then invoke this command plugin using the `swift` `package` invocation ❯ swift package generate-documentation ``` -Since no `--target` options are provided, SwiftPM passes all the package’s regular targets to the plugin (in this simple example, just the `MyLibrary` target). - The plugin would usually print the path at which it generated the documentation. ## Example 2: Formatting Source Code @@ -595,9 +591,9 @@ let package = Package( ], targets: [ .plugin( - "MyFormatterPlugin", + name: "MyFormatterPlugin", capability: .command( - intent: .sourceCodeFormatting, + intent: .sourceCodeFormatting(), permissions: [ .writeToPackageDirectory(reason: "This command reformats source files") ] @@ -619,22 +615,21 @@ import Foundation @main struct MyFormatterPlugin: CommandPlugin { func performCommand( - context: PluginContext, - targets: [Target], - arguments: [String] + context: PluginContext, + arguments: [String] ) async throws { // We'll be invoking `swift-format`, so start by locating it. let swiftFormatTool = try context.tool(named: "swift-format") - + // By convention, use a configuration file in the package directory. let configFile = context.package.directory.appending(".swift-format.json") - - // Iterate over the targets we've been asked to format. - for target in targets { + + // Iterate over the targets in the package. + for target in context.package.targets { // Skip any type of target that doesn't have source files. // Note: We could choose to instead emit a warning or error here. guard let target = target as? SourceModuleTarget else { continue } - + // Invoke `swift-format` on the target directory, passing a configuration // file from the package directory. let swiftFormatExec = URL(fileURLWithPath: swiftFormatTool.path.string) @@ -647,9 +642,14 @@ struct MyFormatterPlugin: CommandPlugin { let process = try Process.run(swiftFormatExec, arguments: swiftFormatArgs) process.waitUntilExit() - // The plugin should also report non-zero exit codes from `swift-format` here. - - print("Formatted the source code in \(target.directory).") + // Check whether the subprocess invocation was successful. + if process.terminationReason == .exit && process.terminationStatus == 0 { + print("Formatted the source code in \(target.directory).") + } + else { + let problem = "\(process.terminationReason):\(process.terminationStatus)" + Diagnostics.error("swift-format invocation failed: \(problem)") + } } } } @@ -681,17 +681,17 @@ let package = Package( targets: [ // This is the hypothetical executable we want to distribute. .executableTarget( - "MyExec" + name: "MyExec" ), // This is the plugin that defines a custom command to distribute the executable. .plugin( - "MyDistributionArchiveCreator", + name: "MyDistributionArchiveCreator", capability: .command( intent: .custom( verb: "create-distribution-archive", - description: "Creates a .tar file containing release binaries" + description: "Creates a .zip containing release builds of products" ) - ), + ) ) ] ) @@ -706,44 +706,49 @@ import Foundation @main struct MyDistributionArchiveCreator: CommandPlugin { func performCommand( - context: PluginContext, - targets: [Target], - arguments: [String] + context: PluginContext, + arguments: [String] ) async throws { // Check that we were given the name of a product as the first argument // and the name of an archive as the second. guard arguments.count == 2 else { - throw Error("Expected two arguments: product name and archive name") + throw Error("Expected two arguments: product name and archive name") } let productName = arguments[0] let archiveName = arguments[1] - + // Ask the plugin host (SwiftPM or an IDE) to build our product. - let result = await packageManager.build( + let result = try await packageManager.build( .product(productName), parameters: .init(configuration: .release, logging: .concise) ) // Check the result. Ideally this would report more details. guard result.succeeded else { throw Error("couldn't build product") } - + // Get the list of built executables from the build result. - let builtExecutables = result.builtArtifacts.first{ $0.kind == .executable } - + let builtExecutables = result.builtArtifacts.filter{ $0.kind == .executable } + // Decide on the output path for the archive. - let outputPath = context.pluginWorkDirectory.appending("\(archiveName).tar") - - // Use Foundation to run `tar`. The exact details of using the Foundation + let outputPath = context.pluginWorkDirectory.appending("\(archiveName).zip") + + // Use Foundation to run `zip`. The exact details of using the Foundation // API aren't relevant; the point is that the built artifacts can be used // by the script. - let tarTool = try context.tool(named: "tar") - let tarArgs = ["-czf", outputPath.string] + builtExecutables.map{ $0.path.string } - let process = Process.run(URL(fileURLWithPath: tarTool.path.string), arguments: tarArgs) + let zipTool = try context.tool(named: "zip") + let zipArgs = ["-j", outputPath.string] + builtExecutables.map{ $0.path.string } + let zipToolURL = URL(fileURLWithPath: zipTool.path.string) + let process = try Process.run(zipToolURL, arguments: zipArgs) process.waitUntilExit() - - // The plugin should also report errors from the creation of the archive. - - print("Created archive at \(outputPath).") + + // Check whether the subprocess invocation was successful. + if process.terminationReason == .exit && process.terminationStatus == 0 { + print("Created distribution archive at \(outputPath).") + } + else { + let problem = "\(process.terminationReason):\(process.terminationStatus)" + Diagnostics.error("zip invocation failed: \(problem)") + } } } ``` @@ -790,6 +795,8 @@ Since SwiftPM currently has only a single-layered package dependency graph, it i Once this is possible, a future direction might be to have a command plugin use SwiftArgumentParser to declare a supported set of input parameters. This could allow SwiftPM (or possibly an IDE) to present an interface for those plugin options — IDEs, in particular, could construct user interfaces for well-defined options (possibly in the manner of the archaic MPW `Commando` tool). +Another direction might be for the PackagePlugin API to define its own facility for a plugin to declare externally visible properties. This might include considerations particular to plugins, such as whether or not a particular path property is intended to be writable (requiring permission from the user before the plugin runs). As with SwiftArgumentParser, a natural approach would be to declare such properties on the type that implements the plugin, with their values having been set by the plugin host at the time the plugin is invoked. + ### Additional access to Package Manager services The API in the `PackageManager` type that this proposal defines is just a start. The idea is to, over time, offer plugins a variety of functionality and derivable information that they can request and then further process. @@ -801,3 +808,7 @@ Extending the `PackageManager` API does need to be done in a way that is possibl ### Providing access to build and test progress and structured results The initial proposed API for having plugins run builds and tests is fairly minimal. In particular, the build log is returned at the end of the build as a single text string, and the plugin has no way to cancel the build. Future proposals should extend this, ideally to the point at which `swift` `build` and `swift` `test` could themselves be implemented using the same API as for custom commands. + +### Allowing a plugin to report progress + +While a plugin can emit diagnostics using the `Diagnostics` type, there is currently no way for a plugin to report progress while it is running. This would be very useful for long-running plugins, and should be addressed in a future proposal. diff --git a/proposals/0333-with-memory-rebound.md b/proposals/0333-with-memory-rebound.md index 688fc3a778..850529c3a5 100644 --- a/proposals/0333-with-memory-rebound.md +++ b/proposals/0333-with-memory-rebound.md @@ -3,21 +3,19 @@ * Proposal: [SE-0333](0333-with-memory-rebound.md) * Authors: [Guillaume Lessard](https://github.com/glessard), [Andrew Trick](https://github.com/atrick) * Review Manager: [Ben Cohen](https://github.com/airspeedswift) -* Status: **Active review (November 30 - December 9, 2021)** -* Implementation: [draft pull request][draft-pr] +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Acceptance](https://forums.swift.org/t/54699) +* Implementation: [apple/swift#39529](https://github.com/apple/swift/pull/39529) * Bugs: [SR-11082](https://bugs.swift.org/browse/SR-11082), [SR-11087](https://bugs.swift.org/browse/SR-11087) -[draft-pr]: https://github.com/apple/swift/pull/39529 -[pitch-thread]: https://forums.swift.org/t/pitch-expand-usability-of-withmemoryrebound/52500 - ## Introduction The function `withMemoryRebound(to:capacity:_ body:)` executes a closure while temporarily binding a range of memory to a different type than the callee is bound to. We propose to lift some notable limitations of `withMemoryRebound` and enable rebinding to a larger set of types, -as well as rebinding from raw memory pointers and buffers. +as well as rebinding the memory pointed to by raw memory pointers and buffers. -Swift-evolution thread: [Pitch thread][pitch-thread] +Swift-evolution threads: [Pitch thread](https://forums.swift.org/t/52500), [Review thread](https://forums.swift.org/t/53799) ## Motivation @@ -36,33 +34,19 @@ func withMemoryRebound( ) rethrows -> Result ``` -This function is currently more limited than necessary. -It requires that the stride of `Pointee` and `T` be equal. -This requirement makes many legitimate use cases technically illegal, +In its current incarnation, this function is more limited than necessary. +It requires that the stride of `Pointee` and `T` be equal, +and that requirement makes many legitimate use cases technically illegal, even though they could be supported by the compiler. -We propose to allow temporarily binding to a type `T` whose stride is -a whole fraction or whole multiple of `Pointee`'s stride, -when the starting address is properly aligned for type `T`. -As before, `T`'s memory layout must be compatible with that of`Pointee`. - - - -For example, suppose that a buffer of `Double` consisting of a series of (x,y) pairs is returned from data analysis code written in C. +We propose to expand and better define the rules by which the function can be used, +including to allow temporarily binding to a type `T` that is a homogeneous aggregate of `Pointee`, +or a type `T` of which `Pointee` is a homogeneous aggregate. +For instance, the tuple `(Int, Int, Int)` is a homogeneous aggregate. + +As an example of rebinding, suppose that a buffer of `Double` consisting of a series of (x,y) pairs is returned from data analysis code written in C. The next step might be to display it in a preview graph, which needs to read `CGPoint` values. -We need to copy the `Double` values as pairs to values of type `CGPoint` (when executing on a 64-bit platform): +We need to copy pairs of `Double` values to values of type `CGPoint` (when executing on a 64-bit platform): ```swift var count = 0 @@ -81,7 +65,7 @@ var points = Array(unsafeUninitializedCapacity: count/2) { We could do better with an improved version of `withMemoryRebound`. Since `CGPoint` values consist of a pair of `CGFloat` values, -and `CGFloat` values are themselves layout-compatible with `Double` (when executing on a 64-bit platform): +and `CGFloat` values are themselves layout-equivalent with `Double` (when executing on a 64-bit platform): ```swift var points = Array(unsafeUninitializedCapacity: data.count/2) { buffer, initializedCount in @@ -128,15 +112,44 @@ var points = Array(unsafeUninitializedCapacity: data.count/MemoryLayout ## Proposed solution -We propose to lift the restriction that the strides of `T` and `Pointee` must be equal. -This means that it will now be considered correct to re-bind from a homogeneous aggregate type to the type of its constitutive elements, -as they are layout compatible, even though their stride is different. +`withMemoryRebound` is currently defined for `UnsafePointer`, `UnsafeMutablePointer`, +`UnsafeBufferPointer` and `UnsafeMutableBufferPointer`. +The type to which the memory is bound by the `Pointer` types is called `Pointee`, +while it is `Element` for the `BufferPointer` types. +For simplicity the following discussion calls both `Pointee`. + +In the general case, the runtime performs housekeeping tasks when initializing, deinitializing or updating a value of a type. +Initializing and deinitialization of a type that is or stores a reference type means that type-specific code is executed, +and therefore in general data cannot be accessed as another type. + +`withMemoryRebound` can be used safely with pairs of types `Pointee` and `T` that do _not_ require initialization or deinitialization. +These types do not yet have a formal name in Swift, +but are referred to as "trivial" types in some API documentation. + +In order to safely use `withMemoryRebound`, the current rule +is that the destination type, `T`, must be _layout equivalent_ with `Pointee`. +To this we add that, as an alternative, `T` can be a homogeneous aggregate of `Pointee`, or `Pointee` can be a homogeneous aggregate of `T`. + +Two types A and B are layout equivalent when they are, for example: +- identical types; +- one is a typealias for the other; +- trivial scalar types with the same size and alignment, such as floating-point, integer and pointer types; +- one is a class type, and the other is one of its superclass types, or `AnyObject`; +- optional references whose underlying types are layout equivalent; +- pointer types, such as `UnsafePointer` and `OpaquePointer`; +- optional pointer types, such as `UnsafePointer?` and `UnsafeRawPointer?`; +- one is a struct with a single stored property, the other is the type of its stored property; + +Homogeneous aggregate types (tuples, array storage, and frozen structs) are layout equivalent if they have the same number of layout-equivalent elements. + ### Instance methods of `UnsafePointer` and `UnsafeMutablePointer` -We propose to lift the restriction that the strides of `T` and `Pointee` must be equal, when calling `withMemoryRebound`. +We propose to lift the restriction that the strides of `T` and `Pointee` must be equal when calling `withMemoryRebound`. +`T` and `Pointee` must either be layout equivalent (see above,) +or one must be a homogeneous aggregate of the other. The function declarations remain the same on these two types, -though given the relaxed restriction, +though given the updated rules, we must clarify the meaning of the `capacity` argument. `capacity` shall mean the number of strides of elements of the temporary type (`T`) to be temporarily bound. The documentation will be updated to reflect the changed behaviour. @@ -165,6 +178,8 @@ extension UnsafeMutablePointer { We propose adding a `withMemoryRebound` method, which currently does not exist on these types. Since it operates on raw memory, this version of `withMemoryRebound` places no restriction on the temporary type (`T`). It is therefore up to the program author to ensure type safety when using these methods. +When applied to memory that is initialized but viewed as raw memory, +the relation between the initialized type and `T` must be valid under the `UnsafePointer.withMemoryRebound` rules. As in the `UnsafePointer` case, `capacity` means the number of strides of elements of the temporary type (`T`) to be temporarily bound. ```swift @@ -187,9 +202,11 @@ extension UnsafeMutableRawPointer { ### Instance methods of `UnsafeBufferPointer` and `UnsafeMutableBufferPointer` -We propose to lift the restriction that the strides of `T` and `Pointee` must be equal, when calling `withMemoryRebound`. +We propose to lift the restriction that the strides of `T` and `Element` must be equal when calling `withMemoryRebound`. +`T` and `Element` must either be layout equivalent (see above,) +or one must be a homogeneous aggregate of the other. The function declarations remain the same on these two types. -The capacity of the buffer to the temporary type will be calculated using the length of the `UnsafeBufferPointer` and the stride of the temporary type. +The capacity of the buffer to the temporary type will be calculated using the capacity of the `UnsafeBufferPointer` and the stride of the temporary type. The documentation will be updated to reflect the changed behaviour. We will add parameter labels to the closure type declaration to benefit code completion (a source compatible change.) @@ -214,9 +231,11 @@ extension UnsafeMutableBufferPointer { We propose adding a `withMemoryRebound` method, which currently does not exist on these types. Since it operates on raw memory, this version of `withMemoryRebound` places no restriction on the temporary type (`T`). It is therefore up to the program author to ensure type safety when using these methods. -The capacity of the buffer to the temporary type will be calculated using the length of the `UnsafeRawBufferPointer` and the stride of the temporary type. +When applied to memory that is initialized but viewed as raw memory, +the relation between the initialized type and `T` must be valid under the `UnsafePointer.withMemoryRebound` rules. +The capacity of the buffer to the temporary type will be calculated using the capacity of the `UnsafeRawBufferPointer` and the stride of the temporary type. -Finally the set, we propose to add an `assumingMemoryBound` function that calculates the capacity of the returned `UnsafeBufferPointer`. +To complete the set, we propose to add an `assumingMemoryBound` function that calculates the capacity of the returned `UnsafeBufferPointer`. ```swift extension UnsafeRawBufferPointer { @@ -241,8 +260,6 @@ extension UnsafeMutableRawBufferPointer { ## Detailed design -Note: please see the [draft PR][draft-pr] to visualize the proposed changes rather than the proposed final state. - ```swift extension UnsafePointer { /// Executes the given closure while temporarily binding memory to @@ -282,21 +299,21 @@ extension UnsafePointer { /// After executing `body`, this method rebinds memory back to the original /// `Pointee` type. /// - /// - Note: Only use this method to rebind the pointer's memory to a type - /// that is layout compatible with the `Pointee` type. The stride of the + /// - Note: Only use this method to rebind the pointer's memory to a type `T` + /// that is layout equivalent with the `Pointee` type, or a type `T` that + /// is an aggregate of `Pointee` instances, or a type `T` such that `Pointee` + /// is an aggregate of `T` instances. As such, the stride of the /// temporary type (`T`) may be an integer multiple or a whole fraction - /// of `Pointee`'s stride, for example to point to one element of - /// an aggregate. + /// of `Pointee`'s stride. /// To bind a region of memory to a type that does not match these - /// requirements, convert the pointer to a raw pointer and use the - /// `bindMemory(to:)` method. + /// requirements, convert the pointer to a raw pointer and use its + /// `withMemoryRebound(to:)` method. /// If `T` and `Pointee` have different alignments, this pointer /// must be aligned with the larger of the two alignments. /// /// - Parameters: /// - type: The type to temporarily bind the memory referenced by this - /// pointer. The type `T` must be layout compatible - /// with the pointer's `Pointee` type. + /// pointer. This pointer must be correctly aligned for `type`. /// - count: The number of instances of `T` in the re-bound region. /// - body: A closure that takes a typed pointer to the /// same memory as this pointer, only bound to type `T`. The closure's @@ -351,21 +368,21 @@ extension UnsafeMutablePointer { /// After executing `body`, this method rebinds memory back to the original /// `Pointee` type. /// - /// - Note: Only use this method to rebind the pointer's memory to a type - /// that is layout compatible with the `Pointee` type. The stride of the + /// - Note: Only use this method to rebind the pointer's memory to a type `T` + /// that is layout equivalent with the `Pointee` type, or a type `T` that + /// is an aggregate of `Pointee` instances, or a type `T` such that `Pointee` + /// is an aggregate of `T` instances. As such, the stride of the /// temporary type (`T`) may be an integer multiple or a whole fraction - /// of `Pointee`'s stride, for example to point to one element of - /// an aggregate. + /// of `Pointee`'s stride. /// To bind a region of memory to a type that does not match these - /// requirements, convert the pointer to a raw pointer and use the - /// `bindMemory(to:)` method. + /// requirements, convert the pointer to a raw pointer and use its + /// `withMemoryRebound(to:)` method. /// If `T` and `Pointee` have different alignments, this pointer /// must be aligned with the larger of the two alignments. /// /// - Parameters: /// - type: The type to temporarily bind the memory referenced by this - /// pointer. The type `T` must be layout compatible - /// with the pointer's `Pointee` type. + /// pointer. This pointer must be correctly aligned for `type`. /// - count: The number of instances of `T` in the re-bound region. /// - body: A closure that takes a mutable typed pointer to the /// same memory as this pointer, only bound to type `T`. The closure's @@ -415,20 +432,21 @@ extension UnsafeBufferPointer { /// After executing `body`, this method rebinds memory back to the original /// `Element` type. /// - /// - Note: Only use this method to rebind the buffer's memory to a type - /// that is layout compatible with the currently bound `Element` type. - /// The stride of the temporary type (`T`) may be an integer multiple - /// or a whole fraction of `Element`'s stride. + /// - Note: Only use this method to rebind the pointer's memory to a type `T` + /// that is layout equivalent with the `Element` type, or a type `T` that + /// is an aggregate of `Element` instances, or a type `T` such that `Element` + /// is an aggregate of `T` instances. As such, the stride of the + /// temporary type (`T`) may be an integer multiple or a whole fraction + /// of `Element`'s stride. /// To bind a region of memory to a type that does not match these - /// requirements, convert the buffer to a raw buffer and use the - /// `bindMemory(to:)` method. + /// requirements, convert the pointer to a raw buffer and use its + /// `withMemoryRebound(to:)` method. /// If `T` and `Element` have different alignments, this buffer's /// `baseAddress` must be aligned with the larger of the two alignments. /// /// - Parameters: - /// - type: The type to temporarily bind the memory referenced by this - /// buffer. The type `T` must be layout compatible - /// with the pointer's `Element` type. + /// - type: The type to temporarily bind the memory referenced by this pointer. + /// This buffer's `baseAddress` must be correctly aligned for `type`. /// - body: A closure that takes a typed buffer to the /// same memory as this buffer, only bound to type `T`. The buffer /// parameter contains a number of complete instances of `T` based @@ -480,20 +498,21 @@ extension UnsafeMutableBufferPointer { /// After executing `body`, this method rebinds memory back to the original /// `Element` type. /// - /// - Note: Only use this method to rebind the buffer's memory to a type - /// that is layout compatible with the currently bound `Element` type. - /// The stride of the temporary type (`T`) may be an integer multiple - /// or a whole fraction of `Element`'s stride. + /// - Note: Only use this method to rebind the pointer's memory to a type `T` + /// that is layout equivalent with the `Element` type, or a type `T` that + /// is an aggregate of `Element` instances, or a type `T` such that `Element` + /// is an aggregate of `T` instances. As such, the stride of the + /// temporary type (`T`) may be an integer multiple or a whole fraction + /// of `Element`'s stride. /// To bind a region of memory to a type that does not match these - /// requirements, convert the buffer to a raw buffer and use the - /// `bindMemory(to:)` method. + /// requirements, convert the pointer to a raw buffer and use its + /// `withMemoryRebound(to:)` method. /// If `T` and `Element` have different alignments, this buffer's /// `baseAddress` must be aligned with the larger of the two alignments. /// /// - Parameters: - /// - type: The type to temporarily bind the memory referenced by this - /// buffer. The type `T` must be layout compatible - /// with the pointer's `Element` type. + /// - type: The type to temporarily bind the memory referenced by this pointer. + /// This buffer's `baseAddress` must be correctly aligned for `type`. /// - body: A closure that takes a mutable typed buffer to the /// same memory as this buffer, only bound to type `T`. The buffer /// parameter contains a number of complete instances of `T` based @@ -548,14 +567,15 @@ extension UnsafeRawPointer { /// must equal zero. /// /// - Note: The region of memory starting at this pointer may have been - /// bound to a type. If that is the case, then `T` must be - /// layout compatible with the type to which the memory has been bound. + /// bound to a type (the prebound type). If that is the case, then `T` must be + /// layout equivalent with the prebound type, or `T` must be an aggregate of + /// the prebound type, or the the prebound type is an aggregate of `T`. /// This requirement does not apply if the region of memory /// has not been bound to any type. /// /// - Parameters: /// - type: The type to temporarily bind the memory referenced by this - /// pointer. This pointer must be a multiple of this type's alignment. + /// pointer. This pointer must be correctly aligned for `type`. /// - count: The number of instances of `T` in the re-bound region. /// - body: A closure that takes a typed pointer to the /// same memory as this pointer, only bound to type `T`. The closure's @@ -607,14 +627,15 @@ extension UnsafeMutableRawPointer { /// must equal zero. /// /// - Note: The region of memory starting at this pointer may have been - /// bound to a type. If that is the case, then `T` must be - /// layout compatible with the type to which the memory has been bound. + /// bound to a type (the prebound type). If that is the case, then `T` must be + /// layout equivalent with the prebound type, or `T` must be an aggregate of + /// the prebound type, or the the prebound type is an aggregate of `T`. /// This requirement does not apply if the region of memory /// has not been bound to any type. /// /// - Parameters: /// - type: The type to temporarily bind the memory referenced by this - /// pointer. This pointer must be a multiple of this type's alignment. + /// pointer. This pointer must be correctly aligned for `type`. /// - count: The number of instances of `T` in the re-bound region. /// - body: A closure that takes a typed pointer to the /// same memory as this pointer, only bound to type `T`. The closure's @@ -662,14 +683,16 @@ extension UnsafeRawBufferPointer { /// must equal zero. /// /// - Note: A raw buffer may represent memory that has been bound to a type. - /// If that is the case, then `T` must be layout compatible with the - /// type to which the memory has been bound. This requirement does not - /// apply if the raw buffer represents memory that has not been bound - /// to any type. + //// (the prebound type). If that is the case, then `T` must be + /// layout equivalent with the prebound type, or `T` must be an aggregate of + /// the prebound type, or the the prebound type is an aggregate of `T`. + /// This requirement does not apply if the region of memory + /// has not been bound to any type. /// /// - Parameters: /// - type: The type to temporarily bind the memory referenced by this - /// pointer. This pointer must be a multiple of this type's alignment. + /// pointer. This buffer's `baseAddress` must be correctly aligned + /// for `type`. /// - body: A closure that takes a typed pointer to the /// same memory as this pointer, only bound to type `T`. The closure's /// pointer argument is valid only for the duration of the closure's @@ -735,14 +758,16 @@ extension UnsafeMutableRawBufferPointer { /// must equal zero. /// /// - Note: A raw buffer may represent memory that has been bound to a type. - /// If that is the case, then `T` must be layout compatible with the - /// type to which the memory has been bound. This requirement does not - /// apply if the raw buffer represents memory that has not been bound - /// to any type. + //// (the prebound type). If that is the case, then `T` must be + /// layout equivalent with the prebound type, or `T` must be an aggregate of + /// the prebound type, or the the prebound type is an aggregate of `T`. + /// This requirement does not apply if the region of memory + /// has not been bound to any type. /// /// - Parameters: /// - type: The type to temporarily bind the memory referenced by this - /// pointer. This pointer must be a multiple of this type's alignment. + /// pointer. This buffer's `baseAddress` must be correctly aligned + /// for `type`. /// - body: A closure that takes a typed pointer to the /// same memory as this pointer, only bound to type `T`. The closure's /// pointer argument is valid only for the duration of the closure's diff --git a/proposals/0334-pointer-usability-improvements.md b/proposals/0334-pointer-usability-improvements.md index ad958c40b4..ffae2587ad 100644 --- a/proposals/0334-pointer-usability-improvements.md +++ b/proposals/0334-pointer-usability-improvements.md @@ -3,7 +3,8 @@ * Proposal: [SE-0334](0334-pointer-usability-improvements.md) * Authors: [Guillaume Lessard](https://github.com/glessard), [Andrew Trick](https://github.com/atrick) * Review Manager: [Ben Cohen](https://github.com/airspeedswift) -* Status: **Active review (November 30 - December 9, 2021)** +* Status: **Implemented (Swift 5.7)** +* Decision notes: [Acceptance](https://forums.swift.org/t/54700) * Implementation: [Draft pull request][draft-pr] * Bugs: [rdar://64342031](rdar://64342031), [SR-11156](https://bugs.swift.org/browse/SR-11156) ([rdar://53272880](rdar://53272880)), [rdar://22541346](rdar://22541346) @@ -18,7 +19,7 @@ This proposal introduces some quality-of-life improvements for `UnsafePointer` a 2. Add an API to obtain a pointer to a stored property of an aggregate `T`, given an `UnsafePointer`. 3. Add the ability to compare pointers of any two types. -Swift-evolution thread: [Discussion][pitch-thread] +Swift-evolution threads: [Discussion][pitch-thread], [Review](https://forums.swift.org/t/53800) ## Motivation @@ -87,12 +88,12 @@ Calculating the offset between the start of the data structure to the field of t We propose to add a function to help perform this operation on raw pointer types: ```swift extension UnsafeRawPointer { - public func alignedUp(for: T.type) -> Self + public func alignedUp(for: T.Type) -> Self } ``` This function will round the current pointer up to the next address properly aligned to access an instance of `T`. -When applied to a `self` already aligned for `T`, `UnsafeRawPointer.aligned(for:)` will return `self`. +When applied to a `self` already aligned for `T`, `UnsafeRawPointer.alignedUp(for:)` will return `self`. The new function would make identifying the storage location of `T` much more straightforward than in the example above: ```swift @@ -192,6 +193,7 @@ To remedy this, we propose to add the following static functions, scoped to the ```swift extension _Pointer { public static func == (lhs: Self, rhs: Other) -> Bool + public static func != (lhs: Self, rhs: Other) -> Bool public static func < (lhs: Self, rhs: Other) -> Bool public static func <= (lhs: Self, rhs: Other) -> Bool @@ -274,7 +276,7 @@ extension UnsafeMutableRawPointer { /// - Parameters: /// - type: the type to be stored at the returned address. /// - Returns: a pointer properly aligned to store a value of type `T`. - public func alingedDown(for type: T.Type) -> UnsafeMutableRawPointer + public func alignedDown(for type: T.Type) -> UnsafeMutableRawPointer /// Obtain the next pointer whose bit pattern is a multiple of `alignment`. /// @@ -348,6 +350,26 @@ extension UnsafeMutablePointer { #### Allow comparisons of pointers of any type ```swift + /// Returns a Boolean value indicating whether two pointers represent + /// the same memory address. + /// + /// - Parameters: + /// - lhs: A pointer. + /// - rhs: Another pointer. + /// - Returns: `true` if `lhs` and `rhs` reference the same memory address; + /// otherwise, `false`. + public static func == (lhs: Self, rhs: Other) -> Bool + + /// Returns a Boolean value indicating whether two pointers represent + /// different memory addresses. + /// + /// - Parameters: + /// - lhs: A pointer. + /// - rhs: Another pointer. + /// - Returns: `true` if `lhs` and `rhs` reference different memory addresses; + /// otherwise, `false`. + public static func != (lhs: Self, rhs: Other) -> Bool + /// Returns a Boolean value indicating whether the first pointer references /// a memory location earlier than the second pointer references. /// diff --git a/proposals/0335-existential-any.md b/proposals/0335-existential-any.md new file mode 100644 index 0000000000..eaf480380e --- /dev/null +++ b/proposals/0335-existential-any.md @@ -0,0 +1,303 @@ +# Introduce existential `any` + +* Proposal: [SE-0335](0335-existential-any.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.6)** +* Upcoming Feature Flag: `ExistentialAny` (implemented in Swift 5.8) +* Implementation: [apple/swift#40282](https://github.com/apple/swift/pull/40282) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0335-introduce-existential-any/54504) + +## Contents + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [Grammar of explicit existential types](#grammar-of-explicit-existential-types) + - [Semantics of explicit existential types](#semantics-of-explicit-existential-types) + - [`Any` and `AnyObject`](#any-and-anyobject) + - [Metatypes](#metatypes) + - [Type aliases and associated types](#type-aliases-and-associated-types) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Alternatives considered](#alternatives-considered) + - [Rename `Any` and `AnyObject`](#rename-any-and-anyobject) + - [Use `Any

` instead of `any P`](#use-anyp-instead-of-any-p) + - [Future Directions](#future-directions) + - [Extending existential types](#extending-existential-types) + - [Re-purposing the plain protocol name](#re-purposing-the-plain-protocol-name) + - [Revisions](#revisions) + - [Changes from the pitch discussion](#changes-from-the-pitch-discussion) + - [Acknowledgments](#acknowledgments) + +## Introduction + +Existential types in Swift have an extremely lightweight spelling: a plain protocol name in type context means an existential type. Over the years, this has risen to the level of **active harm** by causing confusion, leading programmers down the wrong path that often requires them to re-write code once they hit a fundamental [limitation of value-level abstraction](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--limits-of-existentials). This proposal makes the impact of existential types explicit in the language by annotating such types with `any`. + +Swift evolution discussion thread: [[Pitch] Introduce existential `any`](https://forums.swift.org/t/pitch-introduce-existential-any/53520). + +## Motivation + +Existential types in Swift have significant limitations and performance implications. Some of their limitations are missing language features, but many are fundamental to their type-erasing semantics. For example, given a protocol with associated type requirements, the existential type cannot conform to the protocol itself without a manual conformance implementation, because there is not an obvious concrete associated type that works for any value conforming to the protocol, as shown by the following example: + +```swift +protocol P { + associatedtype A + func test(a: A) +} + +func generic(p: ConcreteP, value: ConcreteP.A) { + p.test(a: value) +} + +func useExistential(p: P) { + generic(p: p, value: ???) // what type of value would P.A be?? +} +``` + +Existential types are also significantly more expensive than using concrete types. Because they can store any value whose type conforms to the protocol, and the type of value stored can change dynamically, existential types require dynamic memory unless the value is small enough to fit within an inline 3-word buffer. In addition to heap allocation and reference counting, code using existential types incurs pointer indirection and dynamic method dispatch that cannot be optimized away. + +Despite these significant and often undesirable implications, existential types have a minimal spelling. Syntactically, the cost of using one is hidden, and the similar spelling to generic constraints has caused many programmers to confuse existential types with generics. In reality, the need for the dynamism they provided is relatively rare compared to the need for generics, but the language makes existential types too easy to reach for, especially by mistake. The cost of using existential types should not be hidden, and programmers should explicitly opt into these semantics. + +## Proposed solution + +I propose to make existential types syntactically explicit in the language using the `any` keyword. This proposal introduces the new syntax in the Swift 5 language mode, and this syntax should be required for existential types under a future language mode. + +In Swift 5, anywhere that an existential type can be used today, the `any` keyword can be used to explicitly denote an existential type: + +```swift +// Swift 5 mode + +protocol P {} +protocol Q {} +struct S: P, Q {} + +let p1: P = S() // 'P' in this context is an existential type +let p2: any P = S() // 'any P' is an explicit existential type + +let pq1: P & Q = S() // 'P & Q' in this context is an existential type +let pq2: any P & Q = S() // 'any P & Q' is an explicit existential type +``` + +In a future language mode, existential types are required to be explicitly spelled with `any`: + +```swift +// Future language mode + +protocol P {} +protocol Q {} +struct S: P, Q {} + +let p1: P = S() // error +let p2: any P = S() // okay + +let pq1: P & Q = S() // error +let pq2: any P & Q = S() // okay +``` + +This behavior can be enabled in earlier language modes with the [upcoming feature flag](0362-piecemeal-future-features.md) `ExistentialAny`. + +## Detailed design + +### Grammar of explicit existential types + +This proposal adds the following production rules to the grammar of types: + +``` +type -> existential-type + +existential-type -> 'any' type +``` + +### Semantics of explicit existential types + +The semantics of `any` types are the same as existential types today. Explicit `any` can only be applied to protocols and protocol compositions, or metatypes thereof; `any` cannot be applied to nominal types, structural types, type parameters, and protocol metatypes: + +```swift +struct S {} + +let s: any S = S() // error: 'any' has no effect on concrete type 'S' + +func generic(t: T) { + let x: any T = t // error: 'any' has no effect on type parameter 'T' +} + +let f: any ((Int) -> Void) = generic // error: 'any' has no effect on concrete type '(Int) -> Void' +``` + +#### `Any` and `AnyObject` + +`any` is unnecessary for `Any` and `AnyObject` (unless part of a protocol composition): + +```swift +struct S {} +class C {} + +let value: any Any = S() +let values: [any Any] = [] +let object: any AnyObject = C() + +protocol P {} +extension C: P {} + +let pObject: any AnyObject & P = C() // okay +``` + +> **Rationale**: `any Any` and `any AnyObject` are redundant. `Any` and `AnyObject` are already special types in the language, and their existence isn’t nearly as harmful as existential types for regular protocols because the type-erasing semantics is already explicit in the name. + +#### Metatypes + +The existential metatype, i.e. `P.Type`, becomes `any P.Type`. The protocol metatype, i.e. `P.Protocol`, becomes `(any P).Type`. The protocol metatype value `P.self` becomes `(any P).self`: + +```swift +protocol P {} +struct S: P {} + +let existentialMetatype: any P.Type = S.self + +protocol Q {} +extension S: Q {} + +let compositionMetatype: any (P & Q).Type = S.self + +let protocolMetatype: (any P).Type = (any P).self +``` + +> **Rationale**: The existential metatype is spelled `any P.Type` because it's an existential type that is a generalization over metatypes. The protocol metatype is the singleton metatype of the existential type `any P` itself, which is naturally spelled `(any P).Type`. + +Under this model, the `any` keyword conceptually acts like an existential quantifier `∃ T`. Formally, `any P.Type` means `∃ T:P . T.Type`, i.e. for some concrete type `T` conforming to `P`, this is the metatype of that concrete type.`(any P).Type` is formally `(∃ T:P . T).Type`, i.e. the metatype of the existential type itself. + +The distinction between `any P.Type` and `(any P).Type` is syntactically very subtle. However, `(any P).Type` is rarely useful in practice, and it's helpful to explain why, given a generic context where a type parameter `T` is substituted with an existential type, `T.Type` is the singleton protocol metatype. + +##### Metatypes for `Any` and `AnyObject` + +Like their base types, `Any.Type` and `AnyObject.Type` remain valid existential metatypes; writing `any` on these metatypes in unnecessary. The protocol metatypes for `Any` and `AnyObject` are spelled `(any Any).Type` and `(any AnyObject).Type`, respectively. + +#### Type aliases and associated types + +Like plain protocol names, a type alias to a protocol `P` can be used as both a generic constraint and an existential type. Because `any` is explicitly an existential type, a type alias to `any P` can only be used as an existential type, it cannot be used as a generic conformance constraint, and `any` does not need to be written at the use-site: + +```swift +protocol P {} +typealias AnotherP = P +typealias AnyP = any P + +struct S: P {} + +let p2: any AnotherP = S() +let p1: AnyP = S() + +func generic(value: T) { ... } +func generic(value: T) { ... } // error +``` + +Once the `any` spelling is required under a future language mode, a type alias to a plain protocol name is not a valid type witness for an associated type requirement; existential type witnesses must be explicit in the `typealias` with `any`: + +```swift +protocol P {} + +protocol Requirements { + associatedtype A +} + +struct S1: Requirements { + typealias A = P // error: associated type requirement cannot be satisfied with a protocol +} + +struct S2: Requirements { + typealias A = any P // okay +} +``` + +## Source compatibility + +Enforcing that existential types use the `any` keyword will require a source change. To ease the migration, I propose to start allowing existential types to be spelled with `any` with the Swift 5.6 compiler, and require existential types to be spelled with `any` under a future language mode. The old existential type syntax will continue to be supported under the Swift 5 language mode, and the transition to the new syntax is mechanical, so it can be performed automatically by a migrator. + +[SE-0309 Unlock existentials for all protocols](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md) enables more code to be written using existential types. To minimize the amount of new code written that will become invalid under `ExistentialAny`, I propose requiring `any` immediately for protocols with `Self` and associated type requirements. This introduces an inconsistency for protocols under the Swift 5 language mode, but this inconsistency already exists today (because you cannot use certain protocols as existential types at all), and the syntax difference serves two purposes: + +1. It saves programmers time in the long run by preventing them from writing new code that will become invalid later. +2. It communicates the existence of `any` and encourages programmers to start using it for other existential types before adopting `ExistentialAny`. + +### Transitioning to `any` + +The new `any` syntax will be staged in over several major Swift releases. In the release where `any` is introduced, the compiler will not emit diagnostics for the lack of `any` on existential types, save for the aforementioned cases. After `any` is introduced, warnings will be added to guide programmers toward the new syntax. Finally, a missing `any` will become an unconditional error, or [plain protocol names may be repurposed](#re-purposing-the-plain-protocol-name) — in Swift 6 or a later language mode. + +## Effect on ABI stability + +None. + +## Effect on API resilience + +None. + +## Alternatives considered + +### Rename `Any` and `AnyObject` + +Instead of leaving `Any` and `AnyObject` in their existing spelling, an alternative is to spell these types as `any Value` and `any Object`, respectively. Though this is more consistent with the rest of the proposal, this change would have an even bigger source compatibility impact. Given that `Any` and `AnyObject` aren’t as harmful as other existential types, changing the spelling isn’t worth the churn. + +### Use `Any

` instead of `any P` + +A common suggestion is to spell existential types with angle brackets on `Any`, e.g. `Any`. However, an important aspect of the proposed design is that `any` has symmetry with `some`, where both keywords can be applied to protocol constraints. This symmetry is important for helping programmers understand and remember the syntax, and for future extensions of the `some` and `any` syntax. Opaque types and existential types would both greatly benefit from being able to specify constraints on associated types. This could naturally be done in angle brackets, e.g. `some Sequence` and `any Sequence`, or `some Sequence<.Element == Int>` and `any Sequence<.Element == Int>`. + +Using the same syntax between opaque types and exsitential types also makes it very easy to replace `any` with `some`, and it is indeed the case that many uses of existential types today could be replaced with opaque types instead. + +Finally, the `Any

` syntax is misleading because it appears that `Any` is a generic type, which is confusing to the mental model for 2 reasons: + +1. A generic type is something programmers can implement themselves. In reality, existential types are a built-in language feature that would be _very_ difficult to replicate with regular Swift code. +2. This syntax creates the misconception that the underlying concrete type is a generic argument to `Any` that is preserved statically in the existential type. The `P` in `Any

` looks like an implicit type parameter with a conformance requirement, but it's not; the underlying type conforming to `P` is erased at compile-time. + +## Future Directions + +### Extending existential types + +This proposal provides an obvious syntax for extending existential types in order to manually implement protocol conformances: + +```swift +extension any Equatable: Equatable { ... } +``` + +### Re-purposing the plain protocol name + +In other places in the language, a plain protocol name is already sugar for a type parameter conforming to the protocol. Consider a normal protocol extension: + +```swift +extension Collection { ... } +``` + +This extension is a form of universal quantification; it extends all types that conform to `Collection`. This extension introduces a generic context with a type parameter ``, which means the above syntax is effectively sugar for a parameterized extension: + +```swift +extension Self where Self: Collection { ... } +``` + +Changing the syntax of existential types creates an opportunity to expand upon this sugar. If existential types are spelled explicitly with `any`, a plain protocol name could always mean sugar for a type parameter on the enclosing context with a conformance requirement to the protocol. For example, consider the declaration of `append(contentsOf:)` from the standard library: + +```swift +extension Array { + mutating func append(contentsOf newElements: S) where S.Element == Element +} +``` + +Combined with a syntax for constraining associated types in angle brackets, such as in [[Pitch] Light-weight same-type constraint syntax](https://forums.swift.org/t/pitch-light-weight-same-type-constraint-syntax/52889), the above declaration could be simplified to: + +```swift +extension Array { + mutating func append(contentsOf newElements: Sequence) +} +``` + +This sugar eliminates a lot of noise in cases where a type parameter is only referred to once in a generic signature, and it enforces a natural model of abstraction, where programmers only need to name an entity when they need to refer to it multiple times. + +## Revisions + +### Changes from the pitch discussion + +* Spell the existential metatype as `any P.Type`, and the protocol metatype as `(any P).Type`. +* Preserve `any` through type aliases. +* Allow `any` on `Any` and `AnyObject`. + +## Acknowledgments + +Thank you to Joe Groff, who originally suggested this direction and syntax in [Improving the UI of generics](https://forums.swift.org/t/improving-the-ui-of-generics/22814), and to those who advocated for this change in the recent discussion about [easing the learning curve for generics](https://forums.swift.org/t/discussion-easing-the-learning-curve-for-introducing-generic-parameters/52891). Thank you to John McCall and Slava Pestov, who helped me figure out the implementation model. diff --git a/proposals/0336-distributed-actor-isolation.md b/proposals/0336-distributed-actor-isolation.md new file mode 100644 index 0000000000..bc7b91c704 --- /dev/null +++ b/proposals/0336-distributed-actor-isolation.md @@ -0,0 +1,1822 @@ +# Distributed Actor Isolation + +* Proposal: [SE-0336](0336-distributed-actor-isolation.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [Pavel Yaskevich](https://github.com/xedin), [Doug Gregor](https://github.com/DougGregor), [Kavon Farvardin](https://github.com/kavon) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0336-distributed-actor-isolation/54726) +* Implementation: + * Partially available in [recent `main` toolchain snapshots](https://swift.org/download/#snapshots) behind the `-enable-experimental-distributed` feature flag. + * This flag also implicitly enables `-enable-experimental-concurrency`. +* Sample app: + * A sample app, showcasing how the various "pieces" work together is available here: + [https://github.com/apple/swift-sample-distributed-actors-transport](https://github.com/apple/swift-sample-distributed-actors-transport) + +## Table of Contents + +- [Distributed Actor Isolation](#distributed-actor-isolation) + - [Table of Contents](#table-of-contents) + - [Introduction](#introduction) + - [Useful links](#useful-links) + - [Motivation](#motivation) + - [Location Transparency](#location-transparency) + - [Remote and Local Distributed Actors](#remote-and-local-distributed-actors) + - [Proposed solution](#proposed-solution) + - [Distributed Actors](#distributed-actors) + - [Complete isolation of state](#complete-isolation-of-state) + - [Distributed Methods](#distributed-methods) + - [Detailed design](#detailed-design) + - [Distributed Actors and Distributed Actor Systems](#distributed-actors-and-distributed-actor-systems) + - [Distributed Actor Initializers](#distributed-actor-initializers) + - [Distributed Actors implicitly conform to Codable](#distributed-actors-implicitly-conform-to-codable) + - [Distributed Methods](#distributed-methods-1) + - [Distributed Method Serialization Requirements](#distributed-method-serialization-requirements) + - [Distributed Methods and Generics](#distributed-methods-and-generics) + - [Distributed Methods and Existential Types](#distributed-methods-and-existential-types) + - [Implicit effects on Distributed Methods](#implicit-effects-on-distributed-methods) + - [Isolation states and Implicit effects on Distributed Methods](#isolation-states-and-implicit-effects-on-distributed-methods) + - [Distributed Actor Properties](#distributed-actor-properties) + - [Stored properties](#stored-properties) + - [Computed properties](#computed-properties) + - [Protocol Conformances](#protocol-conformances) + - [The `DistributedActor` protocol and protocols inheriting from it](#the-distributedactor-protocol-and-protocols-inheriting-from-it) + - [Breaking through Location Transparency](#breaking-through-location-transparency) + - [Future Directions](#future-directions) + - [Versioning and Evolution of Distributed Actors and Methods](#versioning-and-evolution-of-distributed-actors-and-methods) + - [Evolution of parameter values only](#evolution-of-parameter-values-only) + - [Evolution of distributed methods](#evolution-of-distributed-methods) + - [Introducing the `local` keyword](#introducing-the-local-keyword) + - [Alternatives Considered](#alternatives-considered) + - [Implicitly `distributed` methods / "opt-out of distribution"](#implicitly-distributed-methods--opt-out-of-distribution) + - [Introducing "wrapper" type for `Distributed`](#introducing-wrapper-type-for-distributedsomeactor) + - [Creating only a library and/or source-generation tool](#creating-only-a-library-andor-source-generation-tool) + - [Acknowledgments & Prior Art](#acknowledgments--prior-art) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Changelog](#changelog) + +## Introduction + +With the recent introduction of [actors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md) to the language, Swift gained powerful and foundational building blocks for expressing *thread-safe* concurrent programs. This proposal is the first in a series of proposals aiming to extend Swift's actor runtime with the concept of *distributed actors*, allowing developers leverage the actor model not only in local, but also distributed settings. + +With distributed actors, we acknowledge that the world we live in is increasingly built around distributed systems, and that we should provide developers with better tools to work within those environments. We aim to simplify and push the state-of-the-art for distributed systems programming in Swift as we did with concurrent programming with local actors and Swift’s structured concurrency approach embedded in the language. + +> The distributed actor proposals will be structured similarly to how Swift Concurrency proposals were: as a series of interconnected proposals that build on top of each other. + +This proposal focuses on the extended actor isolation and type-checking aspects of distributed actors. + +#### Useful links + +Swift Evolution: + +- [Distributed Actors: Pitch #1](https://forums.swift.org/t/pitch-distributed-actors/51669) - a comprehensive, yet quite large, pitch encompassing all pieces of the distributed actor feature; It will be split out into smaller proposals going into the details of each subject, such that we can focus on, and properly review, its independent pieces step by step. + +While this pitch focuses _only_ on the actor isolation rules, we have work-in-progress transport implementations for distributed actors available as well. While they are work-in-progress and do not make use of the complete model described here, they may be useful to serve as reference for how distributed actors might be used. + +- [Swift Distributed Actors Library](https://www.swift.org/blog/distributed-actors/) - a reference implementation of a *peer-to-peer cluster* for distributed actors. Its internals depend on the work in progress language features and are dynamically changing along with these proposals. It is a realistic implementation that we can use as reference for these design discussions. +- "[Fishy Transport](https://github.com/apple/swift-sample-distributed-actors-transport)" Sample - a simplistic example transport implementation that is easier to follow the basic integration pieces than the realistic cluster implementation. Feel free to refer to it as well, while keeping in mind that it is very simplified in its implementation approach. + +## Motivation + +Distributed actors are necessary to expand Swift's actor model to distributed environments. The new `distributed` keyword offers a way for progressively disclosing the additional complexities that come with multiprocess or multi-node environments, into the local-only actor model developers are already familiar with. + +Distributed actors need stronger isolation guarantees than those that are offered by Swift's "local-only" actors. This was a conscious decision, as part of making sure actors are convenient to use in the common scenario where they are only used as concurrency isolation domains. This convenience though is too permissive for distributed programming. + +This proposal introduces the additional isolation checks necessary to allow a distributed runtime to utilize actors as its primary building block, while keeping the convenience and natural feel of such actor types. + +### Location Transparency + +The design of distributed actors intentionally does not provide facilities to easily determine whether an instance is local or remote. The programmer should not _need_ to think about where the instance is located, because Swift will make it work in either case. There are numerous benefits to embracing location transparency: + +- The programmer can write a complex distributed systems algorithm and test it locally. Running that program on a cluster becomes merely a configuration and deployment change, without any additional source code changes. +- Distributed actors can be used with multiple transports without changing the actor's implementation. +- Actor instances can be balanced between nodes once capacity of a cluster changes, or be passivated when not in use, etc. There are many more advanced patterns for allocating instances, such as the "virtual actor" style as popularized by Orleans or Akka's cluster sharding. + +Swift's take on location transparency is expressed and enforced in terms of actor isolation. The same way as actors isolate their state to protect from local race conditions, distributed actors must isolate their state because the state "might not actually be available locally" while we're dealing with a remote distributed actor reference. + +It is also possible to pass distributed actors to distributed methods, if the actor is able to conform to the serialization requirements imposed on it by the actor system. + +### Remote and Local Distributed Actors + +For the purpose of this proposal, we omit the implementation details of a remote actor reference, however as the purpose of actor isolation is to erase the observable difference between a local and remote instance (to achieve location transparency), we need to at least introduce the general concept. + +It is, by design, not possible to *statically* determine if a distributed actor instance is remote or local, therefore all programming against a distributed actor must be done as-if it was remote. This is the root reason for most of the isolation rules introduced in this proposal. For example, the following snippet illustrates location transparency in action, where in our tests we use a local instance, but in a real deployment they would be remote instances communicating: + +```swift +distributed actor TokenRange { + let range: (Token, Token) + var storage: [Token: Data] + + init(...) { ... } + + distributed func read(at loc: Token) -> Data? { + return storage[loc] + } + + distributed func write(to loc: Token, data: Data) -> Data? { + let prev = storage[loc] + storage[loc] = data + return prev + } +} +``` + +Which can be used in a local test: + +```swift +func test_distributedTokenRange() async throws {} + let range = TokenRange(...) + try await assert(range.read(at: testToken) == nil) + + try await write(to: testToken, someData) + try await assert(range.read(at: testToken) == someData) +} +``` + +Distributed functions must be marked with `try` and `await` because they imply asynchronous network calls which may fail. While the `await` rule is the same as with local-only actors, the rule about distributed methods throwing is unique to them because of the assumption that underlying transport mechanisms can fail (i.e. network or serialization errors), regardless if the called function is able to throw or not. + +Note that the even though this test is strictly local -- there are no remote actors involved here at all -- the call-sites of distributed methods have implicitly gained the async and throwing effects, which means that we must invoke them with `try await dist.` This is an important aspect of the design, as it allows us to surface any potential network issues that might occur during these calls, such as timeouts, network failures or other issues that may have caused these calls to fail. This failure is a natural consequence of the calls potentially having to cross process or network boundaries. The asynchronous effect is similar, because we might be waiting for a long time for a response to arrive, distributed calls must be potential suspension points. + +We could write the same unit-test using a distributed remote actor, and the test would remain exactly the same: + +```swift +func test_distributedTokenRange() async throws {} + // the range is actually 'remote' now + let range: TokenRange = + try await assert(range.read(at: testToken) == nil) + + try await write(to: testToken, someData) + try await assert(range.read(at: testToken) == someData) +} +``` + +During this proposal, we will be using the following phrases which have well-defined meanings, so in order to avoid confusion, let us define them explicitly up-front: + +- _distributed actor type_ - any `distributed actor` declaration, or `protocol` declaration that also conforms to `DistributedActor` because they can only be implemented by specific distributed actors, e.g. `protocol Worker: DistributedActor` as well as `distributed actor Worker`, both, can be referred to as "distributed actor type" +- _distributed actor reference_ - any variable, or parameter referring to a distributed actor instance (regardless if remote or local), +- _known-to-be-local distributed actor_, or "_distributed local actor_" for short - a specific known to be local instance of a distributed actor. A distributed actor reference can be checked at runtime if it is remote or local, but in certain situations it is also known in the type system that an actor is "definitely local" and not all isolation checks need to be applied, +- "_distributed remote actor_" - an instance of a distributed actor type, that is actually "remote" and therefore does not have any storage allocated and effectively functions like a "proxy" object. This state does not exist anywhere explicitly in the type-system explicitly, and is what we assume every distributed actor is, unless proven to be "known to be local". + +Keeping this in mind, let us proceed to discussing the specific isolation rules of distributed actors. + +## Proposed solution + +### Distributed Actors + +Distributed actors are a flavor of the `actor` type that enforces additional rules on the type and its instances in order to enable location transparency. Thanks to this, it is possible to program against a `distributed actor` without *statically* knowing if a specific instance is remote or local. All calls are made to look as-if they were remote, and in the local case simply no networking s performed and the calls execute the same as if they were a normal local-only actor. + +Distributed actors are declared by prepending `distributed` to an `actor` declaration: + +```swift +distributed actor Player { + // ... + let name: String +} +``` + +While we do not deep dive into the runtime representation in this proposal, we need to outline the general idea behind them: a `distributed actor` is used to represent an actor which may be either *local* or *remote*. + +This property of hiding away information about the location of the actual instance is called _location transparency_. Under this model, we must program against such location transparent type as-if it was remote, even when it might not be. This allows us to develop and test distributed algorithms locally, without having to resort to networking (unless we want to), vastly simplifying the testing of such systems. + +> **Note:** This is not the same as making "remote calls look like local ones" which has been a failure of many RPC systems. Instead, it is the opposite! Pessimistically assuming that all calls made cross-actor to a distributed actor may be remote, and offering specific ways to guarantee that some calls are definitely local (and thus have the usual, simpler isolation rules). + +Distributed actor isolation checks introduced by this proposal serve the purpose of enforcing the property of location transparency, and helping developers not accidentally break it. For example, the above `Player` actor could be used to represent an actor in a remote host, where the same game state is stored and references to player's devices are managed. As such, the _state_ of a distributed actor is not known locally. This brings us to the first of the additional isolation checks: properties. + +### Complete isolation of state + +Because a distributed actor, along with its actual state, may be located on a remote host, some conveniences local-only actors allow cannot be allowed for distributed ones. Let's consider the following `Player` type: + +```swift +public distributed actor Player { + public let name: String + public var score: Int +} +``` + +Such actor may be running on some remote host, meaning that if we have a "remote reference" to it we _do not_ have its state available, and any attempt to get it would involve network communication. Because of that, stored properties are not accessible across distributed actors: + +```swift +let player: Player = // ... get remote reference to Player +player.name // ❌ error: distributed actor state is only available within the actor instance +``` + +Developers should think carefully about operations that cross into the actor's isolation domain, because the cost of each operation can be very expensive (e.g., if the actor is on a machine across the internet). Properties make it very easy to accidentally make multiple round-trips: + +```swift +func example1(p: Player) async throws -> (String, Int) { + try await (p.name, p.score) // ❌ might make two slow network round-trips to `p` +} +``` + +Instead, the use of methods to perform a batched read is strongly encouraged. + +Stored properties can only be accessed when the actor is known-to-be-local, a property that is possible to check at runtime using the `whenLocal` function that we'll discuss later during this proposal. The following snippet illustrates one example of such known-to-be-local actor access, though there can be different situations where this situation occurs: + +```swift +distributed actor Counter { + var count = 0 + + func publishNextValue() { + count += 1 + Task.detached { @MainActor in + ui.countLabel.text = "Count is now \(await self.count)" + } + } +} +``` + +Stored properties cannot be declared `distributed` nor `nonisolated`. Computed properties however can be either of the two. However, computed properties can only be `distributed` if they are `get`-only due to limitations in how effectful properties work, in which case they function effectively the same as distributed methods which we'll discuss next. + +### Distributed Methods + +In order to enforce the distributed "*maybe remote*" nature of distributed actors, this proposal introduces a new flavor of method declaration called a *distributed method*. Other than a few special cases (such as `nonisolated` members), distributed methods are the only members that can be invoked cross-actor on distributed actors. + +It is necessary to give developers tight control over the distributed nature of methods they write, and it must be a conscious opt-in step. It is also possible to declared computed properties as `distributed`. A distributed method or property is defined within a distributed actor type by writing `distributed` in front of the method's declaration: + +```swift +distributed actor Player { + + distributed func yourTurn() -> Move { + return thinkOfNextMove() + } + + func thinkOfNextMove() -> Move { + // ... + } + + distributed var currentTurn: Int { + // ... + } +} +``` + +It is not possible to invoke the `thinkOfNextMove()` method cross-actor, because the target of the invocation may be remote, and it was not "exposed" for distribution using the `distributed func` keywords. This is checked at compile time and is a more restrictive form of actor-isolation checking: + +```swift +func test(p: Player) async throws { + try await p.yourTurn() + // ✅ ok, distributed func + + try await p.currentTurn + // ✅ ok, distributed computed property + + try await p.thinkOfNextMove() + // ❌ error: only 'distributed' instance methods can be called on a potentially remote distributed actor +} +``` + +Distribution must not be simply inferred from access-control, because the concept of distribution is orthogonal to access control. For example, it is very much common to have `internal distributed func` (or even `private distributed func`) declarations, which are useful for actors within a module communicating with each other (remotely), however those methods should be invoked be end-users of such library. + +Distributed methods may be subject to additional type-checking, specifically a distributed actor infers a `SerializationRequirement` from the ActorSystem it is associated with. One common serialization requirement is `Codable`. + +Such `SerializationRequirement` typealias defined on the actor system the actor is associated with causes additional type-checks to be enforced on distributed methods: all parameter types and return type of such method must be or conform to the SerializationRequirement type. This allows the compiler to fail compilation early, rather than leaving serialization crashes to the runtime, easing development and analysis of distributed actor systems: + +```swift +distributed actor Player { + typealias ActorSystem = CodableMessagingSystem + // inferred: typealias SerializationRequirement = Codable + + distributed func test(not: NotCodable) {} + // ❌ error: parameter 'not' of type 'NotCodable' in distributed instance method + // does not conform to 'Codable' +} +``` + + + +## Detailed design + +Unless otherwise specified in this proposal, the semantics of a distributed actor are the same as a regular actor, as described in [SE-0306](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md). + +### Distributed Actors + +Distributed actors can only be declared using the `distributed actor` keywords. Such types automatically conform to the `DistributedActor` protocol. The protocol is defined in the `_Distributed` module as follows: + +```swift +/// Common protocol to which all distributed actors conform. +/// +/// The `DistributedActor` protocol generalizes over all distributed actor types. +/// All distributed actor types implicitly conform to this protocol. +/// +/// It is not possible to explicitly conform to this protocol using any other declaration +/// other than a 'distributed actor', e.g. it cannot be conformed to by a plain 'actor' or 'class'. +/// +/// ### Implicit Codable conformance +/// If the 'ID' conforms to `Codable` then the concrete distributed actor adopting this protocol +/// automatically gains a synthesized Codable conformance as well. This is because the only reasonable +/// way to implement coding of a distributed actor is to encode it `ID`, and decoding can make use of +/// decoding the same ID, and resolving it using an actor system found in the Decoder's `userInfo`. +/// +/// This works well with `Codable` serialization requirements, and allows actor references to be +/// sent to other distributed actors. +protocol DistributedActor: AnyActor, Identifiable, Hashable + where ID == ActorSystem.ActorID { + + /// Type of the distributed actor system this actor is able to operate with. + /// It can be a type erased, or existential actor system (through a type-eraser wrapper type), + /// if the actor is able to work with different ones. + associatedtype ActorSystem: DistributedActorSystem + + /// The serialization requirement to apply to all distributed declarations inside the actor. + typealias SerializationRequirement = ActorSystem.SerializationRequirement + + /// Unique identity of this distributed actor, used to resolve remote references to it from other peers, + /// and also enabling the Hashable and (optional) Codable conformances of a distributed actor. + /// + /// The id may be freely shard across tasks and processes, and resolving it should return a reference + /// to the actor where it originated from. + nonisolated override var id: ID { get } + + /// Distributed Actor System responsible for managing this distributed actor. + /// + /// It is responsible for assigning and managing the actor's id, + /// as well as delivering incoming messages as distributed method invocations on the actor. + nonisolated var actorSystem: ActorSystem { get } +} +``` + +All distributed actors are *explicitly* part of some specific distributed actor system. The term "actor system" originates from both early, and current terminology relating to actor runtimes and loosely means "group of actors working together", which carries a specific meaning for distributed actors, because it implies they must be able to communicate over some (network or ipc) protocol they all understand. In Swift's local-only actor model, the system is somewhat implicit, because it simply is "the runtime", as all local objects can understand and invoke each other however they see fit. In distribution this needs to become a little more specific: there can be different network protocols and "clusters" to which actors belong, and as such, they must be explicit about their actor system use. We feel this is an expected and natural way to introduce the concept of actor systems only once we enter distribution, because previously (in local only actors) the concept would not have added much value, but in distribution it is the *core* of everything distributed actors do. + +The protocol also includes two nonisolated property requirements: `id` and `actorSystem`. Witnesses for these requirements are nonisolated computed properties that the compiler synthesizes in specific distributed actor declarations. They store the actor system the actor was created with, and its id, which is crucial to its lifecycle and messaging capabilities. We will not discuss in depth how the id is assigned in this proposal, but in short: it is created and assigned by the actor system during the actor's initialization. + +Note, that the `DistributedActor` protocol does *not* refine the `Actor` protocol, but instead it refines the `AnyActor` protocol, also introduced in this proposal. This detail is very important to upholding the soundness of distributed actor isolation. + +Sadly, just refining the Actor protocol results in the following unsound isolation behavior: + +```swift +// Illustrating isolation violation, IF 'DistributedActor' were to refine 'Actor': +extension Actor { + func f() -> SomethingSendable { ... } +} +func g(a: A) async { + print(await a.f()) +} + +// given any distributed actor: +actor MA: DistributedActor {} // : Actor implicitly (not proposed, for illustration purposes only) + +func h(ma: MA) async { + await g(ma) // 💥 would be allowed because a MA is an Actor, but can't actually work at runtime +} +``` + +The general issue here is that a distributed actor type must uphold its isolation guarantees, because the actual instance of such type may be remote, and therefore cannot be allowed to have non-distributed calls made on it. One could argue for the inverse relationship, that `Actor: DistributedActor` as the Actor is more like "`LocalActor`", however this idea also breaks down rather quickly, as one would expect "any IS-A distributed actor type, to have distributed actor isolation", however we definitely would NOT want `actor Worker {}` suddenly exhibit distributed actor isolation. In a way, this way of inheritance breaks the substitution principle in weird ways which could be hacked together to make work, but feel fragile and would lead to hard to understand isolation issues. + +In order to prevent this hole in the isolation model, we must prevent `DistributedActor` from being downcast to `Actor` and the most natural way of doing so, is introducing a shared super-type for the two Actor-like types: `AnyActor`. + +```swift +@_marker +@available(SwiftStdlib 5.6, *) +public protocol AnyActor: Sendable, AnyObject {} + +public protocol Actor: AnyActor { ... } +public protocol DistributedActor: AnyActor, ... { ... } +``` + +Thanks to this protocol we gain an understandable, and complete, type hierarchy for all actor-like behaviors, that is, types that perform a kind of isolation checking and guarantee data-race freedom to invocations on them by serializing them through an actor mailbox. This does not incur much implementation complexity in practice because functionality wise, distributed actors mirror actors exactly, however their customization of e.g. executors only applies to local instances. + +## Distributed Actor Systems + +Libraries aiming to implement distributed actor systems, and act as the runtime for distributed actors must implement the `DistributedActorSystem`. We will expand the definition of this protocol with important lifecycle functions in the runtime focused proposal, however for now let us focus on its aspects which affect type checking and isolation of distributed actors. The protocol is defined as: + +```swift +public protocol DistributedActorSystem: Sendable { + associatedtype ActorID: Hashable & Sendable // discussed below + + /// The serialization requirement that will be applied to all distributed targets used with this system. + typealias SerializationRequirement = // (simplified, actually an associatetype) + + // ... many lifecycle related functions, to be defined in follow-up proposals ... +} +``` + +Every distributed actor must declare what distributed actor system it is able to work with, this is expressed as an `associatedtype` requirement on the `DistributedActor` protocol, to which all `distributed actor` declarations conform implicitly. For example, this distributed actor works with some `ClusterSystem`: + +```swift +distributed actor Worker { + typealias ActorSystem = ClusterSystem +} +``` + +The necessity of declaring this statically will become clear as we discuss the serialization requirements and details of the typechecking mechanisms in the sections below. + +Please note that it is possible to use a protocol or type eraser as the actor system, which allows actors to swap-in completely different actor system implementations, as long as their serialization mechanisms are compatible. Using existential actor systems though comes at a slight performance penalty (as do all uses of existentials). + +It is possible to declare a module-wide `typealias DefaultDistributedActorSystem` in order to change this "default" actor system type, for all distributed actor types declared within a module: + +```swift +// in 'Cluster' module: +typealias DefaultDistributedActorSystem = ClusterSystem + +// in 'Cluster' module, clearly we want to use the 'ClusterSystem' +distributed actor Example { + // synthesized: + // typealias DistributedActorSystem = DefaultDistributedActorSystem // ClusterSystem + + // synthesized initializers (discussed below) also accept the expected type then: + // init(system: DefaultDistributedActorSystem) { ... } +} +``` + +It is also possible to declare protocols which refine the general `DistributedActor` concept to some specific transport, such as: + +```swift +protocol ClusterActor: DistributedActor where DistributedActorSystem == ClusterSystem {} + +protocol XPCActor: DistributedActor where DistributedActorSystem == XPCSystem { } +``` + +Those protocols, because they refine the `DistributedActor` protocol, can also only be conformed to by other distributed actors. It allows developers to declare specific requirements to their distributed actor's use, and even provide extensions based on the actor system type used by those actors, e.g.: + +```swift +extension DistributedActor where DistributedActorSystem == ClusterSystem { + /// Returns the node on which this distributed actor instance is located. + nonisolated var node: Cluster.Node? { ... } +} +``` + +> **Note:** We refer to `distributed actor` declarations or protocols refining the `DistributedActor` protocol as any "distributed actor type" - wherever this phrase is used, it can apply to a specific actor or such protocol. + +### Distributed Actor Initializers + +Distributed actor initializers are always _local_, therefore no special rules are applied to their isolation checking. + +Distributed actor initializers are subject to the same isolation rules as actor initializers, as outlined in [SE-0327: On Actors and Initialization](https://forums.swift.org/t/se-0327-on-actors-and-initialization/53053). Please refer to that proposal for details about when it is safe to escape `self` out of an actor initializer, as well as when it is permitted to call other functions on the actor during its initialization. + +A distributed actor's *designated initializer* must always contain exactly one `DistributedActorSystem` parameter. This is because the lifecycle and messaging of a distributed actor is managed by the system. It also assigns every newly initialized distributed actor instance an identity, that the actor then stores and makes accessible via the compiler-synthesized computed property `id`. The system is similarly available to the actor via the compiler synthesized computed property `actorSystem`. + +Similar to classes and local-only actors, a distributed actor gains an implicit default designated initializer when no user-defined initializer is found. This initializer accepts an actor system as parameter, in order to conform to the requirement stated above: + +```swift +// default system for this module: +typealias DefaultDistributedActorSystem = SomeSystem + +distributed actor Worker { + // synthesized default designated initializer: + // init(system: DefaultDistributedActorSystem) +} +``` + +if no module-wide `DefaultDistributedActorSystem` is defined, such declaration would request the developer to provide one at compile time: + +```swift +distributed actor Worker { + typealias ActorSystem = SomeSystem + + // synthesized default designated initializer: + // init(system: SomeSystem) +} +``` + +Alternatively, we can infer this typealias from a user-defined initializer, like this: + +```swift +distributed actor Worker { + // inferred typealias from explicit initializer declaration + // typealias ActorSystem = SomeSystem + + init(system: SomeSystem) { self.name = "Alice" } +} +``` + +The necessity to pass an actor system to each newly created distributed actor is because the system is the one assigning and managing identities. While we don't discuss those details in depth in this proposal, here is a short pseudocode of why passing this system is necessary: + +```swift +// Lifecycle interactions with the system during initialization +// NOT PART OF THIS PROPOSAL; These will be discussed in-depth in a forthcoming proposal focused on the runtime. +distributed actor Worker { + init(system: SomeSystem) { + // self._system = system + // the actor is assigned an unique identity as it initializes: + // self._id = system.assignID(Self.self) + self.name = "Alice" + // once fully initialized, the actor is ready to receive remote calls: + // system.actorReady(self) + } +} +``` + +Having that said, here are a few example of legal and illegal initializer declarations: + +```swift +distributed actor InitializeMe { + init() + // ❌ error: designated distributed actor initializer 'init()' is missing required 'DistributedActorSystem' parameter + + init(x: String) + // ❌ error: designated distributed actor initializer 'init(x:)' is missing required 'DistributedActorSystem' parameter + + init(system: AnyDistributedActorSystem, too many: AnyDistributedActorSystem) + // ❌ error: designated distributed actor initializer 'init(system:too:)' must accept exactly one DistributedActorSystem parameter, found 2 + + // -------- + + + init(system: AnyDistributedActorSystem) // ✅ ok + init(y: Int, system: AnyDistributedActorSystem) // ✅ ok + init(canThrow: Bool, system: AnyDistributedActorSystem) async throws // ✅ ok, effects are ok too + + // 'convenience' may or may not be necessary, depending on SE-0327 review outcome. + convenience init() { + self.init(system: SomeSystem(...)) // legal, but not recommended + } +} +``` + +*Remote* distributed actor references are not obtained via initializers, but rather through a static `resolve(_:using:)` function that is available on any distributed type: + +```swift +extension DistributedActor { + + /// Resolves the passed in `id` using the passed distributed actor `system`, + /// returning either a local or remote distributed actor reference. + /// + /// The system will be asked to `resolve` the identity and return either + /// a local instance or request a "proxy" to be created for this identity. + /// + /// A remote distributed actor reference will forward all invocations through + /// the system, allowing it to take over the remote messaging with the + /// remote actor instance. + /// + /// - Parameter id: identity uniquely identifying a, potentially remote, actor in the system + /// - Parameter system: distributed actor system which must resolve and manage the returned distributed actor reference + static func resolve(id: ID, using system: DistributedActorSystem) throws -> Self +} +``` + +The specifics of resolving, and remote actor runtime details will be discussed in a follow-up proposal focused on the runtime aspects of distributed actors. We mention it here to share a complete picture how Identities, systems, and remote references all fit into the picture. + +### Distributed Actors implicitly conform to Codable + +If a distributed actor's `ID` conforms to `Codable`, the distributed actor automatically gains a `Codable` conformance as well. + +This conformance is synthesized by the compiler, for every specific `distributed actor` declaration. It is not possible to express such conformance using the conditional conformances. + +> **Note:** It is not possible to implement such conformance semantics on the DistributedActor protocol using conditional conformances (like this `extension DistributedActor: Codable where ID: Codable`), and it is unlikely to be supported in the future. As such, we currently opt to synthesize the conformance for specific distributed actor declarations. + +```swift +distributed actor Player /*: DistributedActor, Codable */ { + // typealias ID = SomeCodableID +} +``` + +The synthesized `Codable` conformance strictly relies on the implementation of the actors' identity `Codable` conformance. When we "encode" a distributed actor, we never encode "the actor", but rather only its identity: + +```swift +// distributed actor Player: Codable, ... { + nonisolated public func encode(to encoder: Encoder) throws { + var container = encoder.singleValueContainer() + try container.encode(self.id) + } +// } +``` + +And similarly, decoding a distributed actor has the specific meaning of attempting to `resolve(_:using:)` a reference of the specific actor type, using the decoded id: + +```swift +// distributed actor Player: Codable, ... { + nonisolated public init(from decoder: Decoder) throws { + // ~~~ pseudo code for illustration purposes ~~~ + guard let system = decoder.userInfo[.distributedActorSystemKey] as? Self.ActorSystem else { + throw DistributedActorCodingError(message: + "Missing DistributedActorSystem (for key .distributedActorSystemKey) " + + "in \(decoder).userInfo, while decoding \(Self.self)!") + } + + // [1] decode the identity + let id: ID = try Self.ID(from: decoder) + // [2] resolve the identity using the current system; this usually will return a "remote reference" + self = try Self.resolve(id: id, using: system) // (!) + } +// } +``` + +The Decodable's `init(from:)` implementation is actually not possible to express in plain Swift today, because the restriction on self assignment in class initializers (and therefore also actor initializers). + +> **Note:** We could eventually generalize this more mutable `self` in class/actor initializer mechanism, however that would be done as separate Swift Evolution proposal. We are aware [this feature was requested before](https://forums.swift.org/t/allow-self-x-in-class-convenience-initializers/15924), and feels like a natural follow up to this proposal to generalize this capability. + +Note also that, realistically, there is only one correct way to implement a distributed actor's codability (as well as `Hashable` and `Equatable` conformances), because the only property that is related to its identity, and is known to both local and remote "sides" is the identity, as such implementations of those protocols must be directly derived from the `id` property of a distributed actor. + +The capability, to share actor references across to other (potentially remote) distributed actors, is crucial for location-transparency and the ability to "send actor references around" which enables developers to implement "call me later" style patterns (since we cannot do so with closures, as they are not serializable). In a way, this is similar to the delegate pattern, known to developers on Apple platforms: where we offer an instance to some other object, that will call lifecycle or other types of methods on the delegate whenever certain events happen. + +To illustrate how this capability is used in practice, let us consider the following turn-based distributed `Game` example, which waits until it has enough players gathered, and then kicks off the game by notifying all the players (regardless _where_ they are located) that the game is now starting. + +```swift +typealias DefaultDistributedActorSystem = SomeCodableDistributedActorSystem +struct SomeCodableDistributedActorSystem: DistributedActorSystem { + typealias ActorID = SomeCodableID + typealias SerializationRequirement = Codable +} + +distributed actor Player { + distributed func play(turn: Int) -> Move { ... } + distributed func opponentMoved(_ move: Move) { ... } +} + +distributed actor Game { + let minPlayers = 2 + var players: Set = [] + + distributed func join(player: Player) async throws { + guard players.count < 2 else { + throw ... + } + + players.insert(player) + + if players.count == 2 { + await play() // keep asking players for their move via 'play(turn:)' until one of them wins + } + } + + func play() async throws { + // keep asking players for their move via 'play(turn:)' until one of them wins + } + + distributed var result: GameResult { + ... + } +} + +func play(game: Game) async throws { + try await game.join(player: Player(system: ...)) + try await game.join(player: Player(system: ...)) + // the game begins, players are notified about it + + let result = try await game.result + print("Winner of \(game) was: \(result.winner)") +} +``` + +The `Player` distributed actor automatically gained a Codable conformance, because it is using the `SomeCodableDistributedActorSystem` that assigns it a `SomeCodableID`. Other serialization mechanisms are also able to implement this "encode the ID" and "decode the ID, and resolve it" pattern, so this pattern is equally achievable using Codable, or other serialization mechanisms. + +### Distributed Methods + +The primary way a distributed actor can be interacted with are distributed methods. Most notably, invoking a non-distributed method (i.e. those declared with *just* the `func` keyword by itself), is not allowed as it may be potentially violating distributed actor isolation rules, that is unless the target of the invocation is known to be a *local* distributed actor - a topic we'll explore later on in this proposal: + +```swift +distributed actor IsolationExample { + func notDistributed() {} + distributed func accessible() {} + distributed var computed: String { "" } +} + +func test(actor: IsolationExample) async throws { + try await actor.notDistributed() + // ❌ error: only 'distributed' instance methods can be called on a potentially remote distributed actor + + try await actor.accessible() + // ✅ ok, method is distributed + + try await actor.computed + // ✅ ok, distributed get-only computed property +} +``` + +Distributed methods are declared by writing the `distributed` keyword in the place of a declaration modifier, under the `actor-isolation-modifier` production rule as specified by [the grammar in TSPL](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_declaration-modifiers). Only methods can use `distributed` as a declaration modifier, and no order is specified for this modifier. + +It is also possible to declare distributed get-only properties, and they obey the same rules as a parameter-less `distributed func` would. It is not permitted to make get/set computed properties, or stored properties `distributed`. + +Distributed actor types are the only types in which a distributed method declaration is allowed. This is because, in order to implement a distributed method, an actor system and identity must be associated with the values carrying the method. Distributed methods can synchronously refer to any of the state isolated to the distributed actor instance. + +The following distributed method declarations are not allowed: + +```swift +actor/class/enum/struct NotDistributedActor { + distributed func test() {} + // ❌ error: 'distributed' function can only be declared within 'distributed actor' +} + +protocol NotDistributedActorProtocol { + distributed func test() + // ❌ error: 'distributed' function can only be declared within 'distributed actor' + // 💡 fixit: add ': DistributedActor' to protocol inheritance clause +} +``` + +While these are all proper declarations: + +```swift +distributed actor Worker { + distributed func work() { ... } +} + +extension Worker { + distributed func reportWorkedHours() -> Duration { ... } +} + +protocol TypicalGreeter: DistributedActor { + distributed func greet() +} +``` + +The last example, the `TypicalGreeter` protocol, can *only* be implemented by a `distributed actor`, because of the `DistributedActor` requirement. We will discuss distributed actors conforming to protocols in great detail below. + +It is not allowed to combine `distributed` with `nonisolated`, as a distributed function is _always_ isolated to the actor in which it is defined. + +```swift +distributed actor Charlie { + distributed nonisolated func cantDoThat() {} + // ❌ error: 'distributed' function must not be 'nonisolated' + // 💡 fixit: remove 'nonisolated' or 'distributed' +} +``` + +It is possible to declare a nonisolated method though. Such function can only access other `nonisolated` members of the instance. Two important members which are such nonisolated computed properties are the actor's identity, and associated actor system. Those are synthesized by the compiler, however they just follow the same isolation rules as laid out in this proposal: + +```swift +distributed actor Charlie: CustomStringConvertible { + // synthesized: nonisolated var id: Self.ID { get } + // synthesized: nonisolated var actorSystem: Self.ActorSystem { get } + + nonisolated var description: String { + "Charlie(\(self.id))" // ok to refer to `self.id` since also nonisolated + } +} +``` + +Distributed methods may be declared explicitly `async` or `throws` and this has the usual effect on the declaration and method body. It has no effect on cross distributed actor calls, because such calls are implicitly asynchronous and throwing to begin with. + +The `distributed` nature of a method is completely orthogonal to access control. It is even possible to declare a `private distributed func` because the following pattern may make it an useful concept to have: + +```swift +distributed actor Robot { + + nonisolated async throws isHuman(caller: Caller) -> String { + guard isTrustworthy(caller) else { + return "It is a mystery!" // no remote call needs to be performed + } + + return try await self.checkHumanity() + } + + private distributed func checkHumanity() -> String { + "Human, after all!" + } +} +``` + +Such methods allow us avoiding remote calls if some local validation already can short-circuit them. While not a common pattern, it definitely can have its uses. Note that the ability to invoke distributed methods remotely, also directly translates into such methods being "effectively public", even if access control wise they are not. This makes sense, and distributed methods must always be audited and carefully checked if they indeed should be allowed to execute when invoked remotely, e.g. they may need to perform caller authentication – a feature we do not provide out of the box yet, but are definitely interested in exploring in the future. + +It is not allowed to declare distributed function parameters as `inout` or varargs: + +```swift +distributed actor Charlie { + distributed func varargs(int: Int...) {} + // ❌ error: cannot declare variadic argument 'int' in distributed instance method 'varargs(int:)' + + distributed func noInout(inNOut burger: inout String) {} + // ❌ error: cannot declare 'inout' argument 'burger' in distributed instance method 'noInout(inNOut:)' + // 💡 fixit: remove 'inout' +} +``` + +While subscripts share many similarities with methods, they can lead to complex and potentially impossible to support invocations, meaning that they are currently also not allowed to be `distributed`. Such subscripts' usefulness would, in any case, be severely limited by both their lack of support for being `async` (e.g., could only support read-only subscripts, because no coroutine-style accessors) and their lightweight syntax can lead to the same problems as properties. + +Distributed functions _may_ be combined with property wrappers to function parameters (which were introduced by [SE-0293: Extend Property Wrappers to Function and Closure Parameters](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0293-extend-property-wrappers-to-function-and-closure-parameters.md)), and their semantics are what one would expect: they are a transformation on the syntactical level, meaning that the actual serialized parameter value is what the property wrapper has wrapped the parameter in. This is especially interesting for implementing eager validation of specific parameters, such that calls with illegal argument values can be synchronously prevented before even sending the message. Of course, the recipient should still validate the incoming arguments using the same logic, but thanks to this we are able to avoid sending wrong values in non-adversarial situations, and just validate some values on the client side eagerly. + +#### Distributed Method Serialization Requirements + +An important goal of the distributed actor design is being able to enforce some level of compile time safety onto distributed methods calls, which helps prevent unexpected runtime failures, and aides developers make conscious decisions which types should be exposed to remote peers and which not. + +This feature is applied to `distributed` methods, and configured by declaring a `SerializationRequirement` typealias on the actor system, from which specific actors infer it. This type alias informs the type-checker to ensure that all parameters, as well as return type of distributed methods must conform to the type that is provided as `SerializationRequirement`. This is in addition to the usual `Sendable` conformance requirements enforced on any values passed to/from actors). + +Another interesting capability this unlocks is being able to confine actors to sending only well-known types, if we wanted to enforce such closed-world assumptions onto the permissible messages exchanged between actors. + +Most frequently, the serialization requirement is going to be `Codable`, so for the rest of this proposal we'll focus mostly on this use-case. It is equally possible and supported to provide e.g. an external serialization systems top-level protocol as requirement here, e.g. a Protocol Buffer `Message`. The following snippet illustrates how this can work in practice: + +```swift +protocol CodableDistributedActorSystem: DistributedActorSystem { + typealias SerializationRequirement = Codable +} + +distributed actor Worker { + typealias ActorSystem = CodableDistributedActorSystem + typealias SerializationRequirement = SpecificActorSystem.SerializationRequirement + // = Codable +} +``` + +It is possible, albeit not recommended, to disable this checking by setting the `SerializationRequirement` to `Any` in which case no additional checks are performed on distributed methods. + +This section will discuss the implications of the `SerializationRequirement` on distributed method declarations. + +A serialization requirement means that all parameter types and return type of distributed method must conform to the requirement. With the `CodableDistributedActorSystem` in mind, let us write a few methods and see how this works: + +```swift +distributed actor Worker { + typealias ActorSystem = CodableDistributedActorSystem + + distributed func ok() // ✅ ok, no parameters + distributed func greet(name: String) -> String // ✅ ok, String is Codable + + struct NotCodable {} + + distributed func reject(not: NotCodable) + // ❌ error: parameter 'not' of type 'NotCodable' in distributed instance method + // does not conform to 'Codable' + // 💡 fixit: add ': Codable' to 'struct NotCodable' +} +``` + +This also naturally extends to closures without any the need of introducing any special rules, because closures do not conform to protocols (such as `Codable`), the following is naturally ill-formed and rejected: + +```swift +distributed actor Worker { + typealias ActorSystem = CodableDistributedActorSystem + + distributed func take(_ closure: (String) -> String) + // ❌ error: parameter 'closure' of type '(String) -> String' in distributed instance method + // does not conform to 'Codable' +} +``` + +Thrown errors are not enforced to be `Codable`, however a distributed actor system may detect that an error is Codable at runtime, and attempt to transfer it back entirely. For throws of non-Codable types, systems should attempt some form of best-effort description of the error, while keeping in mind privacy of error descriptions. I.e. errors should never be sent back to the caller by just getting their description, as that may leak sensitive information from the server system. A recommended approach here is to send back the type of the thrown error and throwing some generic `NotCodableError("\(type(of: error))")` or similar. + +Distributed actors may also witness protocol requirements (discussed in more detail below), however their method declarations must then also conform to the `SerializationRequirement`: + +```swift +protocol Greetings { + func greet(name: String) async throws + func synchronous() +} + +distributed actor Greeter: Greetings { + // typealias SerializationRequirement = Codable + distributed func greet(name: String) { // may or may not be async/throws, it always is when cross-actor + // ✅ ok, String is Codable + } + + nonisolated func synchronous() {} // nonisolated func may be used the same as on normal actors +} +``` + +Note that while every `distributed actor` must be associated with some specific distributed actor system, protocols need not be so strict and we are allowed to specify a distributed actor protocol like this: + +```swift +protocol Greetings: DistributedActor { + // no specific ActorSystem requirement (!) + func greet(name: String) +} +``` + +At the declaration site of such protocol the distributed functions are *not* subject to any `SerializationRequirement` checks. However once it is implemented by a distributed actor, that actor will be associated with a specific actor system, and thus also a specific SerializationRequirement, and could potentially not be able to implement such protocol because of the serializability checks, e.g.: + +```swift +protocol Greetings: DistributedActor { + // no specific ActorSystem requirement (!) + func greet(name: String) +} + +distributed actor Greeter { + // typealias SerializationRequirement = MagicMessage + distributed func greet(name: String) {} + // ❌ error: parameter 'name' of type 'String' in distributed instance method + // does not conform to 'MagicMessage' +} +``` + +A similar mechanism will exist for resolving remote actor references only based on a protocol. + +#### Distributed Methods and Generics + +It is possible to declare and use distributed methods that make use of generics. E.g. we could define an actor that picks an element out of a collection, yet does not really care about the element type: + +```swift +distributed actor Picker { + func pickOne(from items: [Item]) -> Item? { // Is this ok? It depends... + ... + } +} +``` + +This is possible to implement in general, however the `Item` parameter will be subject to the same `SerializableRequirement` checking as any other parameter. Depending on the associated distributed actor system's serialization requirement, this declaration may fail to compile, e.g. because `Item` was not guaranteed to be `Codable`: + +```swift +distributed actor Picker { + // typealias ActorSystem = CodableMessagingSystem + func pickOne(from items: [Item]) -> Item? { nil } + // ❌ error: parameter 'items' of type '[Item]' in distributed instance method + // does not conform to 'Codable' + // ❌ error: return type 'Item' in distributed instance method does not conform to 'Codable' + + func pickOneFixed(from items: [Item]) -> Item? + where Item: Codable { nil } // ✅ ok, we declared that the generic 'Item' is 'Codable' +} +``` + +This is the same rule about serialization requirements really, but spelled out explicitly. + + The runtime implementation of such calls is more complicated than non-generic calls, and does incur a slight wire envelope size increase, because it must carry the *specific type identifier* that was used to perform the call (e.g. that it was invoked using the *specific* `struct MyItem: Item` and not just some item). Generic distributed function calls will perform the deserialization using the *specific type* that was used to perform the remote invocation. + +As with any other type involved in message passing, actor systems may also perform additional inspections at run time of the types and check if they are trusted or not before proceeding to decode them (i.e. actor systems have the possibility to inspect incoming message envelopes and double-check involved types before proceeding tho decode the parameters). + +It is also allowed to make distributed actors themselves generic, and it works as one would expect: + +```swift +distributed actor Worker { // ✅ ok + func work() -> Item { ... } +} +``` + + + +#### Distributed Methods and Existential Types + +It is worth calling out that due to existential types not conforming to themselves, it is not possible to just pass a `Codable`-conforming existential as parameter to distributed functions. It will result in the following compile time error: + +```swift +protocol P: Codable {} + +distributed actor TestExistential { + typealias ActorSystem = CodableMessagingSystem + + distributed func compute(s: String, i: Int, p: P) {} + // ❌ error: parameter 'p' of type 'P' in distributed instance method does not conform to 'Codable' +} +``` + +The way to deal with this, as with usual local-only Swift programming, is to make the `P` existential generic, like this: + +```swift +protocol P: Codable {} + +distributed actor TestExistential { + typealias ActorSystem = CodableMessagingSystem + + distributed func compute(s: String, i: Int, p: Param) {} + // ✅ ok, the generic allows us getting access to the specific underlying type +} +``` + +which will compile, and work as expected. + +#### Implicit effects on Distributed Methods + +Local-only actor methods can be asynchronous , throwing or both, however invoking them cross-actor always causes them to become implicitly asynchronous: + +```swift +// Reminder about implicit async on actor functions +actor Greeter { + func greet() -> String { "Hello!" } + func inside() { + greet() // not asynchronous, we're not crossing an actor boundary + } +} + +Task { + await Greeter().hi() // implicitly asynchronous +} +``` + +The same mechanism is extended to the throwing behavior of distributed methods. Distributed cross-actor calls may fail not only because of the remote side actively throwing an error, but also because of transport errors such as network issues or serialization failures. Therefore, distributed cross-actor calls also implicitly gain the throwing effect, and must be marked with `try` when called: + +```swift +distributed actor Greeter { + distributed func greet() -> String { "Hello!" } + + func inside() { + greet() // not asynchronous or throwing, we're inside the actual local instance + } +} + +Task { + try await Greeter().greet() // cross-actor distributed function call: implicitly async throws +} +``` + +It is also possible to declare distributed functions as either `throws` or `async` (or both). The implicitly added effect is a no-op then, as the function always was, respectively, throwing or asynchronous already. + +The following snippets illustrate all cases how effects are applied to distributed actor methods: + +```swift +distributed actor Worker { + distributed func simple() {} + distributed func funcAsync() async {} + distributed func funcThrows() throws {} + distributed func funcAsyncThrows() async throws {} +} +``` + +Cross distributed-actor calls behave similar to cross actor calls, in the sense that they gain those implicit effects. This is because we don't know if the callee is remote or local, and thus assume that it might be remote, meaning that there may be transport errors involved in the call, making the function call implicitly throwing: + +```swift +func outside(worker: Worker) async throws { + // wrong invocation: + worker.simple() + // ❌ error: expression is 'async' but is not marked with 'await' + // ❌ error: call can throw but is not marked with 'try' + // 💡 note: calls to distributed instance method 'simple()' from outside of its actor context are implicitly asynchronous + + // proper invocations: + try await worker.simple() + try await worker.funcAsync() + try await worker.funcThrows() + try await worker.funcAsyncThrows() +} +``` + +These methods may be also be called from *inside* the actor, as well as on an `isolated` parameter of that actor type, without any implicit effects applied to them. This is the same idea applies that actor methods becoming implicitly asynchronous but only during cross-actor calls. + +```swift +extension Worker { + distributed func inside() async throws { + self.simple() + await self.funcAsync() + try self.funcThrows() + try await self.funcAsyncThrows() + } +} + +func isolatedFunc(worker: isolated Worker) async throws { + worker.simple() + await worker.funcAsync() + try worker.funcThrows() + try await worker.funcAsyncThrows() +} +``` + +The isolated function parameter works because the only way to offer an `isolated Worker` to a function, is for a real local actor instance to offer its `self` to `isolatedFunc`, and because of that it is known that it is a real local instance (after all, only a real local instance has access to `self`). + +It is not allowed to declare `isolated` parameters on distributed methods, because distributed methods _must_ be isolated to the actor they are declared on. This can be thought of always using an `isolated self: Self` parameter, and in combination of a func only being allowed to be isolated to a single actor instance, this means that there cannot be another isolated parameter on such functions. Following this logic a `nonisolated func` declared on a distributed actor, _is_ allowed to accept `isolated` parameters, however such call will not be crossing process boundaries. + +It is also worth calling out the interactions with `Task` and `async let`. Their context may be the same asynchronous context as the actor, in which case we also do not need to cause the implicit asynchronous effect. When it is known the invocation is performed on an `isolated` distributed actor reference, we infer the fact that it indeed is "known to be local", and do not need to apply the implicit throwing effect either: + +```swift +extension Worker { + func test(other: Philosopher) async throws { + // self -------------------------------------------------------------------- + async let alet = self.simple() // implicitly async; async let introduced concurrent context + _ = await alet // not throwing, but asynchronous! + + Task { + _ = self.hi() // no implicit effects, Task inherited the Actor's execution context + } + + Task.detached { + _ = await self.hi() // implicitly async, different Task context than the actor + // however not implicitly throwing; we know there is no networking involved in a call on self + } + + // other ------------------------------------------------------------------- + async let otherLet = other.hi() // implicitly async and throws; other may be remote + _ = try await otherLet // forced to 'try await' here, as per usual 'async let' semantics + + Task { + _ = try await other.hi() // implicitly async and throws + } + + Task.detached { + _ = try await other.hi() // implicitly async and throws + } + } +} +``` + +#### Isolation states and Implicit effects on Distributed Methods + +A distributed actor reference. such as a variable or function parameter, effectively can be in one of three states: + +- `isolated` – as defined by Swift's local-only actors. The `isolated` also implies the following "local" state, because it is not possible to pass isolated members across distributed boundaries, +- "local" – not explicitly modeled in the type-system in this proposal, though we might end up wanting to do so (see Future Directions), or +- "potentially remote" – which is the default state of any distributed actor variable. + +These states determine the implicit effects that function invocations, and general distributed actor isolation checking, need to apply when checking accesses through the distributed actor reference. + +Let us discuss the implications of these states on the effects applied to method calls on such distributed actor references, starting from the last "potentially remote" state, as it is the default and most prominent state which enables location-transparency. + +By default, any call on a ("potentially remote") distributed actor must be assumed to be crossing network boundaries. Thus, the type system pessimistically applies implicit throwing and async effects to such call-sites: + +```swift +func test(actor: Greeter) async throws { + try await actor.greet(name: "Asa") // ✅ call could be remote +} +``` + +In special circumstances, a reference may be "known to be local", even without introducing a special "local" keyword in the language this manifests itself for example in closures which capture `self`. For example, we may capture `self` in a detached task, meaning that the task's closure will be executing on some different execution context than the actor itself -- and thus `self` is *not* isolated, however we *know* that it definitely is local, because there is no way we could ever refer to `self` from a remote actor: + +```swift +distributed actor Closer { + distributed func check() -> Bool { true } + + func test() { + Task.detached { + await self.check() // ✅ call is definitely local, but it must be asynchronous + } + } +} +``` + +In the above situation, we know for sure that the `self.check()` will not be crossing any process boundaries, and therefore there cannot be any implicit errors emitted by the underlying distributed actor system transport. This manifests in the type-system by the `distributed func` call not being throwing (!), however it remains asynchronous because of the usual local-only actor isolation rules. + +The last case is `isolated` distributed actor references. This is relatively simple, because it just reverts all isolation checking to the local-only model. Instance members of actors are effectively methods which take an `isolated Self`, and in the same way functions which accept an `isolated Some(Distributed)Actor` are considered to be isolated to that actor. For the purpose of distributed actor isolation checking it effectively means there are no distributed checks at all, and we can even access stored properties synchronously on such reference: + +```swift +distributed actor Namer { + let baseName: String = ... +} + +func bad(n: Namer) { + n.baseName // ❌ error, as expected we cannot access the distributed actor-isolated state +} + +func good(n: isolated Namer) { + n.baseName // ✅ ok; we are isolated to the specific 'n' Namer instance +} +``` + +### Distributed Actor Properties + +#### Stored properties + +Distributed actors may declare any kind of stored property, and the declarations themselves are *not restricted in any way*. This is important and allows distributed actors to store any kind of state, even if it were not serializable. Access to such state from the outside though is only allowed through distributed functions, meaning that cross-network access to such non-serializable state must either be fully encapsulated or "packaged up" into some serializable format that leans itself to transporting across the network. + +One typical example of this is a distributed actor storing a live database connection, and being unable to send this connection across to other nodes, it should send the results of querying the database to its callers. This is a very natural way to think about actor storage, and will even be possible to enforce at compile time, which we'll discuss in follow-up proposals discussing serialization and runtime aspects of distributed actor messages. + +To re-state the rule once again more concisely: It is not possible to reach a distributed actors stored properties cross-actor. This is because stored properties may be located on a remote host, and we do not want to subject them to the same implicit effects, and serialization type-checking as distributed methods. + +```swift +distributed actor Properties { + let fullName: String + var age: Int +} +``` + +Trying to access those properties results in isolation errors at compile time: + +```swift +Properties().fullName +// ❌ error: distributed actor-isolated property 'fullName' can only be referenced inside the distributed actor +Properties().age +// ❌ error: distributed actor-isolated property 'age' can only be referenced inside the distributed actor +``` + +Unlike with local-only actors, it is *not* allowed to declare `nonisolated` *stored properties*, because a nonisolated stored property implies the ability to access it without any synchronization, and would force the remote "proxy" instance to have such stored property declared and initialized, however there is no meaningful good way to initialize such variable, because a remote reference is _only_ the actor's identity and associated transport (which will be explored in more depth in a separate proposal): + +```swift +distributed actor Properties { + nonisolated let fullName: String // ❌ error: distributed actor cannot declare nonisolated stored properties +} +``` + +It is allowed to declare static properties on distributed actors, and they are not isolated to the actor. This is the same as static properties on local-only actors. + +```swift +distributed actor Worker { + static let MAX_ITEMS: Int = 12 // ⚠️ static properties always refer to the value in the *local process* + var workingOnItems: Int = 0 + + distributed func work(on item: Item) throws { + guard workingOnItems < Self.MAX_ITEMS else { + throw TooMuchWork(max: Self.MAX_ITEMS) + } + + workingonItems += 1 + } +} +``` + +Be aware though that any such `static` property on a `distributed actor` always refers to whatever the property was initialized with _locally_ (in the current process). i.e. if the remote node is running a different version of the software, it may have the `MAX_ITEMS` value set to something different. So keep this in mind when debugging code while rolling out new versions across a cluster. Static properties are useful for things like constants, so feel free to use them in the same manner as you would with local-only actors. + +It is permitted, same as with local-only actors, to declare `static` methods and even `static` variables on distributed actors, although please be advised that currently static variables are equally thread-*unsafe* as global properties and Swift Concurrency currently does not perform any checks on those. + +```swift +// Currently allowed in Swift 5.x, but dangerous (for now) +[distributed] actor Glass { + var contents: String = Glass.defaultContents + + static var defaultContents: String { "water" } // ⚠️ not protected from data-races in Swift 5.x +} +``` + +As such, please be very careful with such mutable declarations. Swift Concurrency will eventually also check for shared global and static state, and devise a model preventing races in such declarations as well. Static properties declared on distributed actors will be subject to the same checks as any other static properties or globals once this has been proposed and implemented (via a separate Swift Evolution proposal). + +#### Computed properties + +Distributed _computed properties_ are possible to support in a very limited fashion because of the effectful nature of the distributed keyword. It is only possible to make *read-only* properties distributed, because only such properties may be effectful (as introduced by [SE-0310: Effectful Read-only Properties](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0310-effectful-readonly-properties.md)). + +```swift +distributed actor Chunk { + let chunk: NotSerializableDataChunk + + distributed var size: Int { self.chunk.size } +} +``` + +A distributed computed property is similar to a method accepting zero arguments, and returning a value. + +Distributed computed properties are subject to the same isolation rules, and implicit async and throwing effects. As such, accessing such variable (even across the network) is fairly explicitly telling the developer something is going on here, and they should re-consider if e.g. doing this in a loop truly is a good idea: + +```swift +var i = 0 +while i < (try await chunk.size) { // very bad idea, don't do this + // logic here + i += 1 +} + +// better, only check the size once: +var i = 0 +let max = try await chunk.size // implicitly 'async throws', same as distributed methods +while i < max { + // logic here + i += 1 +} +``` + +Because distributed methods and properties are statically known, we could envision IDEs giving explicit warnings, and even do some introspection and analysis detecting such patterns if they really wanted to. + +Any value returned by such computed property needs to be able to be serialized, similarly to distributed method parameters and return values, and would be subject to the same checks. + +It is not possible to declare read/write computed properties, because of underlying limitations of effectful properties. + +### Protocol Conformances + +Distributed actors can conform to protocols in the same manner as local-only actors can. + +As calls "through" protocols are always cross-actor, requirements that are possible to witness by a `distributed actor` must be `async throws`. The following protocol shows a few examples of protocol requirements, and whether they are possible to witness using a distributed actor's distributed function: + +```swift +protocol Example { + func synchronous() + func justAsync() async -> Int + func justThrows() throws -> Int + func asyncThrows() async throws -> String +} +``` + +We can attempt to conform to this protocol using a distributed actor: + +```swift +distributed actor ExampleActor: Example { + distributed func synchronous() {} + // ❌ error: actor-isolated instance method 'synchronous()' cannot be used to satisfy a protocol requirement + // cross-actor calls to 'justThrows()' are 'async throws' yet protocol requirement is synchronous + + distributed func justAsync() async -> Int { 2 } + // ❌ error: actor-isolated instance method 'justAsync()' cannot be used to satisfy a protocol requirement + // cross-actor calls to 'justAsync()' are 'async throws' yet protocol requirement is only 'async' + + distributed func justThrows() throws -> Int { 2 } + // ❌ error: actor-isolated instance method 'justThrows()' cannot be used to satisfy a protocol requirement}} + // cross-actor calls to 'justThrows()' are 'async throws' yet protocol requirement is only 'throws' + + distributed func asyncThrows() async throws -> String { "two" } // ✅ +} +``` + +Let us focus on the last example, `asyncThrows()` which is declared as a throwing and asynchronous protocol requirement, and returns a `String`. We are able to witness this requirement, but we should mention the future direction of compile time serialization checking while discussing this function as well. + +If we recall the previously mentioned serialization conformance checking mechanism, we could imagine that the `ExampleActor` configured itself to use e.g. `Codable` for its message serialization. This means that the method declarations are subject to `Codable` checking: + +```swift +distributed actor CodableExampleActor: Example { + typealias SerializationRequirement = Codable + + distributed func asyncThrows() async throws -> String { "two" } // ✅ ok, String is Codable +} +``` + +As we can see, we were still able to successfully witness the `asyncThrows` protocol requirement, since the signature matches our serialization requirement. This allows us to conform to existing protocol requirements with distributed actors, without having to invent complicated wrappers. + +If we used a different serialization mechanism, we may have to provide a `nonisolated` witness, that converts the types expected by the protocol, to whichever types we are able to serialize (e.g. protocol buffer messages, or anything else, including custom serialization formats). Either way, we are able to work our way through and conform to protocols if necessary. + +It is possible to utilize `nonisolated` functions to conform to synchronous protocol requirements, however those have limited use in practice on distributed actors since they cannot access any isolated state. In practice such functions are implementable by accessing the actor's identity or actor system it belongs to, but not much else. + +```swift +protocol CustomStringConvertible { + var description: String { get } +} + +distributed actor Example: CustomStringConvertible { + nonisolated var description: String { + "distributed actor Example: \(self.identity)" + } +} +``` + +The above example conforms a distributed actor to the well-known `CustomStringConvertible` protocol, and we can use similar techniques to implement protocols like `Hashable`, `Identifiable`, and even `Codable`. We will discuss these in the following proposals about distributed actor runtime details though. + +#### The `DistributedActor` protocol and protocols inheriting from it + +This proposal mentioned the `DistributedActor` protocol a few times, however without going into much more depth about its design. We will leave this to the *actor runtime* focused proposals, however in regard to isolation we would like do discuss its relation to protocols and protocol conformances: + +The `DistributedActor` protocol cannot be conformed to explicitly by any other type other than a `distributed actor` declaration. This is similar to the `Actor` protocol and `actor` declarations. + +It is possible however to express protocols that inherit from the `DistributedActor` protocol, like this: + +```swift +protocol Worker: DistributedActor { + distributed func work(on: Item) -> Int + + nonisolated func same(as other: Worker) -> Bool + + static func isHardWorking(_ worker: Worker) -> Bool +} +``` + +Methods definitions inside distributed actor inheriting protocols must be declared either:`distributed`, `static`or `nonisolated`. Again, we value the explicitness of the definitions, and the compiler will guide and help you decide how the method shall be isolated. + +Note that it is always possible to conform to a distributed protocol requirement with a witness with "more" effects, since the cross-actor API remains the same - thanks to the implicit effects caused by the distributed keyword. + +```swift +protocol Arnold: Worker { + distributed func work(on: Item) async -> Int { + // turns out we need this to be async internally, this is okay + } +} +``` + +This witness works properly, because the `distributed func` requirement in the protocol is always going to be `async throws` due to the `distributed func`'s effect on the declaration. Therefore the declaration "inside the actor" can make use of `async` or `throws` without changing how the protocol can be used. + +### Breaking through Location Transparency + +Programs based on distributed actors should always be written to respect location transparency, but sometimes it is useful to break through that abstraction. The most common situation where breaking through location transparency can be useful is when writing unit tests. Such tests may need to inspect state, or call non-distributed methods, of a distributed actor instance that is known to be local. + +To support this kind of niche circumstance, all distributed actors offer a `whenLocal` method, which executes a provided closure based on whether it is a local instance: + +```swift +extension DistributedActor { + /// Runs the 'body' closure if and only if the passed 'actor' is a local instance. + /// + /// Returns `nil` if the actor was remote. + @discardableResult + nonisolated func whenLocal( + _ body: (isolated Self) async throws -> T + ) async rethrows -> T? + + /// Runs the 'body' closure if and only if the passed 'actor' is a local instance. + /// + /// Invokes the 'else' closure if the actor instance was remote. + @discardableResult + nonisolated func whenLocal( + _ body: (isolated Self) async throws -> T, + else whenRemote: (Self) async throws -> T + ) async rethrows -> T +``` + +When the instance is local, the `whenLocal` method exposes the distributed actor instance to the provided closure, as if it were a regular actor instance. This means you can invoke non-distributed methods when the actor instance is local, without relying on hacks that would trigger a crash if invoked on a remote instance. + +> **Note:** We would like to explore a slightly different shape of the `whenLocal` functions, that would allow _not_ hopping to the actor unless necessary, however we are currently lacking the implementation ability to do so. So this proposal for now shows the simple, `isolated` based approach. The alternate API we are considering would have the following shape: +> +> ```swift +> @discardableResult +> nonisolated func whenLocal( +> _ body: (local Self) async throws -> T +> ) reasync rethrows -> T? +> ``` +> +> This API could enable us to treat such `local DistActor` exactly the same as a local-only actor type; We could even consider allowing nonisolated stored properties, and allow accessing them synchronously like that: +> +> ```swift +> // NOT part of this proposal, but a potential future direction +> distributed actor FamousActor { +> let name: String = "Emma" +> } +> +> FamousActor().whenLocal { fa /*: local FamousActor*/ in +> fa.name // OK, known to be local, distributed-isolation does not apply +> } +> ``` + +## Future Directions + +### Versioning and Evolution of Distributed Actors and Methods + +Versioning and evolution of exposed `distributed` functionality is a very important, and quite vast topic to tackle. This proposal by itself does not include new capabilities - we are aware this might be limiting adoption in certain use-cases. + +#### Evolution of parameter values only + +In today's proposal, it is possible to evolve data models *inside* parameters passed through distributed method calls. This completely relies on the serialization mechanism used for the individual parameters. Most frequently, we expect Codable, or some similar mechanism, to be used here and this evolution of those values relies entirely on what the underlying encoders/decoders can do. As an example, we can define a `Message` struct like this: + +```swift +struct Message: Codable { + let oldVersion: String + let onlyInNewVersion: String +} + +distributed func accept(_: Message) { ... } +``` + +and the usual backwards / forwards evolution techniques used with `Codable` can be applied here. Most coders are able to easily ignore new unrecognized fields when decoding. It is also possible to improve or implement a different decoder that would also store unrecognized fields in some other container, e.g. like this: + +```swift +struct Message: Codable { + let oldVersion: String + let unknownFields: [String: ...] +} + +JSONDecoderAwareOfUnknownFields().decode(Message.self, from: ...) +``` + +and the decoder could populate the `unknownFields` if necessary. There are various techniques to perform schema evolution here, and we won't be explaining them in more depth here. We are aware of limitations and challenges related to `Codable` and might revisit it for improvements. + +#### Evolution of distributed methods + +The above-mentioned techniques apply only for the parameter values themselves though. With distributed methods we need to also take care of the method signatures being versioned, this is because when we declare + +```swift +distributed actor Greeter { + distributed func greet(name: String) +} +``` + +we exposed the ability to invoke `greet(name:)` to other peers. Such normal, non-generic signature will *not* cause the transmission of `String`, over the wire. They may be attempting to invoke this method, even as we roll out a new version of the "greeter server" which now has a new signature: + +```swift +distributed actor Greeter { + distributed func greet(name: String, in language: Language) +} +``` + +This is a breaking change as much in API/ABI and of course also a break in the declared wire protocol (message) that the actor is willing to accept. + +Today, Swift does not have great facilities to move between such definitions without manually having to keep around the forwarder methods, so we'd do the following: + +```swift +distributed actor Greeter { + + @available(*, deprecated, renamed: "greet(name:in:)") + distributed func greet(name: String) { + self.greet(name: name, in: .defaultLanguage) + } + + distributed func greet(name: String, in language: Language) { + print("\(language.greeting), name!") + } +} +``` + +This manual pattern is used frequently today for plain old ABI-compatible library evolution, however is fairly manual and increasingly annoying to use as more and more APIs become deprecated and parameters are added. It also means we are unable to use Swift's default argument values, and have to manually provide the default values at call-sites instead. + +Instead, we are interested in extending the `@available` annotation's capabilities to be able to apply to method arguments, like this: + +```swift +distributed func greet( + name: String, + @available(macOS 12.1, *) in language: Language = .defaultLanguage) { + print("\(language.greeting), name!") +} + +// compiler synthesized: +// // "Old" API, delegating to `greet(name:in:)` +// distributed func greet(name: String) { +// self.greet(name: name, in: .defaultLanguage) +// } +``` + +This functionality would address both ABI stable library development, and `distributed` method evolution, because effectively they share the same concern -- the need to introduce new parameters, without breaking old API. For distributed methods specifically, this would cause the emission of metadata and thunks, such that the method `greet(name:)` can be resolved from an incoming message from an "old" peer, while the actual local invocation is performed on `greet(name:in:)`. + +Similar to many other runtimes, removing parameters is not going to be supported, however we could look into automatically handling optional parameters, defaulting them to `nil` if not present incoming messages. + +In order to serve distribution well, we might have to extend what notion of "platform" is allowed in the available annotation, because these may not necessarily be specific to "OS versions" but rather "version of the distributed system cluster", which can be simply sem-ver numbers that are known to the cluster runtime: + +```swift +distributed func greet( + name: String, + @available(distributed(cluster) 1.2.3, *) in language: Language = .defaultLanguage) { + print("\(language.greeting), name!") +} +``` + +During the initial handshake peers in a distributed system exchange information about their runtime version, and this can be used to inform method lookups, or even reject "too old" clients. + +## Introducing the `local` keyword + +It would be possible to expand the way distributed actors can conform to protocols which are intended only for the actor's "local side" if we introduced a `local` keyword. It would be used to taint distributed actor variables as well as functions in protocols with a local bias. + +For example, `local` marked distributed actor variables could simplify the following (surprisingly common in some situations!) pattern: + +```swift +distributed actor GameHost { + let myself: local Player + let others: [Player] + + init(system: GameSystem) { + self.myself = Player(system: GameSystem) + self.others = [] + } + + distributed func playerJoined(_ player: Player) { + others.append(player) + if others.count >= 2 { // we need 2 other players to start a game + self.start() + } + } + + func start() { + // start the game somehow, inform the local and all remote players + // ... + // Since we know `myself` is local, we can send it a closure with some logic + // (or other non-serializable data, like a connection etc), without having to use the whenLocal trick. + myself.onReceiveMessage { ... game logic here ... } + } +} +``` + +The above example makes use of the `myself: local Player` stored property, which propagates the knowledge that the player instance stored in this property *definitely* is local, and therefore we can call non-distributed methods on it, which is useful when we need to pass it closures or other non-serializable state -- as we do in the `start()` method. + +An `isolated Player` where Player is a `distributed actor` would also automatically be known to be `local`, and the `whenLocal` function could be expressed more efficiently (without needing to hop to the target actor at all): + +```swift +// WITHOUT `local`: +// extension DistributedActor { +// public nonisolated func whenLocal(_ body: @Sendable (isolated Self) async throws -> T) +// async rethrows -> T? where T: Sendable + +// WITH local, we're able to not "hop" when not necessary: +extension DistributedActor { + public nonisolated func whenLocal(_ body: @Sendable (local Self) async throws -> T) + reasync rethrows -> T? where T: Sendable // note the reasync (!) +} +``` + +This version of the `whenLocal` API is more powerful, since it would allow accessing actor state without hops, if we extended the model to allow this. This would allow treating `local AnyDistributedActor` the same way as we treat any local-only actor, and can be very useful in testing. + +We would not have to wrap APIs in `whenLocal` or provide wrapper APIs that are `nonisolated` but actually invoke things on self, like this real problem example, from implementing a Cluster "receptionist" actor where certain calls shall only be made by the "local side", however the entire actor is accessible remotely for other peers to communicate with: + +```swift +distributed actor Receptionist { + distributed func receiveGossip(...) { ... } + + // only to be invoked by "local" actors + func registerLocalActor(actor: Act) where Act: DistributedActor { ... } +} +``` + +Since it is too annoying to tell end-users to "always use `whenLocal` to invoke the local receptionist", library developers are forced to provide the following wrapper: + +```swift +extension Receptionist { + + // annoying forwarder/wrapper func; potentially unsafe, intended only for local use. + nonisolated func register(actor: Act) async where Act: DistributedActor { + await self.whenLocal { myself in + myself.registerLocalActor(actor: actor) + } else: { + fatalError("\(#function) must only be called on the local receptionist!") + } + } +} + +// ------------------------------------ +final class System: DistributedActorSystem { + // ... + let receptionist: Receptionist +} + +distributed actor Worker { + init(system: System) async { + receptionist.register(self) // ✅ OK + } +} +``` + +This mostly works, but the implementation of the `nonisolated func register` leaves much to be desired. Rather, we want to express the following: + +```swift +final class System: DistributedActorSystem { + // ... + let receptionist: local Receptionist +} + +distributed actor Worker { + init(system: System) async { + await receptionist.registerLocalActor(self) // ✅ OK + } +} +``` + +Without the need of manually implementing the "discard the distributed nature" of such actors. + +We see this as a natural follow up and future direction, which may take a while to implement, but would vastly improve the ergonomics of distributed actors in those special yet common enough few cases where such actors make an appearance. + +## Alternatives Considered + +This section summarizes various points in the design space for this proposal that have been considered, but ultimately rejected from this proposal. + +### Implicitly `distributed` methods / "opt-out of distribution" + +After initial feedback that `distributed func` seems to be "noisy", we actively explored the idea of alternative approaches which would reduce this perceived noise. We are convinced that implicitly distributed functions are a bad idea for the overall design, understandability, footprint and auditability of systems expressed using distributed actors. + +A promising idea, described by Pavel Yaskevich in the [Pitch #1](https://forums.swift.org/t/pitch-distributed-actors/51669/129) thread, was to inverse the rule, and say that _all_ functions declared on distributed actors are `distributed` by default (except `private` functions), and introduce a `local` keyword to opt-out from the distributed nature of actors. This listing exemplifies the idea: + +```swift +distributed actor Worker { + func work(on: Item) {} // "implicitly distributed" + private func actualWork() {} // not distributed + + local func shouldWork(on item: Item) -> Bool { ... } // NOT distributed +} +``` + +However, this turns out to complicate the understanding of such a system rather than simplify it. + +[1] We performed an analysis of a real distributed actor runtime (that we [open sourced recently](https://swift.org/blog/distributed-actors/)), and noticed that complex distributed actors have by far more non-distributed functions, than distributed ones. It is typical for a single distributed function, to invoke multiple non distributed functions in the same actor - simply because good programming style causes the splitting out of small pieces of logic into small functions with good names; Special care would have to be taken to mark those methods local. It is easy to forget doing so, since it is not a natural concept anywhere else in Swift to have to mark things "local" -- everything else is local after all. + +For example, the [distributed actor cluster implementation](https://github.com/apple/swift-distributed-actors) has a few very complex actors, and their sizes are more or less as follows: + +- ClusterShell - a very complex actor, orchestrating node connections etc. + - 14 distributed methods (it's a very large and crucial actor for the actor system) + - ~25 local methods +- SWIMShell, thee actor orchestrating the SWIM failure detection mechanism, + - 5 distributed methods + - 1 public local-only methods used by local callers + - ~12 local methods + +- ClusterReceptionist, responsible for discovering and gossiping information about actors + - 2 distributed methods + - 3 public local-only methods + - ~30 internal and private methods (lots of small helpers) + +- NodeDeathWatcher, responsible for monitoring node downing, and issuing associated actor termination events, + - 5 distributed functions + - no local-only methods + +[2] We are concerned about the auditability and review-ability of implicit distributed methods. In a plain text review it is not possible to determine whether the following introduces a distributed entry point or not. Consider the following diff, that one might be reviewing when another teammate submits a pull request: + +```swift ++ extension Worker { ++ func runShell(cmd: String) { // did this add a remotely invocable endpoint? we don't know from this patch! ++ // execute in shell ++ } ++ } +``` + +Under implicit `distributed func` rules, it is impossible to know if this function is possible to be invoked remotely. And if it were so, it could be a potential exploitation vector. Of course transports do and will implement their own authentication and authorization mechanisms, however nevertheless the inability to know if we just added a remotely invokable endpoint is worrying. + +In order to know if we just introduced a scary security hole in our system, we would have to go to the `Worker` definition and check if it was an `actor` or `distributed actor`. + +The accidental exposing can have other, unintended, side effects such as the following declaration of a method which is intended only for the actor itself to invoke it when some timer tick is triggered: + +```swift +// inside some distributed actor +func onPeriodicAckTick() { ... } +``` + +The method is not declared `private`, because in tests we want to be able to trigger the ticks manually. Under the implicit `distributeed func` rule, we would have to remember to make it local, as otherwise we accidentally made a function that is only intended for our own timers as remotely invocable, which could be misunderstood and/or be abused by either mistake, or malicious callers. + +Effectively, the implicitly-distributed rule causes more cognitive overhead to developers, every time having to mark and think about local only functions, rather than only think about the few times they actively want to _expose_ methods. + +[3] We initially thought we could delay additional type checks of implicit distributed functions until their first use. This would be similar to `Sendable` checking, where one can define a function accepting not-Sendable values, and only once it is attempted to be used in a cross-actor situation, we get compile errors. + +With distribution this poses a problem though: For example, should we allow the following conformance: + +```swift +struct Item {} // NOT Codable + +protocol Builder { + func build(_: Item) async throws +} + +distributed actor Bob: Builder { + typealias SerializationRequirement = Codable + func build(_: Item) async throws { ... } +} +``` + +Under implicit distributed rules, we should treat this function as distributed, however that means we should be checking `Item` for the `Codable` conformance. We know at declaration time that this conformance is faulty. While in theory we could delay the error until someone actually invoked the build function: + +``` swift +let bob: Bob +try await bob.build(Item()) // ❌ error: parameter type 'Item' does not conform to 'Bob.SerializationRequirement' +``` + +so we have declared a method that is impossible to invoke... however if we attempted to erase `Bob` to `Builder`... + +```swift +let builder: Builder = bob +try await builder.build(Item()) +``` + +there is nothing preventing this call from happening. There is no good way for the runtime to handle this; We would have to invent some defensive throwing modes, throwing in the distributed remote thunk, if the passed parameters do not pass what the type-system should have prevented from happening. + +In other words, the Sendable-like conformance model invites problematic cases which may lead to unsoundness. + +Thus, the only type-checking model of distributed functions, implicit or not, is an eager one. Where we fail during type checking immediately as we see the illegal declaration: + +```swift +struct Item {} // NOT Codable + +protocol Builder { + func build(_: Item) async throws +} + +distributed actor Bob: Builder { + typealias SerializationRequirement = Codable + func build(_: Item) async throws { ... } + // ❌ error: function 'build(_:)' cannot be used to satisfy protocol requirement + // ❌ error: parameter type 'Item' does not conform to 'Bob.SerializationRequirement' +} +``` + +By itself this is fine, however this has a painful effect on common programming patterns in Swift, where we are encouraged to extract small meaningful functions that are re-used in places by the actor. We are forced to annotate _more_ APIs as `local` than we would have been with the _explicit_ `distributed` annotation model (see observation that real world distributed actors often have many small functions, not intended for distribution) + +[4] Since almost all functions are distributed by default in the implicit model, we need to create and store metadata for all of them, regardless if they are used or not. This may cause unnecessary binary size growth, and seems somewhat backwards to Swift's approach to be efficient and minimal in metadata produced. + +We are aware of runtimes where every byte counts, and would not want to prevent them from adopting distributed actors for fear of causing accidental binary size growth. In practice, we would force developers to always write `local func` unless proven that it needs to be distributed, then removing the keyword – this model feels backwards from the explicit distributed marking model, in which we make a conscious decision that "yes, this function is intended for distribution" and mark it as `distributed func` only once we actively need to. + +[5] While it may seem simplistic, an effective method for auditing a distributed "attack surface" of a distributed actor system is enabled by the ability search the codebase for `distributed func` and make sure all functions perform the expected authorization checks. These functions are as important as "service endpoints" and should be treated with extra care. This only works when distributed functions are explicit. + +We should also invest in transport-level authentication and authorization techniques, however some actions are going to be checked action-per-action, so this additional help of quickly locating distributed functions is a feature, not an annoyance. + +Summing up, the primary benefit of the implicit `distributed func` rule was to attempt to save developers a few keystrokes, however it fails to deliver this in practice because frequently (verified by empirical data) actors have many local methods which they do not want to expose as well. The implicit rule makes these more verbose, and results in more additional annotations. Not only that, but it causes greater mental overhead for having to remember if we're in the context of a distributed actor, and if a `func` didn't just accidentally get exposed as remotely accessible endpoint. We also noticed a few soundness and additional complexity in regard to protocol conformances that we found quite tricky. + +We gave this alternative design idea significant thought and strongly favor the explicit distributed rule. + +### Declaring actors and methods as "`distributable`" + +Naming of distributed actors has been debated and while it is true that `distributed` means "may be distributed (meaning 'remote') or not", this is not really the mindset we want to promote with distributed actors. The mental mindset should be that these are distributed and we must treat them this way, and they may happen to be local. Locality is the special case, distribution is the capability we're working with while designing location transparent actors. While we do envision the use of "known to be local" distributed actors, this is better solved with either a `worker.whenLocal { ...` API or allowing marking types with a `local` keyword - either approaches are not part of this proposal and will be pitched in dependently. + +The `distributed` keyword functions the same way as `async` on methods. Async methods are not always asynchronous. The `async` keyword merely means that such method _may suspend_. Similarly, a `distributed func` may or may not perform a remote call, as such the semantics follow the same "beware, the more expensive thing may happen" style of marking methods. + +### Unconditionally conforming `DistributedActor` to `Codable` + +This was part of an earlier design, where the distributed actor protocol was declared as: + +```swift +protocol DistributedActor: AnyActor, Sendable, Codable, ... { ... } +``` + +forcing all implementations of distributed actors to implement the Codable `init(from:)` initializer and `encode(to:)` method. + +While we indeed to expect `Codable` to play a large role in some distributed actor implementations, we have specific use-cases in mind where: + +- Codable might not be used _at all_, thus the re-design and strong focus on being serialization mechanism agnostic in the proposal, by introducing the `SerializationRequirement` associated type. +- Some distributed actor runtimes may behave more like "services" which are _not_ meant to be "passed around" to other nodes. This capability has been explicitly requested by some early adopters in IPC scenarios, where it will help to clean up vague and hacky solutions today, with a clear model where some distributed actors are Codable and thus "pass-around-able" and some are not, depending on the specifics how they were created. + +As such, we are left with no other implementation approach other than the implicit conformance, because it is not possible to add the `Codable` conformance to types managed by a distributed actor system that _wants to_ make distributed actors Codable otherwise (i.e. it is not possible to express `extension DistributedActor: Codable where ID: Codable {}` in today's Swift). Alternative approaches force implementations into casting and doing unsafe tricksy and lose out on the type-safety of only passing Codable actors to distributed methods. + +For distributed actor systems which _do not_ use `Codable`, forcing them to implement Codable methods and initializers would be quite a problem and the implementations would likely be implemented as just crashing. Implementations may force actors to conform to some other protocol, like `IPCServiceDistributedActor` which conforms to the `SerializationRequirement` and attempts to initialize an actor which does not conform to this protocol can crash eagerly, at initialization time. This way actor system authors gain the same developer experience as using `Codable` for passing distributed actors through distributed methods, but the initialization can be specialized -- as it is intended to, because libraries may require specific things from actor types after all. + +### Introducing "wrapper" type for `Distributed` + +We did consider (and have implemented, assisted by swift-syntax based source-generation) the idea of wrapping distributed actors using some "wrapper" type, that would delegate calls to all distributed functions, but prevent access to e.g. stored properties wrapped by such instance. + +This loses the benefit that a proper nominal type distributed actor offers though: the easy to incrementally move actors to distribution as it becomes necessary. The complexity of forming the "call forwarding" functions is also problematic, and extensions to such types would be confusing, would we have to do extensions like this? + +```swift +extension Distributed where Actor == SomeActor { + func hi() { ... } +} +``` + +while _also_ forwarding to functions extended on the `SomeActor` itself? + +```swift +extension SomeActor { + func hi() { ... } // conflict? +} +``` + +What would that mean for when we try to call `hi()` on a distributed actor? It also does not really simplify testing, as we want to test the actual actor, but also the distributed functions actually working correctly (i.e. enforcing serialization constraints on parameters). + +### Creating only a library and/or source-generation tool + +While this may be a highly subjective and sensitive topic, we want to tackle the question up-front, so why are distributed actors better than "just" some RPC library? + +The answer lies in the language integration and the mental model developers can work with when working with distributed actors. Swift already embraces actors for its local concurrency programming, and they will be omni-present and become a familiar and useful tool for developers. It is also important to notice that any async function may be technically performing work over network, and it is up to developers to manage such calls in order to not overwhelm the network etc. With distributed actors, such calls are more _visible_ because IDEs have the necessary information to e.g. underline or otherwise highlight that a function is likely to hit the network and one may need to consider its latency more, than if it was just a local call. IDEs and linters can even use this statically available information to write hints such as "hey, you're doing this distributed actor call in a tight loop - are you sure you want to do that?" + +Distributed actors, unlike "raw" RPC frameworks, help developers think about their distributed applications in terms of a network of collaborating actors, rather than having to think and carefully manage every single serialization call and network connection management between many connected peers - which we envision to be more and more important in the future of device and server programming et al. You may also refer to the [Swift Concurrency Manifesto; Part 4: Improving system architecture](https://gist.github.com/lattner/31ed37682ef1576b16bca1432ea9f782#part-4-improving-system-architecture) section for some other ideas on the topic. + +This does _not_ mean that we shun RPC style libraries or plain-old HTTP clients and libraries similar to them, which may rather be expressed as non-actor types with asynchronous functions. They still absolutely have their place, and we do not envision distributed actors fully replacing them - they are fantastic for cross-language communication, however distributed actors offer a vastly superior programming model, while we remain mostly within Swift and associated actor implementations (we *could*, communicate with non-swift actors over the network, however have not invested into this yet). We do mean however that extending the actor model to its natural habitat (networking) will enable developers to build some kinds of interactive multi-peer/multi-node systems far more naturally than each time having to re-invent a similar abstraction layer, never quite reaching the integration smoothness as language provided integration points such as distributed actors can offer. + +## Acknowledgments & Prior Art + +We would like to acknowledge the prior art in the space of distributed actor systems which have inspired our design and thinking over the years. Most notably we would like to thank the Akka and Orleans projects, each showing independent innovation in their respective ecosystems and implementation approaches. As these are library-only solutions, they have to rely on wrapper types to perform the hiding of information, and/or source generation; we achieve the same goal by expanding the already present in Swift actor-isolation checking mechanisms. + +We would also like to acknowledge the Erlang BEAM runtime and Elixir language for a more modern take built upon the on the same foundations, which have greatly inspired our design, however take a very different approach to actor isolation (i.e. complete isolation, including separate heaps for actors). + +## Source compatibility + +This change is purely additive to the source language. + +The additional use of the keyword `distributed` in `distributed actor` and `distributed func` applies more restrictive requirements to the use of such an actor, however this only applies to new code, as such no existing code is impacted. + +Marking an actor as distributed when it previously was not is potentially source-breaking, as it adds additional type checking requirements to the type. + +## Effect on ABI stability + +None. + +## Effect on API resilience + +None. + +## Changelog + +- 1.3.1 Minor cleanups + - Allow `private distributed func` + - Allow generic distributed actor declarations +- 1.3 More about serialization typechecking and introducing mentioned protocols explicitly + - Revisions Introduce `DistributedActor` and `DistributedActorSystem` protocols properly + - Discuss future directions for versioning and evolving APIs + - Introduce conditional Codable conformance of distributed actors, based on ID + - Discuss `SerializationRequirement` driven typechecking of distributed methods + - Discuss `DistributedActorSystem` parameter requirement in required initializers + - Discuss isolation states in depth "isolated", "known to be local", "potentially remote" and their effect on implicit effects on call-sites +- 1.2 Drop implicitly distributed methods +- 1.1 Implicitly distributed methods +- 1.0 Initial revision +- [Pitch: Distributed Actors](https://forums.swift.org/t/pitch-distributed-actors/51669) + - Which focused on the general concept of distributed actors, and will from here on be cut up in smaller, reviewable pieces that will become their own independent proposals; Similar to how Swift Concurrency is a single coherent feature, however was introduced throughout many interconnected Swift Evolution proposals. diff --git a/proposals/0337-support-incremental-migration-to-concurrency-checking.md b/proposals/0337-support-incremental-migration-to-concurrency-checking.md new file mode 100644 index 0000000000..f3804f5a8f --- /dev/null +++ b/proposals/0337-support-incremental-migration-to-concurrency-checking.md @@ -0,0 +1,286 @@ +# Incremental migration to concurrency checking + +* Proposal: [SE-0337](0337-support-incremental-migration-to-concurrency-checking.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Ben Cohen](https://github.com/AirspeedSwift) +* Status: **Implemented (Swift 5.6)** +* Upcoming Feature Flag: `StrictConcurrency` (Implemented in Swift 6.0) (Enabled in Swift 6 language mode) +* Implementation: [Pull request](https://github.com/apple/swift/pull/40680), [Linux toolchain](https://ci.swift.org/job/swift-PR-toolchain-Linux/761//artifact/branch-main/swift-PR-40680-761-ubuntu16.04.tar.gz), [macOS toolchain](https://ci.swift.org/job/swift-PR-toolchain-osx/1256//artifact/branch-main/swift-PR-40680-1256-osx.tar.gz) + +## Introduction + +Swift 5.5 introduced mechanisms to eliminate data races from the language, including the `Sendable` protocol ([SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md)) to indicate which types have values that can safely be used across task and actor boundaries, and global actors ([SE-0316](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0316-global-actors.md)) to help ensure proper synchronization with (e.g.) the main actor. However, Swift 5.5 does not fully enforce `Sendable` nor all uses of the main actor because interacting with modules which have not been updated for Swift Concurrency was found to be too onerous. We propose adding features to help developers migrate their code to support concurrency and interoperate with other modules that have not yet adopted it, providing a smooth path for the Swift ecosystem to eliminate data races. + +Swift-evolution threads: [[Pitch] Staging in `Sendable` checking](https://forums.swift.org/t/pitch-staging-in-sendable-checking/51341), [Pitch #2](https://forums.swift.org/t/pitch-2-staging-in-sendable-checking/52413), [Pitch #3](https://forums.swift.org/t/pitch-3-incremental-migration-to-concurrency-checking/53610) + +## Motivation + +Swift Concurrency seeks to provide a mechanism for isolating state in concurrent programs to eliminate data races. The primary mechanism is `Sendable` checking. APIs which send data across task or actor boundaries require their inputs to conform to the `Sendable` protocol; types which are safe to send declare conformance, and the compiler checks that these types only contain `Sendable` types, unless the type's author explicitly indicates that the type is implemented so that it uses any un-`Sendable` contents safely. + +This would all be well and good if we were writing Swift 1, a brand-new language which did not need to interoperate with any existing code. Instead, we are writing Swift 6, a new version of an existing language with millions of lines of existing libraries and deep interoperation with C and Objective-C. None of this code specifies any of its concurrency behavior in a way that `Sendable` checking can understand, but until it can be updated, we still want to use it from Swift. + +There are several areas where we wish to address adoption difficulties. + +### Adding retroactive concurrency annotations to libraries + +Many existing APIs should be updated to formally specify concurrency behavior that they have always followed, but have not been able to describe to the compiler until now. For instance, it has always been the case that most UIKit methods and properties should only be used on the main thread, but before the `@MainActor` attribute, this behavior could only be documented and asserted in the implementation, not described to the compiler. + +Thus, many modules should undertake a comprehensive audit of their APIs to decide where to add concurrency annotations. But if they try to do so with the tools they currently have, this will surely cause source breaks. For instance, if a method is marked `@MainActor`, projects which have not yet adopted Swift Concurrency will be unable to call it even if they are using it correctly, because the project does not yet have the annotations to *prove to the compiler* that the call will run in the main actor. + +In some cases, these changes can even cause ABI breaks. For instance, `@Sendable` attributes on function types and `Sendable` constraints on generic parameters are incorporated into mangled function names, even though `Sendable` conformances otherwise have no impact on the calling convention (there isn't an extra witness table parameter, for instance). A mechanism is needed to enforce these constraints during typechecking, but generate code as though they do not exist. + +Here, we need: + +* A formal specification of a "compatibility mode" for pre-concurrency code which imports post-concurrency modules + +* A way to mark declarations as needing special treatment in this "compatibility mode" because their signatures were changed for concurrency + +### Adopting `Sendable` checking before the modules you use have been updated + +The process of auditing libraries to add concurrency annotations will take a long time. We don't think it's realistic for each module to wait until all of its libraries have been updated before they can start adopting `Sendable` checking. + +This means modules need a way to work around incomplete annotations in their imports--either by tweaking the specifications of imported declarations, or by telling the compiler to ignore errors. Whatever mechanism we use, we don't want it to be too verbose, though; for example, marking every single variable of a non-`Sendable` type which we want to treat as `Sendable` would be pretty painful. + +We must also pay special attention to what happens when the library finally *does* add its concurrency annotations, and they reveal that a client has made a mistaken assumption about its concurrency behavior. For instance, suppose you import type `Point` from module `Geometry`. You enable `Sendable` checking before `Geometry`'s maintainers have added concurrency annotations, so it diagnoses a call that sends a `Point` to a different actor. Based on the publicly-known information about `Point`, you decide that this type is probably `Sendable`, so you silence this diagnostic. However, `Geometry`'s maintainers later examine the implementation of `Point` and determine that it is *not* safe to send, so they mark it as non-`Sendable`. What should happen when you get the updated version of `Geometry` and rebuild your project? + +Ideally, Swift should not continue to suppress the diagnostic about this bug. After all, the `Geometry` team has now marked the type as non-`Sendable`, and that is more definitive than your guess that it would be `Sendable`. On the other hand, it probably shouldn't *prevent* you from rebuilding your project either, because this bug is not a regression. The updated `Geometry` module did not add a bug to your code; your code was already buggy. It merely *revealed* that your code was buggy. That's an improvement on the status quo--a diagnosed bug is better than a hidden one. + +But if Swift reacts to this bug's discovery by preventing you from building a module that built fine yesterday, you might have to put off updating the `Geometry` module or even pressure `Geometry`'s maintainers to delay their update until you can fix it, slowing forward progress. So when your module assumes something about an imported declaration that is later proven to be incorrect, Swift should emit a *warning*, not an error, about the bug, so that you know about the bug but do not have to correct it just to make your project build again. + +Here, we need: + +* A mechanism to silence diagnostics about missing concurrency annotations related to a particular declaration or module + +* Rules which cause those diagnostics to return once concurrency annotations have been added, but only as warnings, not errors + +## Proposed solution + +We propose a suite of features to aid in the adoption of concurrency annotations, especially `Sendable` checking. These features are designed to enable the following workflow for adopting concurrency checking: + +1. Enable concurrency checking, by adopting concurrency features (such as `async/await` or actors), enabling Swift 6 mode, or adding the `-warn-concurrency` flag. This causes new errors or warnings to appear when concurrency constraints are violated. + +2. Start solving those problems. If they relate to types from another module, a fix-it will suggest using a special kind of import, `@preconcurrency import`, which silences these warnings. + +3. Once you've solved these problems, integrate your changes into the larger build. + +4. At some future point, a module you import may be updated to add `Sendable` conformances and other concurrency annotations. If it is, and your code violates the new constraints, you will see warnings telling you about these mistakes; these are latent concurrency bugs in your code. Correct them. + +5. Once you've fixed those bugs, or if there aren't any, you will see a warning telling you that the `@preconcurrency import` is unnecessary. Remove the `@preconcurrency` attribute. Any `Sendable`-checking failures involving that module from that point forward will not suggest using `@preconcurrency import` and, in Swift 6 mode, will be errors that prevent your project from building. + +Achieving this will require several features working in tandem: + +* In Swift 6 mode, all code will be checked completely for missing `Sendable` conformances and other concurrency violations, with mistakes generally diagnosed as errors. The `-warn-concurrency` flag will diagnose these violations as warnings in older language versions. + +* When applied to a nominal declaration, the `@preconcurrency` attribute specifies that a declaration was modified to update it for concurrency checking, so the compiler should allow some uses in Swift 5 mode that violate concurrency checking, and generate code that interoperates with pre-concurrency binaries. + +* When applied to an `import` statement, the `@preconcurrency` attribute tells the compiler that it should only diagnose `Sendable`-requiring uses of non-`Sendable` types from that module if the type explicitly declares a `Sendable` conformance that is unavailable or has constraints that are not satisfied; even then, this will only be a warning, not an error. + + +## Detailed design + +### Recovery behavior + +When this proposal speaks of an error being emitted as a warning or suppressed, it means that the compiler will recover by behaving as though (in order of preference): + +* A nominal type that does not conform to `Sendable` does. + +* A function type with an `@Sendable` or global actor attribute doesn't have it. + +### Concurrency checking modes + +Every scope in Swift can be described as having one of two "concurrency checking modes": + +* **Strict concurrency checking**: Missing `Sendable` conformances or global-actor annotations are diagnosed. In Swift 6, these will generally be errors; in Swift 5 mode and with nominal declarations visible via `@preconcurrency import` (defined below), these diagnostics will be warnings. + +* **Minimal concurrency checking**: Missing `Sendable` conformances or global-actor annotations are diagnosed as warnings; on nominal declarations, `@preconcurrency` (defined below) has special effects in this mode which suppress many diagnostics. + +The top level scope's concurrency checking mode is: + +* **Strict** when the module is being compiled in Swift 6 mode or later, when the `-warn-concurrency` flag is used with an earlier language mode, or when the file being parsed is a module interface. + +* **Minimal** otherwise. + +A child scope's concurrency checking mode is: + +* **Strict** if the parent's concurrency checking mode is **Minimal** and any of the following conditions is true of the child scope: + + * It is a closure with an explicit global actor attribute. + + * It is a closure or autoclosure whose type is `async` or `@Sendable`. (Note that the fact that the parent scope is in Minimal mode may affect whether the closure's type is inferred to be `@Sendable`.) + + * It is a declaration with an explicit `nonisolated` or global actor attribute. + + * It is a function, method, initializer, accessor, variable, or subscript which is marked `async` or `@Sendable`. + + * It is an `actor` declaration. + +* Otherwise, the same as the parent scope's. + +> Implementation note: The logic for determining whether a child scope is in Minimal or Strict mode is currently implemented in `swift::contextRequiresStrictConcurrencyChecking()`. + +Imported C declarations belong to a scope with Minimal concurrency checking. + +### `@preconcurrency` attribute on nominal declarations + +To describe their concurrency behavior, maintainers must change some existing declarations in ways which, by themselves, could be source-breaking in pre-concurrency code or ABI-breaking when interoperating with previously-compiled binaries. In particular, they may need to: + +* Add `@Sendable` or global actor attributes to function types +* Add `Sendable` constraints to generic signatures +* Add global actor attributes to declarations + +When applied to a nominal declaration, the `@preconcurrency` attribute indicates that a declaration existed before the module it belongs to fully adopted concurrency, so the compiler should take steps to avoid these source and ABI breaks. It can be applied to any `enum`, enum `case`, `struct`, `class`, `actor`, `protocol`, `var`, `let`, `subscript`, `init` or `func` declaration. + +When a nominal declaration uses `@preconcurrency`: + +* Its name is mangled as though it does not use any of the listed features. + +* At use sites whose enclosing scope uses Minimal concurrency checking, the compiler will suppress any diagnostics about mismatches in these traits. + +* The ABI checker will remove any use of these features when it produces its digests. + +Objective-C declarations are always imported as though they were annotated with `@preconcurrency`. + +For example, consider a function that can only be called on the main actor, then runs the provided closure on a different task: + +```swift +@MainActor func doSomethingThenFollowUp(_ body: @Sendable () -> Void) { + // do something + Task.detached { + // do something else + body() + } +} +``` + +This function could have existed before concurrency, without the `@MainActor` and `@Sendable` annotations. After adding these concurrency annotations, code that worked previously would start producing errors: + +```swift +class MyButton { + var clickedCount = 0 + + func onClicked() { // always called on the main thread by the system + doSomethingThenFollowUp { // ERROR: cannot call @MainActor function outside the main actor + clickedCount += 1 // ERROR: captured 'self' with non-Sendable type `MyButton` in @Sendable closure + } + } +} +``` + +However, if we add `@preconcurrency` to the declaration of `doSomethingThenFollowUp`, its type is adjusted to remove both the `@MainActor` and the `@Sendable`, eliminating the errors and providing the same type inference from before concurrency was adopted by `doSomethingThenFollowUp`. The difference is visible in the type of `doSomethingThenFollowUp` in a minimal vs. a strict context: + +```swift +func minimal() { + let fn = doSomethingThenFollowUp // type is (( )-> Void) -> Void +} + +func strict() async { + let fn = doSomethingThenFollowUp // type is @MainActor (@Sendable ( )-> Void) -> Void +} +``` + +### `Sendable` conformance status + +A type can be described as having one of the following three `Sendable` conformance statuses: + +* **Explicitly `Sendable`** if it actually conforms to `Sendable`, whether via explicit declaration or because the `Sendable` conformance was inferred based on the rules specified in [SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md). + +* **Explicitly non-`Sendable`** if a `Sendable` conformance has been declared for the type, but it is not available or has constraints the type does not satisfy, *or* if the type was declared in a scope that uses Strict concurrency checking.[2] + +* **Implicitly non-`Sendable`** if no `Sendable` conformance has been declared on this type at all. + +> [2] This means that, if a module is compiled with Swift 6 mode or the `-warn-concurrency` flag, all of its types are either explicitly `Sendable` or explicitly non-`Sendable`. + +A type can be made explicitly non-`Sendable` by creating an unavailable conformance to `Sendable`, e.g., + +```swift +@available(*, unavailable) +extension Point: Sendable { } +``` + +Such a conformance suppresses the implicit conformance of a type to `Sendable`. + +### `@preconcurrency` on `Sendable` protocols + +Some number of existing protocols describe types that should all be `Sendable`. When such protocols are updated for concurrency, they will likely inherit from the `Sendable` protocol. However, doing so will break existing types that conform to the protocol and are now assumed to be `Sendable`. This problem was [described in SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md#thrown-errors) because it affects the `Error` and `CodingKey` protocols from the standard library: + +```swift +protocol Error: /* newly added */ Sendable { ... } + +class MutableStorage { + var counter: Int +} +struct ProblematicError: Error { + var storage: MutableStorage // error: Sendable struct ProblematicError has non-Sendable stored property of type MutableStorage +} +``` + +To address this, SE-0302 says the following about the additional of `Sendable` to the `Error` protocol: + +> To ease the transition, errors about types that get their `Sendable` conformances through `Error` will be downgraded to warnings in Swift < 6. + +We propose to replace this bespoke rule for `Error` and `CodingKey` to apply to every protocol that is annotated with `@preconcurrency` and inherits from `Sendable`. These two standard-library protocols will use `@preconcurrency`: + +```swift +@preconcurrency protocol Error: Sendable { ... } +@preconcurrency protocol CodingKey: Sendable { ... } +``` + +### `@preconcurrency` attribute on `import` declarations + +The `@preconcurrency` attribute can be applied to an `import` declaration to indicate that the compiler should reduce the strength of some concurrency-checking violations caused by types imported from that module. You can use it to import a module which has not yet been updated with concurrency annotations; if you do, the compiler will tell you when all of the types you need to be `Sendable` have been annotated. It also serves as a temporary escape hatch to keep your project compiling until any mistaken assumptions you had about that module are fixed. + +When an import is marked `@preconcurrency`, the following rules are in effect: + +* If an implicitly non-`Sendable` type is used where a `Sendable` type is needed: + + * If the type is visible through a `@preconcurrency import`, the diagnostic is suppressed (prior to Swift 6) or emitted as a warning (in Swift 6 and later). + + * Otherwise, the diagnostic is emitted normally, but a separate diagnostic is provided recommending that `@preconcurrency import` be used to work around the issue. + +* If an explicitly non-`Sendable` type is used where a `Sendable` type is needed: + + * If the type is visible through an `@preconcurrency import`, a warning is emitted instead of an error, even in Swift 6. + + * Otherwise, the diagnostic is emitted normally. + +* If the `@preconcurrency` attribute is unused[3], a warning will be emitted recommending that it be removed. + +> [3] We don't define "unused" more specifically because we aren't sure if we can refine it enough to, for instance, recommend removing one of a pair of `@preconcurrency` imports which both import an affected type. + +## Source compatibility + +This proposal is largely motivated by source compatibility concerns. Correct use of `@preconcurrency` should prevent source breaks in code built with Minimal concurrency checking, and `@preconcurrency import` temporarily weakens concurrency-checking rules to preserve source compatibility if a project adopts Full or Strict concurrency checking before its dependencies have finished adding concurrency annotations. + +## Effect on ABI stability + +By itself, `@preconcurrency` does not change the ABI of a declaration. If it is applied to declarations which have already adopted one of the features it affects, that will create an ABI break. However, if those features are added at the same time or after `@preconcurrency` is added, adding those features will *not* break ABI. + +`@preconcurrency`'s tactic of disabling `Sendable` conformance errors is compatible with the current ABI because `Sendable` was designed to not emit additional metadata, have a witness table that needs to be passed, or otherwise impact the calling convention or most other parts of the ABI. It only affects the name mangling. + +This proposal should not otherwise affect ABI. + +## Effect on API resilience + +`@preconcurrency` on nominal declarations will need to be printed into module interfaces. It is effectively a feature to allow the evolution of APIs in ways that would otherwise break resilience. + +`@preconcurrency` on `import` statements will not need to be printed into module interfaces; since module interfaces use the Strict concurrency checking mode, where concurrency diagnostics are warnings, they have enough "wiggle room" to tolerate the missing conformances. (As usual, compiling a module interface silences warnings by default.) + +## Alternatives considered + +### A "concurrency epoch" + +If the evolution of a given module is tied to a version that can be expressed in `@available`, it is likely that there will be some specific version where it retroactively adds concurrency annotations to its public APIs, and that thereafter any new APIs will be "born" with correct concurrency annotations. We could take advantage of this by allowing the module to specify a particular version when it started ensuring that new APIs were annotated and automatically applying `@preconcurrency` to APIs available before this cutoff. + +This would save maintainers from having to manually add `@preconcurrency` to many of the APIs they are retroactively updating. However, it would have a number of limitations: + +1. It would only be useful for modules used exclusively on Darwin. Non-Darwin or cross-platform modules would still need to add `@preconcurrency` manually. + +2. It would only be useful for modules which are version-locked with either Swift itself or a Darwin OS. Modules in the package ecosystem, for instance, would have little use for it. + +3. In practice, version numbers may be insufficiently granular for this task. For instance, if a new API is added at the beginning of a development cycle and it is updated for concurrency later in that cycle, you might mistakenly assume that it will automatically get `@preconcurrency` when in fact you will need to add it by hand. + +Since these shortcomings significantly reduce its applicability, and you only need to add `@preconcurrency` to declarations you are explicitly editing (so you are already very close to the place where you need to add it), we think a concurrency epoch is not worth the trouble. + +### Objective-C and `@preconcurrency` + +Because all Objective-C declarations are implicitly `@preconcurrency`, there is no way to force concurrency APIs to be checked in Minimal-mode code, even if they are new enough that there should be no violating uses. We think this limitation is acceptable to simplify the process of auditing large, existing Objective-C libraries. diff --git a/proposals/0338-clarify-execution-non-actor-async.md b/proposals/0338-clarify-execution-non-actor-async.md new file mode 100644 index 0000000000..d7d8d29aae --- /dev/null +++ b/proposals/0338-clarify-execution-non-actor-async.md @@ -0,0 +1,224 @@ +# Clarify the Execution of Non-Actor-Isolated Async Functions + +* Proposal: [SE-0338](0338-clarify-execution-non-actor-async.md) +* Author: [John McCall](https://github.com/rjmccall) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.7)** ([Decision notes](https://forums.swift.org/t/accepted-se-0338-clarify-the-execution-of-non-actor-isolated-async-functions/54929)) + +## Introduction + +[SE-0306](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md), which introduced actors to Swift, states that `async` functions may be actor-isolated, meaning that they formally run on some actor's executor. Nothing in either SE-0306 or [SE-0296](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md) (`async`/`await`) ever specifies where asynchronous functions that *aren't* actor-isolated run. This proposal clarifies that they do not run on any actor's executor, and it tightens up the rules for [sendability checking](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md) to avoid a potential data race. + +## Motivation + +It is sometimes important that programmers be able to understand which executor is formally responsible for running a particular piece of code. A function that does a large amount of computation on an actor's executor will prevent other tasks from making progress on that actor. The proper isolation of a value may also depend on only accessing it from a particular executor (see note below). Furthermore, in some situations the current executor has other semantic impacts, such as being "inherited" by tasks created with the `Task` initializer. Therefore, Swift needs to provide an intuitive and comprehensible rule for which executors are responsible for running which code. + +> Note: Swift will enforce the correct isolation of data by default with `Sendable` checking. However, this will not be fully enabled until code adopts a future language mode (probably Swift 6). Even under that mode, it will be possible to opt out using `@unsafe Sendable`, making safety the programmer's responsibility. And even if neither of those caveats were true, it would still be important for programmers to be able to understand the execution rules in order to understand how best to fix an isolation error. + +In the current implementation of Swift, `async` functions that aren't actor-isolated never intentionally change the current executor. That is, whenever execution enters such an `async` function, it will continue on whatever the current executor is, with no specific preference for any particular executor. + +To be slightly more precise, we can identify three principle ways that execution can enter an `async` function: + +- It's called by some other `async` function. +- It calls some other `async` function which then returns, resuming the caller. +- It needs to suspend for internal reasons (perhaps it uses `withContinuation` or calls a runtime function that suspends), and it is resumed after that suspension. + +In the current implementation, calls and returns from actor-isolated functions will continue running on that actor's executor. As a result, actors are effectively "sticky": once a task switches to an actor's executor, they will remain there until either the task suspends or it needs to run on a different actor. But if a task suspends within a non-actor-isolated function for a different reason than a call or return, it will generally resume on a non-actor executor. + +This rule perhaps makes sense from the perspective of minimizing switches between executors, but it has several unfortunate consequences. It can lead to unexpected "overhang", where an actor's executor continues to be tied up long after it was last truly needed. An actor's executor can be surprisingly inherited by tasks created during this overhang, leading to unnecessary serialization and contention for the actor. It also becomes unclear how to properly isolate data in such a function: some data accesses may be safe because of the executor the function happens to run on dynamically, but it is unlikely that this is guaranteed by the system. All told, it is a very dynamic rule which interacts poorly with how the rest of concurrency is generally understood, both by Swift programmers and statically by the Swift implementation. + +## Proposed solution + +`async` functions that are not actor-isolated should formally run on a generic executor associated with no actor. Such functions will formally switch executors exactly like an actor-isolated function would: on any entry to the function, including calls, returns from calls, and resumption from suspension, they will switch to a generic, non-actor executor. If they were previously running on some actor's executor, that executor will become free to execute other tasks. + +```swift +extension MyActor { + func update() async { + // This function is actor-isolated, so formally we switch to the actor. + // as soon as it is called. + + // Here we call a function which is not actor-isolated. + let update = await session.readConsistentUpdate() + + // Now we resume executing the function, so formally we switch back to + // the actor. + name = update.name + age = update.age + } +} + +extension MyNetworkSession { + func readConsistentUpdate() async -> Update { + // This function is not actor-isolated, so formally we switch to a + // generic executor when it's called. So if we happen to be called + // from an actor-isolated function, we will immediately switch off the + // actor here. + + // This code runs without any special isolation. + + // Keep calling readUpdate until it returns the same thing twice in a + // row. If that never happens in 1000 different calls, just return the + // last update. This code is just for explanatory purposes; please don't + // expect too much from it. + var update: Update? + for i in 0..<1000 { + // Here we make an async call. + let newUpdate = await readUpdateOnce() + + // Formally, we will switch back to the generic executor after the + // call, so if we happen to have called an actor-isolated function, + // we will immediately switch off of the actor here. + + if update == newUpdate { break } + update = newUpdate + } + return update! + } +} +``` + +## Detailed design + +This proposal changes the semantics of non-actor-isolated `async` functions by specifying that they behave as if they were running on a generic executor not associated with any actor. Technically, the current rule was never written down, so you could say that this proposal *sets* the semantics of these functions; in practice, though, this is an observable change in behavior. + +As a result of this change, the formal executor of an `async` function is always known statically: +- actor-isolated `async` functions always formally run on the actor's executor +- non-actor-isolated `async` functions never formally run on any actor's executor + +This change calls for tasks to switch executors at certain points: +- when the function is called +- when a call made by the function returns +- when the function returns from an internal suspension (e.g. due to a continuation) +As usual, these switches are subject to static and dynamic optimization. These optimizations are the same as are already done with switches to actor executors. + +Statically, if a non-actor-isolated async function doesn't do any significant work before returning, suspending, or making an async call, it can simply remain on the current executor and allow its caller, resumer, or callee to make whatever switches it feels are advisable. This is why this proposal is careful to talk about what executor is *formally* running the task: the actual executor is permitted to be different. Typically, this difference will not be observable, but there are some exceptions. For example, if a function makes two consecutive calls to the same actor, it's possible (but not guaranteed) that the actor will not be given up between them, preventing other work from interleaving. It is outside the scope of this proposal to define what work is "significant". + +Dynamically, a switch will not suspend the task if the task is already on an appropriate executor. Furthermore, some executor changes can be done cheaply without fully suspending the task by giving up the current thread. + +### Sendability + +The `Sendable` rule for calls to non-actor-isolated `async` functions is currently broken. This rule is closely tied to the execution semantics of these functions because of the role of sendability checking in proving the absence of data races. The `Sendable` rule is broken even under the current semantics, but it's arguably even more broken under the proposed rule, so we really do need to fix it as part of this proposal. (There is an alternative which would make the current rule correct, but it doesn't seem advisable; see "Alternatives Considered".) + +It is a basic goal of Swift concurrency that programs should be free of basic data races. In order to achieve this, we must be able to prove that all uses of certain values and memory are totally ordered. All of the code that runs on a particular task is totally ordered with respect to itself. Similarly, all of the code that runs on a particular actor is totally ordered with respect to itself. So, if we can restrict a value/memory to only be used by a single task or actor, we've proven that all of its uses are totally ordered. This is the immediate goal of sendability checking: it prevents non-`Sendable` values from being shared between different concurrent contexts and thus potentially being accessed in non-totally-ordered ways. + +For the purposes of sendability, the concurrent context of an actor-isolated `async` function is the actor. An actor can have non-`Sendable` values in its actor-isolated storage. Actor-isolated functions can read values from that storage into their local state, and similarly they can write values from their local state into actor-isolated storage. Therefore, such functions must strictly separate their "internal" local state from the "external" local state of the task. (It would be possible to be more lenient here, but that is outside the scope of this proposal.) + +The current sendability rule for `async` calls is that the arguments and results of calls to actor-isolated `async` functions must be `Sendable` unless the callee is known to be isolated to the same actor as the caller. Unfortunately, no such restriction is placed on calls to non-isolated `async` functions. That is incorrect under both the current and the proposed execution semantics of such functions because the local state of such functions is not strictly isolated to the actor. + +As a result, the following is allowed: + +```swift +actor MyActor { + var isolated: NonSendableValue + + // Imagine that there are two different tasks calling these two + // functions, and the actor runs the task for `inside_one()` first. + + func inside_one() async { + await outside(argument: isolated) + } + + func inside_two() async { + isolated.operate() + } +} + +// This is a non-actor-isolated async function. +func outside(argument: NonSendableValue) async { + // Under the current execution semantics, when we resume from this + // sleep, we will not be on the actor's executor anymore. + // Under the proposed execution semantics, we will leave the actor's + // executor even before sleeping. + await Task.sleep(nanoseconds: 1_000) + + // In either case, this use of the non-Sendable value can now happen + // concurrently with a use of it on the actor. + argument.operate() +} +``` + +The sendability rule for `async` calls must be changed: the arguments and results of *all* `async` calls must be `Sendable` unless: +- the caller and callee are both known to be isolated to the same actor, or +- the caller and callee are both known to be non-actor-isolated. + +## Source compatibility + +The change to the execution semantics will not break source compatibility. However, it's possible that recompiling code under this proposal will introduce a data race if that code was previously relying on an actor-isolated value passed as an argument to a non-actor-isolation function only being accessed on the actor's executor. There should at least be a warning in this case. + +The change to the sendability rule may break source compatibility for code that has already adopted concurrency. + +In both cases, since Swift's current behavior is clearly undesirable, these seem like necessary changes. There will not be any attempt to maintain compatibility for existing code. + +## Effect on ABI stability + +The change in execution semantics does not require additional runtime support; the compiler will simply emit a different pattern of calls. + +The change in the sendability rule is compile-time and has no ABI impact. + +## Effect on API resilience + +This proposal does not introduce a new feature. + +It may become more difficult to use `async` APIs that take non-`Sendable` arguments. Such APIs are rare and usually aren't a good idea. + +## Alternatives considered + +### Full inheritance of the caller's executor + +One alternative to this would be for `async` functions that aren't actor-isolated to "inherit" the executors of their callers. Essentially, they would record the current executor when they are called, and they would return to that executor whenever they're resumed. + +There are several benefits to this approach: + +- It can be seen as consistent with the behavior of calls to synchronous functions, which of course "inherit" their executor because they have no ability to change it. + +- It significantly improves the overhang problem relative to the current execution semantics. Overhang would be bounded by the end of the call to an actor function, since upon return the caller would resume its own executor. + +- It is the only alternative which would make the current sendability rule for calls to `async` functions correct. + +However, it has three significant drawbacks: + +- While the overhang would be bounded, it would still cover potentially a large amount of code. Everything called by an actor-isolated async function would resume to the actor, which could include a large amount of work that really doesn't need to be actor-isolated. Actors could become heavily contended for artificial and perhaps surprising reasons. + +- It would make it difficult to write code that does leave the actor, since the inheritance would be implicit and recursive. There could be an attribute which avoids the inheritance, but programmers would have to explicitly remember to use it. This is the opposite of Swift's usual language design approach (e.g. with `mutating` methods); it's better to be less permissive by default so that the places which need stronger guarantees are explicit about it. + +- It would substantially impede optimization. Since the current executor would be semantically observable by inheritance, optimizations that remove executor switches would still have to dynamically record the correct executor that should be inherited. Since they currently do not do this, and since there is no efficient support in the runtime for doing this, this would come at a substantial runtime cost. + +### Initial inheritance of the caller's executor + +Another alternative would be to only inherit the executor of the caller for the initial period of execution, from the call to the first suspension. Later resumptions would resume to a generic, non-actor executor. + +This would permit the current sendability rule for arguments, but only if we enforce that the parameters are not used after a suspension in the callee. This is more flexible, but in ways that are highly likely to prove extremely brittle and limiting; a programmer relying on this flexibility is likely to come to regret it. It would also still not permit return values to be non-`Sendable`, so the rule would still need changing. + +The overhang problem would be further improved relative to full inheritance. The only real overhang risk would be a function that does a lot of synchronous work before returning or suspending. + +Sophisticated programmers might be able to use these semantics to avoid some needless switching. It is common for `async` functions to begin with an `async` call, but if Swift has trouble analyzing the code that sets up that call, then under the proposed semantics, Swift might be unable to avoid the initial switch. However, this optimization deficiency equally affects actor-isolated `async` functions, and arguably it ought to have a consistent solution. + +This would still significantly inhibit optimization prior to `async` calls, since the current executor would be observable when (e.g.) creating new tasks with the `Task` initializer. Other situations would be able to optimize freely. + +Using a sendability rule that's sensitive to both data flow and control flow seems like a non-starter; it is far too complex for its rather weak benefits. However, using such a rule is unnecessary, and these execution semantics could instead be combined with the proposed sendability rule. Non-`Sendable` values that are isolated to the actor would not be shareable with the non-actor-isolated function, and uses of non-`Sendable` values created during the initial segment would be totally ordered by virtue of being isolated to the task. + +Overall, while this approach has some benefits over the proposal, it seems better to go with a consistent and wholly static rule for which executor is running any particular `async` function. Allowing a certain amount of inheritance of executors is an interesting future direction. + +## Future directions + +### Explicit inheritance of executors + +There is still room under this proposal for `async` functions to dynamically inherit their executor from their caller. It simply needs to be opt-in rather than opt-out. This does not seem like such an urgent need that it needs to be part of this proposal. + +While `reasync` functions have not yet been proposed, it would probably be reasonable for them to inherit executors, since they deliberately blur the lines between synchronous and asynchronous operation. + +To allow the caller to use a stronger sendability rule, to avoid over-constraining static optimization of switching, and to support a more efficient ABI, this kind of inheritance should be part of the function signature of the callee. + +### Control over executor-switching optimization + +By adding potential switches in non-actor-isolated `async` functions, this proposal puts more pressure on Swift's optimizer to eliminate unnecessary switches. It may be valuable to add a way for programmers to explicitly inform the optimizer that none of the code prior to a suspension is sensitive to the current executor. + +### Distinguishing actor-isolated from task-isolated values + +As discussed above, uses of a non-`Sendable` value may be totally ordered by being restricted to either a consistent task or a consistent actor. The current sendability rules do not distinguish between these cases; instead, all non-`Sendable` values in a function are subject to uniform restrictions. This forces the creation of hard walls between actor-isolated functions and other functions on the same task. A more expressive sendability rule would distinguish these in actor-isolated `async` functions. This would significantly decrease the degree to which this proposal infringes on reasonable expressivity in such functions. + +The default for parameters and return values should probably be task-isolation rather than actor-isolation, so if we're going to consider this, we need to do it soon for optimal results. + +## Acknowledgments + +Many people contributed to the development of this proposal, but I'd like to especially thank Kavon Farvardin for his part in the investigation. diff --git a/proposals/0339-module-aliasing-for-disambiguation.md b/proposals/0339-module-aliasing-for-disambiguation.md new file mode 100644 index 0000000000..a8e1dfdcee --- /dev/null +++ b/proposals/0339-module-aliasing-for-disambiguation.md @@ -0,0 +1,261 @@ +# Module Aliasing For Disambiguation + +* Proposal: [SE-0339](0339-module-aliasing-for-disambiguation.md) +* Authors: [Ellie Shin](https://github.com/elsh) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Pitch: [Module Aliasing](https://forums.swift.org/t/pitch-module-aliasing/51737) +* Implementation: ([toolchain](https://github.com/apple/swift/pull/40899)), +[apple/swift-package-manager#4023](https://github.com/apple/swift-package-manager/pull/4023), others +* Review: ([review](https://forums.swift.org/t/se-0339-module-aliasing-for-disambiguation/54730)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0339-module-aliasing-for-disambiguation/55032)) + +## Introduction + +Swift does not allow multiple modules in a program to share the same name, and attempts to do so will fail to build. These name collisions can happen in a reasonable program when using multiple packages developed independently from each other. This proposal introduces a way to resolve these conflicts without making major, invasive changes to a package's source by turning a module name in source into an alias, a different unique name. + +## Motivation + +As the Swift package ecosystem has grown, programmers have begun to frequently encounter module name clashes, as seen in several forum discussions including [module name 'Logging' clash in Vapor](https://forums.swift.org/t/logging-module-name-clash-in-vapor-3/25466) and [namespacing packages/modules regarding SwiftNIO](https://forums.swift.org/t/namespacing-of-packages-modules-especially-regarding-swiftnio/24726). There are two main use cases where these arise: + +* Two different packages include logically different modules that happen to have the same name. Often, these modules are "internal" dependencies of the package, which would be submodules if Swift supported submodules; for example, it's common to put common utilities into a `Utils` module, which will then collide if more than one package does it. Programmers often run into this problem when adding a new dependency or upgrading an existing one. +* Two different versions of the same package need to be included in the same program. Programmers often run into this problem when trying to upgrade a dependency that another library has pinned to a specific version. Being unable to resolve this collision makes it difficult to gradually update dependencies, forcing migration to be done all at once later. + +In both cases, it is important to be able to resolve the conflict without making invasive changes to the conflicting packages. While submodules might be a better long-term solution for the first case, they are not currently supported by Swift. Even if submodules were supported, they might not always be correctly adopted by packages, and it would not be reasonable for package clients to have to rewrite the package to properly use them. Submodules and other namespacing features would not completely eliminate the need to "retroactively" resolve module name conflicts. + +## Proposed solution + +We believe that module aliasing provides a systematic method for addressing module name collisions. The conflicting modules can be given unique names while still allowing the source code that depends on them to compile. There's already a way to set a module name to a different name, but we need a new aliasing technique that will allow source files referencing the original module names to compile without making source changes. This will be done via new build settings which will then translate to new compiler flags described below. Together, these low-level tools will allow conflicts to be resolved by giving modules a unique name while using aliases to avoid the need to change any source code. + +We propose to introduce the following new settings in SwiftPM. To illustrate the flow, let's go over an example. Consider the following scenario: `App` imports the module `Game`, which imports a module `Utils` from the same package. `App` also imports another module called `Utils` from a different package. This collision might have been introduced when updating to a new version of `Game`'s package, which introduced an "internal" `Utils` module for the first time. + +``` +App + |— Module Game (from package ‘swift-game’) + |— Module Utils (from package ‘swift-game’) + |— Module Utils (from package ‘swift-draw’) +``` + +The modules from each package have the following code: + +```swift +[Module Game] // swift-game + +import Utils // swift-game +public func start(level: Utils.Level) { ... } +``` + +```swift +[Module Utils] // swift-game + +public struct Level { ... } +public var currentLevel: Utils.Level { ... } +``` + +```swift +[Module Utils] // swift-draw + +public protocol Drawable { ... } +public class Canvas: Utils.Drawable { ... } +``` + +Since `App` depends on these two `Utils` modules, we have a conflict, thus we need to rename one. We will introduce a new setting in SwiftPM called `moduleAliases` that will allow setting unique names for dependencies, like so: +```swift + targets: [ + .executableTarget( + name: "App", + dependencies: [ + .product(name: "Game", package: "swift-game", moduleAliases: ["Utils": "GameUtils"]), + .product(name: "Utils", package: "swift-draw"), + ]) + ] +``` + +The setting `moduleAliases` will rename `Utils` from the `swift-game` package as `GameUtils` and alias all its references in the source code to be compiled as `GameUtils`. Since renaming one of the `Utils` modules will resolve the conflict, it is not necessary to rename the other `Utils` module. The references to `Utils` in the `Game` module will be built as `GameUtils` without requiring any source changes. If `App` needs to reference both `Utils` modules in its source code, it can do so by directly including the aliased name: +```swift +[App] + +import GameUtils +import Utils +``` + +Module aliasing relies on being able to change the namespace of all declarations in a module, so initially only pure Swift modules will be supported and users will be required to opt in. Support for languages that give declarations names outside of the control of Swift, such as Objective-C, C, and C++, would be limited as it will require special handling; see the **Requirements / Limitations** section for more details. + + +## Detailed design + +### Changes to Swift Frontend + +Most use cases should just require setting `moduleAliases` in a package manifest. However, it may be helpful to understand how that setting changes the compiler invocations under the hood. In our example scenario, those invocations will change as follows: + +1. First, we need to take the `Utils` module from `swift-game` and rename it `GameUtils`. To do this, we will compile the module as if it were actually named `GameUtils`, while treating any references to `Utils` in its source files as references to `GameUtils`. + 1. The first part (renaming) can be achieved by passing the new module name (`GameUtils`) to `-module-name`. The new module name will also need to be used in any flags specifying output paths, such as `-o`, `-emit-module-path`, or `-emit-module-interface-path`. For example, the binary module file should be built as `GameUtils.swiftmodule` instead of `Utils.swiftmodule`. + 2. The second part (treating references to `Utils` in source files as `GameUtils`) can be achieved with a new compiler flag `-module-alias [name]=[new_name]`. Here, `name` is the module name that appears in source files (`Utils`), while `new_name` is the new, unique name (`GameUtils`). So in our example, we will pass `-module-alias Utils=GameUtils`. + + Putting these steps together, the compiler invocation command would be `swiftc -module-name GameUtils -emit-module-path /path/to/GameUtils.swiftmodule -module-alias Utils=GameUtils ...`. + + For all intents and purposes, the true name of the module is now `GameUtils`. The name `Utils` is no longer associated with it. Module aliases can be used in specific parts of the build to allow source code that still uses the name `Utils` (possibly including the module itself) to continue to compile. +2. Next, we need to build the module `Game`. `Game` contains references to `Utils`, which we need to treat as references to `GameUtils`. We can do this by just passing `-module-alias Utils=GameUtils` without any other changes. The overall compiler invocation command to build `Game` is `swiftc -module-name Game -module-alias Utils=GameUtils ...`. +3. We don't need any build changes when building `App` because the source code in `App` does not expect to use the `Utils` module from `swift-game` under its original name. If `App` tries to import a module named `Utils`, that will refer to the `Utils` module from `swift-draw`, which has not been renamed. If `App` does need to import the `Utils` module from `swift-game`, it must use `import GameUtils`. + + +The arguments to the `-module-alias` flag will be validated against reserved names, invalid identifiers, a wrong format or ordering (`-module-alias Utils=GameUtils` is correct but `-module-alias GameUtils=Utils` is not). The flag can be repeated to allow multiple aliases, e.g. `-module-alias Utils=GameUtils -module-alias Logging=GameLogging`, and will be checked against duplicates. Diagnostics and fix-its will contain the name `Utils` in the error messages as opposed to `GameUtils` to be consistent with the names appearing to users. + +The validated map of aliases will be stored in the AST context and used for dependency scanning/resolution and module loading; from the above scenario, if `Game` is built with `-module-alias Utils=GameUtils` and has `import Utils` in source code, `GameUtils.swiftmodule` should be loaded instead of `Utils.swiftmodule` during import resolution. + +While the name `Utils` appears in source files, the actual binary name `GameUtils` will be used for name lookup, semantic analysis, symbol mangling (e.g. `$s9GameUtils5Level`), and serialization. Since the binary names will be stored during serialization, the aliasing flag will only be needed to build the conflicting modules and their immediate consuming modules; building non-immediate consuming modules will not require the flag. + +Direct references to the renamed modules should only be allowed in source code if multiple conflicting modules need to be imported; in such case, a direct reference in an import statement, e.g. `import GameUtils`, is allowed. Otherwise, the original name `Utils` should be used in source code instead of the binary name `GameUtils`. The module alias map will be used to track when to disallow direct references to the binary module names in source files, and an attempt to use the binary name will result in an error along with a fix-it. This restriction is useful as it can make it easier to rename the module again later if needed, e.g. from `GameUtils` to `SwiftGameUtils`. + +Unlike source files, the generated interface (.swiftinterface) will contain the binary module name in all its references. The binary module name will also be stored for indexing and debugging, and treated as the source of truth. + +### Changes to Code Assistance / Indexing + +The compiler arguments including the new flag `-module-alias` will be available to SourceKit and indexing. The aliases will be stored in the AST context and used to fetch the right results for code completion and other code assistance features. They will also be stored for indexing so features such as `Jump to Definition` can navigate to declarations scoped to the binary module names. + +Generated documentation, quick help, and other assistance features will contain the binary module names, which will be treated as the source of truth. + +### Changes to Swift Driver + +The module aliasing arguments will be used during the dependency scan for both implicit and explicit build modes; the resolved dependency graph will contain the binary module names. In case of the explicit build mode, the dependency input passed to the frontend will contain the binary module names in its json file. Similar to the frontend, validation of the aliasing arguments will be performed at the driver. + +### Changes to SwiftPM + +To make module aliasing more accessible, we will introduce new build configs which can map to the compiler flags for aliasing described above. Let’s go over how they can be adopted by SwiftPM with the above scenario (copied here). +``` +App + |— Module Game (from package ‘swift-game’) + |— Module Utils (from package ‘swift-game’) + |— Module Utils (from package ‘swift-draw’) +``` + +Here are the manifest examples for `swift-game` and `swift-draw`. + +```swift +let package = Package( + name: "swift-game", + dependencies: [], + products: [ + .library(name: "Game", targets: ["Game"]), + .library(name: "Utils", targets: ["Utils"]), + ], + targets: [ + .target(name: "Game", dependencies: ["Utils"]), + .target(name: "Utils", dependencies: []) + ] +) +``` + +```swift +let package = Package( + name: "swift-draw", + dependencies: [], + products: [ + .library(name: "Utils", targets: ["Utils"]), + ], + targets: [ + .target(name: "Utils", dependencies: []) + ] +) +``` + +The `App` manifest needs to explicitly define unique names for the conflicting modules via a new parameter called `moduleAliases`. +```swift +let package = Package( + name: "App", + dependencies: [ + .package(url: https://.../swift-game.git), + .package(url: https://.../swift-draw.git) + ], + products: [ + .executable(name: "App", targets: ["App"]) + ] + targets: [ + .executableTarget( + name: "App", + dependencies: [ + .product(name: "Game", package: "swift-game", moduleAliases: ["Utils": "GameUtils"]), + .product(name: "Utils", package: "swift-draw"), + ]) + ] +) +``` + +SwiftPM will perform validations when it parses `moduleAliases`; for each entry, it will check whether the given alias is a unique name, whether there is a conflict among aliases, whether the specified module is built from source (pre-compiled modules cannot be rebuilt to respect the rename), and whether the module is a pure Swift module (see **Requirements/Limitations** section for more details). + +It will also check if any aliases are defined in upstream packages and override them if necessary. For example, if the `swift-game` package were modified per below and defined its own alias `SwiftUtils` for module `Utils` from a dependency package, the alias defined in `App` will override it, thus the `Utils` module from `swift-utils` will be built as `GameUtils`. + +```swift +let package = Package( + name: "swift-game", + dependencies: [ + .package(url: https://.../swift-utils.git), + ], + products: [ + .library(name: "Game", targets: ["Game"]), + ], + targets: [ + .target(name: "Game", + dependencies: [ + .product(name: "UtilsProduct", + package: "swift-utils", + moduleAliases: ["Utils": "SwiftUtils"]), + ]) + ] +) +``` + +Once the validation and alias overriding steps pass, dependency resolution will take place using the new module names, and the `-module-alias [name]=[new_name]` flag will be passed to the build execution. + + +### Resources + +Tools invoked by a build system to compile resources should be modified to handle the module aliasing. The module name entry should get the renamed value and any references to aliased modules in the resources should correctly map to the corresponding binary names. The resources likely impacted by this are IB, CoreData, and anything that explicitly requires module names. We will initially only support asset catalogs and localized strings as module names are not required for those resources. + +### Debugging + +When module aliasing is used, the binary module name will be stored in mangled symbols, e.g. `$s9GameUtils5Level` instead of `$s5Utils5Level`, which will be stored in Debuginfo. + +For evaluating an expression, the name `Utils` can be used as it appears in source files (which were already compiled with module aliasing); however, the result of the evaluation will contain the binary module name. + +If a module were to be loaded directly into lldb, the binary module name should be used, i.e. `import GameUtils` instead of `import Utils`, since it does not have access to the aliasing flag. + +In REPL, binary module names should be used for importing or referencing; support for aliasing in that mode may be added in the future. + +## Requirements / Limitations + +To allow module aliasing, the following requirements need to be met, which come with some limitations. + +* Only pure Swift modules allowed for aliasing: no ObjC/C/C++/Asm due to potential symbol collision. Similarly, `@objc(name)` is discouraged. +* Building from source only: aliasing distributed binaries is not possible due to the impact on mangling and serialization. +* Runtime: calls to convert String to types in module, i.e direct or indirect calls to `NSClassFromString(...)`, will fail and should be avoided. +* For resources, only asset catalogs and localized strings are allowed. +* Higher chance of running into the following existing issues: + * [Retroactive conformance](https://forums.swift.org/t/retroactive-conformances-vs-swift-in-the-os/14393): this is already not a recommended practice and should be avoided. + * Extension member “leaks”: this is [considered a bug](https://bugs.swift.org/browse/SR-3908) which hasn’t been fixed yet. More discussions [here](https://forums.swift.org/t/pre-pitch-import-access-control-a-modest-proposal/50087). +* Code size increase will be more implicit thus requires a caution, although module aliasing will be opt-in and a size threshold could be added to provide a warning. + +## Source compatibility +This is an additive feature. Currently when there are duplicate module names, it does not compile at all. This feature requires explicitly opting in to allow and use module aliaisng via package manifests or compiler invocation commands and does not require source code changes. + +## Effect on ABI stability +The feature in this proposal does not have impact on the ABI. + +## Effect on API resilience +This proposal does not introduce features that would be part of a public API. + +## Future Directions + +* Currently when a module contains a type with the same name, fully qualifying a decl in the module results in an error; it treats the left most qualifier as a type instead of the module ([SR-14195](https://bugs.swift.org/browse/SR-14195), [pitch](https://forums.swift.org/t/fixing-modules-that-contain-a-type-with-the-same-name/3025), [pitch](https://forums.swift.org/t/pitch-fully-qualified-name-syntax/28482)); `XCTest` is a good example as it contains a class called `XCTest`. Trying to access a top level function `XCTAssertEqual` via `XCTest.XCTAssertEqual(...)` results in `Type 'XCTest' has no member 'XCTAssertEqual'` error. Module aliasing could mitigate this issue by renaming `XCTest` as `XCTestFramework` without requiring source changes in the `XCTest` module and allowing the function access via `XCTestFramework.XCTAssertEqual(...)` in the user code. + +* Introducing new import syntax such as `import Utils as GameUtils` has been discussed in forums to improve module disambiguation. The module aliasing infrastructure described in this proposal paves the way towards using such syntax that could allow more explicit (in source code) aliasing. + +* Visibility change to import decl access level (from public to internal) pitched [here](https://forums.swift.org/t/pre-pitch-import-access-control-a-modest-proposal/50087) could help address the extension leaks issues mentioned in **Requirements / Limitations** section. + +* Swift modules that have C target dependencies could, in a limited capacity, be supported by changing visibility to C symbols. + +* C++ interop support could potentially allow C++ modules to be aliased besides pure Swift modules. + +* Nested namespacing or submodules might be a better long-term solution for some of the collision issues described in **Motivation**. However, it would not completely eliminate the need to "retroactively" resolve module name conflicts. Module aliasing does not introduce any lexical or structural changes that might have an impact on potential future submodules support; it's an orthogonal feature and can be used in conjunction if needed. + +## Acknowledgments +This proposal was improved with feedback and helpful suggestions along with code reviews by Becca Royal-Gordon, Alexis Laferriere, John McCall, Joe Groff, Mike Ash, Pavel Yaskevich, Adrian Prantl, Artem Chikin, Boris Buegling, Anders Bertelrud, Tom Doron, and Johannes Weiss, and others. diff --git a/proposals/0340-swift-noasync.md b/proposals/0340-swift-noasync.md new file mode 100644 index 0000000000..3a95229752 --- /dev/null +++ b/proposals/0340-swift-noasync.md @@ -0,0 +1,375 @@ +# Unavailable From Async Attribute + +* Proposal: [SE-0340](0340-swift-noasync.md) +* Authors: [Evan Wilde](https://github.com/etcwilde) +* Review manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.7)** +* Implementation: [noasync availability](https://github.com/apple/swift/pull/40769) +* Discussion: [Discussion: Unavailability from asynchronous contexts](https://forums.swift.org/t/discussion-unavailability-from-asynchronous-contexts/53088) +* Pitch: [Pitch: Unavailability from asynchronous contexts](https://forums.swift.org/t/pitch-unavailability-from-asynchronous-contexts/53877) +* Review: [SE-0340: Unavailable from Async Attribute](https://forums.swift.org/t/se-0340-unavailable-from-async-attribute/54852) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0340-unavailable-from-async-attribute/55356) + +## Introduction + +The Swift concurrency model allows tasks to resume on different threads from the +one they were suspended on. For this reason, API that relies on thread-local +storage, locks, mutexes, and semaphores, should not be used across suspension +points. + +```swift +func badAsyncFunc(_ mutex: UnsafeMutablePointer, _ op : () async -> ()) async { + // ... + pthread_mutex_lock(mutex) + await op() + pthread_mutex_unlock(mutex) // Bad! May unlock on a different thread! + // ... +} +``` + +The example above exhibits undefined behaviour if `badAsyncFunc` resumes on a +different thread than the one it started on after running `op` since +`pthread_mutex_unlock` must be called from the same thread that locked the +mutex. + +We propose extending `@available` with a new `noasync` availability kind to +indicate API that may not be used directly from asynchronous contexts. + +Swift evolution thread: [Pitch: Unavailability from asynchronous contexts](https://forums.swift.org/t/pitch-unavailability-from-asynchronous-contexts/53877) + +## Motivation + +The Swift concurrency model allows tasks to suspend and resume on different +threads. While this behaviour allows higher utility of computational resources, +there are some nasty pitfalls that can spring on an unsuspecting programmer. One +such pitfall is the undefined behaviour from unlocking a `pthread_mutex_t` from +a different thread than the thread that holds the lock, locking threads may +easily cause unexpected deadlocks, and reading from and writing to thread-local +storage across suspension points may result in unintended behaviour that is +difficult to debug. + +## Proposed Solution + +We propose extending `@available` to accept a `noasync` availability kind. +The `noasync` availability kind is applicable to most declarations, but is not +allowed on destructors as those are not explicitly called and must be callable +from anywhere. + +```swift +@available(*, noasync) +func doSomethingNefariousWithNoOtherOptions() { } + +@available(*, noasync, message: "use our other shnazzy API instead!") +func doSomethingNefariousWithLocks() { } + +func asyncFun() async { + // Error: doSomethingNefariousWithNoOtherOptions is unavailable from + // asynchronous contexts + doSomethingNefariousWithNoOtherOptions() + + // Error: doSomethingNefariousWithLocks is unavailable from asynchronous + // contexts; use our other shanzzy API instead! + doSomethingNefariousWithLocks() +} +``` + +The `noasync` availability attribute only prevents API usage in the immediate +asynchronous context; wrapping a call to an unavailable API in a synchronous +context and calling the wrapper will not emit an error. This allows for cases +where it is possible to use the API safely within an asynchronous context, but +in specific ways. The example below demonstrates this with an example of using a +pthread mutex to wrap a critical section. The function ensures that there cannot +be a suspension point between obtaining and releasing the lock, and therefore is +safe for consumption by asynchronous contexts. + +```swift +func goodAsyncFunc(_ mutex: UnsafeMutablePointer, _ op : () -> ()) async { + // not an error, pthread_mutex_lock is wrapped in another function + with_pthread_mutex_lock(mutex, do: op) +} + +func with_pthread_mutex_lock( + _ mutex: UnsafeMutablePointer, + do op: () throws -> R) rethrows -> R { + switch pthread_mutex_lock(mutex) { + case 0: + defer { pthread_mutex_unlock(mutex) } + return try op() + case EINVAL: + preconditionFailure("Invalid Mutex") + case EDEADLK: + fatalError("Locking would cause a deadlock") + case let value: + fatalError("Unknown pthread_mutex_lock() return value: '\(value)'") + } +} +``` + +The above snippet is a safe wrapper for `pthread_mutex_lock` and +`pthread_mutex_unlock`, since the lock is not held across suspension points. The +critical section operation must be synchronous for this to hold true though. +The following snippet uses a synchronous closure to call the unavailable +function, circumventing the protection provided by the attribute. + +```swift +@available(*, noasync) +func pthread_mutex_lock(_ lock: UnsafeMutablePointer) {} + +func asyncFun(_ mutex : UnsafeMutablePointer) async { + // Error! pthread_mutex_lock is unavailable from async contexts + pthread_mutex_lock(mutex) + + // Ok! pthread_mutex_lock is not called from an async context + _ = { unavailableFun(mutex) }() + + await someAsyncOp() +} +``` + +### Replacement API + +In some cases, it is possible to provide an alternative that is safe. The +`with_pthread_mutex_lock` is an example of a way to provide a safe way to wrap +locking and unlocking pthread mutexes. + +In other cases, it may be safe to use an API from a specific actor. For +example, API that uses thread-local storage isn't safe for consumption by +asynchronous functions in general, but is safe for functions on the MainActor +since it will only use the main thread. + +The unavailable API should still be annotated as such, but an alternative +function can be implemented as an extension of the actors that support the +operation. + +```swift +@available(*, noasync, renamed: "mainactorReadID()", message: "use mainactorReadID instead") +func readIDFromThreadLocal() -> Int { } + +@MainActor +func readIDFromMainActor() -> Int { readIDFromThreadLocal() } + +func asyncFunc() async { + // Bad, we don't know what thread we're on + let id = readIDFromThreadLocal() + + // Good, we know it's coming from the main actor on the main thread. + // Note the suspension due to the jump to the main actor. + let id = await readIDFromMainActor() +} +``` + +Restricting a synchronous API to an actor is done similarly, as demonstrated in +the example below. The synchronous `save` function is part of a public API, so +it can't just be pulled into the `DataStore` actor without causing a source +break. Instead, it is annotated with a `noasync` available attribute. +`DataStore.save` is a thin wrapper around the original synchronous save +function. Calls from an asynchronous context to `save` may only be done through +the `DataStore` actor, ensuring that the cooperative pool isn't tied up with the +save function. The original save function is still available to synchronous code +as it was before. + +```swift +@available(*, noasync, renamed: "DataStore.save()") +public func save(_ line: String) { } + +public actor DataStore { } + +public extension DataStore { + func save(_ line: String) { + save(line) + } +} +``` + +## Additional design details + +Verifying that unavailable functions are not used from asynchronous contexts is +done weakly; only unavailable functions called directly from asynchronous +contexts are diagnosed. This avoids the need to recursively typecheck the bodies +of synchronous functions to determine whether they are implicitly available from +asynchronous contexts, or to verify that they are appropriately annotated. + +While the typechecker doesn't need to emit diagnostics from synchronous +functions, they cannot be omitted entirely. It is possible to declare +asynchronous contexts inside of synchronous contexts, wherein diagnostics should +be emitted. + +```swift +@available(*, noasync) +func bad2TheBone() {} + +func makeABadAsyncClosure() -> () async -> Void { + return { () async -> Void in + bad2TheBone() // Error: Unavailable from asynchronous contexts + } +} +``` + +## Source Compatibility + +Swift 3 and Swift 4 do not have this attribute, so code coming from Swift 3 and +Swift 4 won't be affected. + +The attribute will affect any current asynchronous code that currently contains +use of API that are modified with this attribute later. To ease the transition, +we propose that this attribute emits a warning in Swift 5.6, and becomes a full +error in Swift 6. In cases where someone really wants unsafe behavior and enjoys +living on the edge, the diagnostic is easily circumventable by wrapping the API +in a synchronous closure, noted above. + +## Effect on ABI stability + +This feature has no effect on ABI. + +## Effect on API resilience + +The presence of the attribute has no effect on the ABI. + +## Alternatives Considered + +### Propagation + +The initial discussion focused on how unavailability propagated, including the +following three designs; + - implicitly-inherited unavailability + - explicit unavailability + - thin unavailability + +The ultimate decision is to go with the thin checking; both the implicit and +explicit checking have high performance costs and require far more consideration +as they are adding another color to functions. + +The attribute is expected to be used for a fairly limited set of specialized +use-cases. The goal is to provide some protection without dramatically impacting +the performance of the compiler. + +#### Implicitly inherited unavailability + +Implicitly inheriting unavailability would transitively apply the unavailability +to functions that called an unavailable function. This would have the lowest +developer overhead while ensuring that one could not accidentally use the +unavailable functions indirectly. + +```swift +@unavailableFromAsync +func blarp1() {} + +func blarp2() { + // implicitly makes blarp2 unavailable + blarp1() +} + +func asyncFun() async { + // Error: blarp2 is impicitly unavailable from async because of call to blarp1 + blarp2() +} +``` + +Unfortunately, computing this is very expensive, requiring type-checking the +bodies of every function a given function calls in order to determine if the +declaration is available from an async context. Requiring even partial +type-checking of the function bodies to determine the function declaration is +prohibitively expensive, and is especially detrimental to the performance of +incremental compilation. + +We would need an additional attribute to disable the checking for certain +functions that are known to be usable from an async context, even though they +use contain unavailable functions. An example of a safe, but "unavailable" +function is `with_pthread_mutex_lock` above. + +#### Explicit unavailability + +This design behaves much like availability does today. In order to use an +unavailable function, the calling function must be explicitly annotated with the +unavailability attribute or an error is emitted. + +Like the implicit unavailability propagation, we still need an additional +attribute to indicate that, while a function may contain unsafe API, it uses +them in a way that is safe for use in asynchronous contexts. + +The benefits of this design are that it both ensures that unsafe API are explicitly +handled correctly, avoiding bugs. Additionally, typechecking asynchronous +functions is reasonably performant and does not require recursively +type-checking the bodies of every synchronous function called by the +asynchronous function. + +Unfortunately, we would need to walk the bodies of every synchronous function to +ensure that every synchronous function is correctly annotated. This reverses the +benefits of the implicit availability checking, while having a high developer +overhead. + +### Separate Attribute + +We considered using a separate attribute, spelled `@unavailableFromAsync`, to +annotate the unavailable API. After more consideration, it became apparent that +we would likely need to reimplement much of the functionality of the +`@available` attribute. + +Some thoughts that prompted the move from `@unavailableFromAsync` to an +availability kind include: + + - A given API may have different implementations on different platforms, and + therefore may be implemented in a way that is safe for consumption in + asynchronous contexts in some cases but not others. + - An API may be currently implemented in a way that is unsafe for consumption + in asynchronous contexts, but may be safe in the future. + - We get `message`, `renamed`, and company, with serializations, for free by + merging this with `@available`. + +Challenges to the merge mostly focus on the difference in the verification model +between this and the other availability modes. The `noasync`, as discussed +above, is a weaker check and does not require API that is using the unavailable +function to also be annotated. The other availability checks do require that the +availability information be propagated. + +## Future Directions + +[Custom executors](https://forums.swift.org/t/support-custom-executors-in-swift-concurrency/44425) +are pitched to become part of the language as a future feature. +Restricting an API to a custom executor is the same as restricting that API to +an actor. The difference is that the actor providing the replacement API has +it's `unownedExecutor` overridden with the desired custom executor. + +Hand-waving around some of the syntax, this protection could look something like +the following example: + +```swift +protocol IOActor : Actor { } + +extension IOActor { + nonisolated var unownedExecutor: UnownedSerialExecutor { + return getMyCustomIOExecutor() + } +} + +@available(*, noasync, renamed: "IOActor.readInt()") +func readIntFromIO() -> String { } + +extension IOActor { + // IOActor replacement API goes here + func readInt() -> String { readIntFromIO() } +} + +actor MyIOActor : IOActor { + func printInt() { + // Okay! It's synchronous on the IOActor + print(readInt()) + } +} + +func print(myActor : MyIOActor) async { + // Okay! We only call `readIntFromIO` on the IOActor's executor + print(await myActor.readInt()) +} +``` + +The `IOActor` overrides it's `unownedExecutor` with a specific custom IO +executor and provides a synchronous `readInt` function wrapping a call to the +`readIntFromIO` function. The `noasync` availability attribute ensures that +`readIntFromIO` cannot generally be used from asynchronous contexts. +When `readInt` is called, there will be a hop to the `MyIOActor`, which uses the +custom IO executor. + +## Acknowledgments + +Thank you Becca and Doug for you feedback and help shaping the proposal. diff --git a/proposals/0341-opaque-parameters.md b/proposals/0341-opaque-parameters.md new file mode 100644 index 0000000000..e324461490 --- /dev/null +++ b/proposals/0341-opaque-parameters.md @@ -0,0 +1,270 @@ +# Opaque Parameter Declarations + +* Proposal: [SE-0341](0341-opaque-parameters.md) +* Author: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Ben Cohen](https://github.com/AirspeedSwift) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#40993](https://github.com/apple/swift/pull/40993) + +## Introduction + +Swift's syntax for generics is designed for generality, allowing one to express complicated sets of constraints amongst the different inputs and outputs of a function. For example, consider an eager concatenation operation that builds an array from two sequences: + +```swift +func eagerConcatenate( + _ sequence1: Sequence1, _ sequence2: Sequence2 +) -> [Sequence1.Element] where Sequence1.Element == Sequence2.Element +``` + +There is a lot going on in that function declaration: the two function parameters are of different types determined by the caller, which are captured by `Sequence1` and `Sequence2`, respectively. Both of these types must conform to the `Sequence` protocol and, moreover, the element types of the two sequences must be equivalent. Finally, the result of this operation is an array of the sequence's element type. One can use this operation with many different inputs, so long as the constraints are met: + +```swift +eagerConcatenate([1, 2, 3], Set([4, 5, 6])) // okay, produces an [Int] +eagerConcatenate([1: "Hello", 2: "World"], [(3, "Swift"), (4, "!")]) // okay, produces an [(Int, String)] +eagerConcatenate([1, 2, 3], ["Hello", "World"]) // error: sequence element types do not match +``` + +However, when one does not need to introduce a complex set of constraints, the syntax starts to feel quite heavyweight. For example, consider a function that composes two SwiftUI views horizontally: + +```swift +func horizontal(_ v1: V1, _ v2: V2) -> some View { + HStack { + v1 + v2 + } +} +``` + +There is a lot of boilerplate to declare the generic parameters `V1` and `V2` that are only used once, making this function look far more complex than it really is. The result, on the other hand, is able to use an [opaque result type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md) to hide the specific returned type (which would be complicated to describe), describing it only by the protocols to which it conforms. + +This proposal extends the syntax of opaque result types to parameters, allowing one to specify function parameters that are generic without the boilerplate associated with generic parameter lists. The `horizontal` function above can then be expressed as: + +```swift +func horizontal(_ v1: some View, _ v2: some View) -> some View { + HStack { + v1 + v2 + } +} +``` + +Semantically, this formulation is identical to the prior one, but is simpler to read and understand because the inessential complexity from the generic parameter lists has been removed. It takes two views (the concrete type does not matter) and returns a view (the concrete type does not matter). + +Swift-evolution threads: [Pitch for this proposal](https://forums.swift.org/t/pitch-opaque-parameter-types/54914), [Easing the learning curve for introducing generic parameters](https://forums.swift.org/t/discussion-easing-the-learning-curve-for-introducing-generic-parameters/52891), [Improving UI of generics pitch](https://forums.swift.org/t/improving-the-ui-of-generics/22814) + +## Proposed solution + +This proposal extends the use of the `some` keyword to parameter types for function, initializer, and subscript declarations. As with opaque result types, `some P` indicates a type that is unnamed and is only known by its constraint: it conforms to the protocol `P`. When an opaque type occurs within a parameter type, it is replaced by an (unnamed) generic parameter. For example, the given function: + +```swift +func f(_ p: some P) { } +``` + +is equivalent to a generic function described as follows, with a synthesized (unnamable) type parameter `_T`: + +```swift +func f<_T: P>(_ p: _T) +``` + +Note that, unlike with opaque result types, the caller determines the type of the opaque type via type inference. For example, if we assume that both `Int` and `String` conform to `P`, one can call or reference the function with either `Int` or `String`: + +```swift +f(17) // okay, opaque type inferred to Int +f("Hello") // okay, opaque type inferred to String + +let fInt: (Int) -> Void = f // okay, opaque type inferred to Int +let fString: (String) -> Void = f // okay, opaque type inferred to String +let fAmbiguous = f // error: cannot infer parameter for `some P` parameter +``` + +[SE-0328](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0328-structural-opaque-result-types.md) extended opaque result types to allow multiple uses of `some P` types within the result type, in any structural position. Opaque types in parameters permit the same structural uses, e.g., + +```swift +func encodeAnyDictionaryOfPairs(_ dict: [some Hashable & Codable: Pair]) -> Data +``` + +This is equivalent to: + +```swift +func encodeAnyDictionaryOfPairs<_T1: Hashable & Codable, _T2: Codable, _T3: Codable>(_ dict: [_T1: Pair<_T2, _T3>]) -> Data +``` + +Each instance of `some` within the declaration represents a different implicit generic parameter. + +## Detailed design + +Opaque parameter types can only be used in parameters of a function, initializer, or subscript declaration. They cannot be used in (e.g.) a typealias or any value of function type. For example: + +```swift +typealias Fn = (some P) -> Void // error: cannot use opaque types in a typealias +let g: (some P) -> Void = f // error: cannot use opaque types in a value of function type +``` + +There are additional restrictions on the use of opaque types in parameters where they may conflict with future language features. + +### Variadic generics + +An opaque type cannot be used in a variadic parameter: + +```swift +func acceptLots(_: some P...) +``` + +This restriction is in place because the semantics implied by this proposal might not be the appropriate semantics if Swift gains variadic generics. Specifically, the semantics implied by this proposal itself (without variadic generics) would be equivalent to: + +```swift +func acceptLots<_T: P>(_: _T...) +``` + +where `acceptLots` requires that all of the arguments have the same type: + +```swift +acceptLots(1, 1, 2, 3, 5, 8) // okay +acceptLots("Hello", "Swift", "World") // okay +acceptLots("Swift", 6) // error: argument for `some P` could be either String or Int +``` + +With variadic generics, one might instead make the implicit generic parameter a generic parameter pack, as follows: + +```swift +func acceptLots<_Ts: P...>(_: _Ts...) +``` + +In this case, `acceptLots` accepts any number of arguments, all of which might have different types: + +```swift +acceptLots(1, 1, 2, 3, 5, 8) // okay, Ts contains six Int types +acceptLots("Hello", "Swift", "World") // okay, Ts contains three String types +acceptLots(Swift, 6) // okay, Ts contains String and Int +``` + +### Opaque parameters in "consuming" positions of function types + +The resolution of [SE-0328](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0328-structural-opaque-result-types.md) prohibited the use of opaque parameters in "consuming" positions of function types. For example: + +```swift +func f() -> (some P) -> Void { ... } // error: cannot use opaque type in parameter of function type +``` + +The result of function `f` is fairly hard to use, because there is no way for the caller to easily create a value of an unknown, unnamed type: + +```swift +let fn = f() +fn(/* how do I create a value here? */) +``` + +The same prohibition applies to opaque types that occur within parameters of function type, e.g., + +```swift +func g(fn: (some P) -> Void) { ... } // error: cannot use opaque type in parameter of function type +``` + +The reasoning for this prohibition is similar. In the implementation of `g`, it's hard to produce a value of the type `some P` when that type isn't named anywhere else. + +## Source compatibility + +This is a pure language extension with no backward-compatibility concerns, because all uses of `some` in parameter position are currently errors. + +## Effect on ABI stability + +This proposal has no effect on the ABI or runtime because it is syntactic sugar for generic parameters. + +## Effect on API resilience + +This feature is purely syntactic sugar, and one can switch between using opaque parameter types and the equivalent formulation with explicit generic parameters without breaking either the ABI or API. However, the complete set of constraints must be the same in such cases. + +## Future Directions + +### Constraining the associated types of a protocol + +This proposal composes well with an idea that allows the use of generic syntax to specify the associated type of a protocol, e.g., where `Collection`is "a `Collection` whose `Element` type is `String`". Combined with this proposal, one can more easily express a function that takes an arbitrary collection of strings: + +```swift +func takeStrings(_: some Collection) { ... } +``` + +Recall the complicated `eagerConcatenate` example from the introduction: + +```swift +func eagerConcatenate( + _ sequence1: Sequence1, _ sequence2: Sequence2 +) -> [Sequence1.Element] where Sequence1.Element == Sequence2.Element +``` + +With opaque parameter types and generic syntax on protocol types, one can express this in a simpler form with a single generic parameter representing the element type: + +```swift +func eagerConcatenate( + _ sequence1: some Sequence, _ sequence2: some Sequence +) -> [T] +``` + +And in conjunction with opaque result types, we can hide the representation of the result, e.g., + +```swift +func lazyConcatenate( + _ sequence1: some Sequence, _ sequence2: some Sequence +) -> some Sequence +``` + +### Enabling opaque types in consuming positions + +The prohibition on opaque types in "consuming" positions could be lifted for opaque types both in parameters and in return types, but they wouldn't be useful with their current semantics because in both cases the wrong code (caller vs. callee) gets to choose the parameter. We could enable opaque types in consuming positions by "flipping" who gets to choose the parameter. To understand this, think of opaque result types as a form of "reverse generics", where there is a generic parameter list after a function's `->` and for which the function itself (the callee) gets to choose the type. For example: + +```swift +func f1() -> some P { ... } +// translates to "reverse generics" version... +func f1() -> T { /* callee implementation here picks concrete type for T */ } +``` + +The problem with opaque types in consuming positions of the return type is that the callee picks the concrete type, and the caller can't reason about it. We can see this issue by translating to the reverse-generics formulation: + +```swift +func f2() -> (some P) -> Void { ... } +// translates to "reverse generics" version... +func f2() -> (T) -> Void { /* callee implementation here picks concrete type for T */} +``` + +We could "flip" the caller/callee choice here by translating opaque types in consuming positions to the other side of the `->`. For example, `f2` would be translated into + +```swift +// if we "flip" opaque types in consuming positions +func f2() -> (some P) -> Void { ... } +// translates to +func f2() -> (T) -> Void { ... } +``` + +This is a more useful translation, because the caller picks the type for `T` using type context, and the callee provides a closure that can work with whatever type the caller picks, generically. For example: + +```swift +let fn1: (Int) -> Void == f2 // okay, T == Int +let fn2: (String) -> Void = f2 // okay, T == String +``` + +Similar logic applies to opaque types in consuming positions within parameters. Consider this function: + +```swift +func g2(fn: (some P) -> Void) { ... } +``` + +If this translates to "normal" generics, i.e., then the parameter isn't readily usable: + +```swift +// if we translated to "normal" generics +func g2(fn: (T) -> Void) { /* how do we come up with a T to call fn with? */} +``` + +Again, the problem here is that the caller gets to choose what `T` is, but then the callee cannot use it effectively. We could again "flip" the generics, moving the implicit type parameter for an opaque type in consuming position to the other side of the function's `->`: + +```swift +// if we "flip" opaque types in consuming positions +func g2(fn: (some P) -> Void) { ... } +// translates to +func g2(fn: (T) -> Void) -> Void { ... } +``` + +Now, the implementation of `g2` (the callee) gets to choose the type of `T`, which is appropriate because it will be providing values of type `T` to `fn`. The caller will need to provide a closure or generic function that's able to accept any `T` that conforms to `P`. It cannot write the type out, but it can certainly make use of it via type inference, e.g.: + +```swift +g2 { x in x.doSomethingSpecifiedInP() } +``` diff --git a/proposals/0342-static-link-runtime-libraries-by-default-on-supported-platforms.md b/proposals/0342-static-link-runtime-libraries-by-default-on-supported-platforms.md new file mode 100644 index 0000000000..788c935013 --- /dev/null +++ b/proposals/0342-static-link-runtime-libraries-by-default-on-supported-platforms.md @@ -0,0 +1,232 @@ +# Statically link Swift runtime libraries by default on supported platforms + +* Proposal: [SE-0342](0342-static-link-runtime-libraries-by-default-on-supported-platforms.md) +* Authors: [neonichu](https://github.com/neonichu) [tomerd](https://github.com/tomerd) +* Review Manager: [Ted Kremenek](https://github.com/tkremenek) +* Status: **Accepted** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0342-statically-link-swift-runtime-libraries-by-default-on-supported-platforms/56517) +* Implementation: [apple/swift-package-manager#3905](https://github.com/apple/swift-package-manager/pull/3905) +* Initial discussion: [Forum Thread](https://forums.swift.org/t/pre-pitch-statically-linking-the-swift-runtime-libraries-by-default-on-linux) +* Pitch: [Forum Thread](https://forums.swift.org/t/pitch-package-manager-statically-link-swift-runtime-libraries-by-default-on-supported-platforms) + +## Introduction + +Swift 5.3.1 introduced [statically linking the Swift runtime libraries on Linux](https://forums.swift.org/t/static-linking-on-linux-in-swift-5-3-1/). +With this feature, users can set the `--static-swift-stdlib` flag when invoking SwiftPM commands (or the long form `-Xswiftc -static-stdlib`) in order to statically link the Swift runtime libraries into the program. + +On some platforms, such as Linux, this is often the preferred way to link programs, since the program is easier to deploy to the target server or otherwise share. + +This proposal explores making it SwiftPM's default behavior when building executable programs on such platforms. + +## Motivation + +Darwin based platform ship with the Swift runtime libraries in the _dyld shared cache_. +This allows building smaller Swift programs by dynamically linking the Swift runtime libraries. +The shared cache keeps the cost of loading these libraries low. + +Other platforms, such as conventional Linux distributions, do not ship with the Swift runtime libraries. +Hence, a deployment of a program built with Swift (e.g. a web-service or CLI tool) on such platform requires one of three options: + +1. Package the application with a "bag of shared objects" (the `libswift*.so` files making up Swift's runtime libraries) alongside the program. +2. Statically link the runtime libraries using the `--static-swift-stdlib` flag described above. +3. Use a "runtime" docker image that contain the _correct version_ of the the runtime libraries (matching the compiler version exactly). + +Out of the three options, the most convenient is #2 given that #1 requires manual intervention and/or additional wrapper scripts that use `ldd`, `readelf` or similar tools to deduce the correct list of runtime libraries. +#3 is convenient but version sensitive. +#2 also has cold start performance advantage because there is less dynamic library loading. +However #2 comes at a cost of bigger binaries. + +Stepping outside the Swift ecosystem, deployment of statically linked programs is often the preferred way on server centric platforms such as Linux, as it dramatically simplifies deployment of server workloads. +For reference, Go and Rust both chose to statically link programs by default for this reason. + +### Policy for system shipping with the Swift runtime libraries + +At this time, Darwin based systems are the only ones that ship with the Swift runtime libraries. +In the future, other operating systems may choose to ship with the Swift runtime libraries. +As pointed out by [John_McCall](https://forums.swift.org/u/John_McCall), in such cases the operating system vendors are likely to choose a default that is optimized for their software distribution model. +For example, they may choose to distribute a Swift toolchain defaulting to dynamic linking (against the version of the Swift runtime libraries shipped with that system) to optimize binary size and memory consumption on that system. +To support this, the Swift toolchain build script could include a preset to controls the default linking strategy. + +As pointed out by [Joe_Groff](https://forums.swift.org/u/Joe_Groff), even in such cases there is still an argument for defaulting Swift.org distributed Swift toolchains to static linking for distribution, since we want to preserve the freedom to pursue ABI-breaking improvements in top-of-tree for platforms that aren't ABI-constrained. This situation wouldn't be different from other ecosystems: The Python or Ruby that comes with macOS are set up to provide a stable runtime environment compatible with the versions of various Python- and Ruby-based tools and libraries packaged by the distribution, but developers can and do often install their own local Python/Ruby interpreter if they need a language version different from the vendor's, or a package that's incompatible with the vendor's interpreter. + +## Proposed solution + +Given that at this time Darwin based systems are the only ones that ship with the Swift runtime libraries, +we propose to make statically linking of the Swift runtime libraries SwiftPM's default behavior when building executables in release mode on non-Darwin platforms that support such linking, +with an opt-out way to disable this default behavior. + +Building in debug mode will continue to dynamically link the Swift runtime libraries, given that debugging requires the Swift toolchain to be installed and as such the Swift runtime libraries are by definition present. + +Note that SwiftPM's default behavior could change over time as its designed to reflect the state and available options of Swift runtime distribution on that platform. In other words, whether or not a SwiftPM defaults to static linking of the Swift runtime libraries is a by-product of the state of Swift runtime distribution on a platform. + +Also note this does not mean the resulting program is fully statically linked - only the Swift runtime libraries (stdlib, Foundation, Dispatch, etc) would be statically linked into the program, while external dependencies will continue to be dynamically linked and would remain a concern left to the user when deploying Swift based programs. +Such external dependencies include: +1. Glibc (including `libc.so`, `libm.so`, `libdl.so`, `libutil.so`): On Linux, Swift relies on Glibc to interact with the system and its not possible to fully statically link programs based on Glibc. In practice this is usually not a problem since most/all Linux systems ship with a compatible Glibc. +2. `libstdc++` and `libgcc_s.so`: Swift on Linux also relies on GNU's C++ standard library as well as GCC's runtime library which much like Glibc is usually not a problem because a compatible version is often already installed on the target system. +3. At this time, non-Darwin version of Foundation (aka libCoreFoundation) has two modules that rely on system dependencies (`FoundationXML` on `libxml2` and `FoundationNetworking` on `libcurl`) which cannot be statically linked at this time and require to be installed on the target system. +4. Any system dependencies the program itself brings (e.g. `libsqlite`, `zlib`) would not be statically linked and must be installed on the target system. + + +## Detailed design + +The changes proposed are focused on the behavior of SwiftPM. +We propose to change SwiftPM's default linking of the Swift runtime libraries when building executables as follows: + +### Default behavior + +* Darwin-based platforms used to support statically linking the Swift runtime libraries in the past (and never supported fully static binaries). + Today however, the Swift runtime library is shipped with the operating system and can therefore not easily be included statically in the binary. + Naturally, dynamically linking the Swift runtime libraries will remain the default on Darwin. + +* Linux and WASI support static linking and would benefit from it for the reasons highlighted above. + We propose to change the default on these platforms to statically link the Swift runtime libraries. + +* Windows may benefit from statically linking for the reasons highlighted above but it is not technically supported at this time. + As such the default on Windows will remain dynamically linking the Swift runtime libraries. + +* The default behavior on platforms not listed above will remain dynamically linking the Swift runtime libraries. + +### Opt-in vs. Opt-out + +SwiftPM's `--static-swift-stdlib` CLI flag is designed as an opt-in way to achieve static linking of the Swift runtime libraries. + +We propose to deprecate `--static-swift-stdlib` and introduce a new flag `--disable-static-swift-runtime` which is designed as an opt-out from the default behavior described above. + +Users that want to force static linking (as with `--static-swift-stdlib`) can use the long form `-Xswiftc -static-stdlib`. + +### Example + +Consider the following simple program: + +```bash +$ swift package init --type executable +Creating executable package: test +Creating Package.swift +Creating README.md +Creating .gitignore +Creating Sources/ +Creating Sources/test/main.swift +Creating Tests/ +Creating Tests/testTests/ +Creating Tests/testTests/testTests.swift + +$ cat Sources/test/main.swift +print("Hello, world!") +``` +Building the program with default dynamic linking yields the following: + +```bash +$ swift build -c release +[3/3] Build complete! + +$ ls -la --block-size=K .build/release/test +-rwxr-xr-x 1 root root 17K Dec 3 22:48 .build/release/test* + +$ ldd .build/release/test + linux-vdso.so.1 (0x00007ffc82be4000) + libswift_Concurrency.so => /usr/lib/swift/linux/libswift_Concurrency.so (0x00007f0f5cfb5000) + libswiftCore.so => /usr/lib/swift/linux/libswiftCore.so (0x00007f0f5ca55000) + libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0f5c85e000) + libdispatch.so => /usr/lib/swift/linux/libdispatch.so (0x00007f0f5c7fd000) + libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0f5c7da000) + libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0f5c7d4000) + libswiftGlibc.so => /usr/lib/swift/linux/libswiftGlibc.so (0x00007f0f5c7be000) + libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0f5c5dc000) + libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0f5c48d000) + libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0f5c472000) + /lib64/ld-linux-x86-64.so.2 (0x00007f0f5d013000) + libicui18nswift.so.65 => /usr/lib/swift/linux/libicui18nswift.so.65 (0x00007f0f5c158000) + libicuucswift.so.65 => /usr/lib/swift/linux/libicuucswift.so.65 (0x00007f0f5bf55000) + libicudataswift.so.65 => /usr/lib/swift/linux/libicudataswift.so.65 (0x00007f0f5a4a2000) + librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0f5a497000) + libBlocksRuntime.so => /usr/lib/swift/linux/libBlocksRuntime.so (0x00007f0f5a492000) +``` + +Building the program with static linking of the Swift runtime libraries yields the following: + +```bash +$ swift build -c release --static-swift-stdlib +[3/3] Build complete! + +$ ls -la --block-size=K .build/release/test +-rwxr-xr-x 1 root root 35360K Dec 3 22:50 .build/release/test* + +$ ldd .build/release/test + linux-vdso.so.1 (0x00007fffdaafa000) + libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdd521c5000) + libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdd521a2000) + libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdd51fc0000) + libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdd51e71000) + libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdd51e56000) + libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdd51c64000) + /lib64/ld-linux-x86-64.so.2 (0x00007fdd54211000) +``` + +These snippets demonstrates the following: +1. Statically linking of the Swift runtime libraries increases the binary size from 17K to ~35M. +2. Statically linking of the Swift runtime libraries reduces the dependencies reported by `ldd` to core Linux libraries. + +This jump in binary size may be alarming at first sight, but since the program is not usable without the Swift runtime libraries, the actual size of the deployable unit is similar. + +```bash +$ mkdir deps + +$ ldd ".build/release/test" | grep swift | awk '{print $3}' | xargs cp -Lv -t ./deps +'/usr/lib/swift/linux/libswift_Concurrency.so' -> './deps/libswift_Concurrency.so' +'/usr/lib/swift/linux/libswiftCore.so' -> './deps/libswiftCore.so' +'/usr/lib/swift/linux/libdispatch.so' -> './deps/libdispatch.so' +'/usr/lib/swift/linux/libswiftGlibc.so' -> './deps/libswiftGlibc.so' +'/usr/lib/swift/linux/libicui18nswift.so.65' -> './deps/libicui18nswift.so.65' +'/usr/lib/swift/linux/libicuucswift.so.65' -> './deps/libicuucswift.so.65' +'/usr/lib/swift/linux/libicudataswift.so.65' -> './deps/libicudataswift.so.65' +'/usr/lib/swift/linux/libBlocksRuntime.so' -> './deps/libBlocksRuntime.so' + +$ ls -la --block-size=K deps/ +total 42480K +drwxr-xr-x 2 root root 4K Dec 3 22:59 ./ +drwxr-xr-x 6 root root 4K Dec 3 22:58 ../ +-rw-r--r-- 1 root root 17K Dec 3 22:59 libBlocksRuntime.so +-rw-r--r-- 1 root root 432K Dec 3 22:59 libdispatch.so +-rwxr-xr-x 1 root root 27330K Dec 3 22:59 libicudataswift.so.65* +-rwxr-xr-x 1 root root 4030K Dec 3 22:59 libicui18nswift.so.65* +-rwxr-xr-x 1 root root 2403K Dec 3 22:59 libicuucswift.so.65* +-rwxr-xr-x 1 root root 520K Dec 3 22:59 libswift_Concurrency.so* +-rwxr-xr-x 1 root root 7622K Dec 3 22:59 libswiftCore.so* +-rwxr-xr-x 1 root root 106K Dec 3 22:59 libswiftGlibc.so* +``` + +## Impact on existing packages + +The new behavior will take effect with a new version of SwiftPM, and packages build with that version will be linked accordingly. + +* Deployment of applications using "bag of shared objects" technique (#1 above) will continue to work as before (though would be potentially redundant). +* Deployment of applications using explicit static linking (#2 above) will continue to work and emit a warning that its redundant. +* Deployment of applications using docker "runtime" images (#3 above) will continue to work as before (though would be redundant). + +### Additional validation when linking libraries + +SwiftPM currently performs no validation when linking libraries into an executable that statically links the Swift runtime libraries. +This means that users can mistakenly link a library that already has the Swift runtime libraries statically linked into the executable that will also statically link the Swift runtime libraries, which could lead to runtime errors if the versions of the Swift runtime libraries do not match. +As part of this proposal, SwiftPM will gain a new post build validation checking for this condition and warning the user accordingly. + +## Alternatives considered and future directions + +The most obvious question this proposal brings is why not fully statically link the program instead of statically linking only the runtime libraries. +Go is a good example for creating fully statically linked programs, contributing to its success in the server ecosystem at large. +Swift already offers a flag for this linking mode: `-Xswiftc -static-executable`, but in reality Swift's ability to create fully statically linked programs is constrained. +This is mostly because today, Swift on Linux only supports GNU's libc (Glibc) and GNU's C++ standard library which do not support producing fully static binaries. +A future direction could be to look into supporting the `musl libc` and LLVM's `libc++` which should be able to produce fully static binaries. + +Further, Swift has good support to interoperate with C libraries installed on the system. +Whilst that is a nice feature, it does make it difficult to create fully statically linked programs because it would be necessary to make sure each and every of these dependencies is available in a fully statically linked form with all the common dependencies being compatible. +For example, it is not possible to link a binary that uses the `musl libc` with libraries that expect to be statically linked with Glibc. +As Swift's ability to create fully statically linked programs improves, we should consider changing the default from `-Xswiftc -static-stdlib` to `-Xswiftc -static-executable`. + +A more immediate future direction which would improve programs that need to use of FoundationXML and FoundationNetworking is to replace the system dependencies of these modules with native implementation. +This is outside the scope of this proposal which focuses on SwiftPM's behavior. + +Another alternative is to do nothing. +In practice, this proposal does not add new features, it only changes default behavior which is already achievable with the right knowledge of build flags. +That said, we believe that changing the default will make using Swift on non-Darwin platforms easier, saving time and costs to Swift users on such platforms. + +The spelling of the new flag `--disable-static-swift-runtime` is open to alternative ideas, e.g. `--disable-static-swift-runtime-libraries`. diff --git a/proposals/0343-top-level-concurrency.md b/proposals/0343-top-level-concurrency.md new file mode 100644 index 0000000000..e11e040563 --- /dev/null +++ b/proposals/0343-top-level-concurrency.md @@ -0,0 +1,213 @@ +# Concurrency in Top-level Code + +* Proposal: [SE-0343](0343-top-level-concurrency.md) +* Authors: [Evan Wilde](https://github.com/etcwilde) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Implemented (Swift 5.7)** +* Implementation: [Fix top-level global-actor isolation crash](https://github.com/apple/swift/pull/40963), [Add `@MainActor @preconcurrency` to top-level variables](https://github.com/apple/swift/pull/40998), [Concurrent top-level inference](https://github.com/apple/swift/pull/41061) + +## Introduction + +Bringing concurrency to top-level code is an expected continuation of the +concurrency work in Swift. This pitch looks to iron out the details of how +concurrency will work in top-level code, specifically focusing on how top-level +variables are protected from data races, and how a top-level code context goes +from a synchronous context to an asynchronous context. + +Swift-evolution thread: [Discussion thread topic for concurrency in top-level code](https://forums.swift.org/t/concurrency-in-top-level-code/55001) + +## Motivation + +The top-level code declaration context works differently than other declaration +spaces. As such, adding concurrency features to this spaces results in questions +that have not yet been addressed. + +Variables in top-level code behave as a global-local hybrid variable; they exist +in the global scope and are accessible as global variables within the module, +but are initialized sequentially like local variables. Global variables are +dangerous, especially with concurrency. There are no isolation guarantees made, +and are therefore subject to race conditions. + +As top-level code is intended as a safe space for testing out features and +writing pleasant little scripts, this simply will not do. + +In addition to the strange and dangerous behavior of variables, changing whether +a context is synchronous or asynchronous has an impact on how function overloads +are resolved, so simply flipping a switch could result in some nasty hidden +semantic changes, potentially breaking scripts that already exist. + +## Proposed solution + +The solutions will only apply when the top-level code is an asynchronous +context. As a synchronous context, the behavior of top-level code does not +change. In order to trigger making the top-level context an asynchronous context, I +propose using the presence of an `await` in one of the top-level expressions. + +An await nested within a function declaration or a closure will not trigger the +behavior. + +```swift +func doAsyncStuff() async { + // ... +} + +let countCall = 0 + +let myClosure = { + await doAsyncStuff() // `await` does not trigger async top-level + countCall += 1 +} + +await myClosure() // This `await` will trigger an async top-level +``` + +Top-level global variables are implicitly assigned a `@MainActor` global actor +isolation to prevent data races. To avoid breaking sources, the variable is +implicitly marked as pre-concurrency up to Swift 6. + +```swift +var a = 10 + +func bar() { + print(a) +} + +bar() + +await something() // make top-level code an asynchronous context +``` + +After Swift 6, full actor-isolation checking will take place. The usage of `a` +in `bar` will result in an error due to `bar` not being isolated to the +`MainActor`. In Swift 5, this will compile without errors. + +## Detailed design + +### Asynchronous top-level context inference + +The rules for inferring whether the top-level context is an asynchronous context +are the same for anonymous closures, specified in [SE-0296 Async/Await](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md#closures). + +The top-level code is inferred to be an asynchronous context if it contains a +suspension point in the immediate top-level context. + +```swift +func theAnswer() async -> Int { 42 } + +async let a = theAnswer() // implicit await, top-level is async + +await theAnswer() // explicit await, top-level is async + +let numbers = AsyncStream(Int.self) { continuation in + Task { + for number in 0 .. < 10 { + continuation.yield(number) + } + continuation.finish() + } +} + +for await number in numbers { // explicit await, top-level is asnyc + print(number) +} +``` + +The above example demonstrates each kind of suspension point, triggering an +asynchronous top-level context. Specifically, `async let a = theAnswer()` +involves an implicit suspension, `await theAnswer()` involves an explicit +suspension, as does `for await number in numbers`. Any one of these is +sufficient to trigger the switch to an asynchronous top-level context. + +Not that the inference of `async` in the top-level does not propagate to +function and closure bodies, because those contexts are separably asynchronous +or synchronous. + +```swift +func theAnswer() async -> Int { 42 } + +let closure1 = { @MainActor in print(42) } +let closure2 = { () async -> Int in await theAnswer() } +``` + +The top-level code in the above example is not an asynchronous context because +the top-level does not contain a suspension point, either explicit or implicit. + +The mechanism for inferring whether a closure body is an asynchronous context +lives in the `FindInnerAsync` ASTWalker. With minimal effort, the +`FindInnerAsync` walker can be generalized to handle top-level code bodies, +maintaining the nice parallel inference behaviour between top-level code and +closure body asynchronous detection. + +### Variables + +Variables in top-level code are initialized sequentially like a local variable, +but are in the global scope and are otherwise treated as global variables. To +prevent data races, variables should implicitly be isolated to the main actor. +It would be a shame if every top-level variable access had to go through an +`await` though. Luckily, like the other entrypoints, top-level code runs on the +main thread, so we can make the top-level code space implicitly main-actor +isolated so the variables can be accessed and modified directly. This is still +source-breaking though; a synchronous global function written in the top-level +code will emit an error because the function is not isolated to the main actor +when the variable is. While the diagnostic is correct in stating that there is a +potential data-race, the source-breaking effect is also unfortunate. To +alleviate the source break, the variable is implicitly annotated with the +`@preconcurrency` attribute. The attribute only applies to Swift 5 code, and +once the language mode is updated to Swift 6, these data races will become hard +errors. + +If `-warn-concurrency` is passed to the compiler and there is an `await` in +top-level code, the warnings are hard errors in Swift 5, as they would in any +other asynchronous context. If there is no `await` and the flag is passed, +variables are implicitly protected by the main actor and concurrency checking is +strictly enforced, even though the top-level is not an asynchronous context. +Since the top-level is not an asynchronous context, no run-loops are created +implicitly and the overload resolution behavior does not change. + +In summary, top-level variable declarations behave as though they were declared +with `@MainActor @preconcurrency` in order to strike a nice balance between +data-race safety and reducing source breaks. + +Going back to the global behaviour variables, there are some additional design +details that I should point out. + +I would like to propose removing the ability to explicitly specify a global +actor on top-level variables. Top-level variables are treated like a hybrid of +global and local variables, which has some nasty consequences. The variables are +declared in the global scope, so they are assumed to be available anywhere. This +results in some nasty memory safety issues, like the following example: + +```swift +print(a) +let a = 10 +``` + +The example compiles and prints "0" when executed. The declaration `a` is +available at the `print` statement because it is a global variable, but it is +not yet initialized because initialization happens sequentially. Integer types +and other primitives are implicitly zero-initialized; however, classes are +referential types, initialized to zero, so this results in a segmentation fault +if the variable is a class type. + +Eventually, we would like to plug this hole in the memory model. The design for +that is still in development, but will likely move toward making top-level +variables local variables of the implicit main function. I am proposing that we +disallow explicit global actors to facilitate that change and reduce the source +breakage caused by that change. + +## Source compatibility + +The `await` expression cannot appear in top-level code today since the top-level +is not an asynchronous context. As the features proposed herein are enabled by +the presence of an `await` expression in the top level, there are no scripts +today that will be affected by the changes proposed in this proposal. + +## Effect on ABI stability + +This proposal has no impact on ABI. Functions and variables have the same +signature as before. + +## Acknowledgments + +Thank you, Doug, for lots of discussion on how to break this down into something +that minimizes source breakage to a level where we can introduce this to Swift 5. diff --git a/proposals/0344-distributed-actor-runtime.md b/proposals/0344-distributed-actor-runtime.md new file mode 100644 index 0000000000..b052a3531c --- /dev/null +++ b/proposals/0344-distributed-actor-runtime.md @@ -0,0 +1,1888 @@ +# Distributed Actor Runtime + +* Proposal: [SE-0344](0344-distributed-actor-runtime.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [Pavel Yaskevich](https://github.com/xedin), [Doug Gregor](https://github.com/DougGregor), [Kavon Farvardin](https://github.com/kavon), [Dario Rexin](https://github.com/drexin), [Tomer Doron](https://github.com/tomerd) +* Review Manager: [Joe Groff](https://github.com/jckarter/) +* Status: **Implemented (Swift 5.7)** +* Implementation: + * Partially available in [recent `main` toolchain snapshots](https://swift.org/download/#snapshots) behind the `-enable-experimental-distributed` feature flag. + * This flag also implicitly enables `-enable-experimental-concurrency`. +* Review threads + * [First Review](https://forums.swift.org/t/se-0344-distributed-actor-runtime/55525) ([summary](https://forums.swift.org/t/returned-for-revision-se-0344-distributed-actor-runtime/55836)) + * [Second Review](https://forums.swift.org/t/se-0344-second-review-distributed-actor-runtime/56002) ([summary](https://forums.swift.org/t/accepted-se-0344-distributed-actor-runtime/56416)) + + +## Table of Contents + +- [Distributed Actor Runtime](#distributed-actor-runtime) + - [Table of Contents](#table-of-contents) + - [Introduction](#introduction) + - [Useful links](#useful-links) + - [Motivation](#motivation) + - [Example scenario](#example-scenario) + - [Caveat: Low-level implementation details](#caveat-low-level-implementation-details) + - [Detailed design](#detailed-design) + - [The `DistributedActorSystem` protocol](#the-distributedactorsystem-protocol) + - [Implicit distributed actor properties](#implicit-distributed-actor-properties) + - [Initializing distributed actors](#initializing-distributed-actors) + - [Distributed actor initializers](#distributed-actor-initializers) + - [Initializing `actorSystem` and `id` properties](#initializing-actorsystem-and-id-properties) + - [Ready-ing distributed actors](#ready-ing-distributed-actors) + - [Ready-ing distributed actors, exactly once](#ready-ing-distributed-actors-exactly-once) + - [Resigning distributed actor IDs](#resigning-distributed-actor-ids) + - [Resolving distributed actors](#resolving-distributed-actors) + - [Invoking distributed methods](#invoking-distributed-methods) + - [Sender: Invoking a distributed method](#sender-invoking-a-distributed-method) + - [Sender: Serializing and sending invocations](#sender-serializing-and-sending-invocations) + - [Recipient: Receiving invocations](#recipient-receiving-invocations) + - [Recipient: Deserializing incoming invocations](#recipient-deserializing-incoming-invocations) + - [Recipient: Resolving the recipient actor instance](#recipient-resolving-the-recipient-actor-instance) + - [Recipient: The `executeDistributedTarget` method](#recipient-the-executedistributedtarget-method) + - [Recipient: Executing the distributed target](#recipient-executing-the-distributed-target) + - [Recipient: Collecting result/error from invocations](#recipient-collecting-resulterror-from-invocations) + - [Amendments](#amendments) + - [Initializers no longer need to accept a single DistributedActorSystem](#initializers-no-longer-need-to-accept-a-single-distributedactorsystem) + - [Future work](#future-work) + - [Variadic generics removing the need for `remoteCallVoid`](#variadic-generics-removing-the-need-for-remotecallvoid) + - [Identifying, evolving and versioning remote calls](#identifying-evolving-and-versioning-remote-calls) + - [Default distributed call target identification scheme](#default-distributed-call-target-identification-scheme) + - [Compression techniques to avoid repeatedly sending large identifiers](#compression-techniques-to-avoid-repeatedly-sending-large-identifiers) + - [Overlap with general ABI and versioning needs in normal Swift code](#overlap-with-general-abi-and-versioning-needs-in-normal-swift-code) + - [Discussion: User provided target identities](#discussion-user-provided-target-identities) + - [Resolving `DistributedActor` protocols](#resolving-distributedactor-protocols) + - [Passing parameters to `assignID`](#passing-parameters-to-assignid) + - [Alternatives considered](#alternatives-considered) + - [Define `remoteCall` as protocol requirement, and accept `[Any]` arguments](#define-remotecall-as-protocol-requirement-and-accept-any-arguments) + - [Constraining arguments, and return type with of `remoteCall` with `SerializationRequirement`](#constraining-arguments-and-return-type-with-of-remotecall-with-serializationrequirement) + - [Hardcoding the distributed runtime to make use of `Codable`](#hardcoding-the-distributed-runtime-to-make-use-of-codable) + - [Acknowledgments & Prior art](#acknowledgments--prior-art) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Changelog](#changelog) + +## Introduction + +With the recent introduction of [actors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md) to the language, Swift gained powerful and foundational building blocks for expressing *thread-safe* concurrent programs. Actors guarantee thread-safety thanks to actor-isolation of mutable state they encapsulate. + +In [SE-0336: Distributed Actor Isolation][isolation] we took it a step further, guaranteeing complete isolation of state with distributed actor-isolation, and setting the stage for `distributed` method calls to be performed across process and node boundaries. + +This proposal focuses on the runtime aspects of making such remote calls possible, their exact semantics and how developers can provide their own `DistributedActorSystem` implementations to hook into the same language mechanisms, extending Swift's distributed actor model to various environments (such as cross-process communication, clustering, or even client/server communication). + +#### Useful links + +It is recommended, though not required, to familiarize yourself with the prior proposals before reading this one: + +- [SE-0336: Distributed Actor Isolation][isolation] — a detailed proposal +- Distributed Actor Runtime (this proposal) + +Feel free to reference the following library implementations which implement this proposal's library side of things: + +- [Swift Distributed Actors Library](https://www.swift.org/blog/distributed-actors/) — a reference implementation of a *peer-to-peer cluster* for distributed actors. Its internals depend on the work in progress language features and are dynamically changing along with these proposals. It is a realistic implementation that we can use as reference for these design discussions. + +## Motivation + +With distributed actor-isolation checking laid out in [SE-0336: Distributed Actor Isolation][isolation], we took the first step towards enabling remote calls being made by invoking `distributed func` declarations on distributed actors. The isolation model and serialization requirement checks in that proposal outline how we can guarantee the soundness of such distributed actor model at compile time. + +Distributed actors enable developers to build their applications and systems using the concept of actors that may be "local" or "remote", and communicate with them regardless of their location. Our goal is to set developers free from having to re-invent ad-hoc approaches to networking, serialization and error handling every time they need to embrace distributed computing. + +Instead, we aim to embrace a co-operative approach to the problem, in which: + +1. the Swift language, compiler, and runtime provide the necessary isolation checks and runtime hooks for distributed actor lifecycle management, and distributed method calls that can be turned into "messages" that can be sent to remote peers, +2. `DistributedActorSystem` library implementations, hook into the language provided cut-points, take care of the actual message interactions, e.g. by sending messages representing remote distributed method calls over the network, +3. `distributed actor` authors, who want to focus on getting things done, express their distributed API boundaries and communicate using them. They may have opinions about serialization and specifics of message handling, and should be able to configure and use the `DistributedActorSystem` of their choice to get things done. + +In general, we propose to embrace the actor style of communication for typical distributed system development, and aim to provide the necessary tools in the language, and runtime to make this a pleasant and nice default go-to experience for developers. + +Distributed actors may not serve *all* possible use-cases where networking is involved, but we believe a large group of applications and systems will benefit from them, as the ecosystem gains mature `DistributedActorSystem` implementations. + +#### Example scenario + +In this proposal we will focus only on the runtime aspects of distributed actors and methods, i.e. what happens in order to create, send, and receive messages formed when a distributed method is called on a remote actor. For more details on distributed actor isolation and other compile-time checks, please refer to [SE-0336: Distributed Actor Isolation][isolation]. + +We need to pass around distributed actors in order to invoke methods on them at some later point in time. We need those actors to declare `distributed` methods such that we have something we can message them with, and there must be some lifecycle and registration mechanisms related to them. + +One example use case we can keep in mind is a simple turn-based `Game` which showcases most of the capabilities we come to expect of distributed actors: + +```swift +distributed actor Player { + // ... + + distributed func makeMove() -> Move { ... } + + distributed func gameFinished(result: GameResult) { + if result.winner == self { + print("I WON!") + } else { + print("Player \(result.winner) won the game.") + } + } +} + +distributed actor Game { + var state: GameState = ... + + // players can be located on different nodes + var players: Set = [] + + distributed func playerJoined(_ player: Player) { + others.append(player) + if others.count >= 2 { // we need 2 other players to start a game + Task { try await self.start() } + } + } + + func start() async throws { + state = .makeNewGameState(with: players) + while !state.finished { + for player in players { + let move = try await p.makeMove() // TODO: handle failures, e.g. "move timed-out" + state.apply(move, by: player) + } + } + + let winner = state.winner + try await game.finishedResult + } +} +``` + +This code snippet showcases what kind of distributed actors one might want to implement – they represent addressable identities in a system where players may be hosted on different hosts or devices, and we'd like to communicate with any of them from the `Game` actor which manages the entire game's state. Players may be on the same host as the `Game` actor, or on different ones, but we never have to change the implementation of `Game` to deal with this – thanks to distributed actors and the concept of location transparency, we can implement this piece of code once, and run it all locally, or distributed without changing the code specifically for either of those cases. + +### Caveat: Low-level implementation details + +This proposal includes low-level implementation details in order to showcase how one can use to build a real, efficient, and extensible distributed actor system using the proposed language runtime. It is primarily written for distributed actor system authors, which need to understand the underlying mechanisms which distributed actors use. + +End users, who just want to use _distributed actors_, and not necessarily _implement_ a distributed actor system runtime, do not need to dive deep as deep into this proposal, and may be better served by reading [SE-0366: Distributed Actor Isolation][isolation] which focuses on how distributed actors are used. Reading this — runtime — proposal, however, will provide additional insights as to why distributed actors are isolated the way they are. + +This proposal focuses on how a distributed actor system runtime can be implemented. Because this language feature is extensible, library authors may step in and build their own distributed actor runtimes. It is expected that there will be relatively few, but solid actor system implementations eventually, yet their use would apply to many many more end-users than actor system developers. + +## Detailed design + +This section is going to deep dive into the runtime details and its interaction with user provided `DistributedActorSystem` implementations. Many of these aspects are not strictly necessary to internalize by end-user/developer, who only wants to write some distributed actors and have them communicate using *some* distributed actor system. + +### The `DistributedActorSystem` protocol + +At the core of everything distributed actors do, is the `DistributedActorSystem` protocol. This protocol is open to be implemented by anyone, and can be used to extend the functionality of distributed actors to various environments. + +Building a solid actor system implementation is not a trivial task, and we only expect a handful of mature implementations to take the stage eventually. + +> At the time of writing, we–the proposal authors–have released a work in progress [peer-to-peer cluster actor system implementation](https://www.swift.org/blog/distributed-actors/) that is tracking this evolving language feature. It can be viewed as a reference implementation for the language features and `DistributedActorSystem` protocol discussed in this proposal. + +Below we present the full listing of the `DistributedActorSystem` protocol, and we'll be explaining the specific methods one by one as we go: + +```swift +// Module: _Distributed + +protocol DistributedActorSystem: Sendable { + /// The type of `ID` assigned to a distributed actor while initializing with this actor system. + /// The identity should be meaningfully unique, in the sense that ID equality should mean referring to the + /// same distributed actor. + /// + /// A distributed actor created using a specific actor system will use the system's `ActorID` as + /// the `ID` type it stores and for its `Hashable` implementation. + /// + /// ### Implicit distribute actor `Codable` conformance + /// If the `ActorID` (and therefore also the `DistributedActor.ID`) conforms to `Codable`, + /// the `distributed actor` will gain an automatically synthesized conformance to `Codable` as well. + associatedtype ActorID: Sendable & Hashable + + /// The specific type of the invocation encoder that will be created and populated + /// with details about the invocation when a remote call is about to be made. + /// + /// The populated instance will be passed to the `remoteCall` from where it can be + /// used to serialize into a message format in order to perform the remote invocation. + associatedtype InvocationEncoder: DistributedTargetInvocationEncoder + + /// The specific type of invocation decoder used by this actor system. + /// + /// An instance of this type must be passed to `executeDistributedTarget` which + /// extracts arguments and applies them to the local target of the invocation. + associatedtype InvocationDecoder: DistributedTargetInvocationDecoder + + /// The serialization requirement that will be applied to all distributed targets used with this system. + /// + /// An actor system is still allowed to throw serialization errors if a specific value passed to a distributed + /// func violates some other restrictions that can only be checked at runtime, e.g. checking specific types + /// against an "allow-list" or similar. The primary purpose of the serialization requirement is to provide + /// compile time hints to developers, that they must carefully consider evolution and serialization of + /// values passed to and from distributed methods and computed properties. + associatedtype SerializationRequirement + where SerializationRequirement == InvocationEncoder.SerializationRequirement, + SerializationRequirement == InvocationDecoder.SerializationRequirement + + // ==== --------------------------------------------------------------------- + // - MARK: Actor Lifecycle + + /// Called by a distributed when it begins its initialization (in a non-delegating init). + /// + /// The system should take special care to not assign two actors the same `ID`, and the `ID` + /// must remain valid until it is resigned (see `resignID(_:)`). + func assignID(_ actorType: Actor.Type) -> ActorID + where Actor: DistributedActor, + Actor.ID == ActorID + + /// Automatically called by in every distributed actor's non-delegating initializer. + /// + /// The call is made specifically before the `self` of such distributed actor is about to + /// escape, e.g. via a function call, closure or otherwise. If no such event occurs the + /// call is made at the end of the initializer. + /// + /// The passed `actor` is the `self` of the initialized actor, and its `actor.id` is expected + /// to be of the same value that was assigned to it in `assignID`. + /// + /// After the ready call returns, it must be possible to resolve it using the 'resolve(_:as:)' + /// method on the system. + func actorReady(_ actor: Actor) + where Actor: DistributedActor, + Actor.ID == ActorID + + /// Called when the distributed actor is deinitialized (or has failed to finish initializing). + /// + /// The system may release any resources associated with this actor id, and should not make + /// further attempts to deliver messages to the actor identified by this identity. + func resignID(_ id: ActorID) + + // ==== --------------------------------------------------------------------- + // - MARK: Resolving distributed actors + + /// Resolve a local or remote actor address to a real actor instance, or throw if unable to. + /// The returned value is either a local actor or proxy to a remote actor. + /// + /// Resolving an actor is called when a specific distributed actors `init(from:)` + /// decoding initializer is invoked. Once the actor's identity is deserialized + /// using the `decodeID(from:)` call, it is fed into this function, which + /// is responsible for resolving the identity to a remote or local actor reference. + /// + /// If the resolve fails, meaning that it cannot locate a local actor managed for + /// this identity, managed by this transport, nor can a remote actor reference + /// be created for this identity on this transport, then this function must throw. + /// + /// If this function returns correctly, the returned actor reference is immediately + /// usable. It may not necessarily imply the strict *existence* of a remote actor + /// the identity was pointing towards, e.g. when a remote system allocates actors + /// lazily as they are first time messaged to, however this should not be a concern + /// of the sending side. + /// + /// Detecting liveness of such remote actors shall be offered / by transport libraries + /// by other means, such as "watching an actor for termination" or similar. + func resolve(_ id: ActorID, as actorType: Actor.Type) throws -> Actor? + where Actor: DistributedActor, + Actor.ID: ActorID, + Actor.SerializationRequirement == Self.SerializationRequirement + + // ==== --------------------------------------------------------------------- + // - MARK: Remote Target Invocations + + /// Invoked by the Swift runtime when a distributed remote call is about to be made. + /// + /// The returned `InvocationEncoder` will be populated with all + /// generic substitutions, arguments, and specific error and return types + /// that are associated with this specific invocation. + func makeInvocationEncoder() -> InvocationEncoder + + // We'll discuss the remoteCall method in detail in this proposal. + // It cannot be declared as protocol requirement, and remains an ad-hoc + // requirement like this: + /// Invoked by the Swift runtime when making a remote call. + /// + /// The `invocation` is the arguments container that was previously created + /// by `makeInvocationEncoder` and has been populated with all arguments. + /// + /// This method should perform the actual remote function call, and await for its response. + /// + /// ## Errors + /// This method is allowed to throw because of underlying transport or serialization errors, + /// as well as by re-throwing the error received from the remote callee (if able to). + /// + /// Ad-hoc protocol requirement. + func remoteCall( + on actor: Actor, + target: RemoteCallTarget, + invocation: inout InvocationEncoder, + throwing: Failure.Type, + returning: Success.Type + ) async throws -> Success + where Actor: DistributedActor, + Actor.ID == ActorID, + Failure: Error, + Success: Self.SerializationRequirement + + /// Invoked by the Swift runtime when making a remote call to a `Void` returning function. + /// + /// ( ... Same as `remoteCall` ... ) + /// + /// Ad-hoc protocol requirement. + func remoteCallVoid( + on actor: Actor, + target: RemoteCallTarget, + invocation: inout InvocationEncoder, + throwing: Failure.Type + ) async throws + where Actor: DistributedActor, + Actor.ID == ActorID, + Failure: Error +} + +/// A distributed 'target' can be a `distributed func` or `distributed` computed property. +/// +/// The actor system should encode the identifier however it sees fit, +/// and transmit it to the remote peer in order to invoke identify the target of an invocation. +public struct RemoteCallTarget: Hashable { + /// The mangled name of the invoked distributed method. + /// + /// It contains all information necessary to lookup the method using `executeDistributedActorMethod(...)` + var mangledName: String { ... } + + /// The human-readable "full name" of the invoked method, e.g. 'Greeter.hello(name:)'. + var fullName: String { ... } +} +``` + +In the following sections, we will be explaining how the various methods of a distributed system are invoked by the Swift runtime. + +### Implicit distributed actor properties + +Distributed actors have two properties that are crucial for the inner workings of actors that we'll explore during this proposal: the `id` and `actorSystem`. + +These properties are synthesized by the compiler, in every `distributed actor` instance, and they witness the `nonisolated` property requirements defined on the `DistributedActor` protocol. + +The `DistributedActor` protocol (defined in [SE-0336][isolation]), defines those requirements: + +```swift +protocol DistributedActor { + associatedtype ActorSystem: DistributedActorSystem + + typealias ID = ActorSystem.ActorID + typealias SerializationRequirement: ActorSystem.SerializationRequirement + + nonisolated var id: ID { get } + nonisolated var actorSystem: ActorSystem { get } + + // ... +} +``` + +which are witnessed by *synthesized properties* in every specific distributed actor instance. + +Next, we will discuss how those properties get initialized, and used in effectively all aspects of a distributed actor's lifecycle. + +### Initializing distributed actors + +At runtime, a *local* `distributed actor` is effectively the same as a local-only `actor`. The allocated `actor` instance is a normal `actor`. However, its initialization is a little special, because it must interact with its associated actor system to make itself available for remote calls. + +We will focus on non-delegating initializers, as they are the ones where distributed actors cause additional things to happen. + +> Please note that **initializing** a distributed actor with its `init` always returns a **local** reference to a new actor. The only way to obtain a a remote reference is by using the `resolve(id:using:)` method, which is discussed in [Resolving Distributed Actors](#resolving-distributed-actors). + +Distributed actor initializers inject a number of calls into specific places of the initializer's body. These calls allow for the associated actor system to manage the actor's identity, and availability to remote calls. Before we dive into the details, the following diagram outlines the various calls that will be explained in this section: + +``` +┌────────────────────────────┐ ┌──────────────────────────┐ +│ distributed actor MyActor │ │ MyDistributedActorSystem │ +└────────────────────────────┘ └──────────────────────────┘ + │ │ + init(...) │ + │ │ + │── // self.id = actorSystem.assignID(Self.self) ───▶│ Generate and reserve ID + │ │ + │ // self.actorSystem = system │ + │ │ + │ │ + │ │ + │── // actorSystem.actorReady(self) ────────────────▶│ Store a mapping (ID -> some DistributedActor) + │ │ + ... ... + │ │ + ◌ deinit ─ // actorSystem.resignID(self.id) ────────▶│ Remove (ID -> some DistributedActor) mapping + │ + ... +``` + +### Distributed actor initializers + +A non-delegating initializer of a type must *fully initialize* it. The place in code where an actor becomes fully initialized has important and specific meaning to actor isolation which is defined in depth in [SE-0327: On Actors and Initialization](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md). Not only that, but once fully initialized it is possible to escape `self` out of a (distributed) actor's initializer. This aspect is especially important for distributed actors, because it means that once fully initialized they _must_ be registered with the actor system as they may be sent to other distributed actors and even sent messages to. + +All non-delegating initializers must store an instance of a `DistributedActorSystem` conforming type into their `self.actorSystem` property. + +This is an improvement over the initially proposed semantics in [SE-0336: Distributed Actor Isolation][isolation], where the initializers must have accepted a _single_ distributed actor system argument and would initialize the synthesized stored property automatically. + +This proposal amends this behavior to the following semantics: + +- the default initializer gains a `actorSystem: Self.ActorSystem` parameter. +- other non-delegating initializers must initialize the `self.actorSystem` property explicitly. + - it is recommended to accept an actor system argument and store it + - technically it is also possible to store a global actor system to this property, however this is generally an anti-pattern as it hurts testability of such actor (i.e. it becomes impossible to swap the actor system for a "for testing" one during test execution). + +```swift +distributed actor DA { + // synthesized: + // init(actorSystem: Self.ActorSystem) {} // ✅ no user defined init, so we synthesize a default one +} + +distributed actor DA2 { + init(system: Self.ActorSystem) { // ⚠️ ok declaration, but self.actorSystem was not initialized + // ❌ error: implicit stored property 'actorSystem' was not initialized + } + + init(other: Int, actorSystem system: Self.ActorSystem) { // ✅ ok + self.actorSystem = system + } +} +``` + +Now in the next sections, we will explore in depth why this parameter was necessary to enforce to begin with. + +#### Initializing `actorSystem` and `id` properties + +The two properties (`actorSystem` and `id`) are synthesized _stored_ properties in the body of every actor. + +Users must initialize the `actorSystem`, however management of the `id` is left up to the compiler to synthesize. All properties of an actor have to be initialized for the type to become fully initialized, same as with any other type. However, the initialization of `id` is left to the compiler in order to streamline the initialization, as well as keep the _strict_ contract between the stored actor system and that the `ID` _must_ be assigned by that exact actor system. It would be illegal to just assign an `id` from some other source because of how tightly it relates to the actors lifecycle. + +The compiler synthesizes code that does this in any designated initializer of distributed actors: + +```swift +distributed actor DA { + // let id: ID + // let actorSystem: ActorSystem + + init(actorSystem: Self.ActorSystem) { + self.actorSystem = actorSystem + // ~~~ synthesized ~~~ + // ... + // ~~~ end of synthesized ~~~ + } +} +``` + +Synthesizing the id assignment means that we need to communicate with the `system` used to initialize the distributed actor, for it is the `ActorSystem` that allocates and manages identifiers. In order to obtain a fresh `ID` for the actor being initialized, we need to call `system`'s `assignID` method. This is done before any user-defined code is allowed to run in the actors designated initializer, like this: + +```swift +distributed actor DA { + let number: Int + + init(system: ActorSystem) { + self.actorSystem = system + // ~~ injected property initialization ~~ + // self.id = system.assignID(Self.self) + // ~~ end of injected property initialization ~~ + + // user-defined code follows... + self.number = 42 + + // ... + } +} +``` + +### Ready-ing distributed actors + +So far, the initialization process was fairly straightforward. We only needed to find a way to initialize the stored properties, and that's it. There is one more step though that is necessary to make distributed actors work: "ready-ing" the actor. + +As the actor becomes fully initialized, the type system allows escaping its `self` through method calls or closures. There are a number of rules which govern the isolation state of `self` of any actor during its initializer, which are fully explained in: [SE-0327: On Actor Initialization](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md). Distributed actor initializers are subject to the same rules, and in addition to that they inject an `actorReady(self)` at the point where self would have become nonisolated under the rules explained in SE-0327. + +This call is necessary in order for the distributed actor system to be able to resolve an incoming `ID` (that it knows, since it assigned it) to a specific distributed actor instance (which it does not know, until `actorReady` is called on the system). This means that there is a state between the `assignID` and `actorReady` calls, during which the actor system cannot yet properly resolve the actor. + +A distributed actor becomes "ready", and transparently invokes `actorSystem.ready(self)`, during its non-delegating initializer just _before_ the actor's `self` first escaping use, or at the end of the initializer if no explicit escape is found. + +This rule is not only simple to remember, but also consistent between synchronous and asynchronous initializers. The rule plays also very well with the flow-isolation treatment of self in plain actors. + +The following snippets illustrate where the ready call is emitted by the compiler: + +```swift +distributed actor DA { + let number: Int + + init(sync system: ActorSystem) { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + self.number = 42 + // << system.actorReady(self) + } + + init(sync system: ActorSystem) { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + self.number = 42 + // << system.actorReady(self) + Task.detached { // escaping use of `self` + await self.hello() + } + } +} +``` + +If the self of the actor were to be escaped on multiple execution paths, the ready call is injected in all appropriate paths, like this: + +```swift +distributed actor DA { + let number: Int + init(number: Int, system: ActorSystem) async { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + if number % 2 == 0 { + print("even") + self.number = number + // << system.actorReady(self) + something(self) + } else { + print("odd") + self.number = number + // << system.actorReady(self) + something(self) + } + } +} +``` + +Special care needs to be taken about the distributed actor and actor system interaction in the time between the `assignID` and `actorReady` calls, because during this time the system is unable to *deliver* an invocation to the target actor. However, it is always able to recognize that an ID is known, but just not ready yet – the system did create and assign the ID after all. + +This should not be an issue for developers using distributed actors, but actor system authors need to be aware of this interval between the actor ID being reserved and readies. We suggest "reserving" the ID immediately in `assignID` in order to avoid issuing the same ID to multiple actors which can yield unexpected behavior when handling incoming messages. + +Another thing to be aware of is "long" initializers, which take a long time to complete which may sometimes be the case with asynchronous initializers. For example, consider this initializer which performs a lot of work on the passed in items before returning: + +```swift +init(items: [Item], system: ActorSystem) async { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + for await item in items { + await compute(item) + } + // ... + // ... + // ?? what if init "never" returns" ?? + // ... + // ... + // << system.actorReady(self) +} +``` + +This is arguably problematic for any class, struct or actor, however for distributed actors this also means that the period of time during an ID was assigned and will finally be readied can be potentially quite long. In general, we discourage such "long running" initializers as they make use of the actor in distribution impossible until it is readied. On the other hand, though, it can only be used in distribution once the initializer returns in any case so this is a similar problem to any long running initializer. + +#### Ready-ing distributed actors, exactly once + +Another interesting case the `actorReady` synthesis in initializers needs to take care of is triggering the `actorReady` call only *once*, as the actor first becomes fully initialized. The following snippet does a good job showing an example of where it can manifest: + +```swift +distributed actor DA { + var int: Int + init(system: ActorSystem) async { + self.actorSystem = system + var loops = 10 + while loops > 0 { + self.int = loops + // ~ AT THE FIRST ITERATION ~ + // ~ become fully initialized ~ + // ... + escape(self) + + loops -= 1 + } + } +} +``` + +This actor performs a loop during which it assigns values to `self.int`, the actor becomes fully initialized the first time this loop runs. + +We need to emit the `actorReady(self)` call, only once, and we should not repeatedly call the actor system's `actorReady` method which would force system developers into weirdly defensive implementations of this method. Thankfully, this is possible to track in the compiler, and we can emit the ready call only once, based on internal initialization marking mechanisms (that store specific bits for every initialized field). + +The synthesized (pseudo-)code therefore is something like this: + +```swift +distributed actor DA { + var int: Int + init(system: ActorSystem) { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + // MARK INITMAP[id] = INITIALIZED + + var loops = 10 + while loops > 0 { + self.int = loops + // MARK INITMAP[int] = INITIALIZED + // INITMAP: FULLY INITIALIZED + // + // IF INITMAP[IMPLICIT_HOP_TO_SELF] != DONE { + // << + // MARK INITMAP[IMPLICIT_HOP_TO_SELF] = DONE + // } + // + // IF INITMAP[ACTOR_READY] != INITIALIZED { + // << system.actorReady(self) + // MARK INITMAP[ACTOR_READY] = INITIALIZED + // } + escape(self) + + loops -= 1 + } + } +} +``` + +Using this technique we are able to emit the ready call only once, and put off the complexity of dealing with repeated ready calls from distributed actor system library authors. + +> The same technique is used to avoid hopping to the self executor 10 times, and the implicit hop-to-self is only performed once, on the initial iteration where the actor became fully initialized. + +Things get more complex in the face of failable as well as throwing initializers. Specifically, because we not only have to assign identities, we also need to ensure that they are resigned when the distributed actor is deallocated. In the simple, non-throwing initialization case this is simply done in the distributed actor's `deinit`. However, some initialization semantics make this more complicated. + +### Resigning distributed actor IDs + +In addition to assigning `ID` instances to specific actors as they get created, we must also *always* ensure the `ID`s assigned are resigned as their owning actors get destroyed. + +Resigning an `ID` allows the actor system to release any resources it might have held in association with this actor. Most often this means removing it from some internal lookup table that was used to implement the `resolve(ID) -> Self` method of a distributed actor, but it could also imply tearing down connections, clearing caches, or even dropping any in-flight messages addressed to the now terminated actor. + +In the simple case this is trivially solved by deinitialization: we completely initialize the actor, and once it deinitializes, we invoke `resignID` in the actor's deinitializer: + +```swift +deinit { + // << self.actorSystem.resignID(self.id) +} +``` + +This also works with user defined deinitializers, where the resign call is injected as the *first* operation in the deinitializer: + +```swift +// user-defined deinit +deinit { + // << self.actorSystem.resignID(self.id) + print("deinit \(self.id)") +} +``` + +Things get more complicated once we take into account the existence of *failable* and *throwing* initializers. Existing Swift semantics around those types of initializers, and their effect on if and when `deinit` is invoked mean that we need to take special care of them. + +Let us first discuss [failable initializers](https://docs.swift.org/swift-book/LanguageGuide/Initialization.html#ID224), i.e. initializers which are allowed to assign `nil` during their initialization. As actors allow such initializers, distributed actors should too, in order to make the friction of moving from local-only to distributed actors as small as possible. + +```swift +distributed actor DA { + var int: Int + + init?(int: Int, system: ActorSystem) { + self.actorSystem = system + // << self.id = actorSystem.assignID(Self.self) + // ... + if int < 10 { + // ... + // << self.actorSystem.resignID(self.id) + return nil + } + self.int = int + // << self.actorSystem.actorReady(self) + } + + // deinit { + // << self.actorSystem.resignID(self.id) + // } +} +``` + +Due to rules about actor and class init/deinit, when we `return nil` from a failable initializer, its deinitializer *does not run* (!). Because of this, we cannot rely on the deinit to resign the ID as we'd leave an un-used, but still registered identity hanging in the actor system, and the `resignID` is injected just before the "failing return" from such an initializer. This is done transparently, and neither distributed actor developers nor actor system developers need to worry about this: the ID is always resigned properly. + +> This does mean that `resignID` may be called without `actorReady` having ever been called! The system should react to this as it would to any usual resignID and free any resources associated with the identifier. + +Next, we need to discuss *throwing* initializers, and their multiple paths of execution. Again, rules about class and actor deinitialization, are tightly related to whether a type's `deinit` will be executed or not, so let us analyze the following example: + +```swift +distributed actor DA { + var int: Int + + init(int: Int, system: ActorSystem) throws { + self.actorSystem = system + // << self.id = system.assignID(Self.self) + // ... + if int <= 1 { + // ... + // << self.actorSystem.resignID(self.id) + throw Boom() // [1] + } + + if int <= 2 { + self.int = int + // ~ become fully initialized ~ + throw Boom() // [2] + } + + throw Boom() // Boom for good measure... (same as [2] though) + // theoretically, the ready call is inserted at the end of the init: + // << system.actorReady(self) + } + + init(int: Int, system: ActorSystem) async throws { + // << self.id = system.assignID(Self.self) + // ... + if int <= 1 { + // ... + // << self.actorSystem.resignID(self.id) + throw Boom() // [1] + } + + if int <= 2 { + self.int = int + // ~ become fully initialized ~ + // << system.actorReady(self) + throw Boom() // [2] + } + + throw Boom() // Boom for good measure... (same as [2] though) + } + + // deinit { + // << self.actorSystem.resignID(self.id) + // } +} +``` + +The actor shown above both has state that it needs to initialize, and it is going to throw. It will throw either before becoming fully initialized `[1]`, or after it has initialized all of its stored properties `[2]`. Swift handles those two executions differently. Only a fully initialized reference type's `deinit` is going to be executed. This means that if the `init` throws at `[1]` we need to inject a `resignID` call there, while if it throws after becoming fully initialized, e.g. on line `[2]` we do not need to inject the `resignID` call, because the actor's `deinit` along with the injected-there `resignID` will be executed instead. + +Both the synchronous and asynchronous initializers deal with this situation well, because the resign call must be paired with the assign, and if the actor was called ready before it calls `resignID` it does not really impact the resignation logic. + +To summarize, the following are rules that distributed actor system implementors can rely on: + +- `assignID(_:)` will be called exactly-once, at the very beginning of the initialization of a distributed actor associated with the system. +- `actorReady(_:)` will be called exactly-once, after all other properties of the distributed actor have been initialized, and it is ready to receive messages from other peers. By construction, it will also always be called after `assignID(_:)`. +- `resignID(_:)` will be called exactly-once as the actor becomes deinitialized, or fails to finish its initialization. This call will always be made after an `assignID(_:)` call. While there may be ongoing racy calls to the transport as the actor invokes this method, any such calls after `resignID(_:)` was invoked should be handled as if actor never existed to begin with. + +Note that the system usually should not hold the actor with a strong reference, as doing so inhibits its ability to deinit until the system lets go of it. + +### Resolving distributed actors + +Every distributed actor type has a static "resolve" method with the following signature: + +```swift +extension DistributedActor { + public static func resolve(id: Self.ID, using system: Self.ActorSystem) throws -> Self { + ... + } +} +``` + +This method will return a distributed actor reference, or throw when the actor system is unable to resolve the reference. + +The `resolve(id:using:)` method on distributed actors is an interesting case of the Swift runtime collaborating with the `DistributedActorSystem`. The Swift runtime implements this method as calling the passed-in actor system to resolve the ID, and if the system claims that this is a _remote_ reference, the Swift runtime will allocate a _remote_ distributed actor reference, sometimes called a "proxy" instance. + +Its implementation can be thought of as follows: + +```swift +extension DistributedActor { + // simplified implementation + static func resolve(id: Self.ID, using system: ActorSystem) throws -> Self { + switch try system.resolve(id: id, as: Self.self) { + case .some(let localInstance): + return localInstance + case nil: + return <>(id: id, system: system) + } + } +} +``` + +Specifically, this calls into the `ActorSystem`'s `resolve(id:as:)` method which has a slightly different signature than the one defined on actors, specifically it can return `nil` to signal the instance is not found in this actor system, but we're able to proxy it. + +> The **result** of `resolve(id:using:)` may be a **local instance** of a distributed actor, or a **reference to a remote** distributed actor. + +The resolve implementation should be fast, and should be non-blocking. Specifically, it should *not* attempt to contact the remote peer to confirm whether this actor really exists or not. Systems should blindly resolve remote identifiers assuming the remote peer will be able to handle them. Some systems may after all spin up actor instances lazily, upon the first message sent to them etc. + +Allocating the remote reference is implemented by the Swift runtime, by creating a fixed-size object that serves only the purpose of proxying calls into the `system.remoteCall`. The `_isDistributedRemoteActor()` function always returns `true` for such a reference. + +If the system entirely fails to resolve the ID, e.g. because it was ill-formed or the system is unable to handle proxies for the given ID, it must throw with an error conforming to `DistribtuedActorSystemError`, rather than returning `nil`. An example implementation could look something like this: + +```swift +final class ClusterSystem: DistributedActorSystem { + private let lock: Lock + private var localActors: [ActorID: AnyWeaklyHeldDistributedActor] // stored into during actorReady + + // example implementation; more sophisticated ones can exist, but boil down to the same idea + func resolve(id: ID, as actorType: Actor.Type) + throws -> Actor? where Actor: DistributedActor, + Actor.ID == Self.ActorID, + Actor.SerializationRequirement == Self.SerializationRequirement { + if validate(id) == .illegal { + throw IllegalActorIDError(id) + } + + return lock.synchronized { + guard let known = self.localActors[id] else { + return nil // not local actor, but we can allocate a remote reference for it + } + + return try known.as(Actor.self) // known managed local instance + } + } +} +``` + +The types work out correctly since it is the specific actor system that has _assigned_ the `ID`, and stored the specific distributed actor instance for the specific `ID`. + +> **Note:** Errors thrown by actor systems should conform to the `protocol DistributedActorSystemError: Error {}` protocol. While it is just a marker protocol, it helps end users understand where an error originated. + +Attempting to ready using one type, and resolve using another will cause a throw to happen during the resolve, e.g. like this: + +```swift +distributed actor One {} +distributed actor Two {} + +let one = One(system: cluster) +try Two.resolve(id: one.id, using: cluster) +// throws: DistributedActorResolveError.wrongType(found: One) // system specific error +``` + +This is only the case for local instances, though. For remote instances, by design, the local actor system does not track any information about them and as any remote call can fail anyway, the failures surface at call-site (as the remote recipient will fail to be resolved). + +### Invoking distributed methods + +Invoking a distributed method (or distributed computed property) involves a number of steps that occur on two "sides" of the call. + +The local/remote wording which works well with actors in general can get slightly confusing here, because every call made "locally" on a "remote reference", actually results in a "local" invocation execution on the "remote system". Instead, we will be using the terms "**sender**" and "**recipient**" to better explain which side of a distributed call we are focusing on. + +As was shown earlier, invoking a `distributed func` essentially can follow one of two execution paths: + +- if the distributed actor instance was **local**: + - the call is made directly, as if it was a plain-old local-only `actor` +- if the distributed actor was **remote**: + - the call must be transformed into an invocation that will be offered to the `system.remoteCall(...)` method to execute + +The first case is governed by normal actor execution rules. There might be a context switch onto the actor's executor, and the actor will receive and execute the method call as usual. + +In this section, we will explain all the steps involved in the second, remote, case of a distributed method call. The invocations will be using two very important types that represent the encoding and decoding side of such distributed method invocations. + +The full listing of those types is presented below: + +```swift +protocol DistributedActorSystem: ... { + // ... + associatedtype InvocationEncoder: DistributedTargetInvocationEncoder + associatedtype InvocationDecoder: DistributedTargetInvocationDecoder + + func makeInvocationEncoder() -> InvocationEncoder +} +``` + + + +```swift +public protocol DistributedTargetInvocationEncoder { + associatedtype SerializationRequirement + + /// Record a type of generic substitution which is necessary to invoke a generic distributed invocation target. + /// + /// The arguments must be encoded order-preserving, and once `decodeGenericSubstitutions` + /// is called, the substitutions must be returned in the same order in which they were recorded. + mutating func recordGenericSubstitution(_ type: T.Type) throws + + /// Record an argument of `Argument` type in this arguments storage. + /// + /// The argument name is provided to the `name` argument and can either be stored or ignored. + /// The value of the argument is passed as the `argument`. + /// + /// Ad-hoc protocol requirement. + mutating func recordArgument( + _ argument: RemoteCallArgument + ) throws + + /// Record the error type thrown by the distributed invocation target. + /// If the target does not throw, this method will not be called and the error type can be assumed `Never`. + mutating func recordErrorType(_ type: E.Type) throws + + /// Record the return type of the distributed method. + /// If the target does not return any specific value, this method will not be called and the return type can be assumed `Void`. + /// + /// Ad-hoc protocol requirement. + mutating func recordReturnType(_ type: R.Type) throws + + /// All values and types have been recorded. + /// Optionally "finalize" the recording, if necessary. + mutating func doneRecording() throws +} + +/// Represents an argument passed to a distributed call target. +public struct RemoteCallArgument { + /// The "argument label" of the argument. + /// The label is the name visible name used in external calls made to this + /// target, e.g. for `func hello(label name: String)` it is `label`. + /// + /// If no label is specified (i.e. `func hi(name: String)`), the `label`, + /// value is empty, however `effectiveLabel` is equal to the `name`. + /// + /// In most situations, using `effectiveLabel` is more useful to identify + /// the user-visible name of this argument. + let label: String? + var effectiveLabel: String { + return label ?? name + } + + /// The internal name of parameter this argument is accessible as in the + /// function body. It is not part of the functions API and may change without + /// breaking the target identifier. + /// + /// If the method did not declare an explicit `label`, it is used as the + /// `effectiveLabel`. + let name: String + + /// The value of the argument being passed to the call. + /// As `RemoteCallArgument` is always used in conjunction with + /// `recordArgument` and populated by the compiler, this Value will generally + /// conform to a distributed actor system's `SerializationRequirement`. + let value: Value +} +``` + + + +```swift +public protocol DistributedTargetInvocationDecoder { + associatedtype SerializationRequirement + + mutating func decodeGenericSubstitutions() throws -> [Any.Type] + + /// Attempt to decode the next argument from the underlying buffers into pre-allocated storage + /// pointed at by 'pointer'. + /// + /// This method should throw if it has no more arguments available, if decoding the argument failed, + /// or, optionally, if the argument type we're trying to decode does not match the stored type. + /// + /// Ad-hoc protocol requirement. + mutating func decodeNextArgument() throws -> Argument + + mutating func decodeErrorType() throws -> Any.Type? + + mutating func decodeReturnType() throws -> Any.Type? +} +``` + +#### Sender: Invoking a distributed method + +A call to a distributed method (or computed property) on a remote distributed actor reference needs to be turned into a runtime introspectable representation which will be passed to the `remoteCall` method of a specific distributed actor system implementation. + +In this section, we'll see what happens for the following `greet(name:)` distributed method call: + +```swift +// distributed func greet(name: String) -> String { ... } + +try await greeter.greet(name: "Alice") +``` + +Such invocation is calling the method via a "distributed thunk" rather than directly. The "distributed thunk" is synthesized by the compiler for every `distributed func`, and can be illustrated by the following snippet: + +```swift +// ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ SYNTHESIZED ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +extension Greeter { + // synthesized; not user-accessible thunk for: greet(name: String) -> String + nonisolated func greet_$distributedThunk(name: String) async throws -> String { + guard _isDistributedRemoteActor(self) else { + // the local func was not throwing, but since we're nonisolated in the thunk, + // we must hop to the target actor here, meaning the 'await' is always necessary. + return await self.greet(name: name) + } + + // [1] prepare the invocation object: + var invocation = self.actorSystem.makeInvocationEncoder() + + // [1.1] if method has generic parameters, record substitutions + // e.g. for func generic(a: A, b: B) we would get two substitutions, + // for the generic parameters A and B: + // + // << invocation.recordGenericSubstitution() + // << invocation.recordGenericSubstitution() + + // [1.2] for each argument, synthesize a specialized recordArgument call: + try invocation.recordArgument(RemoteCallArgument(label: nil, name: "name", value: name)) + + // [1.3] if the target was throwing, record Error.self, + // otherwise do not invoke recordErrorType at all. + // + // << try invocation.recordErrorType(Error.self) + + // [1.4] we also record the return type; it may or may not be necessary to + // transmit over the wire but if necessary, the system may choose to do so. + // + // This call is not made when the return type is Void. + try invocation.recordReturnType(String.self) + + // [1.5] done recording arguments + try invocation.doneRecording() + + // [2] invoke the `remoteCall` method of the actor system + return try await self.actorSystem.remoteCall( + on: self, + target: RemoteCallTarget(...), + invocation: invocation, + throwing: Never.self, // the target func was not throwing + returning: String.self + ) + } +} +``` + +The synthesized thunk is always throwing and asynchronous. This is correct because it is only invoked in situations where we might end up calling the `actorSystem.remoteCall(...)` method, which by necessity is asynchronous and throwing. + +The thunk is `nonisolated` because it is a method that can actually run on a *remote* instance, and as such is not allowed to touch any other state than other nonisolated stored properties. This is specifically designed such that the thunk and actor system are able to access the `id` of the actor (and the `actorSystem` property itself) which is necessary to perform the actual remote message send. + +The `nonisolated` aspect of the method has another important role to play: if this invocation happens to be on a local distributed actor, we do not want to "hop" executors twice. If this invocation were on a local actor, only accessing `nonisolated` state, or for other reasons the hop could be optimized away, we want to keep this ability for the optimizer to do as good of a job as it would for local only actors. If the instance was remote, we don't need to suspend early at all, and we leave it to the `ActorSystem` to decide when exactly the task will suspend. For example, the system may only suspend the call after it has sent the bytes synchronously over some IPC channel, etc. The semantics of to suspend are highly dependent on the specific underlying transport, and thanks to this approach we allow system implementations to do the right thing, whatever that might be: they can suspend early, late, or even not at all if the call is known to be impossible to succeed. + +Note that the compiler will pass the `self` of the distributed *known-to-be-remote* actor to the `remoteCall` method on the actor system. This allows the system to check the passed type for any potential, future, customization points that the actor may declare as static properties, and/or conformances affecting how a message shall be serialized or delivered. It is impossible for the system to access any of that actor's state, because it is remote after all. The one piece of state it will need to access though is the actor's `id` because that is signifying the *recipient* of the call. + +The thunk creates the `invocation` container `[1]` into which it records all arguments. Note that all these APIs are using only concrete types, so we never pay for any existential wrapping or other indirections. Arguments are wrapped in a `RemoteCallArgument` which also provides additional information about the argument, such as the labels used in its declaration. This can be useful for transports which wish to encode values in named dictionaries, e.g. such as JSON dictionaries. It is important to recognize that storing names or labels is _not_ required or recommended for the majority of transports, however for those which need it, they are free to use this additional information. + +Since the strings created are string literals, known at compile time, there is no allocation impact for this argument wrapper type and this additional label information. + +> It may seem tempting to simply record arguments into a dictionary using their effective labels, however this can lead to issues due to labels being allowed to be reused. For example, `func sum(a a1: Int, a a2: Int)` is a legal, although not very readable, function declaration. A naive encoding scheme could risk overriding the "first `a`" value with the "second `a` if it strongly relied on the labels only. A distributed actor system implementation should decide how to deal with such situations and may choose to throw at runtime if such a risky signature is detected, or apply some mangling, e.g. use the parameter names to clarify which value was encoded. + +The `record...` calls are expected to serialize the values, using any mechanism they want to, and thanks to the fact that the type performing the recording is being provided by the specific `ActorSystem`, it also knows that it can rely on the arguments to conform to the system's `SerializationRequirement`. + +The first step in the thunk is to record any "generic substitutions" `[1.1]` if they are necessary. This makes it possible for remote calls to support generic arguments, and even generic distributed actors. The substitutions are not recorded for call where the generic context is not necessary for the invocation. For a generic method, however, the runtime will invoke the `recordGenericTypeSubstitution` with _concrete_ generic arguments that are necessary to perform the call. For example, if we declared a generic `echo` method like this: + +```swift +distributed func echo(_ value: T) -> T +``` + +and call it like this: + +```swift +try await greeter.echo("Echo!") // typechecks ok; String: SerializationRequirement +``` + +The Swift runtime would generate the following call: + +```swift +try invocation.recordGenericTypeSubstitution(String.self) +``` + +This method is implemented by a distributed actor system library, and can use this information to double-check this type against an allow-list of types allowed to be transmitted over the wire, and then store and send it over to the recipient such that it knows what type to decode the argument as. + +Next, the runtime will record all arguments of the invocation `[1.2]`. This is done in a series of `recordArgument` calls. If the type of actor the target is declared on also includes a generic parameter that is used by the invocation, this also is recorded. + +As the `recordArgument(_:)` method is generic over the argument type (``), and requires the argument to conform to `SerializationRequirement` (which in turn was enforced at compile time by [SE-0336][isolation]), the actor system implementation will have an easy time to serialize or validate this argument. For example, if the `SerializationRequirement` was codable — this is where one could invoke `SomeEncoder().encode(argument)` because `Argument` is a concrete type conforming to `Codable`! + +Finally, the specific error `[1.3]` and return types `[1.4]` are also recorded. If the function is not throwing, `recordErrorType` is not called. Likewise, if the return type is `Void` the `recordReturnType` is not called. + +Recording the error type is mostly future-proofing and currently will only ever be invoked with the `Error.self` or not at all. It allows informing the system if a throw from the remote side is to be expected, and technically, if Swift were to gain typed throws this method could record specific expected error types as well — although we have no plans with regards to typed throws at this point in time. + +The last encoder call `doneRecording()` is made, to signal to the invocation encoder that no further record calls will be made. This is useful since with the optional nature of some of the calls, it would be difficult to know for a system implementation when the invocation is fully constructed. Operations which may want to be delayed until completion could include serialization, de-duplicating values or similar operations which benefit from seeing the whole constructed invocation state in the encoder. + +Lastly, the populated encoder, along with additional type and function identifying information is passed to the `remoteCall`, or `remoteCallVoid` method on the actor system which should actually perform the message request/response interaction with the remote actor. + +#### Sender: Serializing and sending invocations + +The next step in making a remote call is serializing a representation of the distributed method (or computed property) invocation. This is done through a series of compiler, runtime, and distributed actor system interactions. These interactions are designed to be highly efficient and customizable. Thanks to the `DistributedTargetInvocationEncoder`, we are able to never resort to existential boxing of values, allow serializers to manage and directly write into their destination buffers (i.e. allowing for zero copies to be performed between the message serialization and the underlying networking layer), and more. + +Let us consider a `ClusterSystem` that will use `Codable` and send messages over the network. Most systems will need to form some kind of "Envelope" (easy to remember as: "the thing that contains the **message** and also has knowledge of the **recipient**"). For the purpose of this proposal, we'll define a a `WireEnvelope` and use it in the next snippets to showcase how a typical actor system would work with it. This type is not pre-defined or required by this proposal, but it is something implementations will frequently do on their own: + +```swift +// !! ClusterSystem or WireEnvelope are NOT part of the proposal, but serves as illustration how actor systems might !! +// !! implement the necessary pieces of the DistributedActorSystem protocol. !! + +final struct ClusterSystem: DistributedActorSystem { + // ... + typealias SerializationRequirement = Codable + typealias InvocationEncoder = ClusterTargetInvocationEncoder + typealias InvocationDecoder = ClusterTargetInvocationDecoder + + // Just an example, we can implement this more efficiently if we wanted to. + private struct WireEnvelope: Codable, Sendable { + var recipientID: ClusterSystem.ActorID // is Codable + + /// Mangled method/property identifier, e.g. in a mangled format + var identifier: String + + // Type substitutions matter only for distributed methods which use generics: + var genericSubstitutions: [String] + + // For illustration purposes and simplicity of code snippets we use `[Data]` here, + // but real implementations can be much more efficient here — packing all the data into exact + // byte buffer that will be passed to the networking layer, etc. + var arguments: [Data] // example is using Data, because that's what Codable coders use + + // Metadata can be used by swift-distributed-tracing, or other instrumentations to carry extra information: + var metadata: [String: [Data]] // additional metadata, such as trace-ids + } +} +``` + +Note that `method` property is enough to identify the target of the call, we do not need to carry any extra type information explicitly in the call. The method identifier is sufficient to resolve the target method on the recipient, however in order to support generic distributed methods, we need to carry additional (mangled) type information for any of the generic parameters of this specific method invocation. Thankfully, these are readily provided to us by the Swift runtime, so we'll only need to store and send them over. + +> **Note:** An implementation may choose to define any shape of "envelope" (or none at all) that suits its needs. It may choose to transport mangled names of involved types for validation purposes, or choose to not transfer them at all and impose other limitations on the system and its users for the sake of efficiency. +> +> While advanced implementations may apply compression and other techniques to minimize the overhead of these envelopes — this is a deep topic by itself, and we won't be going in depth on it in this proposal — rest assured though, we have focused on making different kinds of implementations possible with this approach. + +Next, we will discuss how the `InvocationEncoder` can be implemented in order to create such `WireEnvelope`. + +> Note on ad-hoc requirements: Some of the protocol requirements on the encoder, as well as actor system protocols, are so-called "ad-hoc" requirements. This means that they are not directly expressed in Swift source, but instead the compiler is aware of the signatures and specifically enforces that a type conforming to such protocol implements these special methods. +> +> Specifically, methods which fall into this category are functions which use the `SerializationRequirement` as generic type requirement. This is currently not expressible in plain Swift, due to limitations in the type system which are difficult to resolve immediately, but in time as this could become implementable these requirements could become normal protocol requirements. +> +> This tradeoff was discussed at length and we believe it is worth taking, because it allows us to avoid numerous un-necessary type-casts, both inside the runtime and actor system implementations. It also allows us to avoid any existential boxing and thus lessens the allocation footprint of making remote calls which is an important aspect of the design and use cases we are targeting. + +The following listing illustrates how one _could_ implement a `DistributedTargetInvocationEncoder`: + +```swift +extension ClusterSystem { + typealias InvocationEncoder = ClusterTargetInvocationEncoder + + func makeInvocationEncoder() -> Self.InvocationEncoder { + return ClusterTargetInvocation(system: system) + } +} + +struct ClusterTargetInvocationEncoder: DistributedTargetInvocationEncoder { + typealias SerializationRequirement = ClusterSystem.SerializationRequirement + + let system: ClusterSystem + var envelope: Envelope + + init(system: ClusterSystem) { + self.system = system + self.envelope = .init() // new "empty" envelope + } + + /// The arguments must be encoded order-preserving, and once `decodeGenericSubstitutions` + /// is called, the substitutions must be returned in the same order in which they were recorded. + mutating func recordGenericSubstitution(_ type: T.Type) throws { + // NOTE: we are showcasing a pretty simple implementation here... + // advanced systems could use mangled type names or registered type IDs. + envelope.genericSubstitutions.append(String(reflecting: T.self)) + } + + mutating func recordArgument(_ argument: RemoteCallArgument) throws { + // in this implementation, we just encode the values one-by-one as we receive them: + let argData = try system.encoder.encode(argument) // using whichever Encoder the system has configured + envelope.arguments.append(argData) + } + + mutating func recordErrorType(_ errorType: E.Type) throws { + envelope.errorType = String(reflecting: errorType) + } + + mutating func recordReturnType(_ returnType: R.Type) throws { + envelope.returnType = String(reflecting: returnType) + } + + /// Invoked when all the `record...` calls have been completed and the `DistributedTargetInvocation` + /// will be passed off to the `remoteCall` to perform the remote call using this invocation representation. + mutating func doneRecording() throws { + // our impl does not need to do anything here + } +} +``` + +The above encoder is going to be called by the Swift runtime as was explained in the previous section. + +Once that is complete, the runtime will pass the constructed `InvocationEncoder` to the `remoteCall`: + +```swift + extension ClusterSystem { + // 'remoteCall' is not a protocol requirement, however its signature is well known to the compiler, + // and it will invoke the method. We also are guaranteed that the 'Success: Codable' requirement is correct, + // since the type-system will enforce this conformance thanks to the type-level checks on distributed funcs. + func remoteCall( + on actor: Actor, + target: RemoteCallTarget, + invocation: Self.InvocationEncoder, + throwing: Failure.Type, + returning: Success.Type + ) async throws -> Success + where Actor: DistributedActor, + Actor.ID == ActorID. + Failure: Error, + Success: Self.SerializationRequirement { + var envelope = invocation.envelope + + // [1] the recipient is transferred over the wire as its id + envelope.recipient = recipient.id + + // [2] the method is a mangled identifier of the 'distributed func' (or var). + // In this system, we just use the mangled name, but we could do much better in the future. + envelope.target = target.identifier + + // [3] send the envelope over the wire and await the reply: + let responseData = try await self.underlyingTransport.send(envelope, to: actor.id) + + // [4] decode the response from the response bytes + // in our example system, we're using Codable as SerializationRequirement, + // so we can decode the response like this (and never need to cast `as? Codable` etc.): + return try self.someDecoder.decode(as: Success.self, from: responseData) + } +} +``` + +The overall purpose of this `remoteCall` implementation is to create some form of message representation of the invocation and send it off to the remote node (or process) to receive and invoke the target method on the remote actor. + +In our example implementation, the `Invocation` already serialized the arguments and stored them in the `Envelope`, so the `remoteCall` only needs to add the information about the call recipient `[1]`, and the target (method or computed property) of the call `[2]`. In our example implementation, we just store the target's mangled name `[2]`, which is simple, but it has its challenges in regard to protocol evolution. + +One notable issue that mangled names have is that any change in the method signature will result in not being able to resolve the target method anymore. We are very much aware of the issues this may cause to protocol evolution, and we lay out plans in [Future Work](#future-work) to improve the lookup mechanisms in ways that will even allow adding parameters (with default values), in wire (and ABI) compatible ways. + +The final step is handing over the envelope containing the encoded arguments, recipient information, etc., to the underlying transport mechanism `[3]`. The transport does not really have to concern itself with any of the specifics of the call, other than transmitting the bytes to the callee and the response data back. As we get the response data back, we have the concrete type of the expected response and can attempt to decode it `[4]`. + +> Note on `remoteCallVoid`: One limitation in the current implementation approach is that a remote call signature cannot handle void returning methods, because of the `Success: SerializationRequirement` requirement on the method. +> +> This will be possible to solve using the incoming [Variadic Generics](https://forums.swift.org/t/variadic-generics/54511) language feature that is being currently worked on and pitched. With this feature, the return type could be represented as variadic generic and the `Void` return type would be modeled as "empty" tuple, whereas a value return would contain the specific type of the return, this way we would not violate the `Success: SerializationRequirement` when we needed to model `Void` calls. + +#### Recipient: Receiving invocations + +On the remote side, there usually will be some receive loop or similar mechanism that is implemented in the transport layer of the actor system. In practice this often means binding a port and receiving TCP (or UDP) packets, applying some form of framing and eventually decoding the incoming message envelope. + +Since the communication of the sending and receiving side is going to be implemented by the same type of transport and actor system, receiving the envelopes is straightforward: we know the wire protocol, and follow it to receive enough bytes to decode the `Envelope` which we sent a few sections above. + +This part does not have anything specific prescribed in the `DistributedActorSystem` protocol. It is up to every system to implement whichever transport mechanism works for it. While not a "real" snippet, this can be thought of a simple loop over incoming connections, like this: + +```swift +// simplified pseudo code for illustration purposes +func receiveLoop(with node: Node) async throws { + for try await envelopeData in connection(node).receiveEnvelope { + await self.receive(envelopeData) + } +} +``` + +In a real server implementation we'd likely use a [Swift NIO](https://github.com/apple/swift-nio) `ChannelPipeline` to perform this networking, framing and emitting of `Envelope`s, but this is beyond the scope of what we need to explain in this proposal to get the general idea of how this is going to work. + +#### Recipient: Deserializing incoming invocations + +Now that we have received all the bytes for one specific envelope, we need to perform a two-step deserialization on it. + +First, we'll need to decode the target identifier (e.g. method name, mangled method name, or some other form of target identifier), and the actor `ID` of the recipient. These are necessary to decode always, as we need to locate both the method and actor we're trying to invoke. + +Next, the deserialization of the actual message representation of our invocation will take place. However, this is done lazily. Rather than just decoding the values and storing them somewhere in our system implementation, these will be requested by the Swift runtime when it is about to perform the method call. + +Before we dive deeper into this, let us visualize how this two-step process is intended to work, by looking at what might be a typical envelope format on the wire: + +```c ++------------------------------- ENVELOPE --------------------------------------+ +| +---------- HEADER --------++-------------------- MESSAGE ------------------+ | +| | target | recipient | ... || [ ... lazy decoded section: types, args ... ] | | +| +--------------------------++-----------------------------------------------+ | ++-------------------------------------------------------------------------------+ +``` + +We see that as we decode our wire envelope, we are able to get the header section, and all values contained within it eagerly and leave the remaining slice of the buffer untouched. It will be consumed during performing of the invocation soon enough. The nice thing about this design is that we're still able to hold onto the actual buffer handed us from the networking library, and we never had to copy the buffer to our own local copies. + +Next, we need to prepare for the decoding of the message section. This is done by implementing the remaining protocol requirements on the `ClusterTargetInvocation` type we defined earlier, as well as implementing a decoding iterator of type `DistributedTargetInvocationArgumentDecoder`, as shown below: + +```swift +struct ClusterTargetInvocationDecoder: DistributedTargetInvocationDecoder { + typealias SerializationRequirement = Codable + + let system: ClusterSystem + var bytes: ByteBuffer + + mutating func decodeGenericSubstitutions() throws -> [Any.Type] { + let subCount = try self.bytes.readInt() + + var subTypes: [Any.Type] = [] + for _ in 0..() throws -> Argument { + try nextDataLength = try bytes.readInt() + let nextData = try bytes.readData(bytes: nextDataLength) + // since we are guaranteed the values are Codable, so we can just invoke it: + return try system.decoder.decode(as: Argument.self, from: bytes) + } + + mutating func decodeErrorType() throws -> Any.Type? { + let length = try self.bytes.readInt() // read the length of the type + guard length > 0 { + return nil // we don't always transmit it, 0 length means "none" + } + let typeName = try self.bytes.readString(length: length) + return try self.system.summonType(byName: typeName) + } + + mutating func decodeReturnType() throws -> Any.Type? { + let length = try self.bytes.readInt() // read the length of the type + guard length > 0 { + return nil // we don't always transmit it, 0 length means "none" + } + let typeName = try self.bytes.readString(length: length) + return try self.system.summonType(byName: typeName) + } +} +``` + +The general idea here is that the `InvocationDecoder` is *lazy* in its decoding and just stores the remaining bytes of the envelope. All we need to do for now is to implement the Invocation in such way that it expects the decoding methods be invoked in the following order (which is the same as the order on the sending side): + +- 0...1 invocation of `decodeGenericArguments`, +- 0...n invocation(s) of `decoder.decodeNextArgument`, +- 0...1 invocation of `decodeReturnType`, +- 0...1 invocation of `decodeErrorType`. + +Decoding arguments is the most interesting here. This is another case where the compiler and Swift runtime enable us to implement things more easily. Since the `Argument` generic type of the `decodeNextArgument` is ensured to conform to the `SerializationRequirement`, actor system implementations can rely on this fact and have a simpler time implementing the decoding steps. For example, with `Codable` the decoding steps becomes a rather simple task of invoking the usual `Decoder` APIs. + +This decoder must be prepared by the actor system and eventually passed to the `executeDistributedTarget` method which we'll discuss next. That, Swift runtime provided, function is the one which will be calling the `decode...` methods and will is able to ensure all the type requirements are actually met and form the correct generic method invocations. + +> **Note:** This proposal does not include an implementation for the mentioned `summonType(byName:)` function, it is just one way systems may choose to implement these functions. Possible implementations include: registering all "trusted" types, using mangled names, or something else entirely. This proposal has no opinion about how these types are recovered from the transmitted values. + +#### Recipient: Resolving the recipient actor instance + +Now that we have prepared our `InvocationDecoder` we are ready to make the next step, and resolve the recipient actor which the invocation shall be made on. + +We already discussed how resolving actors works in [Resolving distributed actors](#resolving-distributed-actors), however in this section we can tie it into the real process of invoking the target function as well. + +In the example we're following so far, the recipient resolution is simple because we have the recipient ID available in the `Envelope.recipientID`, so we only need to resolve that using the system that is receiving the message: + +```swift +guard let actor: any DistributedActor = try self.knownActors[envelope.recipientID] else { + throw ClusterSystemError.unknownRecipient(envelope.recipientID) +} +``` + +This logic is the same as the internal implementation of the `resolve(id:as:)` method only that we don't have a need to validate the specific type of the actor — this will be handled by the Swift runtime in `executeDistributedTarget`'s implementation the target of the call which we'll explain in the next section. + +#### Recipient: The `executeDistributedTarget` method + +Invoking a distributed method is a tricky task, and involves a lot of type demangling, opening existential types, forming specific generic invocations and tightly managing all of that in order to avoid un-necessary heap allocations to pass the decoded arguments to the target function, etc. After iterating over multiple designs, we decided to expose a single `DistributedActorSystem.executeDistributedTarget` entry point which efficiently performs all the above operations. + +Thanks to abstracting the decoding logic into the `DistributedTargetInvocationDecoder` type, all deserialization can be made directly from the buffers that were received from the underlying network transport. The `executeDistributedTarget` method has no opinion about what serialization mechanism is used either, and any mechanism — be it `Codable` or other external serialization systems — can be used, allowing distributed actor systems developers to implement whichever coding strategy they choose, potentially directly from the buffers obtained from the transport layer. + +The `executeDistributedTarget` method is defined as: + +```swift +extension DistributedActorSystem { + /// Prepare and execute a call to the distributed function identified by the passed arguments, + /// on the passed `actor`, and collect its results using the `ResultHandler`. + /// + /// This method encapsulates multiple steps that are invoked in executing a distributed function, + /// into one very efficient implementation. The steps involved are: + /// + /// - looking up the distributed function based on its name; + /// - decoding, in an efficient manner, all arguments from the `Args` container into a well-typed representation; + /// - using that representation to perform the call on the target method. + /// + /// The reason for this API using a `ResultHandler` rather than returning values directly, + /// is that thanks to this approach it can avoid any existential boxing, and can serve the most + /// latency sensitive-use-cases. + func executeDistributedTarget( + on actor: Actor, + mangledName: String, + invocation: inout Self.InvocationDecoder, + handler: ResultHandler + ) async throws where Actor: DistributedActor, + Actor.ID == ActorID, + ResultHandler: DistributedTargetInvocationResultHandler { + // implemented by the _Distributed library + } +} +``` + +This method encapsulates all the difficult and hard to implement pieces of the target invocation, and it accepts the base actor the call should be performed on, along with a `DistributedTargetInvocationResultHandler`. + +Rather than having the `executeDistributedTarget` method return an `Any` result, we use the result handler in order to efficiently, and type-safely provide the result value to the actor system library implementation. This technique is the same as we did with the `recordArgument` method before, and it allows us to provide the _specific_ type including its `SerializationRequirement` conformance making handling results much simpler, and without having to resort to any casts which can be unsafe if used wrongly, or have impact on runtime performance. + +The `DistributedTargetInvocationResultHandler` is defined as follows: + +```swift +protocol DistributedTargetInvocationResultHandler { + associatedtype SerializationRequirement + + func onReturn(value: Success) async throws + where Success: SerializationRequirement + func onThrow(error: Error) async throws + where Failure: Error +} +``` + +In a way, the `onReturn`/`onThrow` methods can be thought of as the counterparts of the `recordArgument` calls on the sender side. We need to encode the result and send it _back_ to the sender after all. This is why providing the result value along with the appropriate SerializationRequirement conforming type is so important — it makes sending back the reply to a call, as simple as encoding the argument of the call. + +Errors must be handled by informing the sender about the failed call. This is in order to avoid senders waiting and waiting for a reply, and eventually triggering a timeout; rather, they should be informed as soon as possible that a call has failed. Treat an error the same way as you would a valid return in terms of sending the reply back. However, it is not required to actually send back the actual error, as it may not be safe, or a good idea from a security and information exposure perspective, to send back entire errors. Instead, systems are encouraged to send back a reasonable amount of information about a failure, and e.g. optionally, only if the thrown error type is `Codable` and allow-listed to be sent over the wire, transport it directly. + +#### Recipient: Executing the distributed target + +Now that we have completed all the above steps, all building up to actually invoking the target of a remote call: it is finally time to do so, by calling the `executeDistributedTarget` method: + +```swift +// inside recipient actor system +let envelope: IncomingEnvelope = // receive & decode ... +let recipient: DistributedActor = // resolve ... + +let invocationDecoder = InvocationDecoder(system: self, bytes: envelope.bytes) + +try await executeDistributedTarget( + on: recipient, // target instance for the call + mangledName: envelope.targetName, // target func/var for the call + invocation: invocationDecoder // will be used to perform decoding arguments, + handler: ClusterTargetInvocationResultHandler(system, envelope) // handles replying to the caller (omitted in proposal) +) +``` + +This call triggers all the decoding that we discussed earlier, and if any of the decoding, or distributed func/var resolution fails this call will throw. Otherwise, once all decoding has successfully been completed, the arguments are passed through the buffer to a distributed method accessor that actually performs the local method invocation. Once the method returns, its results are moved into the handler where the actor system takes over in order to send a reply to the remote caller — completing the remote call! + +Internally, the execute distributed thunk heavily relies on the lookup and code generated by the compiler for every `distributed func` which we refer to as **distributed method accessor thunk**. This thunk is able to decode incoming arguments using the `InvocationDecoder` and directly apply the target function, all while properly handling generics and other important aspects of function invocations. It is the distributed method accessor thunk that must be located using the "target identifier" when we handle an incoming the remote call, the thunk then calls the actual target function. + +For sake of completeness, the listing below shows the distributed method accessor thunk that is synthesized by the compiler. The thunk contains compiler synthesized logic specific to every distributed function to locate the target function, obtain the expected parameter types and use the passed in decoder to decode the arguments to finally pass them to the final function application. + +The thunk can be thought of in terms of this abstract example. However, it cannot be implemented like this because of various interactions with the generic system as well as how emissions (function calls) actually work. Distributed method accessor thunks are implemented directly in IR as it would not be possible to synthesize the necessary emissions in any higher level part of the compiler (!). Thankfully, the logic contained in those accessors is fairly straightforward and can be imagined as: + +```swift +distributed actor DA { + func myCompute(_ i: Int, _ s: String, _ d: Double) async throws -> String { + "i:\(i), s:\(s), d:\(d)" + } +} + +extension DA { + // Distributed accessor thunk" for 'myCompute(_:_:_:) -> String' + // + // PSEUDO CODE FOR ILLUSTRATION PURPOSES; NOT IMPLEMENTABLE IN PLAIN SWIFT; + // Implemented in directly in IR for expressibility reasons, and not user-accessible. + nonisolated func $distributedFuncAccessor_myCompute( + decoder: UnsafeMutableRawPointer, + argumentTypes: UnsafeRawPointer, + resultBuffer: UnsafeRawPointer, + genericSubstitutions: UnsafeRawPointer, + witnessTables: UnsafeRawPointer, + numWitnessTables: Int, + actorSelf: UnsafeRawPointer) async { + + // - get generic signature of 'myCompute' + // - create storage 'args' for all the parameters; it will be used directly + // - for every argument, get the argumentType + // - invoke 'decoder.decodeArgument()' + // - store in 'args' + // - deal with the generic substitutions, witness tables and prepare the call + // invoke 'myCompute' with 'args', and the prepared 'result' and 'error' buffers + } +} +``` + +As we can see, this thunk is "just" taking care of converting the heterogeneous parameters into the well typed counterparts, and finally performing a plain-old method invocation using those parameters. The actual code emission and handling of generics for all this to work is rather complex and can only be implemented in the IR layer of the compiler. The good part about it is that the compiler is able to prepare and emit good errors in case the types or witness tables seem to be mismatched with the target or other issues are found. Allocations are also kept to a minimum, as no intermediate allocations need to be made for the arguments and they are stored and directly emitted into the call emission of the target. + +The thunk again uses the indirect return, so we can avoid any kind of implicit existential boxing even on those layers. Errors are always returned indirectly, so we do not need to do it explicitly. + +#### Recipient: Collecting result/error from invocations + +Now that the distributed method has been invoked, it eventually returns or throws an error. + +Collecting the return (or error) value is also implemented using the `DistributedMethodInvocationHandler` we passed to the `executeDistributedTarget(...)` method. This is done for the same reason as parameters: we need a concrete type in order to efficiently pass the values to the actor system, so it can encode them without going through existential wrappers. As we cannot implement the `invoke()` method to be codable over the expected types — we don't know them until we've looked up the actual method we were about to invoke (and apply generic substitutions to them). + +The implementation could look as follows: + +```swift +extension ExampleDistributedMethodInvocationHandler { + func onReturn(result: Success) throws { + do { + let replyData = system.encoder.encode(result) + self.reply(replyData) + } catch { + self.replyError("Failed to encode reply: \(type(of: error))") + } + } + + func onError(error: Failure) { + guard Failure is Encodable else { + // best effort error reporting just sends back the type string + // we don't want to send back string representation since it could leak sensitive information + self.replyError("\(Failure.self)") + } + + // ... if possible, `as?` cast to Encodable and return an actual error, + // but only if it is allow-listed, as we don't want to send arbitrary errors back. + } +} +``` + +We omit the implementations of `replyError` and `reply` because they are more of the same patterns that we have already discussed here, and this is a proposal focused on illustrating the language feature, not a complete system implementation after all. + +The general pattern here is the same as with decoding parameters, however in the opposite direction. + +Once the `onError` or `onReturn` methods complete, the `executeDistributedTarget` method returns, and its caller knows the distributed request/response has completed – at least, as far as this peer is concerned. We omit the implementation of the `reply` and `replyError` methods that the actor system would implement here, because they are pretty much the same process as sending the request, except that the message must be sent as a response to a specific request, rather than target a specific actor and method. How this is achieved can differ wildly between transport implementations: some have built-in request/reply mechanisms, while others are uni-directional and rely on tagging replies with identifiers such as "this is a reply for request 123456". + +## Amendments + +### Initializers no longer need to accept a single DistributedActorSystem + +During this review, feedback was received and addressed with regards to the previous implementation limitation that user defined designated actor initializers must have always accepted a single `DistributedActorSystem` conforming parameter. This was defined in [SE-0336: Distributed Actor Isolation](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0336-distributed-actor-isolation.md). + +The limitation used to be that one had to accept such parameter or the system would fail to synthesize the initializer an offer a compile-time error: + +```swift +// PREVIOUSLY, as defined by SE-0336 +init(x: String) {} +// ❌ error: designated distributed actor initializer 'init(x:)' is missing required 'DistributedActorSystem' parameter + +init(system: AnyDistributedActorSystem) {} +// ✅ used to be ok; though the "unused" parameter looks confusing and is actually necessary +``` + +This system argument, picked by type, would have been picked up by the compiler and used in synthesis of necessary calls during a distributed actor's initialization. This led to a weird "dangling" parameter that is not used by visible code, but is crucial for the correctness of the program. + +This implementation restriction is now lifted, and is rephrased as requiring an assignment to the `self.actorSystem` property in non-delegating initializers. This is in line with how normal Swift works, and can be explained to users more intuitively, because every `distributed actor` has a synthesized `let actorSystem: ActorSystem` property, it clearly must be initialized to something as implied by normal initializer rules. + +The checks therefore are now plain property assignment checks, and the compiler is able to pick up the assigned property and synthesize the necessary `assignID` and `actorReady` calls based on the property, e.g.: + +```swift +distributed actor Example { + init(x: String) { + // ❌ error: property 'actorSystem' was not initialized + // more educational notes to be produced here (e.g. what type it expects etc) + } + + init (x: String, cluster: ClusterSystem) { + self.actorSystem = cluster // ✅ ok + } +} +``` + +This means that it is also possible to assign a "global" or static actor system, however it is **strongly discouraged** to do so because it hurts reusability and testability, e.g. by swapping a test instance of an actor system during unit tests. + +```swift +// Possible, but considered an anti-pattern as it hurts testability of such actor +// DON'T DO THIS. +distributed actor NotGoodIdea { + init() { + self.actorSystem = SomeSystem.global // anti-pattern, always accept an actor system via initializer. + } +} +``` + +## Future work + +### Variadic generics removing the need for `remoteCallVoid` + +Once [variadic generics](https://forums.swift.org/t/variadic-generics/54511/2) are fully implemented, we will be able to remove the limitation that we cannot express the `remoteCall<..., Success: SerializationRequirement>(..., returning returnType: Success.Type)` function for the `Void` type, since it cannot always conform to `SerializationRequirement`. + +With variadic generics, it would be natural to conform an "empty tuple" to the `SerializationRequirement` and we'd this way be able to implement only a single method (`remoteCall`) rather than having to provide an additional special case implementation for `Void` return types. + +### Identifying, evolving and versioning remote calls + +At present remote calls use an opaque string identifier (wrapped as `RemoteCallTarget`) to refer to, and invoke remote call targets. This identifier is populated by default using the mangled name of the distributed thunk of the target function. + +#### Default distributed call target identification scheme + +Distributed actor system implementations shall treat the identifier as opaque value that they shall ship from the sender to the recipient node, without worrying too much about their contents. The identification scheme though has a large and important impact on the versioning story of distributed calls, so in this section we'd like to discuss and outline the general plans we foresee here as we will introduce proper and more versioning friendly schemes in the future. + +Swift's default mangling scheme is used for distributed methods is problematic for API evolution, however, it is a great "default" to pick until we devise a different scheme in the future. The mangling scheme is a good default because: + +- it allows callers to use all of the richness of Swift's calling semantics, including: + - overloads by type (e.g. we can call `receive(_:Int)` as well as `receive(_:String)`) and invoke the correct target based on the type. +- it allows us to evolve APIs manually by adding overloads, deprecate methods etc, and we will continue to work towards a feature rich versioning story while adopting limited use-cases where these limitations are not a problem. + +This is also avoids a well-known problem from objective-c, where selectors must not accidentally be the same, otherwise bad-things-happen™. + +One potential alternative would be to use the full names of methods, i.e. `hello(name:surname:)`, similar to objective-c selectors to identify methods. However, this means the loss of any and all type-safety and _intent_ of target methods being used according to their appropriate types. It also means identifiers must be unique and overloads must be banned at compile time, or we risk reusing identifiers and not knowing which function to invoke. This again could be solved by separately shipping type identifiers, but this would mean reinventing what Swift's mangling scheme already does, but in a worse way. Instead, we propose to start with the simple mangling scheme, and in a subsequent proposal, address the versioning story more holistically, in a way that addresses ABI concerns of normal libraries, as well as distributed calls. + +We are aware that the mangling scheme makes the following, what should be possible to evolve without breaking APIs situations not work, that a different scheme could handle well: + +- adding parameters even with default parameters, +- changing a type of argument from struct to class (or enum etc), is also breaking though it need not be. + - We could address this by omitting the kind information from the mangled names. + +The robust versioning and evolution scheme we have in mind for the future must be able to handle these cases, and we will be able to roll out a new identification scheme that handles those in the future, without breaking API or breaking existing remote calls. The metadata to lookup distributed method accessors would still be available using the "most precise" mangled format, even as we introduce a lossy, more versioning friendly format. APIs would remain unchanged, because from the perspective of `DistributedActorSystem` all it does is ship around an opaque String `identifier` of a remote call target. + +There are numerous other versioning and rollout topics to cover, such as "rolling deployments", "green/blue deployments" and other techniques that both the versioning scheme _and_ the specific actor system implementation must be able to handle eventually. However, based on experience implementing other actor systems in the past, we are confident that those are typical to add in later phases of such an endeavor, not as the first thing in the first iteration of the project. + +#### Compression techniques to avoid repeatedly sending large identifiers + +One of the worries brought up with regards to mangled names and string identifiers in general during the review has been that they can impose a large overhead when the actual messages are small. Specifically, as the `RemoteCallTarget.identifier` often can be quite large, and for distributed methods which e.g. accept only a few integers, or even no values at all, the identifiers often can dominate the entire message payload. + +This is not a novel problem – we have seen and solved such issues in the past in other actor runtimes (i.e. Akka). One potential solution is to establish a compression mechanism between peers where message exchanges establish shared knowledge about "long identifiers" and mapping them to unique numbers. The sender in such a system at first pessimistically sends the "long" String-based identifier, and in return may get additional metadata in the response "the next time you want to call this target, send the ID 12345". The sender system then stores in a cache that when invoking the "X...Z" target it does not need to send the long string identifier, but instead can send the number 12345. This solves the long string identifiers problem, as they need not be sent repeatedly over the wire. (There are a lot of details to how such schemes can be implemented that we do not need to dive into here, however we are confident those work well, because we have seen them solve this exact issue before). + +Given the need, and assuming other shared knowledge, we could even implement other identification schemes, which can also avoid that "initial" long string identifier sending. We will be exploring those as specific use-cases and requirements from adopters as they arise. We are confident in our ability to fit such techniques into this design because of the flexibility that APIs based around `RemoteCallTarget` gives us, without requiring us to actually send the entire target object over the wire. + +Such optimization techniques, are entirely implementable in concrete distributed actor system implementations. However, if necessary, we would be able to even extend the capabilities of `executeDistributedTarget` to accommodate other concrete needs and designs. + +#### Overlap with general ABI and versioning needs in normal Swift code + +The general concept binary/wire-compatible evolution of APIs by _adding_ new parameters to methods is not unique to distribution. In fact, this is a feature that would benefit ABI-stable libraries like the ones shipping with the SDK. It is often the case that new APIs are introduced that accept _more_ arguments, perhaps to enable or extend functionality of an existing feature. Developers today have to add new functions with "one more argument", like this: + +```swift +public func f() { + f(x: 0) +} + +@available(macOS 9999) +public func f(x: Int) { + // new implementation +} +``` + +Where the new function is going to ship with a new OS, yet actually contains an implementation that is compatible with the old definition of `f()`. In such situations, developers are forced to manually write forwarding methods. While this seems simple on small APIs, it can easily become rather complex and easy to introduce subtle bugs in the forwarding logic. + +Instead, developers would want to be able to introduce new parameters, with a default value that would be used when the function is invoked on older platforms, like this: + +```swift +public func f(@available(macOS 9999) x: Int = 0) { + // new implementation +} + +// compiler synthesizes: +// +// public func f() { self.f(x: 0) } +// +// @available(macOS 9999) +// public func f( x: Int = 0) { +``` + +Where the compiler would synthesize versions of the methods for the various availabilities, and delegate, in an ABI-compatible way to the new implementation. This is very similar to what would be necessary to help wire-compatible evolution of distributed methods. So the same mechanisms could be used for distributed calls, if instead of OS versions, we could also allow application/node versions in the availability perhaps? This is not completely designed, but definitely a direction we are interested in exploring. + +#### Discussion: User provided target identities + +We also are considering, though are not yet convinced that these would be the optimal way to address some of the evolution concerns, the possibility to use "stable names". This mechanism would allow users to give full control over target identity to developers, where the `RemoteCallTarget.identifier` offered to the `remoteCall` implemented by a distributed actor system library, could be controlled by end users of the library, i.e. those who define distributed methods. + +On one hand, this gives great power to end-users, as they may use specific and minimal identifiers, even using `"A"` and other short names to minimize the impact of the identifiers to the payload size. On the other hand though, it opens up developers to a lot of risks, including conflicts in identifier use – a problem all to well known to objective-c developers where selectors could end up in similar problematic situations. + +We want to take time to design a proper system rather than just open up full control over the remote call target `identifier` to developers. We will aim to provide a flexible, and powerful mechanism, that does not risk giving developers easy ways to get shoot themselves in the foot. The design proposed here, is flexible enough to evolve. + +### Resolving `DistributedActor` protocols + +We want to be able to publish only protocols that contain distributed methods, and allow clients to resolve remote actors based on protocols alone, without having any knowledge about the specific `distributed actor` type implementing the protocol. This allows binary, closed-source frameworks to offer distributed actors as way of communication. Of course, for this to be viable we also need to solve the above ABI and wire-compatible evolution of distributed methods, assuming we solve those though, publishing distributed actor protocols is very useful and interesting for client/server scenarios, where the peers of a communication are not exact mirrors of the same process, but exhibit some asymmetry. + +Currently, we resolve distributed actors using the static method defined on the `DistributedActor` protocol, sadly this method is not possible to invoke on just a protocol: + +```swift +protocol Greeter: DistributedActor { + func greet() -> String +} + +let greeter: any Greeter = try Greeter.resolve(id: ..., using: websocketActorSystem) +// ❌ error: static member 'resolve' cannot be used on protocol metatype 'Greeter.Protocol' +``` + +A "client" peer does not have to know what distributed actor exactly implements this protocol, just that we're able to send a "greet" message to it, we should be able to obtain an existential `any Greeter` and be able to invoke `greet()` on it. + +In order to facilitate this capability, we need to: + +- implement ad-hoc synthesis of a type that effectively works like a "stub" that other RPC systems generally source-generate, yet thanks to our actor model we're able to synthesize it in the compiler on demand; +- find a way to invoke `resolve` on such protocol, for example we could offer a global function `resolveDistributedActorProtocol(Greeter.self, using: websocketActorSystem)` + +The `resolveDistributedActorProtocol` method has to be able to check the serialization requirement at compile-time where we invoke the resolve, because the distributed actor protocols don't have to declare a serialization requirement — they can, but they don't have to (and this is by design). + +It should be possible to resolve the following examples: + +```swift +protocol Greeter: DistributedActor { + distribute func greet() -> String +} + +final class WebsocketActorSystem: DistributedActorSystem { + typealias ActorID: WebsocketID // : Codable + typealias SerializationRequirement: Codable +} + +... = try resolveDistributedActorProtocol(id: ..., as: Greeter.self, using: websocketActorSystem) +``` + +The resolve call would use the types defined for the `ActorID` and `SerializationRequirement` to see if this `Greeter` protocol is even implementable using these, i.e. if its distributed method parameters/return types do indeed conform to `Codable`, and if there isn't a conflict with regards to the actor ID. + +This means that we should reject at compile-time any attempts to resolve a protocol that clearly cannot be implemented over the actor system in question, for example: + +```swift +protocol Greeter: DistributedActor { + distributed func greet() -> NotCodableResponse +} + +final class WebsocketActorSystem: DistributedActorSystem { + typealias ActorID: WebsocketID // : Codable + typealias SerializationRequirement: Codable +} + +... = try resolveDistributedActorProtocol(id: ..., as: Greeter.self, using: websocketActorSystem) +// ❌ error: 'Greeter' cannot be resolved using 'WebsocketActorSystem' +// ❌ error: result type 'NotCodableResponse' of distributed instance method does not conform to 'Codable' +``` + +These are the same checks that are performed on `distributed actor` declarations, but they are performed on the type. We can think of these checks running whenever distributed methods are "combined" with a specific actor system: this is the case in `distributed actor` declarations, as well as this protocol resolution time, because we're effectively creating a not-user-visible actor declaration that combines the given `DistributedActorSystem` with the synthesized "stub" distributed actor, so we need to run the checks here. Thankfully, we can run them at compile time, disallowing any ill-formed and impossible to implement combinations. + +### Passing parameters to `assignID` + +Sometimes, transports may need to get a little of configuration for a specific actor being initialized. + +Since `DistributedActorSystem.assignID` accepts the actor *type* it can easily access any configuration that is static for some specific actor type, e.g. like this: + +```swift +protocol ConfiguredDistributedActor: DistributedActor { + static var globalServiceName: String { get } +} + +distributed actor Cook: DistributedActorConfiguration { + static var globalServiceName: String { "com.apple.example.CookingService" } +} +``` + +This way the `assignID` can detect the static property and e.g. ensure this actor is possible to look up by this static name: + +```swift +extension SpecificDistributedActorSystem { + func assignID(_ type: Actor.Type) -> Actor.ID where Actor: DistributedActor { + let id = <> + if let C = type as ConfiguredDistributedActor.Type { + // for example, we could make sure the actor is discoverable using the service name: + let globalServiceName = C.globalServiceName + self.ensureAccessibleAs(id: id, as: globalServiceName) + } + + return id + } +} +``` + +Or similar configuration patterns. However, it is hard to implement a per instance configuration to be passed to the system. + +One way we could solve this is by introducing an `assignID` overload that accepts the `ActorConfiguration` that may be +passed to the actor initializer, and would be passed along to the actor system like this: + +```swift +extension SpecificDistributedActorSystem { + // associatedtype ActorConfiguration + func assignID(_ type: Actor.Type, _ properties: Self.ActorConfiguration) -> Actor.ID where Actor: DistributedActor { + if let name = properties.name { + return makeID(withName: name) + } + + return makeRandomID() + } +} +``` + +The creation of the actor would then be able to be passed at-most one `ActorConfiguration` instance, and that would be then passed down to this method: + +```swift +distributed actor Worker {...} + +Worker(actorSystem: system, actorProperties: .name("worker-1234")) +``` + +Which can be *very* helpful since now IDs can have user provided information that are meaningful in the user's domain. + +## Alternatives considered + +This section summarizes various points in the design space for this proposal that have been considered, but ultimately rejected from this proposal. + +### Define `remoteCall` as protocol requirement, and accept `[Any]` arguments + +The proposal includes the fairly special `remoteCall` method that is expected to be present on a distributed actor system, however is not part of the protocol requirements because it cannot be nicely expressed in today's Swift, and it suffers from the lack of variadic generics (which are being worked on, see: [Pitching The Start of Variadic Generics](https://forums.swift.org/t/pitching-the-start-of-variadic-generics/51467)), however until they are complete, expressing `remoteCall` in the type-system is fairly painful, and we resort to providing multiple overloads of the method: + +```swift + func remoteCall( + on recipient: Actor, + method: DistributedMethodName, + _ arg1: P1, + throwing errorType: Failure.Type, + returning returnType: Success.Type + ) async throws -> Success where Actor: DistributedActor, Actor.ID = ActorID { ... } + + func remoteCall( + on recipient: Actor, + method: DistributedMethodName, + _ arg1: P1, _ arg2: P2, + throwing errorType: Failure.Type, + returning returnType: Success.Type + ) async throws -> Success where Actor: DistributedActor, Actor.ID = ActorID { ... } + + // ... +``` + +This is annoying for the few distributed actor system developers. However, it allows us to completely avoid any existential boxing that shuttling values through `Any` would imply. We are deeply interested in offering this system to systems that are very concerned about allocations, and runtime overheads, and believe this is the right tradeoff to make, while we await the arrival of variadic generics which will solve this system implementation annoyance. + +We are also able to avoid any heap allocations during the `remoteCall` thanks to this approach, as we do not have to construct type erased `arguments: [Any]` which would have been the alternative: + +```swift + func remoteCall( + on recipient: Actor, + method: DistributedMethodIdentifier, + _ args: [Any], // BAD + throwing errorType: Failure.Type, + returning returnType: Success.Type + ) async throws -> Success where Actor: DistributedActor, Actor.ID = ActorID { ... } +``` + +Not only that, but passing arguments as `[Any]` would force developers into using internal machinery to open the existentials (the not officially supported `_openExistential` feature), in order to obtain their specific types, and e.g. use `Codable` with them. + +### Constraining arguments, and return type with of `remoteCall` with `SerializationRequirement` + +Looking at the signature, one might be tempted to also include a `where` clause to statically enforce that all parameters and return type, conform to the `Self.SerializationRequirement`, like so: + +```swift + func remoteCall( + on recipient: Actor, + method: DistributedMethodName, + _ arg1: P1, + throwing errorType: Failure.Type, + returning returnType: Success.Type + ) async throws -> Success where Actor: DistributedActor, + Actor.ID = ActorID, + P1: SerializationRequirement { ... } +// ❌ error: type 'P1' constrained to non-protocol, non-class type 'Self.R' +``` + +However, this is not expressible today in Swift, because we cannot prove the `associatedtype SerializationRequirement` can be used as constraint. + +Fixing this would require introducing new very advanced type system features, and after consultation with the core team we decided to accept this as a current implementation limitation. + +In practice this is not a problem, because the parameters are guaranteed to succeed being cast to `SerializationRequirement` at runtime thanks to the compile-time guarantee about parameters of distributed methods. + +### Hardcoding the distributed runtime to make use of `Codable` + +`Codable` is a great, useful, and relatively flexible protocol allowing for serialization of Swift native types. However, it may not always be the best serialization system available. For example, we currently do not have a great binary serialization format that works with `Codable`, or perhaps some developers just really want to use a 3rd party serialization format such as protocol buffers, SBE or something entirely custom. + +The additional complexity of the configurable `SerializationRequirement` is pulling its weight, and we are not interested in closing down the system to just use Codable. + +## Acknowledgments & Prior art + +We would like to acknowledge the prior art in the space of distributed actor systems which have inspired our design and thinking over the years. Most notably we would like to thank the Akka and Orleans projects, each showing independent innovation in their respective ecosystems and implementation approaches. As these are library-only solutions, they have to rely on wrapper types to perform the hiding of information, and/or source generation; we achieve the same goal by expanding the actor-isolation checking mechanisms already present in Swift. + +We would also like to acknowledge the Erlang BEAM runtime and Elixir language for a more modern take built upon the on the same foundations, which have greatly inspired our design, however take a very different approach to actor isolation (i.e. complete isolation, including separate heaps for actors). + +## Source compatibility + +This change is purely additive to the source language. + +The language impact has been mostly described in the Distributed Actor Isolation proposal + +## Effect on ABI stability + +TODO + +## Effect on API resilience + +None. + +## Changelog + +- 2.0 Adjustments after first round of review + - Expanded discussion on current semantics, and **future directions for versioning** and wire compatibility of remote calls. + - Amendment to **non-delegating distributed actor initializer semantics** + - Non-delegating Initializers no longer require a single `DistributedActorSystem` conforming argument to be passed, and automatically store and initialize the `self.id` with it. + - Instead, users must initialize the `self.actorSystem` property themselves. + - This should generally be done using the same pattern, by passing _in_ an actor system from the outside which helps in testing and stubbing out systems, however if one wanted to. + - However the actorSystem property is initialized, from an initializer parameter or some global value, task-local etc, the general initialization logic remains the same, and the compiler will inject the assignment of the `self.id` property. + - Useful error messages explaining this decision are reported if users attempt to assign to the `id` property directly. + + - Thanks to YR Chen for recognizing this limitation and helping arrive at a better design here. + + - **Removal of explicit use of mangled names** in the APIs, and instead all APIs use an opaque `RemoteCallTarget` that carries an opaque String `identifier` + - The `RemoteCallTarget` is populated by the compiler using mangled names by default, but other schemes can be used in the future + - The `RemoteCallTarget` now pretty prints as "`hello(name:surname:)`" if able to decode the target from the identifier, otherwise it prints the identifier directly. This can be used to include pretty names of call targets in tracing systems etc. + - This design future-proofs the APIs towards potential new encoding schemed we might come up in the future as we tackle a proper and feature complete versioning story for distributed calls. + + - `recordArgument` is passed a **`RemoteCallArgument`** which carries additional information about the argument in question + - This parameter can be either ignored, or stored along the serialized format which may be useful for non swift targets of invocations. E.g. it is possible to store an invocation as JSON object where the argument names are used as labels or similar patterns. + - Thanks to Slava Pestov for suggesting this improvement. + +- 1.3 Larger revision to match the latest runtime developments + - recording arguments does not need to write into provided pointers; thanks to the calls being made in IRGen, we're able to handle things properly even without the heterogenous buffer approach. Thank you, Pavel Yaskevich + - simplify rules of readying actors across synchronous and asynchronous initializers, we can always ready "just before `self` is escaped", in either situation; This is thanks to the latest developments in actor initializer semantics. Thank you, Kavon Farvardin + - express recording arguments and remote calls as "ad-hoc" requirements which are invoked directly by the compiler + - various small cleanups to reflect the latest implementation state +- 1.2 Drop implicitly distributed methods +- 1.1 Implicitly distributed methods +- 1.0 Initial revision +- [Pitch: Distributed Actors](https://forums.swift.org/t/pitch-distributed-actors/51669) + - Which focused on the general concept of distributed actors, and will from here on be cut up in smaller, reviewable pieces that will become their own independent proposals. Similar to how Swift Concurrency is a single coherent feature, however was introduced throughout many interconnected Swift Evolution proposals. + +[isolation]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0336-distributed-actor-isolation.md diff --git a/proposals/0345-if-let-shorthand.md b/proposals/0345-if-let-shorthand.md new file mode 100644 index 0000000000..265d89d873 --- /dev/null +++ b/proposals/0345-if-let-shorthand.md @@ -0,0 +1,382 @@ +# `if let` shorthand for shadowing an existing optional variable + +* Proposal: [SE-0345](0345-if-let-shorthand.md) +* Author: [Cal Stephens](https://github.com/calda) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0345-if-let-shorthand-for-shadowing-an-existing-optional-variable/56364) +* Implementation: [apple/swift#40694](https://github.com/apple/swift/pull/40694) + +## Introduction + +Optional binding using `if let foo = foo { ... }`, to create an unwrapped variable that shadows an existing optional variable, is an extremely common pattern. This pattern requires the author to repeat the referenced identifier twice, which can cause these optional binding conditions to be verbose, especially when using lengthy variable names. We should introduce a shorthand syntax for optional binding when shadowing an existing variable: + +```swift +let foo: Foo? = ... + +if let foo { + // `foo` is of type `Foo` +} +``` + +Swift-evolution thread: [`if let` shorthand](https://forums.swift.org/t/if-let-shorthand/54230) + +## Motivation + +Reducing duplication, especially of lengthy variable names, makes code both easier to write _and_ easier to read. + +For example, this statement that unwraps `someLengthyVariableName` and `anotherImportantVariable` is rather arduous to read (and was without a doubt arduous to write): + +```swift +let someLengthyVariableName: Foo? = ... +let anotherImportantVariable: Bar? = ... + +if let someLengthyVariableName = someLengthyVariableName, let anotherImportantVariable = anotherImportantVariable { + ... +} +``` + +One approach for dealing with this is to use shorter, less descriptive, names for the unwrapped variables: + +```swift +if let a = someLengthyVariableName, let b = anotherImportantVariable { + ... +} +``` + +This approach, however, reduces clarity at the point of use for the unwrapped variables. Instead of encouraging short variable names, we should allow for the ergonomic use of descriptive variable names. + +## Proposed solution + +If we instead omit the right-hand expression, and allow the compiler to automatically shadow the existing variable with that name, these optional bindings are much less verbose, and noticeably easier to read / write: + +```swift +let someLengthyVariableName: Foo? = ... +let anotherImportantVariable: Bar? = ... + +if let someLengthyVariableName, let anotherImportantVariable { + ... +} +``` + +This is a fairly natural extension to the existing syntax for optional binding conditions. + +## Detailed design + +Specifically, this proposal extends the Swift grammar for [`optional-binding-condition`](https://docs.swift.org/swift-book/ReferenceManual/Statements.html#grammar_optional-binding-condition)s. + +This is currently defined as: + +> optional-binding-condition → **let** [pattern](https://docs.swift.org/swift-book/ReferenceManual/Patterns.html#grammar_pattern) [initializer](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_initializer) | **var** [pattern](https://docs.swift.org/swift-book/ReferenceManual/Patterns.html#grammar_pattern) [initializer](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_initializer) + +and would be updated to: + +> optional-binding-condition → **let** [pattern](https://docs.swift.org/swift-book/ReferenceManual/Patterns.html#grammar_pattern) [initializer](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_initializer)opt | **var** [pattern](https://docs.swift.org/swift-book/ReferenceManual/Patterns.html#grammar_pattern) [initializer](https://docs.swift.org/swift-book/ReferenceManual/Declarations.html#grammar_initializer)opt + +This would apply to all conditional control flow statements: + +```swift +if let foo { ... } +if var foo { ... } + +else if let foo { ... } +else if var foo { ... } + +guard let foo else { ... } +guard var foo else { ... } + +while let foo { ... } +while var foo { ... } +``` + +The compiler would synthesize an initializer expression that references the variable being shadowed. + +For example: + +```swift +if let foo { ... } +``` + +is transformed into: + +```swift +if let foo = foo { ... } +``` + +Explicit type annotations are permitted, like with standard optional binding conditions. + +For example: + +```swift +if let foo: Foo { ... } +``` + +is transformed into: + +```swift +if let foo: Foo = foo { ... } +``` + +The pattern following the introducer serves as both an evaluated expression _and_ an identifier for the newly-defined non-optional variable. Existing precedent for this type of syntax includes closure capture lists, which work the same way: + +```swift +let foo: Foo +let closure = { [foo] in // `foo` is both an expression and the identifier + ... // for a new variable defined within the closure +} +``` + +Because of this, only valid identifiers would be permitted with this syntax. For example, this example would not be valid: + +```swift +if let foo.bar { ... } // 🛑 unwrap condition requires a valid identifier + ^ // fix-it: insert `<#identifier#> = ` +``` + +### Interaction with implicit self + +Like with existing optional bindings, this new syntax would support implifict self references to unwrap optional members of `self`. For example, the usage in this example would be permitted: + +```swift +struct UserView: View { + let name: String + let emailAddress: String? + + var body: some View { + VStack { + Text(user.name) + + // Equivalent to `if let emailAddress = emailAddress { ... }`, + // unwraps `self.emailAddress`. + if let emailAddress { + Text(emailAddress) + } + } + } +} +``` + +## Source compatibility + +This change is purely additive and does not break source compatibility of any valid existing Swift code. + +## Effect on ABI stability + +This change is purely additive, and is a syntactic transformation to existing valid code, so has no effect on ABI stability. + +## Effect on API resilience + +This change is purely additive, and is a syntactic transformation to existing valid code, so has no effect on ABI stability. + +## Future directions + +### Optional casting + +A natural extension of this new syntax could be to support shorthand for optional casting. For example: + +`if let foo as? Bar { ... }` + +could be equivalent to: + +`if let foo = foo as? Bar { ... }` + +This is not included in this proposal, but is a reasonable feature that could be added in the future. + +### Interaction with future borrow introducers + +["A roadmap for improving Swift performance predictability"](https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206#borrow-variables-7) discusses potential new `ref` and `inout` introducers for creating variables that "borrow" existing variables without making a copy (by enforcing exclusive access). For consistency with `let` / `var`, it will likely make sense to support optional binding conditions for these new introducers: + +```swift +if ref foo = foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, immutable variable +} + +if inout foo = &foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, mutable variable +} +``` + +The shorthand syntax for `let` / `var` optional bindings would extend fairly naturally to these new introducers: + +```swift +if ref foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, immutable variable +} + +if inout &foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, mutable variable +} +``` + +### Unwrapping nested members of objects + +This proposal doesn't permit shorthand unwrapping for members nested in other objects. For example: + +`if let foo.bar { ... } // 🛑` + +There are a few different options that could allow us to support this type of syntax in the future. + +One approach could be to automatically synthesize the identifier name for the unwrapped variable in the inner scope. For example. `if let foo.bar` could introduce a new non-optional variable named `bar` or `fooBar`. + +Another approach could be to permit this for potential future borrow introducers `ref` and `inout` (from ["A roadmap for improving Swift performance predictability"](https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206#borrow-variables-7)). These borrows would have compiler-enforced exclusive access to the underlying storage, so they technically do not require a unique identifier name for the inner scope. This could allow us to unwrap members of objects without any new variables or copies. For example: + +```swift +// `mother.father.sister` is optional + +if ref mother.father.sister { + // `mother.father.sister` is non-optional and immutable +} + +if inout &mother.father.sister { + // `mother.father.sister` is non-optional and mutable +} +``` + +## Alternatives considered + +There have been many other proposed spellings for this feature: + +### `if foo` + +The briefest possible spelling for this feature would just be a bare `if foo` condition. This spelling, however, would create ambiguity between optional unwrapping conditions and boolean conditions, and could lead to confusing / conter-intuitive situations: + +```swift +let foo: Bool = true +let bar: Bool? = false + +if foo, bar { + // would succeed +} + +if foo == true, bar == true { + // would fail +} +``` + +To avoid this ambiguity, we need some sort of distinct syntax for optional bindings. + +### `if unwrap foo` + +Another option is to introduce a new keyword or sigil for this purpose, like `if unwrap foo`, `if foo?` or `if have foo`. + +A key benefit of introducing a completely new syntax like `if unwrap foo` is that it gives us the opportunity to also revisit the _semantics_ of how optional binding conditions actually work. Today, optional binding conditions always make a copy of the value. From a performance perspective, it would be more efficient to perform a _borrow_ instead of a copy. + +["A roadmap for improving Swift performance predictability"](https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206#borrow-variables-7) discusses potential future introducers `ref` (to perform an immutable borrow) and `inout` (to perform a mutable borrow). For consistency with `let` / `var`, it will likely make sense to support optional binding conditions for these new introducers: + +```swift +if ref foo = foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, immutable variable +} + +if inout foo = &foo { + // if `foo` is not nil, it is borrowed and made available as a non-optional, mutable variable +} +``` + +Instead of being shorthand for `if let`, this new shorthand syntax could instead be shorthand for `if ref`. This would improve performance in general, and could nudge users towards using borrows instead of copies (since only the borrow form would receive shorthand sugar). + +A key downside of borrows, however, is that they require exclusive access to the borrowed variable. Memory exclusivity violations will result in compiler errors in some cases, but can also manifest as runtime errors in more complex cases. For example: + +```swift +var x: Int? = 1 + +func increment(by number: Int) { + x? += number +} + +if ref x = x { + increment(by: x) +} +``` + +This would trap at runtime, because `increment(by:)` would attempt to modify the value of `x` while it is already being borrowed by the `if ref x = x` optional binding condition. + +Once borrow introducers are added to the language, seeing `ref x` or `inout x` anywhere in Swift will serve as an important visual marker about the exclusivity requirements of the code. On the other hand, a new syntax like `if unwrap x` doesn't explicitly indicate that the variable is being borrowed. This could lead to users being surprised by unexpected exclusivity violations, which could cause confusing compile-time errors or runtime crashes. + +Borrow introducers will be very useful, but adopting them is a tradeoff between performance and conceptual overhead. Borrows are cheap but come with high conceptual overhead. Copies can be expensive but always work as expected without much extra thought. Given this tradeoff, it likely makes sense for this shorthand syntax to provide a way for users to choose between performing a copy or performing a borrow, rather than limiting users to one or the other. + +Additionally, for consistency with existing optional binding conditions, this new shorthand should support the distinction between immutable and mutable variables. Combined with the distinction between copies and borrows, that would give us the same set of options as normal variables: + +```swift +// Included in this proposal: +if let foo { /* foo is an immutable copy */ } +if var foo { /* foo is a mutable copy */ } + +// Potentially added in the future: +if ref foo { /* foo is an immutable borrow */ } +if inout &foo { /* foo is a mutable borrow */ } +``` + +Since we already have syntax for these concepts, we should reuse that syntax in this shorthand rather than create a new syntax that is less expressive (e.g. only supports a subset of the available options) and less explicit (e.g. that users would have to memorize whether this new shorthand performs a copy or a borrow). + +### `if let foo?` + +Another option is to include a `?` to explicitly indicate that this is unwrapping an optional, using `if let foo?`. This is indicative of the existing `case let foo?` pattern matching syntax. + +`if let foo = foo` (the most common existing syntax for this) unwraps optionals without an explicit `?`. This implies that a conditional optional binding is sufficiently clear without a `?` to indicate the presence of an optional. If this is the case, then an additional `?` is likely not strictly necessary in the shorthand `if let foo` case. + +While the symmetry of `if let foo?` with `case let foo?` is nice, consistency with `if let foo = foo` is even more important condiering they will more frequently appear within the same statement: + +```swift +// Consistent +if let user, let defaultAddress = user.shippingAddresses.first { ... } + +// Inconsistent +if let user?, let defaultAddress = user.shippingAddresses.first { ... } +``` + +Additionally, the `?` symbol makes it trickier to support explicit type annotations like in `if let foo: Foo = foo`. `if let foo: Foo` is a natural consequence of the existing grammar. It's less clear how this would work with an additional `?`. `if let foo?: Foo` likely makes the most sense, but doesn't match any existing language constructs. + +### `if foo != nil` + +One somewhat common proposal is to permit `nil`-checks (like `if foo != nil`) to unwrap the variable in the inner scope. Kotlin supports this type of syntax: + +```kt +var foo: String? = "foo" +print(foo?.length) // "3" + +if (foo != null) { + // `foo` is non-optional + print(foo.length) // "3" +} +``` + +This pattern in Kotlin _does not_ define a new variable -- it merely changes the type of the existing variable within the inner scope. So mutations that affect the inner scope also affect the outer scope: + +```kt +var foo: String? = "foo" + +if (foo != null) { + print(foo) // "foo" + foo = "bar" + print(foo) // "bar" +} + +print(foo) // "bar" +``` + +This is different from Swift's optional binding conditions (`if let foo = foo`), which define a new, _separate_ variable in the inner scope. This is a defining characteristic of optional binding conditions in Swift, so any shorthand syntax must make it abundantly clear that a new variable is being declared. + +### Don't permit `if var foo` + +Since `if var foo = foo` is significantly less common than `if let foo = foo`, we could potentially choose to _not_ support `var` in this shorthand syntax. + +`var` shadowing has the potential to be more confusing than `let` shadowing -- `var` introduces a new _mutable_ variable, and any mutations to the new variable are not shared with the original optional variable. On the other hand, `if var foo = foo` already exists, and it seems unlikely that `if var foo` would be more confusing / less clear than the existing syntax. + +Since `let` and `var` are interchangeable elsewhere in the language, that should also be the case here -- disallowing `if var foo` would be inconsistent with existing optional binding condition syntax. If we were using an alternative spelling that _did not_ use `let`, it may be reasonable to exclude `var` -- but since we are using `let` here, `var` should also be allowed. + +## Acknowledgments + +Many thanks to Craig Hockenberry, who recently wrote about this topic in [Let’s fix `if let` syntax](https://forums.swift.org/t/lets-fix-if-let-syntax/48188) which directly informed this proposal. + +Thanks to Ben Cohen for suggesting the alternative `if let foo?` spelling, and for providing valuable feedback on this proposal during the pitch phase. + +Thanks to Chris Lattner for suggesting to consider how this proposal should interact with upcoming language features like potential `ref` and `inout` borrow introducers. + +Thanks to [tera](https://forums.swift.org/u/tera/summary) for suggesting the alternative `if foo` spelling. + +Thanks to Jon Shier for providing the SwiftUI optional binding example. + +Thanks to James Dempsey for providing the "consistency with existing optional binding conditions" example. + +Thanks to Frederick Kellison-Linn for pointing out that variables in closure capture lists are an existing precedent for this type of syntax. diff --git a/proposals/0346-light-weight-same-type-syntax.md b/proposals/0346-light-weight-same-type-syntax.md new file mode 100644 index 0000000000..7199446d35 --- /dev/null +++ b/proposals/0346-light-weight-same-type-syntax.md @@ -0,0 +1,496 @@ +# Lightweight same-type requirements for primary associated types + +* Proposal: [SE-0346](0346-light-weight-same-type-syntax.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin), [Holly Borla](https://github.com/hborla), [Slava Pestov](https://github.com/slavapestov) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Previous Revisions: [1st](https://github.com/swiftlang/swift-evolution/blob/5d86d57cfd6d803df4da90b196682d495e5de9b9/proposals/0346-light-weight-same-type-syntax.md) +* Review: ([first pitch](https://forums.swift.org/t/pitch-light-weight-same-type-constraint-syntax/52889)) ([second pitch](https://forums.swift.org/t/pitch-2-light-weight-same-type-requirement-syntax/55081)) ([first review](https://forums.swift.org/t/se-0346-lightweight-same-type-requirements-for-primary-associated-types/55869)) ([second review](https://forums.swift.org/t/se-0346-second-review-lightweight-same-type-requirements-for-primary-associated-types/56414)) ([acceptance](https://forums.swift.org/t/accepted-se-0346-lightweight-same-type-requirements-for-primary-associated-types/56747)) + +## Introduction + +As a step toward the goal of improving the UI of generics outlined in [Improving the UI of Generics](https://forums.swift.org/t/improving-the-ui-of-generics/22814#heading--directly-expressing-constraints), this proposal introduces a new syntax for conforming a generic parameter and constraining an associated type via a same-type requirement. + +## Motivation + +Consider a function that returns a `Sequence` of lines in a source file: + +```swift +struct LineSequence : Sequence { + struct Iterator : IteratorProtocol { + mutating func next() -> String? { ... } + } + + func makeIterator() -> Iterator { + return Iterator() + } +} + +func readLines(_ file: String) -> LineSequence { ... } +``` + +Suppose you are implementing a syntax highlighting library. You might define another function which wraps the result in a `SyntaxTokenSequence`, whose element type is `[Token]`, representing an array of syntax-highlighted tokens on each line: + +```swift +func readSyntaxHighlightedLines(_ file: String) + -> SyntaxTokenSequence { + ... +} +``` + +At this point, the concrete result type is rather complex, and we might wish to hide it behind an opaque result type using the `some` keyword: + +```swift +func readSyntaxHighlightedLines(_ file: String) -> some Sequence { + ... +} +``` + +However, the resulting definition of `readSyntaxHighlightedLines()` is not as useful as the original, because the requirement that the `Element` associated type of the resulting `Sequence` is equal to `[Token]` cannot be expressed. + +As another example, consider a global function `concatenate` that operates on two arrays of `String`: + +```swift +func concatenate(_ lhs: Array, _ rhs: Array) -> Array { + ... +} +``` + +To generalize this function to arbitrary sequences, one might write: + +```swift +func concatenate(_ lhs: S, _ rhs: S) -> S where S.Element == String { + ... +} +``` + +However, while `where` clauses are very general and allow complex generic requirements to be expressed, they also introduce cognitive overhead when reading and writing the declaration, and looks quite different than the concrete implementation where the type was simply written as `Array`. It would be nice to have a simpler solution for cases where there is only a single same-type requirement, as above. + +## Proposed solution + +We’d like to propose a new syntax for declaring a protocol conformance requirement together with one or more same-type requirements on the protocol's _primary associated types_. This new syntax looks like the application of a concrete generic type to a list of type arguments, allowing you to write `Sequence` or `Sequence<[Token]>`. This builds on the user's previous intuition and understanding of generic types and is analogous to `Array` and `Array<[Token]>`. + +Protocols can declare one or more primary associated types using a syntax similar to a generic parameter list of a concrete type: + +```swift +protocol Sequence { + associatedtype Element + associatedtype Iterator : IteratorProtocol + where Element == Iterator.Element + ... +} + +protocol DictionaryProtocol { + associatedtype Key : Hashable + associatedtype Value + ... +} +``` + +A protocol with primary associated types can be written with a list of type arguments in angle brackets, from any position where a protocol conformance requirement was previously allowed. + +For example, an opaque result type can now constrain the primary associated type: + +```swift +func readSyntaxHighlightedLines(_ file: String) -> some Sequence<[Token]> { + ... +} +``` + +The `concatenate()` function shown earlier can now be written like this: + +```swift +func concatenate>(_ lhs: S, _ rhs: S) -> S { + ... +} +``` + +Primary associated types are intended to be used for associated types which are usually provided by the caller. These associated types are often witnessed by generic parameters of the conforming type. For example, `Element` is a natural candidate for the primary associated type of `Sequence`, since `Array` and `Set` both conform to `Sequence`, with the `Element` associated type witnessed by a generic parameter in the corresponding concrete types. This introduces a clear correspondence between the constrained protocol type `Sequence` on one hand and the concrete types `Array`, `Set` on the other hand. + +## Detailed design + +At the protocol declaration, an optional _primary associated types list_ delimited by angle brackets can follow the protocol name. When present, at least one primary associated type must be named. Multiple primary associated types are separated by commas. Each entry in the primary associated type list must name an existing associated type declared in the body of the protocol or one of its inherited protocols. The formal grammar is amended as follows, adding an optional **primary-associated-type-list** production to **protocol-declaration**: + +- **protocol-declaration** → attributesopt access-level-modifieropt `protocol` protocol-name primary-associated-type-listopt type-inheritance-clauseopt generic-where-clauseopt protocol-body +- **primary-associated-type-list** → `<` primary-associated-type-entry `>` +- **primary-associated-type-entry** → primary-associated-type | primary-associated-type `,` primary-associated-type-entry +- **primary-associated-type** → type-name + +Some examples: + +```swift +// Primary associated type 'Element' is declared inside the protocol +protocol SetProtocol { + associatedtype Element : Hashable + ... +} + +protocol SortedMap { + associatedtype Key + associatedtype Value +} + +// Primary associated types 'Key' and 'Value' are declared inside +// the inherited 'SortedMap' protocol +protocol PersistentSortedMap : SortedMap { + ... +} +``` + +At the usage site, a _constrained protocol type_ may be written with one or more type arguments, like `P`. Omitting the list of type arguments altogether is permitted, and leaves the protocol unconstrained. Specifying fewer or more type arguments than the number of primary associated types is an error. Adding a primary associated type list to a protocol is a source-compatible change; the protocol can still be referenced without angle brackets as before. + +### Constrained protocols in desugared positions + +An exhaustive list of positions where the constrained protocol syntax may appear follows. In the first set of cases, the new syntax is equivalent to the existing `where` clause syntax with a same-type requirement constraining the primary associated types. + +- The extended type of an extension, for example: + + ```swift + extension Collection { ... } + + // Equivalent to: + extension Collection where Element == String { ... } + ``` + +- The inheritance clause of another protocol, for example: + + ```swift + protocol TextBuffer : Collection { ... } + + // Equivalent to: + protocol TextBuffer : Collection where Element == String { ... } + ``` + +- The inheritance clause of a generic parameter, for example: + + ```swift + func sortLines>(_ lines: S) -> S + + // Equivalent to: + func sortLines(_ lines: S) -> S + where S.Element == String + ``` + +- The inheritance clause of an associated type, for example: + + ```swift + protocol Document { + associatedtype Lines : Collection + } + + // Equivalent to: + protocol Document { + associatedtype Lines : Collection + where Lines.Element == String + } + ``` + +- The right-hand side of a conformance requirement in a `where` clause, for example: + + ```swift + func merge(_ sequences: S) + where S.Element : Sequence + + // Equivalent to: + func merge(_ sequences: S) + where S.Element : Sequence, S.Element.Element == String + ``` + +- An opaque parameter declaration (see [SE-0341 Opaque Parameter Declarations](0341-opaque-parameters.md)): + + ```swift + func sortLines(_ lines: some Collection) + + // Equivalent to: + func sortLines>(_ lines: C) + + // In turn equivalent to: + func sortLines(_ lines: C) + where C.Element == String + ``` + +- The protocol arguments can contain nested opaque parameter declarations. For example, + + ```swift + func sort(elements: inout some Collection) {} + + // Equivalent to: + func sort(elements: inout C) {} + where C.Element == E + ``` + +When referenced from one of the above positions, a conformance requirement `T : P` desugars to a conformance requirement `T : P` followed by one or more same-type requirements: + +```swift +T : P +T.PrimaryType1 == Arg1 +T.PrimaryType2 == Arg2 +... +``` + +If the right hand side `Arg1` is itself an opaque parameter type, a fresh generic parameter is introduced for use as the right-hand side of the same-type requirement. See [SE-0341 Opaque Parameter Declarations](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md) for details. + +### Constrained protocols in opaque result types + +- A constrained protocol may appear in an opaque result type specified by the `some` keyword. In this case, the syntax actually allows you to express something that was previously not possible to write, since we do not allow `where` clauses on opaque result types: + + ```swift + func transformElements, E>(_ lines: S) -> some Sequence + ``` + + This example also demonstrates that the argument can itself depend on generic parameters from the outer scope. + + The [SE-0328 Structural Opaque Result Types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0328-structural-opaque-result-types.md) pitch allows multiple occurrences of `some` in a return type. This generalizes to constrained protocol types, whose constraint can be another opaque result type: + + ```swift + func transform(_: some Sequence) -> some Sequence + ``` + + Note that in the above, the opaque result type `some Sequence` is unrelated to the opaque _parameter_ type `some Sequence`. The parameter type is provided by the caller. The opaque result type is a (possibly different) homogeneous sequence of elements, where the element type is known to conform to `some Equatable` but is otherwise opaque to the caller. + +### Other positions + +There are three more places where constrained protocols may appear: + +- In the inheritance clause of a concrete type, for example: + + ```swift + struct Lines : Collection { ... } + ``` + + In this position it is sugar for specifying the associated type witness, similar to explicitly declaring a typealias: + + ```swift + struct Lines : Collection { + typealias Element = String + } + ``` + +- As the underlying type of a typealias: + + ```swift + typealias SequenceOfInt = Sequence + ``` + + The typealias may be used in any position where the constrained protocol type itself would be used. + +- As a member of a protocol composition, when the protocol composition appears in any position where a constrained protocol type would be valid: + + ```swift + func takeEquatableSequence(_ seqs: some Sequence & Equatable) {} + ``` + +### Unsupported positions + +A natural generalization is to enable this syntax for existential types, e.g. `any Collection`. This is a larger feature that needs careful consideration of type conversion behaviors. It will also require runtime support for metadata and dynamic casts. For this reason it will be covered by a separate proposal. + +## Alternatives considered + +### Treat primary associated type list entries as declarations + +In an earlier revision of this proposal, the associated type list entries would declare new associated types, instead of naming associated types declared in the body. That is, you would write + +```swift +protocol SetProtocol { + ... +} +``` + +instead of + +```swift +protocol SetProtocol { + associatedtype Key : Hashable + ... +} +``` + +We felt that allowing declaration of associated types in the primary associated type list promotes confusion that what’s actually happening here is a generic declaration, when it is semantically different in important ways. Another potential source of confusion would be if a primary associated type declared a default type witness: + +```swift +protocol SetProtocol { + ... +} +``` + +This makes it look like a "defaulted generic parameter", which it is not; writing `SetProtocol` means leaving `Key` unconstrained, and is not the same as `SetProtocol`. The new form makes it clearer that what is going on here is that a default is being declared for conformances to the protocol, and not for the usage site of the generic constraint: + +```swift +protocol SetProtocol { + associatedtype Key : Hashable = String + ... +} +``` + +### Require associated type names, e.g. `Collection<.Element == String>` + +Explicitly writing associated type names to constrain them in angle brackets has a number of benefits: + +* Doesn’t require any special syntax at the protocol declaration. +* Explicit associated type names allows constraining arbitrary associated types. + +There are also a number of drawbacks to this approach: + +* No visual clues at the protocol declaration about what associated types are useful. +* The use-site may become onerous. For protocols with only one primary associated type, having to specify the name of it is unnecessarily repetitive. +* The syntax can be confusing when the constrained associated type has the same name as a generic parameter of the declaration. For example, the following: + + ```swift + func adjacentPairs(_: some Sequence, + _: some Sequence) + -> some Sequence<(Element, Element)> {} + ``` + + reads better than the hypothetical alternative: + + ```swift + func adjacentPairs(_: some Sequence<.Element == Element>, + _: some Sequence<.Element == Element>) + -> some Sequence<.Element == (Element, Element)> {} + ``` + +* This more verbose syntax is not as clear of an improvement over the existing syntax today, because most of the where clause is still explicitly written. This may also encourage users to specify most or all generic constraints in angle brackets at the front of a generic signature instead of in the `where` clause, violates a core tenet of [SE-0081 Move where clause to end of declaration](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0081-move-where-expression.md). + +* Finally, this syntax lacks the symmetry between concrete types and generic types; generalizing from `Array` requires learning and writing the novel syntax `some Collection<.Element == Int>` instead of simply `some Collection`. + +Note that nothing in this proposal _precludes_ adding the above syntax in the future; the presence of a leading dot (or some other signifier) should allow unambiguous parsing in either case. + +### Implement more general syntax for opaque result type requirements first + +As previously mentioned, in the case of opaque result types, this proposal introduces new expressive power, since opaque result types cannot have a `where` clause where a same-type requirement on a primary associated type could otherwise be written. + +It would be possible to first introduce a language feature allowing general requirements on opaque result types. One such possibility is "named opaque result types", which can have requirements imposed upon them in a `where` clause: + +```swift +func readLines(_ file: String) -> some Sequence { ... } + +// Equivalent to: +func readLines(_ file: String) -> S + where S : Sequence, S.Element == String { ... } +``` + +However, the goal of this proposal is to make generics more approachable by introducing a symmetry between concrete types and generics, and make generics feel more like a generalization of what programmers coming from other languages are already familiar with. + +A more general syntax for opaque result types can be considered on its own merits, and as with the `some Collection<.Element == Int>` syntax discussed in the previous section, nothing in this proposal precludes opaque result types from being generalized further in the future. + +### Annotate regular `associatedtype` declarations with `primary` + +Adding some kind of modifier to `associatedtype` declaration shifts complexity to the users of an API because it’s still distinct from how generic types declare their parameters, which goes against the progressive disclosure principle, and, if we choose to generalize this proposal to multiple primary associated types in the future, requires an understanding of ordering on the use-site. + +This would also make declaration order significant, in a way that is not currently true for the members of a protocol definition. + +Annotation of associated type declarations could make it easier to conditionally declare a protocol which defines primary associated types in newer compiler versions only. The syntax described in this proposal applies to the protocol declaration itself. As a consequence, a library wishing to adopt this feature in a backwards-compatible manner must duplicate entire protocol definitions behind `#if` blocks: + +```swift +#if swift(>=5.7) +protocol SetProtocol { + associatedtype Element : Hashable + + var count: Int { get } + ... +} +#else +protocol SetProtocol { + associatedtype Element : Hashable + + var count: Int { get } + ... +} +#endif +``` + +With a hypothetical `primary` keyword, only the primary associated types must be duplicated: + +```swift +protocol SetProtocol { +#if swift(>=5.7) + primary associatedtype Element : Hashable +#else + associatedtype Element : Hashable +#if + + var count: Int { get } + ... +} +``` + +However, duplicating the associated type declaration in this manner is still an error-prone form of code duplication, and it makes the code harder to read. We feel that this use case should not unnecessarily hinder the evolution of the language syntax. The concerns of libraries adopting new language features while remaining compatible with older compilers is not unique to this proposal, and would be best addressed with a third-party pre-processor tool. + +With the current proposed syntax, it is sufficient for the pre-processor to strip out everything between angle brackets after the protocol name to produce a backward-compatible declaration. A minimal implementation of such a pre-processor is a simple sed invocation, like `sed -e 's/\(protocol .*\)<.*> {/\1 {/'`. + +### Generic protocols + +This proposal uses the angle-bracket syntax for constraining primary associated types, instead of a hypothetical "generic protocols" feature modeled after Haskell's multi-parameter typeclasses or Rust's generic traits. The idea is that such a "generic protocol" can be parametrized over multiple types, not just a single `Self` conforming type: + +```swift +protocol ConvertibleTo { + static func convert(_: Self) -> Other +} + +extension String : ConvertibleTo { + static func convert(_: String) -> Int +} + +extension String : ConvertibleTo { + static func convert(_: String) -> Double +} +``` + +We believe that constraining primary associated types is a more generally useful feature than generic protocols, and using angle-bracket syntax for constraining primary associated types gives users what they generally expect, with the clear analogy between `Array` and `Collection`. + +Nothing in this proposal precludes introducing generic protocols in the future under a different syntax, perhaps something that does not privilege the `Self` type over other types to make it clear there is no functional dependency between the type parameters like there is with associated types: + +```swift +protocol Convertible(from: Self, to: Other) { + static func convert(_: Self) -> Other +} + +extension Convertible(from: String, to: Int) { + static func convert(_: String) -> Int +} + +extension Convertible(from: String, to: Double) { + static func convert(_: String) -> Double +} +``` + +## Source compatibility + +This proposal does not impact source compatibility for existing code. + +Adding a primary associated type list to an existing protocol is a source-compatible change. + +The following are **source-breaking** changes: + +- Removing the primary associated type list from an existing protocol. +- Changing the order or contents of a primary associated type list of an existing protocol. + +## Effect on ABI stability + +This proposal does not impact ABI stability for existing code. The new feature does not require runtime support and can be backward-deployed to existing Swift runtimes. + +The primary associated type list is not part of the ABI, so all of the following are binary-compatible changes: + +- Adding a primary associated type list to an existing protocol. +- Removing the primary associated type list from a existing protocol. +- Changing the primary associated type list of an existing protocol. + +The last two are source-breaking however, so are not recommended. + +## Effect on API resilience + +This change does not impact API resilience for existing code. + +## Future Directions + +### Standard library adoption + +Actually adopting primary associated types in the standard library is outside of the scope of this proposal. There are the obvious candidates such as `Sequence` and `Collection`, and no doubt others that will require additional discussion. + +### Constrained existentials + +As stated above, this proposal alone does not enable constrained protocol existential types, such as `any Collection`. + +## Acknowledgments + +Thank you to Joe Groff for writing out the original vision for improving generics ergonomics — which included the initial idea for this feature — and to Alejandro Alonso for implementing the lightweight same-type constraint syntax for extensions on generic types which prompted us to think about this feature again for protocols. diff --git a/proposals/0347-type-inference-from-default-exprs.md b/proposals/0347-type-inference-from-default-exprs.md new file mode 100644 index 0000000000..db105e1d92 --- /dev/null +++ b/proposals/0347-type-inference-from-default-exprs.md @@ -0,0 +1,264 @@ +# Type inference from default expressions + +* Proposal: [SE-0347](0347-type-inference-from-default-exprs.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Rationale](https://forums.swift.org/t/accepted-se-0347-type-inference-from-default-expressions/56558) +* Implementation: [apple/swift#41436](https://github.com/apple/swift/pull/41436) + +## Introduction + +It's currently impossible to use a default value expression with a generic parameter type to default the argument and its type: + +```swift +func compute(_ values: C = [0, 1, 2]) { ❌ + ... +} +``` + +An attempt to compile this declaration results in the following compiler error - `default argument value of type '[Int]' cannot be converted to type 'C'` because, under the current semantic rules, the type of a default expression has to work for every possible concrete type replacement of `C` inferred at a call site. There are couple of ways to work around this expressivity limitation, but all of them require overloading which complicates APIs: + +``` +func compute(_ values: C) { // original declaration without default + ... +} + +func compute(_ values: [Int] = [0, 1, 2]) { // concretely typed overload of `compute` with default value + ... +} +``` + +I propose to allow type inference for generic parameters from concretely-typed default parameter values (referred to as default expressions in the proposal) when the call-site omits an explicit argument. Concretely-typed default expressions would still be rejected by the compiler if generic parameters associated with a defaulted parameter could be inferred _at a call site_ from any other location in a parameter list by an implicit or explicit argument. For example, declaration `func compute(_: T = 42, _: U) where U: Collection, U.Element == T` is going to be rejected by the compiler because it's possible to infer a type of `T` from the second argument, but declaration `func compute(_: T = 42, _: U = []) where U: Collection, U.Element == Int` is going to be accepted because `T` and `U` are independent. + +Under the proposed rules, the original `compute` declaration becomes well formed and doesn't require any additional overloads: + +```swift +func compute(_ values: C = [0, 1, 2]) { ✅ + ... +} +``` + +Swift-evolution thread: [Discussion thread topic for that proposal](https://forums.swift.org/t/pitch-type-inference-from-default-expressions/55585) + + +## Motivation + +Interaction between generic parameters and default expressions is confusing when default expression only works for a concrete specialization of a generic parameter. It's possible to spell it in the language (in some circumstances) but requires boiler-plate code and knowledge about nuances of constrained extensions. + +For example, let's define a `Flags` protocol and a container type for default set of flags: + +```swift +protocol Flags { + ... +} + +struct DefaultFlags : Flags { + ... +} +``` + +Now, let's declare a type that accepts a set of flags to act upon during initialization. + +```swift +struct Box { + init(dimensions: ..., flags: F) { + ... + } +} +``` + + +To create a `Box` , the caller would have to pass an instance of type conforming to `Flags` to its initializer call. If the majority of `Box`es doesn’t require any special flags, this makes for subpar API experience, because although there is a `DefaultFlags` type, it’s not currently possible to provide a concretely typed default value for the `flags` parameter, e.g. (`flags: F = DefaultFlags()`). Attempting to do so results in the following error: + +``` +error: default argument value of type 'DefaultFlags' cannot be converted to type 'F' +``` + +This happens because even though `DefaultFlags` does conform to protocol `Flags` the default value cannot be used for _every possible_ `F` that can be inferred at a call site, only when `F` is `DefaultFlags`. + +To avoid having to pass flags, it's possible to "specialize" the initializer over a concrete type of `F` via combination of conditional extension and overloading. + +Let’s start with a direct `where` clause: + +```swift +struct Box { + init(dimensions: ..., flags: F = DefaultFlags()) where F == DefaultFlags { + ... + } +} +``` + +This `init` declaration results in a loss of memberwise initializers for `Box`. + +Another possibility is a constrained extension which makes `F` concrete `DefaultFlags` like so: + +```swift +extension Box where F == DefaultFlags { + init(dimensions: ..., flags: F = DefaultFlags()) { + ... + } +} +``` + +Initialization of `Box` without `flags:` is now well-formed and implicit memberwise initializers are preserved, albeit with `init` now being overloaded, but this approach doesn’t work in situations where generic parameters belong to the member itself. + +Let’s consider that there is an operation on our `Box` type that requires passing a different set of flags: + +```swift +extension Box { + func ship(_ flags: F) { + ... + } +} +``` + + +The aforementioned approach that employs constrained extension doesn’t work in this case because generic parameter `F` is associated with the method `ship` instead of the `Box` type. There is another trick that works in this case - overloading. + + New method would have to have a concrete type for `flags:` like so: + +```swift +extension Box { + func ship(_ flags: DefaultShippingFlags = DefaultShippingFlags()) { + ... + } +} +``` + +This is a usability pitfall - what works for some generic parameters, doesn’t work for others, depending on whether the parameter is declared. This inconsistency sometimes leads to API authors reaching for existential types, potentially without realizing all of the consequences that might entail, because a declaration like this would be accepted by the compiler: + +```swift +extension Box { + func ship(_ flags: any Flags = DefaultShippingFlags()) { + ... + } +} +``` + + Also, there is no other way to associate default value `flags:` parameter without using existential types for enum declarations: + +```swift +enum Box { +} + +extension Box where F == DefaultFlags { + case flatRate(dimensions: ..., flags: F = DefaultFlags()) ❌ // error: enum 'case' is not allowed outside of an enum +} +``` + + +To summarize, there is a expressivity limitation related to default expressions which could be, only in some circumstances, mitigated via constrained extensions feature, its other issues include: + +1. Doesn’t work for generic parameters associated with function, subscript, or case declarations because constrained extensions could only be declared for types i.e. `init(..., flags: F = F()) where F == DefaultFlags` is not allowed. +2. Methods have to be overloaded, which increases API surface of the `Box` , and creates a matrix of overloads if there are more than combination of parameters with default values required i.e. if `dimensions` parameter was to be made generic and defaulted for some box sides. +3. Doesn’t work for `enum` declarations at all because Swift does not support overloading cases or declaring them in extensions. +4. Requires know-how related to constrained extensions and their ability to bind generic parameters to concrete types. + +## Proposed solution + +To address the aforementioned short-comings of the language, I propose to support a more concise and intuitive syntax - to allow concretely typed default expressions to be associated with parameters that refer to generic parameters. + +```swift +struct Box { + init(flags: F = DefaultFlags()) { + ... + } +} + +Box() // F is inferred to be DefaultFlags +Box(flags: CustomFlags()) // F is inferred to be CustomFlags +``` + +This syntax could be achieved by amending the type-checking semantics associated with default expressions to allow type inference from them at call sites in cases where such inference doesn’t interfere with explicitly passed arguments. + +## Detailed design + +Type inference from default expressions would be allowed if: + +1. The generic parameter represents either a direct type of a parameter i.e. `(_: T = ...)` or used in a nested position i.e. `(_: [T?] = ...)` +2. The generic parameter is used only in a single location in the parameter list. For example, `(_: T, _: T = ...)` or `(_: [T]?, _: T? = ...)` are *not* allowed because only an explicit argument is permitted to resolve a type conflict to avoid any surprising behavior related to implicit joining of the types. + 1. Note: A result type is allowed to reference generic parameter types inferable from default expressions to make it possible to use the feature while declaring initializers of generic types or `case`s of generic enums. +3. There are no same-type generic constraints that relate a generic parameter that could be inferred from a default expression with any other parameter that couldn’t be inferred from the same expression. For example, `(_: T = [...], _: U) where T.Element == U` is not allowed because `U` is not associated with defaulted parameter where `T` is used, but `(_: [(K, V?)] = ...) where K.Element == V` is permitted because both generic parameters are associated with one expression. +4. The default expression produces a type that satisfies all of the conformance, layout and other generic requirements placed on each generic parameter it would be used to infer at a call site. + + +With these semantic updates, both the initializer and `ship` method of the `Box` type could be expressed in a concise and easily understandable way that doesn’t require any constrained extensions or overloading: + +```swift +struct Box { + init(dimensions: ..., flags: F = DefaultFlags()) { + ... + } + + func ship(_ flags: F = DefaultShippingFlags()) { + ... + } +} +``` + +`Box` could also be converted to an enum without any loss of expressivity: + +```swift +enum Box { +case flatRate(dimensions: D = [...], flags: F = DefaultFlags()) +case overnight(dimensions: D = [...], flags: F = DefaultFlags()) +... +} +``` + +At the call site, if the defaulted parameter doesn’t have an argument, the type-checker will form an argument conversion constraint from the default expression type to the parameter type, which guarantees that all of the generic parameter types are always inferred. + +```swift +let myBox = Box(dimensions: ...) // F is inferred as DefaultFlags + +myBox.ship() // F is inferred as DefaultShippingFlags +``` + +Note that it is important to establish association between the type of a default expression and a corresponding parameter type not just for inference sake, but to guarantee that there are not generic parameter type clashes with a result type (which is allowed to mention the same generic parameters): + +```swift +func compute(initialValues: T = [0, 1, 2, 3]) -> T { + // A complex computation that uses initial values +} + +let result: Array = compute() ✅ +// Ok both `initialValues` and result type are the same type - `Array` + +let result: Array = compute() ❌ +// This is an error because type of default expression is `Array` and result +// type is `Array` +``` + +## Source compatibility + +Proposed changes to default expression handling do not break source compatibility. + + +## Effect on ABI stability + +No ABI impact since this is an additive change to the type-checker. + + +## Effect on API resilience + +All of the resilience rules associated with adding and removing of default expressions are left unchanged, see https://github.com/apple/swift/blob/main/docs/LibraryEvolution.rst#id12 for more details. + + +## Alternatives considered + +[Default generic arguments](https://github.com/apple/swift/blob/main/docs/GenericsManifesto.md#default-generic-arguments) feature mentioned in the Generics Manifesto should not be confused with type inference rules proposed here. Having an ability to default generic arguments alone is not enough to provide a consistent way to use default expressions when generic parameters are involved. The type inference described in this proposal would still be necessary allow default expressions with concrete type to be used when the parameter references a type parameter, and to determine whether the default expression works with a default generic argument type, which means that default generic arguments feature could be considered an enhancement instead of an alternative approach. + +A number of similar approaches has been discussed on Swift Forums, one of them being [[Pre-pitch] Conditional default arguments - #4 by Douglas_Gregor - Dis...](https://forums.swift.org/t/pre-pitch-conditional-default-arguments/7122/4) which relies on overloading, constrained extensions, and/or custom attributes and therefore has all of the issues outlined in the Motivation section. Allowing type inference from default expressions in this regard is a much cleaner approach that works for all situations without having to introduce any new syntax or custom attributes. + + +## Future Directions + +This proposal limits use of inferable generic parameters to a single location in a parameter list because all default expressions are type-checked independently. It is possible to lift this restriction and type-check all of the default expressions together which means that if generic parameters is inferable from different default expressions its type is going to be a common type that fits all locations (action of obtaining such a type is called type-join). It’s not immediately clear whether lifting this restriction would always adhere to the principle of the least surprise for the users, so it would require a separate discussion if this proposal is accepted. + +The simplest example that illustrates the problem is `test(a: T = 42, b: T = 4.2)-> T` , this declaration creates a matrix of possible calls each of which could be typed differently: + +1. `test()` — T = Double because the only type that fits both `42` and `4.2` is `Double` +2. `test(a: 0.0)` — T = `Double` +3. `test(b: 0)` — T = `Int` +4. `let _: Int = test()` - fails because `T` cannot be `Int` and `Double` at the same time. diff --git a/proposals/0348-buildpartialblock.md b/proposals/0348-buildpartialblock.md new file mode 100644 index 0000000000..4a196e68c6 --- /dev/null +++ b/proposals/0348-buildpartialblock.md @@ -0,0 +1,306 @@ +# `buildPartialBlock` for result builders + +* Proposal: [SE-0348](0348-buildpartialblock.md) +* Author: [Richard Wei](https://github.com/rxwei) +* Implementation: [apple/swift#41576](https://github.com/apple/swift/pull/41576) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** + +## Overview + +We introduce a new result builder customization point that allows components of a block to be combined pairwise. + +```swift +@resultBuilder +enum Builder { + /// Builds a partial result component from the first component. + static func buildPartialBlock(first: Component) -> Component + + /// Builds a partial result component by combining an accumulated component + /// and a new component. + /// - Parameter accumulated: A component representing the accumulated result + /// thus far. + /// - Parameter next: A component representing the next component after the + /// accumulated ones in the block. + static func buildPartialBlock(accumulated: Component, next: Component) -> Component +} +``` + +When `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` are both provided, the [result builder transform](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md#the-result-builder-transform) will transform components in a block into a series of calls to `buildPartialBlock`, combining one subsequent line into the result at a time. + +```swift +// Original +{ + expr1 + expr2 + expr3 +} + +// Transformed +// Note: `buildFinalResult` and `buildExpression` are called only when they are defined, just like how they behave today. +{ + let e1 = Builder.buildExpression(expr1) + let e2 = Builder.buildExpression(expr2) + let e3 = Builder.buildExpression(expr3) + let v1 = Builder.buildPartialBlock(first: e1) + let v2 = Builder.buildPartialBlock(accumulated: v1, next: e2) + let v3 = Builder.buildPartialBlock(accumulated: v2, next: e3) + return Builder.buildFinalResult(v3) +} +``` + +The primary goal of this feature is to reduce the code bloat caused by overloading `buildBlock` for multiple arities, allowing libraries to define builder-based generic DSLs with joy and ease. + +## Motivation + +Among DSLs powered by result builders, it is a common pattern to combine values with generic types in a block to produce a new type that contains the generic parameters of the components. For example, [`ViewBuilder`](https://developer.apple.com/documentation/swiftui/viewbuilder) and [`SceneBuilder`](https://developer.apple.com/documentation/swiftui/scenebuilder) in SwiftUI use `buildBlock` to combine views and scenes without losing strong types. + +```swift +extension SceneBuilder { + static func buildBlock(Content) -> Content + static func buildBlock(_ c0: C0, _ c1: C1) -> some Scene where C0: Scene, C1: Scene + ... + static func buildBlock(_ c0: C0, _ c1: C1, _ c2: C2, _ c3: C3, _ c4: C4, _ c5: C5, _ c6: C6, _ c7: C7, _ c8: C8, _ c9: C9) -> some Scene where C0: Scene, C1: Scene, C2: Scene, C3: Scene, C4: Scene, C5: Scene, C6: Scene, C7: Scene, C8: Scene, C9: Scene +} +``` + +Due to the lack of variadic generics, `buildBlock` needs to be overloaded for any supported block arity. This unfortunately increases code size, causes significant code bloat in the implementation and documentation, and it is often painful to write and maintain the boiletplate. + +While this approach works for types like `ViewBuilder` and `SceneBuilder`, some builders need to define type combination rules that are far too complex to implement with overloads. One such example is [`RegexComponentBuilder`](https://github.com/apple/swift-experimental-string-processing/blob/85c7d906dd871364357156126278d9d427936ca4/Sources/_StringProcessing/RegexDSL/Builder.swift#L13) in [Declarative String Processing](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/DeclarativeStringProcessing.md). + +The regex builder DSL is designed to allow developers to easily compose regex patterns. [Strongly typed captures](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/StronglyTypedCaptures.md#strongly-typed-regex-captures) are represented as part of the `Match` generic parameter in the `Regex` type, which has a builder-based initializer. + +```swift +struct Regex { + init(@RegexComponentBuilder _ builder: () -> Self) +} +``` + +> #### Recap: Regular expression capturing basics +> +> When a regular expression does not contain any capturing groups, its `Match` type is `Substring`, which represents the whole matched portion of the input. +> +> ```swift +> let noCaptures = #/a/# // => Regex +> ``` +> +> When a regular expression contains capturing groups, i.e. `(...)`, the `Match` type is extended as a tuple to also contain *capture types*. Capture types are tuple elements after the first element. +> +> ```swift +> // ________________________________ +> // .0 | .0 | +> // ____________________ _________ +> let yesCaptures = #/a(?:(b+)c(d+))+e(f)?/# // => Regex<(Substring, Substring, Substring, Substring?)> +> // ---- ---- --- --------- --------- ---------- +> // .1 | .2 | .3 | .1 | .2 | .3 | +> // | | | | | | +> // | | |_______________________________ | ______ | ________| +> // | | | | +> // | |______________________________________ | ______ | +> // | | +> // |_____________________________________________| +> // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +> // Capture types +> ``` + +Using the result builder syntax, the regular expression above becomes: + +```swift +let regex = Regex { + "a" // => Regex + OneOrMore { // { + Capture { OneOrMore("b") } // => Regex<(Substring, Substring)> + "c" // => Regex + Capture { OneOrMore("d") } // => Regex<(Substring, Substring)> + } // } => Regex<(Substring, Substring, Substring)> + "e" // => Regex + Optionally { Capture("f") } // => Regex<(Substring, Substring?)> +} // => Regex<(Substring, Substring, Substring, Substring?)> + // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + // Capture types + +let result = "abbcddbbcddef".firstMatch(of: regex) +// => MatchResult<(Substring, Substring, Substring, Substring?)> +``` + +`RegexComponentBuilder` concatenates the capture types of all components as a flat tuple, forming a new `Regex` whose `Match` type is `(Substring, CaptureType...)`. We can define the following `RegexComponentBuilder`: + +```swift +@resultBuilder +enum RegexComponentBuilder { + static func buildBlock() -> Regex + static func buildBlock(_: Regex) -> Regex + // Overloads for non-tuples: + static func buildBlock(_: Regex, _: Regex) -> Regex + static func buildBlock(_: Regex, _: Regex, _: Regex) -> Regex + ... + static func buildBlock(_: Regex, _: Regex, _: Regex, ..., _: Regex) -> Regex + // Overloads for tuples: + static func buildBlock(_: Regex<(W0, C0)>, _: Regex) -> Regex<(Substring, C0)> + static func buildBlock(_: Regex, _: Regex<(W1, C0)>) -> Regex<(Substring, C0)> + static func buildBlock(_: Regex<(W0, C0, C1)>, _: Regex) -> Regex<(Substring, C0, C1)> + static func buildBlock(_: Regex<(W0, C0)>, _: Regex<(W1, C1)>) -> Regex<(Substring, C0, C1)> + static func buildBlock(_: Regex, _: Regex<(W1, C0, C1)>) -> Regex<(Substring, C0, C1)> + ... + static func buildBlock( + _: Regex<(W0, C0, C1)>, _: Regex<(W1, C2)>, _: Regex<(W3, C3, C4, C5)>, _: Regex<(W4, C6)> + ) -> Regex<(Substring, C0, C1, C2, C3, C4, C5, C6)> + ... +} +``` + +Here we just need to overload for all tuple combinations for each arity... Oh my! That is an `O(arity!)` combinatorial explosion of `buildBlock` overloads; compiling these methods alone could take hours. + +## Proposed solution + +This proposal introduces a new block-building approach similar to building heterogeneous lists. Instead of calling a single method to build an entire block wholesale, this approach recursively builds a partial block by taking one new component at a time, and thus significantly reduces the number of overloads. + +We introduce a new customization point to result builders via two user-defined static methods: + +```swift +@resultBuilder +enum Builder { + static func buildPartialBlock(first: Component) -> Component + static func buildPartialBlock(accumulated: Component, next: Component) -> Component +} +``` + +When `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` are both defined, the result builder transform will turn components in a block into a series of calls to `buildPartialBlock`, combining components from top to bottom. + +With this approach, many result builder types with overloaded `buildBlock` can be simplified. For example, the `buildBlock` overloads in SwiftUI's `SceneBuilder` could be simplified as the following: + +```swift +extension SceneBuilder { + static func buildPartialBlock(first: some Scene) -> some Scene + static func buildPartialBlock(accumulated: some Scene, next: some Scene) -> some Scene +} +``` + +Similarly, the overloads of `buildBlock` in `RegexComponentBuilder` can be vastly reduced from `O(arity!)`, down to `O(arity^2)` overloads of `buildPartialBlock(accumulated:next:)`. For an arity of 10, 100 overloads are trivial compared to over 3 million ones. + +```swift +extension RegexComponentBuilder { + static func buildPartialBlock(first regex: Regex) -> Regex + static func buildPartialBlock(accumulated: Regex, next: Regex) -> Regex + static func buildPartialBlock(accumulated: Regex<(W0, C0)>, next: Regex) -> Regex<(Substring, C0)> + static func buildPartialBlock(accumulated: Regex, next: Regex<(W1, C0)>) -> Regex<(Substring, C0)> + static func buildPartialBlock(accumulated: Regex, next: Regex<(W1, C0, C1)>) -> Regex<(Substring, C0, C1)> + static func buildPartialBlock(accumulated: Regex<(W0, C0, C1)>, next: Regex) -> Regex<(Substring, C0, C1)> + static func buildPartialBlock(accumulated: Regex<(W0, C0)>, next: Regex<(W1, C1)>) -> Regex<(Substring, C0, C1)> + ... +} +``` + +### Early adoption feedback + +- In the [regex builder DSL](https://forums.swift.org/t/pitch-regex-builder-dsl/56007), `buildPartialBlock` reduced the number of required overloads from millions (`O(arity!)`) down to hundreds (`O(arity^2)`). +- In the pitch thread, [pointfreeco/swift-parsing reported](https://forums.swift.org/t/pitch-buildpartialblock-for-result-builders/55561/10) that `buildPartialBlock` enabled the deletion of 21K lines of generated code, increased arity support, and reduced compile times from 20 seconds to <2 seconds in debug mode. + +## Detailed design + +When a type is marked with `@resultBuilder`, the type was previously required to have at least one static `buildBlock` method. With this proposal, such a type is now required to have either at least one static `buildBlock` method, or both `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)`. + +In the result builder transform, the compiler will look for static members `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` in the builder type. If the following conditions are met: + +* Both methods `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` exist. +* The availability of the enclosing declaration is greater than or equal to the availability of `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)`. + +Then, a non-empty block will be transformed to the following: + +```swift +// Original +{ + expr1 + expr2 + expr3 +} + +// Transformed +// Note: `buildFinalResult` and `buildExpression` are called only when they are defined, just like how they behave today. +{ + let e1 = Builder.buildExpression(expr1) + let e2 = Builder.buildExpression(expr2) + let e3 = Builder.buildExpression(expr3) + let v1 = Builder.buildPartialBlock(first: e1) + let v2 = Builder.buildPartialBlock(accumulated: v1, next: e2) + let v3 = Builder.buildPartialBlock(accumulated: v2, next: e3) + return Builder.buildFinalResult(v3) +} +``` + +Otherwise, the result builder transform will transform the block to call `buildBlock` instead as proposed in [SE-0289](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md). + +## Source compatibility + +This proposal does not intend to introduce source-breaking changes. Although, if an existing result builder type happens to have static methods named `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)`, the result builder transform will be creating calls to those methods instead and may cause errors depending on `buildPartialBlock`'s type signature and implementation. Nevertheless, such cases should be extremely rare. + +## Effect on ABI stability + +This proposal does not contain ABI changes. + +## Effect on API resilience + +This proposal does not contain API changes. + +## Alternatives considered + +### Prefer viable `buildBlock` overloads to `buildPartialBlock` + +As proposed, the result builder transform will always prefer `buildPartialBlock` to `buildBlock` when they are both defined. One could argue that making a single call to a viable overload of `buildBlock` would be more efficient and more customizable. However, because the result builder transform currently operates before type inference has completed, it would increase the type checking complexity to decide to call `buildBlock` or `buildPairwiseBlock` based on argument types. None of the informal requirements (`buildBlock`, `buildOptional`, etc) of result builders depend on argument types when being transformed to by the result builder transform. + +### Use nullary `buildPartialBlock()` to form the initial value + +As proposed, the result builder transform calls unary `buildPartialBlock(first:)` on the first component in a block before calling `buildPartialBlock(accumulated:next:)` on the rest. While it is possible to require the user to define a nullary `buildPartialBlock()` method to form the initial result, this behavior may be suboptimal for result builders that do not intend to support empty blocks, e.g. SwiftUI's `SceneBuilder`. Plus, the proposed approach does allow the user to define an nullary `buildBlock()` to support building an empty block. + +### Rely on variadic generics + +It can be argued that variadic generics would resolve the motivations presented. However, to achieve the concatenating behavior needed for `RegexComponentBuilder`, we would need to be able to express nested type sequences, perform collection-like transformations on generic parameter packs such as dropping elements and splatting. + +```swift +extension RegexComponentBuilder { + static func buildBlock<(W, (C...))..., R...>(_ components: Regex<(W, C...)>) -> Regex<(Substring, (R.Match.dropFirst()...).splat())> +} +``` + +Such features would greatly complicate the type system. + +### Alternative names + +#### Overload `buildBlock` method name + +Because the proposed feature overlaps `buildBlock`, one could argue for reusing `buildBlock` as the method base name instead of `buildPartialBlock` and using argument labels to distinguish whether it is the pairwise version, e.g. `buildBlock(partiallyAccumulated:next:)` or `buildBlock(combining:into:)`. + +```swift +extension Builder { + static func buildBlock(_: Component) -> Component + static func buildBlock(partiallyAccumulated: Component, next: Component) -> Component +} +``` + +However, the phrase "build block" does not have a clear indication that the method is in fact building a partial block, and argument labels do not have the prominence to carry such indication. + +#### Use `buildBlock(_:)` instead of `buildPartialBlock(first:)` + +The unary base case method `buildPartialBlock(first:)` and `buildBlock(_:)` can be viewed as being functionally equivalent, so one could argue for reusing `buildBlock`. However, as mentioned in [Overload `buildBlock` method name](#overload-buildblock-method-name), the phrase "build block" lacks clarity. + +A more important reason is that `buildPartialBlock(first:)`, especially with its argument label `first:`, leaves space for a customization point where the developer can specify the direction of combination. As a future direction, we could allow `buildPartialBlock(last:)` to be defined instead of `buildPartialBlock(first:)`, and in this scenario the result builder transform will first call `buildPartialBlock(last:)` on the _last_ component and then call `buildPartialBlock(accumulated:next:)` on each preceeding component. + +#### Different argument labels + +The proposed argument labels `accumulated:` and `next:` took inspirations from some precedents in the standard library: +- "accumulated" is used as an argument name of [`Array.reduce(into:_:)`](https://developer.apple.com/documentation/swift/array/3126956-reduce). +- "next" is used as an argument name of [`Array.reduce(_:_:)`](https://developer.apple.com/documentation/swift/array/2298686-reduce) + +Meanwhile, there are a number of alternative argument labels considered in the place of `accumulated:` and `next:`. + +Possible replacements for `accumulated:`: +- `partialResult:` +- `existing:` +- `upper:` +- `_:` + +Possible replacements for `next:`: +- `new:` +- `_:` + +We believe that "accumulated" and "next" have the best overall clarity. diff --git a/proposals/0349-unaligned-loads-and-stores.md b/proposals/0349-unaligned-loads-and-stores.md new file mode 100644 index 0000000000..b542077621 --- /dev/null +++ b/proposals/0349-unaligned-loads-and-stores.md @@ -0,0 +1,277 @@ +# Unaligned Loads and Stores from Raw Memory + +* Proposal: [SE-0349](0349-unaligned-loads-and-stores.md) +* Authors: [Guillaume Lessard](https://github.com/glessard), [Andrew Trick](https://github.com/atrick) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#41033](https://github.com/apple/swift/pull/41033) +* Review: ([pitch](https://forums.swift.org/t/55036/)) ([review](https://forums.swift.org/t/se-0349-unaligned-loads-and-stores-from-raw-memory/56423)) ([acceptance](https://forums.swift.org/t/accepted-se-0349-unaligned-loads-and-stores-from-raw-memory/56748)) + +## Introduction + +Swift does not currently provide a clear way to load data from an arbitrary source of bytes, such as a binary file, in which data may be stored without respect for in-memory alignment. This proposal aims to rectify the situation, making workarounds unnecessary. + +## Motivation + +The method `UnsafeRawPointer.load(fromByteOffset offset: Int, as type: T.Type) -> T` requires the address at `self+offset` to be properly aligned to access an instance of type `T`. Attempts to use a combination of pointer and byte offset that is not aligned for `T` results in a runtime crash. Unfortunately, in general, data saved to files or network streams does not adhere to the same restrictions as in-memory layouts do, and tends to not be properly aligned. When copying data from such sources to memory, Swift users therefore frequently encounter alignment mismatches that require using a workaround. This is a longstanding issue reported in e.g. [SR-10273](https://bugs.swift.org/browse/SR-10273). + +For example, given an arbitrary data stream in which a 4-byte value is encoded between byte offsets 3 through 7: + +```swift +let data = Data([0x0, 0x0, 0x0, 0xff, 0xff, 0xff, 0xff, 0x0]) +``` + +In order to extract all the `0xff` bytes of this stream to an `UInt32`, we would like to be able to use `load(as:)`, as follows: + +```swift +let result = data.dropFirst(3).withUnsafeBytes { $0.load(as: UInt32.self) } +``` + +However, that will currently crash at runtime, because in this case `load` requires the base pointer to be correctly aligned for accessing `UInt32`. A workaround is required, such as the following: + +```swift +let result = data.dropFirst(3).withUnsafeBytes { buffer -> UInt32 in + var storage = UInt32.zero + withUnsafeMutableBytes(of: &storage) { + $0.copyBytes(from: buffer.prefix(MemoryLayout.size)) + } + return storage +} +``` + +The necessity of this workaround (or of others that produce the same outcome) is unsatisfactory for two reasons; firstly it is tremendously non-obvious. Secondly, it requires two copies instead of the expected single copy: the first to a correctly-aligned raw buffer, and then to the final, correctly-typed variable. We should be able to do this with a single copy. + +The kinds of types for which it is important to improve loads from arbitrary alignments are types whose values can be copied bit for bit, without reference counting operations. These types are commonly referred to as "POD" (plain old data) or "trivial" types. We propose to restrict the use of the unaligned loading operation to those types. + +## Proposed solution + +We propose to add an API `UnsafeRawPointer.loadUnaligned(fromByteOffset:as:)` to support unaligned loads from `UnsafeRawPointer`, `UnsafeRawBufferPointer` and their mutable counterparts. These will be explicitly restricted to POD types. Loading a non-POD type remains meaningful only when the source memory is another live object where the memory is, by construction, already correctly aligned. The original API (`load`) will continue to support this case. The new API (`loadUnaligned`) will assert that the return type is POD when run in debug mode. + +`UnsafeMutableRawPointer.storeBytes(of:toByteOffset:)` is documented to only be meaningful for POD types. However, at runtime it enforces storage to an offset correctly aligned to the source type. We propose to remove that alignment restriction and instead enforce the documented POD restriction. The API will otherwise be unchanged, though its documentation will be updated. Please see the ABI stability section for a discussion of binary compatibility with this approach. + +The `UnsafeRawBufferPointer` and `UnsafeMutableRawBufferPointer` types will receive matching changes. + +## Detailed design + +```swift +extension UnsafeRawPointer { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// This function only supports loading trivial types, + /// and will trap if this precondition is not met. + /// A trivial type does not contain any reference-counted property + /// within its in-memory representation. + /// The memory at this pointer plus `offset` must be laid out + /// identically to the in-memory representation of `T`. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance isn't associated + /// with the value in the range of memory referenced by this pointer. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T +} +``` + +```swift +extension UnsafeMutableRawPointer { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// This function only supports loading trivial types, + /// and will trap if this precondition is not met. + /// A trivial type does not contain any reference-counted property + /// within its in-memory representation. + /// The memory at this pointer plus `offset` must be laid out + /// identically to the in-memory representation of `T`. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance isn't associated + /// with the value in the range of memory referenced by this pointer. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T + + /// Stores the given value's bytes into raw memory at the specified offset. + /// + /// The type `T` to be stored must be a trivial type. The memory + /// must also be uninitialized, initialized to `T`, or initialized to + /// another trivial type that is layout compatible with `T`. + /// + /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is + /// initialized to the raw bytes of `value`. If the memory is bound to a + /// type `U` that is layout compatible with `T`, then it contains a value of + /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the + /// bound type of the memory. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// If you need to store a copy of a value of a type that isn't trivial into memory, + /// you cannot use the `storeBytes(of:toByteOffset:as:)` method. Instead, you must know + /// the type of value previously in memory and initialize or assign the + /// memory. For example, to replace a value stored in a raw pointer `p`, + /// where `U` is the current type and `T` is the new type, use a typed + /// pointer to access and deinitialize the current value before initializing + /// the memory with a new value. + /// + /// let typedPointer = p.bindMemory(to: U.self, capacity: 1) + /// typedPointer.deinitialize(count: 1) + /// p.initializeMemory(as: T.self, repeating: newValue, count: 1) + /// + /// - Parameters: + /// - value: The value to store as raw bytes. + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of `value`. + public func storeBytes(of value: T, toByteOffset offset: Int = 0, as type: T.Type) +} +``` + + + +`UnsafeRawBufferPointer` and `UnsafeMutableRawBufferPointer` receive a similar addition of a `loadUnaligned` function. It enables loading from an arbitrary offset with the buffer, subject to the usual index validation rules of `BufferPointer` types: indexes are checked when client code is compiled in debug mode, while indexes are unchecked when client code is compiled in release mode. + +```swift +extension Unsafe{Mutable}RawBufferPointer { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// This function only supports loading trivial types. + /// A trivial type does not contain any reference-counted property + /// within its in-memory stored representation. + /// The memory at `offset` bytes into the buffer must be laid out + /// identically to the in-memory representation of `T`. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// You can use this method to create new values from the buffer pointer's + /// underlying bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the next four bytes. + /// + /// let a = someBytes.load(as: Int32.self) + /// let b = someBytes.load(fromByteOffset: 4, as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the buffer + /// pointer's memory region---that is, `offset + MemoryLayout.size` must + /// be less than or equal to the buffer pointer's `count`. + /// + /// - Parameters: + /// - offset: The offset, in bytes, into the buffer pointer's memory at + /// which to begin reading data for the new instance. The buffer pointer + /// plus `offset` must be properly aligned for accessing an instance of + /// type `T`. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer's + /// memory. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T +} +``` + +Additionally, the semantics of `UnsafeMutableBufferPointer.storeBytes(of:toByteOffset)` will be changed in the same way as its counterpart `UnsafeMutablePointer.storeBytes(of:toByteOffset)`, no longer enforcing alignment at runtime. Again, the index validation behaviour is unchanged: indexes are checked when client code is compiled in debug mode, while indexes are unchecked when client code is compiled in release mode. + +```swift +extension UnsafeMutableRawBufferPointer { + /// Stores a value's bytes into the buffer pointer's raw memory at the + /// specified byte offset. + /// + /// The type `T` to be stored must be a trivial type. The memory must also be + /// uninitialized, initialized to `T`, or initialized to another trivial + /// type that is layout compatible with `T`. + /// + /// The memory written to must not extend beyond the buffer pointer's memory + /// region---that is, `offset + MemoryLayout.size` must be less than or + /// equal to the buffer pointer's `count`. + /// + /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is + /// initialized to the raw bytes of `value`. If the memory is bound to a + /// type `U` that is layout compatible with `T`, then it contains a value of + /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the + /// bound type of the memory. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// If you need to store a copy of a value of a type that isn't trivial into memory, + /// you cannot use the `storeBytes(of:toByteOffset:as:)` method. Instead, you must know + /// the type of value previously in memory and initialize or assign the memory. + /// + /// - Parameters: + /// - offset: The offset in bytes into the buffer pointer's memory to begin + /// reading data for the new instance. The buffer pointer plus `offset` + /// must be properly aligned for accessing an instance of type `T`. The + /// default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + public func storeBytes(of value: T, toByteOffset offset: Int = 0, as: T.Type) +} +``` + + + +## Source compatibility + +This proposal is source compatible. The proposed API modifications relax existing restrictions and keep the same signatures, therefore the changes are compatible. The API additions are source compatible by definition. + +## Effect on ABI stability + +Existing binaries that expect the old behaviour of `storeBytes` will not be affected by the relaxed behaviour proposed here, as we will ensure that the old symbol (with its existing semantics) will remain. + +New binaries that require the new behaviour will correctly backwards deploy by the use of the `@_alwaysEmitIntoClient` attribute. The new API will likewise use the `@_alwaysEmitIntoClient` attribute. + +## Effect on API resilience + +If the added API were removed in a future release, the change would be source-breaking but not ABI-breaking, because the proposed additions will always be inlined. + +## Alternatives considered + +#### Use a marker protocol to restrict unaligned loads to trivial types + +We could enforce the use of unaligned loads at compile time by declaring a new marker protocol for trivial types, and require conformance to this protocol for types loaded through a function that can load from unaligned offsets. While this may be the ideal outcome, we believe this option would take too long to be realized. The approach proposed here can be a stepping stone on the way there. + +#### Relax the alignment restriction on the existing `load` API + +Arguably, user expectations are that the `load` API supports unaligned loads, but since that is not the case with the existing API, source-compatibility considerations dictate that the behaviour of the existing API should not change. If its preconditions were relaxed, a developer would encounter runtime crashes when deploying to a server using Swift 5.5, having tested using a newer toolchain. + +For that reason, we chose to leave the existing API untouched. + +Other programming languages have chosen whether loading from bytes is aligned or unaligned by default depending on their focus. For example, Go's [binary](https://pkg.go.dev/encoding/binary@go1.18) package privileges decoding data from a stream, and accordingly its various `Read` functions perform unaligned loads. [binary](https://pkg.go.dev/encoding/binary@go1.18)'s package documentation acknowledges privileging simplicity over efficiency. On the other hand, Rust's [raw pointer](https://doc.rust-lang.org/core/primitive.pointer.html) primitive type includes both [`read`](https://doc.rust-lang.org/core/ptr/fn.read.html) and [`read_unaligned`](https://doc.rust-lang.org/core/ptr/fn.read_unaligned.html) functions, where the default (with the "good" name) is more strict and more efficient. We believe that Swift's goals align well with having the more performant function (aligned load) be the default one. + +#### Add a separate unaligned store API + +Adding a separate unaligned store API would avoid ABI stability concerns, but the old API would become redundant. The risk of removing the restriction on `storeBytes` is less than it is for `load`, as the restriction is implemented using `_debugPrecondition`, which is compiled away in release mode. + +#### Rename `storeBytes` to `storeUnaligned`, or call `loadUnaligned` `loadFromBytes` instead. + +The idea of making the "load" and the "store" operations have more symmetric names is compelling, however there is a fundamental asymmetry in the operation itself. When a `load` operation completes, a new value is created to be managed by the Swift runtime. On the other hand the `storeBytes` operation is completely transparent to the runtime: the destination is a container of bytes that is _not_ managed by the Swift runtime. For this reason, the "store" operation has the word "bytes" in its name. + +## Acknowledgments + +Thanks to the Swift Standard Library team for valuable feedback and discussion. diff --git a/proposals/0350-regex-type-overview.md b/proposals/0350-regex-type-overview.md new file mode 100644 index 0000000000..2eef59ffb0 --- /dev/null +++ b/proposals/0350-regex-type-overview.md @@ -0,0 +1,576 @@ +# Regex Type and Overview + +* Proposal: [SE-0350](0350-regex-type-overview.md) +* Authors: [Michael Ilseman](https://github.com/milseman) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Implementation: https://github.com/apple/swift-experimental-string-processing + * Available in nightly toolchain snapshots with `import _StringProcessing` + +## Introduction + +Swift strings provide an obsessively Unicode-forward model of programming with strings. String processing with `Collection`'s algorithms is woefully inadequate for many day-to-day tasks compared to other popular programming and scripting languages. + +We propose addressing this basic shortcoming through an effort we are calling regex. What we propose is more powerful, extensible, and maintainable than what is traditionally thought of as regular expressions from other programming languages. This effort is presented as 6 interrelated proposals: + +1. `Regex` and `Regex.Match` types with support for typed captures, both static and dynamic. +2. A best-in-class treatment of traditional, familiar regular expression syntax for run-time construction of regex. +3. A literal for compile-time construction of a regex with statically-typed captures, enabling powerful source tools. +4. An expressive and composable result-builder DSL, with support for capturing strongly-typed values. +5. A modern treatment of Unicode semantics and string processing. +6. A slew of regex-powered string processing algorithms, along with library-extensible protocols enabling industrial-strength parsers to be used seamlessly as regex components. + +This proposal provides details on \#1, the `Regex` type and captures, and gives an overview of how each of the other proposals fit into regex in Swift. + +At the time of writing, these related proposals are in various states of being drafted, pitched, or proposed. For the current status, see [Pitch and Proposal Status][pitches]. + +

Obligatory differentiation from formal regular expressions + +Regular expressions originated in formal language theory as a way to answer yes-or-no whether a string is in a given [regular language](https://en.wikipedia.org/wiki/Regular_language). They are more powerful (and less composable) than [star-free languages](https://en.wikipedia.org/wiki/Star-free_language) and less powerful than [context-free languages](https://en.wikipedia.org/wiki/Context-free_language). Because they just answer a yes-or-no question, _how_ that answer is determined is irrelevant; i.e. their execution model is ambiguous. + +Regular expressions were brought into practical applications for text processing and compiler lexers. For searching within text, where the result (including captures) is a portion of the searched text, _how_ a match happened affects the result. Over time, more and more power was needed and "regular expressions" diverged from their formal roots. + +For compiler lexers, especially when implemented as a [discrete compilation phase](https://en.wikipedia.org/wiki/Lexical_analysis), regular expressions were often ingested by a [separate tool](https://en.wikipedia.org/wiki/Flex_(lexical_analyser_generator)) from the rest of the compiler. Understanding formal regular expressions can help clarify the separation of concerns between lexical analysis and parsing. Beyond that, they are less relevant for structuring modern parsers, which interweave error handling and recovery, debuggability, and fine-grained source location tracking across this traditional separation-of-tools. + +The closest formal analogue to what we are proposing are [Parsing Expression Grammars](https://en.wikipedia.org/wiki/Parsing_expression_grammar) ("PEGs"), which describe a recursive descent parser. Our alternation is ordered choice and we support possessive quantification, recursive subpattern calls, and lookahead. However, we are first and foremost providing a regexy presentation: quantification, by default, is non-possessive. + +
+ + +## Motivation + +Imagine processing a bank statement in order to extract transaction details for further scrutiny. Fields are separated by 2-or-more spaces: + +```swift +struct Transaction { + enum Kind: String { + case credit = "CREDIT" + case debit = "DEBIT" + } + + let kind: Kind + let date: Date + let accountName: String + let amount: Decimal +} + +let statement = """ + CREDIT 03/02/2022 Payroll $200.23 + CREDIT 03/03/2022 Sanctioned Individual A $2,000,000.00 + DEBIT 03/03/2022 Totally Legit Shell Corp $2,000,000.00 + DEBIT 03/05/2022 Beanie Babies Forever $57.33 + """ +``` + +One option is to `split()` around whitespace, hard-coding field offsets for everything except the account name, and `join()`ing the account name fields together to restore their spaces. This carries a lot of downsides such as hard-coded offsets, many unnecessary allocations, and this pattern would not easily expand to supporting other representations. + +Another option is to process an entry in a single pass from left-to-right, but this can get unwieldy: + +```swift +// Parse dates using a simple (localized) numeric strategy +let dateParser = Date.FormatStyle(date: .numeric).parseStrategy + +// Parse currencies as US dollars +let decimalParser = Decimal.FormatStyle.Currency(code: "USD") + +func processEntry(_ s: String) -> Transaction? { + var slice = s[...] + guard let kindEndIdx = slice.firstIndex(of: " "), + let kind = Transaction.Kind(slice[.. Transaction? { + let range = NSRange(line.startIndex..` describes a string processing algorithm. Captures surface the portions of the input that were matched by subpatterns. By convention, capture `0` is the entire match. + +### Creating Regex + +Regexes can be created at run time from a string containing familiar regex syntax. If no output type signature is specified, the regex has type `Regex`, in which captures are existentials and the number of captures is queryable at run time. Alternatively, providing an output type signature produces strongly-typed outputs, where captures are concrete types embedded in a tuple, providing safety and enabling source tools such as code completion. + +```swift +let pattern = #"(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)"# +let regex = try! Regex(pattern) +// regex: Regex + +let regex: Regex<(Substring, Substring, Substring, Substring, Substring)> = + try! Regex(pattern) +``` + +*Note*: The syntax accepted and further details on run-time compilation, including `AnyRegexOutput` and extended syntaxes, are discussed in [Run-time Regex Construction][pitches]. + +Type mismatches and invalid regex syntax are diagnosed at construction time by `throw`ing errors. + +When the pattern is known at compile time, regexes can be created from a literal containing the same regex syntax, allowing the compiler to infer the output type. Regex literals enable source tools, e.g. syntax highlighting and actions to refactor into a result builder equivalent. + +```swift +let regex = /(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)/ +// regex: Regex<(Substring, Substring, Substring, Substring, Substring)> +``` + +*Note*: Regex literals, most notably the choice of delimiter, are discussed in [Regex Literals][pitches]. + +This same regex can be created from a result builder, a refactoring-friendly representation: + +```swift +let fieldSeparator = Regex { + CharacterClass.whitespace + OneOrMore(.whitespace) +} + +let regex = Regex { + Capture(OneOrMore(.word)) + fieldSeparator + + Capture(OneOrMore(.whitespace.inverted)) + fieldSeparator + + Capture { + OneOrMore { + NegativeLookahead(fieldSeparator) + CharacterClass.any + } + } + fieldSeparator + + Capture { OneOrMore(.any) } +} +// regex: Regex<(Substring, Substring, Substring, Substring, Substring)> +``` + +*Note*: The result builder API is discussed in [Regex Builders][pitches]. Character classes and other Unicode concerns are discussed in [Unicode for String Processing][pitches]. + +`Regex` itself is a valid component for use inside a result builder, meaning that embedded literals can be used for concision. + +### Using Regex + +A `Regex.Match` contains the result of a match, surfacing captures by number, name, and reference. + +```swift +func processEntry(_ line: String) -> Transaction? { + // Multiline literal implies `(?x)`, i.e. non-semantic whitespace with line-ending comments + let regex = #/ + (? \w+) \s\s+ + (? \S+) \s\s+ + (? (?: (?!\s\s) . )+) \s\s+ + (? .*) + /# + // regex: Regex<( + // Substring, + // kind: Substring, + // date: Substring, + // account: Substring, + // amount: Substring + // )> + + guard let match = line.wholeMatch(of: regex), + let kind = Transaction.Kind(match.kind), + let date = try? Date(String(match.date), strategy: dateParser), + let amount = try? Decimal(String(match.amount), format: decimalParser) + else { + return nil + } + + return Transaction( + kind: kind, date: date, account: String(match.account), amount: amount) +} +``` + +*Note*: Details on typed captures using tuple labels are covered in [Regex Literals][pitches]. + +The result builder allows for inline failable value construction, which participates in the overall string processing algorithm: returning `nil` signals a local failure and the engine backtracks to try an alternative. This not only relieves the use site from post-processing, it enables new kinds of processing algorithms, allows for search-space pruning, and enhances debuggability. + +Swift regexes describe an unambiguous algorithm, where choice is ordered and effects can be reliably observed. For example, a `print()` statement inside the `TryCapture`'s transform function will run whenever the overall algorithm naturally dictates an attempt should be made. Optimizations can only elide such calls if they can prove it is behavior-preserving (e.g. "pure"). + +`CustomMatchingRegexComponent`, discussed in [String Processing Algorithms][pitches], allows industrial-strength parsers to be used a regex components. This allows us to drop the overly-permissive pre-parsing step: + +```swift +func processEntry(_ line: String) -> Transaction? { + let fieldSeparator = Regex { + CharacterClass.whitespace + OneOrMore(.whitespace) + } + + // Declare strongly-typed references to store captured values into + let kind = Reference() + let date = Reference() + let account = Reference() + let amount = Reference() + + let regex = Regex { + TryCapture(as: kind) { + OneOrMore(.word) + } transform: { + Transaction.Kind($0) + } + fieldSeparator + + TryCapture(as: date) { dateParser } + fieldSeparator + + Capture(as: account) { + OneOrMore { + NegativeLookahead(fieldSeparator) + CharacterClass.any + } + } + fieldSeparator + + TryCapture(as: amount) { decimalParser } + } + // regex: Regex<(Substring, Transaction.Kind, Date, Substring, Decimal)> + + guard let match = line.wholeMatch(of: regex) else { return nil } + + return Transaction( + kind: match[kind], + date: match[date], + account: String(match[account]), + amount: match[amount]) +} +``` + +*Note*: Details on how references work is discussed in [Regex Builders][pitches]. `Regex.Match` supports referring to _all_ captures by position (`match.1`, etc.) whether named or referenced or neither. Due to compiler limitations, result builders do not support forming labeled tuples for named captures. + + +### Regex-powered algorithms + +Regexes can be used right out of the box with a variety of powerful and convenient algorithms, including trimming, splitting, and finding/replacing all matches within a string. + +These algorithms are discussed in [String Processing Algorithms][pitches]. + + +### Unicode handling + +A regex describes an algorithm to be ran over some model of string, and Swift's `String` has a rather unique Unicode-forward model. `Character` is an [extended grapheme cluster](https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) and equality is determined under [canonical equivalence](https://www.unicode.org/reports/tr15/#Canon_Compat_Equivalence). + +Calling `dropFirst()` will not drop a leading byte or `Unicode.Scalar`, but rather a full `Character`. Similarly, a `.` in a regex will match any extended grapheme cluster. A regex will match canonical equivalents by default, strengthening the connection between regex and the equivalent `String` operations. + +Additionally, word boundaries (`\b`) follow [UTS\#29 Word Boundaries](https://www.unicode.org/reports/tr29/#Word_Boundaries). Contractions ("don't") are correctly detected and script changes are separated, without incurring significant binary size costs associated with language dictionaries. + +Regex targets [UTS\#18 Level 2](https://www.unicode.org/reports/tr18/#Extended_Unicode_Support) by default, but provides options to switch to scalar-level processing as well as compatibility character classes. Detailed rules on how we infer necessary grapheme cluster breaks inside regexes, as well as options and other concerns, are discussed in [Unicode for String Processing][pitches]. + + +## Detailed design + +```swift +/// A regex represents a string processing algorithm. +/// +/// let regex = try Regex("a(.*)b") +/// let match = "cbaxb".firstMatch(of: regex) +/// print(match.0) // "axb" +/// print(match.1) // "x" +/// +public struct Regex { + /// Match a string in its entirety. + /// + /// Returns `nil` if no match and throws on abort + public func wholeMatch(in s: String) throws -> Regex.Match? + + /// Match part of the string, starting at the beginning. + /// + /// Returns `nil` if no match and throws on abort + public func prefixMatch(in s: String) throws -> Regex.Match? + + /// Find the first match in a string + /// + /// Returns `nil` if no match is found and throws on abort + public func firstMatch(in s: String) throws -> Regex.Match? + + /// Match a substring in its entirety. + /// + /// Returns `nil` if no match and throws on abort + public func wholeMatch(in s: Substring) throws -> Regex.Match? + + /// Match part of the string, starting at the beginning. + /// + /// Returns `nil` if no match and throws on abort + public func prefixMatch(in s: Substring) throws -> Regex.Match? + + /// Find the first match in a substring + /// + /// Returns `nil` if no match is found and throws on abort + public func firstMatch(in s: Substring) throws -> Regex.Match? + + /// The result of matching a regex against a string. + /// + /// A `Match` forwards API to the `Output` generic parameter, + /// providing direct access to captures. + @dynamicMemberLookup + public struct Match { + /// The range of the overall match + public var range: Range { get } + + /// The produced output from the match operation + public var output: Output { get } + + /// Lookup a capture by name or number + public subscript(dynamicMember keyPath: KeyPath) -> T { get } + + /// Lookup a capture by number + @_disfavoredOverload + public subscript( + dynamicMember keyPath: KeyPath<(Output, _doNotUse: ()), Output> + ) -> Output { get } + // Note: this allows `.0` when `Match` is not a tuple. + + } +} +``` + +*Note*: The below are covered by other proposals, but listed here to help round out intuition. + +```swift + +// Result builder interfaces +extension Regex: RegexComponent { + public var regex: Regex { self } + + /// Result builder interface + public init( + @RegexComponentBuilder _ content: () -> Content + ) where Content.Output == Output + +} +extension Regex.Match { + /// Lookup a capture by reference + public subscript(_ reference: Reference) -> Capture +} + +// Run-time compilation interfaces +extension Regex { + /// Parse and compile `pattern`, resulting in a strongly-typed capture list. + public init(_ pattern: String, as: Output.Type = Output.self) throws +} +extension Regex where Output == AnyRegexOutput { + /// Parse and compile `pattern`, resulting in an existentially-typed capture list. + public init(_ pattern: String) throws +} +``` + +### Cancellation + +Regex is somewhat different from existing standard library operations in that regex processing can be a long-running task. +For this reason regex algorithms may check if the parent task has been cancelled and end execution. + +### On severability and related proposals + +The proposal split presented is meant to aid focused discussion, while acknowledging that each is interconnected. The boundaries between them are not completely cut-and-dry and could be refined as they enter proposal phase. + +Accepting this proposal in no way implies that all related proposals must be accepted. They are severable and each should stand on their own merit. + +## Source compatibility + +Everything in this proposal is additive. Regex delimiters may have their own source compatibility impact, which is discussed in that proposal. + +## Effect on ABI stability + +Everything in this proposal is additive. Run-time strings containing regex syntax are represented in the ABI as strings. For this initial release, literals are strings in the ABI as well (they get re-parsed at run time), which avoids baking an intermediate representation into Swift's ABI as we await better static compilation support (see future work). + +## Effect on API resilience + +N/A + +## Alternatives considered + + +### Regular expressions are a blight upon computing! + +"I had one problem so I wrote a regular expression, now I have two problems!" + +Regular expressions have a deservedly mixed reputation, owing to their historical baggage and treatment as a completely separate tool or subsystem. Despite this, they still occupy an important place in string processing. We are proposing the "regexiest regex", allowing them to shine at what they're good at and providing mitigations and off-ramps for their downsides. + +* "Regular expressions are bad because you should use a real parser" + - In other systems, you're either in or you're out, leading to a gravitational pull to stay in when... you should get out + - Our remedy is interoperability with real parsers via `CustomMatchingRegexComponent` + - Literals with refactoring actions provide an incremental off-ramp from regex syntax to result builders and real parsers +* "Regular expressions are bad because ugly unmaintainable syntax" + - We propose literals with source tools support, allowing for better syntax highlighting and analysis + - We propose result builders and refactoring actions from literals into result builders +* "Regular expressions are bad because Unicode" + - We propose a modern Unicode take on regexes + - We treat regexes as algorithms to be ran over some model of String, like's Swift's default Character-based view. +* "Regular expressions are bad because they're not powerful enough" + - Engine is general-purpose enough to support recursive descent parsers with captures, back-references, and lookahead + - We're proposing a regexy presentation on top of more powerful functionality +* "Regular expressions are bad because they're too powerful" + - We provide possessive quantifications, atomic groups, etc., all the normal ways to prune backtracking + - We provide clear semantics of how alternation works as ordered-choice, allowing for understandable execution + - Pathological behavior is ultimately a run-time concern, better handled by engine limiters (future work) + - Optimization is better done as a compiler problem, e.g. static compilation to DFAs (future work) + - Formal treatment of power is better done by other presentations, like PEGs and linear automata (future work) + + + +### Alternative names + +The generic parameter to `Regex` is `Output` and the erased version is `AnyRegexOutput`. This is... fairly generic sounding. + +An alternative could be `Captures`, doubling down on the idea that the entire match is implicitly capture `0`, but that can make describing and understanding how captures combine in the result builder harder to reason through (i.e. a formal distinction between explicit and implicit captures). + +An earlier prototype used the name `Match` for the generic parameter, but that quickly got confusing with all the match methods and was confusing with the result of a match operation (which produces the output, but isn't itself the generic parameter). We think `Match` works better as the result of a match operation. + + +### What's with all the `String(...)` initializer calls at use sites? + +We're working on how to eliminate these, likely by having API to access ranges, slices, or copies of the captured text. + +We're also looking for more community discussion on what the default type system and API presentation should be. As pitched, `Substring` emphasizes that we're referring to slices of the original input, with strong sharing connotations. + +The actual `Match` struct just stores ranges: the `Substrings` are lazily created on demand. This avoids unnecessary ARC traffic and memory usage. + + +### `Regex` instead of `Regex` + +The generic parameter `Output` is proposed to contain both the whole match (the `.0` element if `Output` is a tuple) and captures. One alternative we have considered is separating `Output` into the entire match and the captures, i.e. `Regex`, and using `Void` for for `Captures` when there are no captures. + +The biggest issue with this alternative design is that the numbering of `Captures` elements misaligns with the numbering of captures in textual regexes, where backreference `\0` refers to the entire match and captures start at `\1`. This design would sacrifice familarity and have the pitfall of introducing off-by-one errors. + +### Encoding `Regex`es into the type system + +During the initial review period the following comment was made: + +> I think the goal should be that, at least for regex literals (and hopefully for the DSL to some extent), one day we might not even need a bytecode or interpreter. I think the ideal case is if each literal was its own function or type that gets generated and optimised as if you wrote it in Swift. + +This is an approach that has been tried a few times in a few different languages (including by a few members of the Swift Standard Library and Core teams), and while it can produce attractive microbenchmarks, it has almost always proved to be a bad idea at the macro scale. In particular, even if we set aside witness tables and other associated swift generics overhead, optimizing a fixed pipeline for each pattern you want to match causes significant codesize expansion when there are multiple patterns in use, as compared to a more flexible byte code interpreter. A bytecode interpreter makes better use of instruction caches and memory, and can also benefit from micro architectural resources that are shared across different patterns. There is a tradeoff w.r.t. branch prediction resources, where separately compiled patterns may have more decisive branch history data, but a shared bytecode engine has much more data to use; this tradeoff tends to fall on the side of a bytecode engine, but it does not always do so. + +It should also be noted that nothing prevents AOT or JIT compiling of the bytecode if we believe it will be advantageous, but compiling or interpreting arbitrary Swift code at runtime is rather more unattractive, since both the type system and language are undecidable. Even absent this rationale, we would probably not encode regex programs directly into the type system simply because it is unnecessarily complex. + +### Future work: static optimization and compilation + +Swift's support for static compilation is still developing, and future work here is leveraging that to compile regex when profitable. Many regex describe simple [DFAs](https://en.wikipedia.org/wiki/Deterministic_finite_automaton) and can be statically compiled into very efficient programs. Full static compilation needs to be balanced with code size concerns, as a matching-specific bytecode is typically far smaller than a corresponding program (especially since the bytecode interpreter is shared). + +Regex are compiled into an intermediary representation and fairly simple analysis and optimizations are high-value. This compilation currently happens at run time (as such the IR is not ABI), but more of this could happen at compile time to save load/compilation time of the regex itself. Ideally, this representation would be shared along the fully-static compilation path and can be encoded in the ABI as a compact bytecode. + + +### Future work: parser combinators + +What we propose here is an incremental step towards better parsing support in Swift using parser-combinator style libraries. The underlying execution engine supports recursive function calls and mechanisms for library extensibility. `CustomMatchingRegexComponent`'s protocol requirement is effectively a [monadic parser](https://homepages.inf.ed.ac.uk/wadler/papers/marktoberdorf/baastad.pdf), meaning `Regex` provides a regex-flavored combinator-like system. + +An issues with traditional parser combinator libraries are the compilation barriers between call-site and definition, resulting in excessive and overly-cautious backtracking traffic. These can be eliminated through better [compilation techniques](https://core.ac.uk/download/pdf/148008325.pdf). As mentioned above, Swift's support for custom static compilation is still under development. + +Future work is a parser combinator system which leverages tiered static compilation and presents a parser-flavored approach, such as limited backtracking by default and more heavily interwoven recursive calls. + + +### Future work: `Regex`-backed enums + +Regexes are often used for tokenization and tokens can be represented with Swift enums. Future language integration could include `Regex` backing somewhat analogous to `RawRepresentable` enums. A Regex-backed enum could conform to `RegexComponent` producing itself upon a match by forming an ordered choice of its cases. + + + +[pitches]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md diff --git a/proposals/0351-regex-builder.md b/proposals/0351-regex-builder.md new file mode 100644 index 0000000000..41a39eae1d --- /dev/null +++ b/proposals/0351-regex-builder.md @@ -0,0 +1,1936 @@ +# Regex builder DSL + +* Proposal: [SE-0351](0351-regex-builder.md) +* Authors: [Richard Wei](https://github.com/rxwei), [Michael Ilseman](https://github.com/milseman), [Nate Cook](https://github.com/natecook1000), [Alejandro Alonso](https://github.com/azoy) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Implementation: [apple/swift-experimental-string-processing](https://github.com/apple/swift-experimental-string-processing/tree/main/Sources/RegexBuilder) + * Available in nightly toolchain snapshots with `import _StringProcessing` +* Status: **Implemented (Swift 5.7)** +* Review: ([pitch](https://forums.swift.org/t/pitch-regex-builder-dsl/56007)) + ([first review](https://forums.swift.org/t/se-0351-regex-builder-dsl/56531)) + ([revision](https://forums.swift.org/t/returned-for-revision-se-0351-regex-builder-dsl/57224)) + ([second review](https://forums.swift.org/t/se-0351-second-review-regex-builder-dsl/58721)) + ([acceptance](https://forums.swift.org/t/accepted-se-0351-regex-builder-dsl/58972)) + +**Table of Contents** +- [Regex builder DSL](#regex-builder-dsl) + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [`RegexComponent` protocol](#regexcomponent-protocol) + - [Concatenation](#concatenation) + - [Capture](#capture) + - [Mapping Output](#mapping-output) + - [Reference](#reference) + - [Alternation](#alternation) + - [Repetition](#repetition) + - [Repetition behavior](#repetition-behavior) + - [Anchors and Lookaheads](#anchors-and-lookaheads) + - [Subpattern](#subpattern) + - [Scoping](#scoping) + - [Composability](#composability) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Future directions](#future-directions) + - [Conversion to textual regex](#conversion-to-textual-regex) + - [Recursive subpatterns](#recursive-subpatterns) + - [Alternatives considered](#alternatives-considered) + - [Operators for quantification and alternation](#operators-for-quantification-and-alternation) + - [Postfix `capture` and `tryCapture` methods](#postfix-capture-and-trycapture-methods) + - [Unify quantifiers under `Repeat`](#unify-quantifiers-under-repeat) + - [Free functions instead of types](#free-functions-instead-of-types) + - [Support `buildOptional` and `buildEither`](#support-buildoptional-and-buildeither) + - [Flatten optionals](#flatten-optionals) + - [Structured rather than flat captures](#structured-rather-than-flat-captures) + - [Unify `Capture` with `TryCapture`](#unify-capture-with-trycapture) + +## Introduction + +[Declarative string processing] aims to offer powerful pattern matching capabilities with expressivity, clarity, type safety, and ease of use. To achieve this, we propose to introduce a result-builder-based DSL, **regex builder**, for creating and composing regular expressions (**regex**es). + +Regex builder is part of the Swift Standard Library but resides in a standalone module named `RegexBuilder`. By importing `RegexBuilder`, you get all necessary API for building a regex. + +```swift +import RegexBuilder + +let emailPattern = Regex { + let word = OneOrMore(.word) + Capture { + ZeroOrMore { + word + "." + } + word + } + "@" + Capture { + word + OneOrMore { + "." + word + } + } +} // => Regex<(Substring, Substring, Substring)> + +let email = "My email is my.name@mail.swift.org." +if let match = try emailPattern.firstMatch(in: email) { + let (wholeMatch, name, domain) = match.output + // wholeMatch: "my.name@mail.swift.org" + // name: "my.name" + // domain: "mail.swift.org" +} +``` + +This proposal introduces all core API for creating and composing regexes that echos the textual [regex syntax] and [strongly typed regex captures], but does not formally specify the matching semantics or define character classes. + +## Motivation + +Regex is a fundamental and powerful tool for textual pattern matching. It is a domain-specific language often expressed as text. For example, given the following bank statement: + +``` +CREDIT 04062020 PayPal transfer $4.99 +CREDIT 04032020 Payroll $69.73 +DEBIT 04022020 ACH transfer $38.25 +DEBIT 03242020 IRS tax payment $52249.98 +``` + +One can write the follow textual regex to match each line: + +``` +(CREDIT|DEBIT)\s+(\d{2}\d{2}\d{4})\s+([\w\s]+\w)\s+(\$\d+\.\d{2}) +``` + +While a regex like this is very compact and expressive, it is very difficult read, write and use: + +1. Syntactic special characters, e.g. `\`, `(`, `[`, `{`, are too dense to be readable. +2. It contains a hierarchy of subpatterns fit into a single line of text. +3. No code completion when typing syntactic components. +4. Capturing groups produce raw data (i.e. a range or a substring) and can only be converted to other data structures after matching. +5. While comments `(?#...)` can be added inline, it only complicates readability. + +## Proposed solution + +We introduce regex builder, a result-builder-based API for creating and composing regexes. This API resides in a new module named `RegexBuilder` that is to be shipped as part of the Swift toolchain. + +With regex builder, the regex for matching a bank statement can be written as the following: + +```swift +import RegexBuilder + +enum TransactionKind: String { + case credit = "CREDIT" + case debit = "DEBIT" +} + +struct Date { + var month, day, year: Int + init?(mmddyyyy: String) { ... } +} + +struct Amount { + var valueTimes100: Int + init?(twoDecimalPlaces text: Substring) { ... } +} + +let statementPattern = Regex { + // Parse the transaction kind. + TryCapture { + ChoiceOf { + "CREDIT" + "DEBIT" + } + } transform: { + TransactionKind(rawValue: String($0)) + } + OneOrMore(.whitespace) + // Parse the date, e.g. "01012021". + TryCapture { + Repeat(.digit, count: 2) + Repeat(.digit, count: 2) + Repeat(.digit, count: 4) + } transform: { Date(mmddyyyy: $0) } + OneOrMore(.whitespace) + // Parse the transaction description, e.g. "ACH transfer". + Capture { + OneOrMore(CharacterClass(.word, .whitespace)) + CharacterClass.word + } transform: { String($0) } + OneOrMore(.whitespace) + "$" + // Parse the amount, e.g. `$100.00`. + TryCapture { + OneOrMore(.digit) + "." + Repeat(.digit, count: 2) + } transform: { Amount(twoDecimalPlaces: $0) } +} // => Regex<(Substring, TransactionKind, Date, String, Amount)> + + +let statement = """ + CREDIT 04062020 PayPal transfer $4.99 + CREDIT 04032020 Payroll $69.73 + DEBIT 04022020 ACH transfer $38.25 + DEBIT 03242020 IRS tax payment $52249.98 + """ +for match in statement.matches(of: statementPattern) { + let (line, kind, date, description, amount) = match.output + ... +} +``` + +Regex builder addresses all of textual regexes' shortcomings presented in the [Motivation](#motivation) section: +1. Capture groups and quantifiers are expressed as API calls that are easy to read. +2. Scoping and indentations clearly distinguish subpatterns in the hierarchy. +3. Code completion is available when the developer types an API call. +4. Capturing groups can be transformed into structured data at the regex declaration site. +5. Normal code comments can be written within a regex declaration to further improve readability. + +## Detailed design + +### `RegexComponent` protocol + +One of the goals of the regex builder DSL is allowing the developers to easily compose regexes from common currency types and literals, or even define custom patterns to use for matching. We introduce `RegexComponent` in the implicitly-imported `Swift` module, a protocol that unifies all types that can represent a component of a regex. Since regexes are composable, the `Regex` type itself conforms to `RegexComponent`. + +```swift +public protocol RegexComponent { + associatedtype RegexOutput + var regex: Regex { get } +} + +extension Regex: RegexComponent { + public typealias RegexOutput = Output + public var regex: Regex { self } +} +``` + +Note: +- `RegexComponent` and `Regex`'s conformance to `RegexComponent` are available without importing `RegexBuilder`. All other types and conformances introduced in this proposal are in the `RegexBuilder` module. +- The associated type `RegexOutput` intentionally has a `Regex` prefix. `Output` would cause confusion in standard library conforming types such as `String`, i.e. `String.Output`. + +By conforming standard library types to `RegexComponent`, we allow them to be used inside the regex builder DSL as a match target. These conformances are available in the `RegexBuilder` module. + +```swift +// A string represents a regex that matches the string. +extension String: RegexComponent { + public var regex: Regex { get } +} + +// A substring represents a regex that matches the substring. +extension Substring: RegexComponent { + public var regex: Regex { get } +} + +// A character represents a regex that matches the character. +extension Character: RegexComponent { + public var regex: Regex { get } +} + +// A unicode scalar represents a regex that matches the scalar. +extension UnicodeScalar: RegexComponent { + public var regex: Regex { get } +} + +// To be introduced in a future pitch. +extension CharacterClass: RegexComponent { + public var regex: Regex { get } +} +``` + +All of the regex builder DSL in the rest of this pitch will accept generic components that conform to `RegexComponent`. + +### Concatenation + +A regex can be viewed as a concatenation of smaller regexes. In the regex builder DSL, `RegexComponentBuilder` is the basic facility to allow developers to compose regexes by concatenation. + +```swift +@resultBuilder +public enum RegexComponentBuilder { ... } +``` + +A closure marked with `@RegexComponentBuilder` will be transformed to produce a `Regex` by concatenating all of its components, where the result type's `Output` type will be a `Substring` followed by concatenated captures (tuple when plural). + +> #### Recap: Regex capturing basics +> +> `Regex` is a generic type with generic parameter `Output`. +> +> ```swift +> struct Regex { ... } +> ``` +> +> When a regex does not contain any capturing groups, its `Output` type is `Substring`, which represents the whole matched portion of the input. +> +> ```swift +> let noCaptures = #/a/# // => Regex +> ``` +> +> When a regex contains capturing groups, i.e. `(...)`, the `Output` type is extended as a tuple to also contain *capture types*. Capture types are tuple elements after the first element. +> +> ```swift +> // ________________________________ +> // .0 | .0 | +> // ____________________ _________ +> let yesCaptures = #/a(?:(b+)c(d+))+e(f)?/# // => Regex<(Substring, Substring, Substring, Substring?)> +> // ---- ---- --- --------- --------- ---------- +> // .1 | .2 | .3 | .1 | .2 | .3 | +> // | | | | | | +> // | | |_______________________________ | ______ | ________| +> // | | | | +> // | |______________________________________ | ______ | +> // | | +> // |_____________________________________________| +> // ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +> // Capture types +> ``` + +We introduce a new initializer `Regex.init(_:)` which accepts a `@RegexComponentBuilder` closure. This initializer is the entry point for creating a regex using the regex builder DSL. + +```swift +extension Regex { + public init( + @RegexComponentBuilder _ content: () -> R + ) where R.RegexOutput == Output +} +``` + +Example: + +```swift +Regex { + regex0 // Regex + regex1 // Regex<(Substring, Int)> + regex2 // Regex<(Substring, Float)> + regex3 // Regex<(Substring, Substring)> +} // Regex<(Substring, Int, Float, Substring)> +``` + +This above regex will be transformed to: + +```swift +Regex { + let e0 = RegexComponentBuilder.buildExpression(regex0) // Regex + let e1 = RegexComponentBuilder.buildExpression(regex1) // Regex<(Substring, Int)> + let e2 = RegexComponentBuilder.buildExpression(regex2) // Regex<(Substring, Float)> + let e3 = RegexComponentBuilder.buildExpression(regex3) // Regex<(Substring, Substring)> + let r0 = RegexComponentBuilder.buildPartialBlock(first: e0) + let r1 = RegexComponentBuilder.buildPartialBlock(accumulated: r0, next: e1) + let r2 = RegexComponentBuilder.buildPartialBlock(accumulated: r1, next: e2) + let r3 = RegexComponentBuilder.buildPartialBlock(accumulated: r2, next: e3) + return r3 +} // Regex<(Substring, Int, Float, Substring)> +``` + +The following example creates a regex by concatenating subpatterns. + +```swift +let regex = Regex { + "regex builder " + "is " + "so easy" +} +let match = try regex.prefixMatch(in: "regex builder is so easy!") +match?.0 // => "regex builder is so easy" +``` + +
+API definition + +Basic methods in `RegexComponentBuilder`, e.g. `buildBlock()`, provides support for creating the most fundamental blocks. The `buildExpression` method wraps a user-provided component in a `RegexComponentBuilder.Component` structure, before passing the component to other builder methods. This is used for saving the source location of the component so that runtime errors can be reported with an accurate location. + +```swift +@resultBuilder +public enum RegexComponentBuilder { + /// Returns an empty regex. + public static func buildBlock() -> Regex + + /// A builder component that stores a regex component and its source location + /// for debugging purposes. + public struct Component { + public var value: Value + public var file: String + public var function: String + public var line: Int + public var column: Int + } + + /// Returns a component by wrapping the component regex in `Component` and + /// recording its source location. + public static func buildExpression( + _ regex: R, + file: String = #file, + function: String = #function, + line: Int = #line, + column: Int = #column + ) -> Component +} +``` + +`RegexComponentBuilder` utilizes `buildPartialBlock` to be able to concatenate all components' capture types to a single result tuple. `buildPartialBlock(first:)` provides support for creating a regex from a single component, and `buildPartialBlock(accumulated:next:)` support for creating a regex from multiple results. + +Before Swift supports variadic generics, `buildPartialBlock(accumulated:next:)` must be overloaded to support concatenating regexes of supported capture quantities (arities). It is overloaded up to `arity^2` times to account for all possible pairs of regexes that make up 10 captures. + +In the initial version of the DSL, we plan to support regexes with up to 10 captures, as 10 captures are sufficient for most use cases. These overloads can be superseded by variadic versions of `buildPartialBlock(first:)` and `buildPartialBlock(accumulated:next:)` in a future release. + +```swift +extension RegexComponentBuilder { + @_disfavoredOverload + public static func buildPartialBlock( + first r: Component + ) -> Regex + + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildPartialBlock< + // AccumulatedWholeMatch, NextWholeMatch, + // AccumulatedCapture..., NextCapture..., + // Accumulated: RegexComponent, Next: RegexComponent + // >( + // accumulated: Accumulated, next: Component + // ) -> Regex<(Substring, AccumulatedCapture..., NextCapture...)> + // where Accumulated.RegexOutput == (AccumulatedWholeMatch, AccumulatedCapture...), + // Next.RegexOutput == (NextWholeMatch, NextCapture...) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0, C1)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0, C1) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0, C1, C2)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0, C1, C2) + + // ... `O(arity^2)` overloads of `buildPartialBlock(accumulated:next:)` +} +``` + +To support `if #available(...)` statements, `buildLimitedAvailability(_:)` is defined with overloads to support up to 10 captures. The overload for non-capturing regexes, due to the lack of generic constraints, must be annotated with `@_disfavoredOverload` in order not shadow other overloads. We expect that a variadic-generic version of this method will eventually superseded all of these overloads. + +```swift +extension RegexComponentBuilder { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildLimitedAvailability< + // Component, WholeMatch, Capture... + // >( + // _ component: Component + // ) where Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?)> + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?, C1?)> + + // ... `O(arity)` overloads of `buildLimitedAvailability(_:)` +} +``` + +`buildOptional` and `buildEither` are intentionally not supported due to ergonomic issues and fundamental semantic differences between regex conditionals and result builder conditionals. Please refer to the [alternatives considered](#support-buildoptional-and-buildeither) section for detailed rationale. + +
+ +### Capture + +Capture is a common regex feature that saves a portion of the input upon match. In regex builder, `Capture` and `TryCapture` are regex components that produce a new regex by inserting the captured pattern's whole match (`.0`) to the `.1` position of `RegexOutput`. When a transform closure is provided, the whole match (`.0`) of the captured content will be transformed to using the closure. + +```swift +public struct Capture: RegexComponent { ... } +public struct TryCapture: RegexComponent { ... } +``` + +To do a simple capture, you provide `Capture` with a regex component or a regex component builder closure. + +```swift +// Equivalent: '(CREDIT|DEBIT)' +Capture { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} // `.RegexOutput == (Substring, Substring)` +``` + +A capture will be represented in the type signature as a slice of the input, i.e. `Substring`. To transform the captured substring into another value during matching, specify a `transform:` closure. + +```swift +// This example is similar to the one above, however in this example we +// transform the result of the capture into: +// "Transaction Kind: CREDIT" or "Transaction Kind: DEBIT" +Capture { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} transform: { + "Transaction Kind: \($0)" +} // `.RegexOutput == (Substring, String)` +``` + +The transform closure can throw. When a transform closure throws during matching, the matching will abort and the error will be propagated directly to the top-level matching API that's being called, e.g. `Regex.wholeMatch(in:)` and `Regex.prefixMatch(in:)`. Aborting is useful for cases where you know that matching can never succeed or when you detect that an important invariant has been violated and the matching procedure needs to be aborted. + +An alternative version of capture is called `TryCapture`, which works in cases where you want to transform the capture, but the transformation may return nil. When a nil is returned, the regex engine backtracks and tries an alternative. For example, `TryCapture` makes it easy to directly transform a capture by calling a failable initializer during matching. + +```swift +enum TransactionKind: String { + case credit = "CREDIT" + case debit = "DEBIT" +} + +TryCapture { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} transform: { + // This initializer may return nil which is why we used TryCapture. + TransactionKind(rawValue: String($0)) +} +``` + +
+API definition + +```swift +public struct Capture: RegexComponent { + public var regex: Regex { get } +} + +public struct TryCapture: RegexComponent { + public var regex: Regex { get } +} +``` + +Below are `Capture` and `TryCapture` initializer variants on capture arity 0. Higher capture arities are omitted for simplicity. + +```swift +extension Capture { + public init( + _ component: R + ) where Output == (Substring, W), R.RegexOutput == W + + public init( + _ component: R, as reference: Reference + ) where Output == (Substring, W), R.RegexOutput == W + + public init( + _ component: R, + transform: @Sendable @escaping (W) throws -> NewCapture + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + @RegexComponentBuilder _ component: () -> R + ) where Output == (Substring, W), R.RegexOutput == W + + // ... `O(arity)` overloads +} + +extension TryCapture { + public init( + _ component: R, + transform: @Sendable @escaping (W) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + @RegexComponentBuilder _ component: () -> R, + transform: @Sendable @escaping (W) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + // ... `O(arity)` overloads +} +``` + +
+ +### Mapping Output + +In addition to transforming individual captures within a regex, you can also map the output of an entire regex to a different output type. You can use the `mapOutput(_:)` methods to reorder captures, flatten nested optionals, or create instances of a custom type. + +This example shows how you can transform the output of a regex with three capture groups into an instance of a custom `SemanticVersion` type, matching strings such as `"1.0.0"` or `"1.0"`: + +```swift +struct SemanticVersion: Hashable { + var major, minor, patch: Int +} + +let semverRegex = Regex { + TryCapture(OneOrMore(.digit)) { Int($0) } + "." + TryCapture(OneOrMore(.digit)) { Int($0) } + Optionally { + "." + TryCapture(OneOrMore(.digit)) { Int($0) } + } +}.mapOutput { _, c1, c2, c3 in + SemanticVersion(major: c1, minor: c2, patch: c3 ?? 0) +} + +let semver1 = "1.11.4".firstMatch(of: semverRegex)?.output +// semver1 == SemanticVersion(major: 1, minor: 11, patch: 4) +let semver2 = "0.6".firstMatch(of: semverRegex)?.output +// semver2 == SemanticVersion(major: 0, minor: 6, patch: 0) +``` + +
+API definition + +Note: This extension is defined in the standard library, not the `RegexBuilder` module. + +```swift +extension Regex { + /// Returns a regex that transforms its matches using the given closure. + /// + /// When you call `mapOutput(_:)` on a regex, you change the type of + /// output available on each match result. The `body` closure is called + /// when each match is found to transform the result of the match. + /// + /// - Parameter body: A closure for transforming the output of this + /// regex. + /// - Returns: A regex that has `NewOutput` as its output type. + func mapOutput(_ body: @escaping (Output) -> NewOutput) -> Regex +} +``` +
+ +### Reference + +Reference is a feature that can be used to achieve named captures and named backreferences from textual regexes. Simply state what type the reference will hold on to and you can use it later once you've matched a string to get back a specific capture. Note the type you pass to reference will be whatever the result of a capture's transform is. A capture with no transform always has a reference type of `Substring`. + +```swift +let kind = Reference(Substring.self) + +let regex = Capture(as: kind) { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} + +let input = "CREDIT" +if let result = try regex.firstMatch(in: input) { + print(result[kind]) // Optional("CREDIT") +} +``` + +Capturing stores the most recently captured content, and references can be used as a name to look up the result of matching. The reference itself can also be used within a regex (commonly called a "backreference") to match the most recently captured content during matching. + +```swift +let a = Reference(Substring.self) +let b = Reference(Substring.self) +let c = Reference(Substring.self) +let regex = Regex { + Capture("abc", as: a) + Capture("def", as: b) + ZeroOrMore { + Capture("hij", as: c) + } + a + Capture(b) +} + +if let result = try regex.firstMatch(in: "abcdefabcdef") { + print(result[a]) // => Optional("abc") + print(result[b]) // => Optional("def") + print(result[c]) // => nil +} +``` + +A regex is considered invalid when it contains a use of reference without it ever being used as the `as:` argument to an initializer of `Capture` or `TryCapture` in the regex. When this occurs in the regex builder DSL, a runtime error will be reported. + +Similarly, the argument to a `Regex.Match.subscript(_:)` must have been used as the `as:` argument to an initializer of `Capture` or `TryCapture` in the regex that produced the match. + +
+API definition + +```swift +/// A reference to a regex capture. +public struct Reference: RegexComponent { + public init(_ captureType: Capture.Type = Capture.self) + public var regex: Regex +} + +extension Capture { + public init( + _ component: R, + as reference: Reference, + transform: @escaping (Substring) throws -> NewCapture + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + as reference: Reference, + @RegexComponentBuilder _ component: () -> R + ) where Output == (Substring, W), R.RegexOutput == W + + // ... `O(arity)` overloads +} + +extension TryCapture { + public init( + _ component: R, + as reference: Reference, + transform: @escaping (Substring) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + as reference: Reference, + @RegexComponentBuilder _ component: () -> R, + transform: @escaping (Substring) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + // ... `O(arity)` overloads +} + +extension Regex.Match { + /// Returns the capture referenced by the given reference. + /// + /// - Precondition: The reference must have been captured in the regex that produced this match. + public subscript(_ reference: Reference) -> Capture? { get } +} +``` + +
+ +### Alternation + +An alternation is used to match one of multiple patterns. When one pattern in an alternation does not match successfully, the regex engine tries the next pattern until there's a successful match. An alternation wraps its underlying patterns' capture types in an `Optional` and concatenates them together, first to last. + +```swift +let choice = ChoiceOf { + regex0 // Regex + regex1 // Regex<(Substring, Int)> + regex2 // Regex<(Substring, Float)> + regex3 // Regex<(Substring, Substring)> +} // => Regex<(Substring, Int?, Float?, Substring?)> +``` + +`AlternationBuilder` is a result builder type for creating alternations from components of a block. + +```swift +@resultBuilder +public struct AlternationBuilder { ... } +``` + +To the developer, the top-level API is a type named `ChoiceOf`. This type has an initializer that accepts an `@AlternationBuilder` closure. + +```swift +public struct ChoiceOf: RegexComponent { + ... + public init( + @AlternationBuilder builder: () -> R + ) where R.RegexOutput == Output +} +``` + +For example, the following code creates an alternation of two subpatterns. + +```swift +let regex = Regex { + ChoiceOf { + "CREDIT" + "DEBIT" + } +} +let match = try regex.prefixMatch(in: "DEBIT 04032020 Payroll $69.73") +match?.0 // => "DEBIT" +``` + +
+API definition + +`AlternationBuilder` is mostly similar to `RegexComponent` with the following distinctions: +- Empty blocks are not supported. +- Capture types are wrapped in a layer of `Optional` before being concatenated in the resulting `Output` type. +- `buildEither(first:)` and `buildEither(second:)` are overloaded for each supported capture arity because they need to wrap capture types in `Optional`. + +```swift +public struct ChoiceOf: RegexComponent { + public var regex: Regex { get } + public init( + @AlternationBuilder builder: () -> R + ) where R.RegexOutput == Output +} + +@resultBuilder +public enum AlternationBuilder { + public typealias Component = RegexComponentBuilder.Component + + /// Returns a component by wrapping the component regex in `Component` and + /// recording its source location. + public static func buildExpression( + _ regex: R, + file: String = #file, + function: String = #function, + line: Int = #line, + column: Int = #column + ) -> Component + + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildPartialBlock< + // R, WholeMatch, Capture... + // >( + // first component: Component + // ) -> Regex<(Substring, Capture?...)> + // where Component.RegexOutput == (WholeMatch, Capture...), + + @_disfavoredOverload + public static func buildPartialBlock( + first r: Component + ) -> Regex + + public static func buildPartialBlock( + first r: Component + ) -> Regex<(Substring, C0?)> where R.RegexOutput == (W, C0) + + public static func buildPartialBlock( + first r: Component + ) -> Regex<(Substring, C0?, C1?)> where R.RegexOutput == (W, C0, C1) + + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildPartialBlock< + // AccumulatedWholeMatch, NextWholeMatch, + // AccumulatedCapture..., NextCapture..., + // Accumulated: RegexComponent, Next: RegexComponent + // >( + // accumulated: Accumulated, next: Component + // ) -> Regex<(Substring, AccumulatedCapture..., NextCapture...)> + // where Accumulated.RegexOutput == (AccumulatedWholeMatch, AccumulatedCapture...), + // Next.RegexOutput == (NextWholeMatch, NextCapture...) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0?)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0?, C1?)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0, C1) + + public static func buildPartialBlock( + accumulated: R0, next: Component + ) -> Regex<(Substring, C0?, C1?, C2?)> where R0.RegexOutput == W0, R1.RegexOutput == (W1, C0, C1, C2) + + // ... `O(arity^2)` overloads of `buildPartialBlock(accumulated:next:)` +} + +extension AlternationBuilder { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single method: + // + // public static func buildLimitedAvailability< + // Component, WholeMatch, Capture... + // >( + // _ component: Component + // ) -> Regex<(Substring, Capture?...)> + // where Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?)> + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?, C1?)> + + // ... `O(arity)` overloads of `buildLimitedAvailability(_:)` + + public static func buildLimitedAvailability( + _ component: Component + ) -> Regex<(Substring, C0?, C1?, C2?, C3?, C4?, C5?, C6?, C7?, C8, C9?)> where R.RegexOutput == (W, C0, C1, C2, C3, C4, C5, C6, C7, C8, C9) +} +``` + +
+ +### Repetition + +One of the most useful features of regex is repetition, aka. quantification, as it allows you to match a specific range of number of occurrences of a subpattern. Regex builder provides 5 repetition components: `One`, `OneOrMore`, `ZeroOrMore`, `Optionally`, and `Repeat`. + +```swift +public struct One: RegexComponent { ... } +public struct OneOrMore: RegexComponent { ... } +public struct ZeroOrMore: RegexComponent { ... } +public struct Optionally: RegexComponent { ... } +public struct Repeat: RegexComponent { ... } +``` + +| Repetition in regex builder | Textual regex equivalent | +|-----------------------------|--------------------------| +| `One(...)` | `...` | +| `OneOrMore(...)` | `...+` | +| `ZeroOrMore(...)` | `...*` | +| `Optionally(...)` | `...?` | +| `Repeat(..., count: n)` | `...{n}` | +| `Repeat(..., n...)` | `...{n,}` | +| `Repeat(..., n...m)` | `...{n,m}` | + +`One`, `OneOrMore` and count-based `Repeat` are quantifiers that produce a new regex with the original capture types. Their `Output` type is `Substring` followed by the component's capture types. `ZeroOrMore`, `Optionally`, and range-based `Repeat` are quantifiers that produce a new regex with optional capture types. Their `Output` type is `Substring` followed by the component's capture types wrapped in `Optional`. + +| Quantifier | Component `Output` | Result `Output` | +|------------------------------------------------------|----------------------------|----------------------------| +| `One`
`OneOrMore`
`Repeat(..., count: ...)` | `(WholeMatch, Capture...)` | `(Substring, Capture...)` | +| `One`
`OneOrMore`
`Repeat(..., count: ...)` | `WholeMatch` (non-tuple) | `Substring` | +| `ZeroOrMore`
`Optionally`
`Repeat(..., n...m)` | `(WholeMatch, Capture...)` | `(Substring, Capture?...)` | +| `ZeroOrMore`
`Optionally`
`Repeat(..., n...m)` | `WholeMatch` (non-tuple) | `Substring` | + +
+API definition + +```swift +public struct One: RegexComponent { + public var regex: Regex { get } +} + +public struct OneOrMore: RegexComponent { + public var regex: Regex { get } +} + +public struct ZeroOrMore: RegexComponent { + public var regex: Regex { get } +} + +public struct Optionally: RegexComponent { + public var regex: Regex { get } +} + +public struct Repeat: RegexComponent { + public var regex: Regex { get } +} +``` + +Due to the lack of variadic generics, initializers must be overloaded for every supported capture arity. + +```swift +extension One { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // _ behavior: RegexRepetitionBehavior = .eager + // ) + // where Output == (Substring, Capture...)>, + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ behavior: RegexRepetitionBehavior = .eager, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture...), + // Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring + + @_disfavoredOverload + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + // ... `O(arity)` overloads +} + +extension OneOrMore { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // _ behavior: RegexRepetitionBehavior = .eager + // ) + // where Output == (Substring, Capture...)>, + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ behavior: RegexRepetitionBehavior = .eager, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture...), + // Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring + + @_disfavoredOverload + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + // ... `O(arity)` overloads +} + +extension ZeroOrMore { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // _ behavior: RegexRepetitionBehavior = nil + // ) + // where Output == (Substring, Capture?...)>, + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ behavior: RegexRepetitionBehavior? = nil, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture?...), + // Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring + + @_disfavoredOverload + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == (Substring, C0?), Component.RegexOutput == (W, C0) + + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0?), Component.RegexOutput == (W, C0) + + // ... `O(arity)` overloads +} + +extension Optionally { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // _ behavior: RegexRepetitionBehavior? = nil + // ) + // where Output == (Substring, Capture?...), + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ behavior: RegexRepetitionBehavior? = nil, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture?...)>, + // Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring + + @_disfavoredOverload + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + _ component: Component, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == (Substring, C0?), Component.RegexOutput == (W, C0) + + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0?), Component.RegexOutput == (W, C0) + + // ... `O(arity)` overloads +} + +extension Repeat { + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // _ component: Component, + // count: Int, + // _ behavior: RegexRepetitionBehavior? = nil + // ) + // where Output == (Substring, Capture...), + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture... + // >( + // count: Int, + // _ behavior: RegexRepetitionBehavior? = nil, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture...), + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture..., RE: RangeExpression + // >( + // _ component: Component, + // _ expression: RE, + // _ behavior: RegexRepetitionBehavior? = nil + // ) + // where Output == (Substring, Capture?...), + // Component.RegexOutput == (WholeMatch, Capture...) + // + // public init< + // Component: RegexComponent, WholeMatch, Capture..., RE: RangeExpression + // >( + // _ expression: RE, + // _ behavior: RegexRepetitionBehavior? = nil, + // @RegexComponentBuilder _ component: () -> Component + // ) + // where Output == (Substring, Capture?...), + // Component.RegexOutput == (WholeMatch, Capture...) + + // Nullary + + @_disfavoredOverload + public init( + _ component: Component, + count: Int, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring, R.Bound == Int + + @_disfavoredOverload + public init( + count: Int, + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring, R.Bound == Int + + @_disfavoredOverload + public init( + _ component: Component, + _ expression: RE, + _ behavior: RegexRepetitionBehavior? = nil + ) where Output == Substring, R.Bound == Int + + @_disfavoredOverload + public init( + _ expression: RE, + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring, R.Bound == Int + + + // Unary + + public init( + _ component: Component, + count: Int, + _ behavior: RegexRepetitionBehavior? = nil + ) + where Output == (Substring, C0), + Component.RegexOutput == (Substring, C0), + R.Bound == Int + + public init( + count: Int, + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) + where Output == (Substring, C0), + Component.RegexOutput == (Substring, C0), + R.Bound == Int + + public init( + _ component: Component, + _ expression: RE, + _ behavior: RegexRepetitionBehavior? = nil + ) + where Output == (Substring, C0?), + Component.RegexOutput == (W, C0), + R.Bound == Int + + public init( + _ expression: RE, + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) + where Output == (Substring, C0?), + Component.RegexOutput == (W, C0), + R.Bound == Int + + // ... `O(arity)` overloads +} +``` + +
+ +#### Repetition behavior + +Repetition behavior defines how eagerly a repetition component should match the input. Behavior can be unspecified, in which case it will default to `.eager` unless an option is provided to change the default (see [Unicode for String Processing](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md#unicode-for-string-processing)). + +```swift +/// Specifies how much to attempt to match when using a quantifier. +public struct RegexRepetitionBehavior { + /// Match as much of the input string as possible, backtracking when + /// necessary. + public static var eager: RegexRepetitionBehavior { get } + + /// Match as little of the input string as possible, expanding the matched + /// region as necessary to complete a match. + public static var reluctant: RegexRepetitionBehavior { get } + + /// Match as much of the input string as possible, performing no backtracking. + public static var possessive: RegexRepetitionBehavior { get } +} +``` + +| Repetition behavior in regex builder | Textual regex equivalent | +|--------------------------------------|--------------------------| +| `.eager` | no suffix | +| `.reluctant` | suffix `?` | +| `.possessive` | suffix `+` | + +To demonstrate how each repetition behavior works, let's look at the following +example. Suppose we want to make a regex that wants to capture an html tag, e.g. +``. We might start with something like the following: + + +```swift +let tag = Reference(Substring.self) + +let htmlRegex = Regex { + "<" + Capture(as: tag) { + // Remember, the default behavior is .eager here! + OneOrMore(.any) + } + ">" +} + +let input = #"print("hello world!")"# + +if let result = htmlRegex.firstMatch(in: input) { + print(result[tag]) +} +``` + +The code above prints `code>print("hello world!")"` because our string was out of characters. This is intended for `.possessive` because it doesn't backtrack the string to find a match for the ending `">"`. + +The desired behavior in this case is `.reluctant`, where the repetition will match as little of the input string as possible. If we use `OneOrMore(.any, .reluctant)`, the code prints expected output ``. + +### Anchors and Lookaheads + +Anchors are a way to constrain a regex, or part of a regex, to matching particular locations within an input string. Regex builder provides anchors that correspond to regex syntax anchors. Regex builder also provides two types that represent look-ahead assertions — essentially a non-consuming sub-regex that has to match (or not match) before the regex can proceed. + +```swift +/// A regex component that matches a specific condition at a particular position +/// in an input string. +/// +/// You can use anchors to guarantee that a match only occurs at certain points +/// in an input string, such as at the beginning of the string or at the end of +/// a line. +public struct Anchor: RegexComponent { + /// An anchor that matches at the start of a line, including the start of + /// the input string. + /// + /// This anchor is equivalent to `^` in regex syntax when the `m` option + /// has been enabled or `anchorsMatchLineEndings(true)` has been called. + public static var startOfLine: Anchor { get } + + /// An anchor that matches at the end of a line, including at the end of + /// the input string. + /// + /// This anchor is equivalent to `$` in regex syntax when the `m` option + /// has been enabled or `anchorsMatchLineEndings(true)` has been called. + public static var endOfLine: Anchor { get } + + /// An anchor that matches at a word boundary. + /// + /// Word boundaries are identified using the Unicode default word boundary + /// algorithm by default. To specify a different word boundary algorithm, + /// see the `RegexComponent.wordBoundaryKind(_:)` method. + /// + /// This anchor is equivalent to `\b` in regex syntax. + public static var wordBoundary: Anchor { get } + + /// An anchor that matches at the start of the input string. + /// + /// This anchor is equivalent to `\A` in regex syntax. + public static var startOfSubject: Anchor { get } + + /// An anchor that matches at the end of the input string. + /// + /// This anchor is equivalent to `\z` in regex syntax. + public static var endOfSubject: Anchor { get } + + /// An anchor that matches at the end of the input string or at the end of + /// the line immediately before the the end of the string. + /// + /// This anchor is equivalent to `\Z` in regex syntax. + public static var endOfSubjectBeforeNewline: Anchor { get } + + /// An anchor that matches at a grapheme cluster boundary. + /// + /// This anchor is equivalent to `\y` in regex syntax. + public static var textSegmentBoundary: Anchor { get } + + /// An anchor that matches at the first position of a match in the input + /// string. + /// + /// This anchor is equivalent to `\y` in regex syntax. + public static var firstMatchingPositionInSubject: Anchor { get } + + /// The inverse of this anchor, which matches at every position that this + /// anchor does not. + /// + /// For the `wordBoundary` and `textSegmentBoundary` anchors, the inverted + /// version corresponds to `\B` and `\Y`, respectively. + public var inverted: Anchor { get } +} + +/// A regex component that allows a match to continue only if its contents +/// match at the given location. +/// +/// A lookahead is a zero-length assertion that its included regex matches at +/// a particular position. Lookaheads do not advance the overall matching +/// position in the input string — once a lookahead succeeds, matching continues +/// in the regex from the same position. +public struct Lookahead: RegexComponent { + /// Creates a lookahead from the given regex component. + public init(_ component: some RegexComponent) + + /// Creates a lookahead from the regex generated by the given builder closure. + public init(@RegexComponentBuilder _ component: () -> some RegexComponent) +} + +/// A regex component that allows a match to continue only if its contents +/// do not match at the given location. +/// +/// A negative lookahead is a zero-length assertion that its included regex +/// does not match at a particular position. Lookaheads do not advance the +/// overall matching position in the input string — once a lookahead succeeds, +/// matching continues in the regex from the same position. +public struct NegativeLookahead: RegexComponent { + /// Creates a negative lookahead from the given regex component. + public init(_ component: some RegexComponent) + + /// Creates a negative lookahead from the regex generated by the given builder + /// closure. + public init(@RegexComponentBuilder _ component: () -> some RegexComponent) +} +``` + +### Subpattern + +In textual regex, one can refer to a subpattern to avoid duplicating the subpattern, for example: + +``` +(you|I) say (goodbye|hello); (?1) say (?2) +``` + +The above regex is equivalent to + +``` +(you|I) say (goodbye|hello); (you|I) say (goodbye|hello) +``` + +With regex builder, there is no special API required to reuse existing subpatterns, as a subpattern can be defined modularly using a `let` binding inside or outside a regex builder closure. + +```swift +Regex { + let subject = ChoiceOf { + "I" + "you" + } + let object = ChoiceOf { + "goodbye" + "hello" + } + subject + "say" + object + ";" + subject + "say" + object +} +``` + +### Scoping + +Because the regex engine backtracks by default when trying to match on a string, sometimes this backtracking can be wasted performance because we don't want to try various possibilities to eventually (maybe) find a match. + +In textual regexes, atomic groups (`(?>...)`) solve this problem by informing the regex engine to actually discard the backtrack location of a group, that is, defining a scope for backtracking. In regex builder, the `Local` type serves this purpose. + +```swift +public struct Local: RegexComponent { ... } +``` + +For example, the following regex matches string `abcc` but not `abc`. + +```swift +Regex { + "a" + Local { + ChoiceOf { + "bc" + "b" + } + } + "c" +} +``` + +If our input is `abcc`, we'll successfully find a match, however if we try to match against `abc` we won't get a match. The reason behind this is that in the `ChoiceOf` we actually matched the "bc" case first, but due to the local group we immediately disregard the backtracking location and continue to try and the rest of the regex. Since we matched the "bc", we don't have anymore string left to match the "c" and our local group will not try and attempt to match the other option, "b". + +
+API definition + +```swift +public struct Local: RegexComponent { + public var regex: Regex + + // The following builder methods implement what would be possible with + // variadic generics (using imaginary syntax) as a single set of methods: + // + // public init( + // @RegexComponentBuilder _ component: () -> Component + // ) where Output == (Substring, Capture...), Component.RegexOutput == (WholeMatch, Capture...) + + @_disfavoredOverload + public init( + @RegexComponentBuilder _ component: () -> Component + ) where Output == Substring + + public init( + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0), Component.RegexOutput == (W, C0) + + public init( + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0, C1), Component.RegexOutput == (W, C0, C1) + + // ... `O(arity)` overloads +} +``` + +
+ +### Composability + +Let's put everything together now and parse this example bank statement. + +``` +CREDIT 04062020 PayPal transfer $4.99 +CREDIT 04032020 Payroll $69.73 +DEBIT 04022020 ACH transfer $38.25 +DEBIT 03242020 IRS tax payment $52249.98 +``` + +Here we have 2 types of transaction kinds, CREDIT and DEBIT, we have a date +denoted by mmddyyyy, a description, and the amount paid. + +```swift +enum TransactionKind: String { + case credit = "CREDIT" + case debit = "DEBIT" +} + +struct Date { + var month: Int + var day: Int + var year: Int + + init?(mmddyyyy: String) { + ... + } +} + +let statementRegex = Regex { + // First, let's capture the transaction kind by wrapping our `ChoiceOf` in a + // `TryCapture` because our initializer can return nil on failure. + TryCapture { + ChoiceOf { + "CREDIT" + "DEBIT" + } + } transform: { + TransactionKind(rawValue: String($0)) + } + + OneOrMore(.whitespace) + + // Next, lets represent our date as 3 separate repeat quantifiers. The first + // two will require 2 digit characters, and the last will require 4. Then + // we'll take the entire substring and try to parse a date out. + TryCapture { + Repeat(.digit, count: 2) + Repeat(.digit, count: 2) + Repeat(.digit, count: 4) + } transform: { + Date(mmddyyyy: String($0)) + } + + OneOrMore(.whitespace) + + // Next, grab the description which can be any combination of word characters, + // digits, etc. + Capture { + OneOrMore(.any, .reluctant) + } + + OneOrMore(.whitespace) + + "$" + + // Finally, we'll grab one or more digits which will represent the whole + // dollars, match the decimal point, and finally get 2 digits which will be + // our cents. + TryCapture { + OneOrMore(.digit) + "." + Repeat(.digit, count: 2) + } transform: { + Double($0) + } +} + +for match in statement.matches(of: statementRegex) { + let (line, kind, date, description, amount) = match.output + ... +} +``` + +## Source compatibility + +Regex builder will be shipped in a new module named `RegexBuilder`, and thus will not affect the source compatibility of the existing code. + +## Effect on ABI stability + +The proposed feature does not change the ABI of existing features. + +## Effect on API resilience + +The proposed feature relies heavily upon overloads of `buildBlock` and `buildPartialBlock(accumulated:next:)` to work for different capture arities. In the fullness of time, we are hoping for variadic generics to supersede existing overloads. Such a change should not involve ABI-breaking modifications as it is merely a change of overload resolution. + +## Future directions + +### Conversion to textual regex + +Sometimes it may be useful to convert a regex created using regex builder to textual regex. This may be achieved in the future by extending `RegexComponent` with a computed property. + +```swift +extension RegexComponent { + public func makeTextualRegex() -> String? +} +``` + +It is worth noting that the internal representation of a `Regex` is _not_ textual regex, but an efficient pattern matching bytecode compiled from an abstract syntax tree. Moreover, not every `Regex` can be converted to textual regex. Regex builder supports arbitrary types that conform to the `RegexComponent` protocol, including `CustomMatchingRegexComponent` (pitched in [String Processing Algorithms]) which can be implemented with arbitrary code. If a `Regex` contains a `CustomMatchingRegexComponent`, it cannot be converted to textual regex. + +### Recursive subpatterns + +Sometimes, a textual regex may also use `(?R)` or `(?0)` to recusively evaluate the entire regex. For example, the following textual regex matches "I say you say I say you say hello". + +``` +(you|I) say (goodbye|hello|(?R)) +``` + +For this, `Regex` offers a special initializer that allows its pattern to recursively reference itself. This is somewhat akin to a fixed-point combinator. + +```swift +extension Regex { + public init( + @RegexComponentBuilder _ content: (Regex) -> R + ) where R.RegexOutput == Match +} +``` + +With this initializer, the above regex can be expressed as the following using regex builder. + +```swift +Regex { wholeSentence in + ChoiceOf { + "I" + "you" + } + "say" + ChoiceOf { + "goodbye" + "hello" + wholeSentence + } +} +``` + +There are some concerns with this design which we need to consider: +- Due to the lack of labeling, the argument to the builder closure can be arbitrarily named and cause confusion. +- When there is an initializer that accepts a result builder closure, overloading that initializer with the same argument labels could lead to bad error messages upon interor type errors. + +## Alternatives considered + +### Semicolons or parentheses instead of `One` + +In the DSL syntax as described in the first version of this proposal, there was a problem with the use of leading-dot syntax for character classes and other "atoms" and the builder syntax: +```swift +Regex { + .digit + OneOrMore(.whitespace) +} +``` +worked as expected, but: +```swift +Regex { + OneOrMore(.whitespace) + .digit +} +``` +did not, because `.digit` parses as a property on `OneOrMore` rather than a regex component. This could have been resolved by making people use either semicolons: +```swift +Regex { + OneOrMore(.whitespace); + .digit +} +``` +or parentheses: +```swift +Regex { + OneOrMore(.whitespace) + (.digit) +} +``` + +Instead we decided to introduce the quantifier `One` to resolve the ambiguity: +```swift +Regex { + OneOrMore(.whitespace) + One(.digit) +} +``` + +This increase the API surface, which is mildly undesirable, but feels much more stylistically consistent with the rest of the DSL and with Swift as whole. We also considered a "two protocol" approach that would force the use of `One` in these cases by making it impossible to use the dot-prefixed "atoms" within builder blocks, but this seems like too much heavy machinery to resolve the problem. + +### Operators for quantification and alternation + +While `ChoiceOf` and quantifier types provide a general way of creating alternations and quantifications, we recognize that some synctactic sugar can be useful for creating one-liners like in textual regexes, e.g. infix operator `|`, postfix operator `*`, etc. + +```swift +// The following functions implement what would be possible with variadic +// generics (using imaginary syntax) as a single function: +// +// public func | < +// R0: RegexComponent, R1: RegexComponent, +// WholeMatch0, WholeMatch1, +// Capture0..., Capture1... +// >( +// _ r0: RegexComponent, +// _ r1: RegexComponent +// ) -> Regex<(Substring, Capture0?..., Capture1?...)> +// where R0.RegexOutput == (WholeMatch0, Capture0...), +// R1.RegexOutput == (WholeMatch1, Capture1...) + +@_disfavoredOverload +public func | (lhs: R0, rhs: R1) -> Regex where R0: RegexComponent, R1: RegexComponent { + +public func | (lhs: R0, rhs: R1) -> Regex<(Substring, C0?)> where R0: RegexComponent, R1: RegexComponent, R1.RegexOutput == (W1, C0) + +public func | (lhs: R0, rhs: R1) -> Regex<(Substring, C0?, C1?)> where R0: RegexComponent, R1: RegexComponent, R1.RegexOutput == (W1, C0, C1) + +// ... `O(arity^2)` overloads. +``` + +However, like `RegexComponentBuilder.buildPartialBlock(accumulated:next:)`, operators such as `|`, `+`, `*`, `.?` require a large number of overloads to work with regexes of every capture arity, compounded by the fact that operator type checking is prone to performance issues in Swift. Here is a list of + +| Opreator | Meaning | Required number of overloads | +|---------------|---------------------------|------------------------------| +| Infix `\|` | Choice of two | `O(arity^2)` | +| Postfix `*` | Zero or more eagerly | `O(arity)` | +| Postfix `*?` | Zero or more reluctantly | `O(arity)` | +| Postfix `*+` | Zero or more possessively | `O(arity)` | +| Postfix `+` | One or more eagerly | `O(arity)` | +| Postfix `+?` | One or more reluctantly | `O(arity)` | +| Postfix `++` | One or more possessively | `O(arity)` | +| Postfix `.?` | Optionally eagerly | `O(arity)` | +| Postfix `.??` | Optionally reluctantly | `O(arity)` | +| Postfix `.?+` | Optionally possessively | `O(arity)` | + + When variadic generics are supported in the future, we may be able to define one function per operator and reduce type checking burdens. + +### Postfix `capture` and `tryCapture` methods + +An earlier iteration of regex builder declared `capture` and `tryCapture` as methods on `RegexComponent`, meaning that you can append `.capture(...)` to any subpattern within a regex to capture it. For example: + +```swift +Regex { + OneOrMore { + r0.capture() + r1 + }.capture() +} // => Regex<(Substring, Substring, Substring)> +``` + +However, there are two shortcomings of this design: + +1. When a subpattern to be captured contains multiple components, the developer has to explicitly group them using a `Regex { ... }` block. + + ```swift + let emailPattern = Regex { + let word = OneOrMore(.word) + Regex { // <= Had to explicitly group multiple components + ZeroOrMore { + word + "." + } + word + }.capture() + "@" + Regex { + word + OneOrMore { + "." + word + } + }.capture() + } // => Regex<(Substring, Substring, Substring)> + ``` + +2. When there are nested captures, it is harder to number the captures visually because the order `capture()` appears is flipped in the postfix (method) notation. + + ```swift + let emailSuffixPattern = Regex { + "@" + Regex { + word + OneOrMore { + "." + word.capture() // top-level domain (.0) + } + }.capture() // full domain (.1) + } // => Regex<(Substring, Substring, Substring)> + // + // full domain ^~~~~~~~~ + // top-level domain ^~~~~~~~~ + ``` + + In comparison, prefix notation (`Capture` and `TryCapture` as a types) makes it easier to visually capture captures as you can number captures in the order they appear from top to bottom. This is consistent with textual regexes where capturing groups are numbered by the left parenthesis of the group from left to right. + + ```swift + let emailSuffixPattern = Regex { + Capture { // full domain (.0) + word + OneOrMore { + "." + Capture(word) // top-level domain (.1) + } + } + } // => Regex<(Substring, Substring, Substring)> + // + // full domain ^~~~~~~~~ + // top-level domain ^~~~~~~~~ + ``` + +### Unify quantifiers under `Repeat` + +Since `Repeat` is the most general version of quantifiers, one could argue for all quantifiers to be unified under the type `Repeat`, for example: + +```swift +Repeat(oneOrMore: r) +Repeat(zeroOrMore: r) +Repeat(optionally: r) +``` + +However, given that one-or-more (`+`), zero-or-more (`*`) and optional (`?`) are the most common quantifiers in textual regexes, we believe that these quantifiers deserve their own type and should be written as a single word instead of two. This can also reduce visual clutter when the quantification is used in multiple places of a regex. + +### Free functions instead of types + +One could argue that type such as `OneOrMore` could be defined as a top-level function that returns `Regex`. While it is entirely possible to do so, it would lose the name scoping benefits of a type and pollute the top-level namespace with `O(arity^2)` overloads of quantifiers, `capture`, `tryCapture`, etc. This could be detrimental to the usefulness of code completion. + +Another reason to use types instead of free functions is consistency with existing result-builder-based DSLs such as SwiftUI. + +### Support `buildOptional` and `buildEither` + +To support `if` statements, an earlier iteration of this proposal defined `buildEither(first:)`, `buildEither(second:)` and `buildOptional(_:)` as the following: + +```swift +extension RegexComponentBuilder { + public static func buildEither< + Component, WholeMatch, Capture... + >( + first component: Component + ) -> Regex<(Substring, Capture...)> + where Component.RegexOutput == (WholeMatch, Capture...) + + public static func buildEither< + Component, WholeMatch, Capture... + >( + second component: Component + ) -> Regex<(Substring, Capture...)> + where Component.RegexOutput == (WholeMatch, Capture...) + + public static func buildOptional< + Component, WholeMatch, Capture... + >( + _ component: Component? + ) where Component.RegexOutput == (WholeMatch, Capture...) +} +``` + +However, multiple-branch control flow statements (e.g. `if`-`else` and `switch`) would need to be required to produce either the same regex type, which is limiting, or an "either-like" type, which can be difficult to work with when nested. Unlike `ChoiceOf`, producing a tuple of optionals is not an option, because the branch taken would be decided when the builder closure is executed, and it would cause capture numbering to be inconsistent with conventional regex. + +Moreover, result builder conditionals does not work the same way as regex conditionals. In regex conditionals, the conditions are themselves regexes and are evaluated by the regex engine during matching, whereas result builder conditionals are evaluated as part of the builder closure. We hope that a future result builder feature will support "lifting" control flow conditions into the DSL domain, e.g. supporting `Regex` as a condition. + +### Flatten optionals + +With the proposed design, `ChoiceOf` with `AlternationBuilder` wraps every component's capture type with an `Optional`. This means that any `ChoiceOf` with optional-capturing components would lead to a doubly-nested optional captures. This could make the result of matching harder to use. + +```swift +ChoiceOf { + OneOrMore(Capture(.digit)) // RegexOutput == (Substring, Substring) + Optionally { + ZeroOrMore(Capture(.word)) // RegexOutput == (Substring, Substring?) + "a" + } // RegexOutput == (Substring, Substring??) +} // RegexOutput == (Substring, Substring?, Substring???) +``` + +One way to improve this could be overloading quantifier initializers (e.g. `ZeroOrMore.init(_:)`) and `AlternationBuilder.buildPartialBlock` to flatten any optionals upon composition. However, this would be non-trivial. Quantifier initializers would need to be overloaded `O(2^arity)` times to account for all possible positions of `Optional` that may appear in the `Output` tuple. Even worse, `AlternationBuilder.buildPartialBlock` would need to be overloaded `O(arity!)` times to account for all possible combinations of two `Output` tuples with all possible positions of `Optional` that may appear in one of the `Output` tuples. + +### Structured rather than flat captures + +We propose inferring capture types in such a way as to align with the traditional numbering of backreferences. This is because much of the motivation behind providing regex in Swift is their familiarity. + +If we decided to deprioritize this motivation, there are opportunities to infer safer, more ergonomic, and arguably more intuitive types for captures. For example, to be consistent with traditional regex backreferences quantifications of multiple or nested captures had to produce parallel arrays rather than an array of tuples. + +```swift +OneOrMore { + Capture { + OneOrMore(.hexDigit) + } + ".." + Capture { + OneOrMore(.hexDigit) + } +} + +// Flat capture types: +// => `RegexOutput == (Substring, Substring, Substring)>` + +// Structured capture types: +// => `RegexOutput == (Substring, (Substring, Substring))` +``` + +Similarly, an alternation of multiple or nested captures could produce a structured alternation type (or an anonymous sum type) rather than flat optionals. + +This is cool, but it adds extra complexity to regex builder and it isn't as clear because the generic type no longer aligns with the traditional regex backreference numbering. We think the consistency of the flat capture types trumps the added safety and ergonomics of the structured capture types. + +### Unify `Capture` with `TryCapture` + +The primary difference between `Capture` and `TryCapture` at the API level is that `TryCapture`'s transform closure returns an `Optional` of the target type, whereas `Capture`'s transform closure returns the target type. `TryCapture` would cause the regex engine to backtrack when the transform closure returns nil, whereas `Capture` does not backtrack. + +It has been argued in the review thread that the distinction between `Capture` and `TryCapture` need not be reflected at the type name level, but could be differentiated by argument label, e.g. `transform:`/`tryTransform:` or `map:`/`compactMap:`. However, doing so may cause ambiguity in cases where the transform closure is not the second, but the first, trailing closure in the initializer. + +```swift +extension Capture { + public init( + _ component: R, + map: @escaping (Substring) throws -> NewCapture + ) where Output == (Substring, NewCapture), R.RegexOutput == W + + public init( + _ component: R, + compactMap: @escaping (Substring) throws -> NewCapture? + ) where Output == (Substring, NewCapture), R.RegexOutput == W +} +``` + +In this case, since the argument label will not be specified for the first trailing closure, using `Capture` where the component is a non-builder-closure may cause type-checking ambiguity. + +```swift +Regex { + Capture(OneOrMore(.digit)) { + Int($0) + } // Which output type, `(Substring, Substring)` or `(Substring, Substring?)`? +} +``` + +Spelling out `TryCapture` also has the benefit of clarity, as it makes clear that a capture's transform closure can cause the regex engine to backtrack. Since backtracking can be expensive, one could choose to throw errors instead and use a normal `Capture`. + +```swift +Regex { + Capture(OneOrMore(.digit)) { + guard let number = Int($0) else { + throw MyCustomParsingError.invalidNumber($0) + } + return number + } +} +``` + +[Declarative String Processing]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/DeclarativeStringProcessing.md +[Strongly Typed Regex Captures]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/StronglyTypedCaptures.md +[Regex Syntax]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md +[String Processing Algorithms]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0357-regex-string-processing-algorithms.md diff --git a/proposals/0352-implicit-open-existentials.md b/proposals/0352-implicit-open-existentials.md new file mode 100644 index 0000000000..ccf49c1942 --- /dev/null +++ b/proposals/0352-implicit-open-existentials.md @@ -0,0 +1,684 @@ +# Implicitly Opened Existentials + +* Proposal: [SE-0352](0352-implicit-open-existentials.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.7)** +* Upcoming Feature Flag: `ImplicitOpenExistentials` (Implemented in Swift 6.0) (Enabled in Swift 6 language mode) +* Implementation: [apple/swift#41996](https://github.com/apple/swift/pull/41996), [macOS toolchain](https://ci.swift.org/job/swift-PR-toolchain-macos/120/artifact/branch-main/swift-PR-41996-120-osx.tar.gz) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0352-implicitly-opened-existentials/57553) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/77374319a7d70c866bd197faada46ecfce461645/proposals/0352-implicit-open-existentials.md) +* Previous Review: [First review](https://forums.swift.org/t/se-0352-implicitly-opened-existentials/56557/52) + +## Table of Contents + + * [Introduction](#introduction) + * [Proposed solution](#proposed-solution) + * [Moving between any and some](#moving-between-any-and-some) + * [Detailed design](#detailed-design) + * [When can we open an existential?](#when-can-we-open-an-existential) + * [Type-erasing resulting values](#type-erasing-resulting-values) + * ["Losing" constraints when type-erasing resulting values](#losing-constraints-when-type-erasing-resulting-values) + * [Contravariant erasure for parameters of function type](#contravariant-erasure-for-parameters-of-function-type) + * [Order of evaluation restrictions](#order-of-evaluation-restrictions) + * [Avoid opening when the existential type satisfies requirements (in Swift 5)](#avoid-opening-when-the-existential-type-satisfies-requirements-in-swift-5) + * [Suppressing explicit opening with as any P / as! any P](#suppressing-explicit-opening-with-as-any-p--as-any-p) + * [Source compatibility](#source-compatibility) + * [Effect on ABI stability](#effect-on-abi-stability) + * [Effect on API resilience](#effect-on-api-resilience) + * [Alternatives considered](#alternatives-considered) + * [Explicitly opening existentials](#explicitly-opening-existentials) + * [Value-dependent opening of existentials](#value-dependent-opening-of-existentials) + * [Revisions](#revisions) + * [Acknowledgments](#acknowledgments) + +## Introduction + +Existential types in Swift allow one to store a value whose specific type is unknown and may change at runtime. The dynamic type of that stored value, which we refer to as the existential's *underlying type*, is known only by the set of protocols it conforms to and, potentially, its superclass. While existential types are useful for expressing values of dynamic type, they are necessarily restricted because of their dynamic nature. Recent proposals have made [existential types more explicit](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md) to help developers understand this dynamic nature, as well as [making existential types more expressive](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md) by removing a number of limitations. However, a fundamental issue with existential types remains, that once you have a value of existential type it is *very* hard to use generics with it. Developers usually encounter this via the error message "protocol 'P' as a type cannot conform to itself": + +```swift +protocol P { + associatedtype A + func getA() -> A +} + +func takeP(_ value: T) { } + +func test(p: any P) { + takeP(p) // error: protocol 'P' as a type cannot conform to itself +} +``` + +This interaction with the generics system makes existentials a bit of a trap in Swift: it's easy to go from generics to existentials, but once you have an existential it is very hard to go back to using it generically. At worst, you need to go back through many levels of functions, changing their parameters or results from `any P` to being generic over `P`, or writing a custom [type eraser](https://www.swiftbysundell.com/articles/different-flavors-of-type-erasure-in-swift/). + +This proposal addresses this existential trap by allowing one to "open" an existential value, binding a generic parameter to its underlying type. Doing so allows us to call a generic function with an existential value, such that the generic function operates on the underlying value of the existential rather than on the existential box itself, making it possible to get out of the existential trap without major refactoring. This capability already exists in the language when accessing a member of an existential (e.g., `p.getA()`), and this proposal extends that behavior to all call arguments in a manner that is meant to be largely invisible: calls to generic functions that would have failed (like `takeP(p)` above) will now succeed. Smoothing out this interaction between existentials and generics can simplify Swift code and make the language more approachable. + +Swift-evolution thread: [Pitch #1](https://forums.swift.org/t/pitch-implicitly-opening-existentials/55412), [Pitch #2](https://forums.swift.org/t/pitch-2-implicitly-opening-existentials/56360) + +## Proposed solution + +To make it easier to move from existentials back to the more strongly-typed generics, we propose to implicitly *open* an existential value when it is passed to a parameter of generic type. In such cases, the generic argument refers to the *underlying* type of the existential value rather than the existential "box". Let's start with a protocol `Costume` that involves `Self` requirements, and write a generic function that checks some property of a costume: + +```swift +protocol Costume { + func withBells() -> Self + func hasSameAdornments(as other: Self) -> Bool +} + +// Okay: generic function to check whether adding bells changes anything +func hasBells(_ costume: C) -> Bool { + return costume.hasSameAdornments(as: costume.withBells()) +} +``` + +This is fine. However, let's write a function that makes sure every costume has bells for the big finale. We run into problems at the boundary between the array of existential values and our generic function: + +```swift +func checkFinaleReadiness(costumes: [any Costume]) -> Bool { + for costume in costumes { + if !hasBells(costume) { // error: protocol 'Costume' as a type cannot conform to the protocol itself + return false + } + } + + return true +} +``` + +In the call to `hasBells`, the generic parameter `C` is getting bound to the type `any Costume`, i.e., a box that contains a value of some unknown underlying type. Each instance of that box type might have a different type at runtime, so even though the underlying type conforms to `Costume`, the box does not. That box itself does not conform to `Costume` because it does not meet the requirement for `hasSameAdornments`., i.e., two boxes aren't guaranteed to store the same the same underlying type. + +This proposal introduces implicitly opened existentials, which allow one to use a value of existential type (e.g., `any Costume`) where its underlying type can be captured in a generic parameter. For example, the call `hasBells(costume)` above would succeed, binding the generic parameter `C` to the underlying type of that particular instance of `costume`. Each iteration of the loop could have a different underlying type bound to `C`: + +```swift +func checkFinaleReadiness(costumes: [any Costume]) -> Bool { + for costume in costumes { + if !hasBells(costume) { // okay with this proposal: C is bound to the type stored inside the 'any' box, known only at runtime + return false + } + } + + return true +} +``` + +Implicitly opening existentials allows one to take a dynamically-typed value and give its underlying type a name by binding it to a generic parameter, effectively moving from a dynamically-typed value to a more statically-typed one. This notion isn't actually new: calling a member of a protocol on a value of existential type implicitly "opens" the `Self` type. In the existing language, one could implement a shim for `hasBells` as a member of a protocol extension: + +```swift +extension Costume { + var hasBellsMember: Bool { + hasBells(self) + } +} + +func checkFinaleReadinessMember(costumes: [any Costume]) -> Bool { + for costume in costumes { + if !costume.hasBellsMember { // okay today: 'Self' is bound to the type stored inside the 'any' box, known only at runtime + return false + } + } + + return true +} +``` + +In that sense, implicitly opening existentials for calls to generic functions is a generalization of this existing behavior to all generic parameters. It isn't strictly more expressive: as the `hasBellsMember` example shows, one *can* always write a member in a protocol extension to get this opening behavior. This proposal aims to make implicit opening of existentials more uniform and more ergonomic, by making it more general. + +Let's consider one last implementation of our "readiness" check, where want to "open code" the check for bells without putting the logic into a separate generic function `hasBells`: + +```swift +func checkFinaleReadinessOpenCoded(costumes: [any Costume]) -> Bool { + for costume in costumes { + let costumeWithBells = costume.withBells() // returned type is 'any Costume' + if !costume.hasSameAdornments(costumeWithBells) { // error: 'any Costume' isn't necessarily the same type as 'any Costume' + return false + } + } + + return true +} +``` + +There are two things to notice here. First, the method `withBells()` returns type `Self`. When calling that method on a value of type `any Costume`, the concrete result type is not known, so it is type-erased to `any Costume` (which becomes the type of `costumeWithBells`). Second, on the next line, the call to `hasSameAdornments` produces a type error because the function expects a value of type `Self`, but there is no statically-typed link between `costume` and `costumeWithBells`: both are of type `any Costume`. Implicit opening of existential arguments only occurs in calls, so that its effects can be type-erased at the end of the call. To have the effects of opening persist over multiple statements, factor that code out into a generic function that gives a name to the generic parameter, as with `hasBells`. + +### Moving between `any` and `some` + +One of the interesting aspects of this proposal is that it allows one to refactor `any` parameters into `some` parameters (as introduced by [SE-0341](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md)) without a significant effect on client code. Let's rewrite our generic `hasBells` function using `some`: + +```swift +func hasBells(_ costume: some Costume) -> Bool { + return costume.hasSameAdornments(as: costume.withBells()) +} +``` + +With this proposal, we can now call `hasBells` given a value of type `any Costume`: + +```swift +func isReadyForFinale(_ costume: any Costume) -> Bool { + return hasBells(costume) // implicit opening of the existential value +} +``` + +It's always the case that one can go from a statically-typed `some Costume` to an `any Costume`. This proposal also allows one to go the other way, opening up an `any Costume` into a `some Costume` parameter. Therefore, with this proposal, we could refactor `isReadyForFinale` to make it generic via `some`: + +```swift +func isReadyForFinale(_ costume: some Costume) -> Bool { + return hasBells(costume) // okay, `T` binds to the generic argument +} +``` + +Any callers to `isReadyForFinale` that provided concrete types now avoid the overhead of "boxing" their type in an `any Costume`, and any callers that provided an `any Costume` will now implicitly open up that existential in the call to `isReadyForFinale`. This allows existential operations to be migrated to generic ones without having to also make all clients generic at the same time, offering an incremental way out of the "existential trap". + +## Detailed design + +Fundamentally, opening an existential means looking into the existential box to find the dynamic type stored within the box, then giving a "name" to that dynamic type. That dynamic type name needs to be captured in a generic parameter somewhere, so it can be reasoned about statically, and the value with that type can be passed along to the generic function being called. The result of such a call might also refer to that dynamic type name, in which case it has to be erased back to an existential type. The After the call, any values described in terms of that dynamic type opened existential type has to be type-erased back to an existential so that the opened type name doesn't escape into the user-visible type system. This both matches the existing language feature (opening an existential value when accessing one of its members) and also prevents this feature from constituting a major extension to the type system itself. + +This section describes the details of opening an existential and then type-erasing back to an existential. These details of this change should be invisible to the user, and manifest only as the ability to use existentials with generics in places where the code would currently be rejected. However, there are a *lot* of details, because moving from dynamically-typed existential boxes to statically-typed generic values must be carefully done to maintain type identity and the expected evaluation semantics. + +### When can we open an existential? + +To open an existential, the argument (or source) must be of existential type (e.g., `any P`) or existential metatype (e.g., `any P.Type`) and must be provided to a parameter (or target) whose type involves a generic parameter that can bind directly to the underlying type of the existential. This means that, for example, we can open an existential when its underlying type would directly bind to a generic parameter: + +```swift +protocol P { + associatedtype A + + func getA() -> A +} + +func openSimple(_ value: T) { } + +func testOpenSimple(p: any P) { + openSimple(p) // okay, opens 'p' and binds 'T' to its underlying type +} +``` + +It's also possible to open an `inout` parameter. The generic function will operate on the underlying type, and can (e.g.) call `mutating` methods on it, but cannot change its *dynamic* type because it doesn't have access to the existential box: + +```swift +func openInOut(_ value: inout T) { } +func testOpenInOut(p: any P) { + var mutableP: any P = p + openInOut(&mutableP) // okay, opens to 'mutableP' and binds 'T' to its underlying type +} +``` + +However, we cannot open when there might be more than one value of existential type or no values at all, because we need to be guaranteed to have a single underlying type to infer. Here are several such examples where the generic parameter is used in multiple places in a manner that prevents opening the existential argument: + +```swift +func cannotOpen1(_ array: [T]) { .. } +func cannotOpen2(_ a: T, _ b: T) { ... } +func cannotOpen3(_ values: T...) { ... } + +struct X { } +func cannotOpen4(_ x: X) { } + +func cannotOpen5(_ x: T, _ a: T.A) { } + +func cannotOpen6(_ x: T?) { } + +func testCannotOpenMultiple(array: [any P], p1: any P, p2: any P, xp: X, pOpt: (any P)?) { + cannotOpen1(array) // each element in the array can have a different underlying type, so we cannot open + cannotOpen2(p1, p2) // p1 and p2 can have different underlying types, so there is no consistent binding for 'T' + cannotOpen3(p1, p2) // similar to the case above, p1 and p2 have different types, so we cannot open them + cannotOpen4(xp) // cannot open the existential in 'X' there isn't a specific value there. + cannotOpen5(p1, p2.getA()) // cannot open either argument because 'T' is used in both parameters + cannotOpen6(pOpt) // cannot open the existential in '(any P)?' because it might be nil, so there would not be an underlying type +} +``` + +The case of optionals is somewhat interesting. It's clear that the call `cannotOpen6(pOpt)` cannot work because `pOpt` could be `nil`, in which case there is no type to bind `T` to. We *could* choose to allow opening a non-optional existential argument when the parameter is optional, e.g., + +```swift +cannotOpen6(p1) // we *could* open here, binding T to the underlying type of p1, but choose not to +``` + +but this proposal doesn't allow this because it would be odd to allow this call but not the `cannotOpen6(pOpt)` call. + +A value of existential metatype can also be opened, with the same limitations as above. + +```swift +func openMeta(_ type: T.Type) { } + +func testOpenMeta(pType: any P.Type) { + openMeta(pType) // okay, opens 'pType' and binds 'T' to its underlying type +} +``` + +### Type-erasing resulting values + +The result type of a generic function can involve generic parameters and their associated types. For example, here's a generic function that returns the original value and some values of its associated types: + +```swift +protocol Q { + associatedtype B: P + func getB() -> B +} + +func decomposeQ(_ value: T) -> (T, T.B, T.B.A) { + (value, value.getB(), value.getB().getA()) +} +``` + +When calling `decomposeQ` with an existential value, the existential is opened and `T` will bind to its underlying type. `T.B` and `T.B.A` are types derived from that underlying type. Once the call completes, however, the types `T`, `T.B`, and `T.B.A` are *type-erased* to their upper bounds, i.e., the existential type that captures all of their requirements. For example: + +```swift +func testDecomposeQ(q: any Q) { + let (a, b, c) = decomposeQ(q) // a is any Q, b is any P, c is Any +} +``` + +This is identical to the [covariant erasure of associated types described in SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md#covariant-erasure-for-associated-types), and the rules specified there apply equally here. We can restate those requirements more generally for an arbitrary generic parameter as: + +When binding a generic parameter `T` to an opened existential, `T`, `T` and `T`-rooted associated types that + +- are **not** bound to a concrete type, and +- appear in covariant position within the result type of the generic function + +will be type-erased to their upper bounds as per the generic signature of the existential that is used to access the member. The upper bounds can be either a class, protocol, protocol composition, or `Any`, depending on the *presence* and *kind* of generic constraints on the associated type. + +When `T` or a `T`-rooted associated type appears in a non-covariant position in the result type, `T` cannot be bound to the underlying type of an existential value because there would be no way to represent the type-erased result. This is essentially the same property as described for the parameter types that prevents opening of existentials, as described above. For example: + +```swift +func cannotOpen7(_ value: T) -> X { /*...*/ } +``` + +However, because the return value is permitted a conversion to erase to an existential type, optionals, tuples, and even arrays *are* permitted: + +```swift +func openWithCovariantReturn1(_ value: T) -> T.B? { /*...*/ } +func openWithCovariantReturn2(_ value: T) -> [T.B] { /*...*/ } + +func covariantReturns(q: any Q){ + let r1 = openWithCovariantReturn1(q) // okay, 'T' is bound to the underlying type of 'q', resulting type is 'any P' + let r2 = openWithCovariantReturn2(q) // okay, 'T' is bound to the underlying type of 'q', resulting type is '[any Q]' +} +``` + +### "Losing" constraints when type-erasing resulting values + +When the result of a call involving an opened existential is type-erased, it is possible that some information about the returned type cannot be expressed in an existential type, so the "upper bound" described above will lose information. For example, consider the type of `b` in this example: + +```swift +protocol P { + associatedtype A +} + +protocol Q { + associatedtype B: P where B.A == Int +} + +func getBFromQ(_ q: T) -> T.B { ... } + +func eraseQAssoc(q: any Q) { + let b = getBFromQ(q) +} +``` + +When type-erasing `T.B`, the most specific upper bound would be "a type that conforms to `P` where the associated type `A` is known to be `Int`". However, Swift's existential types cannot express such a type, so the type of `b` will be the less-specific `any P`. + +It is likely that Swift's existentials will grow in expressivity over time. For example, [SE-0353 "Constrained Existential Types"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0353-constrained-existential-types.md) allows one to express existential types that involve bindings for [primary associated types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md). If we were to adopt that feature for protocol `P`, the most specific upper bound would be expressible: + +```swift +// Assuming SE-0353... +protocol P { + associatedtype A +} + +// ... same as above ... +``` + +Now, `b` would be expected to have the type `any P`. Future extensions of existential types might make the most-specific upper bound expressible even without any source code changes, and one would expect that the type-erasure after calling a function with an implicitly-opened existential would become more precise when those features are added. + +However, this kind of change presents a problem for source compatibility, because code might have come to depend on the type of `b` being the less-precise `any P` due to, e.g., overloading: + +```swift +func f(_: T) -> Int { 17 } +func f(_: T) -> Double where T.A == Int { 3.14159 } + +// ... +func eraseQAssoc(q: any Q) { + let b = getBFromQ(q) + f(b) +} +``` + +With the less-specific upper bound (`any P`), the call `f(b)` would choose the first overload that returns an `Int`. With the more-specific upper bound (`any P` where `A` is known to be `Int`), the call `f(b)` would choose the second overload that returns a `Double`. + +Due to overloading, the source-compatibility impacts of improving the upper bound cannot be completely eliminated without (for example) holding the upper bound constant until a new major language version. However, we propose to mitigate the effects by requiring a specific type coercion on any call where the upper bound is unable to express some requirements due to limitations on existentials. Specifically, the call `getBFromQ(q)` would need to be written as: + +```swift +getBFromQ(q) as any P +``` + +This way, if the upper bound changes due to increased expressiveness of existential types in the language, the overall expression will still produce a value of the same type---`any P`---as it always has. A developer would be free to remove the `as any P` at the point where Swift can fully capture all of the information known about the type in an existential. + +Note that this requirement for an explicit type coercion also applies to all type erasure due to existential opening, including ones that existed prior to this proposal. For example, `getBFromQ` could be written as a member of a protocol extension. The code below has the same issues (and the same resolution) as our example, as was first made well-formed with [SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md): + +```swift +extension Q { + func getBFromQ() -> B { ... } +} + +func eraseQAssocWithSE0309(q: any Q) { + let b = q.getBFromQ() +} +``` + +### Contravariant erasure for parameters of function type + +While covariant erasure applies to the result type of a generic function, the opposite applies to other parameters of the generic function. This affects parameters of function type that reference the generic parameter binding to the opened existential, which will be type-erased to their upper bounds. For example: + +```swift +func acceptValueAndFunction(_ value: T, body: (T) -> Void) { ... } + +func testContravariantErasure(p: any P) { + acceptValueAndFunction(p) { innerValue in // innerValue has type 'any P' + // ... + } +} +``` + +Like the covariant type erasure applied to result types, this type erasure ensures that the "name" assigned to the dynamic type doesn't escape into the user-visible type system through the inferred closure parameter. It effectively maintains the illusion that the generic type parameter `T` is binding to `any P`, while in fact it is binding to the underlying type of that specific value. + +There is one exception to this rule: if the argument to such a parameter is a reference to a generic function, the type erasure does not occur. In such cases, the dynamic type name is bound directly to the generic parameter of this second generic function, effectively doing the same implicit opening of existentials again. This is best explained by example: + +```swift +func takeP(_: U) -> Void { ... } + +func implicitOpeningArguments(p: any P) { + acceptValueAndFunction(p, body: takeP) // okay: T and U both bind to the underlying type of p +} +``` + +This behavior subsumes most of the behavior of the hidden `_openExistential` operation, which specifically only supports opening one existential value and passing it to a generic function. `_openExistential` might still have a few scattered use cases when opening an existential that doesn't have conformance requirements on it. + + ### Order of evaluation restrictions + +Opening an existential box requires evaluating that the expression that produces that box and then peering inside it to extract its underlying type. The evaluation of the expression might have side effects, for example, if one calls the following `getP()` function to produce a value of existential box type `any P`: + +```swift +extension Int: P { } + +func getP() -> any P { + print("getP()") + return 17 +} +``` + +Now consider a generic function for which we want open an existential argument: + +```swift +func acceptFunctionStringAndValue(body: (T) -> Void, string: String, value: T) { ... } + +func hello() -> String { + print("hello()") +} + +func implicitOpeningArgumentsBackwards() { + acceptFunctionStringAndValue(body: takeP, string: hello(), value: getP()) // will be an error, see later +} +``` + +Opening the argument to the `value` parameter requires performing the call to `getP()`. This has to occur *before* the argument to the `body` parameter can be formed, because `takeP`'s generic type parameter `U` is bound to the underlying type of that existential box. Doing so means that the program would produce side effects in the following order: + +``` +getP() +hello() +``` + +However, this would contradict Swift's longstanding left-to-right evaluation order. Rather than do this, we instead place another limitation on the implicit opening of existentials: an existential argument cannot be opened if the generic type parameter bound to its underlying type is used in any function parameter preceding the one corresponding to the existential argument. In the `implicitOpeningArgumentsBackwards` above, the call to `acceptFunctionStringAndValue` does not permit opening the existential argument to the `value` parameter because its generic type parameter, `T`, is also used in the `body` parameter that precedes `value`. This ensures that the underlying type is not needed for any argument prior to the opened existential argument, so the left-to-right evaluation order is maintained. + +### Avoid opening when the existential type satisfies requirements (in Swift 5) + +As presented thus far, opening of existential values can change the behavior of existing programs that relied on passing the existential box to a generic function. For example, consider the effect of passing an existential box to an unconstrained generic function that puts the parameter into the returned array: + +```swift +func acceptsBox(_ value: T) -> Any { [value] } + +func passBox(p: any P) { + let result = acceptsBox(p) // currently infers 'T' to be 'any P', returns [any P] + // unrestricted existential opening would infer 'T' to be the underlying type of 'p', returns [T] +} +``` + +Here, the dynamic type of the result of `acceptsBox` would change if the existential box is opened as part of the call. The change itself is subtle, and would not be detected until runtime, which could cause problems for existing Swift programs that rely on binding generic parameters. Therefore, in Swift 5, this proposal prevents opening of existential values when the existential types themselves would satisfy the conformance requirements of the corresponding generic parameter, making it a strictly additive change: calls to generic functions with existential values that previously worked will continue to work with the same semantics, but calls that didn't work before will open the existential and can therefore succeed. + +Most of the cases in today's Swift where a generic parameter binds to an existential type succeed because there are no conformance requirements on the generic parameter, as with the `T` generic parameter to `acceptsBox`. For most protocols, an existential referencing the corresponding type does not conform to that protocol, i.e., `any Q` does not conform to `Q`. However, there are a small number of exceptions: + +* The existential type `any Error` conforms to the `Error` protocol, as specified in [SE-0235](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0235-add-result.md#adding-swifterror-self-conformance). +* An existential type `any Q` of an `@objc` protocol `Q`, where `Q` contains no `static` requirements, conforms to `Q`. + +For example, consider an operation that takes an error. Passing a value of type `any Error` to it succeeds without opening the existential: + +```swift +func takeError(_ error: E) { } + +func passError(error: any Error) { + takeError(error) // okay without opening: 'E' binds to 'any Error' because 'any Error' conforms to 'Error' +} +``` + +This proposal preserves the semantics of the call above by not opening the existential argument in cases where the existential type satisfies the corresponding generic parameter's conformance requirements according to the results above. Should Swift eventually grow a mechanism to make existential types conform to protocols (e.g., so that `any Hashable` conforms to `Hashable`), then such conformances will **not** suppress implicit opening, because any code that made use of these conformances would be newly-valid code and would start with implicit-opening semantics. + +Swift 6 will be a major language version change that can incorporate some semantics- and source-breaking changes. In Swift 6, the suppression mechanism described in this section will *not* apply, so the `passBox` example above would open the value of `p` and bind `T` to that opened existential type. This provides a more consistent semantics that, additionally, subsumes all of the behavior of `type(of:)` and the hidden `_openExistential` operation. + +### Suppressing explicit opening with `as any P` / `as! any P` + +If for some reason one wants to suppress the implicit opening of an existential value, one can explicitly write a coercion or forced cast to an existential type directly on the call argument. For example: + +```swift +func f1(_: T) { } // #1 +func f1(_: T) { } // #2 + +func test(p: any P) { + f1(p) // opens p and calls #1, which is more specific + f1(p as any P) // suppresses opening of 'p', calls #2 which is the only valid candidate + f1((p as any P)) // parentheses disable this suppression mechanism, so this opens p and calls #1 +} +``` + +Given that implicit opening of existentials is defined to occur in those cases where a generic function would not otherwise be callable, this suppression mechanism should not be required often in Swift 5. In Swift 6, where implicit opening will be more eagerly performed, it can be used to provide the Swift 5 semantics. + +An extra set of parentheses will disable this suppression mechanism, which can be important when `as any P` is required for some other reason. For example, because it acknowledges when information is lost from the result type due to type erasure. This can help break ambiguities when both meanings of `as` could apply: + +```swift +protocol P { + associatedtype A +} +protocol Q { + associatedtype B: P where B.A == Int +} + +func getP(_ p: T) +func getBFromQ(_ q: T) -> T.B { ... } + +func eraseQAssoc(q: any Q) { + getP(getBFromQ(q)) // error, must specify "as any P" due to loss of constraint T.B.A == Int + getP(getBFromQ(q) as any P) // suppresses error above, but also suppresses opening, so it produces + // error: now "any P does not conform to P" and op + getP((getBFromQ(q) as any P)) // okay! original error message should suggest this +} + +``` + +## Source compatibility + +This proposal is defined specifically to avoid most impacts on source compatibility, especially in Swift 5. Some calls to generic functions that would previously have been ill-formed (e.g., they would fail because `any P` does not conform to `P`) will now become well-formed, and existing code will behavior in the same manner as before. As with any such change, it's possible that overload resolution that would have succeeded before will continue to succeed but will now pick a different function. For example: + +```swift +protocol P { } + +func overloaded1(_: T, _: U) { } // A +func overloaded1(_: Any, _: U) { } // B + +func changeInResolution(p: any P) { + overloaded1(p, 1) // used to choose B, will choose A with this proposal +} +``` + +Such examples are easy to construct in the abstract for any feature that makes ill-formed code well-formed, but these examples rarely cause problems in practice. + +## Effect on ABI stability + +This proposal changes the type system but has no ABI impact whatsoever. + +## Effect on API resilience + +This proposal changes the use of APIs, but not the APIs themselves, so it doesn't impact API resilience per se. + +## Alternatives considered + +This proposal opts to open existentials implicitly and locally, type-erasing back to existentials after the immediate call, as a generalization of opening when using a member of an existential value. There are alternative designs that are explicit or open the existential more broadly, with different tradeoffs. + +### Explicitly opening existentials + +This proposal implicitly opens existentials at call sites. Instead, we could provide an explicit syntax for opening an existential, e.g., via [an `as` coercion to `some P`](https://forums.swift.org/t/pitch-implicitly-opening-existentials/55412/8). For example, + +```swift +protocol P { + associatedtype A +} + +func takesP(_ value: T) { } + +func hasExistentialP(p: any P) { + takesP(p) // error today ('any P' does not conform to 'P'), would be well-formed with implicit opening +} +``` + +could be written to explicitly open the existential, e.g., + +```swift +func hasExistentialP(p: any P) { + takesP(p) // error today ('any P' does not conform to 'P'), would still be an error + takesP(p as some P) // explicitly open the existential +} +``` + +There are two advantages to this approach over the implicit opening in this proposal. The first is that it is a purely additive feature and completely opt-in feature, which one can read and reason about when it is encountered in source code. The second is that the opened existential could persist throughout the body of the function. This would allow one to write the "open-coded" finale check from earlier in the proposal without having to factor the code into a separate (generic) function: + +```swift +func checkFinaleReadinessOpenCoded(costumes: [any Costume]) -> Bool { + for costume in costumes { + let openedCostume = costume as some Costume // type is "opened type of costume at this point" + let costumeWithBells = openedCostume.withBells() // returned type is the same as openedCostume + if !openedCostume.hasSameAdornments(costumeWithBells) { // okay, both types are known to be the same + return false + } + } + + return true +} +``` + +The type of `openedCostume` is based on the dynamic type of the the value in the variable `costume` at the point where the `as some Costume` expression occurred. That type must not be allowed to "escape" the scope where the value is created, which implies several restrictions: + +* Only non-`static` local variables can have opened existential type. Any other kind of variable can be referenced at some later point in time where the dynamic type might have changed. +* A value of opened existential type cannot be returned from a function that has an opaque result type (e.g., `some P`), because then the underlying type of the opaque type would be dependent on runtime values provided to the function. + +Additionally, having an explicit opening expression means that opened existential types become part of the user-visible type system: the type of `openedCostume` can only be reasoned about based on its constraints (`P`) and the location in the source code where the expression occurred. Two subsequent openings of the same variable would produce two different types: + +```swift +func f(eq: any Equatable) { + let x1 = eq as some Equatable + if x1 == x1 { ... } // okay + + let x2 = eq as some Equatable + if x1 == x2 { ... } // error: "eq as some Equatable" produces different types in x1 and x2 +} +``` + +An explicit opening syntax is more expressive within a single function than the proposed implicit opening, because one can work with different values that are statically known to be derived from the same opened existential without having to introduce a new generic function to do so. However, this explicitness comes with a corresponding increase in the surface area of the language: not only the expression that performs the explicit opening (`as some P`), but the notion of opened types in the type system, which has heretofore been an implementation detail of the compiler not exposed to users. + +In contrast, the proposed implicit opening improves the expressivity of the language without increasing it's effective surface area. The opening is implicit, and the opened types remain an implementation detail. + +This "alternative Considered" could perhaps be expressed as a potential future direction. Nothing in this proposal prevents us from adding explicitly opened existentials in the future, should they prove to be useful, and we would still want the implicitly opening with type erasure as described in this proposal. Should that happen, the implicit behavior in this proposal could be retroactively understood as inferring something that could be written in the explicit syntax: + +```swift +protocol Q { } + +protocol P { + associatedtype A: Q +} + +func getA(_ value: T) -> T.A { ... } + +func unwrap(p: any P) { + let a = getA(p) // implicitly the same as "getA(p as some P) as any Q" +} +``` + +### Value-dependent opening of existentials + +Implicit opening in this proposal is always scoped to a particular binding of a specific generic parameter (`T`) and is erased thereafter. For example, this means that two invocations of the same generic function on the same existential value will return values of existential type that are not (statically) known to be equivalent: + +```swift +func identity(_ value: T) -> T { value } +func testIdentity(p: any Equatable) { + let p1 = identity(p) // p1 gets type-erased type 'any Equatable' + let p2 = identity(p) // p2 gets type-erased type 'any Equatable' + if p1 == p2 { ... } // error: p1 and p2 aren't known to have the same concrete type + + let openedP1: some P = identity(p) // openedP1 has an opaque type binding to the underlying type of the call + let openedP2: some P = identity(p) // openedP2 has an opaque type binding to the underlying type of the call + if openedP1 == openedP2 { ... } // error: openedP1 and openedP2 aren't known to have the same concrete type +} +``` + +One could imagine tying the identity of the opened existential type to the *value* of the existential. For example, the two calls to `identity(p)` could produce opaque types that are identical because they are based on the underlying type of the value `p`. This is a form of dependent typing, because the (static) types of some entities are determined by their values. It begins to break down if there is any way in which the value can change, e.g., + +```swift +func identityTricks(p: any Equatable) { + let openedP1 = identity(p) // openedP1 has the underlying type of 'p' + let openedP2 = identity(p) // openedP2 has the underlying type of 'p' + if openedP1 == openedP2 { ... } // okay because both values have the underlying type of 'p' + + var q = p // q has the underlying type of 'p' + let openedQ1: some P = identity(q) // openedQ1 has the underlying type of 'q' and therefore 'p' + if openedP1 == openedQ1 { ... } // okay because both values have the underlying type of 'p' + + if condition { + q = 17 // different underlying type for 'q' + } + + let openedQ2: some P = identity(q) + if openedQ1 == openedQ2 { } // error: openedQ1 has the underlying type of 'p', but + // openedQ2 has the underlying type of 'q', which now might be different from 'p' +} +``` + +This approach is much more complex because it introduces value tracking into the type system (where was this existential value produced?), at which point mutations to variables can affect the static types in the system. + +## Revisions + +Fifth revision: + +* Note that parentheses disable the `as any P` suppression mechanism, avoiding the problem where `as any P` is both required (because type erasure lost information from the return type) and also has semantic effect (suppressing opening). + +Fourth revision: + +* Add discussion about type erasure losing constraints and the new requirement to introduce an explicit `as` coercion when the upper bound loses information. + +Third revision: + +* Only apply the source-compatibility rule, which avoids opening an existential argument when the existential box would have sufficed, in Swift 5. In Swift 6, we will open the existential argument whenever we can, providing a consistent and desirable semantics. +* Re-introduce `as any P` and `as! any P` , now that they will be useful in Swift 6. +* Clarify more about the relationship to the explicit opening syntax, which could also be a future direction. + +Second revision: + +* Remove the discussion about `type(of:)`, whose special behavior is no longer subsumed by this proposal. Weaken statements about fully subsuming `_openExistential`. +* Removed `as any P` and `as! any P` as syntaxes to suppress the implicit opening of an existential value. It isn't needed given that we only open when the existential type doesn't meet the generic function's constraints. + +First revision: + +* Describe contravariant erasure for parameters +* Describe the limitation on implicit existential opening to maintain order of evaluation +* Avoid opening an existential argument when the existential type already satisfies the conformance requirements of the corresponding generic parameter, to better maintain source compatibility +* Introduce `as any P` and `as! any P` as syntaxes to suppress the implicit opening of an existential value. +* Added discussion on the relationship with `some` parameters ([SE-0341](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md)). +* Expand discussion of an explicit opening syntax. + +## Acknowledgments + +This proposal builds on the difficult design work of [SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md), which charted most of the detailed semantics for working with values of existential type and dealing with (e.g.) covariant erasure and the restrictions that must be placed on opening existentials. Moreover, the implementation work from one of SE-0309's authors, [Anthony Latsis](https://github.com/AnthonyLatsis), formed the foundation of the implementation work for this feature, requiring only a small amount of generalization. Ensan highlighted the issue with losing information in upper bounds and [suggested an approach](https://forums.swift.org/t/se-0352-implicitly-opened-existentials/56557/7) similar to what is used here. diff --git a/proposals/0353-constrained-existential-types.md b/proposals/0353-constrained-existential-types.md new file mode 100644 index 0000000000..44a784e86e --- /dev/null +++ b/proposals/0353-constrained-existential-types.md @@ -0,0 +1,231 @@ +# Constrained Existential Types + +* Proposal: [SE-0353](0353-constrained-existential-types.md) +* Authors: [Robert Widmann](https://github.com/codafi) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.7)** +* Implementation: implemented in `main` branch, under flag `-enable-parameterized-existential-types` +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0353-constrained-existential-types/57560) + +## Introduction + +Existential types complement the Swift type system’s facilities for abstraction. Like generics, they enable a function to take and return multiple possible types. Unlike generic parameter types, existential types need not be known up front when passed as inputs to a function. Further, concrete types can be *erased* (hidden behind the interface of a protocol) when returned from a function. There has been a flurry of activity in this space with[SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md#covariant-erasure-for-associated-types) unblocking the remaining restrictions on using protocols with associated types as existential types, and [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md) paving the way for a lightweight constraint syntax for the associated types of protocols. Building directly upon those ideas, this proposal seeks to re-use the syntax of lightweight associated type constraints in the context of existential types. + +```swift +any Collection +``` + +In essence, this proposal seeks to provide the same expressive power that [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md) gives to `some` types to `any` types. + +Swift-evolution pitch thread: https://forums.swift.org/t/pitch-constrained-existential-types/56361 + +## Motivation + +Though [SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md#covariant-erasure-for-associated-types) provides the ability to use protocols with associated types freely, it does not leave any room for authors to further constrain the associated types of those protocols, creating a gap in expressiveness between generics and existentials. Consider the implementation of a type-erased stack of event producers and consumers: + +```swift +protocol Producer { + associatedtype Event + + func poll() -> Self.Event? +} + +protocol Consumer { + associatedtype Event + + func respond(to event: Self.Event) +} +``` + +If a hypothetical event system type wishes to accept an arbitrary mix of `Producer`s and an arbitrary mix of `Consumer`s, it is free to do so with existential types: + +```swift +struct EventSystem { + var producers: [any Producer] + var consumers: [any Consumer] + + mutating func add(_ producer: any Producer) { + self.producers.append(producer) + } +} +``` + +However, we run into trouble when trying to compose producers and consumers with one another. As any given `Producer` yields data of an unspecified and unrelated `Event` type when `poll`’ed, Swift will (rightly) tell us that none of our consumers can safely accept any events. One solution would be to make `EventSystem` generic over the type of events and require `Producer` and `Consumer` instances to only return those events. As it stands, this also means restricting the producers and consumers to be concrete, with the added downside of requiring us to homogenize their types - ad-hoc type erasure strikes again: + +```swift +struct EventSystem { + var producers: [AnyProducer] + var consumers: [AnyConsumer] + + mutating func add(_ producer: P) + where P.Event == Event + { + self.producers.append(AnyProducer(erasing: producer)) + } +} +``` + +In this example, we have sacrificed quite a lot for type safety - and also have to maintain two extra type erasing wrappers for producers and consumers. Really, what is missing is the ability to express the fact that the producer and consumer types don’t matter (existential types) but the data they operate on *does* (generic constraints). This is where constrained existential types shine. When combined with the power of primary associated types from [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md), it allows us to write the code we wanted to in the first place: + +```swift +struct EventSystem { + var producers: [any Producer] + var consumers: [any Consumer] + + mutating func add(_ producer: any Producer) { + self.producers.append(producer) + } +} +``` + +## Proposed solution + +Existential types will be augmented with the ability to specify constraints on their primary associated types. When an existential type appears with such constraints, they will be converted into same-type requirements. + +```swift +protocol P { } + +var xs: [any P] // "Equivalent" to [any P] where P.T == B, P.U == N, P.V == J +``` + +## Detailed design + +The syntax of existential types will be updated to accept constraint clauses. Type inference procedures will be updated to apply inference rules to generic parameters appearing as part of parameterized existential types. + +The Swift type system and runtime will accept casts from parameterized existential types to non-parameterized existential types and vice versa, as well as casts that refine any constrained primary associated types. Upcasts and downcasts to, from, and between existential types will be updated to take these additional constraints into account: + +```swift +var x: any Sequence +_ = x as any Sequence // trivially true +_ = x as! any Sequence // requires examining Sequence.Element at runtime +``` + +### Equality of constrained protocol types + +The language must define when two types that are derived differently in code are in fact the same type. In principle, it would make sense to say that two constrained protocol types are the same if and only if they have exactly the same set of possible conforming types. Unfortunately, this rule is impractical in Swift’s type system for complex technical reasons. This means that some constrained protocol types which are logically equivalent to each other will be considered different types in Swift. + +The exact rule is still being determined, but for example, it is possible that the type `any P & Q` might be considered different from the type `any P & Q` even if the associated types of these protocols are known to be equal. Because these types have equivalent logical content, however, there will be an implicit conversion between them in both directions. As a result, this is not expected to pose a large practical difficulty. + +Substitutions of constrained protocol types written with the same basic “shape”, such as `any P`and `any P` in a generic context where `T == Int`, will always be the same type. + +### Variance + +One primary use-case for constrained existential types is their the Swift Standard Library’s Collection types. The Standard Library’s *concrete* collection types have built-in support for covariant coercions. For example, + +```swift +func up(from values: [NSView]) -> [Any] { return values } +``` + +At first blush, it would seem like constrained existential types should support variance as well: + +```swift +func up(from values: any Collection) -> any Collection { return values } +``` + +But this turns out to be quite a technical feat. There is a naive implementation of this coercion that recasts the input collection as an `Array` of the appropriate type, but this would be deeply surprising and would bake the fact that `Array` is always returned into the ABI of the standard library forever. + +Constrained existential types will behave as normal generic types with respect to variance - that is, they are *invariant -* and the code above will be rejected. + +### Covariant Erasure with Constrained Existentials + +[SE-0309](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0309-unlock-existential-types-for-all-protocols.md) specifies that one can use a member on a value of existential type only when references to `Self` or its associated types are in *covariant* positions, such as the return type of a method defined a protocol. In such positions, its associated type is *erased* to its upper bound, which is the type that most closely describes the capabilities of that associated type. For example, consider a use of the `first` property on a collection. + +```swift +extension Collection {} + var first: Element? { get } +} + +func test(collection: any Collection, stringCollection: any Collection) { + let x = collection.first // Previously an error. With SE-0309, erases to 'Any?' + let y = stringCollection.first // With SE-0309, relies on Element == String to produce 'String?' +} +``` + +However, when `Self` or its associated type occurs in an *invariant* position (defined in SE-0309), one cannot use the member of an existential type unless the concrete type is known. SE-0309 provides the following example: + +```swift +var collection: any RangeReplaceableCollection = [1, 2, 3] +// error: member 'append' cannot be used on value of protocol type 'RangeReplaceableCollection' +// because it references associated type 'Element' in contravariant position; use a conformance +// constraint instead. +collection.append(4) +``` + +With constrained existentials, one could append to a `RangeReplaceableCollection`: + +```swift +var intCollection: any RangeReplaceableCollection = [1, 2, 3] +collection.append(4) // okay: the Element type is concrete (Int) within the existential +``` + +The principle here is that a use of an associated type in the member is not considered invariant if that associated type has been made concrete by the existential type. This allows the use of `append` above, because `Element` has been made concrete by the type `any RangeReplaceableCollection`. Additionally, this means that the existential type that results from erasing an associated type can make use of constrained existentials. For example: + +```swift +extension Sequence { + func eagerFilter(_ isIncluded: @escaping (Element) -> Bool) -> [Element] { ... } + func lazyFilter(_ isIncluded: @escaping (Element) -> Bool) -> some Sequence { ... } +} + +func doFilter(sequence: any Sequence, intSequence: any Sequence) { + let e1 = sequence.eagerFilter { _ in true } // error: 'Element' is used in an invariant position + let e2 = intSequence.eagerFilter { _ in true } // okay, returns '[Int]' + let l1 = sequence.lazyFilter { _ in true } // error: 'Element' is used in an invariant position + let l2 = intSequence.lazyFilter { _ in true } // okay, returns 'any Sequence' +} +``` + +The same erasure effects with the implicitly opened existentials introduced in [SE-0352](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md). + +## Effect on ABI stability + +As constrained existential types are an entirely additive concept, there is no impact upon ABI stability. + +It is worth noting that this feature requires revisions to the Swift runtime and ABI that are not backwards-compatible nor backwards-deployable to existing OS releases. + +## Alternatives considered + +Aside from the obvious of not accepting this proposal, we could imagine many different kinds of spellings to introduce same-type requirements on associated types. For example, a where-clause based approach as in: + +```swift +any (Collection where Self.Element == Int) +``` + +Syntax like this is hard to read and use in context and the problem becomes worse as it is made to compose with other existential types and constraints. Further it would conflict with the overall direction that generic constraints in Swift are taking as of [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md). Generalized constraint syntaxes are out of scope for this proposal and are mentioned later as future directions. + +## Future directions + +#### Generalized Constraints + +This proposal intentionally does not take a position on the generalized constraint syntax considered during the review of [SE-0341](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md#constraining-the-associated-types-of-a-protocol). To take one spelling: + +```swift +any Collection<.Index == String.Index> +``` + +Though when and if such a syntax is available we expect it to apply to constrained existential types. Possible designs for generalized constraints on existential types are discussed in https://forums.swift.org/t/generalized-opaque-and-existential-type-constraints/55494. + +#### Opaque Constraints + +One particularly interesting construction is the composition of opaque types and constrained existential types. This combo allows for a particularly powerful form of type abstraction: + +```swift +any Collection +``` + +This type describes any value that implements the `Collection` protocol but whose element type is an opaque instance of the `View` protocol. Today, Swift’s generics system lacks the ability to express same-type constraints with opaque types as an operand. + +#### Even More Generalized Existentials + +Constraints on existing primary associated types are hardly the only thing existential types can express. Swift’s type system can be given the ability to open arbitrary (constrained) type parameters into scope via an existential. This enables not just top-level usages as in + +```swift +any Collection +``` + +But also nested usages as in + +```swift +any Collection Collection> +``` + +Essentially enabling ad-hoc abstraction over generic types of *any shape* at any point in the program. diff --git a/proposals/0354-regex-literals.md b/proposals/0354-regex-literals.md new file mode 100644 index 0000000000..752e22f085 --- /dev/null +++ b/proposals/0354-regex-literals.md @@ -0,0 +1,668 @@ +# Regex Literals + +* Proposal: [SE-0354](0354-regex-literals.md) +* Authors: [Hamish Knight](https://github.com/hamishknight), [Michael Ilseman](https://github.com/milseman), [David Ewing](https://github.com/DaveEwing) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Upcoming Feature Flag: `BareSlashRegexLiterals` (implemented in Swift 5.8) +* Implementation: [apple/swift#42119](https://github.com/apple/swift/pull/42119), [apple/swift#58835](https://github.com/apple/swift/pull/58835) + * Bare slash syntax `/.../` available with `-enable-bare-slash-regex` +* Review: ([first pitch](https://forums.swift.org/t/pitch-regular-expression-literals/52820)) + ([second pitch](https://forums.swift.org/t/pitch-2-regex-literals/56736)) + ([first review](https://forums.swift.org/t/se-0354-regex-literals/57037)) + ([revision](https://forums.swift.org/t/returned-for-revision-se-0354-regex-literals/57366)) + ([second review](https://forums.swift.org/t/se-0354-second-review-regex-literals/57367)) + ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0354-regex-literals/58537)) + +## Introduction + +We propose the introduction of regex literals to Swift source code, providing compile-time checks and typed-capture inference. Regex literals help complete the story told in *[Regex Type and Overview][regex-type]*. + +## Motivation + +In *[Regex Type and Overview][regex-type]* we introduced the `Regex` type, which is able to dynamically compile a regex pattern: + +```swift +let pattern = #"(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)"# +let regex = try! Regex(pattern) +// regex: Regex +``` + +The ability to compile regex patterns at run time is useful for cases where it is e.g provided as user input, however it is suboptimal when the pattern is statically known for a number of reasons: + +- Regex syntax errors aren't detected until run time, and explicit error handling (e.g `try!`) is required to deal with these errors. +- No special source tooling support, such as syntactic highlighting, code completion, and refactoring support, is available. +- Capture types aren't known until run time, and as such a dynamic `AnyRegexOutput` capture type must be used. +- The syntax is overly verbose, especially for e.g an argument to a matching function. + +## Proposed solution + +A regex literal may be written using `/.../` delimiters: + +```swift +// Matches " = ", extracting the identifier and hex number +let regex = /(?[[:alpha:]]\w*) = (?[0-9A-F]+)/ +// regex: Regex<(Substring, identifier: Substring, hex: Substring)> +``` + +Forward slashes are a regex term of art. The association between forward slashes and regexes dates back to 1969's ed, the first Unix editor, and it was inherited by subsequent interactive text tools like less and vim. The syntax was also adopted by the sed language; from there it passed to Perl, and then to Ruby and Javascript. Forward slash is instantly recognizable as a regex; the only common alternative is an ordinary string literal passed to a library API, which usually has extra overhead, requires more escaping, and defers regex syntax errors to runtime. The proposed Swift regex literals do not have these limitations, so forward slash provides the right behavioral cues to developers. There are over fifty years of precedents for forward slash and very little for anything else. + +Perl and Ruby additionally allow for [user-selected delimiters](https://perldoc.perl.org/perlop#Quote-and-Quote-like-Operators) to avoid having to escape any slashes inside a regex. For that purpose, we propose the extended literal `#/.../#`. + +An extended literal, `#/.../#`, avoids the need to escape forward slashes within the regex. It allows an arbitrary number of balanced `#` characters around the literal and escape. When the opening delimiter is followed by a new line, it supports a multi-line literal where whitespace is non-semantic and line-ending comments are ignored. + +The compiler will parse the contents of a regex literal using regex syntax outlined in *[Regex Construction][internal-syntax]*, diagnosing any errors at compile time. The capture types and labels are automatically inferred based on the capture groups present in the regex. Regex literals allows editors and source tools to support features such as syntax coloring inside the literal, highlighting sub-structure of the regex, and conversion of the literal to an equivalent result builder DSL (see *[Regex builder DSL][regex-dsl]*). + +A regex literal also allows for seamless composition with the Regex DSL, enabling lightweight intermixing of a regex syntax with other elements of the builder: + +```swift +// A regex for extracting a currency (dollars or pounds) and amount from input +// with precisely the form /[$£]\d+\.\d{2}/ +let regex = Regex { + Capture { /[$£]/ } + TryCapture { + /\d+/ + "." + /\d{2}/ + } transform: { + Amount(twoDecimalPlaces: $0) + } +} +``` + +This flexibility allows for terse matching syntax to be used when it's suitable, and more explicit syntax where clarity and strong types are required. + +Due to the existing use of `/` in comment syntax and operators, there are some syntactic ambiguities to consider. While there are quite a few cases to consider, we do not feel that the impact of any individual case is sufficient to disqualify the syntax. Some of these ambiguities require a couple of source breaking language changes, and as such the `/.../` syntax requires upgrading to a new language mode in order to use. + +## Detailed design + +### Typed captures + +Regex literals have their capture types statically determined by the capture groups present. This follows a similar inference behavior to [the DSL][regex-dsl], and is explored in more detail in *[Strongly Typed Captures][strongly-typed-captures]*. We are proposing the following inference behavior for regex literals: + +- A `Substring` is always present for the entire match. +- If any captures are present, a tuple is formed with the `Substring`, with subsequent elements representing the capture types. Captures are ordered according to [their numbering][capture-numbering]. + +The type of a capture is `Substring` by default, however it gets wrapped in an optional if it is not guaranteed to have a value on a successful match. This occurs when it is nested within a quantification that may be zero, which includes `?`, `*`, and any range quantifier with a `0` lower bound, e.g `{0,n}`. It also occurs when it appears in a branch of an alternation. For example: + +```swift +let regex1 = /([ab])?/ +// regex1: Regex<(Substring, Substring?)> + +let regex2 = /([ab])|\d+/ +// regex2: Regex<(Substring, Substring?)> +``` + +A zero quantifier or alternation nested within a capture do not produce an optional capture, unless the capture itself is inside a zero quantifier or alternation: + +```swift +let regex = /([ab]*)cd/ +// regex: Regex<(Substring, Substring)> +``` + +In this case, if the `*` quantifier is matched zero times, the resulting capture will be an empty string. + +The optional wrapping does not become nested, at most one layer of optionality is applied. For example: + +```swift +let regex = /(.)*|\d/ +// regex: Regex<(Substring, Substring?)> +``` + +This behavior differs from that of the DSL, which does apply multiple layers of optionality in such cases due to a current limitation of result builders. + +### Named captures + +One additional feature of typed captures that is currently unique to the literal is the ability to infer labeled tuple elements for named capture groups. For example: + +```swift +func matchHexAssignment(_ input: String) -> (String, Int)? { + let regex = /(?[[:alpha:]]\w*) = (?[0-9A-F]+)/ + // regex: Regex<(Substring, identifier: Substring, hex: Substring)> + + guard let match = input.wholeMatch(of: regex), + let hex = Int(match.hex, radix: 16) + else { return nil } + + return (String(match.identifier), hex) +} +``` + +This allows the captures to be referenced as `match.identifier` and `match.hex`, in addition to numerically (like unnamed capture groups) as `match.1` and `match.2`. This label inference behavior is not available in the DSL, however users are able to [bind captures to named variables instead][dsl-captures]. + +### Extended delimiters `#/.../#`, `##/.../##` + +Backslashes may be used to write forward slashes within the regex literal, e.g `/foo\/bar/`. However, this can be quite syntactically noisy and confusing. To avoid this, a regex literal may be surrounded by an arbitrary number of balanced number signs. This changes the delimiter of the literal, and therefore allows the use of forward slashes without escaping. For example: + +```swift +let regex = #/usr/lib/modules/([^/]+)/vmlinuz/# +// regex: Regex<(Substring, Substring)> +``` + +The number of `#` characters may be further increased to allow the use of e.g `/#` within the literal. This is similar in style to the raw string literal syntax introduced by [SE-0200], however it has a couple of key differences. Backslashes do not become literal characters. Additionally, a multi-line literal, where whitespace and line-ending comments are ignored, is supported when the opening delimiter is followed by a newline. + +```swift +let regex = #/ + usr/lib/modules/ # Prefix + (? [^/]+) + /vmlinuz # The kernel +/# +// regex: Regex<(Substring, subpath: Substring)> +``` + +#### Escaping of backslashes + +This syntax differs from raw string literals `#"..."#` in that it does not treat backslashes as literal within the regex. A string literal `#"\n"#` represents the literal characters `\n`. However a regex literal `#/\n/#` remains a newline escape sequence. + +One of the primary motivations behind this escaping behavior in raw string literals is that it allows the contents to be easily transportable to/from e.g external files where escaping is unnecessary. For string literals, this suggests that backslashes be treated as literal by default. For regex literals however, it instead suggests that backslashes should retain their semantic meaning. This enables interoperability with regexes taken from outside your code without having to adjust escape sequences to match the delimiters used. + +With string literals, escaping can be tricky without the use of raw syntax, as backslashes may have semantic meaning to the consumer, rather than the compiler. For example: + +```swift +// Matches '\' * '=' * + +let regex = try NSRegularExpression(pattern: "\\\\\\w\\s*=\\s*\\d+", options: []) +``` + +In this case, the intent is not for the compiler to recognize any of these sequences as string literal escapes, it is instead for `NSRegularExpression` to interpret them as regex escape sequences. As such, a raw string may be used to treat the backslashes literally, allowing `NSRegularExpression` to directly process the escapes, e.g `#"\\\w\s*=\s*\d+"#`. + +However this is not an issue for regex literals, as the regex parser is the only possible consumer of such escape sequences. Such a regex can be directly spelled as: + +```swift +let regex = /\\\w\s*=\s*\d+/ +// regex: Regex +``` + +Backslashes still require escaping to be treated as literal, however we don't expect this to be as common of an occurrence as needing to write a regex escape sequence such as `\s`, `\w`, or `\p{...}`, within a regex literal with extended delimiters `#/.../#`. + +#### Multi-line literals + +Extended regex delimiters additionally support a multi-line literal when the opening delimiter is followed by a new line. For example: + +```swift +let regex = #/ + # Match a line of the format e.g "DEBIT 03/03/2022 Totally Legit Shell Corp $2,000,000.00" + (? \w+) \s\s+ + (? \S+) \s\s+ + (? (?: (?!\s\s) . )+) \s\s+ # Note that account names may contain spaces. + (? .*) +/# +``` + +In such a literal, [extended regex syntax][extended-regex-syntax] `(?x)` is enabled. This means that whitespace in the regex becomes non-semantic (including within character classes), and end-of-line comments are supported with `# comment` syntax. + +This mode is supported with any (non-zero) number of `#` characters in the delimiter. Similar to multi-line strings introduced by [SE-0168], the closing delimiter must appear on a new line. To avoid parsing confusion, such a literal will not be parsed if a closing delimiter is not present. This avoids inadvertently treating the rest of the file as regex if you only type the opening. + +Extended syntax in such a literal may not be disabled with `(?-x)`, however it may be disabled for the contents of a group `(?-x:...)` or quoted sequence `\Q...\E`, as long as they do not span multiple lines. Supporting semantic whitespace over multiple lines would require stripping leading and trailing whitespace while maintaining the verbatim newlines. This could feasibly be supported, however we feel that its behavior could potentially be confusing. + +If desired, newlines may be written using `\n`, or by using a backslash to escape the literal newline character: + +```swift +let regex = #/ + a\ + b\ + c +/# +// regex = /a\nb\nc/ +``` + +### Ambiguities of `/.../` with comment syntax + +Line comment syntax `//` and block comment syntax `/*` will continue to be parsed as comments. An empty regex literal is not a particularly useful thing to express, but can be written as `#//#` if desired. `*` would be an invalid starting character of a regex, and therefore does not pose an issue. + +A parsing conflict does however arise when a block comment surrounds a regex literal ending with `*`, for example: + + ```swift + /* + let regex = /[0-9]*/ + */ + ``` + +In this case, the block comment prematurely ends on the second line, rather than extending all the way to the third line as the user would expect. This is already an issue today with `*/` in a string literal, though it is more likely to occur in a regex given the prevalence of the `*` quantifier. This issue can be avoided in many cases by using line comment syntax `//` instead, which it should be noted is the syntax that Xcode uses when commenting out multiple lines. + + +### Ambiguity of `/.../` with infix operators + +There is a minor ambiguity when infix operators are used with regex literals. When used without whitespace, e.g `x+/y/`, the expression will be treated as using an infix operator `+/`. Whitespace is therefore required for regex literal interpretation, e.g `x + /y/`. Alternatively, extended literals may be used, e.g `x+#/y/#`. + +### Regex syntax limitations in `/.../` + +In order to help avoid a parsing ambiguity, a `/.../` regex literal will not be parsed if it starts or ends with a space or tab character. This restriction may be avoided by using the extended `#/.../#` literal. + +#### Rationale + +The restriction on the ending character helps avoid breaking source compatibility with prefix and infix `/` operators in certain cases. Such cases are explored in the next section. The restriction on the starting character is due to a parsing ambiguity that arises when a `/.../` regex literal starts a new line. This is particularly problematic for result builders, where we expect it to be frequently used, in particular within a `Regex` builder: + +```swift +let digit = Regex { + TryCapture(OneOrMore(.digit)) { Int($0) } +} +// Matches against + (' + ' | ' - ') + +let regex = Regex { + digit + / [+-] / + digit +} +``` + +Instead of being parsed as 3 result builder elements, the second of which being a regex literal, this is instead parsed as a single operator chain with the operands `digit`, `[+-]`, and `digit`. This will therefore be diagnosed as semantically invalid. + +To avoid this issue, a regex literal may not start with a space or tab character. If a space or tab is needed as the first character, it must be either escaped, e.g: + +```swift +let regex = Regex { + digit + /\ [+-] / + digit +} +``` + +or an extended literal must be used, e.g: + +```swift +let regex = Regex { + digit + #/ [+-] /# + digit +} +``` + +This restriction takes advantage of the fact that infix operators require consistent spacing on either side. This includes both space characters as well as newlines. For example: + +```swift +let a = 0 + 1 // Valid +let b = 0+1 // Also valid +let c = 0 ++ 1 // Valid operator chain because the newline before '+' is whitespace. + +let d = 0 +1 // Not valid, '+' is treated as prefix, which cannot then appear next to '0'. +let e = 0+ 1 // Same but postfix +let f = 0 ++1 // Not a valid operator chain, same as 'd', except '+1' is no longer sequenced with '0'. +``` + +In much the same way as `f`, by requiring the first character of a regex literal not to be space or tab, we ensure it cannot be treated as an operator chain: + +```swift +let g = 0 +/1 + 2/ // Must be a regex +``` + +### How `/.../` is parsed + +A `/.../` regex literal will be parsed when an opening `/` is encountered in expression position, and there is a closing `/` present. As such, the following will continue to parse as normal: + +```swift +// Infix '/' is never in an expression position in valid code (unless unapplied). +let a = 1 / 2 / 3 + +// None of these '/^/' cases are in expression position. +infix operator /^/ +func /^/ (lhs: Int, rhs: Int) -> Int { 0 } +let b = 0 /^/ 1 + +// Also fine. +prefix operator / +prefix func / (_ x: Int) -> Int { x } +let c = /0 // No closing '/', so not a regex literal. The '//' of this comment doesn't count either. +``` + +But `let r = /^/` will be parsed as a regex. + +A regex literal may be used with a prefix operator, e.g `let r = ^^/x/` is parsed as `let r = ^^(/x/)`. In this case, when encountering operator characters containing `/` in an expression position, the characters up to the first `/` are split into a prefix operator, and regex literal parsing continues as normal. + +As already discussed, a regex literal may not start or end with a space or tab. This means that the following will continue to be parsed as normal: + +```swift +// Unapplied '/' in a call to 'reduce': +let x = array.reduce(1, /) / 5 +let y = array.reduce(1, /) + otherArray.reduce(1, /) + +// Prefix '/' with another '/' on the same line: +foo(/a, /b) +bar(/x) / 2 + +// Unapplied operators: +baz(!/, 1) / 2 +qux(/, /) +qux(/^, /) +qux(!/, /) + +let d = hasSubscript[/] / 2 // Unapplied infix '/' and infix '/' + +let e = !/y / .foo() // Prefix '!/' with infix '/' and operand '.foo()' +``` + +However this is not sufficient to disambiguate cases such as: + +```swift +// Prefix '/' used multiple times on the same line without trailing whitespace: +(/x).foo(/y) +bar(/x) + bar(/y) + +// Cases where the closing '/' is not used with whitespace: +bar(/x)/2 +baz(!/, 1)/2 + +// Prefix '/^' with postfix '/': +let f = (/^x)/ +``` + +In all of these cases, the opening `/` appears in expression position, and there is a potential closing `/` that is used without whitespace. To avoid source breakage for such cases, one further heuristic is employed. A regex literal will not be parsed if it contains an unbalanced `)`. This takes both escapes and custom character classes into consideration, and therefore only applies to syntax that would already be invalid for a regex. As such, all of the above cases will continue to be parsed as normal. + +This additional heuristic also allows for straightforward disambiguation in source breaking cases where the regex is valid. For example, the following cases will become regex literals: + +```swift +foo(/a, b/) // Will become regex literal '/a, b/' +qux(/, !/) // Will become regex literal '/, !/' +qux(/,/) // Will become regex literal '/,/' + +let g = hasSubscript[/]/2 // Will become regex literal '/]/' + +let h = /0; let f = 1/ // Will become the regex literal '/0; let y = 1/' +let i = /^x/ // Will become the regex literal '/^x/' +``` + +However they can be readily disambiguated by inserting parentheses: + +```swift +// Now a prefix and postfix '/': +foo((/a), b/) + +// Now unapplied operators: +qux((/), !/) +qux((/),/) +let g = hasSubscript[(/)]/2 + +let h = (/0); let f = 1/ // Now prefix '/' and postfix '/' +let i = (/^x)/ // Now prefix '/^' and postfix '/' +``` + +or, in some cases, by inserting whitespace: + +```swift +qux(/, /) +let g = hasSubscript[/] / 2 +``` + +We however expect these cases will be fairly uncommon. A similar case is the use of an unapplied infix operator with two `/` characters, for example: + +```swift +baz(/^/) // Will become the regex literal '/^/' rather than an unapplied operator +``` + +This cannot be disambiguated with parentheses or whitespace, however it can be disambiguated using a closure. For example: + +```swift +baz({ $0 /^/ $1 }) // Is now infix '/^/' +``` + +This takes advantage of the fact that a regex literal will not be parsed in an infix operator position. + +## Source Compatibility + +As explored above, the parsing of `/.../` does have potential to break source in cases where all of the following hold: + +- `/` appears in an expression position. +- There is a closing `/` on the same line. +- The first and last character of the literal is not space or tab. +- There are no unbalanced `)` characters within the literal. + +However we expect these cases will be uncommon, and can be disambiguated with parentheses or closures if needed. + +To accommodate the cases where source may be broken, `/.../` regex literals will be introduced in Swift 6 mode. However, projects may adopt the syntax earlier by passing the compiler flag `-enable-bare-slash-regex` or the [upcoming feature flag](0362-piecemeal-future-features.md) `BareSlashRegexLiterals`. Note this does not affect the extended delimiter syntax `#/.../#`, which will be usable immediately. + +## Future Directions + +### Modern literal syntax + +We could support a more modern Swift-like syntax in regex literals. For example, comments could be done with `//` and `/* ... */`, and quoted sequences could be done with `"..."`. This would however be incompatible with the syntactic superset of regex syntax we intend to parse, and as such may need to be introduced using a new literal kind, with no obvious choice of delimiter. + +However, such a syntax would lose out on the familiarity benefits of standard regex, and as such may lead to an "uncanny valley" effect. It's also possible that the ability to use regex literals in the DSL lessens the benefit that this syntax would bring. + +### Typed captures for duplicate named group + +PCRE allows duplicate capture group names when `(?J)` is set. However this would be incompatible with labeled tuple elements for the captures, as tuples may not have duplicate names. Given we do not currently support `(?J)` in regex literals, the handling of typed captures here is left as future work. + +### Typed captures for branch reset alternations + +PCRE and Perl support a branch reset construct `(?|(a)|(b))` where a child alternation resets the capture numbering for each branch, allowing `(a)` and `(b)` to share the same capture number. This would require unifying their types for the purposes of typed captures. Given we do not currently support this construct, the handling of typed captures here is left as future work. + +### Library-extensible protocol support + +A regex literal describes a string processing algorithm which can be ran over some model of String. The precise semantics of running over extended grapheme clusters vs Unicode scalar values is part of [Unicode for String Processing][regex-unicode]. Libraries may wish to extend this behavior, but the approach presented by various `ExpressibleBy*` protocols is underpowered as libraries would need access to the structure of the algorithm itself. + +A better (and future) approach is to open up the regex parser's AST, API, and AST actions to libraries. Here's some examples of why a library might want to customize regex: + +A library may wish to provide support for a different or higher level model of string. For example, using localized comparison or tailored grapheme-cluster breaks. Such a use case would need access to the structure of the string processing algorithm literal. + +A library may wish to provide support for running over another engine, such as ICU, PCRE, or Javascript. Such a use case would want to pretty-print Swift's regex syntax into one of these syntax variants. + +A library may wish to provide their own higher-level structure around which regex literals can be embedded for the purpose of multi-tier processing. For example, processing URLs where regex literal-character portions would be converted into percent-encoded equivalents (with some kind of character class customization/mapping as well). Additionally, a library may have the desire to explicitly delineate patterns that evaluate within a component vs patterns spanning multiple components. Such an approach would benefit from access to the real AST and rich semantic API. + +## Alternatives Considered + +### Alternative delimiter to `/.../` + +Given the fact that `/.../` is an existing term of art for regular expressions, we feel it should be the preferred delimiter syntax. It should be noted that the syntax has become less popular in some communities such as Perl, however we still feel that it is a compelling choice, especially with extended delimiters `#/.../#`. Additionally, while there are some syntactic ambiguities, we do not feel they are sufficient to disqualify the syntax. To evaluate this trade-off, below is a list of alternative delimiters that would not have the same ambiguities, and would not therefore require source breaking changes. + +#### Extended literal delimiters only `#/.../#` + +We could choose to avoid adding the bare forward slash syntax, and instead require at least one `#` character to be present in the delimiter. This would retain some of the familiarity of `/.../` while avoiding the parsing ambiguities and source breaking changes. + +However we feel that `/.../` is the better choice of default syntax, especially for simple regex where the additional noise of the `#` characters would be undesirable. While there are some parsing ambiguities to contend with, we do not feel they outweigh the benefits of having a lightweight and instantly recognizable syntax for regex. + +#### Prefixed quote `re'...'` + +We could choose to use `re'...'` delimiters, for example: + +```swift +// Matches " = ", extracting the identifier and hex number +let regex = re'([[:alpha:]]\w*) = ([0-9A-F]+)' +``` + +The use of two letter prefix could potentially be used as a namespace for future literal types. It would also have obvious extensions to extended and multi-line literals using `re#'...'#` and `re'''...'''` respectively. However, it is unusual for a Swift literal to be prefixed in this way. We also feel that its similarity to a string literal might have users confuse it with a raw string literal. + +Also, there are a few items of regex grammar that use the single quote character as a metacharacter. These include named group definitions and references such as `(?'name')`, `(?('name'))`, `\g'name'`, `\k'name'`, as well as callout syntax `(?C'arg')`. The use of a single quote conflicts with the `re'...'` delimiter as it will be considered the end of the literal. However, alternative syntax exists for all of these constructs, e.g `(?)`, `\k`, and `(?C"arg")`. Those could be required instead. An extended regex literal syntax e.g `re#'...'#` would also avoid this issue. + +#### Prefixed double quote `re"...."` + +This would be a double quoted version of `re'...'`, more similar to string literal syntax. This has the advantage that single quote regex syntax e.g `(?'name')` would continue to work without requiring the use of the alternative syntax or extended literal syntax. However it could be argued that regex literals are distinct from string literals in that they introduce their own specific language to parse. As such, regex literals are more like "program literals" than "data literals", and the use of single quote instead of double quote may be useful in expressing this difference. + +#### Single letter prefixed quote `r'...'` + +This would be a slightly shorter version of `re'...'`. While it's more concise, it could potentially be confused to mean "raw", especially as Python uses this syntax for raw strings. + +#### Single quotes `'...'` + +This would be an even more concise version of `re'...'` that drops the prefix entirely. However, given how close it is to string literal syntax, it may not be entirely clear to users that `'...'` denotes a regex as opposed to some different form of string literal (e.g some form of character literal, or a string literal with different escaping rules). + +We could help distinguish it from a string literal by requiring e.g `'/.../'`, though it may not be clear that the `/` characters are part of the delimiters rather than part of the literal. Additionally, this would potentially rule out the use of `'...'` as a future literal kind. + +#### Magic literal `#regex(...)` + +We could opt for for a more explicitly spelled out literal syntax such as `#regex(...)`. This is a more heavyweight option, similar to `#selector(...)`. As such, it may be considered syntactically noisy as e.g a function argument `str.match(#regex([abc]+))` vs `str.match(/[abc]+/)`. + +Such a syntax would require the containing regex to correctly balance parentheses for groups, otherwise the rest of the line might be incorrectly considered a regex. This could place additional cognitive burden on the user, and may lead to an awkward typing experience. For example, if the user is editing a previously written regex, the syntax highlighting for the rest of the line may change, and unhelpful spurious errors may be reported. With a different delimiter, the compiler would be able to detect and better diagnose unbalanced parentheses in the regex. + +We could avoid the parenthesis balancing issue by requiring an additional internal delimiter such as `#regex(/.../)`. However this is even more heavyweight, and it may be unclear that `/` is part of the delimiter rather than part of an argument. Alternatively, we could replace the internal delimiter with another character such as ```#regex`...` ```, `#regex{...}`, or `#regex/.../`. However those would be inconsistent with the existing `#literal(...)` syntax and the first two would overload the existing meanings for the ``` `` ``` and `{}` delimiters. + +It should also be noted that `#regex(...)` would introduce a syntactic inconsistency where the argument of a `#literal(...)` is no longer necessarily valid Swift syntax, despite being written in the form of an argument. + +##### On future extensibility to other foreign language snippets + +One of the benefits of `#regex(...)` or `re'...'` is the extensibility to other kinds of foreign language snippets, such as SQL. Nothing in this proposal precludes a scalable approach to foreign language snippets using `#lang(...)` or `lang'...'`. If or when that happens, regex could participate as well, but the proposed syntax would still be valuable as regex literals *are* unique in their prevalence as fragments passed directly to API, as well as components of a result builder DSL. + + +#### Shortened magic literal `#(...)` + +We could reduce the visual weight of `#regex(...)` by only requiring `#(...)`. However it would still retain the same issues, such as still looking potentially visually noisy as an argument, and having suboptimal behavior for parenthesis balancing. It is also not clear why regex literals would deserve such privileged syntax. + +#### Double slash `// ... //` + +Rather than using single forward slash delimiters `/.../`, we could use double slash delimiters. This would have previously been comment syntax, and would therefore be potentially source breaking. In particular, file header comments frequently use this style. Even if they successfully parse as a regex, they would receive different syntax highlighting, and emit a spurious error about being unused. + +This would also significantly impact a variety of commonly occurring comments, some examples from the Swift repository include: + +```swift +// rdar://41219750 + +// Please submit a bug report (https://swift.org/contributing/#reporting-bugs) + +// let pt = CGPoint(x: 1.0, y: 2.0) // Here we query for CGFloat. +``` + +This syntax also means the editor would not be able to automatically complete the closing delimiter, as it would initially appear to be a regular comment. This further means that typing the literal would receive comment syntax highlighting until the closing delimiter is written. + +#### Reusing string literal syntax + +Instead of supporting a first-class literal kind for regex, we could instead allow users to write a regex in a string literal, and parse, diagnose, and generate the appropriate code when it's coerced to the `Regex` type. + +```swift +let regex: Regex = #"([[:alpha:]]\w*) = ([0-9A-F]+)"# +``` + +However we decided against this because: + +- We would not be able to easily apply custom syntax highlighting and other editor features for the regex syntax. +- It would require a `Regex` contextual type to be treated as a regex, otherwise it would be defaulted to `String`, which may be undesired. +- In an overloaded context it may be ambiguous or unclear whether a string literal is meant to be interpreted as a literal string or regex. +- Regex-specific escape sequences such as `\w` would likely require the use of raw string syntax `#"..."#`, as they are otherwise invalid in a string literal. +- It wouldn't be compatible with other string literal features such as interpolations. + +### No custom literal + +Instead of adding a custom regex literal, we could require users to explicitly write `try! Regex("[abc]+")`. This would be similar to `NSRegularExpression`, and loses all the benefits of parsing the literal at compile time. This would mean: + +- No source tooling support (e.g syntax highlighting, refactoring actions) would be available. +- Parse errors would be diagnosed at run time rather than at compile time. +- We would lose the type safety of typed captures. +- More verbose syntax is required. + +We therefore feel this would be a much less compelling feature without first class literal support. + +### Non-semantic whitespace by default for single-line literals + +We could choose to enable non-semantic whitespace by default for single-line literals, matching the behavior of multi-line literals. While this is quite compelling for better readability, we feel that it would lose out on the familiarity and compatibility of the single-line literal. + +Non-semantic whitespace can always be enabled explicitly with `(?x)`: + +```swift +let r = /(?x) abc | def/ +``` + +or by writing a multi-line literal: + +```swift +let r = #/ + abc | def +/# +``` + +### Multi-line literal with semantic whitespace by default + +We could choose semantic whitespace by default within a multi-line regex literal. Such a literal would require a whitespace stripping rule, while keeping newlines of the contents verbatim. To enable non-semantic whitespace in such a literal, you would either have to explicitly write `(?x)` at the very start of the literal: + +```swift +let regex = #/ +(?x) abc | def +/# +``` + +Or we could support an explicit specifier as part of the delimiter syntax. For example: + +```swift +let regex = #/x + abc | def +/# +``` + +However, we don't find either of these options particularly compelling. The former is somewhat verbose considering we expect it to be a common mode for multi-line literals, and it would change meaning if indented at all. The latter wouldn't extend to other matching options, and wouldn't be usable within a single-line literal. + +We ultimately feel that non-semantic whitespace is a much more useful default for a multi-line regex literal, and unlike the single-line case, does not lose out on compatibility or familiarity. We could still enforce the specification of `(?x)` or `x`, however they would retain the same drawbacks. We are therefore not convinced they would be beneficial, and feel that the literal being split over multiple lines provides enough signal to indicate different semantics. + +#### Supporting the full matching option syntax as part of the delimiter + +Rather than supporting a specifier such as `x` on the delimiter, we could support the full range of matching option syntax on the delimiter. For example: + +```swift +let regex = #/(?xi) + abc | def +/# +``` + +However this would be more verbose, and would add additional complexity to the lexing logic which needs to be able to distinguish between an unterminated single-line literal, and a multi-line literal. It would also be limited to the isolated syntax, and e.g wouldn't support `(?xi:...)`. As we expect non-semantic whitespace to be the frequently desired mode in such a literal, we are not convinced the extra complexity or verbosity is beneficial. + +### Allow matching option flags on the literal `/.../x` + +We could choose to support Perl-style specification of matching options on the literal. This could feasibly be supported without introducing source compatibility issues, as identifiers cannot normally be sequenced with a regex literal. However it is unusual for a Swift literal to be suffixed like that. For matching options that affect runtime matching, e.g `i`, we intend on exposing API such as `/.../.ignoresCase()`. The only remaining options that affect parsing instead of matching are `x`, `xx`, `n`, and `J`. These cannot be exposed as API, however the multi-line literal already provides a way to enter extended syntax mode, and we feel writing `(?n)` or `(?J)` at the start of the literal is a suitable alternative to `/.../n` and `/.../J`. + +For the multi-line literal, we could require the specification of the `x` flag to enable extended syntax mode. However this would still require the `#/` delimiter. As such, it would lose out on the familiarity of the `/.../x` syntax, and wouldn't provide much visual signal for non-trivial literals. For example: + +```swift +let regex = #/ + # Match a line of the format e.g "DEBIT 03/03/2022 Totally Legit Shell Corp $2,000,000.00" + (? \w+) \s\s+ + (? \S+) \s\s+ + (? (?: (?!\s\s) . )+) \s\s+ # Note that account names may contain spaces. + (? .*) +/x# +``` + +### Using `///` or `#///` for multi-line + +Instead of re-using the extended delimiter syntax `#/.../#` for multi-line regex literals, we could choose a delimiter that more closely parallels the multi-line string delimiter `"""`. `///` would be the obvious choice, but unfortunately already signifies a documentation comment. As such, it would likely not be viable without further heuristics and regex syntax limitations. A possible alternative is to require at least one `#` character, e.g `#///`. This would be more syntactically viable at the cost of being more verbose. However it may seem odd that a `///` delimiter does not exist for such a literal. + +In either case, we are not convinced that drawing a parallel to multi-line string literals is particularly desirable, as multi-line regex literals have considerably different semantics. For example, whitespace is non-semantic and backslashes treat newlines as literal, rather than eliding them: + +```swift +let str = """ + a\ + b\ + c +""" +// str = " a b c" + +let re = #/ + a\ + b\ + c +/# +// re = /a\nb\nc/ +``` + +For multi-line string literals, the two main reasons for choosing `"""` over `"` were: + +1. Editing: It was felt that typing `"` and temporarily messing up the source highlighting of the rest of the file was a bad experience. +2. Visual weight: It was felt that a single `"` written after potentially paragraphs of text would be difficult to notice. + +However we do not feel that these are serious issues for regex literals. The `#/` delimiter has plenty of visual weight, and we require a closing `/#` before the literal is treated as multi-line. While it may be possible for the closing `/#` of an existing multi-line regex literal to be treated as a closing delimiter when typing `#/` above, we feel such cases will be quite a bit less common than the string literal `"` case. + +### No multi-line literal + +We could choose to only support single-line regex literals, with more complex multi-line cases requiring the DSL. However we feel that the ability to write non-semantic whitespace multi-line regex literals is quite a compelling feature that is not covered by the DSL. We feel that confining the literal's ability to work with non-semantic whitespace to the single-line case would lose a lot of the benefits of the extended syntax. + +### Restrict feature set to that of the builder DSL + +The regex builder DSL is unable to provide some of the features presented such as named captures as tuble labels. An alternative could be to cut those features from the literal out of concern they may lead to an over-use of the literals. However, to do so would remove the clearest demonstration of the need for better type-level operations including working with labeled tuples. + +Similarly, there is no literal equivalent for some of the regex builder features, but that isn't an argument against them. The regex builder DSL has references which serves this role (though not as concisely) and they are useful beyond just naming captures. + +Regex literals should not be outright avoided, they should be used well. Artificially hampering their usage doesn't provide any benefit and we wouldn't want to lock these limitations into Swift's ABI. + + + +[SE-0168]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0168-multi-line-string-literals.md +[SE-0200]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0200-raw-string-escaping.md + +[pitch-status]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md +[regex-type]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md +[strongly-typed-captures]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/StronglyTypedCaptures.md +[regex-unicode]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md#unicode-for-string-processing + +[internal-syntax]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md +[extended-regex-syntax]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#extended-syntax-modes + +[capture-numbering]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#group-numbering + +[regex-dsl]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0351-regex-builder.md +[dsl-captures]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0351-regex-builder.md#capture-and-reference diff --git a/proposals/0355-regex-syntax-run-time-construction.md b/proposals/0355-regex-syntax-run-time-construction.md new file mode 100644 index 0000000000..a3ed0c09ce --- /dev/null +++ b/proposals/0355-regex-syntax-run-time-construction.md @@ -0,0 +1,1075 @@ +# Regex Syntax and Run-time Construction + +* Proposal: [SE-0355](0355-regex-syntax-run-time-construction.md) +* Authors: [Hamish Knight](https://github.com/hamishknight), [Michael Ilseman](https://github.com/milseman) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Implementation: https://github.com/apple/swift-experimental-string-processing + * Available in nightly toolchain snapshots with `import _StringProcessing` +* Review: ([first pitch](https://forums.swift.org/t/pitch-regex-syntax/55711)) + ([second pitch](https://forums.swift.org/t/pitch-2-regex-syntax-and-run-time-construction/56624)) + ([review](https://forums.swift.org/t/se-0355-regex-syntax-and-runtime-construction/57038)) + ([acceptance](https://forums.swift.org/t/accepted-se-0355-regex-syntax-and-runtime-construction/59232)) + +## Introduction + +A regex declares a string processing algorithm using syntax familiar across a variety of languages and tools throughout programming history. We propose the ability to create a regex at run time from a string containing regex syntax (detailed here), API for accessing the match and captures, and a means to convert between an existential capture representation and concrete types. + +The overall story is laid out in [SE-0350 Regex Type and Overview][overview] and each individual component is tracked in [Pitch and Proposal Status][pitches]. + +## Motivation + +Swift aims to be a pragmatic programming language, striking a balance between familiarity, interoperability, and advancing the art. Swift's `String` presents a uniquely Unicode-forward model of string, but currently suffers from limited processing facilities. + +`NSRegularExpression` can construct a processing pipeline from a string containing [ICU regular expression syntax][icu-syntax]. However, it is inherently tied to ICU's engine and thus it operates over a fundamentally different model of string than Swift's `String`. It is also limited in features and carries a fair amount of Objective-C baggage, such as the need to translate between `NSRange` and `Range`. + +```swift +let pattern = #"(\w+)\s\s+(\S+)\s\s+((?:(?!\s\s).)*)\s\s+(.*)"# +let nsRegEx = try! NSRegularExpression(pattern: pattern) + +func processEntry(_ line: String) -> Transaction? { + let range = NSRange(line.startIndex.. + +let regex: Regex<(Substring, Substring, Substring, Substring, Substring)> = + try! Regex(pattern) +``` + +### Syntax + +We propose accepting a syntactic "superset" of the following existing regular expression engines: + +- [PCRE 2][pcre2-syntax], an "industry standard" and a rough superset of Perl, Python, etc. +- [Oniguruma][oniguruma-syntax], a modern engine with additional features. +- [ICU][icu-syntax], used by NSRegularExpression, a Unicode-focused engine. +- [.NET][.net-syntax], which adds delimiter-balancing and some interesting minor details around conditional patterns. + +To our knowledge, all other popular regex engines support a subset of the above syntaxes. + +We also support [UTS#18][uts18]'s full set of character class operators (to our knowledge no other engine does). Beyond that, UTS#18 deals with semantics rather than syntax, and what syntax it uses is covered by the above list. We also parse Java's properties (e.g. `\p{javaLowerCase}`), meaning we support a superset of Java 8 as well. + +Note that there are minor syntactic incompatibilities and ambiguities involved in this approach. Each is addressed in the relevant sections below. + +Regex syntax will be part of Swift's source-compatibility story as well as its binary-compatibility story. Thus, we present a detailed and comprehensive design. + +## Detailed Design + +We propose initializers to declare and compile a regex from syntax. Upon failure, these initializers throw compilation errors, such as for syntax or type errors. API for retrieving error information is future work. + +```swift +extension Regex { + /// Parse and compile `pattern`, resulting in a strongly-typed capture list. + public init(_ pattern: String, as: Output.Type = Output.self) throws +} +extension Regex where Output == AnyRegexOutput { + /// Parse and compile `pattern`, resulting in a type-erased capture list. + public init(_ pattern: String) throws +} +``` + +We propose `AnyRegexOutput` for capture types not known at compilation time, alongside casting API to convert to a strongly-typed capture list. + +```swift +/// A type-erased regex output +public struct AnyRegexOutput { + /// Creates a type-erased regex output from an existing match. + /// + /// Use this initializer to fit a strongly-typed regex match into the + /// use site of a type-erased regex output. + public init(_ match: Regex.Match) + + /// Returns a strongly-typed output by converting type-erased values to the specified type. + /// + /// - Parameter type: The expected output type. + /// - Returns: The output, if the underlying value can be converted to the + /// output type; otherwise `nil`. + public func extractValues( + as type: Output.Type = Output.self + ) -> Output? +} + +extension AnyRegexOutput: RandomAccessCollection { + /// An individual type-erased output value. + public struct Element { + /// The range over which a value was captured. `nil` for no-capture. + public var range: Range? { get } + + /// The slice of the input over which a value was captured. `nil` for no-capture. + public var substring: Substring? { get } + + /// The captured value. `nil` for no-capture. + public var value: Any? { get } + + /// The name of this capture, if it has one, otherwise `nil`. + public var name: String? + } + + // Trivial collection conformance requirements + + public var startIndex: Int { get } + + public var endIndex: Int { get } + + public var count: Int { get } + + public func index(after i: Int) -> Int + + public func index(before i: Int) -> Int + + public subscript(position: Int) -> Element { get } +} +``` + +We propose adding an API to `Regex` and `Regex.Match` to cast the output type to a concrete one. A regex match will lazily create a `Substring` on demand, so casting the match itself saves ARC traffic vs extracting and casting the output. + +```swift +extension Regex.Match where Output == AnyRegexOutput { + /// Creates a type-erased regex match from an existing match. + /// + /// Use this initializer to fit a regex match with strongly-typed captures into the + /// use site of a type-erased regex match. + public init(_ match: Regex.Match) +} + +extension Regex where Output == AnyRegexOutput { + /// Creates a type-erased regex from an existing regex. + /// + /// Use this initializer to fit a regex with strongly-typed captures into the + /// use site of a type-erased regex, i.e. one that was created from a string. + public init(_ regex: Regex) +} + +extension Regex { + /// Creates a strongly-typed regex from a type-erased regex. + /// + /// Use this initializer to create a strongly-typed regex from + /// one that was created from a string. Returns `nil` if the types + /// don't match. + public init?(_ erased: Regex, as: Output.Type = Output.self) +} +``` + +We propose adding API to query and access captures by name in an existentially typed regex and match: + +```swift +extension Regex where Output == AnyRegexOutput { + /// Returns whether a named-capture with `name` exists. + public func contains(captureNamed name: String) -> Bool +} + +extension Regex.Match where Output == AnyRegexOutput { + /// Access a capture by name. Returns `nil` if there's no capture with that name. + public subscript(_ name: String) -> AnyRegexOutput.Element? { get } +} + +extension AnyRegexOutput { + /// Access a capture by name. Returns `nil` if no capture with that name was present in the Regex. + public subscript(_ name: String) -> AnyRegexOutput.Element? { get } +} +``` + +Finally, we propose API for creating a regex containing literal string content. This produces an equivalent regex to a string literal embedded in the result builder DSL. As this is much less common than run-time compilation or an embedded literal in the DSL, it has an explicit argument label. + +```swift +extension Regex { + /// Produces a regex that matches `verbatim` exactly, as though every + /// metacharacter in it was escaped. + public init(verbatim: String) +} +``` + +The rest of this proposal will be a detailed and exhaustive definition of our proposed regex syntax. + +
Grammar Notation + +For the grammar sections, we use a modified PEG-like notation, in which the grammar also describes an unambiguous top-down parsing algorithm. + +- ` -> ` gives the definition of `Element` +- The `|` operator specifies a choice of alternatives +- `'x'` is the literal character `x`, otherwise it's a reference to x + + A literal `'` is spelled `"'"` +- Postfix `*` `+` and `?` denote zero-or-more, one-or-more, and zero-or-one +- Range quantifiers, like `{1...4}`, use Swift range syntax as convention. +- Basic custom character classes are written like `[0-9a-zA-Z]` +- Prefix `!` operator means the next element must not appear (a zero-width assertion) +- Parenthesis group for the purposes of quantification +- Builtins use angle brackets: + - `` refers to an integer, `` a character, etc. + - `` is any whitespace character + - `` is the end-of-line anchor (e.g. `$` in regex). + +For example, `(!'|' !')' ConcatComponent)*` means any number (zero or more) occurrences of `ConcatComponent` so long as the initial character is neither a literal `|` nor a literal `)`. + +
+ +### Top-level regular expression + +``` +Regex -> GlobalMatchingOptionSequence? RegexNode +RegexNode -> '' | Alternation +Alternation -> Concatenation ('|' Concatenation)* +Concatenation -> (!'|' !')' ConcatComponent)* +``` + +A regex may be prefixed with a sequence of [global matching options](#pcre-global-matching-options). Its contents can be empty or a sequence of alternatives separated by `|`. + +Alternatives are a series of expressions concatenated together. The concatenation ends with either a `|` denoting the end of the alternative or a `)` denoting the end of a recursively parsed group. + +Alternation has a lower precedence than concatenation or other operations, so e.g `abc|def` matches against `abc` or `def`. + +### Concatenated subexpressions + +``` +ConcatComponent -> Trivia | Quote | Interpolation | Quantification + +Trivia -> Comment | NonSemanticWhitespace +Comment -> '(?#' (!')')* ')' | EndOfLineComment +Interpolation -> '<{' (!'}>')* '}>' + +(extended syntax only) EndOfLineComment -> '#' (! .)* +(extended syntax only) NonSemanticWhitespace -> + + +Quote -> '\Q' (!'\E' .)* '\E'? +``` + +Each component of a concatenation may be "trivia" (comments and non-semantic whitespace, if applicable), a quoted run of literal content, or a potentially-quantified subexpression. + +In-line comments, similarly to C, are lexical and are not recursively nested like normal groups are. A closing `)` cannot be escaped. + +Quotes are similarly lexical, non-nested, and the `\` before a `\E` cannot be escaped. For example, `\Q^[xy]+$\E`, is treated as the literal characters `^[xy]+$` rather than an anchored quantified character class. `\Q\\E` is a literal `\`. A quoted sequence `\Q` may not have a closing `\E`, in which case it extends to the end of the regex. A quote may appear in a custom character class, but such a quote may not be empty. + +An interpolation sequence `<{...}>` is syntax that is reserved for a potential future interpolation feature. As such, the details surrounding it are future work, and it will currently be rejected for both literals and run-time compiled patterns. It may however be made available in the future as the literal characters. + +### Quantified subexpressions + +``` +Quantification -> QuantOperand Quantifier? +Quantifier -> QuantAmount QuantKind? +QuantAmount -> '?' | '*' | '+' | '{' Range '}' +QuantKind -> '?' | '+' +Range -> ',' | ',' ? | + +QuantOperand -> AbsentFunction | Atom | Conditional | CustomCharClass | Group +``` + +Subexpressions can be quantified, meaning they will be repeated some number of times: + +- `?`: 0 or 1 times. +- `*`: 0 or more times. +- `+`: 1 or more times. +- `{n,m}`: Between `n` and `m` (inclusive) times. +- `{n,}`: `n` or more times. +- `{,m}`: Up to `m` times. +- `{n}`: Exactly `n` times. + +Behavior can further be refined by a subsequent `?` or `+`: + +- `x*` _eager_: consume as much of input as possible. +- `x*?` _reluctant_: consume as little of the input as possible. +- `x*+`: _possessive_: eager and never relinquishes any input consumed. + +### Atoms + +``` +Atom -> Anchor + | Backreference + | BacktrackingDirective + | BuiltinCharacterClass + | Callout + | CharacterProperty + | EscapeSequence + | NamedScalar + | Subpattern + | UnicodeScalar + | '\K' + | '\'? +``` + +Atoms are the smallest units of regex syntax. They include escape sequences, metacharacters, backreferences, etc. The most basic form of atom is a literal character. A metacharacter may be treated as literal by preceding it with a backslash. Other literal characters may also be preceded by a backslash, in which case it has no effect, e.g `\%` is literal `%`. However this does not apply to either non-whitespace Unicode characters, or to unknown ASCII letter and number character escapes, e.g `\I` is invalid and would produce an error. `(...)[\1]` is similarly invalid, as a backreference may not appear in a custom character class. + +#### Anchors + +``` +Anchor -> '^' | '$' | '\A' | '\b' | '\B' | '\G' | '\y' | '\Y' | '\z' | '\Z' +``` + +Anchors match against a certain position in the input rather than on a particular character of the input. + +- `^`: Matches at the very start of the input string, or the start of a line when in multi-line mode. +- `$`: Matches at the very end of the input string, or the end of a line when in multi-line mode. +- `\A`: Matches at the very start of the input string. +- `\Z`: Matches at the very end of the input string, in addition to before a newline at the very end of the input string. +- `\z`: Like `\Z`, but only matches at the very end of the input string. +- `\G`: Like `\A`, but also matches against the start position of where matching resumes in global matching mode (e.g `\Gab` matches twice in `abab`, `\Aab` would only match once). +- `\b` matches a boundary between a word character and a non-word character. The definitions of which vary depending on matching engine. +- `\B` matches a non-word-boundary. +- `\y` matches a text segment boundary, the definition of which varies based on the `y{w}` and `y{g}` matching option. +- `\Y` matches a non-text-segment-boundary. + +#### Escape sequences + +``` +EscapeSequence -> '\a' | '\b' | '\c' | '\e' | '\f' | '\n' | '\r' | '\t' +``` + +These escape sequences each denote a specific scalar value. + +- `\a`: The alert (bell) character `U+7`. +- `\b`: The backspace character `U+8`. Note this may only be used in a custom character class, otherwise it represents a word boundary. +- `\c `: A control character sequence, which denotes a scalar from `U+00` - `U+7F` depending on the ASCII character provided. +- `\e`: The escape character `U+1B`. +- `\f`: The form-feed character `U+C`. +- `\n`: The newline character `U+A`. +- `\r`: The carriage return character `U+D`. +- `\t`: The tab character `U+9`. + +#### Builtin character classes + +``` +BuiltinCharClass -> '.' | '\C' | '\d' | '\D' | '\h' | '\H' | '\N' | '\O' | '\R' | '\s' | '\S' | '\v' | '\V' | '\w' | '\W' | '\X' +``` + +- `.`: Any character excluding newlines. +- `\C`: A single UTF code unit. +- `\d`: Digit character. +- `\D`: Non-digit character. +- `\h`: Horizontal space character. +- `\H`: Non-horizontal-space character. +- `\N`: Non-newline character. +- `\O`: Any character (including newlines). This is syntax from Oniguruma. +- `\R`: Newline sequence. +- `\s`: Whitespace character. +- `\S`: Non-whitespace character. +- `\v`: Vertical space character. +- `\V`: Non-vertical-space character. +- `\w`: Word character. +- `\W`: Non-word character. +- `\X`: Any extended grapheme cluster. + +Precise definitions of character classes is discussed in [Unicode for String Processing][pitches]. + +#### Unicode scalars + +``` +UnicodeScalar -> '\u{' UnicodeScalarSequence '}' + | '\u' HexDigit{4} + | '\x{' HexDigit{1...} '}' + | '\x' HexDigit{0...2} + | '\U' HexDigit{8} + | '\o{' OctalDigit{1...} '}' + | '\0' OctalDigit{0...3} + +UnicodeScalarSequence -> * UnicodeScalarSequencElt+ +UnicodeScalarSequencElt -> HexDigit{1...} * + +HexDigit -> [0-9a-fA-F] +OctalDigit -> [0-7] + +NamedScalar -> '\N{' ScalarName '}' +ScalarName -> 'U+' HexDigit{1...8} | [\s\w-]+ +``` + +These sequences define a unicode scalar value using hexadecimal or octal notation. + +In addition to a regular scalar literal e.g `\u{65}`, `\u{...}` also supports a scalar sequence syntax. This is syntactic sugar that implicitly expands a whitespace separated list of scalars e.g `\u{A B C}` into `\u{A}\u{B}\u{C}`. Such a sequence is currently only valid outside of a custom character class, their behavior within a custom character class is left as future work. + +`\x`, when not followed by any hexadecimal digit characters, is treated as `\0`, matching PCRE's behavior. + +`\N{...}` allows a specific Unicode scalar to be specified by name or hexadecimal code point. + +#### Character properties + +``` +CharacterProperty -> '\' ('p' | 'P') '{' PropertyContents '}' +POSIXCharacterProperty -> '[:' PropertyContents ':]' + +PropertyContents -> PropertyName ('=' PropertyName)? +PropertyName -> [\s\w-]+ +``` + +A character property specifies a particular Unicode, POSIX, or PCRE property to match against. We propose supporting: + +- The full range of Unicode character properties. +- The POSIX properties `alnum`, `blank`, `graph`, `print`, `word`, `xdigit` (note that `alpha`, `lower`, `upper`, `space`, `punct`, `digit`, and `cntrl` are covered by Unicode properties). +- The UTS#18 special properties `any`, `assigned`, `ascii`. +- The special PCRE2 properties `Xan`, `Xps`, `Xsp`, `Xuc`, `Xwd`. +- The special Java properties, including e.g `javaLowerCase`, `javaUpperCase`, `javaWhitespace`, `javaMirrored`. + +We follow [UTS#18][uts18]'s guidance for character properties, including fuzzy matching for property name parsing, according to rules set out by [UAX44-LM3]. The following property names are equivalent: + +- `whitespace` +- `isWhitespace` +- `is-White_Space` +- `iSwHiTeSpaCe` +- `i s w h i t e s p a c e` + +Unicode properties consist of both a key and a value, e.g `General_Category=Whitespace`. Each component follows the fuzzy matching rule, and additionally may have an alternative alias spelling, as defined by Unicode in [PropertyAliases.txt][unicode-prop-key-aliases] and [PropertyValueAliases.txt][unicode-prop-value-aliases]. + +There are some Unicode properties where the key or value may be inferred. These include: + +- General category properties e.g `\p{Whitespace}` is inferred as `\p{General_Category=Whitespace}`. +- Script properties e.g `\p{Greek}` is inferred as `\p{Script_Extensions=Greek}`. +- Boolean properties that are inferred to have a `True` value, e.g `\p{Lowercase}` is inferred as `\p{Lowercase=True}`. +- Block properties that begin with the prefix `in`, e.g `\p{inBasicLatin}` is inferred to be `\p{Block=Basic_Latin}`. + +Other Unicode properties however must specify both a key and value. + +For non-Unicode properties, only a value is required. These include: + +- The UTS#18 special properties `any`, `assigned`, `ascii`. +- The POSIX compatibility properties `alnum`, `blank`, `graph`, `print`, `word`, `xdigit`. The remaining POSIX properties are already covered by boolean Unicode property spellings. +- The special PCRE2 properties `Xan`, `Xps`, `Xsp`, `Xuc`, `Xwd`. +- The special Java properties `javaLowerCase`, `javaUpperCase`, `javaWhitespace`, `javaMirrored`. + +Note that the internal `PropertyContents` syntax is shared by both the `\p{...}` and POSIX-style `[:...:]` syntax, allowing e.g `[:script=Latin:]` as well as `\p{alnum}`. Both spellings may be used inside and outside of a custom character class. + +#### `\K` + +The `\K` escape sequence is used to drop any previously matched characters from the final matching result. It does not affect captures, e.g `a(b)\Kc` when matching against `abc` will return a match of `c`, but with a capture of `b`. + +### Groups + +``` +Group -> GroupStart RegexNode ')' +GroupStart -> '(' GroupKind | '(' +GroupKind -> '' | '?' BasicGroupKind | '*' PCRE2GroupKind ':' + +BasicGroupKind -> ':' | '|' | '>' | '=' | '!' | '*' | '<=' | ' 'atomic' + | 'pla' | 'positive_lookahead' + | 'nla' | 'negative_lookahead' + | 'plb' | 'positive_lookbehind' + | 'nlb' | 'negative_lookbehind' + | 'napla' | 'non_atomic_positive_lookahead' + | 'naplb' | 'non_atomic_positive_lookbehind' + | 'sr' | 'script_run' + | 'asr' | 'atomic_script_run' + +NamedGroup -> 'P<' GroupNameBody '>' + | '<' GroupNameBody '>' + | "'" GroupNameBody "'" + +GroupNameBody -> Identifier | BalancingGroupBody + +Identifier -> [\w--\d] \w* +``` + +Groups define a new scope that contains a recursively nested regex. Groups have different semantics depending on how they are introduced. + +Note there are additional constructs that may syntactically appear similar to groups, such as backreferences and PCRE backtracking directives, but are distinct. + +#### Basic group kinds + +- `()`: A capturing group. +- `(?:)`: A non-capturing group. +- `(?|)`: A group that, for a direct child alternation, resets the numbering of groups at each branch of that alternation. See [Group Numbering](#group-numbering). + +Capturing groups produce captures, which remember the range of input matched for the scope of that group. + +A capturing group may be named using any of the `NamedGroup` syntax. The characters of the group name may be any letter or number characters or the character `_`. However the name must not start with a number. This restriction follows the behavior of other regex engines and avoids ambiguities when it comes to named and numeric group references. Duplicate group names are only permitted when either `(?J)` is set, or when the captures share the same numbering, e.g within a branch reset alternation `(?|)`. Otherwise, they are considered invalid. + +#### Atomic groups + +An atomic group e.g `(?>...)` specifies that its contents should not be re-evaluated for backtracking. This has the same semantics as a possessive quantifier, but applies more generally to any regex pattern. + +#### Lookahead and lookbehind + +These groups evaluate the input ahead or behind the current matching position, without advancing the input. + +- `(?=`: A lookahead, which matches against the input following the current matching position. +- `(?!`: A negative lookahead, which ensures a negative match against the input following the current matching position. +- `(?<=`: A lookbehind, which matches against the input prior to the current matching position. +- `(? Identifier? '-' Identifier +``` + +Introduced by .NET, [balancing groups][balancing-groups] extend the `GroupNameBody` syntax to support the ability to refer to a prior group. Upon matching, the prior group is deleted, and any intermediate matched input becomes the capture of the current group. + +#### Group numbering + +Capturing groups are implicitly numbered according to the position of their opening `(` in the regex. For example: + +``` +(a((?:b)(?c)d)(e)f) +^ ^ ^ ^ +1 2 3 4 +``` + +Non-capturing groups are skipped over when counting. + +Branch reset groups can alter this numbering, as they reset the numbering in the branches of an alternation child. Outside the alternation, numbering resumes at the next available number not used in one of the branches. For example: + +``` +(a()(?|(b)(c)|(?:d)|(e)))(f) +^ ^ ^ ^ ^ ^ +1 2 3 4 3 5 +``` + +Because this construct allows multiple capture groups to share the same number, it allows a capture to share the same name in both branches. For example: + +``` +(?|(?a)|(?b)) +``` + +which produces a single capture result named `x`. This would be otherwise be invalid in a regular alternation, as the captures would have distinct numberings. + +### Custom character classes + +``` +CustomCharClass -> Start Set (SetOp Set)* ']' +Start -> '[' '^'? +Set -> Member+ +Member -> CustomCharClass | Quote | Range | Atom +Range -> RangeElt `-` RangeElt +RangeElt -> | UnicodeScalar | EscapeSequence +SetOp -> '&&' | '--' | '~~' +``` + +Custom characters classes introduce their own sublanguage, in which most regular expression metacharacters become literal. The basic element in a custom character class is an `Atom`, though only some atoms are considered valid: + +- Builtin character classes, except for `.`, `\R`, `\O`, `\X`, `\C`, and `\N`. +- Escape sequences, including `\b` which becomes the backspace character (rather than a word boundary). +- Unicode scalars. +- Named scalars. +- Character properties. +- Plain literal characters. + +Atoms may be used to compose other character class members, including ranges, quoted sequences, and even nested custom character classes `[[ab]c\d]`. Adjacent members form an implicit union of character classes, e.g `[[ab]c\d]` is the union of the characters `a`, `b`, `c`, and digit characters. + +Custom character classes may not be empty, e.g `[]` is forbidden. + +Quoted sequences may be used to escape the contained characters, e.g `[a\Q]\E]` is the character class of `]` and `a`. + +Ranges of characters may be specified with `-`, e.g `[a-z]` matches against the letters from `a` to `z`. Only unicode scalars and literal characters are valid range operands. If `-` cannot be used to form a range, it is interpreted as literal, e.g `[-a-]` is the character class of `-` and `a`. `[a-c-d]` is the character class of `a`...`c`, `-`, and `d`. + +Operators may be used to apply set operations to character class members. The operators supported are: + +- `&&`: Intersection of the LHS and RHS. +- `--`: Subtraction of the RHS from the LHS. +- `~~`: Symmetric difference of the RHS and LHS. + +These operators have a lower precedence than the implicit union of members, e.g `[ac-d&&a[d]]` is an intersection of the character classes `[ac-d]` and `[ad]`. + +Note that a custom character class may begin with the `:` character, and only becomes a POSIX character property if a closing `:]` is present. For example, `[:a]` is the character class of `:` and `a`. + +### Matching options + +``` +MatchingOptionSeq -> '^' MatchingOption* + | MatchingOption+ + | MatchingOption* '-' MatchingOption* + +MatchingOption -> 'i' | 'J' | 'm' | 'n' | 's' | 'U' | 'x' | 'xx' | 'w' | 'D' | 'P' | 'S' | 'W' | 'y{' ('g' | 'w') '}' +``` + +A matching option sequence may be used as a group specifier, and denotes a change in matching options for the scope of that group. For example `(?x:a b c)` enables extended syntax for `a b c`. A matching option sequence may be part of an "isolated group" which has an implicit scope that wraps the remaining elements of the current group. For example, `(?x)a b c` also enables extended syntax for `a b c`. + +If used in the branch of an alternation, an isolated group affects all the following branches of that alternation. For example, `a(?i)b|c|d` is treated as `a(?i:b)|(?i:c)|(?i:d)`. + +We support all the matching options accepted by PCRE, ICU, and Oniguruma. In addition, we accept some matching options unique to our matching engine. + +#### PCRE options + +- `i`: Case insensitive matching. +- `J`: Allows multiple groups to share the same name, which is otherwise forbidden. +- `m`: Enables `^` and `$` to match against the start and end of a line rather than only the start and end of the entire string. +- `n`: Disables the capturing behavior of `(...)` groups. Named capture groups must be used instead. +- `s`: Changes `.` to match any character, including newlines. +- `U`: Changes quantifiers to be reluctant by default, with the `?` specifier changing to mean greedy. +- `x`, `xx`: Enables extended syntax mode, which allows non-semantic whitespace and end-of-line comments. See [Extended Syntax Modes](#extended-syntax-modes) for more info. + +#### ICU options + +- `w`: Enables the Unicode interpretation of word boundaries `\b`. + +#### Oniguruma options + +- `D`: Enables ASCII-only digit matching for `\d`, `\p{Digit}`, `[:digit:]`. +- `S`: Enables ASCII-only space matching for `\s`, `\p{Space}`, `[:space:]`. +- `W`: Enables ASCII-only word matching for `\w`, `\p{Word}`, `[:word:]`, and `\b`. +- `P`: Enables ASCII-only for all POSIX properties (including `digit`, `space`, and `word`). +- `y{g}`, `y{w}`: Changes the meaning of `\X`, `\y`, `\Y`. These are mutually exclusive options, with `y{g}` specifying extended grapheme cluster mode, and `y{w}` specifying word mode. + +#### Swift options + +These options are specific to the Swift regex matching engine and control the semantic level at which matching takes place. + +- `X`: Grapheme cluster matching. +- `u`: Unicode scalar matching. +- `b`: Byte matching. + +Further details on these are TBD and outside the scope of this pitch. + +### References + +``` +NamedOrNumberRef -> NamedRef | NumberRef +NamedRef -> Identifier RecursionLevel? +NumberRef -> ('+' | '-')? RecursionLevel? +RecursionLevel -> '+' | '-' +``` + +A reference is an abstract identifier for a particular capturing group in a regular expression. It can either be named or numbered, and in the latter case may be specified relative to the current group. For example `-2` refers to the capture group `N - 2` where `N` is the number of the next capture group. References may refer to groups ahead of the current position e.g `+3`, or the name of a future group. These may be useful in recursive cases where the group being referenced has been matched in a prior iteration. If a referenced capture does not exist anywhere in the regular expression, the reference is diagnosed as invalid. + +A backreference may optionally include a recursion level in certain cases, which is a syntactic element inherited [from Oniguruma][oniguruma-syntax] that allows the reference to specify a capture relative to a given recursion level. + +#### Backreferences + +``` +Backreference -> '\g{' NamedOrNumberRef '}' + | '\g' NumberRef + | '\k<' NamedOrNumberRef '>' + | "\k'" NamedOrNumberRef "'" + | '\k{' NamedRef '}' + | '\' [1-9] [0-9]+ + | '(?P=' NamedRef ')' +``` + +A backreference evaluates to the value last captured by the referenced capturing group. If the referenced capture has not been evaluated yet, the match fails. + +#### Subpatterns + +``` +Subpattern -> '\g<' NamedOrNumberRef '>' + | "\g'" NamedOrNumberRef "'" + | '(?' GroupLikeSubpatternBody ')' + +GroupLikeSubpatternBody -> 'P>' NamedRef + | '&' NamedRef + | 'R' + | NumberRef +``` + +A subpattern causes the referenced capture group to be re-evaluated at the current position. The syntax `(?R)` is equivalent to `(?0)`, and causes the entire pattern to be recursed. + +### Conditionals + +``` +Conditional -> ConditionalStart Concatenation ('|' Concatenation)? ')' +ConditionalStart -> KnownConditionalStart | GroupConditionalStart + +KnownConditionalStart -> '(?(' KnownCondition ')' +GroupConditionalStart -> '(?' GroupStart + +KnownCondition -> 'R' + | 'R' NumberRef + | 'R&' NamedRef + | '<' NamedOrNumberRef '>' + | "'" NamedOrNumberRef "'" + | 'DEFINE' + | 'VERSION' VersionCheck + | NumberRef + +PCREVersionCheck -> '>'? '=' PCREVersionNumber +PCREVersionNumber -> '.' +``` + +A conditional evaluates a particular condition, and chooses a branch to match against accordingly. 1 or 2 branches may be specified. If 1 branch is specified e.g `(?(...)x)`, it is treated as the true branch. Note this includes an empty true branch, e.g `(?(...))` which is the null pattern as described in the [Top-Level Regular Expression](#top-level-regular-expression) section. If 2 branches are specified, e.g `(?(...)x|y)`, the first is treated as the true branch, the second being the false branch. + +A condition may be: + +- A numeric or delimited named reference to a capture group, which checks whether the group matched successfully. +- A recursion check on either a particular group or the entire regex. In the former case, this checks to see if the last recursive call is through that group. In the latter case, it checks if the match is currently taking place in any kind of recursive call. +- A PCRE version check. + +If the condition does not syntactically match any of the above, it is treated as an arbitrary recursive regular expression. This will be matched against, and evaluates to true if the match is successful. It may contain capture groups that add captures to the match. + +The `DEFINE` keyword is not used as a condition, but rather a way in which to define a group which is not evaluated, but may be referenced by a subpattern. + +### PCRE backtracking directives + +``` +BacktrackingDirective -> '(*' BacktrackingDirectiveKind (':' )? ')' +BacktrackingDirectiveKind -> 'ACCEPT' | 'FAIL' | 'F' | 'MARK' | '' | 'COMMIT' | 'PRUNE' | 'SKIP' | 'THEN' +``` + +This is syntax specific to PCRE, and is used to control backtracking behavior. Any of the directives may include an optional tag, however `MARK` must have a tag. The empty directive is treated as `MARK`. Only the `ACCEPT` directive may be quantified, as it can use the backtracking behavior of the engine to be evaluated only if needed by a reluctant quantification. + +- `ACCEPT`: Causes matching to terminate immediately as a successful match. If used within a subpattern, only that level of recursion is terminated. +- `FAIL`, `F`: Causes matching to fail, forcing backtracking to occur if possible. +- `MARK`: Assigns a label to the current matching path, which is passed back to the caller on success. Subsequent `MARK` directives overwrite the label assigned, so only the last is passed back. +- `COMMIT`: Prevents backtracking from reaching any point prior to this directive, causing the match to fail. This does not allow advancing the input to try a different starting match position. +- `PRUNE`: Similar to `COMMIT`, but allows advancing the input to try and find a different starting match position. +- `SKIP`: Similar to `PRUNE`, but skips ahead to the position of `SKIP` to try again as the starting position. +- `THEN`: Similar to `PRUNE`, but when used inside an alternation will try to match in the subsequent branch before attempting to advance the input to find a different starting position. + +### PCRE global matching options + +``` +GlobalMatchingOptionSequence -> GlobalMatchingOption+ +GlobalMatchingOption -> '(*' GlobalMatchingOptionKind ')' + +GlobalMatchingOptionKind -> LimitOptionKind '=' + | NewlineKind | NewlineSequenceKind + | 'NOTEMPTY_ATSTART' | 'NOTEMPTY' + | 'NO_AUTO_POSSESS' | 'NO_DOTSTAR_ANCHOR' + | 'NO_JIT' | 'NO_START_OPT' | 'UTF' | 'UCP' + +LimitOptionKind -> 'LIMIT_DEPTH' | 'LIMIT_HEAP' | 'LIMIT_MATCH' +NewlineKind -> 'CRLF' | 'CR' | 'ANYCRLF' | 'ANY' | 'LF' | 'NUL' +NewlineSequenceKind -> 'BSR_ANYCRLF' | 'BSR_UNICODE' +``` + +This is syntax specific to PCRE, and allows a set of global options to appear at the start of a regular expression. They may not appear at any other position. + +- `LIMIT_DEPTH`, `LIMIT_HEAP`, `LIMIT_MATCH`: These place certain limits on the resources the matching engine may consume, and matches it may make. +- `CRLF`, `CR`, `ANYCRLF`, `ANY`, `LF`, `NUL`: These control the definition of a newline character, which is used when matching e.g the `.` character class, and evaluating where a line ends in multi-line mode. +- `BSR_ANYCRLF`, `BSR_UNICODE`: These change the definition of `\R`. +- `NOTEMPTY`: Does not consider the empty string to be a valid match. +- `NOTEMPTY_ATSTART`: Like `NOT_EMPTY`, but only applies to the first matching position in the input. +- `NO_AUTO_POSSESS`: Disables an optimization that treats a quantifier as possessive if the following construct clearly cannot be part of the match. In other words, disables the short-circuiting of backtracks in cases where the engine knows it will not produce a match. This is useful for debugging, or for ensuring a callout gets invoked. +- `NO_DOTSTAR_ANCHOR`: Disables an optimization that tries to automatically anchor `.*` at the start of a regex. Like `NO_AUTO_POSSESS`, this is mainly used for debugging or ensuring a callout gets invoked. +- `NO_JIT`: Disables JIT compilation +- `NO_START_OPT`: Disables various optimizations performed at the start of matching. Like `NO_DOTSTAR_ANCHOR`, is mainly used for debugging or ensuring a callout gets invoked. +- `UTF`: Enables UTF pattern support. +- `UCP`: Enables Unicode property support. + +### Callouts + +``` +Callout -> PCRECallout | NamedCallout | InterpolatedCallout + +PCRECallout -> '(?C' CalloutBody ')' +PCRECalloutBody -> '' | + | '`' '`' + | "'" "'" + | '"' '"' + | '^' '^' + | '%' '%' + | '#' '#' + | '$' '$' + | '{' '}' + +NamedCallout -> '(*' Identifier CalloutTag? CalloutArgs? ')' +CalloutArgs -> '{' CalloutArgList '}' +CalloutArgList -> CalloutArg (',' CalloutArgList)* +CalloutArg -> [^,}]+ +CalloutTag -> '[' Identifier ']' + +InterpolatedCallout -> '(?' '{' Interpolation '}' CalloutTag? CalloutDirection? ')' +Interpolation -> | '{' Interpolation '}' +CalloutDirection -> 'X' | '<' | '>' +``` + +A callout is a feature that allows a user-supplied function to be called when matching reaches that point in the pattern. We supported parsing 3 types of callout: + +- PCRE callout syntax, which accepts a string or numeric argument that is passed to the function. +- Oniguruma named callout syntax, which accepts an identifier with an optional tag and argument list. +- Interpolated callout syntax, which is equivalent to Oniguruma's "callout of contents". This callout accepts an arbitrary interpolated program. This is an expanded version of Perl's interpolation syntax, and allows an arbitrary nesting of delimiters in addition to an optional tag and direction. + +While we propose parsing these for the purposes of issuing helpful diagnostics, we are deferring full support for the interpolated syntax for the future. + +### Absent functions + +``` +AbsentFunction -> '(?~' RegexNode ')' + | '(?~|' Concatenation '|' Concatenation ')' + | '(?~|' Concatenation ')' + | '(?~|)' +``` + +An absent function is an [Oniguruma][oniguruma-syntax] feature that allows for the easy inversion of a given pattern. There are 4 variants of the syntax: + +- `(?~|absent|expr)`: Absent expression, which attempts to match against `expr`, but is limited by the range that is not matched by `absent`. +- `(?~absent)`: Absent repeater, which matches against any input not matched by `absent`. Equivalent to `(?~|absent|\O*)`. +- `(?~|absent)`: Absent stopper, which limits any subsequent matching to not include `absent`. +- `(?~|)`: Absent clearer, which undoes the effects of the absent stopper. + + +## Syntactic differences between engines + +The proposed "syntactic superset" introduces some minor ambiguities, as each engine supports a slightly different set of features. When a particular engine's parser sees a feature it doesn't support, it typically has a fall-back behavior, such as treating the unknown feature as literal contents. + +Explicit compatibility modes, i.e. precisely mimicking emergent behavior from a specific engine's parser, is deferred as future work from this proposal. Conversion from this "syntactic superset" to a particular engine's syntax (e.g. as an AST "pretty printer") is deferred as future work from this proposal. + +Below is an exhaustive treatment of every syntactic ambiguity we have encountered. + +### Character class set operations + +In a custom character class, some engines allow for binary set operations that take two character class inputs, and produce a new character class output. However which set operations are supported and the spellings used differ by engine. + +| PCRE | ICU | UTS#18 | Oniguruma | .NET | Java | +|------|-----|--------|-----------|------|------| +| ❌ | Intersection `&&`, Subtraction `--` | Intersection, Subtraction | Intersection `&&` | Subtraction via `-` | Intersection `&&` | + + +[UTS#18][uts18] requires intersection and subtraction, and uses the operation spellings `&&` and `--` in its examples, though it doesn't mandate a particular spelling. In particular, conforming implementations could spell the subtraction `[[x]--[y]]` as `[[x]&&[^y]]`. UTS#18 also suggests a symmetric difference operator `~~`, and uses an explicit `||` operator in examples, though doesn't require either. + +Engines that don't support a particular operator fallback to treating it as literal, e.g `[x&&y]` in PCRE is the character class of `["x", "&", "y"]` rather than an intersection. + +Unlike other engines, .NET supports the use of `-` to denote both a range as well as a set subtraction. .NET disambiguates this by only permitting its use as a subtraction if the right hand operand is a nested custom character class, otherwise it is a range operator. This conflicts with e.g ICU where `[x-[y]]`, in which the `-` is treated as literal. + +We propose supporting the operators `&&`, `--`, and `~~`. This means that any regex literal containing these sequences in a custom character class while being written for an engine not supporting that operation will have a different semantic meaning in our engine. However this ought not to be a common occurrence, as specifying a character multiple times in a custom character class is redundant. + +In order to help avoid confusion between engines, we will reject the use of .NET style `-` for subtraction. Users will be required to write `--` instead, or escape with `\-`. + +### Nested custom character classes + +This allows e.g `[[a]b[c]]`, which is interpreted the same as `[abc]`. It also allows for more complex set operations with custom character classes as the operands. + +| PCRE | ICU | UTS#18 | Oniguruma | .NET | Java | +|------|-----|--------|-----------|------|------| +| ❌ | ✅ | 💡 | ✅ | ❌ | ✅ | + + +UTS#18 doesn't require this, though it does suggest it as a way to clarify precedence for chains of character class set operations e.g `[\w--\d&&\s]`, which the user could write as `[[\w--\d]&&\s]`. + +PCRE does not support this feature, and as such treats `]` as the closing character of the custom character class. Therefore `[[a]b[c]]` is interpreted as the character class `["[", "a"]`, followed by literal `b`, and then the character class `["c"]`, followed by literal `]`. + +.NET does not support nested character classes in general, although allows them as the right-hand side of a subtraction operation. + +We propose allowing nested custom character classes. + +### `\U` + +In PCRE, if `PCRE2_ALT_BSUX` or `PCRE2_EXTRA_ALT_BSUX` are specified, `\U` matches literal `U`. However in ICU, `\Uhhhhhhhh` matches a hex sequence. We propose following the ICU behavior. + +### `{,n}` + +This quantifier is supported by Oniguruma, but in PCRE it matches the literal characters `{`, `,`, `n`, and `}` in sequence. We propose supporting it as a quantifier. + +### `\DDD` + +This syntax is implemented in a variety of different ways depending on the engine. In ICU and Java, it is always a backreference unless prefixed with `0`, in which case it is an octal sequence. + +In PCRE, Oniguruma, and .NET, it is also always an octal sequence if prefixed with `0`, however there are other cases where it may be treated as octal. These cases vary slightly between the engines. In PCRE, it will be treated as backreference if any of the following hold: + +- Its value is `0 < n < 10`. +- Its first digit is `8` or `9`. +- Its value corresponds to a valid *prior* group number. + +Otherwise it is treated as an octal sequence. + +Oniguruma follows all of these except the second. If the first digit is `8` or `9`, it is instead treated as the literal number, e.g `\81` is `81`. .NET also follows this behavior, but additionally has the last condition consider *all* groups, not just prior ones (as backreferences can refer to future groups in recursive cases). + +We propose a simpler behavior more inline with ICU and Java. A `\DDD` sequence that does not start with a `0` will be treated as a backreference, otherwise it will be treated as an octal sequence. If an invalid backreference is formed with this syntax, we will suggest prefixing with a `0` if an octal sequence is desired. + +One further difference exists between engines in the octal sequence case. In ICU, up to 3 additional digits are read after the `0`. In PCRE, only 2 additional digits may be interpreted as octal, the last is literal. We will follow the ICU behavior, as it is necessary when requiring a `0` prefix. + +### `\x` + +In PCRE, a bare `\x` denotes the NUL character (`U+00`). In Oniguruma, it denotes literal `x`. We propose following the PCRE behavior. + +### Whitespace in ranges + +In PCRE, `x{2,4}` is a range quantifier meaning that `x` can be matched from 2 to 4 times. However if any whitespace is introduced within the braces e.g `x{2, 4}`, it becomes an invalid range and is then treated as the literal characters instead. We find this behavior to be unintuitive, and therefore propose parsing any intermixed whitespace in the range. + +### Implicitly-scoped matching option scopes + +PCRE and Oniguruma both support changing the active matching options through an isolated group e.g `(?i)`. However, they have differing semantics when it comes to their scoping. In Oniguruma, it is treated as an implicit new scope that wraps everything until the end of the current group. In PCRE, it is treated as changing the matching option for all the following expressions until the end of the group. + +These sound similar, but have different semantics around alternations, e.g for `a(?i)b|c|d`, in Oniguruma this becomes `a(?i:b|c|d)`, where `a` is no longer part of the alternation. However in PCRE it becomes `a(?i:b)|(?i:c)|(?i:d)`, where `a` remains a child of the alternation. + +We propose matching the PCRE behavior. + +### Backreference condition kinds + +PCRE and .NET allow for conditional patterns to reference a group by its name without any form of delimiter, e.g: + +``` +(?x)?(?(group1)y) +``` + +where `y` will only be matched if `(?x)` was matched. PCRE will always treat such syntax as a backreference condition, however .NET will only treat it as such if a group with that name exists somewhere in the regex (including after the conditional). Otherwise, .NET interprets `group1` as an arbitrary regular expression condition to try match against. Oniguruma on the other hand will always treat `group1` as an regex condition to match against. + +We propose parsing such conditions as an arbitrary regular expression condition, as long as they do not conflict with other known condition spellings such as `R&name`. If the condition has a name that matches a named group in the regex, we will emit a warning asking users to explicitly use the syntax `(?()y)` if they want a backreference condition. This more explicit syntax is supported by both PCRE and Oniguruma. + +### `\N` + +PCRE supports `\N` meaning "not a newline", however there are engines that treat it as a literal `N`. We propose supporting the PCRE behavior. + +### Extended character property syntax + +ICU unifies the character property syntax `\p{...}` with the syntax for POSIX character classes `[:...:]`. This has two effects: + +- They share the same internal grammar, which allows the use of any Unicode character properties in addition to the POSIX properties. +- The POSIX syntax may be used outside of custom character classes, unlike in PCRE and Oniguruma. + +We propose following both of these rules. The former is purely additive, and therefore should not conflict with regex engines that implement a more limited POSIX syntax. The latter does conflict with other engines, but we feel it is much more likely that a user would expect e.g `[:space:]` to be a character property rather than the character class `[:aceps]`. We do however feel that a warning might be warranted in order to avoid confusion. + +### POSIX character property disambiguation + +PCRE, Oniguruma and ICU allow `[:` to be part of a custom character class if a closing `:]` is not present. For example, `[:a]` is the character class of `:` and `a`. However they each have different rules for detecting the closing `:]`: + +- PCRE will scan ahead until it hits either `:]`, `]`, or `[:`. +- Oniguruma will scan ahead until it hits either `:]`, `]`, or the length exceeds 20 characters. +- ICU will scan ahead until it hits a known escape sequence (e.g `\a`, `\e`, `\Q`, ...), or `:]`. Note this excludes character class escapes e.g `\d`. It also excludes `]`, meaning that even `[:a][:]` is parsed as a POSIX character property. + +We propose unifying these behaviors by scanning ahead until we hit either `[`, `]`, `:]`, or `\`. Additionally, we will stop on encountering `}` or a second occurrence of `=`. These fall out the fact that they would be invalid contents of the alternative `\p{...}` syntax. + + +### Script properties + +Shorthand script property syntax e.g `\p{Latin}` is treated as `\p{Script=Latin}` by PCRE, ICU, Oniguruma, and Java. These use [the Unicode Script property][unicode-scripts], which assigns each scalar a particular script value. However, there are scalars that may appear in multiple scripts, e.g U+3003 DITTO MARK. These are often assigned to the `Common` script to reflect this fact, which is not particularly useful for matching purposes. To provide more fine-grained script matching, Unicode provides [the Script Extension property][unicode-script-extensions], which exposes the set of scripts that a scalar appears in. + +As such we feel that the more desirable default behavior of shorthand script property syntax e.g `\p{Latin}` is for it to be treated as `\p{Script_Extension=Latin}`. This matches Perl's default behavior. Plain script properties may still be written using the more explicit syntax e.g `\p{Script=Latin}` and `\p{sc=Latin}`. + +### Extended syntax modes + +Various regex engines offer an "extended syntax" where whitespace is treated as non-semantic (e.g `a b c` is equivalent to `abc`), in addition to allowing end-of-line comments `# comment`. In both PCRE and Perl, this is enabled through the `(?x)`, and in later versions, `(?xx)` matching options. The former allows non-semantic whitespace outside of character classes, and the latter also allows non-semantic whitespace in custom character classes. + +ICU and Java however enable the more broad behavior under `(?x)`. We propose following this behavior, with `(?x)` and `(?xx)` being treated the same. + +Different regex engines also have different rules around what characters are considered non-semantic whitespace. When compiled with Unicode support, PCRE follows the `Pattern_White_Space` Unicode property, which consists of the following scalars: + +- The space character `U+20` +- Whitespace characters `U+9...U+D` +- Next line `U+85` +- Left-to-right mark `U+200E` +- Right-to-left mark `U+200F` +- Line separator `U+2028` +- Paragraph separator `U+2029` + +This is the same set of scalars matched by `UnicodeScalar.Properties.isPatternWhitespace`. Additionally, in a custom character class, PCRE only considers the space and tab characters as whitespace. Other engines do not differentiate between whitespace characters inside and outside custom character classes, and appear to follow a subset of this list. Therefore we propose supporting exactly the characters in this list for the purposes of non-semantic whitespace parsing. + +### Group numbering + +In PCRE, groups are numbered according to the position of their opening parenthesis. .NET also follows this rule, with the exception that named groups are numbered after unnamed groups. For example: + +``` +(a(?x)b)(?y)(z) +^ ^ ^ ^ +1 3 4 2 +``` + +The `(z)` group gets numbered before the named groups get numbered. + +We propose matching the PCRE behavior where groups are numbered purely based on order. + +### Duplicate group names + +By default, Oniguruma, Perl, and .NET allow duplicate capture group names for differently numbered captures. PCRE also allows this when `(?J)` is set. However, each engine has a different backreference behavior to such captures: + +- PCRE and Perl refer to the first matched group with that name. +- .NET refers to the last matched group with that name. +- Oniguruma allows a reference to any of the previously matched values of the groups with that name. + +We feel that this behavior can be unintuitive, and therefore intend to make duplicate group names invalid by default for differently numbered captures. This follows the behavior of ICU, Java, and PCRE's default behavior. + +## Swift canonical syntax + +The proposed syntactic superset means there will be multiple ways to write the same thing. Below we discuss what Swift's preferred spelling could be, a "Swift canonical syntax". + +We are not formally proposing this as a distinct syntax or concept, rather it is useful for considering compiler features such as fixits, pretty-printing, and refactoring actions. We're hoping for further discussion with the community here. Useful criteria include how well the choice fits in with the rest of Swift, whether there's an existing common practice, and whether one choice is less confusing in the context of others. + +[Unicode scalar literals](#unicode-scalars) can be spelled in many ways. We propose treating Swift's string literal syntax of `\u{HexDigit{1...}}` as the preferred spelling. + +Character properties can be spelled `\p{...}` or `[:...:]`. We recommend preferring `\p{...}` as the bracket syntax historically meant POSIX-defined character classes, and still has that connotation in some engines. The [spelling of properties themselves can be fuzzy](#character-properties) and we (weakly) recommend the shortest spelling (no opinion on casing yet). For script extensions, we (weakly) recommend e.g. `\p{Greek}` instead of `\p{Script_Extensions=Greek}`. We would like more discussion with the community here. + +[Lookaround assertions](#lookahead-and-lookbehind) have common shorthand spellings, while PCRE2 introduced longer more explicit spellings. We are (very weakly) recommending the common short-hand syntax of e.g. `(?=...)` as that's wider spread. We are interested in more discussion with the community here. + +Named groups may be specified with a few different delimiters: `(?...)`, `(?P...)`, `(?'name'...)`. We (weakly) recommend `(?...)`, but the final preference may be influenced by choice of delimiter for the regex itself. We'd appreciate any insight from the community. + +[Backreferences](#backreferences) have multiple spellings. For absolute numeric references, `\DDD` seems to be a strong candidate for the preferred syntax due to its familiarity. For relative numbered references, as well as named references, either `\k<...>` or `\k'...'` seem like the better choice, depending on the syntax chosen for named groups. This avoids the confusion between `\g{...}` and `\g<...>` referring to a backreferences and subpatterns respectively, as well as any confusion with group syntax. + +For [subpatterns](#subpatterns), we recommend either `\g<...>` or `\g'...'` depending on the choice for named group syntax. We're unsure if we should prefer `(?R)` as a spelling for e.g. `\g<0>` or not, as it is more widely used and understood, but less consistent with other subpatterns. + +[Conditional references](#conditionals) have a choice between `(?('name'))` and `(?())`. The preferred syntax in this case would likely reflect the syntax chosen for named groups. + +We are deferring runtime support for callouts from regex literals as future work, though we will correctly parse their contents. We have no current recommendation for a preference of PCRE-style [callout syntax](#callouts), and would like to discuss with the community whether we should have one. + +## Alternatives Considered + +### Failable inits + +There are many ways for compilation to fail, from syntactic errors to unsupported features to type mismatches. In the general case, run-time compilation errors are not recoverable by a tool without modifying the user's input. Even then, the thrown errors contain valuable information as to why compilation failed. For example, swiftpm presents any errors directly to the user. + +As proposed, the errors thrown will be the same errors presented to the Swift compiler, tracking fine-grained source locations with specific reasons why compilation failed. Defining a rich error API is future work, as these errors are rapidly evolving and it is too early to lock in the ABI. + + +### Skip the syntax + +The top alternative is to just skip regex syntax altogether by only shipping the result builder DSL and forbidding run-time regex construction from strings. However, doing so would miss out on the familiarity benefits of existing regex syntax. Additionally, without support for run-time strings containing regex syntax, important domains would be closed off from better string processing, such as command-line tools and user-input searches. This would land us in a confusing world where NSRegularExpression, even though it operates over a fundamentally different model of string than Swift's `String` and exhibits different behavior than Swift regexes, is still used for these purposes. + +We consider our proposed direction to be more compelling, especially when coupled with refactoring actions to convert literals into regex DSLs. + +### Introduce a novel regex syntax + +Another alternative is to invent a new syntax for regex. This would similarly lose out on the familiarity benefit, though a few simple adjustments could aid readability. + +We are prototyping an "experimental" Swift extended syntax, which is future work and outside the scope of this proposal. Every syntactic extension, while individually compelling, does introduce incompatibilities and can lead to an "uncanny valley" effect. Further investigation is needed and such support can be built on top of what is presented here. + +### Support a minimal syntactic subset + +Regex syntax will become part of Swift's source and binary-compatibility story, so a reasonable alternative is to support the absolute minimal syntactic subset available. However, we would need to ensure that such a minimal approach is extensible far into the future. Because syntax decisions can impact each other, we would want to consider the ramifications of this full syntactic superset ahead of time anyways. + +Even though it is more work up-front and creates a longer proposal, it is less risky to support the full intended syntax. The proposed superset maximizes the familiarity benefit of regex syntax. + +### Future: Capture descriptions on Regex + +Future API could include a description of the capture list that a regex contains, provided as a collection of optionally-named captures and their types. This would further enhance dynamic regexes. + + + + + +[pcre2-syntax]: https://www.pcre.org/current/doc/html/pcre2syntax.html +[oniguruma-syntax]: https://github.com/kkos/oniguruma/blob/master/doc/RE +[icu-syntax]: https://unicode-org.github.io/icu/userguide/strings/regexp.html +[uts18]: https://www.unicode.org/reports/tr18/ +[.net-syntax]: https://docs.microsoft.com/en-us/dotnet/standard/base-types/regular-expressions +[UAX44-LM3]: https://www.unicode.org/reports/tr44/#UAX44-LM3 +[unicode-prop-key-aliases]: https://www.unicode.org/Public/UCD/latest/ucd/PropertyAliases.txt +[unicode-prop-value-aliases]: https://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt +[unicode-scripts]: https://www.unicode.org/reports/tr24/#Script +[unicode-script-extensions]: https://www.unicode.org/reports/tr24/#Script_Extensions +[balancing-groups]: https://docs.microsoft.com/en-us/dotnet/standard/base-types/grouping-constructs-in-regular-expressions#balancing-group-definitions +[overview]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md +[pitches]: https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md + + + diff --git a/proposals/0356-swift-snippets.md b/proposals/0356-swift-snippets.md new file mode 100644 index 0000000000..e1ab8692c2 --- /dev/null +++ b/proposals/0356-swift-snippets.md @@ -0,0 +1,427 @@ +# Swift Snippets + +* Proposal: [SE-0356](0356-swift-snippets.md) +* Authors: [Ashley Garland](https://github.com/bitjammer) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.7)** +* Implementation: + Available in [recent nightly](https://swift.org/download/#snapshots) snapshots. Requires `--enable-experimental-snippet-support` feature flag when using the [Swift DocC Plugin](https://github.com/apple/swift-docc-plugin). Related pull requests: + * Swift DocC + * [Add snippet support](https://github.com/apple/swift-docc/pull/61) + * Swift Package Manager: + * [Introduce the snippet target type](https://github.com/apple/swift-package-manager/pull/3694) + * [Rename _main symbol when linking snippets](https://github.com/apple/swift-package-manager/pull/3732) + * SymbolKit: + * [Add snippet mixin for symbols](https://github.com/apple/swift-docc-symbolkit/pull/10) + * [Add .snippet and .snippetGroup kind](https://github.com/apple/swift-docc-symbolkit/pull/15) + * Swift DocC Plugin + * [Swift DocC Plugin Snippets](https://github.com/apple/swift-docc-plugin/pull/7) +* Review threads + * [Pitch](https://forums.swift.org/t/pitch-swift-snippets/56348) + * [Review](https://forums.swift.org/t/se-0356-swift-snippets/57097) + + +## Introduction + +This proposal describes a convention for writing a new form of sample code called *snippets*. Snippets are short, single-file examples that can build and run from within a Swift package, with access to other code within that package, and can be used in a variety of ways. + +## Motivation + +There are two main vehicles people employ when they want to use code to demonstrate an idea or API: + +* Complete sample projects +* Bits of code displayed inline within documentation + +Both of these are critical tools, and snippets aren’t intended to replace either of them. Instead, snippets offer an additional method for teaching with code, that fits somewhere between both ideas. First, let’s look at the current options available to most developers. + +### Sample code projects + +Sample code is often created as a full project with build configuration files, resources, and multiple source files to produce a finished “app”. These sample projects are useful when you want to show code in a specific, complete scenario. However, these projects also tend to be a lot of effort to create and maintain. For this reason, developers often simply don’t build great samples. + +Because sample code projects require more time and effort, they tend to become "kitchen sink" examples that show anything and everything around a particular topic, or grow to exemplify multiple topics and libraries. Not only does this make the project increasingly difficult to maintain, it also makes it more difficult for a reader to navigate and find the gems that may be hidden in a sample code project. + +### Code listings within documentation + +Code listings are generally presented as a few lines of code printed inline within larger bits of documentation prose, most often carved out in a small “code box” area. This code is generally authored right along side the prose by a writer, in their favorite word processor. While these bits of code are incredibly helpful while reading the documentation — often this is the best sort of documentation — there are downsides, too. + +**Code listings tend to go stale.** Code listings, once put into the documentation, tend to be treated as regular text. The code isn’t built regularly, and may not be revisited by the author for a long period of time. Over time, changes to the programming language or APIs can easily make code listings go stale and stop compiling successfully. A larger documentation team may build bespoke systems to extract and test this code — time better spent writing great new documentation. Making it easy to add code listings to documentation that can also be built and run (and validated) is one of the main goals of snippets. + +**Code listings don't get good editor support.** As most code listings are typed into an editor focused on writing prose, the author misses out on the coding features typically available in a code editor or IDE. Missing inline error checking and syntax highlighting means it is much more likely the code sample will have an error. + +**Code listings tend to be more like pseudocode.** This happens because the author knows they aren’t actually building running code, and all the explanation for the code happens in the surrounding prose. This results in code that is much less useful for the reader to copy into their own projects. + +### Snippets combine the best of both + +Snippets are designed to provide some of the best features of each of the above approaches. Each snippet is a single file, making it easy to think about as an author and as a reader, but it is also a fully valid program. Each snippet can access the full code and features of the rest of the package, so behavior can be powerful, while the code in each snippet remains simple. This means the code should also be small enough to present inline within documentation — perfect to act as a code listing. This code is able to be tested and maintained as fully-functional code ready to be copied and used by the reader. + +Snippets fill a gap in the spectrum of example-oriented documentation, shown below roughly in decreasing granularity: + +* **API reference and inline code fragments.** Not typically compilable, these usually occur in lists, an index, or perhaps a link to a symbol page. These are not compositional in nature. +* **Snippets.** Here, one file demonstrates one task, composing one or more APIs, across no more than a handful of modules. A snippet should basically be something a reader can copy and paste or use as a starting point. Some examples of a snippet might be: + * An implementation of a sort algorithm + * An interesting SwiftUI view hierarchy + * A quick recipe for displaying a 3D model in a view + * A struct that demonstrates a Swift Argument Parser option + * An example of Swift Coding with custom coding keys + * Many of the examples in [Swift Algorithms’s “Guides”](https://github.com/apple/swift-algorithms/tree/main/Guides) + * Many StackOverflow answers +* **Full sample projects and tutorials.** Here, a project demonstrates a full application, composing not just one or more APIs, but potentially many technologies or modules. A sample project also demonstrates multiple, potentially independent scenarios. Most developers are already familiar with these. + +**Once written, snippets are useful in many contexts.** Both sample projects and inline code listings are written once and meant to be consumed in one particular manner. In contrast, snippets are meant to be written once but read (or even run) anywhere. Snippets are just simple bits of code (but with access to the full package), great for importing within documentation prose, runnable from the command line, or copied and edited within an IDE. + +**Short, focused, single files.** This versatility comes with constraints — snippets should be small and focused, for instance. They should also stand on their own and not require a complex scenario to be understood. With these constraints, snippets can then be easily shuttled around, shown inline in docs, used in interactive code tutorials, run from the command line, quickly gleaned, and provide useful code for a developer to take on the spot. As soon as a snippet feels like it needs multiple files or resources, a traditional sample project starts to become appropriate. But with full access to the rest of the package, it may make sense to group “big” functionality elsewhere in the package, allowing each snippet to remain small, focused, and easily understood. + +**The possibility of snippet-only packages.** While sample code projects' strength is their depth in a specific scenario (often application development), packages consisting mostly or entirely of snippets provide breadth. Examples of snippet-only packages might be a collection of recipes for composing UI elements in new and interesting ways, teaching the Swift language snippet by snippet, or providing exercises for a textbook. Again, since snippets get access to the package's shared code libraries, it is possible to demonstrate even powerful concepts in an easy-to-read snippet. + +## Proposed Solution + +This proposal is a definition of a sample code convention that offers a bite-sized hybrid of sample code and API reference, with a sprinkle of prose: snippets. Snippets are individual `.swift` files that can build and run as executable targets in a Swift package. Each snippet is a complete program, that stands on its own as a token of documentation. + +### Writing a snippet + +A snippet file might look like the following: + +```swift +// The first contiguous line comments +// serve as the snippet's short description. + +func someCodeToShow() { + print("Hello, world!") +} + +// snippet.hide + +func someCodeToHide() { + print("Some demo message") +} + +// Still hidden +someCodeToHide() + +// snippet.show + +someCodeToShow() +``` + +At the top is the snippet's description written in Markdown, typically a short paragraph that may appear with the snippet. `// snippet.hide` and `// snippet.show` toggle hiding and showing code when displaying a snippet in documentation or other tools that may support snippets in the future. This lets the author add some additional demo logic when running the snippet while still keeping its presentation clean when it shows up the finished documentation. + +The above snippet would end up looking something like this within the docs: + +> The first contiguous line comments serve as the snippet's short description. +> +> ```swift +> func someCodeToShow() { +> print("Hello, world!") +> } +> someCodeToShow() +> ``` + +This code extracted after resolving hiding markers is known as a snippet’s *presentation code*. + +### Slices + +When snippets exist in documentation, each code block often continues from the previous one in a sequential narrative, alternating between code and prose. For example: + +> First, call `setup()` to initialize the context: +> +> ```swift +> let context = setup() +> ``` +> +> Then, call `request(_:)` with the desired mode: +> +> ```swift +> context.request(.immediate) +> ``` + +The second code block refers to `context` defined in the first so, for the purposes of compilation, the snippet is comprised of two code blocks. To support this, an author can write the code in a single file and "slice" it with an identifiers, referring to them in the documentation. Here is what the snippet for the above might look like in the Swift source file: + +```swift +// snippet.setup +let context = setup() + +// snippet.request +context.request(.immediate) +``` + +The special comment marker takes the form `// snippet.IDENTIFIER`, where `IDENTIFIER` is a URL-compatible path component in order to be compatible with DocC link resolution logic. Starting a new slice automatically terminates the previous slice. For slices that aren't adjacent, one can use `// snippet.end` to end the current slice: + +```swift +// snippet.setup +let context = setup() +// snippet.end + +// More code here... + +// snippet.request +context.request(.immediate) +``` + +You can also mix show/hide markers with slices: + +```swift +// snippet.setup +let context = setup() + +// snippet.hide +// More code here... +// snippet.show + +// snippet.request +context.request(.immediate) +``` + +### Getting started + +To start adding snippets to a package, first create a `Snippets` directory alongside a package's familiar `Sources` and `Tests` directories. From here, you can start dropping in `.swift` files. Base filenames must be unique from one another. SwiftPM will assume each of these comprise their own executable targets, so it can build and run them for the host platform as it would any other executable. + +After getting started, a package might start to look like the following: + + +``` +📁 MyPackage + 📁 Package.swift + 📁 Sources + 📁 Tests + 📂 Snippets + 📄 Snippet1.swift + 📄 Snippet2.swift + 📄 Snippet3.swift + +``` + +### Grouping + +To help organizing a growing number of snippets, you can also create one additional level of subdirectories under `Snippets` . This does not affect snippet links as shown below. + + +``` +📁 MyPackage + 📁 Package.swift + 📁 Sources + 📁 Tests + 📂 Snippets + 📁 Group1 + 📄 Snippet1.swift + 📄 Snippet2.swift + 📄 Snippet3.swift + 📁 Group2 + 📄 Snippet4.swift + 📄 Snippet5.swift +``` + +### Overriding the location of snippets + +Similar to the `./Sources` and `./Tests` directories, a user may want to override the location of the `./Snippets` directory with a new, optional `snippetsDirectory` argument to the `Package` initializer. Since snippet targets aren't declared individually in the manifest, the setting exists at the package level. + +```swift +let package = Package( + name: "MyPackage", + snippetsDirectory: "Examples", + products: [ + // ... + ], + dependencies: [ + // ... + ], + targets: [ + // ... + ] +) +``` + +> For the remainder of the document, examples will assume the default `Snippets` directory. + +### Using snippets in Swift-DocC documentation + +Swift-DocC (or other documentation tools) can then import snippets within prose Markdown files. For DocC, a snippet's description and code will appear anywhere you use a new block directive called `@Snippet`, with a single required `path` argument: + + + +```swift +@Snippet(path: "my-package/Snippets/Snippet1") +``` + +The `path` argument consists of the following three components: + + +* `my-package` : The package name, as taken from the `Package.swift` manifest. +* `Snippets`: An informal namespace to differentiate snippets from symbols and articles. This is the same regardless of the `snippetsDirectory` override mentioned above. +* `Snippet1`: The snippet name taken from the snippet file basename without extension. + +To insert a snippet slice, add the optional `slice` argument with the matching identifier in the source: + +```swift +@Snippet(path: "my-package/Snippets/Snippet1", slice: "setup") +``` + +### Building and running snippets + +After creating snippets, the Swift Package Manager can build and run them in the same way as executable targets. + +Snippet targets will be built by default when running `swift build --build-snippets`. This is consistent with how building tests is an explicit choice, like when running `swift test` or `swift build --build-tests`. It’s recommended to build snippets in all CI build processes or at least when building documentation. + +Example usage: + +```bash +swift build # Builds source targets as usual, + # but excluding tests and snippets. + +swift build --build-snippets # Builds source targets, including snippets. + +swift build Snippet1 # Build the snippet, Snippet1.swift, as an executable. + +swift run Snippet1 # Run the snippet, Snippet1.swift. +``` + +### Testing snippets + +While the code exemplified in snippets should already be covered by tests, an author may want to assert specific behavior when running a snippet. While we could use a test library like XCTest, it comes with platform-specific considerations and difficulties with execution–XCTest assertions can’t be collected and logged without a platform-specific test harness to execute the tests. Again, thinking about snippets as a kind of executable, how does one assert behavior in an executable? With asserts and preconditions. These should be enough for a majority of use cases, while letting interactive and non-interactive snippets to live side-by-side and treated the same for now. It is important that snippets are testable within CI and external testing solutions to provide additional automation to make this happen in one step. + +**Example:** + +```swift +let numbers = [20, 19, 7, 12] +let numbersMap = numbers.map({ (number: Int) -> Int in + return 3 * number +}) + +// snippet.hide +print(numbersMap) +precondition(numbersMap == [60, 57, 21, 36]) +``` + +## Detailed Design + +### Swift Package Manager + +When constructing the set of available targets for a package, SwiftPM will automatically find snippet files with the following pattern: + +* `./Snippets/**.swift*` +* `./Snippets/*/*.swift` + +These will each become a new kind of `.snippet` target behaving more or less as existing executable targets. A single level of subdirectories is allowed to balance filesystem organization and further subdirectories for snippet-related resources, which are expected to be found informally using relative paths. + +Snippet targets automatically depend on any library targets declared in their host package, so snippets are free to import those modules. In the future, in order to support snippet-only packages, packages that illustrate combining two independenct packages, or packages that require helper libraries for snippets, snippets will be able to import libraries from dependent packages declared in the manifest as well (see Future Directions below). + +### SymbolKit + +Snippets will be communicated to DocC via Symbol Graph JSON, with each snippet becoming a kind of symbol. + +Snippet symbols will include two primary pieces of information: a description carried as the symbol’s “documentation comment”, and presentation code via new mix-in called `Snippet`: + +```swift +public struct Snippet: Mixin, Codable { + public struct Slice: Codable { + public var name: String? + public var language: String? + public var code: String + } + public var slices: [Slice] +} +``` + +When a snippet doesn't have any slice comments, the above snippet model will consist of one slice containing all of the visible code. + +### Swift-DocC + +Swift-DocC will need to do the following to support snippets: + +**Look for and register occurrences of the new snippet mix-ins in symbol graph JSON.** By treating snippets as symbols, this mostly comes for free with the SymbolKit data model. + +**Add support for the new `@Snippet` directive**, checking the `path` and `slice` arguments with the same logic as symbol links. This comes in the form of a new `Semantic` instance: + +```swift +public final class Snippet: Semantic, DirectiveConvertible { + public static let directiveName = "Snippet" + // etc. +} +``` + +**Convert** `@Snippet` **occurrences to paragraphs and code blocks** as needed in the `RenderContentCompiler`, resulting in the following content for each occurrence: + +If the `@Snippet` is a slice, only: +* The slice code as a `CodeBlock`. + +If the `@Snippet` is not a slice: +* The documentation comment Markdown processed as normal for a symbol, a list of block elements. +* For each snippet slice: + * The slice code as a `CodeBlock`. + +### Swift-DocC Plugin + +The recently added [Swift DocC Plugin](https://github.com/apple/swift-docc-plugin) is a new [SwiftPM command plugin](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md) that builds documentation for SwiftPM libraries and executables. + +In order to forward a package's snippet information to DocC, a new tool, `snippet-build`, is added to convert `.swift` files into Symbol Graph JSON, which the plugin will run before `docc`. + +The `snippet-build` tool crawls the `Snippets` directory structure in the same way as SwiftPM, looking for `.swift` files. For each file, a snippet symbol entry is created in a Symbol Graph, and emitted into an output directory. The tool’s usage looks like the following: + +``` +USAGE: snippet-build + +ARGUMENTS: + - The directory containing Swift snippets + - The directory in which to place Symbol Graph JSON file(s) representing the snippets + - The module name to use for the Symbol Graph (typically should be the package name) +``` + +It’s not expected that a person will run this command manually. + +#### A note on Swift plugin dependencies + +Because SwiftPM plugins fold their dependencies into the plugin client’s dependency graph, some useful but minor dependencies were dropped to prevent the possibility for dependency cycles or conflicts: + +* **Swift Argument Parser.** This is a common dependency for lots of packages so the `snippet-build` tool implements argument parsing manually using positional arguments. It’s not expected that the usage will change over time. +* **Swift Syntax.** This could be useful for tokenizing code blocks, but DocC implements syntactic highlighting in the `Swift-DocC-Render` project. + +This current restriction on dependencies is one motivating factor for investigating moving Symbol Graph generation from `.swift` files down to the compiler. This would have nearly identical usage to [existing functionality to emit library and executable symbol graphs](https://github.com/apple/swift/tree/main/lib/SymbolGraphGen) today. More on this below. + +## Source compatibility + +Proposed changes to enable snippets do not break source compatibility. + +## Effect on ABI stability + +Proposed changes to enable snippets do not break ABI stability. + +## Effect on API resilience + +Proposed changes to enable snippets do not impact API resilience. + +## Alternatives considered + +### Literate approach: snippets within Markdown + +Another option was to support something like “[literate programming](http://www.literateprogramming.com/)” where source code is embedded in Markdown documentation files. In this approach, new tools and workflows would be created to extract code from the documentation, assemble that code into valid Swift files or packages, then build, run, and test that code. That tooling would likely use a custom file format with the ability to hide setup and test code, control imports, and more. The goal is to let documentation authors write bits of code inline, but to add tooling to validate the code. Literate programming is very interesting, and may be a good project for Swift, but it is not a small undertaking, and not likely to integrate well with existing tooling. + +Snippets, in contrast, are intended to primarily act as small sample programs that work with existing tooling. It should be super easy for anyone to look at a snippet as just source code, see how it works, remix it, and run it. Snippets should be easy to share, and even paste into a StackOverflow answer. + +At their core, snippets are simply `.swift` files, with conventions in place to make them really easy to fit into existing documentation tools, editors, IDEs, CLI commands, and CI systems. Code conforming to the snippets convention is straight forward to support within [Swift-DocC](https://github.com/apple/swift-docc) documentation tooling, as well as to build a nice CLI to discover, view, and quickly run snippets within a package. + +### Snippets in documentation comments + +Writing snippets exclusively in documentation comments limits their utility, putting too much focus on only documenting APIs within a module. More interesting uses for snippets would be left behind, such as composing functionality across multiple modules, or packages of just snippets for educational purposes. + +### Snippets as playgrounds + +Why aren’t these just playgrounds? While playgrounds started out very similarly to snippets, they have evolved into something more powerful, more tied to custom tooling, and a bit more complex. Playgrounds tend to tell a story, and are stand-alone entities with their own supporting files and sources. + +For open source Swift, packages already have a model for building targets that have multiple files and resources, and in fact, we’re seeing playgrounds migrating more toward looking like packages. + +Snippets are intentionally small programs written as a simple `.swift` file, integrated closely with the Swift Package Manager approach, as is Swift-DocC. + +### Tests acting as snippets + +Snippets are not meant to be tests or come directly from tests, although they may include their own testing and assertions to validate behavior. While tests may use public API in similar ways, the context in which one writes and thinks about tests is usually different from writing example code. For those tests that do match common use cases very well, it may be possible in the future to extract snippets from multiple sources (see below). + +## Future Directions + +**Multiple snippets per file.** In the future there is the option to create multiple snippets per file, where each snippet’s identifier is expressed as a kind of start/end marker in source code. + +**Multi-file snippets.** This could manifest in a couple ways. First, requiring several files to build a snippet already exists in the form sample target or project, so this is probably not a future goal. However, for snippets embedded within existing multi-file projects, it may be possible extract those snippets during build time. This will likely require that the snippet extraction move down to the compiler. + +**Extract snippets while building.** To facilitate some of the above future possibilities and others, the `snippet-build` tool may move down to the `SymbolGraphGen` library that coverts modules into Symbol Graph JSON. Since snippets are communicated with the same Symbol Graph format, moving the implementation down to the compiler will allow utilizing shared implementation and semantic information for future enhancements. This would allow snippets to be pulled from different kinds of sources: from libraries, unit tests, larger sample projects, etc. + +**Build snippets when building documentation.** The current Swift-DocC implementation only requires reading snippet source files when rendering documentation, so building is not required. Depending on whether the implementation is moved down to the compiler, this could be implemented by having the Swift-DocC plugin request snippet builds before generating documentation, or implicitly as the compiler builds snippets to generate symbol graphs. + +**Snippet dependencies.** While snippets automatically depend on any libraries defined in their host package, there may be packages that exist solely to illustrate using one or more libraries from other packages. In the future, we can add the ability to declare external dependencies to which snippets have access. diff --git a/proposals/0357-regex-string-processing-algorithms.md b/proposals/0357-regex-string-processing-algorithms.md new file mode 100644 index 0000000000..1cbd60d3f3 --- /dev/null +++ b/proposals/0357-regex-string-processing-algorithms.md @@ -0,0 +1,1153 @@ +# Regex-powered string processing algorithms + +* Proposal: [SE-0357](0357-regex-string-processing-algorithms.md) +* Authors: [Tina Liu](https://github.com/itingliu), [Michael Ilseman](https://github.com/milseman), [Nate Cook](https://github.com/natecook1000), [Tim Vermeulen](https://github.com/timvermeulen) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift-experimental-string-processing](https://github.com/apple/swift-experimental-string-processing/) + * Available in nightly toolchain snapshots with `import _StringProcessing` +* Review: ([pitch](https://forums.swift.org/t/pitch-regex-powered-string-processing-algorithms/55969)) + ([review](https://forums.swift.org/t/se-0357-regex-string-processing-algorithms/57225)) + ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0357-regex-string-processing-algorithms/58706)) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/7741017763f528dfbdfa54c6d11f559918ab53e4/proposals/0357-regex-string-processing-algorithms.md) + +## Introduction + +The Swift standard library's string processing algorithms are underpowered compared to other popular programming and scripting languages. Some of these omissions can be found in `NSString`, but these fundamental algorithms should have a place in the standard library. + +We propose: + +1. New regex-powered algorithms over strings, bringing the standard library up to parity with scripting languages +2. Generic `Collection` equivalents of these algorithms in terms of subsequences +3. `protocol CustomConsumingRegexComponent`, which allows 3rd party libraries to provide their industrial-strength parsers as intermixable components of regexes + +This proposal is part of a larger [regex-powered string processing initiative](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md), the status of each proposal is tracked [here](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md). Further discussion of regex specifics is out of scope of this proposal and better discussed in their relevant reviews. + +## Motivation + +A number of common string processing APIs are missing from the Swift standard library. While most of the desired functionalities can be accomplished through a series of API calls, every gap adds a burden to developers doing frequent or complex string processing. For example, here's one approach to find the number of occurrences of a substring ("banana") within a string: + +```swift +let str = "A banana a day keeps the doctor away. I love bananas; banana are my favorite fruit." + +var idx = str.startIndex +var ranges = [Range]() +while let r = str.range(of: "banana", options: [], range: idx.. + Comparison of how Swift's APIs stack up with Python's. + +Note: Only a subset of Python's string processing API are included in this table for the following reasons: + +- Functions to query if all characters in the string are of a specified category, such as `isalnum()` and `isalpha()`, are omitted. These are achievable in Swift by passing in the corresponding character set to `allSatisfy(_:)`, so they're omitted in this table for simplicity. +- String formatting functions such as `center(length, character)` and `ljust(width, fillchar)` are also excluded here as this proposal focuses on matching and searching functionalities. + +##### Search and replace + +|Python |Swift | +|--- |--- | +| `count(sub, start, end)` | | +| `find(sub, start, end)`, `index(sub, start, end)` | `firstIndex(where:)` | +| `rfind(sub, start, end)`, `rindex(sub, start, end)` | `lastIndex(where:)` | +| `expandtabs(tabsize)`, `replace(old, new, count)` | `Foundation.replacingOccurrences(of:with:)` | +| `maketrans(x, y, z)` + `translate(table)` | + +##### Prefix and suffix matching + +|Python |Swift | +|--- |--- | +| `startswith(prefix, start, end)` | `starts(with:)` or `hasPrefix(:)`| +| `endswith(suffix, start, end)` | `hasSuffix(:)` | +| `removeprefix(prefix)` | Test if string has prefix with `hasPrefix(:)`, then drop the prefix with `dropFirst(:)`| +| `removesuffix(suffix)` | Test if string has suffix with `hasSuffix(:)`, then drop the suffix with `dropLast(:)` | + +##### Strip / trim + +|Python |Swift | +|--- |--- | +| `strip([chars])`| `Foundation.trimmingCharacters(in:)` | +| `lstrip([chars])` | `drop(while:)` | +| `rstrip([chars])` | Test character equality, then `dropLast()` iteratively | + +##### Split + +|Python |Swift | +|--- |--- | +| `partition(sep)` | `Foundation.components(separatedBy:)` | +| `rpartition(sep)` | | +| `split(sep, maxsplit)` | `split(separator:maxSplits:...)` | +| `splitlines(keepends)` | `split(separator:maxSplits:...)` | +| `rsplit(sep, maxsplit)` | | + + + + + +### Complex string processing + +Even with the API additions, more complex string processing quickly becomes unwieldy. String processing in the modern world involves dealing with localization, standards-conforming validation, and other concerns for which a dedicated parser is required. + +Consider parsing the date field `"Date: Wed, 16 Feb 2022 23:53:19 GMT"` in an HTTP header as a `Date` type. The naive approach is to search for a substring that looks like a date string (`16 Feb 2022`), and attempt to post-process it as a `Date` with a date parser: + +```swift +let regex = Regex { + Capture { + OneOrMore(.digit) + " " + OneOrMore(.word) + " " + OneOrMore(.digit) + } +} + +let dateParser = Date.ParseStrategy(format: "\(day: .twoDigits) \(month: .abbreviated) \(year: .padded(4))" +if let dateMatch = header.firstMatch(of: regex)?.0 { + let date = try? Date(dateMatch, strategy: dateParser) +} +``` + +This requires writing a simplistic pre-parser before invoking the real parser. The pre-parser will suffer from being out-of-sync and less featureful than what the real parser can do. + +Or consider parsing a bank statement to record all the monetary values in the last column: + +```swift +let statement = """ +CREDIT 04/06/2020 Paypal transfer $4.99 +CREDIT 04/03/2020 Payroll $69.73 +DEBIT 04/02/2020 ACH transfer ($38.25) +DEBIT 03/24/2020 IRX tax payment ($52,249.98) +""" +``` + +Parsing a currency string such as `$3,020.85` with regex is also tricky, as it can contain localized and currency symbols in addition to accounting conventions. This is why Foundation provides industrial-strength parsers for localized strings. + + +## Proposed solution + +### Complex string processing + +We propose a `CustomConsumingRegexComponent` protocol which allows types from outside the standard library participate in regex builders and `RegexComponent` algorithms. This allows types, such as `Date.ParseStrategy` and `FloatingPointFormatStyle.Currency`, to be used directly within a regex: + +```swift +let dateRegex = Regex { + Capture(dateParser) +} + +let date: Date = header.firstMatch(of: dateRegex).map(\.result.1) + +let currencyRegex = Regex { + Capture(.localizedCurrency(code: "USD").sign(strategy: .accounting)) +} + +let amount: [Decimal] = statement.matches(of: currencyRegex).map(\.result.1) +``` + +### String algorithm additions + +We also propose the following regex-powered algorithms as well as their generic `Collection` equivalents. See the Detailed design section for a complete list of variation and overloads . + +|Function | Description | +|--- |--- | +|`contains(_:) -> Bool` | Returns whether the collection contains the given sequence or `RegexComponent` | +|`starts(with:) -> Bool` | Returns whether the collection contains the same prefix as the specified `RegexComponent` | +|`trimPrefix(_:)`| Removes the prefix if it matches the given `RegexComponent` or collection | +|`firstRange(of:) -> Range?` | Finds the range of the first occurrence of a given sequence or `RegexComponent`| +|`ranges(of:) -> some Collection` | Finds the ranges of the all occurrences of a given sequence or `RegexComponent` within the collection | +|`replace(:with:subrange:maxReplacements)`| Replaces all occurrences of the sequence matching the given `RegexComponent` or sequence with a given collection | +|`split(by:)`| Returns the longest possible subsequences of the collection around elements equal to the given separator | +|`firstMatch(of:)`| Returns the first match of the specified `RegexComponent` within the collection | +|`wholeMatch(of:)`| Matches the specified `RegexComponent` in the collection as a whole | +|`prefixMatch(of:)`| Matches the specified `RegexComponent` against the collection at the beginning | +|`matches(of:)`| Returns a collection containing all matches of the specified `RegexComponent` | + +## Detailed design + +### `CustomConsumingRegexComponent` + +`CustomConsumingRegexComponent` inherits from `RegexComponent` and satisfies its sole requirement. Conformers can be used with all of the string algorithms generic over `RegexComponent`. + +```swift +/// A protocol allowing custom types to function as regex components by +/// providing the raw functionality backing `prefixMatch`. +public protocol CustomConsumingRegexComponent: RegexComponent { + /// Process the input string within the specified bounds, beginning at the given index, and return + /// the end position (upper bound) of the match and the produced output. + /// - Parameters: + /// - input: The string in which the match is performed. + /// - index: An index of `input` at which to begin matching. + /// - bounds: The bounds in `input` in which the match is performed. + /// - Returns: The upper bound where the match terminates and a matched instance, or `nil` if + /// there isn't a match. + func consuming( + _ input: String, + startingAt index: String.Index, + in bounds: Range + ) throws -> (upperBound: String.Index, output: RegexOutput)? +} +``` + +
+Example for protocol conformance + +We use Foundation `FloatingPointFormatStyle.Currency` as an example for protocol conformance. It would implement the `match` function with `Match` being a `Decimal`. It could also add a static function `.localizedCurrency(code:)` as a member of `RegexComponent`, so it can be referred as `.localizedCurrency(code:)` in the `Regex` result builder: + +```swift +extension FloatingPointFormatStyle.Currency : CustomConsumingRegexComponent { + public func consuming( + _ input: String, + startingAt index: String.Index, + in bounds: Range + ) -> (upperBound: String.Index, match: Decimal)? +} + +extension RegexComponent where Self == FloatingPointFormatStyle.Currency { + public static func localizedCurrency(code: Locale.Currency) -> Self +} +``` + +Matching and extracting a localized currency amount, such as `"$3,020.85"`, can be done directly within a regex: + +```swift +let regex = Regex { + Capture(.localizedCurrency(code: "USD")) +} +``` + +
+ + +### String and Collection algorithm additions + +#### Contains + +We propose a `contains` variant over collections that tests for subsequence membership. The second algorithm allows for specialization using e.g. the [two way search algorithm](https://en.wikipedia.org/wiki/Two-way_string-matching_algorithm). + +```swift +extension Collection where Element: Equatable { + /// Returns a Boolean value indicating whether the collection contains the + /// given sequence. + /// - Parameter other: A sequence to search for within this collection. + /// - Returns: `true` if the collection contains the specified sequence, + /// otherwise `false`. + public func contains(_ other: C) -> Bool + where S.Element == Element +} +extension BidirectionalCollection where Element: Comparable { + /// Returns a Boolean value indicating whether the collection contains the + /// given sequence. + /// - Parameter other: A sequence to search for within this collection. + /// - Returns: `true` if the collection contains the specified sequence, + /// otherwise `false`. + public func contains(_ other: C) -> Bool + where S.Element == Element +} +``` + +We propose a regex-taking variant over string types (those that produce a `Substring` upon slicing). + +```swift +extension Collection where SubSequence == Substring { + /// Returns a Boolean value indicating whether the collection contains the + /// given regex. + /// - Parameter regex: A regex to search for within this collection. + /// - Returns: `true` if the regex was found in the collection, otherwise + /// `false`. + public func contains(_ regex: some RegexComponent) -> Bool +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a Boolean value indicating whether this collection contains a + /// match for the regex, where the regex is created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to search for within + /// this collection. + /// - Returns: `true` if the regex returned by `content` matched anywhere in + /// this collection, otherwise `false`. + public func contains( + @RegexComponentBuilder _ content: () -> some RegexComponent + ) -> Bool +} +``` + +#### Starts with + +We propose a regex-taking `starts(with:)` variant for string types: + +```swift +extension Collection where SubSequence == Substring { + /// Returns a Boolean value indicating whether the initial elements of the + /// sequence are the same as the elements in the specified regex. + /// - Parameter regex: A regex to compare to this sequence. + /// - Returns: `true` if the initial elements of the sequence matches the + /// beginning of `regex`; otherwise, `false`. + public func starts(with regex: some RegexComponent) -> Bool +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a Boolean value indicating whether the initial elements of this + /// collection are a match for the regex created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to match at + /// the beginning of this collection. + /// - Returns: `true` if the initial elements of this collection match + /// regex returned by `content`; otherwise, `false`. + public func starts( + @RegexComponentBuilder with content: () -> some RegexComponent + ) -> Bool +} +``` + +#### Trim prefix + +We propose generic `trimmingPrefix` and `trimPrefix` methods for collections that trim elements matching a predicate or a possible prefix sequence. + +```swift +extension Collection { + /// Returns a new collection of the same type by removing initial elements + /// that satisfy the given predicate from the start. + /// - Parameter predicate: A closure that takes an element of the sequence + /// as its argument and returns a Boolean value indicating whether the + /// element should be removed from the collection. + /// - Returns: A collection containing the elements of the collection that are + /// not removed by `predicate`. + public func trimmingPrefix(while predicate: (Element) throws -> Bool) rethrows -> SubSequence +} + +extension Collection where SubSequence == Self { + /// Removes the initial elements that satisfy the given predicate from the + /// start of the sequence. + /// - Parameter predicate: A closure that takes an element of the sequence + /// as its argument and returns a Boolean value indicating whether the + /// element should be removed from the collection. + public mutating func trimPrefix(while predicate: (Element) throws -> Bool) rethrows +} + +extension RangeReplaceableCollection { + /// Removes the initial elements that satisfy the given predicate from the + /// start of the sequence. + /// - Parameter predicate: A closure that takes an element of the sequence + /// as its argument and returns a Boolean value indicating whether the + /// element should be removed from the collection. + public mutating func trimPrefix(while predicate: (Element) throws -> Bool) rethrows +} + +extension Collection where Element: Equatable { + /// Returns a new collection of the same type by removing `prefix` from the + /// start. + /// - Parameter prefix: The collection to remove from this collection. + /// - Returns: A collection containing the elements that does not match + /// `prefix` from the start. + public func trimmingPrefix(_ prefix: Prefix) -> SubSequence + where Prefix.Element == Element +} + +extension Collection where SubSequence == Self, Element: Equatable { + /// Removes the initial elements that matches `prefix` from the start. + /// - Parameter prefix: The collection to remove from this collection. + public mutating func trimPrefix(_ prefix: Prefix) + where Prefix.Element == Element +} + +extension RangeReplaceableCollection where Element: Equatable { + /// Removes the initial elements that matches `prefix` from the start. + /// - Parameter prefix: The collection to remove from this collection. + public mutating func trimPrefix(_ prefix: Prefix) + where Prefix.Element == Element +} +``` + +We propose regex-taking variants for string types: + +```swift +extension Collection where SubSequence == Substring { + /// Returns a new subsequence by removing the initial elements that matches + /// the given regex. + /// - Parameter regex: The regex to remove from this collection. + /// - Returns: A new subsequence containing the elements of the collection + /// that does not match `prefix` from the start. + public func trimmingPrefix(_ regex: some RegexComponent) -> SubSequence +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a subsequence of this collection by removing the elements + /// matching the regex from the start, where the regex is created by + /// the given closure. + /// + /// - Parameter content: A closure that returns the regex to search for at + /// the start of this collection. + /// - Returns: A collection containing the elements after those that match + /// the regex returned by `content`. If the regex does not match at + /// the start of the collection, the entire contents of this collection + /// are returned. + public func trimmingPrefix( + @RegexComponentBuilder _ content: () -> some RegexComponent + ) -> SubSequence +} + +extension RangeReplaceableCollection where SubSequence == Substring { + /// Removes the initial elements that matches the given regex. + /// - Parameter regex: The regex to remove from this collection. + public mutating func trimPrefix(_ regex: some RegexComponent) +} + +// In RegexBuilder module +extension RangeReplaceableCollection where SubSequence == Substring { + /// Removes the initial elements matching the regex from the start of + /// this collection, if the initial elements match, using the given closure + /// to create the regex. + /// + /// - Parameter content: A closure that returns the regex to search for + /// at the start of this collection. + public mutating func trimPrefix( + @RegexComponentBuilder _ content: () -> some RegexComponent + ) +} +``` + +#### First range + +We propose a generic collection algorithm for finding the first range of a given subsequence: + +```swift +extension Collection where Element: Equatable { + /// Finds and returns the range of the first occurrence of a given sequence + /// within the collection. + /// - Parameter sequence: The sequence to search for. + /// - Returns: A range in the collection of the first occurrence of `sequence`. + /// Returns nil if `sequence` is not found. + public func firstRange(of other: C) -> Range? + where C.Element == Element +} + +extension BidirectionalCollection where Element: Comparable { + /// Finds and returns the range of the first occurrence of a given sequence + /// within the collection. + /// - Parameter other: The sequence to search for. + /// - Returns: A range in the collection of the first occurrence of `sequence`. + /// Returns `nil` if `sequence` is not found. + public func firstRange(of other: C) -> Range? + where C.Element == Element +} +``` + +We propose a regex-taking variant for string types. + +```swift +extension Collection where SubSequence == Substring { + /// Finds and returns the range of the first occurrence of a given regex + /// within the collection. + /// - Parameter regex: The regex to search for. + /// - Returns: A range in the collection of the first occurrence of `regex`. + /// Returns `nil` if `regex` is not found. + public func firstRange(of regex: some RegexComponent) -> Range? +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns the range of the first match for the regex within this collection, + /// where the regex is created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to search for. + /// - Returns: A range in the collection of the first occurrence of the first + /// match of if the regex returned by `content`. Returns `nil` if no match + /// for the regex is found. + public func firstRange( + @RegexComponentBuilder of content: () -> some RegexComponent + ) -> Range? +} +``` + +#### Ranges + +We propose a generic collection algorithm for iterating over all (non-overlapping) ranges of a given subsequence. + +```swift +extension Collection where Element: Equatable { + /// Finds and returns the ranges of the all occurrences of a given sequence + /// within the collection. + /// - Parameter other: The sequence to search for. + /// - Returns: A collection of ranges of all occurrences of `other`. Returns + /// an empty collection if `other` is not found. + public func ranges(of other: C) -> some Collection> + where C.Element == Element +} + +extension BidirectionalCollection where Element: Comparable { + /// Finds and returns the ranges of the all occurrences of a given sequence + /// within the collection. + /// - Parameter other: The sequence to search for. + /// - Returns: A collection of ranges of all occurrences of `other`. Returns + /// an empty collection if `other` is not found. + public func ranges(of other: C) -> some Collection> + where C.Element == Element +} +``` + +And of course regex-taking versions for string types: + +```swift +extension Collection where SubSequence == Substring { + /// Finds and returns the ranges of the all occurrences of a given sequence + /// within the collection. + /// - Parameter regex: The regex to search for. + /// - Returns: A collection or ranges in the receiver of all occurrences of + /// `regex`. Returns an empty collection if `regex` is not found. + public func ranges(of regex: some RegexComponent) -> some Collection> +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns the ranges of the all non-overlapping matches for the regex + /// within this collection, where the regex is created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to search for. + /// - Returns: A collection of ranges of all matches for the regex returned by + /// `content`. Returns an empty collection if no match for the regex + /// is found. + public func ranges( + @RegexComponentBuilder of content: () -> some RegexComponent + ) -> some Collection> +} +``` + +#### Match + +We propose algorithms for extracting a `Match` instance from a given regex from the start, anywhere in the middle, or over the entire `self`. + +```swift +extension Collection where SubSequence == Substring { + /// Returns the first match of the specified regex within the collection. + /// - Parameter regex: The regex to search for. + /// - Returns: The first match of `regex` in the collection, or `nil` if + /// there isn't a match. + public func firstMatch(of regex: R) -> Regex.Match? + + /// Match a regex in its entirety. + /// - Parameter regex: The regex to match against. + /// - Returns: The match if there is one, or `nil` if none. + public func wholeMatch(of regex: R) -> Regex.Match? + + /// Match part of the regex, starting at the beginning. + /// - Parameter regex: The regex to match against. + /// - Returns: The match if there is one, or `nil` if none. + public func prefixMatch(of regex: R) -> Regex.Match? +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns the first match for the regex within this collection, where + /// the regex is created by the given closure. + /// + /// - Parameter content: A closure that returns the regex to search for. + /// - Returns: The first match for the regex created by `content` in this + /// collection, or `nil` if no match is found. + public func firstMatch( + @RegexComponentBuilder of content: () -> R + ) -> Regex.Match? + + /// Matches a regex in its entirety, where the regex is created by + /// the given closure. + /// + /// - Parameter content: A closure that returns a regex to match against. + /// - Returns: The match if there is one, or `nil` if none. + public func wholeMatch( + @RegexComponentBuilder of content: () -> R + ) -> Regex.Match? + + /// Matches part of the regex, starting at the beginning, where the regex + /// is created by the given closure. + /// + /// - Parameter content: A closure that returns a regex to match against. + /// - Returns: The match if there is one, or `nil` if none. + public func prefixMatch( + @RegexComponentBuilder of content: () -> R + ) -> Regex.Match? +} +``` + +#### Matches + +We propose an algorithm for iterating over all (non-overlapping) matches of a given regex: + +```swift +extension Collection where SubSequence == Substring { + /// Returns a collection containing all matches of the specified regex. + /// - Parameter regex: The regex to search for. + /// - Returns: A collection of matches of `regex`. + public func matches(of regex: R) -> some Collection.Match> +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a collection containing all non-overlapping matches of + /// the regex, created by the given closure. + /// + /// - Parameter content: A closure that returns the regex to search for. + /// - Returns: A collection of matches for the regex returned by `content`. + /// If no matches are found, the returned collection is empty. + public func matches( + @RegexComponentBuilder of content: () -> R + ) -> some Collection.Match> +} +``` + +#### Replace + +We propose generic collection algorithms that will replace all occurrences of a given subsequence: + +```swift +extension RangeReplaceableCollection where Element: Equatable { + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - subrange: The range in the collection in which to search for `other`. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: C, + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max + ) -> Self where C.Element == Element, Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: C, + with replacement: Replacement, + maxReplacements: Int = .max + ) -> Self where C.Element == Element, Replacement.Element == Element + + /// Replaces all occurrences of a target sequence with a given collection + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + public mutating func replace( + _ other: C, + with replacement: Replacement, + maxReplacements: Int = .max + ) where C.Element == Element, Replacement.Element == Element +} +extension RangeReplaceableCollection where Self: BidirectionalCollection, Element: Comparable { + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - subrange: The range in the collection in which to search for `other`. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: C, + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max + ) -> Self where C.Element == Element, Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of `other` in + /// `subrange` of the collection are replaced by `replacement`. + public func replacing( + _ other: C, + with replacement: Replacement, + maxReplacements: Int = .max + ) -> Self where C.Element == Element, Replacement.Element == Element + + /// Replaces all occurrences of a target sequence with a given collection + /// - Parameters: + /// - other: The sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of `other` + /// to replace. Default is `Int.max`. + public mutating func replace( + _ other: C, + with replacement: Replacement, + maxReplacements: Int = .max + ) where C.Element == Element, Replacement.Element == Element +} +``` + +We propose regex-taking variants for string types as well as variants that take a closure which will generate the replacement portion from a regex match (e.g. by reading captures). + +```swift +extension RangeReplaceableCollection where SubSequence == Substring { + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - subrange: The range in the collection in which to search for `regex`. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` in `subrange` are replaced by `replacement`. + public func replacing( + _ r: some RegexComponent, + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max + ) -> Self where Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` are replaced by `replacement`. + public func replacing( + _ r: some RegexComponent, + with replacement: Replacement, + maxReplacements: Int = .max + ) -> Self where Replacement.Element == Element + + /// Replaces all occurrences of the sequence matching the given regex with + /// a given collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - replacement: The new elements to add to the collection. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + public mutating func replace( + _ r: some RegexComponent, + with replacement: Replacement, + maxReplacements: Int = .max + ) where Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another regex match. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - subrange: The range in the collection in which to search for `regex`. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` are replaced by `replacement`. + public func replacing( + _ regex: R, + subrange: Range, + maxReplacements: Int = .max, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows -> Self where Replacement.Element == Element + + /// Returns a new collection in which all occurrences of a sequence matching + /// the given regex are replaced by another collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - Returns: A new collection in which all occurrences of subsequence + /// matching `regex` are replaced by `replacement`. + public func replacing( + _ regex: R, + maxReplacements: Int = .max, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows -> Self where Replacement.Element == Element + + /// Replaces all occurrences of the sequence matching the given regex with + /// a given collection. + /// - Parameters: + /// - regex: A regex describing the sequence to replace. + /// - maxReplacements: A number specifying how many occurrences of the + /// sequence matching `regex` to replace. Default is `Int.max`. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + public mutating func replace( + _ regex: R, + maxReplacements: Int = .max, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows where Replacement.Element == Element +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns a new collection in which all matches for the regex + /// are replaced, using the given closure to create the regex. + /// + /// - Parameters: + /// - replacement: The new elements to add to the collection in place of + /// each match for the regex, using `content` to create the regex. + /// - subrange: The range in the collection in which to search for + /// the regex. + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - Returns: A new collection in which all matches for regex in `subrange` + /// are replaced by `replacement`, using `content` to create the regex. + public func replacing( + with replacement: Replacement, + subrange: Range, + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> some RegexComponent + ) -> Self where Replacement.Element == Element + + /// Returns a new collection in which all matches for the regex + /// are replaced, using the given closure to create the regex. + /// + /// - Parameters: + /// - replacement: The new elements to add to the collection in place of + /// each match for the regex, using `content` to create the regex. + /// - maxReplacements: A number specifying how many occurrences of regex + /// to replace. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - Returns: A new collection in which all matches for regex in `subrange` + /// are replaced by `replacement`, using `content` to create the regex. + public func replacing( + with replacement: Replacement, + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> some RegexComponent + ) -> Self where Replacement.Element == Element + + /// Replaces all matches for the regex in this collection, using the given + /// closure to create the regex. + /// + /// - Parameters: + /// - replacement: The new elements to add to the collection in place of + /// each match for the regex, using `content` to create the regex. + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace. + /// - content: A closure that returns the collection to search for + /// and replace. + public mutating func replace( + with replacement: Replacement, + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> some RegexComponent + ) where Replacement.Element == Element + + /// Returns a new collection in which all matches for the regex + /// are replaced, using the given closures to create the replacement + /// and the regex. + /// + /// - Parameters: + /// - subrange: The range in the collection in which to search for the + /// regex, using `content` to create the regex. + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - Returns: A new collection in which all matches for regex in `subrange` + /// are replaced by the result of calling `replacement`, where regex + /// is the result of calling `content`. + public func replacing( + subrange: Range, + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> R, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows -> Self where Replacement.Element == Element + + /// Returns a new collection in which all matches for the regex + /// are replaced, using the given closures to create the replacement + /// and the regex. + /// + /// - Parameters: + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace, using `content` to create the regex. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + /// - Returns: A new collection in which all matches for regex in `subrange` + /// are replaced by the result of calling `replacement`, where regex is + /// the result of calling `content`. + public func replacing( + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> R, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows -> Self where Replacement.Element == Element + + /// Replaces all matches for the regex in this collection, using the + /// given closures to create the replacement and the regex. + /// + /// - Parameters: + /// - maxReplacements: A number specifying how many occurrences of + /// the regex to replace, using `content` to create the regex. + /// - content: A closure that returns the collection to search for + /// and replace. + /// - replacement: A closure that receives the full match information, + /// including captures, and returns a replacement collection. + public mutating func replace( + maxReplacements: Int = .max, + @RegexComponentBuilder content: () -> R, + with replacement: (Regex.Match) throws -> Replacement + ) rethrows where Replacement.Element == Element +} +``` + +#### Split + +We propose a generic collection `split` that can take a subsequence separator: + +```swift +extension Collection where Element: Equatable { + /// Returns the longest possible subsequences of the collection, in order, + /// around elements equal to the given separator collection. + /// + /// - Parameters: + /// - separator: A collection of elements to be split upon. + /// - maxSplits: The maximum number of times to split the collection, + /// or one less than the number of subsequences to return. + /// - omittingEmptySubsequences: If `false`, an empty subsequence is + /// returned in the result for each consecutive pair of separator + /// sequences in the collection and for each instance of separator + /// sequences at the start or end of the collection. If `true`, only + /// nonempty subsequences are returned. + /// - Returns: A collection of subsequences, split from this collection's + /// elements. + public func split( + separator: C, + maxSplits: Int = Int.max, + omittingEmptySubsequences: Bool = true + ) -> some Collection where C.Element == Element +} +extension BidirectionalCollection where Element: Comparable { + /// Returns the longest possible subsequences of the collection, in order, + /// around elements equal to the given separator collection. + /// + /// - Parameters: + /// - separator: A collection of elements to be split upon. + /// - maxSplits: The maximum number of times to split the collection, + /// or one less than the number of subsequences to return. + /// - omittingEmptySubsequences: If `false`, an empty subsequence is + /// returned in the result for each consecutive pair of separator + /// sequences in the collection and for each instance of separator + /// sequences at the start or end of the collection. If `true`, only + /// nonempty subsequences are returned. + /// - Returns: A collection of subsequences, split from this collection's + /// elements. + public func split( + separator: C, + maxSplits: Int = Int.max, + omittingEmptySubsequences: Bool = true + ) -> some Collection where C.Element == Element +} +``` + +And a regex-taking variant for string types: + +```swift +extension Collection where SubSequence == Substring { + /// Returns the longest possible subsequences of the collection, in order, + /// around subsequence that match the given separator regex. + /// + /// - Parameters: + /// - separator: A regex to be split upon. + /// - maxSplits: The maximum number of times to split the collection, + /// or one less than the number of subsequences to return. + /// - omittingEmptySubsequences: If `false`, an empty subsequence is + /// returned in the result for each consecutive pair of matches + /// and for each match at the start or end of the collection. If + /// `true`, only nonempty subsequences are returned. + /// - Returns: A collection of substrings, split from this collection's + /// elements. + public func split( + separator: some RegexComponent, + maxSplits: Int = Int.max, + omittingEmptySubsequences: Bool = true + ) -> some Collection +} + +// In RegexBuilder module +extension Collection where SubSequence == Substring { + /// Returns the longest possible subsequences of the collection, in order, + /// around subsequence that match the regex created by the given closure. + /// + /// - Parameters: + /// - maxSplits: The maximum number of times to split the collection, + /// or one less than the number of subsequences to return. + /// - omittingEmptySubsequences: If `false`, an empty subsequence is + /// returned in the result for each consecutive pair of matches + /// and for each match at the start or end of the collection. If + /// `true`, only nonempty subsequences are returned. + /// - separator: A closure that returns a regex to be split upon. + /// - Returns: A collection of substrings, split from this collection's + /// elements. + public func split( + maxSplits: Int = Int.max, + omittingEmptySubsequences: Bool = true, + @RegexComponentBuilder separator: () -> some RegexComponent + ) -> some Collection +} +``` + +**Note:** We plan to adopt the new generics features enabled by [SE-0346][] for these proposed methods when the standard library adopts primary associated types, [pending a forthcoming proposal][stdlib-pitch]. For example, the first method in the _Replacement_ section above would instead be: + +```swift +extension RangeReplaceableCollection where Element: Equatable { + /// Returns a new collection in which all occurrences of a target sequence + /// are replaced by another collection. + public func replacing( + _ other: some Collection, + with replacement: some Collection, + subrange: Range, + maxReplacements: Int = .max + ) -> Self +} +``` + +#### Searching for empty strings and matches + +Empty matches and inputs are an important edge case for several of the algorithms proposed above. For example, what is the result of `"123.firstRange(of: /[a-z]*/)`? How do you split a collection separated by an empty collection, as in `"1234".split(separator: "")`? For the Swift standard library, this is a new consideration, as current algorithms are `Element`-based and cannot be passed an empty input. + +Languages and libraries are nearly unanimous about finding the location of an empty string, with Ruby, Python, C#, Java, Javascript, etc, finding an empty string at each index in the target. Notably, Foundation's `NSString.range(of:)` does _not_ find an empty string at all. + +The methods proposed here follow the consensus behavior, which makes sense if you think of `a.firstRange(of: b)` as returning the first subrange `r` where `a[r] == b`. If a regex can match an empty substring, like `/[a-z]*/`, the behavior is the same. + +```swift +let hello = "Hello" +let emptyRange = hello.firstRange(of: "") +// emptyRange is equivalent to '0..<0' (integer ranges shown for readability) +``` + +Because searching again at the same index would yield that same empty string, we advance one position after finding an empty string or matching an empty pattern when finding all ranges. This yields the position of every valid index in the string. + +```swift +let allRanges = hello.ranges(of: "") +// allRanges is equivalent to '[0..<0, 1..<1, 2..<2, 3..<3, 4..<4, 5..<5]' +``` + +Splitting with an empty separator (or a pattern that matches empty string), uses this same behavior, resulting in a collection of single-element substrings. Interestingly, a couple languages make different choices here. C# returns the original string instead of its parts, and Python rejects an empty separator (though it permits regexes that match empty strings). + +```swift +let parts = hello.split(separator: "") +// parts == ["h", "e", "l", "l", "o"] + +let moreParts = hello.split(separator: "", omittingEmptySubsequences: false) +// parts == ["", "h", "e", "l", "l", "o", ""] +``` + +Finally, searching for an empty string within an empty string yields, as you might imagine, the empty string: + +```swift +let empty = "" +let range = empty.firstRange(of: empty) +// empty == empty[range] +``` + +[SE-0346]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md +[stdlib-pitch]: https://forums.swift.org/t/pitch-primary-associated-types-in-the-standard-library/56426 + +## Alternatives considered + +### Extend `Sequence` instead of `Collection` + +Most of the proposed algorithms are necessarily on `Collection` due to the use of indices or mutation. `Sequence` does not support multi-pass iteration, so even `trimmingPrefix` would problematic on `Sequence` because it needs to look one `Element` ahead to know when to stop trimming and would need to return a wrapper for the in-progress iterator instead of a subsequence. + +### Cross-proposal API naming consistency + +The regex work is broken down into 6 proposals based on technical domain, which is advantageous for deeper technical discussions and makes reviewing the large body of work manageable. The disadvantage of this approach is that relatively-shallow cross-cutting concerns, such as API naming consistency, are harder to evaluate until we've built up intuition from multiple proposals. + +We've seen the [Regex type and overview](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md), the [Regex builder DSL](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0351-regex-builder.md), and here we present lots of ways to use regex. Now's a good time to go over API naming consistency. + +(The other proposal with a significant amount of API is [Unicode for String Processing](https://forums.swift.org/t/pitch-unicode-for-string-processing/56907), which is in the pitch phase. It is a technical niche and less impactful on these naming discussions. We'll still want to design those names for consistency, of course.) + + +```swift +protocol RegexComponent { + associatedtype RegexOutput +} +``` + +The associatedtype name is "RegexOutput" to help libraries conform their parsers to this protocol (e.g. via `CustomConsumingRegexComponent`). Regex's capture representation is regexy: it has the overall matched portion as the first capture and the regex builders know how to combine these kinds of capture lists together. This could be different than how e.g. a parser combinator library's output types might be represented. Thus, we chose a more specific name to avoid any potential conflicts. + +The name "RegexComponent" accentuates that any conformer can be used as part of a larger regex, while it de-emphasizes that `Regex` instances themselves can be used directly. We propose methods that are generic over `RegexComponent` and developers will be considering whether they should make their functions that otherwise take a `Regex` also be generic over `RegexComponent`. + +It's possible there might be some initial confusion around the word "component", i.e. a developer may have a regex and not be sure how to make it into a component or how to get the component out. The word "component" carries a lot of value in the context of the regex DSL. An alternative name might be `RegexProtocol`, which implies that a Regex can be used at the site and would be clearly the way to make a function taking a concrete `Regex` generic. But, it's otherwise a naming workaround that doesn't carry the additional regex builder connotations. + +The protocol requirement is `var regex: Regex`, i.e. any type that can produce a regex or hook into the engine's customization hooks (this is what `consuming` does) can be used as a component of the DSL and with these generic API. An alternative name could be "CustomRegexConvertible", but we don't feel that communicates component composability very well, nor is it particularly enlightening when encountering these generic API. + +Another alternative is to have a second protocol just for generic API. But without a compelling semantic distinction or practical utility, we'd prefer to avoid adding protocols just for names. If a clearly superior name exists, we should just choose that. + + +```swift +protocol CustomConsumingRegexComponent { + func consuming(...) +} +``` + +This is not a normal developer-facing protocol or concept; it's an advanced library-extensibility feature. Explicit, descriptive, and careful names are more important than concise names. The "custom" implies that we're not just vending a regex directly ourselves, we're instead customizing behavior by hooking into the run-time engine directly. + +Older versions of the pitch had `func match(...) -> (String.Index, T)?` as the protocol requirement. As [Regex type and overview](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md) went through review, naming convention settled on using the word "match" as a noun and in context with operations that produce a `Match` instance. Since this is the engine's customization hook, it produces the value and position to resume execution from directly, and hence different terminology is apt and avoids confusion or future ambiguities. "Consuming" is the nomenclature we're going with for something that chews off the front of its input in order to produces a value. + +This protocol customizes the basic consume-from-the-front functionality. A protocol for customizing search is future work and involves accommodating different kinds of state and ways that a searcher may wish to speed up subsequent searches. Alternative names for the protocol include `CustomRegexComponent`, `CustomConsumingRegex`, etc., but we don't feel brevity is the key consideration here. + + +### Why `where SubSequence == Substring`? + +A `Substring` slice requirement allows the regex engine to produce indices in the original collection by operating over a portion of the input. Unfortunately, this is not one of the requirements of `StringProtocol`. + +A new protocol for types that can produce a `Substring` on request (e.g. from UTF-8 contents) would have to eagerly produce a `String` copy first and would need requirements to translate indices. When higher-level algorithms are implemented via multiple calls to the lower-level algorithms, these copies could happen many times. Shared strings are future work but a much better solution to this. + +## Future directions + +### Backward algorithms + +It would be useful to have algorithms that operate from the back of a collection, including ability to find the last non-overlapping range of a pattern in a string, and/or that to find the first range of a pattern when searching from the back, and trimming a string from both sides. They are deferred from this proposal as the API that could clarify the nuances of backward algorithms are still being explored. + +
+ Nuances of backward algorithms + +There is a subtle difference between finding the last non-overlapping range of a pattern in a string, and finding the first range of this pattern when searching from the back. + +The currently proposed algorithm that finds a pattern from the front, e.g. `"aaaaa".ranges(of: "aa")`, produces two non-overlapping ranges, splitting the string in the chunks `aa|aa|a`. It would not be completely unreasonable to expect to introduce a counterpart, such as `"aaaaa".lastRange(of: "aa")`, to return the range that contains the third and fourth characters of the string. This would be a shorthand for `"aaaaa".ranges(of: "aa").last`. Yet, it would also be reasonable to expect the function to return the first range of `"aa"` when searching from the back of the string, i.e. the range that contains the fourth and fifth characters. + +Trimming a string from both sides shares a similar story. For example, `"ababa".trimming("aba")` can return either `"ba"` or `"ab"`, depending on whether the prefix or the suffix was trimmed first. +
+ +### Split preserving the separator + +Future work is a split variant that interweaves the separator with the separated portions. For example, when splitting over `\p{punctuation}` it might be useful to be able to preserve the punctionation as a separate entry in the returned collection. + +### Future API + +Some common string processing functions are not currently included in this proposal, such as trimming the suffix from a string/collection, and finding overlapping ranges of matched substrings. This pitch aims to establish a pattern for using `RegexComponent` with string processing algorithms, so that further enhancement can to be introduced to the standard library easily in the future, and eventually close the gap between Swift and other popular scripting languages. diff --git a/proposals/0358-primary-associated-types-in-stdlib.md b/proposals/0358-primary-associated-types-in-stdlib.md new file mode 100644 index 0000000000..a95f941c09 --- /dev/null +++ b/proposals/0358-primary-associated-types-in-stdlib.md @@ -0,0 +1,218 @@ +# Primary Associated Types in the Standard Library + +* Proposal: [SE-0358](0358-primary-associated-types-in-stdlib.md) +* Authors: [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#41843](https://github.com/apple/swift/pull/41843) +* Review: ([pitch](https://forums.swift.org/t/pitch-primary-associated-types-in-the-standard-library/56426/)) ([review](https://forums.swift.org/t/se-0358-primary-associated-types-in-the-standard-library/57432)) ([partial acceptance](https://forums.swift.org/t/se-0358-primary-associated-types-in-the-standard-library/57432/14)) ([revision and extension](https://forums.swift.org/t/se-0358-primary-associated-types-in-the-standard-library/57432/32)) ([acceptance](https://forums.swift.org/t/accepted-se-0358-primary-associated-types-in-the-standard-library/58547)) +* Related Proposals: + - [SE-0023] API Design Guidelines + - [SE-0346] Lightweight same-type requirements for primary associated types + +[SE-0023]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0023-api-guidelines.md +[SE-0346]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md + +## Introduction + +[SE-0346] introduced the concept of primary associated types to the language. This document proposes to adopt this feature in the Swift Standard Library, adding primary associated types to select existing protocols. Additionally, we provide some general API design recommendations that protocol authors may find helpful when adding support for this language feature. + +## Motivation + +In order for the lightweight constraint syntax introduced in [SE-0346] to be actually usable, protocol definitions inside and outside the Standard Library need to be extended with primary associated type declarations. + +See [SE-0346] for several motivating examples for these changes. + +## API Design Guidelines + +Primary associated types add a new facet to the design of protocols. For every public protocol with associated type requirements, we need to carefully consider which of them (if any) we want to mark as primary. On the one hand, we want to allow people to use the shorthand syntax whenever possible; on the other hand, we only get one chance to decide this: once a protocol gains a primary associated type annotation, most subsequent changes would be source-breaking. + +We've found the following guidelines helpful when considering the adoption of primary associated types within the Standard Library. We haven't had enough real-life experience with this new feature to propose these guidelines for general use -- however, the recommendations below can still serve as a useful starting point. + +(Aside: If you decide to follow these guidelines when annotating your own protocols, and they lead you to a choice that you later regret, please post a note on the Swift forums! Negative examples are going to be extremely helpful while revising the guidelines for general use. We're also looking for (positive or negative) examples for multiple primary associated types on a single protocol.) + +1. **Let usage inform your design.** + + If you are considering adding a primary associated type declaration to a preexisting protocol, then look at its existing clients to discover which associated types get typically constrained. Is there one particular type that is used overwhelmingly more than any other? If so, then it will probably be a good choice for the primary. + + For example, in the case of `Sequence`, use sites overwhelmingly tend to constrain `Element` -- `Iterator` is almost never mentioned in `where` clauses. This makes it fairly clear that `Element` is the right choice for the primary type. + + If you're designing a new protocol, think about which type people will most likely want to constrain. Sometimes it may not even be one you planned to have as an associated type! + + For example, protocol `Clock` in [SE-0329](0329-clock-instant-duration.md) initially only had `Instant` as an associated type. As it turns out, in actual use cases, people are far more likely to want to constrain `Instant.Duration` rather than `Instant` itself. Clocks tend to be far too closely coupled to their instants for it to serve as a useful constraint target -- `some Clock` is effectively just a circuitous way of spelling `ContinuousClock`. On the other hand, `some Clock` captures all clocks that measure elapsed time in physical seconds -- a far more useful abstraction. Therefore, we decided to add `Clock.Duration` for the express purpose to serve as the primary associated type. + +2. **Consider clarity at the point of use.** To prevent persistent confusion, _people familiar with the protocol_ ought to be able to correctly intuit the meaning of a same-type constraint such as `some Sequence`. + + Lightweight constraint specifications share the same angle-bracketed syntax as generic type arguments, including the same limitations. In particular, the language does not support argument labels in such lists, which prevents us from clarifying the role of the type names provided. A type name such as `Foo` on its own provides no hints about the role of its generic arguments `Int` and `String`; likewise, it isn't possible to decipher the role of `Character` in a same-type requirement such as `some Bar`, unless the reader is already somewhat familiar with the protocol `Bar`. + + The best candidates for primary associated types tend to be those that have a simple, obvious relationship to the protocol itself. A good heuristic is that if the relationship can be described using a simple preposition, then the associated type will probably make a viable primary: + + - `Collection` *of* `Int` + - `Identifiable` *by* `String` + - `SIMD` *of* `Float` + - `RawRepresentable` *by* `Int32` + + Associated types that don't support this tend to have a more complex / idiosyncratic role in their protocol, and often make poor choices for a primary associated type. + + For example, `Numeric` has an associated type called `Magnitude` that does sometimes appear in associated type constraints. However, its role seems too subtle and non-obvious to consider marking it as primary. The meaning of `Int` in `some Numeric` is unlikely to be clear to readers, even if they are deeply familiar with Swift's numeric protocol hierarchy. + +3. **Not every protocol needs primary associated types.** Don't feel obligated to add a primary associated type just because it is possible to do so. If you don't expect people will want to constrain an associated type in practice, there is little reason to mark it as a primary. Similarly, if there are multiple possible choices that seem equally useful, it might be best not to select one. (See point 2 above.) + + For example, `ExpressibleByIntegerLiteral` is not expected to be mentioned in generic function declarations, so there is no reason to mark its sole associated type (`IntegerLiteral`) as the primary. + +4. **Limit yourself to just one primary associated type.** In most cases, it's best not to declare more than one primary associated type on any protocol. + + While the language does allow this, [SE-0346] requires clients using the lightweight syntax to always explicitly constrain all primary associated types, which may become an obstacle. Clients don't have an easy way to indicate that they want to leave one of the types unconstrained -- to do that, they need to revert to classic generic syntax, partially or entirely giving up on the lightweight variant: + + ```swift + protocol MyDictionaryProtocol { + associatedtype Key: Equatable + associatedtype Value + ... + } + + // This function is happy to work on any dictionary-like thing + // as long as it has string keys. + func twiddle(_ items: some MyDictionaryProtocol) -> Int { ... } + + // Possible approaches: + func twiddle(_ items: some MyDictionaryProtocol) -> Int { ... } + func twiddle(_ items: T) -> Int where T.Key == String { ... } + ``` + + Of course, if the majority of clients actually do want to constrain both `Key` and `Value`, then having them both marked primary can be an appropriate choice. + + +## Proposed solution + +The table below lists all public protocols in the Standard Library with associated type requirements, along with their proposed primary associated type, as well as a list of other associated types. + +[note]: #alternatives-considered + +| Protocol | Primary | Others | +|------------------------------------------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `Sequence` | `Element` | `Iterator` | +| `IteratorProtocol` | `Element` | -- | +| `Collection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `MutableCollection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `BidirectionalCollection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `RandomAccessCollection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `RangeReplaceableCollection` | `Element` | `Index`, `Iterator`, `SubSequence`, `Indices` | +| `LazySequenceProtocol` | -- [(1)][note] | `Element`, `Iterator`, `Elements` | +| `LazyCollectionProtocol` | -- [(1)][note] | `Element`, `Index`, `Iterator`, `SubSequence`, `Indices`, `Elements` | +| `Identifiable` | `ID` | -- | +| `RawRepresentable` | `RawValue` | -- | +| `RangeExpression` | `Bound` | -- | +| `Strideable` | `Stride` | -- | +| `SetAlgebra` | `Element` | `ArrayLiteralElement` | +| `OptionSet` | -- [(2)][note] | `Element`, `ArrayLiteralElement`, `RawValue` | +| `Numeric` | -- | `IntegerLiteralType`, `Magnitude` | +| `SignedNumeric` | -- | `IntegerLiteralType`, `Magnitude` | +| `BinaryInteger` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Words` | +| `UnsignedInteger` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Words` | +| `SignedInteger` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Words` | +| `FixedWidthInteger` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Words` | +| `FloatingPoint` | -- | `IntegerLiteralType`, `Magnitude`, `Stride`, `Exponent` | +| `BinaryFloatingPoint` | -- | `IntegerLiteralType`, `FloatLiteralType`, `Magnitude`, `Stride`, `Exponent`, `RawSignificand`, `RawExponent` | +| `SIMD` | `Scalar` | `ArrayLiteralElement`, `MaskStorage` | +| `SIMDStorage` | -- | `Scalar` | +| `SIMDScalar` | -- | `SIMDMaskScalar`, `SIMD2Storage`, `SIMD4Storage`, ..., `SIMD64Storage` | +| `KeyedEncodingContainerProtocol` | -- | `Key` | +| `KeyedDecodingContainerProtocol` | -- | `Key` | +| `ExpressibleByIntegerLiteral` | -- | `IntegerLiteralType` | +| `ExpressibleByFloatLiteral` | -- | `FloatLiteralType` | +| `ExpressibleByBooleanLiteral` | -- | `BooleanLiteralType` | +| `ExpressibleByUnicodeScalarLiteral` | -- | `UnicodeScalarLiteralType` | +| `ExpressibleByExtended-`
`GraphemeClusterLiteral` | -- | `UnicodeScalarLiteralType`, `ExtendedGraphemeClusterLiteralType` | +| `ExpressibleByStringLiteral` | -- | `UnicodeScalarLiteralType`, `ExtendedGraphemeClusterLiteralType`, `StringLiteralType` | +| `ExpressibleByStringInterpolation` | -- | `UnicodeScalarLiteralType`, `ExtendedGraphemeClusterLiteralType`, `StringLiteralType`, `StringInterPolation` | +| `ExpressibleByArrayLiteral` | -- | `ArrayLiteralElement` | +| `ExpressibleByDictionaryLiteral` | -- | `Key`, `Value` | +| `StringInterpolationProtocol` | -- | `StringLiteralType` | +| `Unicode.Encoding` | -- | `CodeUnit`, `EncodedScalar`, `ForwardParser`, `ReverseParser` | +| `UnicodeCodec` | -- | `CodeUnit`, `EncodedScalar`, `ForwardParser`, `ReverseParser` | +| `Unicode.Parser` | -- | `Encoding` | +| `StringProtocol` | -- | `Element`, `Index`, `Iterator`, `SubSequence`, `Indices`, `UnicodeScalarLiteralType`, `ExtendedGraphemeClusterLiteralType`, `StringLiteralType`, `StringInterPolation`, `UTF8View`, `UTF16View`, `UnicodeScalarView` | +| `CaseIterable` | -- | `AllCases` | +| `Clock` | `Duration` | `Instant` | +| `InstantProtocol` | `Duration` | -- | +| `AsyncIteratorProtocol` | -- [(3)][note] | `Element` | +| `AsyncSequence` | -- [(3)][note] | `AsyncIterator`, `Element` | +| `GlobalActor` | -- | `ActorType` | +| `DistributedActor` | -- [(4)][note] | `ID`, `ActorSystem`, `SerializationRequirement` | +| `DistributedActorSystem` | -- [(4)][note] | `ActorID`, `SerializationRequirement`, `InvocationEncoder`, `InvocationDecoder`, `ResultHandler` | +| `DistributedTargetInvocationEncoder` | -- [(4)][note] | `SerializationRequirement` | +| `DistributedTargetInvocationDecoder` | -- [(4)][note] | `SerializationRequirement` | +| `DistributedTargetInvocationResultHandler` | -- [(4)][note] | `SerializationRequirement` | + +As of Swift 5.6, the following public protocols don't have associated type requirements, so they are outside of the scope of this proposal. + +```swift +Equatable, Hashable, Comparable, Error, AdditiveArithmetic, +DurationProtocol, Encodable, Decodable, Encoder, Decoder, +UnkeyedEncodingContainer, UnkeyedDecodingContainer, +SingleValueEncodingContainer, SingleValueDecodingContainer, +ExpressibleByNilLiteral, CodingKeyRepresentable, +CustomStringConvertible, LosslessStringConvertible, TextOutputStream, +TextOutputStreamable, CustomPlaygroundDisplayConvertible, +CustomReflectable, CustomLeafReflectable, MirrorPath, +RandomNumberGenerator, CVarArg, Sendable, UnsafeSendable, Actor, +AnyActor, Executor, SerialExecutor, DistributedActorSystemError +``` + +## Detailed design + +```swift +public protocol Sequence +public protocol IteratorProtocol +public protocol Collection: Sequence +public protocol MutableCollection: Collection +public protocol BidirectionalCollection: Collection +public protocol RandomAccessCollection: BidirectionalCollection +public protocol RangeReplaceableCollection: Collection + +public protocol Identifiable +public protocol RawRepresentable +public protocol RangeExpression +public protocol Strideable: Comparable + +public prococol SetAlgebra: Equatable, ExpressibleByArrayLiteral + +public protocol SIMD: ... + +public protocol Clock: Sendable +public protocol InstantProtocol: Comparable, Hashable, Sendable +``` + +## Source compatibility + +None. The new annotations enable new ways to use these protocols, but they are tied to new syntax, and they do not affect existing code. + +## Effect on ABI stability + +None. The annotations aren't ABI impacting, and the new capabilities deploy back to any previous Swift Standard Library release. + +## Effect on API resilience + +Once introduced, primary associated types cannot be removed from a protocol or reordered without breaking source compatibility. + +[SE-0346] requires usage sites to always list every primary associated type defined by a protocol. Until/unless this restriction is lifted, adding a new primary associated type to a protocol that already has some will also be a source breaking change. + +Therefore, we will not be able to make any changes to the list of primary associated types of any of the protocols that are affected by this proposal once this ships in a Standard Library release. + +## Alternatives considered + +(1) It is tempting to declare `Element` as the primary associated type for `LazySequenceProtocol` and `LazyCollectionProtocol`, for consistency with other protocols in the collection hierarchy. However, in actual use, `Elements` seems just as useful (if not more) to be easily constrained. We left the matter of selecting one of these as primary unresolved for now; as we get more experience with the lightweight constraint syntax, we may revisit these protocols. + +(2) In the `OptionSet` protocol, the `Element` type is designed to always be `Self`, so `RawValue` would be the most practical choice for the primary associated type. However, to avoid potential confusion, we left `OptionSet` without a primary associated type annotation. + +(3) `AsyncSequence` and `AsyncIteratorProtocol` logically ought to have `Element` as their primary associated type. However, we have [ongoing evolution discussions][rethrows] about adding a precise error type to these. If those discussions bear fruit, then it's possible we may want to _also_ mark the potential new `Error` associated type as primary. To prevent source compatibility complications, adding primary associated types to these two protocols is deferred to a future proposal. + +[rethrows]: https://forums.swift.org/t/se-0346-lightweight-same-type-requirements-for-primary-associated-types/55869/70 + +(4) Declaring primary associated types on the distributed actor protocols would be desirable, but it was [deferred to a future proposal](https://forums.swift.org/t/pitch-primary-associated-types-in-the-standard-library/56426/47), to prevent interfering with potential future language improvements that would make them more useful in this use case. + +## Revisions + +- [2022-05-28](https://github.com/swiftlang/swift-evolution/blob/716db41ccefde348ac38bd2fd1eb5bd7842be7b6/proposals/0358-primary-associated-types-in-stdlib.md): Initial proposal version. +- 2022-06-22: Removed the primary associated type declaration from the `OptionSet` protocol. The API guidelines section has revised wording; it no longer proposes the new guidelines for inclusion in the official Swift API Guidelines document. Adjusted wording to prefer the term "lightweight constraint syntax" to "lightweight same-type requirements", as the new syntax can be used for more than just to express same-type constraints. diff --git a/proposals/0359-build-time-constant-values.md b/proposals/0359-build-time-constant-values.md new file mode 100644 index 0000000000..5778ef2da4 --- /dev/null +++ b/proposals/0359-build-time-constant-values.md @@ -0,0 +1,328 @@ +# Build-Time Constant Values + +* Proposal: [SE-0359](0359-build-time-constant-values.md) +* Authors: [Artem Chikin](https://github.com/artemcm), [Ben Cohen](https://github.com/airspeedswift), [Xi Ge](https://github.com/nkcsgexi) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Returned for revision** +* Decision notes: [First review rationale](https://forums.swift.org/t/returned-for-revision-se-0359-build-time-constant-values/58976) +* Implementation: Implemented on `main` as `_const` + +## Introduction + +A Swift language feature for requiring certain values to be knowable at compile-time. This is achieved through an attribute, `@const`, constraining properties and function parameters to have compile-time knowable values. Such information forms a foundation for richer compile-time features in the future, such as extraction and validation of values at compile time. + +Related forum threads: + +* [[Pitch] Compile-Time Constant Values](https://forums.swift.org/t/pitch-compile-time-constant-values/53606) +* [[Pitch #2] Build-Time Constant Values](https://forums.swift.org/t/pitch-2-build-time-constant-values/56762) + +## Motivation + +Compile-time constant values are values that can be known or computed during compilation and are guaranteed to not change after compilation. Use of such values can have many purposes, from enforcing desirable invariants and safety guarantees to enabling users to express arbitrarily-complex compile-time algorithms. + +The first step towards building out support for compile-time constructs in Swift is a basic primitives consisting of an attribute to declare function parameters and properties to require being known at *compile-time*. While this requirement explicitly calls out the compiler as having additional guaranteed information about such declarations, it can be naturally extended into a broader sense of *build-time* known values - with the compiler being able to inform other tools with type-contextual value information. For an example of the latter, see the “Declarative Package Manifest” motivating use-case below. + +## Proposed Solution + +The Swift compiler will recognize declarations of properties, local variables, and function parameters declared with a `@const` attribute as having an additional requirement to be known at compile-time. If a `@const` property or variable is initialized with a runtime value, the compiler will emit an error. Similarly, if a runtime value is passed as an argument to a `@const` function parameter, the compiler will emit an error. Aside from participating in name mangling, the attribute has no runtime effect. + +For example, a `@const` property can provide the compiler and relevant tooling build-time knowledge of a type-specific value: + +```swift +struct DatabaseParams { + @const let encoding: String = "utf-8" + @const let capacity: Int = 256 +} +``` + +A `@const` parameter provides a guarantee of a build-time-known argument being specified at all function call-sites, allowing future APIs to enforce invariants and provide build-time correctness guarantees: + +```swift +func acceptingURLString(@const _ url: String) +``` + +And a `@const static let` protocol property requirement allows protocol authors to enforce get the benefits of build-time known properties for all conforming types: + +```swift +protocol DatabaseSerializableWithKey { + @const static let key: String +} +``` + +## Detailed Design + +### Property `@const` attribute + +A stored property on a `struct` or a `class` can be marked with a `@const` attribute to indicate that its value is known at compile-time. +```swift +struct Foo { + @const let title: String = "foo" +} +``` +The value of such a property must be default-initialized with a compile-time-known value, unlike a plain `let` property, which can also be assigned a value in the type's initializer. + +```swift +struct Foo { + // 👍 + @const let superTitle: String = "Encyclopedia" + // ❌ error: `title` must be initialized with a const value + @const let title: String + // ❌ error: `subTitle` must be initialized with a const value + @const let subTitle: String = bar() +} +``` + +Similarly to Implicitly-Unwrapped Optionals, the mental model for semantics of this attribute is that it is a flag on the declaration that guarantees that the compiler is able to know its value as shared by all instance of the type. For now, `@const let` and `@const static let` are equivalent in what information the `@const` attribute conveys to the compiler. + +### Parameter `@const` attribute + +A function parameter can be marked with a `@const` keyword to indicate that values passed to this parameter at the call-site must be compile-time-known values. + +```swift +func foo(@const input: Int) {...} +``` + +Passing in a runtime value as an argument to `foo` will result in a compilation error: +```swift +foo(11) // 👍 + +let x: Int = computeRuntimeCount() +foo(x) // ❌ error: 'greeting' must be initialized with a const value +``` + +### Protocol `@const` property requirement + +A protocol author may require conforming types to default initialize a given property with a compile-time-known value by specifying it as `@const static let` in the protocol definition. For example: + +```swift +protocol NeedsConstGreeting { + @const static let greeting: String +} +``` +Unlike other property declarations on protocols that require the use of `var` and explicit specification of whether the property must provide a getter `{ get }` or also a setter `{ get set }`, using `var` for build-time-known properties whose values are known to be fixed at runtime is counter-intuitive. Moreover, `@const` implies the lack of a run-time setter and an implicit presence of the value getter. + +If a conforming type initializes `greeting` with something other than a compile-time-known value, a compilation error is produced: + +```swift +struct Foo: NeedsConstGreeting { + // 👍 + static let greeting = "Hello, Foo" +} +struct Bar: NeedsConstGreeting { + // error: ❌ 'greeting' must be initialized with a const value + static let greeting = "\(Bool.random ? "Hello" : "Goodbye"), Bar" +} +``` + +### Supported Types + +The requirement that values of `@const` properties and parameters be known at compile-time restricts the allowable types for such declarations. The current scope of the proposal includes: + +* Enum cases with no associated values +* Certain standard library types that are expressible with literal values +* Integer and Floating-Point types (`(U)?Int(\d*)`, `Float`, `Double`, `Half`), `String` (excluding interpolated strings), `Bool`. +* `Array` and `Dictionary` literals consisting of literal values of above types. +* Tuple literals consisting of the above list items. + +This list will expand in the future to include more literal-value kinds or potential new compile-time valued constructs. + +## Motivating Example Use-Cases + +### Enforcement of Compile-Time Attribute Parameters + +Attribute definitions can benefit from additional guarantees of compile-time constant values. +For example, a `@const` version of the [@Clamping](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md#clamping-a-value-within-bounds) property wrapper that requires that lower and upper bounds be compile-time-known values can ensure that the clamp values are fixed and cannot change for different instantiations of wrapped properties which may occur if runtime values are used to specify the bounds: + +```swift +@propertyWrapper +struct Clamping { + var value: V + let min: V + let max: V + + init(wrappedValue: V, @const min: V, @const max: V) { + value = wrappedValue + self.min = min + self.max = max + assert(value >= min && value <= max) + } + ... +``` +It could also allow the compiler to generate more-efficient comparison and clamping code, in the future. + +Or imagine a property wrapper that declares a property is to be serialized and that it must be stored/retrieved using a specific string key. `Codable` requires users to provide a `CodingKeys` `enum` boilerplate, relying on the `enum`’s `String` raw values. Alternatively, such key can be specified on the property wrapper itself: + +```swift +struct Foo { + @SpecialSerializationSauce(key: "title") + var someSpecialTitleProperty: String +} + +@propertyWrapper +struct SpecialSerializationSauce { + init(@const key: String) {...} +} +``` + +Having the compiler enforce the compile-time constant property of the `key` parameter eliminates the possibility of an error where a run-time value is specified which can cause serialized data to not be able to be deserialized, for example. + +Enforcing compile-time constant nature of the parameters is also the first step to allowing attribute/library authors to be able to check uses by performing compile-time sanity checking and having the capability to emit custom build-time error messages. + +### Enforcement of Non-Failable Initializers + +Ergonomics of the recently-pitched [Foundation.URL](https://forums.swift.org/t/foundation-url-improvements/54057) would benefit greatly from the ability to require the string argument to be compile-time constant. With evolving compile-time evaluation facilities, Swift may even gain an ability to perform compile-time validation of such URLs even though the user may never be able to express a fully compile-time constant `Foundation.URL` type because this type is a part of an ABI-stable SDK. While a type like `StaticString` may be used to require that the argument string must be static, which string is chosen can still be determined at runtime, e.g.: + +```swift +URL(Bool.random() ? "https://valid.url.com" : "invalid url . com") +``` + +### Facilitate Compile-time Extraction of Values + +The [Result Builder-based SwiftPM Manifest](https://forums.swift.org/t/pre-pitch-swiftpm-manifest-based-on-result-builders/53457) pre-pitch outlines a proposal for a manifest format that encodes package model/structure using Swift’s type system via Result Builders. Extending the idea to use the builder pattern throughout can result in a declarative specification that exposes the entire package structure to build-time tools, for example: + +```swift +let package = Package { + Modules { + Executable("MyExecutable", public: true, include: { + Internal("MyDataModel") + }) + Library("MyLibrary", public: true, include: { + Internal("MyDataModel", public: true) + }) + Library("MyDataModel") + Library("MyTestUtilities") + Test("MyExecutableTests", for: "MyExecutable", include: { + Internal("MyTestUtilities") + External("SomeModule", from: "some-package") + }) + Test("MyLibraryTests", for: "MyLibrary") + } + Dependencies { + SourceControl(at: "https://git-service.com/foo/some-package", upToNextMajor: "1.0.0") + } +} +``` + +A key property of this specification is that all the information required to know how to build this package is encoded using compile-time-known concepts: types and literal (and therefore compile-time-known) values. This means that for a category of simple packages where such expression of the package’s model is possible, the manifest does not need to be executed in a sandbox by the Package Manager - the required information can be extracted at manifest *build* time. + +To *ensure* build-time extractability of the relevant manifest structure, a form of the above API can be provided that guarantees the compile-time known properties. For example, the following snippet can guarantee the ability to extract complete required knowledge at build time: + +```swift +Test("MyExecutableTests", for: "MyExecutable", include: { + Internal("MyTestUtilities") + External("SomeModule", from: "some-package") + }) +``` +By providing a specialized version of the relevant types (`Test`, `Internal`, `External`) that rely on parameters relevant to extracting the package structure being `const`: + +```swift +struct Test { + init(@const _ title: String, @const for: String, @DependencyBuilder include: ...) {...} +} +struct Internal { + init(@const _ title: String) +} +struct External { + init(@const _ title: String, @const from: String) +} +``` +This could, in theory, allow SwiftPM to build such packages without executing their manifest. Some packages, of course, could still require run-time (execution at package build-time) Swift constructs. More-generally, providing the possibility of declarative APIs that can express build-time-knowable abstractions can both eliminate (in some cases) the need for code execution - reducing the security surface area - and allow for further novel use-cases of Swift’s DSL capabilities (e.g. build-time extractable database schema, etc.). + +### Guaranteed Optimization Hints + +Similarly, ergonomics of numeric intrinsics can benefit from allowing only certain function parameters to be required to be compile-time known. For example, requiring a given numeric operation to specify a `@const` parameter for the rounding mode of an operation as an enum case, while allowing the operands of the operation be runtime values, allowing the compiler to generate more-efficient code. + +## Source compatibility + +This is a purely additive change and has no source compatibility impacts. + +## Effect on ABI stability and API resilience + +The new function parameter attribute is a part of name mangling. The *value* of `public @const` properties is a part of a module's ABI. See discussion on [*Memory placement*](#memory-placement-and-runtime-initialization) for details. + +## Effect on SwiftPM packages + +There is no impact on SwiftPM packages. + +## Alternatives Considered + +### Using a keyword or an introducer instead of an attribute +`@const` being an attribute, as opposed to a keyword or a new introducer (such as `const` instead of `let`), is an approach that is more amenable to applying to a greater variety of constructs in the futures, in addition to property and parameter declarations, such as `@const func`. In addition, as described in comparison to Implicitly-Unwrapped Optionals above, this attribute does not fundamentally change the behavior of the declaration, rather it restricts its handling by the compiler, similar to `@objc`. + +### Difference to `StaticString`-like types +As described in the **Enforcement of Non-Failable Initializers**, the key difference to types like `StaticString` that require a literal value is the `@const` attribute's requirement that the exact value be known at compile-time. `StaticString` allows for a runtime selection of multiple compile-time known values. + +### Placing `@const` on the declaration type +One alternative to declaring compile-time known values as proposed here with the declaration attribute: + +```swift +@const let x = 11 +``` +Is to instead shift the annotation to declared property's type: + +```swift +let x: @const Int = 11 +``` +This shifts the information conveyed to the compiler about this declaration to be carried by the declaration's type. Semantically, this departs from, and widely broadens the scope from what we intend to capture: the knowability of the declared *value*. Encoding the compile-time property into the type system would force us to reckon with a great deal of complexity and unintended consequences. Consider the following example: + +```swift +typealias CI = @const Int +let x: CI? +``` +What is the type of `x`? It appears to be Optional<@const Int>, which is not a meaningful or useful type, and the programmer most likely intended to have a @const Optional. And although today Implicitly-Unwrapped optional syntax conveys an additional bit of information about the declared value using a syntactic indicator on the declared type, without affecting the declaration's type, the [historical context](https://www.swift.org/blog/iuo/) of that feature makes it a poor example to justify requiring consistency with it. + +### Alternative attribute names +More-explicit spellings of the attribute's intent were proposed in the form of `@buildTime`/`@compileTime`/`@comptime`, and the use of `const` was also examined as a holdover from its use in C++. + +While build-time knowability of values this attribute covers is one of the intended semantic takeaways, the potential use of this attribute for various optimization purposes also lends itself to indicate the additional immutability guarantees on top of a plain `let` (which can be initialized with a dynamic value), as well as capturing the build-time evaluation/knowledge signal. For example, in the case of global variables, thread-safe lazy initialization of `@const` variables may no longer be necessary, in which case the meaning of the term `const` becomes even more explicit. + +Similarly with the comparison to C++, where the language uses the term `const` to describe a runtime behaviour concept, rather than convey information about the actual value. The use of the term `const` is more in line with the mathematical meaning of having the value be a **defined** constant. + +## Forward-Looking Design Aspects and Future Directions + +### Future Inference/Propagation Rules +Though this proposal does **not** itself introduce rules of `@const` inference, their future existence is worth discussing in this design. Consider the example: + +```swift +@const let i = 1 +let j = i +``` +While not valid under this proposal, our intent is to allow the use of `i` where `@const` values are expected in the future, for example `@const let k = i` or `f(i)` where `f` is `func f(@const _: Int)`. It is therefore important to consider whether `@const` is propagated to values like `j` in the above example, which determines whether or not statements like `f(j)` and `@const let k = j` are valid code. While it is desirable to allow such uses of the value within the same compilation unit, if `j` is `public`, automatically inferring it to be `@const` is problematic at the module boundary: it creates a contract with the module's clients that the programmer may not have intended. Therefore, `public` properties must explicitly be marked `@const` in order to be accessible as such outside the defining module. This is similar in nature to `Sendable` inference - `internal` or `private` entities can automatically be inferred by the compiler as `Sendable`, while `public` types must explicitly opt-in. + +### Memory placement and runtime initialization +Effect on runtime placement of `@const` values is an implementation detail that this proposal does not cover beyond indicating that today this attribute has no effect on memory layout of such values at runtime. It is however a highly desirable future direction for the implementation of this feature to allow the use of read-only memory for `@const` values. With this in mind, it is important to allow semantics of this attribute to allow such implementation in the future. For example, a global `@const let`, by being placed into read-only memory removes the need for synchronization on access to such data. Moreover, using read-only memory reduces memory pressure that comes from having to maintain all mutable state in-memory at a given program point - read-only data can be evicted on-demand to be read back later. These are desirable traits for optimization of existing programs which become increasingly important for enabling of low-level system programs to be written in Swift. + +In order to allow such implementation in the future, this proposal makes the *value* of `public` `@const` values/properties a part of a module's ABI. That is, a resilient library that vends `@const let x = 11` changing the value of `x` is considered an ABI break. This treatment allows `public` `@const` data to exist in a single read-only location shared by all library clients, without each client having to copy the value or being concerned with possible inconsistency in behavior across library versions. + +## Future Directions + +### Constant-propagation +Allow default-initialization of `@const` properties using other `@const` values and allow passing `@const` values to `@const` parameters. The [Future Inference/Propagation Rules](#future-inferencepropagation-rules) section discusses a direction for enabling inference of the attribute on values. This is a necessary next building-block to generalizing the use of compile-time values. + +```swift +func foo(@const i: Int) { + @const let j = i +} +``` + +### Toolchain support for extracting compile-time values at build time. +The current proposal covers an attribute that allows clients to build an API surface that is capable of carrying semantic build-time information that may be very useful to build-time tooling, such as [SwiftPM plugins](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md). The next step towards this goal would include toolchain support for tooling that extracts such information in a client-agnostic fashion so that it can be adopted equally by use-cases like the manifest example in [Facilitate Compile-time Extraction of Values](#facilitate-compile-time-extraction-of-values) and others. + +### Compile-time expressions and functions +Building on propagation and inference of `@const`, some of the most interesting use-cases for compile-time-known values emerge with the ability to perform operations on them that result in other compile-time-known values. For example, the [Compiler Diagnostic Directives](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md) could be expanded to trigger conditionally based on a value of a compile-time-known input expression: + +```swift +func foo(@const input: Int) { + #const_assert(input <= 0, "'foo()' expects a positive input") +} +``` +Which would require that it is possible to evaluate the `input <= 0` expression at compile-time, which would also require that certain functions can be evaluated at compile-time if their arguments are known at compile-time. For example: + + +```swift +func <=(@const lhs: Int, @const rhs: Int) -> Bool +``` +Inference on which functions can or cannot be evaluated at compile time will be defined in a future proposal and can follow similar ideas to those described in [Future Inference/Propagation Rules](#future-inferencepropagation-rules) section. + +### Compile-time types +Finally, the flexibility of build-time values and code that operates on them can be greatly expanded by allowing entire user-defined types to form compile-time-known values via either custom literal syntax or having a `@const` initializer. \ No newline at end of file diff --git a/proposals/0360-opaque-result-types-with-availability.md b/proposals/0360-opaque-result-types-with-availability.md new file mode 100644 index 0000000000..d326954662 --- /dev/null +++ b/proposals/0360-opaque-result-types-with-availability.md @@ -0,0 +1,245 @@ +# Opaque result types with limited availability + +* Proposal: [SE-0360](0360-opaque-result-types-with-availability.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Implementation: [apple/swift#42072](https://github.com/apple/swift/pull/42072), [apple/swift#42104](https://github.com/apple/swift/pull/42104), [apple/swift#42167](https://github.com/apple/swift/pull/42167), [apple/swift#42456](https://github.com/apple/swift/pull/42456) +* Status: **Implemented (Swift 5.7)** +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0360-opaque-result-types-with-limited-availability/58712) + +## Introduction + +Since their introduction in [SE-0244](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md), opaque result types have become a powerful tool of type-level abstraction that allows library authors to hide implementation details of their APIs. + +Under the rules described in SE-0244 - a function returning an opaque result type *must return a value of the same concrete type `T`* from each `return` statement, and `T` must meet all of the constraints stated on the opaque type. + +The same-type `return` requirement is unnecessarily strict when it comes to availability conditions. SE-0244 states that it should be possible to change the underlying type in the future version of the library, but that would only work with pre-existing types. In other words, the same-type condition does not have to apply across executions of the same program, the same way that `Hashable` must produce the same output for the same value during program execution, but may produce a different value in the next execution. `#available` is special because it's a checkable form of that: dynamic availability will not change while the program is running, but may be different the next time the program runs. + +Current model and implementation limit usefulness of opaque result types as an abstraction mechanism, because it prevents frameworks from introducing new types and using them as underlying types in existing APIs. To bridge this usability gap, I propose to relax same-type restriction for `return`s inside of availability conditions. + +Swift-evolution thread: [ +[Pitch] Opaque result types with limited availability](https://forums.swift.org/t/pitch-opaque-result-types-with-limited-availability/57286) + +## Motivation + +To illustrate the problematic interaction between opaque result types and availability conditions, let's consider a framework that already has a `Shape` protocol and a `Square` type that conforms to the `Shape` protocol. + +``` +protocol Shape { + func draw(to: Surface) +} + +struct Square : Shape { + ... +} +``` + +In a new version of the framework, the library authors decided to introduce a new shape - `Rectangle` with limited availability: + +``` +@available(macOS 100, *) +struct Rectangle : Shape { + ... +} +``` + +Since a `Rectangle` is generalization of a `Square` it makes sense to allow transforming a `Square` into a `Rectangle` but that currently requires extension with limited availability: + +``` +@available(macOS 100, *) +extension Square { + func asRectangle() -> some Shape { + return Rectangle(...) + } +} +``` + +The fact that the new method has to be declared in availability context to return `Rectangle` limits its usefulness because all uses of `asRectangle` would have to be encapsulated into `if #available` blocks. + +If `asRectangle` already existed in the original version of the framework, it wouldn’t be possible to use a new type at all without declaring `if #available` block in its body: + +``` +struct Square { + func asRectangle() -> some Shape { + if #available(macOS 100, *) { + return Rectangle(...) + } + + return self + } +} +``` + +But doing so is not allowed because all of the `return` statements in the body of the `asRectangle` function have to return the same concrete type: + +``` + error: function declares an opaque return type 'some Shape', but the return statements in its body do not have matching underlying types + func asRectangle() -> some Shape { + ^ ~~~~~~~~~~ +note: return statement has underlying type 'Rectangle' + return Rectangle() + ^ +note: return statement has underlying type 'Square' + return Square() + ^ +``` + +This is a dead-end for the library author although SE-0244 states that it should be possible to change underlying result type in the future version of the library/framework but that assumes that the type already exists so it could be used in all `return` statements. + +## Proposed solution + +To bridge this usability gap, I propose to relax the same-type restriction for functions with `if #available` conditions: if an `if #available` condition is always executed, it can return a different type than the type returned by the rest of the function. + +The proposed changes allow functions to: + +* use multiple `if #available` conditions to return different types based on their dynamic availability and +* safely fall back to a return type with no availability restrictions if none of the availability conditions are met. + +Because the return type must be decidable without running the code in the function, mixing availability conditions with other conditions (such as `if`, `guard`, or `switch`) removes this special power and requires `return`s in the `if #available` to return the same type as the rest of the function. + +This example satisfies these rules: + +```swift +func test() -> some Shape { + if #available(macOS 100, *) { ✅ + return Rectangle() + } + + return self +} +``` + +## Detailed design + +An *unconditional availability clause* is an `if` or `else if` clause that satisfies the following conditions: + + - The clause is part of an `if` statement at the top level of the containing function. + - There are no `return` statements in the containing function prior to the `if` statement. + - The condition of the clause is an `#available` condition. + - The clause is either the initial `if` clause or an `else if` clause immediately following an unconditional availability clause. + - The clause contains at least one `return` statement. + - All paths through the block controlled by the clause terminate by either returning or throwing. + +All `return` statements outside of unconditional availability clauses must return the same type as each other, and this type must be as available as the containing function. + +All `return` statements within a given unconditional availability clause must return the same type as each other, and this type must be as available as the `#available` condition of the clause. This type need not be the same type returned by any `return` statement outside of the clause. + +There must be at least one `return` statement in the containing function. If there are no `return` statements outside of unconditional availability clauses, then at least one of the return types within unconditional availability clauses must be as available as the containing function. + +Dynamically, the return type of the containing function is: + - the return type of `return` statements in the first unconditional availability clause whose condition is dynamically satisfied, or if none are satisfied then + - the return type of `return` statements outside of all unconditional availability clauses, or if there are no such statements then + - the return type of `return` statements in the first unconditional availability clause that is as available as the containing function. + +Now let's consider a couple of examples to better demonstrate the difference between well-formed and invalid functions under the proposed rules. + +The following example is well-formed because the first `if #available` statement terminates with a `return` and the second one is associated with a valid `if #available` and also terminates with a `return`. + + ```swift + func test() -> some Shape { + if #available(macOS 100, *) { ✅ + return Rectangle() + } else if #available(macOS 99, *) { ✅ + return Square() + } + return self + } + ``` + + But + + ```swift + func test() -> some Shape { + if cond { + ... + } else if #available(macOS 100, *) { ❌ + return Rectangle() + } + return self + } + ``` + +is not accepted by the compiler because `if #available` associated with a dynamic condition. + +The following is incorrect because `if #available` is preceded by a dynamic condition that returns: + +```swift +func test() -> some Shape { + guard let x = else { + return ... + } + + if #available(macOS 100, *) { ❌ + return Rectangle() + } + + return self +} +``` + +Similarly, the following is incorrect because `if #available` appears inside of a loop: + +```swift +func test() -> some Shape { + for ... { + if #available(macOS 100, *) { ❌ + return Rectangle() + } + } + return self +} +``` + +The following `test()` function is well-formed because `if` statement produces the same result in both of its branches and it's statically known that the `if #available` always terminates with a `return` + + ```swift + func test() -> some Shape { + if #available(macOS 100, *) { + if cond { ✅ + return Rectangle(...) + } else { + return Rectangle(...) + } + } + return self + } + ``` + + But: + + ```swift + func test() -> some Shape { + if #available(macOS 100, *) { + if cond { ❌ + return Rectangle() + } else { + return Square() + } + } + return self + } + ``` + +is not going to be accepted by the compiler because return types are different: `Rectangle` vs. `Square`. + +This semantic adjustment fits well into the existing model because it makes sure that there is always a single underlying type per platform and universally. + +## Source compatibility + +Proposed changes do not break source compatibility and allow previously incorrect code to compile. + +## Effect on ABI stability + +No ABI impact since this is an additive change. + +## Effect on API resilience + +All of the resilience rules associated with opaque result types are preserved. + +## Alternatives considered + +* Only alternative is to change the API patterns used in the library, e.g. by exposing the underlying result type and overloading the method. + +## Acknowledgments + +[John McCall](https://forums.swift.org/u/john_mccall) for the help with `Proposed Solution` and `Detail Design` improvements. diff --git a/proposals/0361-bound-generic-extensions.md b/proposals/0361-bound-generic-extensions.md new file mode 100644 index 0000000000..8b91826d08 --- /dev/null +++ b/proposals/0361-bound-generic-extensions.md @@ -0,0 +1,196 @@ +# Extensions on bound generic types + +* Proposal: [SE-0361](0361-bound-generic-extensions.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift#41172](https://github.com/apple/swift/pull/41172), gated behind the frontend flag `-enable-experimental-bound-generic-extensions` +* Review: ([pitch](https://forums.swift.org/t/pitch-extensions-on-bound-generic-types/57535/)) ([review](https://forums.swift.org/t/se-0361-extensions-on-bound-generic-types/58366)) ([acceptance](https://forums.swift.org/t/accepted-se-0361-extensions-on-bound-generic-types/58716)) + +## Contents + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Alternatives considered](#alternatives-considered) + - [Reserving syntax for parameterized extensions](#reserving-syntax-for-parameterized-extensions) + - [Future directions](#future-directions) + - [Parameterized extensions](#parameterized-extensions) + +## Introduction + +Specifying the type arguments to a generic type in Swift is almost always written in angle brackets, such as `Array`. Extensions are a notable exception, and if you attempt to extend `Array`, the compiler reports the following error message: + +```swift +extension Array { ... } // error: Constrained extension must be declared on the unspecialized generic type 'Array' with constraints specified by a 'where' clause +``` + +As the error message suggests, this extension must instead be written using a `where` clause: + +```swift +extension Array where Element == String { ... } +``` + +This proposal removes this limitation on extensions, allowing you to write bound generic extensions the same way you write bound generic types everywhere else in the language. + +Swift evolution discussion thread: [[Pitch] Extensions on bound generic types](https://forums.swift.org/t/pitch-extensions-on-bound-generic-types/57535). + +## Motivation + +Nearly everywhere in the language, you write bound generic types using angle brackets after the generic type name. For example, you can write a typealias to an array of strings using angle brackets, and extend that type using the typealias: + +```swift +typealias StringArray = Array + +extension StringArray { ... } +``` + +With [SE-0346](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md), we can also declare a primary associated type, and bind it in an extension using angle-brackets: + +```swift +protocol Collection { + associatedtype Element +} + +extension Collection { ... } +``` + +Not allowing this syntax directly on generic type extensions is clearly an artificial limitation, and even the error message produced by the compiler suggests that the compiler understood what the programmer was trying to do: + +```swift +extension Array { ... } // error: Constrained extension must be declared on the unspecialized generic type 'Array' with constraints specified by a 'where' clause +``` + +This limitation is confusing, because programmers don’t understand why they can write `Array` everywhere *except* to extend `Array`, as evidenced by the numerous questions about this limitation here on the forums, such as [this thread](https://forums.swift.org/t/why-doesnt-eg-extension-array-int-compile-even-though-using-a-typealias-does/56049). + +## Proposed solution + +I propose to allow extending bound generic types using angle-brackets for binding type arguments, or using sugared types such as `[String]` and `Int?`. + +The following declarations all express an extension over the same type: + +```swift +extension Array where Element == String { ... } + +extension Array { ... } + +extension [String] { ... } +``` + +## Detailed design + +A generic type name in an extension can be followed by a comma-separated type argument list in angle brackets. The type argument list binds the type parameters of the generic type to each of the specified type arguments. This is equivalent to writing same-type requirements in a `where` clause. For example: + +```swift +struct Pair {} + +extension Pair {} +``` + +is equivalent to + +```swift +extension Pair where T == Int, U == String {} +``` + +A type argument list applied to an extended type name must specify all type arguments; constraining only a subset of the type parameters in an extension must still use a `where` clause: + +```swift +struct Pair {} + +extension Pair {} // error: Generic type 'Pair' specialized with too few type parameters (got 1, but expected 2) + +extension Pair where T == Int {} // okay +``` + +The types specified in the type argument list must be concrete types. For example, you cannot extend a generic type with placeholders as type arguments: + +```swift +extension Pair {} // error: Cannot extend a type that contains placeholders +``` + +> **Rationale**: When `_` is used as a type placeholder, it directs the compiler to infer the type at the position of the underscore. Using `_` in a bound generic extension would introduce a subtly different meaning of `_`, which is to leave the type at that position unconstrained, so `Pair` would mean different things in different contexts. + +Name lookup of the type arguments is performed outside the extension context, so the type parameters of the generic type cannot appear in the type argument list: + +```swift +extension Array {} // error: Cannot find type 'Element' in scope +``` + +If a generic type has a sugared spelling, the sugared type can also be used to extend the generic type: + +```swift +extension [String] { ... } // Extends Array + +extension String? { ... } // Extends Optional +``` + +## Source compatibility + +This change has no impact on source compatibility. + +## Effect on ABI stability + +This is a syntactic sugar change with no impact on ABI. + +## Effect on API resilience + +This change has no impact on API resilience. Changing an existing bound generic extension using a where clause to the sugared syntax and vice versa is a resilient change. + +## Alternatives considered + +### Reserving syntax for parameterized extensions + +Using angle brackets after an extended type name as sugar for same-type requirements prevents this syntax from being used to declare a parameterized extension. Alternatively, `extension Array { ... }` could mean an extension that declares two new type parameters `T` and `U`, rather than an (invalid) application of type arguments to `Array`'s type parameters. However, SE-0346 already introduced this syntax as sugar for same-type requirements on associated types: + +```swift +protocol Collection { + associatedtype Element +} + +// Already sugar for `extension Collection where Element == String` +extension Collection { ... } +``` + +Instead of reserving this syntax for parameterized extensions, type parameters could be declared in angle brackets after the `extension` keyword, which will help indicate that the type parameters belong to the extension: + +```swift +// Introduces new type parameters `T` and `U` for the APIs +// in this extension. +extension Array { ... } +``` + +## Future directions + +### Parameterized extensions + +This proposal does not provide parameterized extensions, but a separate proposal could build upon this proposal to allow extending a generic type with more sophisticated constraints on the type parameters: + +```swift +extension Array> { ... } + +extension [Wrapped?] { ... } +``` + +Parameterized extensions could also allow using the shorthand `some` syntax to write generic extensions where a type parameter has a conformance requirement: + +```swift +extension Array { ... } + +extension [some Equatable] { ... } +``` + +Writing the type parameter list after the `extension` keyword applies more naturally to extensions over structural types. With this syntax, an extension over all two-element tuples could be spelled + +```swift +extension (T, U) { ... } +``` + +This syntax also generalizes to variadic type parameters, e.g. to extend all tuple types to provide a protocol conformance: + +```swift +extension (T...): Hashable where T: Hashable { ... } +``` diff --git a/proposals/0362-piecemeal-future-features.md b/proposals/0362-piecemeal-future-features.md new file mode 100644 index 0000000000..d52071ddd2 --- /dev/null +++ b/proposals/0362-piecemeal-future-features.md @@ -0,0 +1,171 @@ +# Piecemeal adoption of upcoming language improvements + +* Proposal: [SE-0362](0362-piecemeal-future-features.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#59055](https://github.com/apple/swift/pull/59055), [apple/swift-package-manager#5632](https://github.com/apple/swift-package-manager/pull/5632) +* Review: ([pitch](https://forums.swift.org/t/piecemeal-adoption-of-swift-6-improvements-in-swift-5-x/57184)) ([review](https://forums.swift.org/t/se-0362-piecemeal-adoption-of-future-language-improvements/58384)) ([acceptance](https://forums.swift.org/t/accepted-se-0362-piecemeal-adoption-of-future-language-improvements/59076)) + +## Introduction + +Swift 6 is accumulating a number of improvements to the language that have enough source-compatibility impact that they could not be enabled by default in prior language modes (Swift 4.x and Swift 5.x). These improvements are already implemented in the Swift compiler behind the Swift 6 language mode, but they are inaccessible to users, and will remain so until Swift 6 becomes available as a language mode. There are several reasons why we should consider making these improvements available sooner: + +* Developers would like to get the benefits from these improvements soon, rather than wait until Swift 6 is available. +* Making these changes available to developers prior to Swift 6 provides more experience, allowing us to tune them further for Swift 6 if necessary. +* The sum of all changes made in Swift 6 might make migration onerous for some modules, and adopting these language changes one-by-one while in Swift 4.x/5.x can smooth that transition path. + +A few proposals have already introduced bespoke solutions to provide a migration path: [SE-0337](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md) adds `-warn-concurrency` to enable warnings for `Sendable`-related checks in Swift 4.x/5.x. [SE-0354](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0354-regex-literals.md) adds the flag `-enable-bare-slash-regex` to enable the bare `/.../` regular expression syntax. And although it wasn't part of the proposal, the discussion of [SE-0335](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md) included requests for a compiler flag to require `any` on all existentials. These all have the same flavor, of opting existing Swift 4.x/5.x code into improvements that will come in Swift 6. + +This proposal explicitly embraces the piecemeal, intentional adoption of features that were held until Swift 6 for source-compatibility reasons. It establishes a direct path to incrementally adopt Swift 6 features, one-by-one, to gain their benefits in a Swift 4.x/5.x code base and smooth the migration path to a Swift 6 language mode. Developers can use a new compiler flag, `-enable-upcoming-feature X` to enable the specific feature named `X` for that module, and multiple features can be specified in this manner. When the developer moves to the next major language version, `X` will be implied by that language version and the compiler flag will be rejected. This way, upcoming feature flags only accumulate up to the next major Swift language version and are then cleared away, so we don't fork the language into incompatible dialects. + +Swift-evolution thread: [Pitch #1](https://forums.swift.org/t/piecemeal-adoption-of-swift-6-improvements-in-swift-5-x/57184) + +## Language version and tools version + +There are two related kinds of "Swift version" that are distinct, but we often conflate them for convenience. However, both kinds of version have a bearing on this proposal: + +- *Swift tools version*: the version number of the compiler itself. For example, the Swift 5.6 compiler was introduced in March 2022. +- *Swift language version*: the language version with which we are providing source compatibility. For example, Swift version 5 is the most current language version supported by Swift tools version 5.6. + +The Swift tools support multiple Swift language versions. All recent versions (since Swift tools version 5.0) have supported multiple Swift language versions, of which there are currently only three: 4, 4.2, and 5. As the tools evolve, they try to avoid making source-incompatible changes within a Swift language version, and this is also reflected in the evolution process itself: proposals that change the meaning of existing source code, or make it invalid, are generally not accepted for existing language modes. Many proposals do *extend* the Swift language within an existing language mode. For example, `async`/`await` became available in Swift tools version 5.5, and is available in all language versions (4, 4.2, 5). + +This proposal involves source-incompatible changes that are waiting for the introduction of a new Swift language version, e.g., 6. Swift tools version 6.0 will be the first tools to officially allow the use of Swift language version 6. Those tools will continue to support Swift language versions 4, 4.2, and 5. Code does not need to move to Swift language version 6 to use Swift tools version 6.0, or 6.1, and so on, and code written to Swift language version 6 will interoperate with code written to Swift language version 4, 4.2, or 5. + +## Proposed solution + +Introduce a compiler flag `-enable-upcoming-feature X`, where `X` is a name for the feature to enable. Each proposal will document what `X` is, so it's clear how to enable that feature. For example, SE-0274 could use `ConciseMagicFile`, so that `-enable-upcoming-feature ConciseMagicFile` will enable that change in semantics. One can of course pass multiple `-enable-upcoming-feature` flags to the compiler to enable multiple features. + +Unrecognized upcoming features will be ignored by the compiler. This allows older tools to use the same command lines as newer tools for Swift code that has started adopting new features, but has appropriate workarounds to still work with older tools. Sometimes this is possible because older compilers will still have a reasonable interpretation of the code, other times one will need a way to [detect features in source code](#feature-detection-in-source-code), the subject of a later section. + +All "upcoming" features are enabled by default in some language version. The compiler will produce an error if `-enable-upcoming-feature X` is provided and the language version enables the feature `X` by default. This will make it clear to developers when their expectations about when a feature is available, and clean up projects and manifests that have evolved from from earlier language versions, adopted features piecemeal, and then moved to later language versions. + +### Proposals define their own feature identifier + +Amend the [Swift proposal template](https://github.com/swiftlang/swift-evolution/blob/main/proposal-templates/0000-swift-template.md) with a new, optional field that defines the feature identifier: + +* **Feature identifier**: `UpperCamelCaseFeatureName` + +Amend the following proposals, which are partially or wholly delayed until Swift 6, with the following feature identifiers: + +* [SE-0274 "Concise magic file names"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0274-magic-file.md) (`ConciseMagicFile`) delayed the semantic change to `#file` until Swift 6. Enabling this feature changes `#file` to mean `#fileID` rather than `#filePath`. +* [SE-0286 "Forward-scan matching for trailing closures"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0286-forward-scan-trailing-closures.md) (`ForwardTrailingClosures`) delays the removal of the "backward-scan matching" rule of trailing closures until Swift 6. Enabling this feature removes the backward-scan matching rule. +* [SE-0335 "Introduce existential `any`"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md) (`ExistentialAny`) delays the requirement to use `any` for all existentials until Swift 6. Enabling this feature requires `any` for existential types. +* [SE-0337 "Incremental migration to concurrency checking"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md) (`StrictConcurrency`) delays some checking of the concurrency model to Swift 6 (with a flag to opt in to warnings about it in Swift 5.x). Enabling this feature is equivalent to `-warn-concurrency`, performing complete concurrency checking. +* [SE-0352 "Implicitly Opened Existentials"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md) (`ImplicitOpenExistentials`) expands implicit opening to more cases in Swift 6, because we didn't want to change the semantics of well-formed code in Swift 5.x. Enabling this feature performs implicit opening in these additional cases. +* [SE-0354 "Regex Literals"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0354-regex-literals.md) (`BareSlashRegexLiterals`) delays the introduction of the `/.../` regex literal syntax until Swift 6. Enabling this feature is equivalent to `-enable-bare-regex-syntax`, making the `/.../` regex literal syntax available. If this proposal and SE-0354 are accepted in the same release, `-enable-bare-regex-syntax` can be completely removed in favor of this approach. + +### Swift Package Manager support for upcoming features + +SwiftPM targets should be able to specify the upcoming language features they require. Extend `SwiftSetting` with an API to enable an upcoming feature: + +```swift +extension SwiftSetting { + public static func enableUpcomingFeature( + _ name: String, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting +} +``` + +SwiftPM would then pass each of the upcoming features listed there to the compiler via the `-enable-upcoming-feature` flag when building a module using this setting. Other targets that depend on this one do not need to pass the features when they build, because the effect of upcoming features does not cross module boundaries. + +The features are provided as strings here so that SwiftPM's manifest format doesn't need to change each time a new feature is added to the compiler. Package authors can add upcoming features while still supporting older tools without creating a new, versioned manifest. + +### Feature detection in source code + +When adopting a new feature, it's common to want code to still compile with older tools where that feature is not available. Doing so requires a way to check whether the feature is enabled, either by `-enable-upcoming-feature` or by enabling a suitable language version. + +We should extend Swift's `#if` with explicit support for a `hasFeature(X)` check, which evaluates true whenever the feature with identifier `X` is available. Code that needs to check for a specific feature can use `#if hasFeature` like this: + +```swift +#if hasFeature(ImplicitOpenExistentials) + f(aCollectionOfInts) +#else + f(AnyCollection(aCollectionOfInts)) +#endif +``` + +The `hasFeature(X)` check indicates the presence of features, but by itself an older compiler will still attempt to parse the `#if` branch even if the feature isn't known. That's fine for this feature (implicitly opened existentials) because it doesn't add any syntax, but other features that add syntax might require something more. `hasFeature` can be composed with the `compiler` directive introduced by [SE-0212](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0212-compiler-version-directive.md), e.g., + +```swift +#if compiler(>=5.7) && hasFeature(BareSlashRegexLiterals) +let regex = /.../ +#else +let regex = try NSRegularExpression(pattern: "...") +#endif +``` + +There is an issue with the above, because `hasFeature` *itself* is not understood by tools that predate this proposal, so the code above will fail to compile with any Swift compiler that predates the introduction of `hasFeature`. It is possible to avoid this problem by nesting the `hasFeature` check like this (assuming that Swift 5.7 introduced `hasFeature`): + +```swift +#if compiler(>=5.7) + #if hasFeature(BareSlashRegexLiterals) + let regex = /.../ + #else + let regex = #/.../# + #endif +#else +let regex = try NSRegularExpression(pattern: "...") +#endif +``` + +In the worst case, this does involve some code duplication for libraries that need to work on Swift versions that predate the introduction of `hasFeature`, but it is possible to handle those compilers, and over time that limitation will go away. + +To prevent this issue for any upcoming extensions to the `#if` syntax, the compiler should not attempt to interpret any "call" syntax on the right-hand side of a `&&` or `||` whose left-hand side disables parsing of the `#if` body, such as `compiler(>=5.7)` or `swift(>=6.0)`, and where the right-hand term is not required to determine the result of the whole expression. For example, if we invent something like `#if hasAttribute(Y)` in the future, one can use this formulation: + +```swift +#if compiler(>=5.8) && hasAttribute(Sendable) +... +#endif +``` + +On Swift 5.8 or newer compilers (which we assume will support `hasAttribute`), the full condition will be evaluated. On prior Swift compilers (i.e., ones that support this proposal but not something newer like `hasAttribute`), the code after the `&&` or `||` will be parsed as an expression, but will not be evaluated, so such compilers will not reject this `#if` condition. + +### Embracing experimental features + +It is common for language features in the compiler to be staged in behind an "experimental" flag as they are developed. This is usually done in an ad hoc manner, and the flag is removed before the feature finally ships. However, we should embrace the experimental feature model further: when a feature is under development, provide it with a feature identifier that allows it to be enabled with a new flag, `-enable-experimental-feature X`, or its SwiftPM counterpart `enableExperimentalFeature`. + +Experimental features are still to be considered unstable, and should not be available in released compilers. However, by unifying the manner in which experimental and upcoming features are introduced, we can rely on the same staging mechanisms: a way to enable the feature and to check for its presence in source code, making it easier to experiment with these features. If a feature then "graduates" to a complete, supported language feature, `hasFeature` can return true for it and, if part of it was delayed until the next major language version, `-enable-upcoming-feature` will work with it, too. + +## Source compatibility + +For the language itself, `hasFeature` is the only addition, and it occurs in a constrained syntactic space (`#if`) where there are no user-definable functions. Therefore, there is no source-compatibility issue in the traditional sense, where a newer compiler rejects existing, well-formed code. + +For SwiftPM, the addition of the `enableUpcomingFeature` and `enableExperimentalFeature` functions to `SwiftSetting` represents a one-time break in the manifest file format. Packages that wish to adopt these functions and support tools versions that predate the introduction of `enableUpcomingFeature` and `enableExperimentalFeature` can use versioned manifest, e.g., `Package@swift-5.6.swift`, to adopt the feature for newer tools versions. Once `enableUpcomingFeature` and `enableExperimentalFeature` have been added, adopting additional features this way won't require another copy of the manifest file. + +## Alternatives considered + +### `$X` instead of `hasFeature(X)` + +The original pitch for this proposal used special identifiers `$X` for feature detection instead of `hasFeature(X)`. `$X` has been used in the compiler implementation to help stage in Swift's concurrency features, especially when producing Swift interface files that might need to be understood by older tools versions. The compiler still defines `$AsyncAwait`, for example, which can be used with `#if` to check for async/await support: + +```swift +#if compiler(>=5.3) && $AsyncAwait +func f() async -> String +#endif +``` + +The primary advantage to the `$` syntax is that all Swift compilers already treat `$` as an acceptable leading character for an identifier. The compiler can define names with a leading `$`, but developers aren't technically supposed to, so it's effectively a reserved space for "magic" names. This means that, unlike the `hasFeature` formulation of the above, older compilers can process the code above without producing an error. + +However, this proposal introduces `hasFeature` because it's clearer in the code, and makes the forward-looking changes to the way `#if` conditions are processed to make it easier for additional `hasFeature`-like features to be introduced in the future without having this problem with older compilers. + +### Enabling optional features + +This proposal narrowly introduces `-enable-upcoming-feature` to only describe accepted features that will be enabled with a newer language version, but that were held back (partially or in full) due to source compatibility concerns. It is not meant to be used to enable "optional" features, which would create permanent dialects, and is designed to be somewhat self-healing: as folks move to newer language modes (e.g., Swift 6), the upcoming feature flags are eliminated with the new baseline. + +### Enabling all upcoming features + +The set of upcoming features will expand over time, as Swift introduces new features with source-compatibility impact that are staged in via a new major language version. For developers who want to be on the leading edge, it would be more convenient to have a single flag that enables all upcoming features, rather than having to specify each upcoming feature as they get added. However, the introduction of such a flag would create a shifting dialect of Swift: features are only "upcoming" features if they have source-compatibility impact, so code that adopted this flag could break with every new Swift release. That would directly cut against our source-compatibility goals for Swift, so we do not propose such a flag. Instead, we should find a central place to document all upcoming features on swift.org, updated with each release, so that developers know where to go to learn about the new upcoming features they want to enable. + +## Revision History + +* Changes from first reviewed version: + * Changed the SwiftPM manifest API to be based on `SwiftSettings` rather than the target. + * Use the term "upcoming feature" rather than "future feature" to reduce confusion. + * Don't parse the right-hand side of a `&&` or `||` that doesn't affect the result. + * Add some discussion of language and tools versions. + +## Acknowledgments + +Becca Royal-Gordon designed the original `#if compiler(>=5.5) && $AsyncAwait` approach to adopting features without breaking compatibility with older tools, and helped shape this design. Ben Rimmington provided the design for the SwiftPM API, replacing the less-flexible design from the original reviewed proposal. diff --git a/proposals/0363-unicode-for-string-processing.md b/proposals/0363-unicode-for-string-processing.md new file mode 100644 index 0000000000..a56e074ddf --- /dev/null +++ b/proposals/0363-unicode-for-string-processing.md @@ -0,0 +1,1361 @@ +# Unicode for String Processing + +* Proposal: [SE-0363](0363-unicode-for-string-processing.md) +* Authors: [Nate Cook](https://github.com/natecook1000), [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.7)** +* Implementation: [apple/swift-experimental-string-processing][repo] +* Review: ([pitch](https://forums.swift.org/t/pitch-unicode-for-string-processing/56907)), ([review](https://forums.swift.org/t/se-0363-unicode-for-string-processing/58520)), ([acceptance](https://forums.swift.org/t/accepted-se-0363-unicode-for-string-processing/59998)) + +### Version History + +- Version 1: Initial version +- Version 2: + - Improved option method API names + - Added Unicode property APIs to match regex syntax + - Added `CharacterClass.noneOf(_:)` and sequence-based `init` + - Clarified default state of options + - Added detail around switching semantic modes + - Added detail about Unicode property matching in character mode + - Revised details of custom character class matching + - Removed `\O`/`.anyUnicodeScalar` + +### Table of Contents + + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [Options](#options) + - [Case insensitivity](#case-insensitivity) + - [Single line mode (`.` matches newlines)](#single-line-mode--matches-newlines) + - [Multiline mode](#multiline-mode) + - [ASCII-only character classes](#ascii-only-character-classes) + - [Unicode word boundaries](#unicode-word-boundaries) + - [Matching semantic level](#matching-semantic-level) + - [Default repetition behavior](#default-repetition-behavior) + - [Character Classes](#character-classes) + - [“Any”](#any) + - [Digits](#digits) + - ["Word" characters](#word-characters) + - [Whitespace and newlines](#whitespace-and-newlines) + - [Unicode properties](#unicode-properties) + - [POSIX character classes: `[:NAME:]`](#posix-character-classes-name) + - [Custom classes](#custom-classes) + - [Source compatibility](#source-compatibility) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Effect on API resilience](#effect-on-api-resilience) + - [Future directions](#future-directions) + - [Alternatives considered](#alternatives-considered) + +## Introduction + +This proposal describes `Regex`'s rich Unicode support during regex matching, along with the character classes and options that define and modify that behavior. + +This proposal is one component of a larger [regex-powered string processing initiative](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0350-regex-type-overview.md). For the status of each proposal, [see this document](https://github.com/apple/swift-experimental-string-processing/blob/main/Documentation/Evolution/ProposalOverview.md) — discussion of other facets of the overall regex design is out of scope of this proposal and better discussed in the most relevant review. + +## Motivation + +Swift's `String` type provides, by default, a view of `Character`s or [extended grapheme clusters][graphemes] whose comparison honors [Unicode canonical equivalence][canoneq]. Each character in a string can be composed of one or more Unicode scalar values, while still being treated as a single unit, equivalent to other ways of formulating the equivalent character: + +```swift +let str = "Cafe\u{301}" // "Café" +str == "Café" // true +str.dropLast() // "Caf" +str.last == "é" // true (precomposed e with acute accent) +str.last == "e\u{301}" // true (e followed by composing acute accent) +``` + +This default view is fairly novel. Most languages that support Unicode strings generally operate at the Unicode scalar level, and don't provide the same affordance for operating on a string as a collection of grapheme clusters. In Python, for example, Unicode strings report their length as the number of scalar values, and don't use canonical equivalence in comparisons: + +```python +cafe = u"Cafe\u0301" +len(cafe) # 5 +cafe == u"Café" # False +``` + +Existing regex engines follow this same model of operating at the Unicode scalar level. To match canonically equivalent characters, or have equivalent behavior between equivalent strings, you must normalize your string and regex to the same canonical format. + +```python +# Matches a four-element string +re.match(u"^.{4}$", cafe) # None +# Matches a string ending with 'é' +re.match(u".+é$", cafe) # None + +cafeComp = unicodedata.normalize("NFC", cafe) +re.match(u"^.{4}$", cafeComp) # +re.match(u".+é$", cafeComp) # +``` + +With Swift's string model, this behavior would surprising and undesirable — Swift's default regex semantics must match the semantics of a `String`. + +
Other engines + +Other regex engines match character classes (such as `\w` or `.`) at the Unicode scalar value level, or even the code unit level, instead of recognizing grapheme clusters as characters. When matching the `.` character class, other languages will only match the first part of an `"e\u{301}"` grapheme cluster. Some languages, like Perl, Ruby, and Java, support an additional `\X` metacharacter, which explicitly represents a single grapheme cluster. + +| Matching `"Cafe\u{301}"` | Pattern: `^Caf.` | Remaining | Pattern: `^Caf\X` | Remaining | +|---|---|---|---|---| +| C#, Rust, Go, Python | `"Cafe"` | `"´"` | n/a | n/a | +| NSString, Java, Ruby, Perl | `"Cafe"` | `"´"` | `"Café"` | `""` | + +Other than Java's `CANON_EQ` option, the vast majority of other languages and engines are not capable of comparing with canonical equivalence. + +
+ +## Proposed solution + +In a regex's simplest form, without metacharacters or special features, matching behaves like a test for equality. A string always matches a regex that simply contains the same characters. + +```swift +let str = "Cafe\u{301}" // "Café" +str.contains(/Café/) // true +``` + +From that point, small changes continue to comport with the element counting and comparison expectations set by `String`: + +```swift +str.contains(/Caf./) // true +str.contains(/.+é/) // true +str.contains(/.+e\u{301}/) // true +str.contains(/\w+é/) // true +``` + +For compatibility with other regex engines and the flexibility to match at both `Character` and Unicode scalar level, you can switch between matching levels for an entire regex or within select portions. This powerful capability provides the expected default behavior when working with strings, while allowing you to drop down for Unicode scalar-specific matching. + +By default, literal characters and Unicode scalar values (e.g. `\u{301}`) are coalesced into characters the same way as a normal string, as shown above. Metacharacters, like `.` and `\w`, and custom character classes each match a single element at the current matching level. + +For example, these matches fail, because by the time the engine attempts to match the "`\u{301}`" Unicode scalar literal in the regex, the full `"é"` character in `str` has been matched, even though that character is made up of two Unicode scalar values: + +```swift +str.contains(/Caf.\u{301}/) // false - `.` matches "é" character +str.contains(/Caf\w\u{301}/) // false - `\w` matches "é" character +str.contains(/.+\u{301}/) // false - `.+` matches each character +``` + +Alternatively, we can drop down to use Unicode scalar semantics if we want to match specific Unicode sequences. For example, these regexes match an `"e"` followed by any modifier with the specified parameters: + +```swift +str.contains(/e[\u{300}-\u{314}]/.matchingSemantics(.unicodeScalar)) +// true - matches an "e" followed by a Unicode scalar in the range U+0300 - U+0314 +str.contains(/e\p{Nonspacing Mark}/.matchingSemantics(.unicodeScalar)) +// true - matches an "e" followed by a Unicode scalar with general category "Nonspacing Mark" +``` + +Matching in Unicode scalar mode is analogous to comparing against a string's `UnicodeScalarView` — individual Unicode scalars are matched without combining them into characters or testing for canonical equivalence. + +```swift +str.contains(/Café/.matchingSemantics(.unicodeScalar)) +// false - "e\u{301}" doesn't match with /é/ +str.contains(/Cafe\u{301}/.matchingSemantics(.unicodeScalar)) +// true - "e\u{301}" matches with /e\u{301}/ +``` + +Swift's `Regex` follows the level 2 guidelines for Unicode support in regular expressions described in [Unicode Technical Standard #18][uts18], with support for Unicode character classes, canonical equivalence, grapheme cluster matching semantics, and level 2 word boundaries enabled by default. In addition to selecting the matching semantics, `Regex` provides options for selecting different matching behaviors, such as ASCII character classes or Unicode scalar semantics, which corresponds more closely with other regex engines. + +## Detailed design + +First, we'll discuss the options that let you control a regex's behavior, and then explore the character classes that define the your pattern. + +As detailed below, there are a few differences in defaults between Swift's `Regex` and the typical regex engine. In particular: + +- `Regex` matches at the Swift `Character` level, instead of matching Unicode scalars, UTF-16 code units, or bytes. A regex that deliberately matches multi-scalar characters may need to switch to Unicode scalar semantics. +- `Regex` uses "default" word boundaries, instead of "simple" word boundaries. A regex that expects `\b` to always match the boundary between a word character (`\w`) and a non-word character (`\W`) may need to switch to simple word boundaries. +- For multi-line regex literals, extended syntax is automatically enabled, which ignores whitespace both in patterns and within custom character classes. To use semantic whitespace, you can temporarily disable extended mode (`(?-x:...)`), quote a section of your pattern (`\Q...\E`), or escape a space explicitly (`a\ b`). + +### Options + +Options can be enabled and disabled in two different ways: as part of [regex internal syntax][internals], or applied as methods when declaring a `Regex`. For example, both of these `Regex`es are declared with case insensitivity: + +```swift +let regex1 = /(?i)banana/ +let regex2 = Regex { + "banana" +}.ignoresCase() +``` + +Because the `ignoresCase()` option method is defined on `Regex`, you can always use the more readable option-setting interface in conjunction with regex literals or run-time compiled `Regex`es: + +```swift +let regex3 = /banana/.ignoresCase() +``` + +Calling an option-setting method like `ignoresCase()` acts like wrapping the callee in an option-setting group `(?:...)`. That is, while it sets the behavior for the callee, it doesn’t override options that are applied to more specific regions. In this example, the middle `"na"` in `"banana"` matches case-sensitively, despite the outer call to `ignoresCase()`: + +```swift +let regex4 = Regex { + "ba" + Regex { "na" }.ignoresCase(false) + "na" +}.ignoresCase() + +"banana".contains(regex4) // true +"BAnaNA".contains(regex4) // true +"BANANA".contains(regex4) // false + +// Equivalent to: +let regex5 = /(?i)ba(?-i:na)na/ +``` + +The options that `Regex` supports are shown in the table below, in three groups: Options that affect matching behavior for _both regex syntax and APIs_, options that affect the matching behavior of _regex syntax only_, and options with _structural_ or _syntactic_ effects that are only supported through regex syntax. + +| **Matching Behavior** | | | Default | +|------------------------------|----------------|---------------------------------|--------------------| +| Case insensitivity | `(?i)` | `ignoresCase()` | disabled | +| ASCII-only character classes | `(?DSWP)` | `asciiOnlyClasses(_:)` | `.none` | +| Unicode word boundaries | `(?w)` | `wordBoundaryKind(_:)` | `.default` | +| Semantic level | n/a | `matchingSemantics(_:)` | `.graphemeCluster` | +| Default repetition behavior | n/a | `defaultRepetitionBehavior(_:)` | `.eager` | +| **Regex Syntax Only** | | | | +| Single-line mode | `(?s)` | `dotMatchesNewlines()` | disabled | +| Multi-line mode | `(?m)` | `anchorsMatchNewlines()` | disabled | +| Swap eager/reluctant | `(?U)` | n/a | disabled | +| **Structural/Syntactic** | | | | +| Extended syntax | `(?x)`,`(?xx)` | n/a | `xx` enabled in multi-line regex literals; otherwise, off | +| Named captures only | `(?n)` | n/a | disabled | + +#### Case insensitivity + +Regexes perform case sensitive comparisons by default. The `i` option or the `ignoresCase(_:)` method enables case insensitive comparison. + +```swift +let str = "Café" + +str.firstMatch(of: /CAFÉ/) // nil +str.firstMatch(of: /(?i)CAFÉ/) // "Café" +str.firstMatch(of: /cAfÉ/.ignoresCase()) // "Café" +``` + +Case insensitive matching uses case folding to ensure that canonical equivalence continues to operate as expected. + +**Regex syntax:** `(?i)...` or `(?i:...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression that ignores casing when matching. + public func ignoresCase(_ ignoresCase: Bool = true) -> Regex +} +``` + +#### ASCII-only character classes + +With one or more of these options enabled, the default character classes match only ASCII values instead of the full Unicode range of characters. Four options are included in this group: + +* Regex syntax `(?D)`: Match only ASCII members for `\d`, `[:digit:]`, and `CharacterClass.digit`. +* Regex syntax `(?S)`: Match only ASCII members for `\s`, `[:space:]`, and any of the whitespace-representing `CharacterClass` members. +* Regex syntax `(?W)`: Match only ASCII members for `\w`, `[:word:]`, and `CharacterClass.word`. Also only considers ASCII characters for `\b`, `\B`, and `Anchor.wordBoundary`. +* Regex syntax `(?D)`: Match only ASCII members for all POSIX properties (including `digit`, `space`, and `word`). + +This option affects the built-in character classes listed in the "Character Classes" section below. When one or more of these options is enabled, the set of characters matched by those character classes is constrained to the ASCII character set. For example, `CharacterClass.hexDigit` usually matches `0...9`, `a-f`, and `A-F`, in either the ASCII or half-width variants. When the `(?D)` or `.asciiOnlyClasses(.digit)` options are enabled, only the ASCII characters are matched. + +```swift +let str = "0x35AB" +str.contains(/0x(\d+)/.asciiOnlyClasses()) +``` + +**Regex syntax:** `(?DSWP)...` or `(?DSWP...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression that only matches ASCII characters as digits. + public func asciiOnlyClasses(_ kinds: RegexCharacterClassKind = .all) -> Regex +} + +/// A built-in regex character class kind. +/// +/// Pass one or more `RegexCharacterClassKind` classes to `asciiOnlyClasses(_:)` +/// to control whether character classes match any character or only members +/// of the ASCII character set. +public struct RegexCharacterClassKind: OptionSet, Hashable { + public var rawValue: Int { get } + + /// Regex digit-matching character classes, like `\d`, `[:digit:]`, and + /// `\p{HexDigit}`. + public static var digit: RegexCharacterClassKind { get } + + /// Regex whitespace-matching character classes, like `\s`, `[:space:]`, + /// and `\p{Whitespace}`. + public static var whitespace: RegexCharacterClassKind { get } + + /// Regex word character-matching character classes, like `\w`. + public static var wordCharacter: RegexCharacterClassKind { get } + + /// All built-in regex character classes. + public static var all: RegexCharacterClassKind { get } + + /// No built-in regex character classes. + public static var none: RegexCharacterClassKind { get } +} +``` + +#### Unicode word boundaries + +By default, matching word boundaries with the `\b` and `Anchor.wordBoundary` anchors uses Unicode _default word boundaries,_ specified as [Unicode level 2 regular expression support][level2-word-boundaries]. + +Disabling the `w` option switches to _[simple word boundaries][level1-word-boundaries],_ finding word boundaries at points in the input where `\b\B` or `\B\b` match. Depending on the other matching options that are enabled, this may be more compatible with the behavior other regex engines. + +As shown in this example, the default matching behavior finds the whole first word of the string, while the match with simple word boundaries stops at the apostrophe: + +```swift +let str = "Don't look down!" + +str.firstMatch(of: /D\S+\b/) +// "Don't" +str.firstMatch(of: /D\S+\b/.wordBoundaryKind(.simple)) +// "Don" +``` + +You can see more differences between level 1 (simple) and level 2 (default) word boundaries in the following table, generated by calling `matches(of: /\b.+\b/)` on the strings in the first column: + +| Example | Level 1 | Level 2 | +|---------------------|---------------------------------|-------------------------------------------| +| I can't do that. | ["I", "can", "t", "do", "that"] | ["I", "can't", "do", "that", "."] | +| 🔥😊👍 | ["🔥😊👍"] | ["🔥", "😊", "👍"] | +| 👩🏻👶🏿👨🏽🧑🏾👩🏼 | ["👩🏻👶🏿👨🏽🧑🏾👩🏼"] | ["👩🏻", "👶🏿", "👨🏽", "🧑🏾", "👩🏼"] | +| 🇨🇦🇺🇸🇲🇽 | ["🇨🇦🇺🇸🇲🇽"] | ["🇨🇦", "🇺🇸", "🇲🇽"] | +| 〱㋞ツ | ["〱", "㋞", "ツ"] | ["〱㋞ツ"] | +| hello〱㋞ツ | ["hello〱", "㋞", "ツ"] | ["hello", "〱㋞ツ"] | +| 나는 Chicago에 산다 | ["나는", "Chicago에", "산다"] | ["나", "는", "Chicago", "에", "산", "다"] | +| 眼睛love食物 | ["眼睛love食物"] | ["眼", "睛", "love", "食", "物"] | +| 아니ㅋㅋㅋ네 | ["아니ㅋㅋㅋ네"] | ["아", "니", "ㅋㅋㅋ", "네"] | +| Re:Zero | ["Re", "Zero"] | ["Re:Zero"] | +| \u{d}\u{a} | ["\u{d}", "\u{a}"] | ["\u{d}\u{a}"] | +| €1 234,56 | ["1", "234", "56"] | ["€", "1", "234,56"] | + + +**Regex syntax:** `(?-w)...` or `(?-w...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression that uses the specified word boundary algorithm. + /// + /// A simple word boundary is a position in the input between two characters + /// that match `/\w\W/` or `/\W\w/`, or between the start or end of the input + /// and `\w` character. Word boundaries therefore depend on the option-defined + /// behavior of `\w`. + /// + /// The default word boundaries use a Unicode algorithm that handles some cases + /// better than simple word boundaries, such as words with internal + /// punctuation, changes in script, and Emoji. + public func wordBoundaryKind(_ wordBoundaryKind: RegexWordBoundaryKind) -> Regex +} + +public struct RegexWordBoundaryKind: Hashable { + /// A word boundary algorithm that implements the "simple word boundary" + /// Unicode recommendation. + /// + /// A simple word boundary is a position in the input between two characters + /// that match `/\w\W/` or `/\W\w/`, or between the start or end of the input + /// and a `\w` character. Word boundaries therefore depend on the option- + /// defined behavior of `\w`. + public static var simple: Self { get } + + /// A word boundary algorithm that implements the "default word boundary" + /// Unicode recommendation. + /// + /// Default word boundaries use a Unicode algorithm that handles some cases + /// better than simple word boundaries, such as words with internal + /// punctuation, changes in script, and Emoji. + public static var default: Self { get } +} +``` + +#### Matching semantic level + +To support both matching on `String`'s default character-by-character view and more broadly-compatible Unicode scalar-based matching, you can select a matching level for an entire regex or a portion of a regex constructed with the `RegexBuilder` API. + +When matching with *grapheme cluster semantics* (the default), metacharacters like `.` and `\w`, custom character classes, and character class instances like `.any` match a grapheme cluster, corresponding with the default string representation. In addition, matching with grapheme cluster semantics compares characters using their canonical representation, corresponding with the way comparing strings for equality works. + +When matching with *Unicode scalar semantics*, metacharacters and character classes match a single Unicode scalar value, even if that scalar comprises part of a grapheme cluster. Canonical representations are _not_ used, corresponding with the way comparison would work when using a string's `UnicodeScalarView`. + +These specific levels of matching, and the options to switch between them, are unique to Swift, but not unprecedented in other regular expression engines. Several engines, including Perl, Java, and ICU-based engines like `NSRegularExpression`, support the `\X` metacharacter for matching a grapheme cluster within otherwise Unicode scalar semantic matching. Rust has a related concept in its [`regex::bytes` type][regexbytes], which matches over arbitrary bytes by default but allows switching into Unicode mode for segments of the regular expression. + +These semantic levels lead to different results when working with strings that have characters made up of multiple Unicode scalar values, such as Emoji or decomposed characters. In the following example, `queRegex` matches any 3-character string that begins with `"q"`. + +```swift +let composed = "qué" +let decomposed = "que\u{301}" + +let queRegex = /^q..$/ + +print(composed.contains(queRegex)) +// Prints "true" +print(decomposed.contains(queRegex)) +// Prints "true" +``` + +When using Unicode scalar semantics, however, the regex only matches the composed version of the string, because each `.` matches a single Unicode scalar value. + +```swift +let queRegexScalar = queRegex.matchingSemantics(.unicodeScalar) +print(composed.contains(queRegexScalar)) +// Prints "true" +print(decomposed.contains(queRegexScalar)) +// Prints "false" +``` + +The index boundaries of the overall match and capture groups are affected by the matching semantic level. With grapheme cluster semantics, the start and end index of the overall match and each capture is `Character`-aligned. Matching with Unicode scalar semantics, on the other hand, can yield string indices that aren't aligned to character boundaries. Take care when using indices that aren't aligned with grapheme cluster boundaries, as they may have to be rounded to a boundary if used in a `String` instance. + +```swift +let family = "👨‍👨‍👧‍👦 is a family" + +// Grapheme-cluster mode: Yields a character +let firstCharacter = /^./ +let characterMatch = family.firstMatch(of: firstCharacter)!.output +print(characterMatch) +// Prints "👨‍👨‍👧‍👦" + +// Unicode-scalar mode: Yields only part of a character +let firstUnicodeScalar = /^./.matchingSemantics(.unicodeScalar) +let unicodeScalarMatch = family.firstMatch(of: firstUnicodeScalar)!.output +print(unicodeScalarMatch) +// Prints "👨" + +// The end of `unicodeScalarMatch` is not aligned on a character boundary +print(unicodeScalarMatch.endIndex == family.index(after: family.startIndex)) +// Prints "false" +``` + +When there is a boundary between Unicode scalar semantic and grapheme scalar semantic matching in the middle of a regex, an implicit grapheme cluster boundary assertion is added at the start of the grapheme scalar semantic section. That is, the two regexes in the following example are equivalent; each matches a single "word" scalar, followed by a combining mark scalar, followed by one or more grapheme clusters. + +```swift +let explicit = Regex { + Regex { + CharacterClass.word + CharacterClass.generalCategory(.combiningMark) + }.matchingSemantics(.unicodeScalar) + Anchor.graphemeClusterBoundary // explicit grapheme cluster boundary + OneOrMore(.any) +} + +let implicit = Regex { + Regex { + CharacterClass.word + CharacterClass.generalCategory(.combiningMark) + }.matchingSemantics(.unicodeScalar) + OneOrMore(.any) +} + +try implicit.wholeMatch(in: "e\u{301} abc") // match +try implicit.wholeMatch(in: "e\u{301}\u{327} abc") // no match +``` + +The second call to `wholeMatch(in:)` fails because at the point the matching engine exits the inner regex, the matching position is still in the middle of the `"e\u{301}\u{327}` character. This implicit grapheme cluster boundary assertion maintains the guarantee that capture groups over grapheme cluster semantic sections will have valid character-aligned indices. + +If a regex starts or ends with a Unicode scalar semantic section, there is no assertion added at the start or end of the pattern. Consider the following regex, which has Unicode scalars for the entire pattern except for a section in the middle that matches a purple heart emoji. When applied to a string with a multi-scalar character before or after the `"💜"`, the resulting match includes a partial character at its beginning and end. + +```swift +let regex = Regex { + CharacterClass.any + Regex { // <-- Implicit grapheme cluster boundary assertion, as above + CharacterClass.binaryProperty(\.isEmoji) + }.matchingSemantics(.graphemeCluster) + CharacterClass.any +}.matchingSemantics(.unicodeScalar) + +let borahae = "태형💜아미" // Note: These hangeul characters are decomposed +if let match = borahae.firstMatch(of: regex) { + print(match.0) +} +// Prints "ᆼ💜ᄋ" +``` + +Boundaries from a grapheme cluster section into a Unicode scalar also imply a grapheme cluster boundary, but in this case no assertion is needed. This boundary is an emergent property of the fact that under grapheme cluster semantics, matching always happens one character at a time. + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression that matches with the specified semantic + /// level. + public func matchingSemantics(_ semanticLevel: RegexSemanticLevel) -> Regex +} + +public struct RegexSemanticLevel: Hashable { + /// Match at the default semantic level of a string, where each matched + /// element is a `Character`. + public static var graphemeCluster: RegexSemanticLevel + + /// Match at the semantic level of a string's `UnicodeScalarView`, where each + /// matched element is a `UnicodeScalar` value. + public static var unicodeScalar: RegexSemanticLevel +} +``` + +#### Default repetition behavior + +The `defaultRepetitionBehavior(_:)` method lets you set the default behavior for all quantifiers that don't explicitly provide their own behavior. For example, you can make all quantifiers behave possessively, eliminating any quantification-caused backtracking. This option applies both to quanitifiers in regex syntax that don't include an additional `?` or `+` (indicating reluctant or possessive quantification, respectively) and quantifiers in `RegexBuilder` syntax without an explicit behavior parameter. + +In the following example, both regexes use possessive quantification: + +```swift +let regex1 = /[0-9a-f]+\s*$/.defaultRepetitionBehavior(.possessive) + +let regex2 = Regex { + OneOrMore { + CharacterClass.anyOf( + "0"..."9", + "a"..."f" + ) + } + ZeroOrMore(.whitespace) + Anchor.endOfInput +}.defaultRepetitionBehavior(.possessive) +``` + +This option is related to, but independent from, the regex syntax option `(?U)`. See below for more about that regex-syntax-only option. + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression where quantifiers use the specified behavior + /// by default. + /// + /// You can call this method to change the default repetition behavior for + /// quantifier operators in regex syntax and `RegexBuilder` quantifier + /// methods. For example, in the following example, both regexes use + /// possessive quantification when matching a quotation surround by `"` + /// quote marks: + /// + /// let regex1 = /"[^"]*"/.defaultRepetitionBehavior(.possessive) + /// + /// let quoteMark = "\"" + /// let regex2 = Regex { + /// quoteMark + /// ZeroOrMore(.noneOf(quoteMark)) + /// quoteMark + /// }.defaultRepetitionBehavior(.possessive) + /// + /// This setting only changes the default behavior of quantifiers, and does + /// not affect regex syntax operators with an explicit behavior indicator, + /// such as `*?` or `++`. Likewise, calls to quantifier methods such as + /// `OneOrMore` always use the explicit `behavior`, when given. + /// + /// - Parameter behavior: The default behavior to use for quantifiers. + public func defaultRepetitionBehavior(_ behavior: RegexRepetitionBehavior) -> Regex +} + +public struct RegexRepetitionBehavior { + /// Match as much of the input string as possible, backtracking when + /// necessary. + public static var eager: RegexRepetitionBehavior { get } + + /// Match as little of the input string as possible, expanding the matched + /// region as necessary to complete a match. + public static var reluctant: RegexRepetitionBehavior { get } + + /// Match as much of the input string as possible, performing no backtracking. + public static var possessive: RegexRepetitionBehavior { get } +} +``` + +As described in the [Regex Builder proposal][regexbuilder], `RegexBuilder` quantifier APIs include a `nil`-defaulted optional `behavior` parameter. When you pass `nil`, the quantifier uses the default behavior as set by this option. If an explicit behavior is passed, that behavior is used regardless of the default. + +```swift +// Example `OneOrMore` initializer +extension OneOrMore { + public init( + _ behavior: RegexRepetitionBehavior? = nil, + @RegexComponentBuilder _ component: () -> Component + ) where Output == (Substring, C0), Component.Output == (W, C0) +} +``` + +#### Single line mode (`.` matches newlines) + +The "any" metacharacter (`.`) matches any character in a string *except* newlines by default. With the `s` option enabled, `.` matches any character including newlines. + +```swift +let str = """ + <> + """ + +str.firstMatch(of: /<<.+>>/) +// nil +str.firstMatch(of: /<<.+>>/.dotMatchesNewLines()) +// "This string\nuses double-angle-brackets\nto group text." +``` + +This option applies only to `.` used in regex syntax and does _not_ affect the behavior of `CharacterClass.any`, which always matches any character or Unicode scalar. To get the default `.` behavior when using `RegexBuilder` syntax, use `CharacterClass.anyNonNewline`. + +**Regex syntax:** `(?s)...` or `(?s...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression where the start and end of input + /// anchors (`^` and `$`) also match against the start and end of a line. + public func dotMatchesNewlines(_ dotMatchesNewlines: Bool = true) -> Regex +} +``` + +#### Multiline mode + +By default, the start and end anchors (`^` and `$`) match only the beginning and end of a string. With the `m` or the option, they also match the beginning and end of each line. + +```swift +let str = """ + abc + def + ghi + """ + +str.firstMatch(of: /^abc/) +// "abc" +str.firstMatch(of: /^abc$/) +// nil +str.firstMatch(of: /^abc$/.anchorsMatchLineEndings()) +// "abc" + +str.firstMatch(of: /^def/) +// nil +str.firstMatch(of: /^def$/.anchorsMatchLineEndings()) +// "def" +``` + +This option applies only to anchors used in regex syntax. The anchors defined in `RegexBuilder` are specific about matching at the start/end of the input or the line, and therefore are not affected by this option. + +```swift +str.firstMatch(of: Regex { Anchor.startOfInput ; "def" }) // nil +str.firstMatch(of: Regex { Anchor.startOfLine ; "def" }) // "def" +``` + +**Regex syntax:** `(?m)...` or `(?m...)` + +**Standard Library API:** + +```swift +extension Regex { + /// Returns a regular expression where the start and end of input + /// anchors (`^` and `$`) also match against the start and end of a line. + public func anchorsMatchLineEndings(_ matchLineEndings: Bool = true) -> Regex +} +``` + +#### Eager/reluctant toggle + +Regex quantifiers (`+`, `*`, and `?`) match eagerly by default when they repeat, such that they match the longest possible substring. Appending `?` to a quantifier makes it reluctant, instead, so that it matches the shortest possible substring. + +```swift +let str = "A value." + +// By default, the '+' quantifier is eager, and consumes as much as possible. +str.firstMatch(of: /<.+>/) // "A value." + +// Adding '?' makes the '+' quantifier reluctant, so that it consumes as little as possible. +str.firstMatch(of: /<.+?>/) // "" +``` + +The `U` option toggles the "eagerness" of quantifiers, so that quantifiers are reluctant by default, and only become eager when `?` is added to the quantifier. This change only applies within regex syntax. See the `defaultRepetitionBehavior(_:)` method, described above, for broader control over repetition behavior, including setting the default for `RegexBuilder` syntax. + +```swift +// '(?U)' toggles the eagerness of quantifiers: +str.firstMatch(of: /(?U)<.+>/) // "" +str.firstMatch(of: /(?U)<.+?>/) // "A value." +``` + +**Regex syntax:** `(?U)...` or `(?U...)` + + +--- + +### Character Classes + +We propose the following definitions for regex character classes, along with a `CharacterClass` type as part of the `RegexBuilder` module, to encapsulate and simplify character class usage within builder-style regexes. + +The two regexes defined in this example will match the same inputs, looking for one or more word characters followed by up to three digits, optionally separated by a space: + +```swift +let regex1 = /\w+\s?\d{,3}/ +let regex2 = Regex { + OneOrMore(.word) + Optionally(.whitespace) + Repeat(.digit, ...3) +} +``` + +You can build custom character classes by combining regex-defined classes with individual characters or ranges, or by performing common set operations such as subtracting or negating a character class. + + +#### “Any” + +The simplest character class, representing **any character**, is written as `.` and is sometimes referred to as the "dot" metacharacter. This class always matches a single `Character` or Unicode scalar value, depending on the matching semantic level. This class excludes newlines, unless "single line mode" is enabled (see section above). + +When using the `CharacterClass` type in a `RegexBuilder`-defined regex, the `.any` and `.anyNonNewline` provide separate APIs for the two behaviors of `.`, and are therefore unaffected by the current "single line mode" setting. + +```swift +"Cafe\u{301}".contains(/C.../) +// true +``` + +For this example, using Unicode scalar semantics, a dot matches only a single Unicode scalar value, so the combining marks don't get grouped with the commas before them: + +```swift +let data = "\u{300},\u{301},\u{302},\u{303},..." +for match in data.matches(of: /(.),/.matchingSemantics(.unicodeScalar)) { + print(match.1) +} +// Prints: +// ̀ +// ́ +// ̂ +// ... +``` + +#### Any grapheme cluster + +`Regex` also provides a way to match a single grapheme cluster, regardless of the current semantic level. The **any grapheme cluster** character class is written as `\X` or `CharacterClass.anyGraphemeCluster`, and matches from the current location up to the next grapheme cluster boundary. This includes matching newlines, regardless of any option settings. This metacharacter is equivalent to the regex syntax `(?Xs:.)`. + + +#### Digits + +The **decimal digit** character class is matched by `\d` or `CharacterClass.digit`. Both regexes in this example match one or more decimal digits followed by a colon: + +```swift +let regex1 = /\d+:/ +let regex2 = Regex { + OneOrMore(.digit) + ":" +} +``` + +_Unicode scalar semantics:_ Matches a Unicode scalar that has a `numericType` property equal to `.decimal`. This includes the digits from the ASCII range, from the _Halfwidth and Fullwidth Forms_ Unicode block, as well as digits in some scripts, like `DEVANAGARI DIGIT NINE` (U+096F). This corresponds to the general category `Decimal_Number`. + +_Grapheme cluster semantics:_ Matches a character made up of a single Unicode scalar that fits the decimal digit criteria above. + + +To invert the decimal digit character class, use `\D` or `CharacterClass.digit.inverted`. + + +The **hexadecimal digit** character class is matched by `CharacterClass.hexDigit`. + +_Unicode scalar semantics:_ Matches a decimal digit, as described above, or an uppercase or small `A` through `F` from the _Halfwidth and Fullwidth Forms_ Unicode block. Note that this is a broader class than described by the `UnicodeScalar.properties.isHexDigit` property, as that property only include ASCII and fullwidth decimal digits. + +_Grapheme cluster semantics:_ Matches a character made up of a single Unicode scalar that fits the hex digit criteria above. + +To invert the hexadecimal digit character class, use `CharacterClass.hexDigit.inverted`. + +*
Rationale* + +Unicode's recommended definition for `\d` is its [numeric type][numerictype] of "Decimal" in contrast to "Digit". It is specifically restricted to sets of ascending contiguously-encoded scalars in a decimal radix positional numeral system. Thus, it excludes "digits" such as superscript numerals from its [definition][derivednumeric] and is a proper subset of `Character.isWholeNumber`. + +We interpret Unicode's definition of the set of scalars, especially its requirement that scalars be encoded in ascending chains, to imply that this class is restricted to scalars which meaningfully encode base-10 digits. Thus, we choose to make the grapheme cluster interpretation *restrictive*. + +
+ + +#### "Word" characters + +The **word** character class is matched by `\w` or `CharacterClass.word`. This character class and its name are essentially terms of art within regexes, and represents part of a notional "word". Note that, by default, this is distinct from the algorithm for identifying word boundaries. + +_Unicode scalar semantics:_ Matches a Unicode scalar that has one of the Unicode properties `Alphabetic`, `Digit`, or `Join_Control`, or is in the general category `Mark` or `Connector_Punctuation`. + +_Grapheme cluster semantics:_ Matches a character that begins with a Unicode scalar value that fits the criteria above. + +To invert the word character class, use `\W` or `CharacterClass.word.inverted`. + +*
Rationale* + +Word characters include more than letters, and we went with Unicode's recommended scalar semantics. Following the Unicode recommendation that nonspacing marks remain with their base characters, we extend to grapheme clusters similarly to `Character.isLetter`. That is, combining scalars do not change the word-character-ness of the grapheme cluster. + +
+ + +#### Whitespace and newlines + +The **whitespace** character class is matched by `\s` and `CharacterClass.whitespace`. + +_Unicode scalar semantics:_ Matches a Unicode scalar that has the Unicode properties `Whitespace`, including a space, a horizontal tab (U+0009), `LINE FEED (LF)` (U+000A), `LINE TABULATION` (U+000B), `FORM FEED (FF)` (U+000C), `CARRIAGE RETURN (CR)` (U+000D), and `NEWLINE (NEL)` (U+0085). Note that under Unicode scalar semantics, `\s` only matches the first scalar in a `CR`+`LF` pair. + +_Grapheme cluster semantics:_ Matches a character that begins with a `Whitespace` Unicode scalar value. This includes matching a `CR`+`LF` pair. + +The **horizontal whitespace** character class is matched by `\h` and `CharacterClass.horizontalWhitespace`. + +_Unicode scalar semantics:_ Matches a Unicode scalar that has the Unicode general category `Zs`/`Space_Separator` as well as a horizontal tab (U+0009). + +_Grapheme cluster semantics:_ Matches a character that begins with a Unicode scalar value that fits the criteria above. + +The **vertical whitespace** character class is matched by `\v` and `CharacterClass.verticalWhitespace`. Additionally, `\R` and `CharacterClass.newline` provide a way to include the `CR`+`LF` pair, even when matching with Unicode scalar semantics. + +_Unicode scalar semantics:_ Matches a Unicode scalar that has the Unicode general category `Zl`/`Line_Separator` or `Zp`/`Paragraph_Separator`, as well as any of the following control characters: `LINE FEED (LF)` (U+000A), `LINE TABULATION` (U+000B), `FORM FEED (FF)` (U+000C), `CARRIAGE RETURN (CR)` (U+000D), and `NEWLINE (NEL)` (U+0085). Only when specified as `\R` or `CharacterClass.newline` does this match the whole `CR`+`LF` pair. + +_Grapheme cluster semantics:_ Matches a character that begins with a Unicode scalar value that fits the criteria above. + +To invert these character classes, use `\S`, `\H`, and `\V`, respectively, or the `inverted` property on a `CharacterClass` instance. + +
Rationale + +Note that "whitespace" is a term-of-art and is not correlated with visibility, which is a completely separate concept. + +We use Unicode's recommended scalar semantics for horizontal and vertical whitespace, extended to grapheme clusters as in the existing `Character.isWhitespace` property. + +
+ + +#### Unicode properties + +Character classes that match **Unicode properties** are written as `\p{PROPERTY}` or `\p{PROPERTY=VALUE}`, as described in the [Run-time Regex Construction proposal][internals-properties]. + +While most Unicode properties are only defined at the scalar level, some are defined to match an extended grapheme cluster. For example, `\p{RGI_Emoji_Flag_Sequence}` will match any flag emoji character, which are composed of two Unicode scalar values. Such property classes will match multiple scalars, even when matching with Unicode scalar semantics. + +Unicode property matching is extended to `Character`s with a goal of consistency with other regex character classes, and as dictated by prior standard library additions to the `Character` type. For example, for `\p{Decimal}` and `\p{Hex_Digit}`, only single-scalar `Character`s can match, for the reasons described in that section, above. For other Unicode property classes, like `\p{Whitespace}`, the character matches when the first scalar has that Unicode property. Open the following disclosure area to see the full list of properties, along with the rubric for extending them to grapheme clusters. + +
Unicode properties + +We can choose to extend a Unicode property to a grapheme cluster in one of several ways: + +- *single-scalar*: Only a character that comprises a single Unicode scalar value can match +- *first-scalar*: If the first Unicode scalar in a character matches, then the character matches +- *any-scalar*: If any Unicode scalar in a character matches, then the character matches +- *all-scalars*: A character matches if and only if all its Unicode scalar members match + +With a few guidelines, we can make headway on classifying Unicode properties: + +- Numeric-related properties, like `Numeric_Value`, should only apply to single-scalar characters, for the reasons described in the "Digit" character class section, above. +- Any other properties that directly or approximately correspond to regex or POSIX character classes should use the first-scalar rule. This corresponds with the way `Character.isWhitespace` is implemented and generally matches the perceived categorization of characters. +- Properties that resolve to a unique Unicode scalar, such as `Name`, should only apply to single-scalar characters. +- Properties that govern the way Unicode scalars combine into characters, such as `Canonical_Combining_Class`, or are otherwise only relevant when examining specific Unicode data, such as `Age`, should only apply to single-scalar characters. +- Properties that can naturally apply to a sequence of Unicode scalars, such as `Lowercase_Mapping`, should use an all-scalars approach. This corresponds with the way `Character.isLowercased` and other casing properties are implemented. + +In many cases, properties with a *single-scalar* treatment won't match any characters at all, and will only be useful when matching with Unicode scalar semantics. For example, `/\p{Emoji_Modifier}/` matches the five Fitzpatrick skin tone modifier Unicode scalar values that affect the appearance of emoji within a grapheme cluster. When matching with grapheme cluster semantics, no match for the pattern will be found. Using Unicode scalar semantics, however, you can search for all characters that include such a modifier: + +``` +let regex = /(?u)\y.+?\p{Emoji_Modifier}.+?\y/ +for ch in "👩🏾‍🚀🚀 👨🏻‍🎤🎸 🧑🏻‍💻📲".matches(of: regex) { + print(ch) +} +// Prints: +// 👩🏾‍🚀 +// 👨🏻‍🎤 +// 🧑🏻‍💻 +``` + +The table below shows our best effort at choosing the right manner of extending. + +| Property | Extension | Notes | +|-------------------------------------|-------------------------------|-----------------------------------| +| **General** | | | +| `Name` | single-scalar | | +| `Name_Alias` | single-scalar | | +| `Age` | single-scalar | | +| `General_Category` | first-scalar | Numeric categories: single-scalar | +| `Script` | first-scalar | | +| `White_Space` | first-scalar | Existing `Character` API | +| `Alphabetic` | first-scalar | | +| `Noncharacter_Code_Point` | single-scalar | | +| `Default_Ignorable_Code_Point` | single-scalar | | +| `Deprecated` | single-scalar | | +| `Logical_Order_Exception` | single-scalar | | +| `Variation_Selector` | single-scalar | | +|
**Numeric** | | | +| `Numeric_Value` | single-scalar | | +| `Numeric_Type` | single-scalar | | +| `Hex_Digit` | single-scalar | Existing `Character` API | +| `ASCII_Hex_Digit` | single-scalar | | +|
**Identifiers** | | | +| `ID_Start` | single-scalar | | +| `ID_Continue` | single-scalar | | +| `XID_Start` | single-scalar | | +| `XID_Continue` | single-scalar | | +| `Pattern_Syntax` | single-scalar | | +| `Pattern_White_Space` | single-scalar | | +|
**CJK** | | | +| `Ideographic` | first-scalar | | +| `Unified_Ideograph` | first-scalar | | +| `Radical` | first-scalar | | +| `IDS_Binary_Operator` | single-scalar | | +| `IDS_Trinary_Operator` | single-scalar | | +|
**Case** | | | +| `Lowercase` | first-scalar | | +| `Uppercase` | first-scalar | | +| `Lowercase_Mapping` | all-scalars | | +| `Titlecase_Mapping` | all-scalars | | +| `Uppercase_Mapping` | all-scalars | | +| `Soft_Dotted` | first-scalar | | +| `Cased` | any-scalar | | +| `Case_Ignorable` | all-scalars | | +| `Changes_When_Lowercased` | all-scalars | | +| `Changes_When_Uppercased` | all-scalars | | +| `Changes_When_Titlecased` | all-scalars | | +| `Changes_When_Casefolded` | all-scalars | | +| `Changes_When_Casemapped` | all-scalars | | +|
**Normalization** | | | +| `Canonical_Combining_Class` | single-scalar | | +| `Full_Composition_Exclusion` | single-scalar | | +| `Changes_When_NFKC_Casefolded` | all-scalars | | +|
**Emoji** | | | +| `Emoji` | first-scalar | | +| `Emoji_Presentation` | any-scalar | | +| `Emoji_Modifier` | single-scalar | | +| `Emoji_Modifier_Base` | single-scalar | | +|
**Shaping and Rendering** | | | +| `Join_Control` | single-scalar | | +|
**Bidirectional** | | | +| `Bidi_Control` | single-scalar | | +| `Bidi_Mirrored` | first-scalar | | +|
**Miscellaneous** | | | +| `Math` | first-scalar | | +| `Quotation_Mark` | first-scalar | | +| `Dash` | first-scalar | | +| `Sentence_Terminal` | first-scalar | | +| `Terminal_Punctuation` | first-scalar | | +| `Diacritic` | single-scalar | | +| `Extender` | single-scalar | | +| `Grapheme_Base` | single-scalar | | +| `Grapheme_Extend` | single-scalar | | + +
+ +To invert a Unicode property character class, use `\P{...}`. + +When using `RegexBuilder` syntax, Unicode property classes are available through the following methods on `CharacterClass`: + +- `static func generalCategory(_: Unicode.GeneralCategory) -> CharacterClass` +- `static func binaryProperty(_: KeyPath, value: Bool = true) -> CharacterClass` +- `static func named(_: String) -> CharacterClass` +- `static func age(_: Unicode.Version) -> CharacterClass` +- `static func numericType(_: Unicode.NumericType) -> CharacterClass` +- `static func numericValue(_: Double) -> CharacterClass` +- `static func lowercaseMapping(_: String) -> CharacterClass` +- `static func uppercaseMapping(_: String) -> CharacterClass` +- `static func titlecaseMapping(_: String) -> CharacterClass` +- `static func canonicalCombiningClass(_: Unicode.CanonicalCombiningClass) -> CharacterClass` + +You can see the full `CharacterClass` API with documentation comments in the **Custom Classes** section, below. + +#### POSIX character classes: `[:NAME:]` or `\p{NAME}` + +**POSIX character classes** represent concepts that we'd like to define at all semantic levels. We propose the following definitions, some of which have been described above. When matching with grapheme cluster semantics, Unicode properties are extended to `Character`s as described in the rationale above, and as shown in the table below. That is, for POSIX class `[:word:]`, any `Character` that starts with a matching scalar is a match, while for `[:digit:]`, a matching `Character` must only comprise a single Unicode scalar value. + +| POSIX class | Unicode property class | Character behavior | ASCII mode value | +|--------------|-----------------------------------|----------------------|-------------------------------| +| `[:lower:]` | `\p{Lowercase}` | starts-with | `[a-z]` | +| `[:upper:]` | `\p{Uppercase}` | starts-with | `[A-Z]` | +| `[:alpha:]` | `\p{Alphabetic}` | starts-with | `[A-Za-z]` | +| `[:alnum:]` | `[\p{Alphabetic}\p{Decimal}]` | starts-with | `[A-Za-z0-9]` | +| `[:word:]` | See \* below | starts-with | `[[:alnum:]_]` | +| `[:digit:]` | `\p{DecimalNumber}` | single-scalar | `[0-9]` | +| `[:xdigit:]` | `\p{Hex_Digit}` | single-scalar | `[0-9A-Fa-f]` | +| `[:punct:]` | `\p{Punctuation}` | starts-with | `[-!"#%&'()*,./:;?@[\\\]{}]` | +| `[:blank:]` | `[\p{Space_Separator}\u{09}]` | starts-with | `[ \t]` | +| `[:space:]` | `\p{Whitespace}` | starts-with | `[ \t\n\r\f\v]` | +| `[:cntrl:]` | `\p{Control}` | starts-with | `[\x00-\x1f\x7f]` | +| `[:graph:]` | See \*\* below | starts-with | `[^ [:cntrl:]]` | +| `[:print:]` | `[[:graph:][:blank:]--[:cntrl:]]` | starts-with | `[[:graph:] ]` | + +\* The Unicode scalar property definition for `[:word:]` is `[\p{Alphanumeric}\p{Mark}\p{Join_Control}\p{Connector_Punctuation}]`. +\*\* The Unicode scalar property definition for `[:cntrl:]` is `[^\p{Space}\p{Control}\p{Surrogate}\p{Unassigned}]`. + +#### Custom classes + +Custom classes function as the set union of their individual components, whether those parts are individual characters, individual Unicode scalar values, ranges, Unicode property classes or POSIX classes, or other custom classes. + +- Individual characters and scalars will be tested using the same behavior as if they were listed in an alternation. That is, a custom character class like `[abc]` is equivalent to `(a|b|c)` under the same options and modes. +- Metacharacters that represent built-in character classes keep their same function inside custom character classes. For example, in `[abc\d]+`, the `\d` matches any digit, so the regex matches the entirety of the string `"0a1b2c3"`, and `[\t\R]` matches a tab or any newline character or newline sequence. +- Metacharacters that represent zero-width assertions have their literal meaning in custom character classes, if one exists. For example, `[\b^]` matches either the BEL control character or a literal carat (`^`), while `\B` is an invalid member of a custom character class. + +Ranges in a custom character class require special consideration to avoid unexpected or dangerous results. Using simple lexicographical ordering for comparison is unintuitive when working with multi-scalar characters. For example, +the custom character class `[0-9]` is intended to match only the ten ASCII digits, but because of lexicographical ordering, complex characters like `"3̠̄"` and `"5️⃣"` would fall into that range. Ranges in custom character classes therefore having the following requirements: + +- Range endpoints must be single Unicode scalar values. When parsing a regex, endpoints will be converted to their canonical composed form, so that characters that have a multi-Unicode scalar form in source but a single-scalar canonical representation will still be permitted. +- When matching with grapheme cluster semantics, only single-scalar characters will match a range. The same conversion to canonical composed form will be used to support the expectation of matching with canonical equivalence. + +```swift +let allDigits = /^[0-9]+$/ +"1230".contains(allDigits) // true +"123̠̄0".contains(allDigits) // false +"5️⃣".contains(allDigits) // false + +let cafeExtended = /Caf[à-ÿ]/ +"Café".contains(cafeExtended) // true +"Cafe\u{301}".contains(cafeExtended) // true +``` + +Inside regexes, custom classes are enclosed in square brackets `[...]`, and can be nested or combined using set operators like `&&`. For more detail, see the [Run-time Regex Construction proposal][internals-charclass]. + +With `RegexBuilder`'s `CharacterClass` type, you can use built-in character classes with ranges and groups of characters. For example, to parse a valid octodecimal number, you could define a custom character class that combines `.digit` with a range of characters. + +```swift +let octoDecimalRegex: Regex<(Substring, Int?)> = Regex { + let charClass = CharacterClass(.digit, "a"..."h").ignoresCase() + Capture { + OneOrMore(charClass) + } transform: { Int($0, radix: 18) } +} +``` + +The full `CharacterClass` API is as follows: + +```swift +/// A class of characters that match in a regex. +/// +/// A character class can represent individual characters, a group of +/// characters, the set of character that match some set of criteria, or +/// a set algebraic combination of all of the above. +public struct CharacterClass: RegexComponent { + public var regex: Regex { get } + + /// A character class that matches any character that does not match this + /// character class. + public var inverted: CharacterClass { get } +} + +// MARK: Built-in character classes + +extension RegexComponent where Self == CharacterClass { + /// A character class that matches any element. + /// + /// This character class is unaffected by the `dotMatchesNewlines()` method. + /// To match any character that isn't a newline, see + /// ``CharacterClass.anyNonNewline``. + /// + /// This character class is equivalent to the regex syntax "dot" + /// metacharacter in single-line mode: `(?s:.)`. + public static var any: CharacterClass { get } + + /// A character class that matches any element that isn't a newline. + /// + /// This character class is unaffected by the `dotMatchesNewlines()` method. + /// To match any character, including newlines, see ``CharacterClass.any``. + /// + /// This character class is equivalent to the regex syntax "dot" + /// metacharacter with single-line mode disabled: `(?-s:.)`. + public static var anyNonNewline: CharacterClass { get } + + /// A character class that matches any single `Character`, or extended + /// grapheme cluster, regardless of the current semantic level. + /// + /// This character class is equivalent to `\X` in regex syntax. + public static var anyGraphemeCluster: CharacterClass { get } + + /// A character class that matches any digit. + /// + /// This character class is equivalent to `\d` in regex syntax. + public static var digit: CharacterClass { get } + + /// A character class that matches any hexadecimal digit. + public static var hexDigit: CharacterClass { get } + + /// A character class that matches any element that is a "word character". + /// + /// This character class is equivalent to `\w` in regex syntax. + public static var word: CharacterClass { get } + + /// A character class that matches any element that is classified as + /// whitespace. + /// + /// This character class is equivalent to `\s` in regex syntax. + public static var whitespace: CharacterClass { get } + + /// A character class that matches any element that is classified as + /// horizontal whitespace. + /// + /// This character class is equivalent to `\h` in regex syntax. + public static var horizontalWhitespace: CharacterClass { get } + + /// A character class that matches any element that is classified as + /// vertical whitespace. + /// + /// This character class is equivalent to `\v` in regex syntax. + public static var verticalWhitespace: CharacterClass { get } + + /// A character class that matches any newline sequence. + /// + /// This character class is equivalent to `\R` or `\n` in regex syntax. + public static var newlineSequence: CharacterClass { get } +} + +// MARK: anyOf(_:) / noneOf(_:) + +extension RegexComponent where Self == CharacterClass { + /// Returns a character class that matches any character in the given string + /// or sequence. + /// + /// Calling this method with a group of characters is equivalent to listing + /// those characters in a custom character class in regex syntax. For example, + /// the two regexes in this example are equivalent: + /// + /// let regex1 = /[abcd]+/ + /// let regex2 = OneOrMore(.anyOf("abcd")) + public static func anyOf(_ s: S) -> CharacterClass + where S.Element == Character + + /// Returns a character class that matches any Unicode scalar in the given + /// sequence. + /// + /// Calling this method with a group of Unicode scalars is equivalent to + /// listing them in a custom character class in regex syntax. + public static func anyOf(_ s: S) -> CharacterClass + where S.Element == UnicodeScalar + + /// Returns a character class that matches none of the characters in the given + /// string or sequence. + /// + /// Calling this method with a group of characters is equivalent to listing + /// those characters in a negated custom character class in regex syntax. For + /// example, the two regexes in this example are equivalent: + /// + /// let regex1 = /[^abcd]+/ + /// let regex2 = OneOrMore(.noneOf("abcd")) + public static func noneOf(_ s: S) -> CharacterClass + where S.Element == Character + + /// Returns a character class that matches none of the Unicode scalars in the + /// given sequence. + /// + /// Calling this method with a group of Unicode scalars is equivalent to + /// listing them in a negated custom character class in regex syntax. + public static func noneOf(_ s: S) -> CharacterClass + where S.Element == UnicodeScalar +} + +// MARK: Unicode properties + +extension CharacterClass { + /// Returns a character class that matches any element with the given Unicode + /// general category. + /// + /// For example, when passed `.uppercaseLetter`, this method is equivalent to + /// `/\p{Uppercase_Letter}/` or `/\p{Lu}/`. + public static func generalCategory(_ category: Unicode.GeneralCategory) -> CharacterClass + + /// Returns a character class that matches any element with the given Unicode + /// binary property. + /// + /// For example, when passed `\.isAlphabetic`, this method is equivalent to + /// `/\p{Alphabetic}/` or `/\p{Is_Alphabetic=true}/`. + public static func binaryProperty( + _ property: KeyPath, + value: Bool = true + ) -> CharacterClass + + /// Returns a character class that matches any element with the given Unicode + /// name. + /// + /// This method is equivalent to `/\p{Name=name}/`. + public static func name(_ name: String) -> CharacterClass + + /// Returns a character class that matches any element that was included in + /// the specified Unicode version. + /// + /// This method is equivalent to `/\p{Age=version}/`. + public static func age(_ version: Unicode.Version) -> CharacterClass + + /// Returns a character class that matches any element with the given Unicode + /// numeric type. + /// + /// This method is equivalent to `/\p{Numeric_Type=type}/`. + public static func numericType(_ type: Unicode.NumericType) -> CharacterClass + + /// Returns a character class that matches any element with the given numeric + /// value. + /// + /// This method is equivalent to `/\p{Numeric_Value=value}/`. + public static func numericValue(_ value: Double) -> CharacterClass + + /// Returns a character class that matches any element with the given Unicode + /// canonical combining class. + /// + /// This method is equivalent to + /// `/\p{Canonical_Combining_Class=combiningClass}/`. + public static func canonicalCombiningClass( + _ combiningClass: Unicode.CanonicalCombiningClass + ) -> CharacterClass + + /// Returns a character class that matches any element with the given + /// lowercase mapping. + /// + /// This method is equivalent to `/\p{Lowercase_Mapping=value}/`. + public static func lowercaseMapping(_ value: String) -> CharacterClass + + /// Returns a character class that matches any element with the given + /// uppercase mapping. + /// + /// This method is equivalent to `/\p{Uppercase_Mapping=value}/`. + public static func uppercaseMapping(_ value: String) -> CharacterClass + + /// Returns a character class that matches any element with the given + /// titlecase mapping. + /// + /// This method is equivalent to `/\p{Titlecase_Mapping=value}/`. + public static func titlecaseMapping(_ value: String) -> CharacterClass +} + +// MARK: Set algebra methods + +extension CharacterClass { + /// Returns a character class that combines the characters classes in the + /// given sequence or collection via union. + public init(_ characterClasses: some Sequence) + + /// Creates a character class that combines the given classes in a union. + public init(_ first: CharacterClass, _ rest: CharacterClass...) + + /// Returns a character class from the union of this class and the given class. + public func union(_ other: CharacterClass) -> CharacterClass + + /// Returns a character class from the intersection of this class and the given class. + public func intersection(_ other: CharacterClass) -> CharacterClass + + /// Returns a character class by subtracting the given class from this class. + public func subtracting(_ other: CharacterClass) -> CharacterClass + + /// Returns a character class matching elements in one or the other, but not both, + /// of this class and the given class. + public func symmetricDifference(_ other: CharacterClass) -> CharacterClass +} + +// MARK: Range syntax + +public func ...(lhs: Character, rhs: Character) -> CharacterClass + +@_disfavoredOverload +public func ...(lhs: UnicodeScalar, rhs: UnicodeScalar) -> CharacterClass +``` + +## Source compatibility + +Everything in this proposal is additive, and has no compatibility effect on existing source code. + +## Effect on ABI stability + +Everything in this proposal is additive, and has no effect on existing stable ABI. + +## Effect on API resilience + +N/A + +## Future directions + +### Expanded options and modifiers + +The initial version of `Regex` includes only the options described above. Filling out the remainder of options described in the [Run-time Regex Construction proposal][internals] could be completed as future work, as well as additional improvements, such as adding an option that makes a regex match only at the start of a string. + +### Extensions to Character and Unicode Scalar APIs + +An earlier version of this pitch described adding standard library APIs to `Character` and `UnicodeScalar` for each of the supported character classes, as well as convenient static members for control characters. In addition, regex literals support Unicode property features that don’t currently exist in the standard library, such as a scalar’s script or extended category, or creating a scalar by its Unicode name instead of its scalar value. These kinds of additions have value outside of just their relationship to the `Regex` additions, so they can be pitched and considered in a future proposal. + +### Byte semantic mode + +A future `Regex` version could support a byte-level semantic mode in addition to grapheme cluster and Unicode scalar semantics. Byte-level semantics would allow matching individual bytes, potentially providing the capability of parsing string and non-string data together. + +### More general `CharacterSet` replacement + +Foundation's `CharacterSet` type is in some ways similar to the `CharacterClass` type defined in this proposal. `CharacterSet` is primarily a set type that is defined over Unicode scalars, and can therefore sometimes be awkward to use in conjunction with Swift `String`s. The proposed `CharacterClass` type is a `RegexBuilder`-specific type, and as such isn't intended to be a full general purpose replacement. Future work could involve expanding upon the `CharacterClass` API or introducing a different type to fill that role. + +## Alternatives considered + +### Operate on String.UnicodeScalarView instead of using semantic modes + +Instead of providing APIs to select whether `Regex` matching is `Character`-based vs. `UnicodeScalar`-based, we could instead provide methods to match against the different views of a string. This different approach has multiple drawbacks: + +* As the scalar level used when matching changes the behavior of individual components of a `Regex`, it’s more appropriate to specify the semantic level at the declaration site than the call site. +* With the proposed options model, you can define a Regex that includes different semantic levels for different portions of the match, which would be impossible with a call site-based approach. + +### Binary word boundary option method + +A prior version of this proposal used a binary method for setting the word boundary algorithm, called `usingSimpleWordBoundaries()`. A method taking a `RegexWordBoundaryKind` instance is included in the proposal instead, to leave room for implementing other word boundary algorithms in the future. + +### More "Swifty" default option settings + +Swift's `Regex` includes some default behaviors that don't match other regex engines — in particular, matching characters with `.` and using Unicode's default word boundary algorithm. For other option-based behaviors, `Regex` adheres to the general standard set by other regular expression engines, like having `.` not match newlines and `^` and `$` only match the start and end of the input instead of the beginning and end of each line. This is to ease the process of bringing existing regular expressions and existing knowledge into Swift. + +Instead, we could use this opportunity to choose default options that are more ergonomic or intuitive, and provide a `compatibilityOptions()` API that reverts back to the typical settings, including matching based on Unicode scalars instead of characters. This method could additionally be a point of documentation for Swift's choices of default behaviors. + +### Include `\O` and `CharacterClass.anyUnicodeScalar` + +An earlier draft of this proposal included a metacharacter and `CharacterClass` API for matching an individual Unicode scalar value, regardless of the current matching level, as a counterpart to `\X`/`.anyGraphemeCluster`. The behavior of this character class, particularly when matching with grapheme cluster semantics, is still unclear at this time, however. For example, when matching the expression `\O*`, does the implicit grapheme boundary assertion apply between the `\O` and the quantification operator, or should we treat the two as a single unit and apply the assertion after the `*`? + +At the present time, we prefer to allow authors to write regexes that explicitly shift into and out of Unicode scalar mode, where those kinds of decisions are handled by the explicit scope of the setting. If common patterns emerge that indicate some version of `\O` would be useful, we can add it in the future. + +## Future Work + +### Additional protocol to limit option methods + +The option-setting methods, like `ignoresCase()`, are implemented as extensions of the `Regex` type instead of on the `RegexComponent` protocol. This makes sure that nonsensical formulations like `"abc".defaultRepetitionBehavior(.possessive)"` are impossible to write, but is somewhat inconvenient when working with `RegexBuilder` syntax, as you need to add an additional `Regex { ... }` block around a quantifier or other grouping scope that you want to have a particular behavior. + +One possible future addition would be to add another protocol that refines `RegexComponent`, with a name like `RegexCompoundComponent`, representing types that can hold or more other regex components. Types like `OneOrMore`, `CharacterClass`, and `Regex` itself would all conform, and the option-setting methods would move to an extension on that new protocol, permitting more convenient usage where appropriate. + +### API for current options + +As we gather information about how regexes are used and extended, we may find it useful to query an existing regex instance for the set of options that are present globally, or at the start of the regex. Likewise, if `RegexBuilder` gains the ability to use a predicate or other call out to other code, that may require providing the current set of options at the time of execution. + +Such an options type could have a simple read-write, property accessor interface: + +```swift +/// A set of options that affect matching behavior and semantics. +struct RegexOptions { + /// A Boolean value indicating whether casing is ignored while matching. + var ignoresCase: Bool + /// An option set representing any character classes that are matched as ASCII-only. + var asciiOnlyClasses: RegexCharacterClassKind + /// The current matching semantics. + var matchingSemantics: RegexMatchingSemantics + // etc... +} +``` + +### Regex syntax for matching level + +An earlier draft of this proposal included options within the regex syntax that are equivalent to calling the `matchingSemantics(_:)` method: `(?X)` for switching to grapheme cluster more and `(?u)` for switching to Unicode scalar mode. As these are new additions to regex syntax, and their exclusive behavior has yet to be determined, they are not included in the proposed functionality at this time. + +### API for overriding Unicode property mapping + +We could add API in the future to change how individual Unicode scalar properties are extended to characters. One such approach could be to provide a modifier method that takes a key path and a strategy: + +```swift +// Matches only the character "a" +let regex1 = /\p{name=latin lowercase a}/ + +// Matches any character with "a" as its first scalar +let regex1 = /\p{name=latin lowercase a}/.extendUnicodeProperty(\.name, by: .firstScalar)`. +``` + +[repo]: https://github.com/apple/swift-experimental-string-processing/ +[option-scoping]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#matching-options +[internals]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md +[internals-properties]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#character-properties +[internals-charclass]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#custom-character-classes +[level1-word-boundaries]:https://unicode.org/reports/tr18/#Simple_Word_Boundaries +[level2-word-boundaries]:https://unicode.org/reports/tr18/#RL2.3 + +[overview]: https://forums.swift.org/t/declarative-string-processing-overview/52459 +[charprops]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0221-character-properties.md +[regexbuilder]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0351-regex-builder.md +[charpropsrationale]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0221-character-properties.md#detailed-semantics-and-rationale +[canoneq]: https://www.unicode.org/reports/tr15/#Canon_Compat_Equivalence +[graphemes]: https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries +[meaningless]: https://forums.swift.org/t/declarative-string-processing-overview/52459/121 +[scalarprops]: https://github.com/swiftlang/swift-evolution/blob/master/proposals/0211-unicode-scalar-properties.md +[ucd]: https://www.unicode.org/reports/tr44/tr44-28.html +[numerictype]: https://www.unicode.org/reports/tr44/#Numeric_Type +[derivednumeric]: https://www.unicode.org/Public/UCD/latest/ucd/extracted/DerivedNumericType.txt + + +[uts18]: https://unicode.org/reports/tr18/ +[proplist]: https://www.unicode.org/Public/UCD/latest/ucd/PropList.txt +[pcre]: https://www.pcre.org/current/doc/html/pcre2pattern.html +[perl]: https://perldoc.perl.org/perlre +[raku]: https://docs.raku.org/language/regexes +[rust]: https://docs.rs/regex/1.5.4/regex/ +[regexbytes]: https://docs.rs/regex/1.5.4/regex/bytes/ +[python]: https://docs.python.org/3/library/re.html +[ruby]: https://ruby-doc.org/core-2.4.0/Regexp.html +[csharp]: https://docs.microsoft.com/en-us/dotnet/standard/base-types/regular-expression-language-quick-reference +[icu]: https://unicode-org.github.io/icu/userguide/strings/regexp.html +[posix]: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html +[oniguruma]: https://www.cuminas.jp/sdk/regularExpression.html +[go]: https://pkg.go.dev/regexp/syntax@go1.17.2 +[cplusplus]: https://www.cplusplus.com/reference/regex/ECMAScript/ +[ecmascript]: https://262.ecma-international.org/12.0/#sec-pattern-semantics +[re2]: https://github.com/google/re2/wiki/Syntax +[java]: https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html diff --git a/proposals/0364-retroactive-conformance-warning.md b/proposals/0364-retroactive-conformance-warning.md new file mode 100644 index 0000000000..6300d01b9e --- /dev/null +++ b/proposals/0364-retroactive-conformance-warning.md @@ -0,0 +1,150 @@ +# Warning for Retroactive Conformances of External Types + +* Proposal: [SE-0364](0364-retroactive-conformance-warning.md) +* Author: [Harlan Haskins](https://github.com/harlanhaskins) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.0)** +* Review: ([first pitch](https://forums.swift.org/t/warning-for-retroactive-conformances-if-library-evolution-is-enabled/45321)) + ([second pitch](https://forums.swift.org/t/pitch-warning-for-retroactive-conformances-of-external-types-in-resilient-libraries/56243)) + ([first review](https://forums.swift.org/t/se-0364-warning-for-retroactive-conformances-of-external-types/58922)) + ([second review](https://forums.swift.org/t/second-review-se-0364-warning-for-retroactive-conformances-of-external-types/64615)) + ([acceptance](https://forums.swift.org/t/accepted-se-0364-warning-for-retroactive-conformances-of-external-types/65015)) + ([amendment](https://forums.swift.org/t/amendment-se-0364-allow-same-package-conformances/71877)) + ([acceptance](https://forums.swift.org/t/accepted-amendment-se-0364-allow-same-package-conformances/72880)) + +## Introduction + +Many Swift libraries vend currency protocols, like Equatable, Hashable, Codable, +among others, that unlock worlds of common functionality for types that conform +to them. Sometimes, if a type from another module does not conform to a common +currency protocols, developers will declare a conformance of that type to that +protocol within their module. However, protocol conformances are globally unique +within a process in the Swift runtime, and if multiple modules declare the same +conformance, it can cause major problems for library clients and hinder the +ability to evolve libraries over time. + +## Motivation + +Consider a library that, for one of its core APIs, declares a conformance of +`Date` to `Identifiable`, in order to use it with an API that diffs elements +of a collection by their identity. + +```swift +// Not a great implementation, but I suppose it could be useful. +extension Date: Identifiable { + public var id: TimeInterval { timeIntervalSince1970 } +} +``` + +Now that this client has declared this conformance, if Foundation decides to +add this conformance in a later revision, this client will fail to build. +Before the client removes their conformance and rebuilds, however, their +application will exhibit undefined behavior, as it is indeterminate which +definition of this conformance will "win". Foundation may well have defined +it to use `Date.timeIntervalSinceReferenceDate`, and if the client had persisted +these IDs to a database or some persistent storage beyond the lifetime of the process, +then their dates will have completely different IDs. + +Worse, if this is a library target, this conformance propagates down to every +client that imports the library. This is especially bad for frameworks that +are built with library evolution enabled, as their clients link against +binary frameworks and usually are not aware these conformances don't come from +the actual owning module. + +## Proposed solution + +This proposal adds a warning that explicitly calls out this pattern as +problematic and unsupported. + +```swift +/tmp/retro.swift:3:1: warning: extension declares a conformance of imported type +'Date' to imported protocol 'Identifiable'; this will not behave correctly if +the owners of 'Foundation' introduce this conformance in the future +extension Date: Identifiable { +^ +``` + +If absolutely necessary, clients can silence this warning by adding a new attribute, +`@retroactive`, to the protocol in question. + +The compiler will enforce that there is an explicit `@retroactive` conformance +for each protocol included up the hierarchy. If needed, it will emit a fix-it to +generate extensions for each retroactive conformance in the hierarchy. + +```swift +extension Date: @retroactive Identifiable { + // ... +} +``` + +## Detailed design + +This warning will appear only if all of the following conditions are met, with a few exceptions. + +- The type being extended was declared in a different module from the extension. +- The protocol for which the extension introduces the conformance is declared in a different + module from the extension. + +The following exceptions apply to either the conforming type or the protocol: + +- If it is declared in a Clang module, and the extension in question is declared + in a Swift overlay of that module, this is not considered a retroactive conformance. +- If it is declared or transitively imported in a bridging header or through the + `-import-objc-header` flag, and the type does not belong to any other module, the warning is not + emitted. This could be a retroactive conformance, but since these are added to an implicit module + called `__ObjC`, we have to assume the client takes responsibility for these declaration. +- If it is declared in one module, but uses the `@_originallyDefined(in:)` attribute to + signify that it has moved from a different module, then this will not warn. +- If it is declared in a module that is part of the same package as the conformance, + this is not retroactive. Duplicated same-package conformances will be detected at link or load + time. + +For clarification, the following are still valid, safe, and allowed: +- Conformances of external types to protocols defined within the current module. +- Extensions of external types that do not introduce a conformance. These do not introduce runtime conflicts, since the + module name is mangled into the symbol. + +The `@retroactive` attribute may only be used in extensions, and only when used +to introduce a conformance that requires its existence. It will be an error to +use `@retroactive` outside of the declaration of a retroactive conformance. + +## Source compatibility + +`@retroactive` is a new attribute, but it is purely additive; it can be accepted +by all language versions. It does mean projects building with an older Swift +will not have access to this syntax, so as a source compatible fallback, +a client can silence this warning by fully-qualifying all types in the extension. +As an example, the above conformance can also be written as + +```swift +extension Foundation.Date: Swift.Identifiable { + // ... +} +``` + +This will allow projects that need to build with multiple versions of Swift, and +which have valid reason to declare such conformances, to declare them without +tying their project to a newer compiler build. + +## Effect on ABI stability + +This proposal has no effect on ABI stability. + +## Effect on API resilience + +This proposal has no direct effect on API resilience, but has the indirect effect of reducing +the possible surface of client changes introduced by the standard library adding new conformances. + +## Alternatives considered + +#### Enabling this warning only for resilient libraries + +A previous version of this proposal proposed enabling this warning only for resilient libraries, as those +are meant to be widely distributed and such a conformance is much more difficult to remove from clients +who expect ABI stability. However, review feedback showed a clear preference to enable this warning always, +to give library authors more freedom to introduce conformances. + +#### Putting it behind a flag + +This warning could very well be enabled by a flag, but there's not much +precedent in Swift for flags to disable individual warnings. diff --git a/proposals/0365-implicit-self-weak-capture.md b/proposals/0365-implicit-self-weak-capture.md new file mode 100644 index 0000000000..bb86d69bcb --- /dev/null +++ b/proposals/0365-implicit-self-weak-capture.md @@ -0,0 +1,231 @@ +# Allow implicit `self` for `weak self` captures, after `self` is unwrapped + +* Proposal: [SE-0365](0365-implicit-self-weak-capture.md) +* Author: [Cal Stephens](https://github.com/calda) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#40702](https://github.com/apple/swift/pull/40702), [apple/swift#61520](https://github.com/apple/swift/pull/61520) + +## Introduction + +As of [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md), implicit `self` is permitted in closures when `self` is written explicitly in the capture list. We should extend this support to `weak self` captures, and permit implicit `self` as long as `self` has been unwrapped. + +```swift +class ViewController { + let button: Button + + func setup() { + button.tapHandler = { [weak self] in + guard let self else { return } + dismiss() + } + } + + func dismiss() { ... } +} +``` + +Swift-evolution thread: [Allow implicit `self` for `weak self` captures, after `self` is unwrapped](https://forums.swift.org/t/allow-implicit-self-for-weak-self-captures-after-self-is-unwrapped/54262) + +## Motivation + +Explicit `self` has historically been required in closures, in order to help prevent users from inadvertently creating retain cycles. [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md) relaxed these rules in cases where implicit `self` is unlikely to introduce a hidden retain cycle, such as when `self` is explicitly captured in the closure's capture list: + +```swift +button.tapHandler = { [self] in + dismiss() +} +``` + +SE-0269 left the handling of `weak self` captures as a future direction, so explicit `self` is currently required in this case: + +```swift +button.tapHandler = { [weak self] in + guard let self else { return } + self.dismiss() +} +``` + +Since `self` has already been captured explicitly, there is limited value in requiring authors to use explicit `self`. This is inconsistent, and adds unnecessary visual noise to the body of closures using `weak self` captures. + +## Proposed solution + +We should permit implicit `self` for `weak self` captures, once `self` has been unwrapped. + +This code would now be allowed to compile: + +```swift +class ViewController { + let button: Button + + func setup() { + button.tapHandler = { [weak self] in + guard let self else { return } + dismiss() + } + } + + func dismiss() { ... } +} +``` + +## Detailed design + +### Enabling implicit `self` + +All of the following forms of optional unwrapping are supported, and enable implicit self for the following scope where `self` is non-optional: + +```swift +button.tapHandler = { [weak self] in + guard let self else { return } + dismiss() +} + +button.tapHandler = { [weak self] in + guard let self = self else { return } + dismiss() +} + +button.tapHandler = { [weak self] in + if let self { + dismiss() + } +} + +button.tapHandler = { [weak self] in + if let self = self { + dismiss() + } +} + +button.tapHandler = { [weak self] in + while let self { + dismiss() + } +} + +button.tapHandler = { [weak self] in + while let self = self { + dismiss() + } +} +``` + +Like with implicit `self` for `strong` and `unowned` captures, the compiler will synthesize an implicit `self.` for calls to properties / methods on `self` inside a closure that uses `weak self`. + +If `self` has not been unwrapped yet, the following error will be emitted: + +```swift +button.tapHandler = { [weak self] in + // error: explicit use of 'self' is required when 'self' is optional, + // to make control flow explicit + // fix-it: reference 'self?.' explicitly + dismiss() +} +``` + +### Nested closures + +Nested closures can be a source of subtle retain cycles, so have to be handled more carefully. For example, if the following code was allowed to compile, then the implicit `self.bar()` call could introduce a hidden retain cycle: + +```swift +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { + bar() + } +} +``` + +Following the precedent of [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md), additional closures nested inside the `[weak self]` closure must capture `self` explicitly in order to use implicit `self`. + +```swift +// Not allowed: +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { + // error: call to method 'method' in closure requires + // explicit use of 'self' to make capture semantics explicit + bar() + } +} + +// Allowed: +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { [weak self] in + guard let self else { return } + bar() + } +} + +// Also allowed: +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { + self.bar() + } +} + +// Also allowed: +couldCauseRetainCycle { [weak self] in + guard let self else { return } + foo() + + couldCauseRetainCycle { [self] in + bar() + } +} +``` + +Also following [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md), implicit `self` will only be permitted if the `self` optional binding specifically, and exclusively, refers the closure's `self` capture: + +```swift +button.tapHandler = { [weak self] in + guard let self = self ?? someOptionalWithSameTypeOfSelf else { return } + + // error: call to method 'method' in closure requires explicit use of 'self' + // to make capture semantics explicit + method() +} +``` + +## Source compatibility + +This change is purely additive and does not break source compatibility of any valid existing Swift code. + +## Effect on ABI stability + +This change is purely additive, and is a syntactic transformation to existing valid code, so has no effect on ABI stability. + +## Effect on API resilience + +This change is purely additive, and is a syntactic transformation to existing valid code, so has no effect on API resilience. + +## Alternatives considered + +It is technically possible to also support implicit `self` _before_ `self` has been unwrapped, like: + +```swift +button.tapHandler = { [weak self] in + dismiss() // as in `self?.dismiss()` +} +``` + +That would effectively add implicit control flow, however. `dismiss()` would only be executed when `self` is not nil, without any indication that it may not run. We could create a new way to spell this that still implies optional chaining, like `?.dismiss()`, but that is not meaningfully better than the existing `self?.dismiss()` spelling. + +## Acknowledgments + +Thanks to the authors of [SE-0269](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0269-implicit-self-explicit-capture.md) for laying the foundation for this proposal. + +Thanks to Kyle Sluder for [the suggestion](https://forums.swift.org/t/allow-implicit-self-for-weak-self-captures-after-self-is-unwrapped/54262/2) to not permit implicit `self` in cases where the unwrapped `self` value doesn't necessarily refer to the closure's `self` capture, like in `let self = self ?? C.global`. + +Many thanks to Pavel Yaskevich, John McCall, and Xiaodi Wu for providing significant feedback and advice regarding the implementation of this proposal. \ No newline at end of file diff --git a/proposals/0366-move-function.md b/proposals/0366-move-function.md new file mode 100644 index 0000000000..fa899f24c7 --- /dev/null +++ b/proposals/0366-move-function.md @@ -0,0 +1,758 @@ +# `consume` operator to end the lifetime of a variable binding + +* Proposal: [SE-0366](0366-move-function.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm), [Andrew Trick](https://github.com/atrick), [Joe Groff](https://github.com/jckarter) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.9)** +* Implementation: in main branch of compiler +* Review: ([pitch](https://forums.swift.org/t/pitch-move-function-use-after-move-diagnostic/53983)) ([first review](https://forums.swift.org/t/se-0366-move-function-use-after-move-diagnostic/59202)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0366-move-operation-use-after-move-diagnostic/59687)) ([second review](https://forums.swift.org/t/se-0366-second-review-take-operator-to-end-the-lifetime-of-a-variable-binding/61021)) ([third review](https://forums.swift.org/t/combined-se-0366-third-review-and-se-0377-second-review-rename-take-taking-to-consume-consuming/61904)), ([acceptance](https://forums.swift.org/t/accepted-se-0366-consume-operator-to-end-the-lifetime-of-a-variable-binding/62758)) +* Previous Revisions: + * [1](https://github.com/swiftlang/swift-evolution/blob/567fb1a66c784bcc5394491d24f72a3cb393674f/proposals/0366-move-function.md) + * [2](https://github.com/swiftlang/swift-evolution/blob/43849aa9ae3e87c434866c5a5e389af67537ca26/proposals/0366-move-function.md) + * [3](https://github.com/swiftlang/swift-evolution/blob/7af91127d693bffcb01aa87978d75d5a3170c4d1/proposals/0366-move-function.md) + +## Introduction + +In this document, we propose adding a new operator, marked by the +context-sensitive keyword `consume`, to the +language. `consume` ends the lifetime of a specific local `let`, +local `var`, or function parameter, and enforces this +by causing the compiler to emit a diagnostic upon any use after the +consume. This allows for code that relies on **forwarding ownership** +of values for performance or correctness to communicate that requirement to +the compiler and to human readers. As an example: + +```swift +useX(x) // do some stuff with local variable x + +// Ends lifetime of x, y's lifetime begins. +let y = consume x // [1] + +useY(y) // do some stuff with local variable y +useX(x) // error, x's lifetime was ended at [1] + +// Ends lifetime of y, destroying the current value. +_ = consume y // [2] +useX(x) // error, x's lifetime was ended at [1] +useY(y) // error, y's lifetime was ended at [2] +``` + +## Motivation + +Swift uses reference counting and copy-on-write to allow developers to +write code with value semantics and not normally worry too much +about performance or memory management. However, in performance sensitive code, +developers want to be able to control the uniqueness of COW data structures and +reduce retain/release calls in a way that is future-proof against changes to +the language implementation or source code. Consider the following +example: + +```swift +func test() { + var x: [Int] = getArray() + + // x is appended to. After this point, we know that x is unique. We want to + // preserve that property. + x.append(5) + + // Pass the current value of x off to another function, that could take + // advantage of its uniqueness to efficiently mutate it further. + doStuffUniquely(with: x) + + // Reset x to a new value. Since we don't use the old value anymore, + // it could've been uniquely referenced by the callee. + x = [] + doMoreStuff(with: &x) +} + +func doStuffUniquely(with value: [Int]) { + // If we received the last remaining reference to `value`, we'd like + // to be able to efficiently update it without incurring more copies. + var newValue = value + newValue.append(42) + + process(newValue) +} +``` + +In the example above, a value is built up in the variable `x` and then +handed off to `doStuffUniquely(with:)`, which makes further modifications to +the value it receives. `x` is then set to a new value. It should be possible for +the caller to **forward ownership** of the value of `x` to `doStuffUniquely`, +since it no longer uses the value as is, to avoid unnecessary retains or +releases of the array buffer and unnecessary copy-on-writes of the array +contents. `doStuffUniquely` should in turn be able to move its parameter into +a local mutable variable and modify the unique buffer in place. The compiler +could make these optimizations automatically, but a number of analyses have to +align to get the optimal result. The programmer may want to guarantee that this +series of optimizations occurs, and receive diagnostics if they wrote the code +in a way that would interfere with these optimizations being possible. + +Swift-evolution pitch threads: + +- [https://forums.swift.org/t/pitch-move-function-use-after-move-diagnostic](https://forums.swift.org/t/pitch-move-function-use-after-move-diagnostic) +- [https://forums.swift.org/t/selective-control-of-implicit-copying-behavior-take-borrow-and-copy-operators-noimplicitcopy/60168](https://forums.swift.org/t/selective-control-of-implicit-copying-behavior-take-borrow-and-copy-operators-noimplicitcopy/60168) + +## Proposed solution: `consume` operator + +That is where the `consume` operator comes into play. `consume` consumes +the current value of a **binding with static lifetime**, which is either +an unescaped local `let`, unescaped local `var`, or function parameter, with +no property wrappers or get/set/read/modify/etc. accessors applied. It then + provides a compiler guarantee that the current value will +be unable to be used again locally. If such a use occurs, the compiler will +emit an error diagnostic. We can modify the previous example to use `consume` to +explicitly end the lifetime of `x`'s current value when we pass it off to +`doStuffUniquely(with:)`: + +```swift +func test() { + var x: [Int] = getArray() + + // x is appended to. After this point, we know that x is unique. We want to + // preserve that property. + x.append(5) + + // Pass the current value of x off to another function, that + doStuffUniquely(with: consume x) + + // Reset x to a new value. Since we don't use the old value anymore, + x = [] + doMoreStuff(with: &x) +} +``` + +The `consume x` operator syntax deliberately mirrors the +proposed [ownership modifier](https://forums.swift.org/t/borrow-and-take-parameter-ownership-modifiers/59581) +parameter syntax, `(x: consuming T)`, because the caller-side behavior of `consume` +operator is analogous to a callee’s behavior receiving a `consuming` parameter. +`doStuffUniquely(with:)` can use the `consume` operator, combined with +the `consuming` parameter modifier, to preserve the uniqueness of the parameter +as it moves it into its own local variable for mutation: + +```swift +func doStuffUniquely(with value: consuming [Int]) { + // If we received the last remaining reference to `value`, we'd like + // to be able to efficiently update it without incurring more copies. + var newValue = consume value + newValue.append(42) + + process(newValue) +} +``` + +This takes the guesswork out of the optimizations discussed above: in the +`test` function, the final value of `x` before reassignment is explicitly +handed off to `doStuffUniquely(with:)`, ensuring that the callee receives +unique ownership of the value at that time, and that the caller can't +use the old value again. Inside `doStuffUniquely(with:)`, the lifetime of the +immutable `value` parameter is ended to initialize the local variable `newValue`, +ensuring that the assignment doesn't cause a copy. +Furthermore, if a future maintainer modifies the code in a way that breaks +this transfer of ownership chain, then the compiler will raise an +error. For instance, if a maintainer later introduces an additional use of +`x` after it's consumed, but before it's reassigned, they will see an error: + +```swift +func test() { + var x: [Int] = getArray() + x.append(5) + + doStuffUniquely(with: consume x) + + // ERROR: x used after being consumed + doStuffInvalidly(with: x) + + x = [] + doMoreStuff(with: &x) +} +``` + +Likewise, if the maintainer tries to access the original `value` parameter inside +of `doStuffUniquely` after being consumed to initialize `newValue`, they will +get an error: + +```swift +func doStuffUniquely(with value: consuming [Int]) { + // If we received the last remaining reference to `value`, we'd like + // to be able to efficiently update it without incurring more copies. + var newValue = consume value + newValue.append(42) + + process(newValue) +} +``` + +`consume` can also end the lifetime of local immutable `let` bindings, which become +unavailable after their value is consumed since they cannot be reassigned. +Also note that `consume` operates on bindings, not values. If we declare a +constant `x` and another local constant `other` with the same value, +we can still use `other` after we consume the value from `x`, as in: + +```swift +func useX(_ x: SomeClassType) -> () {} + +func f() { + let x = ... + useX(x) + let other = x // other is a new binding used to extend the lifetime of x + _ = consume x // x's lifetime ends + useX(other) // other is used here... no problem. + useX(other) // other is used here... no problem. +} +``` + +We can also consume `other` independently of `x`, and get separate diagnostics for both +variables: + +```swift +func useX(_ x: SomeClassType) -> () {} + +func f() { + let x = ... + useX(x) + let other = x + _ = consume x + useX(consume other) + useX(other) // error: 'other' used after being consumed + useX(x) // error: 'x' used after being consumed +} +``` + +`inout` function parameters can also be used with `consume`. Like a `var`, an +`inout` parameter can be reassigned after being consumed from and used again; +however, since the final value of an `inout` parameter is passed back to the +caller, it *must* be reassigned by the callee before it +returns. So this will raise an error because `buffer` doesn't have a value at +the point of return: + +```swift +func f(_ buffer: inout Buffer) { // error: 'buffer' not reinitialized after consume! + let b = consume buffer // note: consume was here + b.deinitialize() + ... write code ... +} // note: return without reassigning inout argument `buffer` +``` + +But we can reinitialize `buffer` by writing the following code: + +```swift +func f(_ buffer: inout Buffer) { + let b = consume buffer + b.deinitialize() + // ... write code ... + // We re-initialized buffer before end of function so the checker is satisfied + buffer = getNewInstance() +} +``` + +`defer` can also be used to reinitialize an `inout` or `var` after a consume, +in order to ensure that reassignment happens on any exit from scope, including +thrown errors or breaks out of loops. So we can also write: + +```swift +func f(_ buffer: inout Buffer) { + let b = consume buffer + // Ensure the buffer is reinitialized before we exit. + defer { buffer = getNewInstance() } + try b.deinitializeOrError() + // ... write code ... +} +``` + +## Detailed design + +At runtime, `consume x` evaluates to the current value bound to `x`, just like the +expression `x` does. However, at compile time, the presence of a `consume` forces +ownership of the argument to be transferred out of the binding at the given +point so. Any ensuing use of the binding that's reachable from the `consume` +is an error. The operand to `consume` is required to be a reference +to a *binding with static lifetime*. The following kinds of declarations can +currently be referenced as bindings with static lifetime: + +- a local `let` constant in the immediately-enclosing function, +- a local `var` variable in the immediately-enclosing function, +- one of the immediately-enclosing function's parameters, or +- the `self` parameter in a `mutating` or `__consuming` method. + +A binding with static lifetime also must satisfy the following requirements: + +- it cannot be captured by an `@escaping` closure or nested function, +- it cannot have any property wrappers applied, +- it cannot have any accessors attached, such as `get`, `set`, + `didSet`, `willSet`, `_read`, or `_modify`, +- it cannot be an `async let`. + +Possible extensions to the set of operands that can be used with `consume` are +discussed under Future Directions. It is an error to use `consume` with an operand +that doesn't reference a binding with static lifetime. + +Given a valid operand, `consume` enforces that there are no other +references to the binding after it is consumed. The analysis is +flow sensitive, so one is able to end the lifetime of a value conditionally: + +```swift +if condition { + let y = consume x + // I can't use x anymore here! + useX(x) // !! ERROR! Use after consume. +} else { + // I can still use x here! + useX(x) // OK +} +// But I can't use x here. +useX(x) // !! ERROR! Use after consume. +``` + +If the binding is a `var`, the analysis additionally allows for code to +conditionally reinitialize the var and thus use it in positions +that are dominated by the reinitialization. Consider the +following example: + +```swift +if condition { + _ = consume x + // I can't use x anymore here! + useX(x) // !! ERROR! Use after consume. + x = newValue + // But now that I have re-assigned into x a new value, I can use the var + // again. + useX(x) // OK +} else { + // I can still use x here, since it wasn't consumed on this path! + useX(x) // OK +} +// Since I reinitialized x along the `if` branch, and it was never consumed +// from on the `else` branch, I can use it here too. +useX(x) // OK +``` + +Notice how in the above, we are able to use `x` both in the true block AND the +code after the `if` block, since over both paths through the `if`, `x` ends up +with a valid value before proceeding. + +For an `inout` parameter, the analysis behaves the same as for a `var`, except +that all exits from the function (whether by `return` or by `throw`) are +considered to be uses of the parameter. Correct code therefore *must* reassign +inout parameters after they are consumed from. + +Using `consume` on a binding without using the result raises a warning, +just like a function call that returns an unused non-`Void` result. +To "drop" a value without using it, the `consume` can be assigned to +`_` explicitly. + +## Source compatibility + +`consume` behaves as a contextual keyword. In order to avoid interfering +with existing code that calls functions named `consume`, the operand to +`consume` must begin with another identifier, and must consist of an +identifier or postfix expression: + +```swift +consume x // OK +consume [1, 2, 3] // Subscript access into property named `consume`, not a consume operation +consume (x) // Call to function `consume`, not a consume operation +consume x.y.z // Syntactically OK (although x.y.z is not currently semantically valid) +consume x[0] // Syntactically OK (although x[0] is not currently semantically valid +consume x + y // Parses as (consume x) + y +``` + +## Effect on ABI stability + +`consume` requires no ABI additions. + +## Effect on API resilience + +None, this is additive. + +## Alternatives considered + +### Alternative spellings + +The [first reviewed revision](https://github.com/swiftlang/swift-evolution/blob/567fb1a66c784bcc5394491d24f72a3cb393674f/proposals/0366-move-function.md) +of this proposal offered `move(x)` as a special +function with semantics recognized by the compiler. Based on initial feedback, +we pivoted to the contextual keyword spelling. As a function, this operation +would be rather unusual, since it only accepts certain forms of expression as +its argument, and it doesn't really have any runtime behavior of its own, +acting more as a marker for the compiler to perform additional analysis. + +The community reviewed the contextual keyword syntax, using the name `move x`, +and through further discussion the alternative name `take` arose. This name +aligns with the term used in the Swift compiler internals, and also reads well +as the analogous parameter ownership modifier, `(x: take T)`, so the authors +now favor this name. However, during review of SE-0377, reviewers found that +`take` was too generic, and could be confused with the common colloquial language +of talking about function calls as "taking their arguments". Of alternative +names reviewers offered, the language workgroup favors `consume`. + +Many have suggested alternative spellings that also make `consume`'s special +nature more syntactically distinct, including: + +- an expression attribute, like `useX(@consume x)` +- a compiler directive, like `useX(#consume x)` +- an operator, like `useX(<-x)` + +### Use of scoping to end lifetimes + +It is possible in the language today to foreshorten the lifetime of local +variables using lexical scoping: + +```swift +func test() { + var x: [Int] = getArray() + + // x is appended to. After this point, we know that x is unique. We want to + // preserve that property. + x.append(5) + + // We create a new variable y so we can write an algorithm where we may + // change the value of y (causing a COW copy of the buffer shared with x). + do { + var y = x + longAlgorithmUsing(&y) + consumeFinalY(y) + } + + // We no longer use y after this point. Ideally, x would be guaranteed + // unique so we know we can append again without copying. + x.append(7) +} +``` + +However, there are a number of reasons not to rely solely on lexical scoping +to end value lifetimes: + +- Adding lexical scopes requires nesting, which can lead to "pyramid of doom" + situations when managing the lifetimes of multiple variables. +- Value lifetimes don't necessarily need to nest, and may overlap or interleave + with control flow. This should be valid: + + ```swift + let x = foo() + let y = bar() + // end x's lifetime before y's + consume(consume x) + consume(consume y) + ``` + +- Lexical scoping cannot be used by itself to shorten the lifetime of function + parameters, which are in scope for the duration of the function body. +- Lexical scoping cannot be used to allow for taking from and reinitializing + mutable variables or `inout` parameters. + +Looking outside of Swift, the Rust programming language originally only had +strictly scoped value lifetimes, and this was a significant ergonomic +problem until "non-lexical lifetimes" were added later, which allowed for +value lifetimes to shrinkwrap to their actual duration of use. + +## Future directions + +### Dynamic enforcement of `consume` for other kinds of bindings + +In the future, we may want to accommodate the ability to dynamically +consume bindings with dynamic lifetime, such as escaped local +variables, and class stored properties, although doing so in full +generality would require dynamic enforcement in addition to static +checking, similar to how we need to dynamically enforce exclusivity +when accessing globals and class stored properties. Since this +dynamic enforcement turns misuse of `consume`s into runtime errors +rather than compile-time guarantees, we might want to make those +dynamic cases syntactically distinct, to make the possibility of +runtime errors clear. + +`Optional` and other types with a canonical "no value" or "empty" +state can use the static `consume` operator to provide API that +dynamically takes ownership of the current value inside of them +while leaving them in their empty state: + +```swift +extension Optional { + mutating func take() -> Wrapped { + switch consume self { + case .some(let x): + self = .none + return x + case .none: + fatalError("trying to consume an empty Optional") + } + } +} +``` + +### Piecewise `consume` of frozen structs and tuples + +For frozen structs and tuples, both aggregates that the compiler can statically +know the layout of, we could do finer-grained analysis and allow their +individual fields to be consumed independently: + +```swift +struct TwoStrings { + var first: String + var second: String +} + +func foo(x: TwoStrings) { + use(consume x.first) + // ERROR! part of x was consumed + use(x) + // OK, this part wasn't + use(x.second) +} +``` + +### Destructuring methods for move-only types with `deinit` + +Move-only types would allow for the possibility of value types with custom +`deinit` logic that runs at the end of a value of the type's lifetime. +Typically, this logic would run when the final owner of the value is finished +with it, which means that a function which `consume`s an instance, or a +`taking func` method on the type itself, would run the deinit if it does not +forward ownership anywhere else: + +```swift +moveonly struct FileHandle { + var fd: Int32 + + // close the fd on deinit + deinit { close(fd) } +} + +func dumpAndClose(to fh: consume FileHandle, contents: Data) { + write(fh.fd, contents) + // fh implicitly deinit-ed here, closing it +} +``` + +However, this may not always be desirable, either because the function performs +an operation that invalidates the value some other way, making it unnecessary +or incorrect for `deinit` to run on it, or because the function wants to be able +to take ownership of parts away from the value: + +```swift +extension FileHandle { + // Return the file descriptor back to the user for manual management + // and disable automatic management with the FileHandle value. + + taking func giveUp() -> Int32 { + return fd + // How do we stop the deinit from running here? + } +} +``` + +Rust has the magic function `mem::forget` to suppress destruction of a value, +though `forget` in Rust still does not allow for the value to be destructured +into parts. We could come up with a mechanism in Swift that both suppresses +implicit deinitialization, and allows for piecewise taking of its components. +This doesn't require a new parameter convention (since it fits within the +ABI of a `consume T` parameter), but could be spelled as a new operator +inside of a `taking func`: + +```swift +extension FileHandle { + // Return the file descriptor back to the user for manual management + // and disable automatic management with the FileHandle value. + + taking func giveUp() -> Int32 { + // `deinit fd` is strawman syntax for consuming a value without running + // its deinitializer. it is only allowed inside of `taking func` methods + // that have visibility into the type layout. + return (deinit self).fd + } +} + +moveonly struct SocketPair { + var input, output: FileHandle + + deinit { /* ... */ } + + // Break the pair up into separately-managed FileHandles + taking func split() -> (input: FileHandle, output: FileHandle) { + // Break apart the value without running the standard deinit + return ((deinit self).input, (deinit self).output) + } +} +``` + +Suppressing the normal deinit logic for a type should be done carefully. +Although Rust allows `mem::forget` to be used anywhere, and considers it "safe" +based on the observation that destructors may not always run if a program +crashes or leaks a value anyway, that observation is less safe in the case +of values whose lifetime depends on a parent value, and where the child value's +deinit must be sequenced with operations on the parent. As such, I think +the ability should be limited to methods defined on the type itself. Also, +within the constraints of Swift's stable ABI and library evolution model, +code that imports a module may not always have access to the concrete layout +of a type, which would make partial destructuring impossible, which further +limits the contexts in which such an operator can be used. + +### `consume` from computed properties, property wrappers, properties with accessors, etc. + +It would potentially be useful to be able to `consume` variables +and properties with modified access behavior, such as computed +properties, properties with didSet/willSet observers, property +wrappers, and so on. Although we could do lifetime analysis on these +properties, we wouldn't be able to get the full performance benefits +from consuming a computed variable without allowing for some +additional accessors to be defined, such as a "consuming getter" +that can consume its `self` in order to produce the property value, +and an initializer to reinitialize `self` on reassignment after a +`move`. + +### Additional selective controls for implicit copying behavior + +Pitch: [Selective control of implicit copying behavior](https://forums.swift.org/t/selective-control-of-implicit-copying-behavior-take-borrow-and-copy-operators-noimplicitcopy/60168) + +The `consume` operator is one of a number of implicit copy controls +we're considering: + +- A value that isn't modified can generally be "borrowed" and + shared in-place by multiple bindings, or between a caller and + callee, without copying. However, the compiler will + pass shared mutable state by copying the current value, and + passing that copy to a callee. We do this to avoid + potential rule-of-exclusivity violations, since it is difficult to + know for sure whether a callee will go back and try to mutate the + same global variable, object, or other bit of shared mutable + state: + + ```swift + var global = Foo() + + func useFoo(x: Foo) { + // We would need exclusive access to `global` to do this: + + /* + global = Foo() + */ + } + + func callUseFoo() { + // callUseFoo doesn't know whether `useFoo` accesses global, + // so we want to avoid imposing shared access to it for longer + // than necessary. So by default the compiler will + // pass a copy instead, and this: + useFoo(x: global) + + // will compile more like: + + /* + let copyOfGlobal = copy(global) + useFoo(x: copyOfGlobal) + destroy(copyOfGlobal) + */ + } + ``` + + Although the compiler is allowed to eliminate the defensive copy + inside callUseFoo if it proves that useFoo doesn't try to write + to the global variable, it is unlikely to do so in practice. The + developer however knows that useFoo doesn't modify global, and + may want to suppress this copy in the call site. An explicit + `borrow` operator would let the developer communicate this to the + compiler: + + ```swift + var global = Foo() + + func useFoo(x: Foo) { + /* global not used here */ + } + + func callUseFoo() { + // The programmer knows that `useFoo` won't + // touch `global`, so we'd like to pass it without copying + useFoo(x: borrow global) + } + ``` + +- `consume` and `borrow` operators can eliminate copying in + common localized situations, but it is also useful to be able to + suppress implicit copying altogether for certain variables, types, + and scopes. We could define an attribute to specify that bindings + with static lifetime, types, or scopes should not admit implicit + copies: + + ```swift + // we're not allowed to implicitly copy `x` + func foo(@noImplicitCopy x: String) { + } + + // we're not allowed to implicitly copy values (statically) of + // type Gigantic + @noImplicitCopy struct Gigantic { + var fee, fie, fo, fum: String + } + + // we're not allowed to implicitly copy inside this hot loop + for item in items { + @noImplicitCopy do { + } + } + ``` + +### `borrow` and `consume` argument modifiers + +Pitch: [`borrow` and `consume` parameter ownership modifiers](https://forums.swift.org/t/borrow-and-take-parameter-ownership-modifiers/59581) + +Swift currently only makes an explicit distinction between +pass-by-value and pass-by-`inout` parameters, leaving the mechanism +for pass-by-value up to the implementation. But there are two broad +conventions that the compiler uses to pass by value: + +- The callee can **borrow** the parameter. The caller guarantees that + its argument object will stay alive for the duration of the call, + and the callee does not need to release it (except to balance any + additional retains it performs itself). +- The callee can **consume** the parameter. The callee becomes + responsible for either releasing the parameter or passing + ownership of it along somewhere else. If a caller doesn't want to + give up its own ownership of its argument, it must retain the + argument so that the callee can consume the extra reference count. + +In order to allow for manual optimization of code, and to support +move-only types where this distinction becomes semantically +significant, we plan to introduce explicit parameter modifiers to +let developers specify explicitly which convention a parameter +should use. + +## Acknowledgments + +Thanks to Nate Chandler, Tim Kientzle, and Holly Borla for their help with this! + +## Revision history + +Changes from the [third revision](https://github.com/swiftlang/swift-evolution/blob/7af91127d693bffcb01aa87978d75d5a3170c4d1/proposals/0366-move-function.md): + +- `take` is renamed again to `consume`. + +Changes from the [second revision](https://github.com/swiftlang/swift-evolution/blob/43849aa9ae3e87c434866c5a5e389af67537ca26/proposals/0366-move-function.md): + +- `move` is renamed to `take`. +- Dropping a value without using it now requires an explicit + `_ = take x` assignment again. +- "Movable bindings" are referred to as "bindings with static lifetime", + since this term is useful and relevant to other language features. +- Additional "alternatives considered" raised during review + and pitch discussions were added. +- Expansion of "related directions" section contextualizes the + `take` operator among other planned features for selective copy + control. +- Now that [ownership modifiers for parameters](https://forums.swift.org/t/borrow-and-take-parameter-ownership-modifiers/59581) + are being pitched, this proposal ties into that one. Based on + feedback during the first review, we have gone back to only allowing + parameters to be used with the `take` operator if the parameter + declaration is `take` or `inout`. + +Changes from the [first revision](https://github.com/swiftlang/swift-evolution/blob/567fb1a66c784bcc5394491d24f72a3cb393674f/proposals/0366-move-function.md): + +- `move x` is now proposed as a contextual keyword, instead of a magic function + `move(x)`. +- The proposal no longer mentions `__owned` or `__shared` parameters, which + are currently an experimental language feature, and leaves discussion of them + as a future direction. `move x` is allowed to be used on all function + parameters. +- `move x` is allowed as a statement on its own, ignoring the return value, + to release the current value of `x` without forwarding ownership without + explicitly assigning `_ = move x`. diff --git a/proposals/0367-conditional-attributes.md b/proposals/0367-conditional-attributes.md new file mode 100644 index 0000000000..09b915db61 --- /dev/null +++ b/proposals/0367-conditional-attributes.md @@ -0,0 +1,101 @@ +# Conditional compilation for attributes + +* Proposal: [SE-0367](0367-conditional-attributes.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.8)** + +* Implementation: [apple/swift#60208](https://github.com/apple/swift/pull/60208) +* Swift-evolution thread: [Pitch](https://forums.swift.org/t/pitch-conditional-compilation-for-attributes-and-modifiers/58339) +* Decision Notes: [Acceptance](https://forums.swift.org/t/accepted-se-0367-conditional-compilation-for-attributes/59756) + +## Introduction + +Over time, Swift has introduced a number of new attributes to communicate additional information in source code. Existing code can then be updated to take advantage of these new constructs to improve its behavior, providing more expressive capabilities, better compile-time checking, better performance, and so on. + +However, adopting a new attribute in existing source code means that source code won't compile with an older compiler. Conditional compilation can be used to address this problem, but the result is verbose and unsatisfactory. For example, one could use `#if` to check the compiler version to determine whether to use the `@preconcurrency` attribute: + +```swift +#if compiler(>=5.6) +@preconcurrency protocol P: Sendable { + func f() + func g() +} +#else +protocol P: Sendable { + func f() + func g() +} +#endif +``` + +This is unsatisfactory for at least two reasons. First, it's a lot of code duplication, because the entire protocol `P` needs to be duplicated just to provide the attribute. Second, the Swift 5.6 compiler is the first to contain the `@preconcurrency` attribute, but that is somewhat incidental and not self-documenting: the attribute could have been enabled by a compiler flag or partway through the development of Swift 5.6, making that check incorrect. Moreover, the availability of some attributes can depend not on compiler version, but on platform and configuration flags: for example, `@objc` is only available when the Swift runtime has been compiled for interoperability with Objective-C. Although these are small issues in isolation, they make adopting new attributes in existing code harder than it needs to be. + +## Proposed solution + +I propose two related changes to make it easier to adopt new attributes in existing code: + +* Allow `#if` checks to surround attributes on a declaration wherever they appear, eliminating the need to clone a declaration just to adopt a new attribute. +* Add a conditional directive `hasAttribute(AttributeName)` that evaluates `true` when the compiler has support for the attribute with the name `AttributeName` in the current language mode. + +The first two of these can be combined to make the initial example less repetitive and more descriptive: + +```swift +#if hasAttribute(preconcurrency) +@preconcurrency +#endif +protocol P: Sendable { + func f() + func g() +} +``` + +## Detailed design + +The design of these features is relatively straightforward, but there are a few details to cover. + +### Grammar changes + +The current production for an attribute list: + +``` +attributes → attribute attributes[opt] +``` + +will be augmented with an additional production for a conditional attribute: + +``` +attributes → conditional-compilation-attributes attributes[opt] + +conditional-compilation-attributes → if-directive-attributes elseif-directive-attributes[opt] else-directive-attributes[opt] endif-directive +if-directive-attributes → if-directive compilation-condition attributes[opt] +elseif-directive-attributes → elseif-directive-attributes elseif-directive-attributes[opt] +elseif-directive-attributes → elseif-directive compilation-condition attributes[opt] +else-directive-attributes → else-directive attributes[opt] +``` + +i.e., within an attribute list one can have a conditional clause `#if...#endif` that wraps another attribute list. + +### `hasAttribute` only considers attributes that are part of the language + +A number of Swift language features, including property wrappers, result builders, and global actors, all introduce forms of custom attributes. For example, a type `MyWrapper` that has been marked with the `@propertyWrapper` attribute (and meets the other requirements for a property wrapper type) can be used with the attribute syntax `@MyWrapper`. While the built-in attribute that enables the feature will be recognized by `hasAttribute` (e.g., `hasAttribute(propertyWrapper)` will evaluate `true`), the custom attribute will not (e.g., `hasAttribute(MyWrapper)` will evaluate `false`). + +### Parsing the conditionally-compiled branches not taken + +Due to support for custom attributes, attributes have a very general grammar that should suffice for any new attributes we introduce into Swift: + +``` +attribute → @ attribute-name attribute-argument-clause[opt] +attribute-name → identifier +attribute-argument-clause → ( balanced-tokens[opt] ) +``` + +Therefore, a conditionally-compiled branch based on `#if hasAttribute(UnknownAttributeName)` can still be parsed by an existing compiler, even though it will not be applied to the declaration because it isn't understood: + +```swift +#if hasAttribute(UnknownAttributeName) +@UnknownAttributeName(something we do not understand) // okay, we parse this but don't reject it +#endif +func f() +``` + diff --git a/proposals/0368-staticbigint.md b/proposals/0368-staticbigint.md new file mode 100644 index 0000000000..4c86fb5ff0 --- /dev/null +++ b/proposals/0368-staticbigint.md @@ -0,0 +1,179 @@ +# StaticBigInt + +* Proposal: [SE-0368](0368-staticbigint.md) +* Author: [Ben Rimmington](https://github.com/benrimmington) +* Review Manager: [Doug Gregor](https://github.com/DougGregor), [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#40722](https://github.com/apple/swift/pull/40722), [apple/swift#62733](https://github.com/apple/swift/pull/62733) +* Review: ([pitch](https://forums.swift.org/t/staticbigint/54545)) ([review](https://forums.swift.org/t/se-0368-staticbigint/59421)) ([acceptance](https://forums.swift.org/t/accepted-se-0368-staticbigint/59962)) ([amendment](https://forums.swift.org/t/pitch-amend-se-0368-to-remove-prefix-operator/62173)), ([amendment review](https://forums.swift.org/t/amendment-se-0368-staticbigint/62992)), ([amendment acceptance](https://forums.swift.org/t/accepted-amendment-se-0368-staticbigint/63246)) + +
+Revision history + +| | | +| ---------- | ------------------------------------------------- | +| 2022-01-10 | Initial pitch. | +| 2022-02-01 | Updated with an "ABI-neutral" abstraction. | +| 2022-04-23 | Updated with an "infinitely-sign-extended" model. | +| 2022-08-18 | Updated with a "non-generic" subscript. | +| 2023-02-03 | Amended to remove the prefix `+` operator. | + +
+ +## Introduction + +Integer literals in Swift source code can express an arbitrarily large value. However, types outside of the standard library which conform to `ExpressibleByIntegerLiteral` are restricted in practice in how large of a literal value they can be built with, because the value passed to `init(integerLiteral:)` must be of a type supported by the standard library. This makes it difficult to write new integer types outside of the standard library. + +## Motivation + +Types in Swift that want to be buildable with an integer literal can conform to the following protocol: + +```swift +public protocol ExpressibleByIntegerLiteral { + associatedtype IntegerLiteralType: _ExpressibleByBuiltinIntegerLiteral + init(integerLiteral value: IntegerLiteralType) +} +``` + +The value passed to `init(integerLiteral:)` must have a type that knows how to manage the primitive interaction with the Swift compiler so that it can be built from an arbitrary literal value. That constraint is expressed with the `_ExpressibleByBuiltinIntegerLiteral` protocol, which cannot be implemented outside of the standard library. All of the integer types in the standard library conform to `_ExpressibleByBuiltinIntegerLiteral` as well as `ExpressibleByIntegerLiteral`. A type outside of the standard library must select one of those types as the type it takes in `init(integerLiteral:)`. As a result, such types cannot be built from an integer literal if there isn't a type in the standard library big enough to express that integer. + +For example, if larger fixed-width integers (such as `UInt256`) were added to the [Swift Numerics][] package, they would currently have to use smaller literals (such as `UInt64`). + +```swift +let value: UInt256 = 0x1_0000_0000_0000_0000 +// ^ +// error: integer literal '18446744073709551616' overflows when stored into 'UInt256' +``` + +## Proposed solution + +We propose adding a new type to the standard library called `StaticBigInt` which is capable of expressing any integer value. This can be used as the associated type of an `ExpressibleByIntegerLiteral` conformance. For example: + +```swift +extension UInt256: ExpressibleByIntegerLiteral { + + public init(integerLiteral value: StaticBigInt) { + precondition( + value.signum() >= 0 && value.bitWidth <= Self.bitWidth + 1, + "integer literal '\(value)' overflows when stored into '\(Self.self)'" + ) + self.words = Words() + for wordIndex in 0.. Int + + /// Returns the minimal number of bits in this value's binary representation, + /// including the sign bit, and excluding the sign extension. + /// + /// The following examples show the least significant byte of each value's + /// binary representation, separated (by an underscore) into excluded and + /// included bits. Negative values are in two's complement. + /// + /// * `-4` (`0b11111_100`) is 3 bits. + /// * `-3` (`0b11111_101`) is 3 bits. + /// * `-2` (`0b111111_10`) is 2 bits. + /// * `-1` (`0b1111111_1`) is 1 bit. + /// * `+0` (`0b0000000_0`) is 1 bit. + /// * `+1` (`0b000000_01`) is 2 bits. + /// * `+2` (`0b00000_010`) is 3 bits. + /// * `+3` (`0b00000_011`) is 3 bits. + public var bitWidth: Int { get } + + /// Returns a 32-bit or 64-bit word of this value's binary representation. + /// + /// The words are ordered from least significant to most significant, with + /// an infinite sign extension. Negative values are in two's complement. + /// + /// let negative: StaticBigInt = -0x0011223344556677_8899AABBCCDDEEFF + /// negative.signum() //-> -1 + /// negative.bitWidth //-> 118 + /// negative[0] //-> 0x7766554433221101 + /// negative[1] //-> 0xFFEEDDCCBBAA9988 + /// negative[2] //-> 0xFFFFFFFFFFFFFFFF + /// + /// let positive: StaticBigInt = 0x0011223344556677_8899AABBCCDDEEFF + /// positive.signum() //-> +1 + /// positive.bitWidth //-> 118 + /// positive[0] //-> 0x8899AABBCCDDEEFF + /// positive[1] //-> 0x0011223344556677 + /// positive[2] //-> 0x0000000000000000 + /// + /// - Parameter wordIndex: A nonnegative zero-based offset. + public subscript(_ wordIndex: Int) -> UInt { get } +} +``` + +## Effect on ABI stability + +This feature adds to the ABI of the standard library, and it won't back-deploy (by default). + +The integer literal type has to be selected statically as the associated type. There is currently no way to conditionally use a different integer literal type depending on the execution environment. Types will not be able to adopt this and use the most flexible possible literal type dynamically available. + +## Alternatives considered + +- Modeling the original source text instead of a mathematical value would allow this type to support a wide range of use cases, such as fractional values, decimal values, and other things such as arbitrary binary strings expressed in hexadecimal. However, it is not a goal of Swift's integer literals design to support these use cases. Supporting them would burden integer types with significant code size, dynamic performance, and complexity overheads. For example, either the emitted code would need to contain both the original source text and a more optimized representation used by ordinary integer types, or ordinary integer types would need to fall back on parsing numeric values from source at runtime. + +- Along similar lines, it is intentional that `StaticBigInt` cannot represent fractional values. Integer types should not be constructible with fractional literals, and allowing that simply adds unnecessary costs and introduces a new way for construction to fail. It is still a language goal for Swift to someday support dynamically flexible floating-point literals the way it does for integer literals, but that is a separable project from introducing `StaticBigInt`. + +- A prior design had a `words` property, initially as a contiguous buffer, subsequently as a custom collection. John McCall requested an "ABI-neutral" abstraction, and suggested the current "infinitely-sign-extended" model. Xiaodi Wu convincingly argued for a "non-generic" subscript, rather than over-engineering a choice of element type. + +- Xiaodi Wu [suggested](https://forums.swift.org/t/staticbigint/54545/23) that a different naming scheme and API design be chosen to accommodate other similar types, such as IEEE 754 interchange formats. However, specific alternatives weren't put forward for consideration. Using non-arithmetic types for interchange formats would seem to be a deliberate choice; whereas for `StaticBigInt` it's because of an inherent limitation. + +- A previously accepted version of this proposal included the following operator, for symmetry between negative and positive literals. + + ```swift + extension StaticBigInt { + /// Returns the given value unchanged. + public static prefix func + (_ rhs: Self) -> Self + } + ``` + + It was later discovered to be a source-breaking change. For example: + + ```swift + let a = -7 // inferred as `a: Int` + let b = +6 // inferred as `b: StaticBigInt` + let c = a * b + // ^ + // error: Cannot convert value of type 'StaticBigInt' to expected argument type 'Int' + ``` + + The prefix `+` operator on [`AdditiveArithmetic`][numeric protocols] was no longer chosen, because concrete overloads are preferred over generic overloads. + +## Acknowledgments + +John McCall made significant improvements to this proposal; and (in Swift 5.0) implemented arbitrary-precision integer literals. `StaticBigInt` is a thin wrapper around the existing [`Builtin.IntLiteral`][] type. + +Stephen Canon proposed an amendment to remove the prefix `+` operator. + + + +[`Builtin.IntLiteral`]: + +[numeric protocols]: + +[Swift Numerics]: diff --git a/proposals/0369-add-customdebugdescription-conformance-to-anykeypath.md b/proposals/0369-add-customdebugdescription-conformance-to-anykeypath.md new file mode 100644 index 0000000000..6f58d7aff8 --- /dev/null +++ b/proposals/0369-add-customdebugdescription-conformance-to-anykeypath.md @@ -0,0 +1,160 @@ +# Add CustomDebugStringConvertible conformance to AnyKeyPath + +* Proposal: [SE-0369](0369-add-customdebugdescription-conformance-to-anykeypath.md) +* Author: [Ben Pious](https://github.com/benpious) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#60133](https://github.com/apple/swift/pull/60133) +* Review: ([pitch](https://forums.swift.org/t/pitch-add-customdebugdescription-conformance-to-anykeypath/58705)) ([review](https://forums.swift.org/t/se-0369-add-customdebugstringconvertible-conformance-to-anykeypath/59704)) ([acceptance](https://forums.swift.org/t/accepted-se-0369-add-customdebugstringconvertible-conformance-to-anykeypath/60001)) + +## Introduction + +This proposal is to add conformance to the protocol `CustomDebugStringConvertible` to `AnyKeyPath`. + +## Motivation + +Currently, passing a keypath to `print()`, or to the `po` command in LLDB, yields the standard output for a Swift class. This is not very useful. For example, given +```swift +struct Theme { + + var backgroundColor: Color + var foregroundColor: Color + + var overlay: Color { + backgroundColor.withAlpha(0.8) + } +} +``` +`print(\Theme.backgroundColor)` would have an output of roughly +``` +Swift.KeyPath +``` +which doesn't allow `foregroundColor` to be distinguished from any other property on `Theme`. + +Ideally, the output would be +``` +\Theme.backgroundColor +``` +exactly as it was written in the program. + +## Proposed solution + +Take advantage of whatever information is available in the binary to implement the `debugDescription` requirement of `CustomDebugStringConvertible`. In the best case, roughly the output above will be produced, in the worst cases, other, potentially useful information will be output instead. + +## Detailed design + +### Implementation of `CustomDebugStringConvertible` + +Much like the `_project` functions currently implemented in `KeyPath.swift`, this function would loop through the keypath's buffer, handling each segment as follows: + +For offset segments, the implementation is simple: use `_getRecursiveChildCount`, `_getChildOffset`, and `_getChildMetadata` to get the string name of the property. I believe these are the same mechanisms used by `Mirror` today. + +For optional chain, force-unwrap, etc. the function appends a hard coded "?" or "!", as appropriate. + +For computed segments, call `swift::lookupSymbol()` on the result of `getter()` in the `ComputedAccessorsPtr`. Demangle the result to get the property name. + +### Changes to the Swift Runtime + +To implement descriptions for computed segments, it is necessary to make two changes to the runtime: + +1. Expose a Swift calling-convention function to call `swift::lookupSymbol()`. +2. Implement and expose a function to demangle keypath functions without the ornamentation the existing demangling functions produce. + +### Dealing with missing data + +There are two known cases where data might not be available: + +1. type metadata has not been emitted because the target was built using the `swift-disable-reflection-metadata` flag +2. the linker has stripped the symbol names we're trying to look up + +In these cases, we would print the following: + +#### Offset case +`` where `x` is the memory offset we read from the reflection metadata, and `typename` is the type that will be returned. +So +``` +print(\Theme.backgroundColor) // outputs "\Theme." +``` + +#### `lookupSymbol` failure case + +In this case we'll print the address-in-memory as hex, plus the type name: +``` +print(\Theme.overlay) // outputs \Theme. +``` + +As it might be difficult to correlate a memory address with the name of the function, the type name may be useful here to provide extra context. + +## Source compatibility + +Programs that extend `AnyKeyPath` to implement `CustomDebugStringConvertible` themselves will no longer compile and the authors of such code will have to delete the conformance. Based on a search of Github, there are currently no publicly available Swift projects that do this. + +Calling `print` on a KeyPath will, of course, produce different results than before. + +It is unlikely that any existing Swift program is depending on the existing behavior in a production context. While it is likely that someone, somewhere has written code in unit tests that depends on the output of this function, any issues that result will be easy for the authors of such code to identify and fix, and will likely represent an improvement in the readability of those tests. + +## Effect on ABI stability + +This proposal will add a new var & protocol conformance to the Standard Library's ABI. It will be availability guarded appropriately. + +The new debugging output will not be backdeployed, so Swift programs running on older ABI stable versions of the OS won't be able to rely on the new output. + +## Effect on API resilience + +The implementation of `debugDescription` might change after the initial work to implement this proposal is done. In particular, the output format will not be guaranteed to be stable. Here are a few different changes we might anticipate making: + +- As new features are added to the compiler, there may be new metadata available in the binary to draw from. One example would be lookup tables of KeyPath segment to human-readable-name or some other unique, stable identifier +- Whenever a new feature is added to `KeyPath`, it will need to be reflected in the output of this function. For example, the `KeyPath`s produced by [\_forEachFieldWithKeyPath](https://github.com/apple/swift/blob/main/stdlib/public/core/ReflectionMirror.swift#L324) are incomplete, in the sense that they merely set a value at an offset in memory and do not call `didSet` observers. If this function were ever publicly exposed, it would be useful if this was surfaced in the debugging information. +- The behavior of subscript printing might be changed: for example, we might always print out the value of the argument to the subscript, or we might do so only in cases where the output is short. We might also change from `.subscript()` to `[]` +- The Swift language workgroup may create new policies around debug descriptions and the output of this function might need to be updated to conform to them + +## Alternatives considered + +### Print fully qualified names or otherwise add more information to the output + +ex. `\ModuleName.MyType.myField`, `> \ModuleName.MyType.myField` `(writable) \Theme.backgroundColor` + +As this is just for debugging, it seems likely that the information currently being provided would be enough to resolve any ambiguities. If ambiguities arose during a debugging session, in most cases the user could figure out exactly which keypath they were dealing with simply by running `po myKeyPath == \MyType.myField` till they found the right one. + +### Modify KeyPaths to include a string description + +This is an obvious solution to this problem, and would likely be very easy to implement, as the compiler already produces `_kvcString`. + +It has the additional advantage of being 100% reliable, to the point where it arguably could be the basis for implementing `description` rather than `debugDescription`. + +However, it would add to the code size of the compiled code, perhaps unacceptably so. Furthermore, it precludes the possibility of someday printing out the arguments of subscript based keypaths, as +these can be created dynamically. It would also add overhead to appending keypaths, as the string would also have to be appended. + +An alternative implementation of this idea would be the output of additional metadata: a lookup table of function -> name. However, this would require a lot of additional work on the compiler for a relatively small benefit. + +I think that most users who might want this _really_ want to use it to build something else, like encodable keypaths. Those features should be provided opt-in on a per-keypath or per-type basis, which will make it much more useful (and in the context of encodable keypaths specifically, eliminate major potential security issues). Such a feature should also include the option to let the user configure this string, so that it can remain backwards compatible with older versions of the program. + +### Make Keypath functions global to prevent the linker from stripping them + +This would also potentially make it feasible to change this proposal from implementing `debugDescription` to implementing `description`. + +This would also potentially bloat the binary and would increase linker times. It could also be a security issue, as `dlysm` would now be able to find these functions. + +I am not very knowledgeable about linkers or how typical Swift builds strip symbols, but I think it might be useful to have this as an option in some IDEs that build Swift programs. But that is beyond the scope of this proposal. + +## Future Directions + +### Add LLDB formatters/summaries + +This would be a good augmentation to this proposal, and might improve the developer experience, as there [might be debug metadata available to the debugger](https://forums.swift.org/t/pitch-add-customdebugdescription-conformance-to-anykeypath/58705/2) that is not available in the binary itself. + +However, I think it might be very difficult to implement this. I see two options: + +1. Implement a public reflection API for KeyPaths in the Swift Standard Library that the formatter can interact with from Python. +2. The formatter parses the raw memory of the KeyPath, essentially duplicating the code in `debugDescription`. + +I think (1) is overkill, especially considering the limited potential applications of this API beyond its use by the formatter. If it's possible to implement this as an `internal` function in the Swift stdlib then this is a much more attractive option. +From personal experience trying to parse KeyPath memory from outside the Standard Library, I think (2) would be extremely difficult to implement, and unsustainable to maintain considering that the [memory layout of KeyPaths](https://github.com/apple/swift/blob/main/docs/ABI/KeyPaths.md) is not ABI stable. + +### Make Keypath functions global in DEBUG builds only + +This may be necessary to allow `swift::lookupSymbol` to function correctly on Windows, Linux and other platforms that use COFF or ELF-like formats. + +## Acknowledgments + +Thanks to Joe Groff for answering several questions on the initial pitch document, and Slava Pestov for answering some questions about the logistics of pitching this. diff --git a/proposals/0370-pointer-family-initialization-improvements.md b/proposals/0370-pointer-family-initialization-improvements.md new file mode 100644 index 0000000000..9786754117 --- /dev/null +++ b/proposals/0370-pointer-family-initialization-improvements.md @@ -0,0 +1,1763 @@ +# Pointer Family Initialization Improvements and Better Buffer Slices + +* Proposal: [SE-0370](0370-pointer-family-initialization-improvements.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#41608](https://github.com/apple/swift/pull/41608) +* Review: ([first pitch](https://forums.swift.org/t/pitch-pointer-family-initialization-improvements/53168)) ([second pitch](https://forums.swift.org/t/pitch-buffer-partial-initialization-better-buffer-slices/53795)) ([third pitch](https://forums.swift.org/t/pitch-pointer-family-initialization-improvements-better-buffer-slices/55689)) ([review](https://forums.swift.org/t/se-0370-pointer-family-initialization-improvements-and-better-buffer-slices/59724)) ([acceptance](https://forums.swift.org/t/accepted-se-0370-pointer-family-initialization-improvements-and-better-buffer-slices/60007)) + +## Introduction + +The types in the `UnsafeMutablePointer` family typically require manual management of memory allocations, including the management of their initialization state. Unfortunately, not every relevant type in the family has the necessary functionality to fully manage the initialization state of the memory it represents. The states involved are, after allocation: + +1. Unbound and uninitialized (as returned from `UnsafeMutableRawPointer.allocate()`) +2. Bound to a type, and uninitialized (as returned from `UnsafeMutablePointer.allocate()`) +3. Bound to a type, and initialized + +Memory can be safely deallocated whenever it is uninitialized. + +We intend to round out initialization functionality for every relevant member of that family: `UnsafeMutablePointer`, `UnsafeMutableRawPointer`, `UnsafeMutableBufferPointer`, `UnsafeMutableRawBufferPointer`, `Slice` and `Slice`. The functionality will allow managing initialization state in a much greater variety of situations, including easier handling of partially-initialized buffers. + +## Motivation + +Memory allocated using `UnsafeMutablePointer`, `UnsafeMutableRawPointer`, `UnsafeMutableBufferPointer` and `UnsafeMutableRawBufferPointer` is passed to the user in an uninitialized state. In the general case, such memory needs to be initialized before it is used in Swift. Memory can be "initialized" or "uninitialized". We hereafter refer to this as a memory region's "initialization state". + +The methods of `UnsafeMutablePointer` that interact with initialization state are: + +- `func initialize(to value: Pointee)` +- `func initialize(repeating repeatedValue: Pointee, count: Int)` +- `func initialize(from source: UnsafePointer, count: Int)` +- `func assign(repeating repeatedValue: Pointee, count: Int)` +- `func assign(from source: UnsafePointer, count: Int)` +- `func move() -> Pointee` +- `func moveInitialize(from source: UnsafeMutablePointer, count: Int)` +- `func moveAssign(from source: UnsafeMutablePointer, count: Int)` +- `func deinitialize(count: Int) -> UnsafeMutableRawPointer` + +This is a fairly complete set. + +- The `initialize` functions change the state of memory locations from uninitialized to initialized, + then assign the corresponding value(s). +- The `assign` functions update the values stored at memory locations that have previously been initialized. +- `deinitialize` changes the state of a range of memory from initialized to uninitialized. +- The `move()` function deinitializes a memory location, then returns its current contents. +- The `move` prefix means that the `source` range of memory will be deinitialized after the function returns. + +Unfortunately, `UnsafeMutablePointer` is the only one of the list of types listed in the introduction to allow full control of initialization state, and this means that complex use cases such as partial initialization of a buffer become needlessly difficult. + +An example of partial initialization is the insertion of elements in the middle of a collection. This is one of the possible operations needed in an implementation of `RangeReplaceableCollection.replaceSubrange(_:with:)`. Given a `RangeReplaceableCollection` whose unique storage can be represented by a partially-initialized `UnsafeMutableBufferPointer`: + +```swift +mutating func replaceSubrange(_ subrange: Range, with newElements: C) + where C: Collection, Element == C.Element { + + // obtain unique storage as UnsafeMutableBufferPointer + let buffer: UnsafeMutableBufferPointer = self.myUniqueStorage() + let oldCount = self.count + let growth = newElements.count - subrange.count + let newCount = oldCount + growth + if growth > 0 { + assert(newCount < buffer.count) + let oldTail = subrange.upperBound..(_ subrange: Range, with newElements: C) + where C: Collection, Element == C.Element { + + // obtain unique storage as UnsafeMutableBufferPointer + let buffer: UnsafeMutableBufferPointer = self.myUniqueStorage() + let oldCount = self.count + let growth = newElements.count - subrange.count + let newCount = oldCount + growth + if growth > 0 { + assert(newCount < buffer.count) + let oldTail = subrange.upperBound..` and `Slice`. + + +## Proposed solution + +Note: the pseudo-diffs presented in this section denotes added functions with `+++` and renamed functions with `---`. Unmarked functions are unchanged. + +##### `UnsafeMutableBufferPointer` + +We propose to modify `UnsafeMutableBufferPointer` as follows: + +```swift +extension UnsafeMutableBufferPointer { + func initialize(repeating repeatedValue: Element) ++++ func initialize(from source: S) -> (unwritten: S.Iterator, index: Index) where S: Sequence, S.Element == Element +--- func initialize(from source: S) -> (S.Iterator, Index) where S: Sequence, S.Element == Element ++++ func initialize(fromContentsOf source: C) -> Index where C: Collection, C.Element == Element +--- func assign(repeating repeatedValue: Element) ++++ func update(repeating repeatedValue: Element) ++++ func update(from source: S) -> (unwritten: S.Iterator, index: Index) where S: Sequence, S.Element == Element ++++ func update(fromContentsOf source: C) -> Index where C: Collection, C.Element == Element ++++ func moveInitialize(fromContentsOf source: UnsafeMutableBufferPointer) -> Index ++++ func moveInitialize(fromContentsOf source: Slice) -> Index ++++ func moveUpdate(fromContentsOf source: Self) -> Index ++++ func moveUpdate(fromContentsOf source: Slice) -> Index ++++ func deinitialize() -> UnsafeMutableRawBufferPointer + ++++ func initializeElement(at index: Index, to value: Element) ++++ func moveElement(from index: Index) -> Element ++++ func deinitializeElement(at index: Index) +} +``` + + + +We would like to use the verb `update` instead of `assign`, in order to better communicate the intent of the API. It is currently a common programmer error to use one of the existing `assign` functions for uninitialized memory; using the verb `update` instead would express the precondition in the API name itself. + +The methods that initialize or update from a `Collection` will have consistent (and strict) semantics, and require the destination buffer (or slice) to have enough storage to copy the entire source collection, and then return the index in the destination that follows the last element copied. The storage available precondition will be strictly enforced, resulting in a consistent behaviour. The existing `Sequence` functions intended such a precondition, but in practice cannot enforce it. + +We also add functions to manipulate the initialization state for single elements of the buffer. There is no `buffer.updateElement(at index: Index, to value: Element)`, because it can already be expressed as `buffer[index] = value`. + +##### `UnsafeMutablePointer` + +The proposed modifications to `UnsafeMutablePointer` are renamings: + +```swift +extension UnsafeMutablePointer { + func initialize(to value: Pointee) + func initialize(repeating repeatedValue: Pointee, count: Int) + func initialize(from source: UnsafePointer, count: Int) +--- func assign(repeating repeatedValue: Pointee, count: Int) ++++ func update(repeating repeatedValue: Pointee, count: Int) +--- func assign(from source: UnsafePointer, count: Int) ++++ func update(from source: UnsafePointer, count: Int) + func move() -> Pointee + func moveInitialize(from source: UnsafeMutablePointer, count: Int) +--- func moveAssign(from source: UnsafeMutablePointer, count: Int) ++++ func moveUpdate(from source: UnsafeMutablePointer, count: Int) + func deinitialize(count: Int) -> UnsafeMutableRawPointer +} +``` + +The motivation for these renamings are explained above. + +##### `UnsafeMutableRawBufferPointer` + +We propose to add new functions to initialize memory referenced by `UnsafeMutableRawBufferPointer` instances. + +```swift +extension UnsafeMutableRawBufferPointer { + func initializeMemory( + as type: T.Type, repeating repeatedValue: T + ) -> UnsafeMutableBufferPointer + + func initializeMemory( + as type: S.Element.Type, from source: S + ) -> (unwritten: S.Iterator, initialized: UnsafeMutableBufferPointer) where S: Sequence + ++++ func initializeMemory( + as type: C.Element.Type, fromContentsOf source: C + ) -> UnsafeMutableBufferPointer where C: Collection + ++++ func moveInitializeMemory( + as type: T.Type, fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + ++++ func moveInitializeMemory( + as type: T.Type, fromContentsOf source: Slice> + ) -> UnsafeMutableBufferPointer +} +``` + +The first addition will initialize raw memory from a `Collection` and have similar behaviour as `UnsafeMutableBufferPointer.initialize(fromContentsOf:)`, described above. The other two initialize raw memory by moving data from another range of memory, leaving that other range of memory deinitialized. + +##### `UnsafeMutableRawPointer` + +```swift +extension UnsafeMutableRawPointer { ++++ func initializeMemory(as type: T.Type, to value: T) -> UnsafeMutablePointer + + func initializeMemory( + as type: T.Type, repeating repeatedValue: T, count: Int + ) -> UnsafeMutablePointer + + func initializeMemory( + as type: T.Type, from source: UnsafePointer, count: Int + ) -> UnsafeMutablePointer + + func moveInitializeMemory( + as type: T.Type, from source: UnsafeMutablePointer, count: Int + ) -> UnsafeMutablePointer +} +``` + +The addition here initializes a single value. + +##### Slices of `BufferPointer` + +We propose to extend slices of `Unsafe[Mutable][Raw]BufferPointer` with all the `BufferPointer`-specific methods of their `Base`. The following declarations detail the additions, which are all intended to behave exactly as the functions on the base BufferPointer types: + +```swift +extension Slice> { + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeBufferPointer) throws -> Result + ) rethrows -> Result +} +``` + +```swift +extension Slice> { + func initialize(repeating repeatedValue: Element) + + func initialize(from source: S) -> (unwritten: S.Iterator, index: Index) + where S.Element == Element + + func initialize(fromContentsOf source: C) -> Index + where C.Element == Element + + func update(repeating repeatedValue: Element) + + func update( + from source: S + ) -> (unwritten: S.Iterator, index: Index) where S.Element == Element + + func update( + fromContentsOf source: C + ) -> Index where C.Element == Element + + func moveInitialize(fromContentsOf source: UnsafeMutableBufferPointer) -> Index + func moveInitialize(fromContentsOf source: Slice>) -> Index + func moveUpdate(fromContentsOf source: UnsafeMutableBufferPointer) -> Index + func moveUpdate(fromContentsOf source: Slice>) -> Index + + func deinitialize() -> UnsafeMutableRawBufferPointer + + func initializeElement(at index: Index, to value: Element) + func moveElement(at index: Index) -> Element + func deinitializeElement(at index: Index) + + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result +} +``` + +Slices of `Unsafe[Mutable]RawBufferPointer` will add memory binding functions, memory initialization functions, and variants of `load`, `loadUnaligned` and `storeBytes`. +```swift +extension Slice { + func bindMemory(to type: T.Type) -> UnsafeBufferPointer + func assumingMemoryBound(to type: T.Type) -> UnsafeBufferPointer + + func withMemoryRebound( + to type: T.Type, _ body: (UnsafeBufferPointer) throws -> Result + ) rethrows -> Result + + func load(fromByteOffset offset: Int = 0, as type: T.Type) -> T + func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T +} +``` + +```swift +extension Slice { + func copyMemory(from source: UnsafeRawBufferPointer) + func copyBytes(from source: C) where C.Element == UInt8 + + func initializeMemory( + as type: T.Type, repeating repeatedValue: T + ) -> UnsafeMutableBufferPointer + + func initializeMemory( + as type: S.Element.Type, from source: S + ) -> (unwritten: S.Iterator, initialized: UnsafeMutableBufferPointer) + + func initializeMemory( + as type: C.Element.Type, fromContentsOf source: C + ) -> UnsafeMutableBufferPointer + + func moveInitializeMemory( + as type: T.Type, fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + + func moveInitializeMemory( + as type: T.Type, fromContentsOf source: Slice> + ) -> UnsafeMutableBufferPointer + + func bindMemory(to type: T.Type) -> UnsafeMutableBufferPointer + func assumingMemoryBound(to type: T.Type) -> UnsafeMutableBufferPointer + + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result + + func load(fromByteOffset offset: Int = 0, as type: T.Type) -> T + func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T + func storeBytes(of value: T, toByteOffset offset: Int = 0, as type: T.Type) +} +``` + +## Detailed design + +##### `UnsafeMutableBufferPointer` + +```swift +extension UnsafeMutableBufferPointer { + /// Initializes the buffer's memory with the given elements. + /// + /// Prior to calling the `initialize(from:)` method on a buffer, + /// the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer up to, but not including, + /// the returned index is initialized. + /// The buffer must contain sufficient memory to accommodate + /// `source.underestimatedCount`. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer, which is one past the last element written. + /// If `source` contains no elements, the returned index is equal to + /// the buffer's `startIndex`. If `source` contains an equal or greater number + /// of elements than the buffer can hold, the returned index is equal to + /// the buffer's `endIndex`. + /// + /// - Parameter source: A sequence of elements with which to initialize the + /// buffer. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and an index to the next uninitialized element in the buffer. + public func initialize( + from source: S + ) -> (unwritten: S.Iterator, index: Index) where S.Element == Element + + /// Initializes the buffer's memory with every element of the source. + /// + /// Prior to calling the `initialize(fromContentsOf:)` method on a buffer, + /// the memory referenced by the buffer must be uninitialized, + /// or the `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer up to, but not including, + /// the returned index is initialized. + /// The buffer must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A collection of elements to be used to + /// initialize the buffer's storage. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + func initialize(fromContentsOf source: C) -> Index + where C: Collection, C.Element == Element + + /// Updates every element of this buffer's initialized memory. + /// + /// The buffer’s memory must be initialized or the buffer's `Element` + /// must be a trivial type. + /// + /// - Note: All buffer elements must already be initialized. + /// + /// - Parameters: + /// - repeatedValue: The value used when updating this pointer's memory. + public func update(repeating repeatedValue: Element) + + /// Updates the buffer's initialized memory with the given elements. + /// + /// The buffer's memory must be initialized or the buffer's `Element` type + /// must be a trivial type. + /// + /// - Parameter source: A sequence of elements to be used to update + /// the buffer's contents. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and the index one past the last updated element in the buffer. + public func update(from source: S) -> (unwritten: S.Iterator, index: Index) + where S: Sequence, S.Element == Element + + /// Updates the buffer's initialized memory with every element of the source. + /// + /// Prior to calling the `update(fromContentsOf:)` method on a buffer, + /// the first `source.count` elements of the buffer's memory must be + /// initialized, or the buffer's `Element` type must be a trivial type. + /// The buffer must reference enough initialized memory to accommodate + /// `source.count` elements. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A collection of elements to be used to update + /// the buffer's contents. + /// - Returns: An index one past the index of the last element updated. + public func update(fromContentsOf source: C) -> Index + where C: Collection, C.Element == Element + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer, leaving the source memory + /// uninitialized and this buffer's memory initialized. + /// + /// Prior to calling the `moveInitialize(fromContentsOf:)` method on a buffer, + /// the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer up to, but not including, + /// the returned index is initialized. The memory referenced by + /// `source` is uninitialized after the function returns. + /// The buffer must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to copy. The memory + /// region underlying `source` must be initialized. The memory regions + /// referenced by `source` and this buffer may overlap. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + public func moveInitialize(fromContentsOf source: Self) -> Index + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer, leaving the source memory + /// uninitialized and this buffer's memory initialized. + /// + /// Prior to calling the `moveInitialize(fromContentsOf:)` method on a buffer, + /// the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer up to, but not including, + /// the returned index is initialized. The memory referenced by + /// `source` is uninitialized after the function returns. + /// The buffer must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to copy. The memory + /// region underlying `source` must be initialized. The memory regions + /// referenced by `source` and this buffer may overlap. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + public func moveInitialize(fromContentsOf source: Slice) -> Index + + /// Updates this buffer's initialized memory initialized memory by + /// moving every element from the source buffer slice, + /// leaving the source memory uninitialized. + /// + /// Prior to calling the `moveUpdate(fromContentsOf:)` method on a buffer, + /// the first `source.count` elements of the buffer's memory must be + /// initialized, or the buffer's `Element` type must be a trivial type. + /// The memory referenced by `source` is uninitialized after the function + /// returns. The buffer must reference enough initialized memory + /// to accommodate `source.count` elements. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to move. + /// The memory region underlying `source` must be initialized. The + /// memory regions referenced by `source` and this pointer must not overlap. + /// - Returns: An index one past the index of the last element updated. + public func moveUpdate(fromContentsOf source: Self) -> Index + + /// Updates this buffer's initialized memory initialized memory by + /// moving every element from the source buffer slice, + /// leaving the source memory uninitialized. + /// + /// Prior to calling the `moveUpdate(fromContentsOf:)` method on a buffer, + /// the first `source.count` elements of the buffer's memory must be + /// initialized, or the buffer's `Element` type must be a trivial type. + /// The memory referenced by `source` is uninitialized after the function + /// returns. The buffer must reference enough initialized memory + /// to accommodate `source.count` elements. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// can hold, the returned index is equal to the buffer's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer slice containing the values to move. + /// The memory region underlying `source` must be initialized. The + /// memory regions referenced by `source` and this pointer must not overlap. + /// - Returns: An index one past the index of the last element updated. + public func moveUpdate(fromContentsOf source: Slice) -> Index + + /// Deinitializes every instance in this buffer. + /// + /// The region of memory underlying this buffer must be fully initialized. + /// After calling `deinitialize(count:)`, the memory is uninitialized, + /// but still bound to the `Element` type. + /// + /// - Note: All buffer elements must already be initialized. + /// + /// - Returns: A raw buffer to the same range of memory as this buffer. + /// The range of memory is still bound to `Element`. + public func deinitialize() -> UnsafeMutableRawBufferPointer + + /// Initializes the buffer element at `index` to the given value. + /// + /// The destination element must be uninitialized or the buffer's `Element` + /// must be a trivial type. After a call to `initialize(to:)`, the + /// memory underlying this element of the buffer is initialized. + /// + /// - Parameters: + /// - value: The value used to initialize the buffer element's memory. + /// - index: The index of the element to initialize + public func initializeElement(at index: Index, to value: Element) + + /// Retrieves and returns the buffer element at `index`, + /// leaving that element's memory uninitialized. + /// + /// The memory underlying buffer the element at `index` must be initialized. + /// After calling `moveElement(from:)`, the memory underlying the buffer + /// element at `index` is uninitialized, and still bound to type `Element`. + /// + /// - Parameters: + /// - index: The index of the buffer element to retrieve and deinitialize. + /// - Returns: The instance referenced by this index in this buffer. + public func moveElement(from index: Index) -> Element + + /// Deinitializes the buffer element at `index`. + /// + /// The memory underlying the buffer element at `index` must be initialized. + /// After calling `deinitializeElement()`, the memory underlying the buffer + /// element at `index` is uninitialized, and still bound to type `Element`. + /// + /// - Parameters: + /// - index: The index of the buffer element to deinitialize. + public func deinitializeElement(at index: Index) +} +``` + +##### `UnsafeMutablePointer` + +```swift +extension UnsafeMutablePointer { + /// Update this pointer's initialized memory with the specified number of + /// instances, copied from the given pointer's memory. + /// + /// The region of memory starting at this pointer and covering `count` + /// instances of the pointer's `Pointee` type must be initialized or + /// `Pointee` must be a trivial type. After calling + /// `update(from:count:)`, the region is initialized. + /// + /// - Note: Returns without performing work if `self` and `source` are equal. + /// + /// - Parameters: + /// - source: A pointer to at least `count` initialized instances of type + /// `Pointee`. The memory regions referenced by `source` and this + /// pointer may overlap. + /// - count: The number of instances to copy from the memory referenced by + /// `source` to this pointer's memory. `count` must not be negative. + public func update(from source: UnsafePointer, count: Int) + + /// Update this pointer's initialized memory by moving the specified number + /// of instances the source pointer's memory, leaving the source memory + /// uninitialized. + /// + /// The region of memory starting at this pointer and covering `count` + /// instances of the pointer's `Pointee` type must be initialized or + /// `Pointee` must be a trivial type. After calling + /// `moveUpdate(from:count:)`, the region is initialized and the memory + /// region `source..<(source + count)` is uninitialized. + /// + /// - Note: The source and destination memory regions must not overlap. + /// + /// - Parameters: + /// - source: A pointer to the values to be moved. The memory region + /// `source..<(source + count)` must be initialized. The memory regions + /// referenced by `source` and this pointer must not overlap. + /// - count: The number of instances to move from `source` to this + /// pointer's memory. `count` must not be negative. + public func moveUpdate(from source: UnsafeMutablePointer, count: Int) +``` + +##### `UnsafeMutableRawPointer` + +```swift +extension UnsafeMutableRawPointer { + /// Initializes the memory referenced by this pointer with the given value, + /// binds the memory to the value's type, and returns a typed pointer to the + /// initialized memory. + /// + /// The memory referenced by this pointer must be uninitialized or + /// initialized to a trivial type, and must be properly aligned for + /// accessing `T`. + /// + /// The following example allocates raw memory for one instance of `UInt`, + /// and then uses the `initializeMemory(as:to:)` method + /// to initialize the allocated memory. + /// + /// let bytePointer = UnsafeMutableRawPointer.allocate( + /// byteCount: MemoryLayout.stride, + /// alignment: MemoryLayout.alignment) + /// let int8Pointer = bytePointer.initializeMemory(as: UInt.self, to: 0) + /// + /// // After using 'int8Pointer': + /// int8Pointer.deallocate() + /// + /// After calling this method on a raw pointer `p`, the region starting at + /// `self` and continuing up to `p + MemoryLayout.stride` is bound + /// to type `T` and initialized. If `T` is a nontrivial type, you must + /// eventually deinitialize the memory in this region to avoid memory leaks. + /// + /// - Parameters: + /// - type: The type to which this memory will be bound. + /// - value: The value used to initialize this memory. + /// - Returns: A typed pointer to the memory referenced by this raw pointer. + public func initializeMemory(as type: T.Type, to value: T) -> UnsafeMutablePointer +} +``` + +##### `UnsafeMutableRawBufferPointer` + +```swift +extension UnsafeMutableRawBufferPointer { + /// Initializes the buffer's memory with every element of the source, + /// binding the initialized memory to the elements' type. + /// + /// When calling the `initializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer must be uninitialized, or initialized + /// to a trivial type. The buffer must reference enough memory to store + /// `source.count` elements, and its `baseAddress` must be properly aligned + /// for accessing `C.Element`. + /// + /// This method initializes the buffer with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A collection of elements to be used to + /// initialize the buffer's storage. + /// - Returns: A typed buffer containing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this buffer, and its count is equal to `source.count` + public func initializeMemory( + as: C.Element.Type, fromContentsOf source: C + ) -> UnsafeMutableBufferPointer + where C: Collection + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer, leaving the source memory + /// uninitialized and this buffer's memory initialized. + /// + /// When calling the `moveInitializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer must be uninitialized, or initialized + /// to a trivial type. The buffer must reference enough memory to store + /// `source.count` elements, and its `baseAddress` must be properly aligned + /// for accessing `C.Element`. After the method returns, + /// the memory referenced by the returned buffer is initialized and the + /// memory region underlying `source` is uninitialized. + /// + /// This method initializes the buffer with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A buffer containing the values to copy. + /// The memory region underlying `source` must be initialized. + /// The memory regions referenced by `source` and this buffer may overlap. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this buffer, and its count is equal to `source.count`. + public func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + + /// Moves every element of an initialized source buffer slice into the + /// uninitialized memory referenced by this buffer, leaving the source memory + /// uninitialized and this buffer's memory initialized. + /// + /// When calling the `moveInitializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer must be uninitialized, or initialized + /// to a trivial type. The buffer must reference enough memory to store + /// `source.count` elements, and its `baseAddress` must be properly aligned + /// for accessing `C.Element`. After the method returns, + /// the memory referenced by the returned buffer is initialized and the + /// memory region underlying `source` is uninitialized. + /// + /// This method initializes the buffer with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A buffer containing the values to copy. + /// The memory region underlying `source` must be initialized. + /// The memory regions referenced by `source` and this buffer may overlap. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this buffer, and its count is equal to `source.count`. + public func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: Slice> + ) -> UnsafeMutableBufferPointer +} +``` + + + +For `Slice` of typed buffers, the functions need to add an additional generic parameter, which is immediately restricted in the `where` clause. This is necessary because "parameterized extensions" are not yet a Swift feature. Eventually, these functions should be able to have exactly the same generic signatures as the counterpart function on their `UnsafeBufferPointer`-family base. This change will be neither source-breaking nor ABI-breaking. + +##### `Slice` + +```swift +extension Slice { + /// Executes the given closure while temporarily binding the memory referenced + /// by this buffer slice to the given type. + /// + /// Use this method when you have a buffer of memory bound to one type and + /// you need to access that memory as a buffer of another type. Accessing + /// memory as type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// The number of instances of `T` referenced by the rebound buffer may be + /// different than the number of instances of `Element` referenced by the + /// original buffer slice. The number of instances of `T` will be calculated + /// at runtime. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. Every instance of `Pointee` overlapping with a given + /// instance of `T` should have the same initialization state (i.e. + /// initialized or uninitialized.) Accessing a `T` whose underlying + /// `Pointee` storage is in a mixed initialization state shall be + /// undefined behaviour. + /// + /// Because this range of memory is no longer bound to its `Element` type + /// while the `body` closure executes, do not access memory using the + /// original buffer slice from within `body`. Instead, + /// use the `body` closure's buffer argument to access the values + /// in memory as instances of type `T`. + /// + /// After executing `body`, this method rebinds memory back to the original + /// `Element` type. + /// + /// - Note: Only use this method to rebind the buffer's memory to a type + /// that is layout compatible with the currently bound `Element` type. + /// The stride of the temporary type (`T`) may be an integer multiple + /// or a whole fraction of `Element`'s stride. + /// To bind a region of memory to a type that does not match these + /// requirements, convert the buffer to a raw buffer and use the + /// `bindMemory(to:)` method. + /// If `T` and `Element` have different alignments, this buffer's + /// `baseAddress` must be aligned with the larger of the two alignments. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// buffer. The type `T` must be layout compatible + /// with the pointer's `Element` type. + /// - body: A closure that takes a typed buffer to the + /// same memory as this buffer, only bound to type `T`. The buffer + /// parameter contains a number of complete instances of `T` based + /// on the capacity of the original buffer and the stride of `Element`. + /// The closure's buffer argument is valid only for the duration of the + /// closure's execution. If `body` has a return value, that value + /// is also used as the return value for the `withMemoryRebound(to:_:)` + /// method. + /// - buffer: The buffer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, _ body: (UnsafeBufferPointer) throws -> Result + ) rethrows -> Result + where Base == UnsafeBufferPointer +} +``` + +##### `Slice>` + +```swift +extension Slice { + /// Initializes every element in this buffer slice's memory to + /// a copy of the given value. + /// + /// The destination memory must be uninitialized or the buffer's `Element` + /// must be a trivial type. After a call to `initialize(repeating:)`, the + /// entire region of memory referenced by this buffer slice is initialized. + /// + /// - Parameter repeatedValue: The value with which to initialize this + /// buffer slice's memory. + public func initialize(repeating repeatedValue: Element) + where Base == UnsafeMutableBufferPointer + + /// Initializes the buffer slice's memory with the given elements. + /// + /// Prior to calling the `initialize(from:)` method on a buffer slice, + /// the memory it references must be uninitialized, + /// or the `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer slice up to, but not including, + /// the returned index is initialized. + /// The buffer must contain sufficient memory to accommodate + /// `source.underestimatedCount`. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer slice, which is one past the last element written. + /// If `source` contains no elements, the returned index is equal to + /// the buffer's `startIndex`. If `source` contains an equal or greater number + /// of elements than the buffer slice can hold, the returned index is equal to + /// the buffer's `endIndex`. + /// + /// - Parameter source: A sequence of elements with which to initialize the + /// buffer. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and an index to the next uninitialized element in the buffer. + public func initialize( + from source: S + ) -> (unwritten: S.Iterator, index: Index) + where S: Sequence, Base == UnsafeMutableBufferPointer + + /// Initializes the buffer slice's memory with with every element of the source. + /// + /// Prior to calling the `initialize(fromContentsOf:)` method on a buffer slice, + /// the memory it references must be uninitialized, + /// or the `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer slice up to, but not including, + /// the returned index is initialized. + /// + /// The returned index is the index of the next uninitialized element + /// in the buffer slice, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to + /// the buffer's `startIndex`. If `source` contains as many elements + /// as the buffer slice can hold, the returned index is equal to + /// to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A collection of elements to be used to + /// initialize the buffer's storage. + /// - Returns: An index to the next uninitialized element in the + /// buffer slice, or `endIndex`. + public func initialize( + fromContentsOf source: C + ) -> Index + where C : Collection, Base == UnsafeMutableBufferPointer + + /// Updates every element of this buffer slice's initialized memory. + /// + /// The buffer slice’s memory must be initialized or its `Element` + /// must be a trivial type. + /// + /// - Note: All buffer elements must already be initialized. + /// + /// - Parameters: + /// - repeatedValue: The value used when updating this pointer's memory. + public func update(repeating repeatedValue: Element) + where Base == UnsafeMutableBufferPointer + + /// Updates the buffer slice's initialized memory with the given elements. + /// + /// The buffer slice's memory must be initialized or its `Element` type + /// must be a trivial type. + /// + /// - Parameter source: A sequence of elements to be used to update + /// the buffer's contents. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and the index one past the last updated element in the buffer. + public func update( + from source: S + ) -> (unwritten: S.Iterator, index: Index) + where S: Sequence, Base == UnsafeMutableBufferPointer + + /// Updates the buffer slice's initialized memory with every element of the source. + /// + /// Prior to calling the `update(fromContentsOf:)` method on a buffer + /// slice, the first `source.count` elements of the referenced memory must be + /// initialized, or the buffer's `Element` type must be a trivial type. + /// The buffer must reference enough initialized memory to accommodate + /// `source.count` elements. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// slice can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A collection of elements to be used to update + /// the buffer's contents. + /// - Returns: An index one past the index of the last element updated. + public func update( + fromContentsOf source: C + ) -> Index + where C: Collection, Base == UnsafeMutableBufferPointer + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer slice, leaving the + /// source memory uninitialized and this buffer slice's memory initialized. + /// + /// Prior to calling the `moveInitialize(fromContentsOf:)` method on a + /// buffer slice, the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer slice up to, but not including, + /// the returned index is initialized. The memory referenced by + /// `source` is uninitialized after the function returns. + /// The buffer slice must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer slice, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// slice's `startIndex`. If `source` contains as many elements as the slice + /// can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to copy. + /// The memory region underlying `source` must be initialized. The memory + /// regions referenced by `source` and this buffer may overlap. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + public func moveInitialize( + fromContentsOf source: UnsafeMutableBufferPointer + ) -> Index + where Base == UnsafeMutableBufferPointer + + /// Moves every element of an initialized source buffer slice into the + /// uninitialized memory referenced by this buffer slice, leaving the + /// source memory uninitialized and this buffer slice's memory initialized. + /// + /// Prior to calling the `moveInitialize(fromContentsOf:)` method on a + /// buffer slice, the memory it references must be uninitialized, + /// or its `Element` type must be a trivial type. After the call, + /// the memory referenced by the buffer slice up to, but not including, + /// the returned index is initialized. The memory referenced by + /// `source` is uninitialized after the function returns. + /// The buffer slice must reference enough memory to accommodate + /// `source.count` elements. + /// + /// The returned index is the position of the next uninitialized element + /// in the buffer slice, one past the index of the last element written. + /// If `source` contains no elements, the returned index is equal to the + /// slice's `startIndex`. If `source` contains as many elements as the slice + /// can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer slice containing the values to copy. + /// The memory region underlying `source` must be initialized. The memory + /// regions referenced by `source` and this buffer slice may overlap. + /// - Returns: An index to the next uninitialized element in the buffer, + /// or `endIndex`. + public func moveInitialize( + fromContentsOf source: Slice> + ) -> Index + where Base == UnsafeMutableBufferPointer + + /// Updates this buffer slice's initialized memory initialized memory by + /// moving every element from the source buffer, + /// leaving the source memory uninitialized. + /// + /// The region of memory starting at the beginning of this buffer slice and + /// covering `source.count` instances of its `Element` type must be + /// initialized, or `Element` must be a trivial type. After calling + /// `moveUpdate(fromContentsOf:)`, + /// the region of memory underlying `source` is uninitialized. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// slice can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer containing the values to move. + /// The memory region underlying `source` must be initialized. The memory + /// regions referenced by `source` and this buffer slice must not overlap. + /// - Returns: An index one past the index of the last element updated. + public func moveUpdate( + fromContentsOf source: UnsafeMutableBufferPointer + ) -> Index + where Base == UnsafeMutableBufferPointer + + /// Updates this buffer slice's initialized memory initialized memory by + /// moving every element from the source buffer slice, + /// leaving the source memory uninitialized. + /// + /// The region of memory starting at the beginning of this buffer slice and + /// covering `source.count` instances of its `Element` type must be + /// initialized, or `Element` must be a trivial type. After calling + /// `moveUpdate(fromContentsOf:)`, + /// the region of memory underlying `source` is uninitialized. + /// + /// The returned index is one past the index of the last element updated. + /// If `source` contains no elements, the returned index is equal to the + /// buffer's `startIndex`. If `source` contains as many elements as the buffer + /// slice can hold, the returned index is equal to the slice's `endIndex`. + /// + /// - Precondition: `self.count` >= `source.count` + /// + /// - Parameter source: A buffer slice containing the values to move. + /// The memory region underlying `source` must be initialized. The memory + /// regions referenced by `source` and this buffer slice must not overlap. + /// - Returns: An index one past the index of the last element updated. + public func moveUpdate( + fromContentsOf source: Slice> + ) -> Index + where Base == UnsafeMutableBufferPointer + + /// Deinitializes every instance in this buffer slice. + /// + /// The region of memory underlying this buffer slice must be fully + /// initialized. After calling `deinitialize(count:)`, the memory + /// is uninitialized, but still bound to the `Element` type. + /// + /// - Note: All buffer elements must already be initialized. + /// + /// - Returns: A raw buffer to the same range of memory as this buffer. + /// The range of memory is still bound to `Element`. + public func deinitialize() -> UnsafeMutableRawBufferPointer + where Base == UnsafeMutableBufferPointer + + /// Initializes the element at `index` to the given value. + /// + /// The memory underlying the destination element must be uninitialized, + /// or `Element` must be a trivial type. After a call to `initialize(to:)`, + /// the memory underlying this element of the buffer slice is initialized. + /// + /// - Parameters: + /// - value: The value used to initialize the buffer element's memory. + /// - index: The index of the element to initialize + public func initializeElement(at index: Int, to value: Element) + where Base == UnsafeMutableBufferPointer + + /// Updates the initialized element at `index` to the given value. + /// + /// The memory underlying the destination element must be initialized, + /// or `Element` must be a trivial type. This method is equivalent to: + /// + /// self[index] = value + /// + /// - Parameters: + /// - value: The value used to update the buffer element's memory. + /// - index: The index of the element to update + public func updateElement(at index: Index, to value: Element) + where Base == UnsafeMutableBufferPointer + + /// Retrieves and returns the element at `index`, + /// leaving that element's underlying memory uninitialized. + /// + /// The memory underlying the element at `index` must be initialized. + /// After calling `moveElement(from:)`, the memory underlying this element + /// of the buffer slice is uninitialized, and still bound to type `Element`. + /// + /// - Parameters: + /// - index: The index of the buffer element to retrieve and deinitialize. + /// - Returns: The instance referenced by this index in this buffer. + public func moveElement(from index: Index) -> Element + where Base == UnsafeMutableBufferPointer + + /// Deinitializes the memory underlying the element at `index`. + /// + /// The memory underlying the element at `index` must be initialized. + /// After calling `deinitializeElement()`, the memory underlying this element + /// of the buffer slice is uninitialized, and still bound to type `Element`. + /// + /// - Parameters: + /// - index: The index of the buffer element to deinitialize. + public func deinitializeElement(at index: Base.Index) + where Base == UnsafeMutableBufferPointer + + /// Executes the given closure while temporarily binding the memory referenced + /// by this buffer slice to the given type. + /// + /// Use this method when you have a buffer of memory bound to one type and + /// you need to access that memory as a buffer of another type. Accessing + /// memory as type `T` requires that the memory be bound to that type. A + /// memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// The number of instances of `T` referenced by the rebound buffer may be + /// different than the number of instances of `Element` referenced by the + /// original buffer slice. The number of instances of `T` will be calculated + /// at runtime. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. Every instance of `Pointee` overlapping with a given + /// instance of `T` should have the same initialization state (i.e. + /// initialized or uninitialized.) Accessing a `T` whose underlying + /// `Pointee` storage is in a mixed initialization state shall be + /// undefined behaviour. + /// + /// Because this range of memory is no longer bound to its `Element` type + /// while the `body` closure executes, do not access memory using the + /// original buffer slice from within `body`. Instead, + /// use the `body` closure's buffer argument to access the values + /// in memory as instances of type `T`. + /// + /// After executing `body`, this method rebinds memory back to the original + /// `Element` type. + /// + /// - Note: Only use this method to rebind the buffer's memory to a type + /// that is layout compatible with the currently bound `Element` type. + /// The stride of the temporary type (`T`) may be an integer multiple + /// or a whole fraction of `Element`'s stride. + /// To bind a region of memory to a type that does not match these + /// requirements, convert the buffer to a raw buffer and use the + /// `bindMemory(to:)` method. + /// If `T` and `Element` have different alignments, this buffer's + /// `baseAddress` must be aligned with the larger of the two alignments. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// buffer. The type `T` must be layout compatible + /// with the pointer's `Element` type. + /// - body: A closure that takes a ${Mutable.lower()} typed buffer to the + /// same memory as this buffer, only bound to type `T`. The buffer + /// parameter contains a number of complete instances of `T` based + /// on the capacity of the original buffer and the stride of `Element`. + /// The closure's buffer argument is valid only for the duration of the + /// closure's execution. If `body` has a return value, that value + /// is also used as the return value for the `withMemoryRebound(to:_:)` + /// method. + /// - buffer: The buffer temporarily bound to `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, _ body: (UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result + where Base == UnsafeMutableBufferPointer +} +``` + +##### `Slice` + +```swift +extension Slice where Base: UnsafeRawBufferPointer { + + /// Binds this buffer’s memory to the specified type and returns a typed buffer + /// of the bound memory. + /// + /// Use the `bindMemory(to:)` method to bind the memory referenced + /// by this buffer to the type `T`. The memory must be uninitialized or + /// initialized to a type that is layout compatible with `T`. If the memory + /// is uninitialized, it is still uninitialized after being bound to `T`. + /// + /// - Warning: A memory location may only be bound to one type at a time. The + /// behavior of accessing memory as a type unrelated to its bound type is + /// undefined. + /// + /// - Parameters: + /// - type: The type `T` to bind the memory to. + /// - Returns: A typed buffer of the newly bound memory. The memory in this + /// region is bound to `T`, but has not been modified in any other way. + /// The typed buffer references `self.count / MemoryLayout.stride` instances of `T`. + public func bindMemory(to type: T.Type) -> UnsafeBufferPointer + + /// Executes the given closure while temporarily binding the buffer to + /// instances of type `T`. + /// + /// Use this method when you have a buffer to raw memory and you need + /// to access that memory as instances of a given type `T`. Accessing + /// memory as a type `T` requires that the memory be bound to that type. + /// A memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. The memory underlying any individual instance of `T` + /// must have the same initialization state (i.e. initialized or + /// uninitialized.) Accessing a `T` whose underlying memory + /// is in a mixed initialization state shall be undefined behaviour. + /// + /// If the byte count of the original buffer is not a multiple of + /// the stride of `T`, then the re-bound buffer is shorter + /// than the original buffer. + /// + /// After executing `body`, this method rebinds memory back to its original + /// binding state. This can be unbound memory, or bound to a different type. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Note: A raw buffer may represent memory that has been bound to a type. + /// If that is the case, then `T` must be layout compatible with the + /// type to which the memory has been bound. This requirement does not + /// apply if the raw buffer represents memory that has not been bound + /// to any type. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This pointer must be a multiple of this type's alignment. + /// - body: A closure that takes a typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - buffer: The buffer temporarily bound to instances of `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, _ body: (UnsafeBufferPointer) throws -> Result + ) rethrows -> Result + + /// Returns a typed buffer to the memory referenced by this buffer, + /// assuming that the memory is already bound to the specified type. + /// + /// Use this method when you have a raw buffer to memory that has already + /// been bound to the specified type. The memory starting at this pointer + /// must be bound to the type `T`. Accessing memory through the returned + /// pointer is undefined if the memory has not been bound to `T`. To bind + /// memory to `T`, use `bindMemory(to:capacity:)` instead of this method. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Parameter to: The type `T` that the memory has already been bound to. + /// - Returns: A typed pointer to the same memory as this raw pointer. + public func assumingMemoryBound(to type: T.Type) -> UnsafeBufferPointer + + /// Returns a new instance of the given type, read from the + /// specified offset into the buffer pointer slice's raw memory. + /// + /// The memory at `offset` bytes into this buffer pointer slice + /// must be properly aligned for accessing `T` and initialized to `T` or + /// another type that is layout compatible with `T`. + /// + /// You can use this method to create new values from the underlying + /// buffer pointer's bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the next four bytes. + /// + /// let a = someBytes[0..<4].load(as: Int32.self) + /// let b = someBytes[4..<8].load(as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the + /// memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// - Parameters: + /// - offset: The offset into the slice's memory, in bytes, at + /// which to begin reading data for the new instance. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer + /// slice's memory. + public func load(fromByteOffset offset: Int = 0, as type: T.Type) -> T + + /// Returns a new instance of the given type, read from the + /// specified offset into the buffer pointer slice's raw memory. + /// + /// This function only supports loading trivial types. + /// A trivial type does not contain any reference-counted property + /// within its in-memory stored representation. + /// The memory at `offset` bytes into the buffer slice must be laid out + /// identically to the in-memory representation of `T`. + /// + /// You can use this method to create new values from the buffer pointer's + /// underlying bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the fourth through seventh bytes. + /// + /// let a = someBytes[..<4].loadUnaligned(as: Int32.self) + /// let b = someBytes[3...].loadUnaligned(as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the + /// memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// - Parameters: + /// - offset: The offset into the slice's memory, in bytes, at + /// which to begin reading data for the new instance. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer's + /// memory. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T +} +``` + +##### `Slice` + +```swift +extension Slice where Base == UnsafeMutableRawBufferPointer { + + /// Copies the bytes from the given buffer to this buffer slice's memory. + /// + /// If the `source.count` bytes of memory referenced by this buffer are bound + /// to a type `T`, then `T` must be a trivial type, the underlying pointer + /// must be properly aligned for accessing `T`, and `source.count` must be a + /// multiple of `MemoryLayout.stride`. + /// + /// The memory referenced by `source` may overlap with the memory referenced + /// by this buffer. + /// + /// After calling `copyMemory(from:)`, the first `source.count` bytes of + /// memory referenced by this buffer are initialized to raw bytes. If the + /// memory is bound to type `T`, then it contains values of type `T`. + /// + /// - Parameter source: A buffer of raw bytes. `source.count` must + /// be less than or equal to this buffer slice's `count`. + public func copyMemory(from source: UnsafeRawBufferPointer) + + /// Copies from a collection of `UInt8` into this buffer slice's memory. + /// + /// If the `source.count` bytes of memory referenced by this buffer are bound + /// to a type `T`, then `T` must be a trivial type, the underlying pointer + /// must be properly aligned for accessing `T`, and `source.count` must be a + /// multiple of `MemoryLayout.stride`. + /// + /// After calling `copyBytes(from:)`, the first `source.count` bytes of memory + /// referenced by this buffer are initialized to raw bytes. If the memory is + /// bound to type `T`, then it contains values of type `T`. + /// + /// - Parameter source: A collection of `UInt8` elements. `source.count` must + /// be less than or equal to this buffer slice's `count`. + public func copyBytes(from source: C) where C.Element == UInt8 + + /// Initializes the memory referenced by this buffer with the given value, + /// binds the memory to the value's type, and returns a typed buffer of the + /// initialized memory. + /// + /// The memory referenced by this buffer must be uninitialized or + /// initialized to a trivial type, and must be properly aligned for + /// accessing `T`. + /// + /// After calling this method on a raw buffer with non-nil `baseAddress` `b`, + /// the region starting at `b` and continuing up to + /// `b + self.count - self.count % MemoryLayout.stride` is bound to type `T` and + /// initialized. If `T` is a nontrivial type, you must eventually deinitialize + /// or move the values in this region to avoid leaks. If `baseAddress` is + /// `nil`, this function does nothing and returns an empty buffer pointer. + /// + /// - Parameters: + /// - type: The type to bind this buffer’s memory to. + /// - repeatedValue: The instance to copy into memory. + /// - Returns: A typed buffer of the memory referenced by this raw buffer. + /// The typed buffer contains `self.count / MemoryLayout.stride` + /// instances of `T`. + func initializeMemory( + as type: T.Type, repeating repeatedValue: T + ) -> UnsafeMutableBufferPointer + + /// Initializes the buffer's memory with the given elements, binding the + /// initialized memory to the elements' type. + /// + /// When calling the `initializeMemory(as:from:)` method on a buffer `b`, + /// the memory referenced by `b` must be uninitialized or initialized to a + /// trivial type, and must be properly aligned for accessing `S.Element`. + /// The buffer must contain sufficient memory to accommodate + /// `source.underestimatedCount`. + /// + /// This method initializes the buffer with elements from `source` until + /// `source` is exhausted or, if `source` is a sequence but not a + /// collection, the buffer has no more room for its elements. After calling + /// `initializeMemory(as:from:)`, the memory referenced by the returned + /// `UnsafeMutableBufferPointer` instance is bound and initialized to type + /// `S.Element`. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A sequence of elements with which to initialize the buffer. + /// - Returns: An iterator to any elements of `source` that didn't fit in the + /// buffer, and a typed buffer of the written elements. The returned + /// buffer references memory starting at the same base address as this + /// buffer. + public func initializeMemory( + as type: S.Element.Type, from source: S + ) -> (unwritten: S.Iterator, initialized: UnsafeMutableBufferPointer) + + /// Initializes the buffer slice's memory with every element of the source, + /// binding the initialized memory to the elements' type. + /// + /// When calling the `initializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer slice must be uninitialized, + /// or initialized to a trivial type. The buffer slice must reference + /// enough memory to store `source.count` elements, and it + /// must be properly aligned for accessing `C.Element`. + /// + /// This method initializes the buffer with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A collection of elements to be used to + /// initialize the buffer's storage. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this slice, and its count is equal to `source.count` + public func initializeMemory( + as type: C.Element.Type, + fromContentsOf source: C + ) -> UnsafeMutableBufferPointer + + /// Moves every element of an initialized source buffer into the + /// uninitialized memory referenced by this buffer slice, leaving + /// the source memory uninitialized and this slice's memory initialized. + /// + /// When calling the `moveInitializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer slice must be uninitialized, + /// or initialized to a trivial type. The buffer slice must reference + /// enough memory to store `source.count` elements, and it must be properly + /// aligned for accessing `C.Element`. After the method returns, + /// the memory referenced by the returned buffer is initialized and the + /// memory region underlying `source` is uninitialized. + /// + /// This method initializes the buffer slice with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type `C.Element` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer slice, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A buffer referencing the values to copy. + /// The memory region underlying `source` must be initialized. + /// The memory regions referenced by `source` and this slice may overlap. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this slice, and its count is equal to `source.count`. + public func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + + /// Moves every element from an initialized source buffer slice into the + /// uninitialized memory referenced by this buffer slice, leaving + /// the source memory uninitialized and this slice's memory initialized. + /// + /// When calling the `moveInitializeMemory(as:fromContentsOf:)` method, + /// the memory referenced by the buffer slice must be uninitialized, + /// or initialized to a trivial type. The buffer slice must reference + /// enough memory to store `source.count` elements, and it must be properly + /// aligned for accessing `C.Element`. After the method returns, + /// the memory referenced by the returned buffer is initialized and the + /// memory region underlying `source` is uninitialized. + /// + /// This method initializes the buffer slice with the contents of `source` + /// until `source` is exhausted. + /// After calling `initializeMemory(as:fromContentsOf:)`, the memory + /// referenced by the returned `UnsafeMutableBufferPointer` instance is bound + /// to the type of `T` and is initialized. This method does not change + /// the binding state of the unused portion of the buffer slice, if any. + /// + /// - Parameters: + /// - type: The type of element to which this buffer's memory will be bound. + /// - source: A buffer referencing the values to copy. + /// The memory region underlying `source` must be initialized. + /// The memory regions referenced by `source` and this buffer may overlap. + /// - Returns: A typed buffer referencing the initialized elements. + /// The returned buffer references memory starting at the same + /// base address as this slice, and its count is equal to `source.count`. + public func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: Slice> + ) -> UnsafeMutableBufferPointer + + /// Binds this buffer’s memory to the specified type and returns a typed buffer + /// of the bound memory. + /// + /// Use the `bindMemory(to:)` method to bind the memory referenced + /// by this buffer to the type `T`. The memory must be uninitialized or + /// initialized to a type that is layout compatible with `T`. If the memory + /// is uninitialized, it is still uninitialized after being bound to `T`. + /// + /// - Warning: A memory location may only be bound to one type at a time. The + /// behavior of accessing memory as a type unrelated to its bound type is + /// undefined. + /// + /// - Parameters: + /// - type: The type `T` to bind the memory to. + /// - Returns: A typed buffer of the newly bound memory. The memory in this + /// region is bound to `T`, but has not been modified in any other way. + /// The typed buffer references `self.count / MemoryLayout.stride` instances of `T`. + public func bindMemory(to type: T.Type) -> UnsafeMutableBufferPointer + + /// Executes the given closure while temporarily binding the buffer to + /// instances of type `T`. + /// + /// Use this method when you have a buffer to raw memory and you need + /// to access that memory as instances of a given type `T`. Accessing + /// memory as a type `T` requires that the memory be bound to that type. + /// A memory location may only be bound to one type at a time, so accessing + /// the same memory as an unrelated type without first rebinding the memory + /// is undefined. + /// + /// Any instance of `T` within the re-bound region may be initialized or + /// uninitialized. The memory underlying any individual instance of `T` + /// must have the same initialization state (i.e. initialized or + /// uninitialized.) Accessing a `T` whose underlying memory + /// is in a mixed initialization state shall be undefined behaviour. + /// + /// If the byte count of the original buffer is not a multiple of + /// the stride of `T`, then the re-bound buffer is shorter + /// than the original buffer. + /// + /// After executing `body`, this method rebinds memory back to its original + /// binding state. This can be unbound memory, or bound to a different type. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Note: A raw buffer may represent memory that has been bound to a type. + /// If that is the case, then `T` must be layout compatible with the + /// type to which the memory has been bound. This requirement does not + /// apply if the raw buffer represents memory that has not been bound + /// to any type. + /// + /// - Parameters: + /// - type: The type to temporarily bind the memory referenced by this + /// pointer. This pointer must be a multiple of this type's alignment. + /// - body: A closure that takes a typed pointer to the + /// same memory as this pointer, only bound to type `T`. The closure's + /// pointer argument is valid only for the duration of the closure's + /// execution. If `body` has a return value, that value is also used as + /// the return value for the `withMemoryRebound(to:capacity:_:)` method. + /// - buffer: The buffer temporarily bound to instances of `T`. + /// - Returns: The return value, if any, of the `body` closure parameter. + public func withMemoryRebound( + to type: T.Type, _ body: (UnsafeMutableBufferPointer) throws -> Result + ) rethrows -> Result + + /// Returns a typed buffer to the memory referenced by this buffer, + /// assuming that the memory is already bound to the specified type. + /// + /// Use this method when you have a raw buffer to memory that has already + /// been bound to the specified type. The memory starting at this pointer + /// must be bound to the type `T`. Accessing memory through the returned + /// pointer is undefined if the memory has not been bound to `T`. To bind + /// memory to `T`, use `bindMemory(to:capacity:)` instead of this method. + /// + /// - Note: The buffer's base address must match the + /// alignment of `T` (as reported by `MemoryLayout.alignment`). + /// That is, `Int(bitPattern: self.baseAddress) % MemoryLayout.alignment` + /// must equal zero. + /// + /// - Parameter to: The type `T` that the memory has already been bound to. + /// - Returns: A typed pointer to the same memory as this raw pointer. + public func assumingMemoryBound(to type: T.Type) -> UnsafeMutableBufferPointer + + /// Returns a new instance of the given type, read from the + /// specified offset into the buffer pointer slice's raw memory. + /// + /// The memory at `offset` bytes into this buffer pointer slice + /// must be properly aligned for accessing `T` and initialized to `T` or + /// another type that is layout compatible with `T`. + /// + /// You can use this method to create new values from the underlying + /// buffer pointer's bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the next four bytes. + /// + /// let a = someBytes[0..<4].load(as: Int32.self) + /// let b = someBytes[4..<8].load(as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the + /// memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// - Parameters: + /// - offset: The offset into the slice's memory, in bytes, at + /// which to begin reading data for the new instance. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer + /// slice's memory. + public func load(fromByteOffset offset: Int = 0, as type: T.Type) -> T + + /// Returns a new instance of the given type, read from the + /// specified offset into the buffer pointer slice's raw memory. + /// + /// This function only supports loading trivial types. + /// A trivial type does not contain any reference-counted property + /// within its in-memory stored representation. + /// The memory at `offset` bytes into the buffer slice must be laid out + /// identically to the in-memory representation of `T`. + /// + /// You can use this method to create new values from the buffer pointer's + /// underlying bytes. The following example creates two new `Int32` + /// instances from the memory referenced by the buffer pointer `someBytes`. + /// The bytes for `a` are copied from the first four bytes of `someBytes`, + /// and the bytes for `b` are copied from the fourth through seventh bytes. + /// + /// let a = someBytes[..<4].loadUnaligned(as: Int32.self) + /// let b = someBytes[3...].loadUnaligned(as: Int32.self) + /// + /// The memory to read for the new instance must not extend beyond the + /// memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// - Parameters: + /// - offset: The offset into the slice's memory, in bytes, at + /// which to begin reading data for the new instance. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + /// - Returns: A new instance of type `T`, copied from the buffer pointer's + /// memory. + public func loadUnaligned(fromByteOffset offset: Int = 0, as type: T.Type) -> T + + /// Stores a value's bytes into the buffer pointer slice's raw memory at the + /// specified byte offset. + /// + /// The type `T` to be stored must be a trivial type. The memory must also be + /// uninitialized, initialized to `T`, or initialized to another trivial + /// type that is layout compatible with `T`. + /// + /// The memory written to must not extend beyond + /// the memory region represented by the buffer pointer slice---that is, + /// `offset + MemoryLayout.size` must be less than or equal + /// to the slice's `count`. + /// + /// After calling `storeBytes(of:toByteOffset:as:)`, the memory is + /// initialized to the raw bytes of `value`. If the memory is bound to a + /// type `U` that is layout compatible with `T`, then it contains a value of + /// type `U`. Calling `storeBytes(of:toByteOffset:as:)` does not change the + /// bound type of the memory. + /// + /// - Note: A trivial type can be copied with just a bit-for-bit copy without + /// any indirection or reference-counting operations. Generally, native + /// Swift types that do not contain strong or weak references or other + /// forms of indirection are trivial, as are imported C structs and enums. + /// + /// If you need to store into memory a copy of a value of a type that isn't + /// trivial, you cannot use the `storeBytes(of:toByteOffset:as:)` method. + /// Instead, you must know either initialize the memory or, + /// if you know the memory was already bound to `type`, assign to the memory. + /// + /// - Parameters: + /// - value: The value to store as raw bytes. + /// - offset: The offset in bytes into the buffer pointer slice's memory + /// to begin writing bytes from the value. The default is zero. + /// - type: The type to use for the newly constructed instance. The memory + /// must be initialized to a value of a type that is layout compatible + /// with `type`. + public func storeBytes(of value: T, toByteOffset offset: Int = 0, as type: T.Type) +} +``` + +## Source compatibility + +This proposal consists mostly of additions, which are by definition source compatible. + +The proposal includes the renaming of four existing functions from `assign` to `update`. The existing function names would be deprecated, producing a warning. A fixit will support an easy transition to the renamed versions of these functions. + +The proposal also includes the addition of labels to the tuple return value of one function. This is technically source breaking. We have tested the change against the source compatibility suite and found no issue. + +## Effect on ABI stability + +The functions proposed here are generally small wrappers around existing functionality. They are implemented as `@_alwaysEmitIntoClient` functions, which means they have no ABI impact. + +The renamed functions can reuse the existing symbol, while the deprecated functions can forward using an `@_alwaysEmitIntoClient` stub to support the functionality under its previous name. The renamings would therefore have no ABI impact. + +In the case of the added tuple labels, we can avoid an ABI impact by reusing the mangled symbol of the previously-existing function that had an unlabeled return tuple type. + +## Effect on API resilience + +All functionality implemented as `@_alwaysEmitIntoClient` will back-deploy. Renamed functions that reuse a previous symbol will also back-deploy. + + +## Alternatives considered + +##### Single element update functions + +An earlier version of this proposal included single-element update functions, `UnsafeMutablePointer.update(to:)` and `UnsafeMutableBufferPointer.updateElement(at:to:)`. These are synonyms for the setters of `UnsafeMutablePointer.pointee` and `UnsafeMutableBufferPointer.subscript(_ i: Index)`, respectively. They were intended to improve the documentation for that operation, in particular the often overlooked initialization requirement. + +##### Renaming `assign` to `update` + +The renaming of `assign` to `update` could be omitted entirely, although we believe the word "update" communicates the API's intent much better than does the word "assign". In _The Swift Programming Language_ (_TSPL_,) the `=` symbol is named "the assignment operator", and its function is described as to either "initialize" or "update" a value. The current name (`assign`) seemingly conflates the two roles of `=` as described in *TSPL*, while the proposed name (`update`) builds on _TSPL_. + +There are only four current symbols to be renamed by this proposal, and their replacements are easily migrated by a fixit. For context, this renaming would change only 6 lines of code in the standard library, outside of the function definitions. If the renaming is omitted, the four new functions proposed in the family should use the name `assign` as well. The two single-element versions would be `assign(_ value:)` and `assignElement(at:_ value:)`. + +##### Element-by-element copies from `Collection` inputs + +The initialization and updating functions that copy from `Collection` inputs use the argument label `fromContentsOf`. This is a different label than used by the pre-existing functions that copy from `Sequence` inputs. We could use the same argument label (`from`) as with the `Sequence` inputs, but that would mean that we must return the `Iterator` for the `Collection` versions, and that is not necessarily desirable, especially if a particular `Iterator` cannot be copied cheaply. If we used the same argument label (`from`) and did not return `Iterator`, then the `Sequence` and `Collection` versions of the `initialize(from:)` would be overloaded by their return type, and that would be source-breaking: +an existing use of the current function that doesn't destructure the returned tuple on assignment could now pick up the `Collection` overload, which would have a return value incompatible with the subsequent code which assumes that the return value is of type `(Iterator, Index)`. + +##### Returned tuple labels + +One of the pre-existing returned tuples does not have element labels, and the original version of the proposal did not change that. New labels are now proposed for this case. This is technically source-breaking, in that if existing source uses exactly the proposed labels in a different position, then the returned tuple value would be shuffled. The chosen labels have sufficiently pointed names that the risk is very small. + +## Acknowledgments + +[Diana Ma](https://github.com/tayloraswift) (aka [Taylor Swift](https://forums.swift.org/u/taylorswift/summary))'s initial versions of the pitch that became SE-0184 included more functions to manipulate initialization state. These were deferred, but much of the deferred functionality has not been pitched again until now. + +Members of the Swift Standard Library team for valuable discussions. diff --git a/proposals/0371-isolated-synchronous-deinit.md b/proposals/0371-isolated-synchronous-deinit.md new file mode 100644 index 0000000000..00b88aca31 --- /dev/null +++ b/proposals/0371-isolated-synchronous-deinit.md @@ -0,0 +1,531 @@ +# Isolated synchronous deinit + +* Proposal: [SE-0371](0371-isolated-synchronous-deinit.md) +* Author: [Mykola Pokhylets](https://github.com/nickolas-pohilets) +* Review Manager: [Frederick Kellison-Linn](https://github.com/jumhyn) +* Status: **Implemented (Swift 6.2)** +* Implementation: [apple/swift#60057](https://github.com/apple/swift/pull/60057) +* Review: ([first pitch](https://forums.swift.org/t/isolated-synchronous-deinit/58177)) ([first review](https://forums.swift.org/t/se-0371-isolated-synchronous-deinit/59754)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0371-isolated-synchronous-deinit/60060)) ([discussion](https://forums.swift.org/t/isolated-synchronous-deinit-2/62565)) ([second pitch](https://forums.swift.org/t/pitch-2-se-0371-isolated-async-deinit/64836)) ([sub-pitch](https://forums.swift.org/t/sub-pitch-task-local-values-in-isolated-synchronous-deinit-and-async-deinit/70060)) ([second review](https://forums.swift.org/t/second-review-se-0371-isolated-synchronous-deinit/73406)) ([accepted with modifications](https://forums.swift.org/t/accepted-with-modifications-se-0371-isolated-synchronous-deinit/74042)) + +## Introduction + +This feature allows `deinit`'s of actors and global-actor isolated classes to access non-sendable isolated state, lifting restrictions imposed by [SE-0327](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md). This is achieved by providing runtime support for hopping onto the relevant actor's executor before executing the `deinit` body. + +## Motivation + +Restrictions imposed by [SE-0327](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md) reduce the usefulness of explicit `deinit`s in actors and global actor isolated types. Workarounds for these limitations may involve creation of `close()`-like methods, or even manual reference counting if the API should be able to serve several clients. + +In cases when `deinit` belongs to a subclass of `UIView` or `UIViewController` which are known to call `dealloc` on the main thread, developers may be tempted to silence the diagnostic by adopting `@unchecked Sendable` in types that are not actually sendable. This undermines concurrency checking by the compiler, and may lead to data races when using incorrectly marked types in other places. + +While this proposal introduces additional control over deinit execution semantics that are necessary in some situations, it also introduces a certain amount of non-determinism to deinitializer execution. Because isolated deinits must potentially be enqueued and executed "later" rather than directly inline, this can cause subtle timing issues in resource reclamation. For example, APIs which require quick and predictable resource cleanup, such as scarce resources such as e.g. file descriptors or connections, should not be managed using isolated deinitializers, as the exact timing of when the resource would be released is non-deterministic, which can lead to subtle timing and resource starvation issues. Instead, for types which require tight control over lifetime and cleanup, one should still prefer using "with-style" APIs (`await withResource { resource }`), explicit `await resource.close()` or non-copyable & non-escapable types. + +## Proposed solution + +Allow users to specify a `deinit` as `isolated deinit`, indicating that the execution of the `deinit` body, destruction of the stored properties and object deallocation should be scheduled on the executor (either that of the actor itself or that of the relevant global actor), if needed. + +Let's consider [examples from SE-0327](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0327-actor-initializers.md#data-races-in-deinitializers): + +In the case of several instances with shared data isolated on a common actor, the problem is completely eliminated: + +```swift +class NonSendableAhmed { + var state: Int = 0 +} + +@MainActor +class Maria { + let friend: NonSendableAhmed + + init() { + self.friend = NonSendableAhmed() + } + + init(sharingFriendOf otherMaria: Maria) { + // While the friend is non-Sendable, this initializer and + // and the otherMaria are isolated to the MainActor. That is, + // they share the same executor. So, it's OK for the non-Sendable value + // to cross between otherMaria and self. + self.friend = otherMaria.friend + } + + isolated deinit { + // Used to be a potential data race. Now, deinit is also + // isolated on the MainActor, so this code is perfectly + // correct. + friend.state += 1 + } +} + +func example() async { + let m1 = await Maria() + let m2 = await Maria(sharingFriendOf: m1) + doSomething(m1, m2) +} +``` + +In the case of escaping `self`, the race condition is eliminated but the problem of dangling reference remains. + +```swift +actor Clicker { + var count: Int = 0 + + func click(_ times: Int) { + for _ in 0..Objective-C boundary in either direction. + +This is still sufficient to create ObjC subclasses in runtime, as KVO does. Or swizzle `dealloc` of the most derived class of an instance. But swizzling `dealloc` of intermediate Swift classes might not work as expected. + +### Isolated synchronous deinit of default actors + +When deinitializing an instance of default actor, runtime attempts to take actor's lock and execute deinit on the current thread. If previous executor was another default actor, it remains locked. So potentially multiple actors can be locked at the same time. This does not lead to deadlocks, because (1) lock is acquired conditionally, without waiting; and (2) object cannot be deinitializer twice, so graph of the deinit calls has no cycles. + +### Interaction with distributed actors + +A `deinit` declared in the code of the distributed actor applies only to the local actor and can be isolated as described above. Remote proxy has an implicit compiler-generated synchronous `deinit` which is never isolated. + +### Interaction with task executor preference + +Ad-hoc job created for isolated synchronous deinit is executed outside a task, so task executor preference does not apply. + +### Task-local values + +Task-local values set by the task/thread that performed last release are blocked inside isolated deinit. +Attempting to read such task-local inside isolated deinit will return a default value without any runtime warnings. +Task-local values set inside the body of the isolated deinit are visible for the corresponding scope. + +This behavior ensures that isolated deinit behaves the same way both when running inline and when hopping, without high runtime costs for copying task-local values. + +The point of the last release of the object can be hard to predict and can be changed by optimizations, leading to different behavior between debug and release builds. +Because of this, developers are discouraged from depending on the set of task-local values available at the point of the last release. +Instead of using task-local values, developers are advised to inject dependencies into deinit using the object's stored properties. +This advice applies to non-isolated deinit as well, but this proposal does not change the behavior of the non-isolated deinit. + +Note that any existing hopping in overridden `retain`/`release` for UIKit classes is unlikely to be aware of task-local values. + +## Source compatibility + +This proposal makes previously invalid code valid. + +## Effect on ABI stability + +This proposal does not change the ABI of existing language features, but does introduce new runtime functions. + +## Effect on API resilience + +Isolation attributes of the `deinit` become part of the public API, but they matter only when inheriting from the class. + +Any changes to the isolation of `deinit` of non-open classes are allowed. + +For open non-@objc classes, it is allowed to change synchronous `deinit` from isolated to nonisolated. Any non-recompiled subclasses will keep calling `deinit` of the superclass on the original actor. Changing `deinit` from nonisolated to isolated or changing identity of the isolating actor is a breaking change. + +For open @objc classes, any change in isolation of the synchronous `deinit` is a breaking change, even changing from isolated to nonisolated. This removes symbol for `__isolated_deallocating_deinit` and clients will fail to link with new framework version. See also [Interaction with ObjC runtime](#interaction-with-objc-runtime). + + + + + + + + + + + + + + + + + + + + + + +
Change|opennon-open
| + @objc + + non-@objc +
remove isolation|breakingokok
add isolation|breakingbreakingok
change actor|breakingbreakingok
+ +Adding an isolation annotation to a `dealloc` method of an imported Objective-C class is a breaking change. Existing Swift subclasses may have `deinit` isolated on different actor, and without recompilation will be calling `[super deinit]` on that actor. When recompiled, subclasses isolating on a different actor will produce a compilation error. Subclasses that had a non-isolated `deinit` (or a `deinit` isolated on the same actor) remain ABI compatible. It is possible to add isolation annotation to UIKit classes now, because currently all their subclasses have nonisolated `deinit`s. + +Removing an isolation annotation from the `dealloc` method (together with `retain`/`release` overrides) is a breaking change. Any existing subclasses would be type-checked as isolated but compiled without isolation thunks. After changes in the base class, subclass `deinit`s could be called on the wrong executor. + +If isolated deinit need to be suppressed in `.swiftinterface` for compatibility with older compilers, then `open` classes are emitted as `public` to prevent subclassing. + +## Future Directions + +### Implicit asynchronous `deinit` + +Currently, if users need to initiate an asynchronous operation from `deinit`, they need to manually start a task. This requires copying all the needed data from the object, which can be tedious and error-prone. If some data is not copied explicitly, `self` will be captured implicitly, leading to a fatal error in runtime. + +```swift +actor Service { + func shutdown() {} +} + +@MainActor +class ViewModel { + let service: Service + + deinit { + // Incorrect: + _ = Task { await service.shutdown() } + + // Corrected version: + _ = Task { [service] in await service.shutdown() } + } +} +``` + +If almost every instance property is copied, then it would be more efficient to reuse original object as a task closure context and make `deinit` asynchronous: + +```swift + ... + deinit async { + await service.shutdown() + + // Destroy stored properties and deallocate memory + // after asynchronous shutdown is complete + } +} +``` + +Similarly to this proposal, `__deallocating_deinit` can be used as a thunk that starts an unstructured task for executing async deinit. But this is out of scope of this proposal. + +### Linear types + +Invoking sequential async cleanup is a suspension point, and needs to be marked with `await`. Explicit method calls fit this role better than implicitly invoked `deinit`. But using such methods can be error-prone without compiler checks that cleanup method is called exactly once on all code paths. Move-only types help to ensure that cleanup method is called **at most once**. Linear types help to ensure that cleanup method is called **exactly once**. + +```swift +@linear // like @moveonly, but consumption is mandatory +struct Connection { + // acts as a named explicit async deinit + consuming func close() async { + ... + } +} + +func communicate() async { + let c = Connection(...) + // error: value of linear type is not consumed +} +``` + +### Improving de-virtualization and inlining of the executor access. + +Consider the following example: + +```swift +import Foundation + +@_silgen_name("do_it") +@MainActor func doIt() + +public class Foo { + @MainActor + public func foo() async { + doIt() + } + @MainActor + deinit {} +} +``` + +Currently both the `foo()` and `deinit` entry points produce two calls to access the `MainActor.shared.unownedExecutor`, with the second one even using dynamic dispatch. These two calls could be replaced with a single call to the statically referenced `swift_task_getMainExecutor()`. + +```llvm +%1 = tail call swiftcc %swift.metadata_response @"type metadata accessor for Swift.MainActor"(i64 0) #6 +%2 = extractvalue %swift.metadata_response %1, 0 +%3 = tail call swiftcc %TScM* @"static Swift.MainActor.shared.getter : Swift.MainActor"(%swift.type* swiftself %2) +%4 = tail call i8** @"lazy protocol witness table accessor for type Swift.MainActor and conformance Swift.MainActor : Swift.Actor in Swift"() #6 +%5 = bitcast %TScM* %3 to %objc_object* +%6 = tail call swiftcc { i64, i64 } @"dispatch thunk of Swift.Actor.unownedExecutor.getter : Swift.UnownedSerialExecutor"(%objc_object* swiftself %5, %swift.type* %2, i8** %4) +``` + +### Improving extended stack trace support + +Developers who put breakpoints in the isolated deinit might want to see the call stack that led to the last release of the object. Currently, if switching of executors was involved, the release call stack won't be shown in the debugger. + +### Implementing API for synchronously scheduling arbitrary work on the actor + +Added runtime function has calling convention optimized for the `deinit` use case, but using a similar runtime function with a slightly different signature, one could implement an API for synchronously scheduling arbitrary work on the actor: + +```swift +extension Actor { + /// Adds a job to the actor queue that calls `work` passing `self` as an argument. + nonisolated func enqueue(_ work: __owned @Sendable @escaping (isolated Self) -> Void) + + /// If actor's executor is already the current one - executes work immediately + /// Otherwise adds a job to the actor's queue. + nonisolated func executeOrEnqueue(_ work: __owned @Sendable @escaping (isolated Self) -> Void) +} + +actor MyActor { + var k: Int = 0 + func inc() { k += 1 } +} + +let a = MyActor() +a.enqueue { aIsolated in + aIsolated.inc() // no await +} +``` + +## Alternatives considered + +### Placing hopping logic in `swift_release()` instead. + +`UIView` and `UIViewController` implement hopping to the main thread by overriding the `release` method. But in Swift there are no vtable/wvtable slots for releasing, and adding them would also affect a lot of code that does not need isolated deinit. + +### Copy task-local values when hopping by default + +This comes with a performance cost, which is unlikely to be beneficial to most of the users. +Leaving behavior of the task-locals undefined allows to potentially change it in the future, after getting more feedback from the users. + +### Keeping behavior of task-local values undefined + +Approach of 'make no promises' is likely to result in users inadvertently relying on implementation details which would turn out to be difficult to change later. + +### Implicitly propagate isolation to synchronous `deinit`. + +This would be a source-breaking change. + +Majority of the `deinit`'s are implicitly synthesized by the compiler and only release stored properties. Global open source search in Sourcegraph, gives is 77.5k deinit declarations for 2.2m classes - 3.5%. Release can happen from any executor/thread and does not need isolation. Isolating implicit `deinit`s would come with a major performance cost. Providing special rules for propagating isolation to synchronous `deinit` unless it is implicit, would complicate propagation rules. diff --git a/proposals/0372-document-sorting-as-stable.md b/proposals/0372-document-sorting-as-stable.md new file mode 100644 index 0000000000..2449d0ae3c --- /dev/null +++ b/proposals/0372-document-sorting-as-stable.md @@ -0,0 +1,77 @@ +# Document Sorting as Stable + +* Proposal: [SE-0372](0372-document-sorting-as-stable.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift PR #60936](https://github.com/apple/swift/pull/60936) +* Review: ([pitch](https://forums.swift.org/t/pitch-document-sorting-as-stable/59880)) ([review](https://forums.swift.org/t/se-0372-document-sorting-as-stable/60165)) ([acceptance](https://forums.swift.org/t/accepted-se-0372-document-sorting-as-stable/60425)) + +## Introduction + +Swift's sorting algorithm was changed to be stable before Swift 5, but we've never updated the documentation to provide that guarantee. Let's commit to the sorting algorithm being stable so that people can rely on that behavior. + +## Motivation + +A *stable sort* is a sort that keeps the original relative order for any elements that compare as equal or unordered. For example, given this list of players that are already sorted by last name, a sort by first name preserves the original order of the two players named "Ashley": + +```swift +var roster = [ + Player(first: "Sam", last: "Coffey"), + Player(first: "Ashley", last: "Hatch"), + Player(first: "Kristie", last: "Mewis"), + Player(first: "Ashley", last: "Sanchez"), + Player(first: "Sophia", last: "Smith"), +] + +roster.sort(by: { $0.first < $1.first }) +// roster == [ +// Player(first: "Ashley", last: "Hatch"), +// Player(first: "Ashley", last: "Sanchez"), +// Player(first: "Kristie", last: "Mewis"), +// Player(first: "Sam", last: "Coffey"), +// Player(first: "Sophia", last: "Smith"), +// ] +``` + +For users who are unaware that many sorting algorithms aren't stable, an unstable sort can be surprising. Preserving relative order is an expectation set by software like spreadsheets, where sorting by one column, and then another, is a way to complete a sort based on multiple properties. + +Sort stability isn't always observable. When a collection is sorted based on the elements' `Comparable` conformance, like sorting an array of integers, "unordered" elements are typically indistinguishable. In general, sort stability is important when elements are sorted based on a subset of their properties. + +The standard library `sort()` has long been stable, but the documentation explicitly [doesn't make this guarantee](https://github.com/apple/swift/blob/release/5.7/stdlib/public/core/Sort.swift#L40-L41): + +> The sorting algorithm is not guaranteed to be stable. A stable sort preserves the relative order of elements that compare as equal. + +This status quo is a problem — developers who are aware of what stability is cannot rely on the current behavior, and developers who are unaware of stability could be surprised by unexpected bugs if stability were to disappear. Guaranteeing stability would resolve both of these issues. + +## Proposed solution + +Let's change the documentation! Since all current versions of the Swift runtime include a stable sort (which was introduced before ABI stability), this change can be made to the standard library documentation only: + +```diff +- /// The sorting algorithm is not guaranteed to be stable. A stable sort ++ /// The sorting algorithm is guaranteed to be stable. A stable sort + /// preserves the relative order of elements that compare as equal. +``` + +## Source compatibility + +This change codifies the existing standard library behavior, so it is compatible with all existing source code. + +## Effect on ABI stability + +The change to make sorting stable was implemented before ABI stability, so all ABI-stable versions of Swift already provide this behavior. + +## Effect on API resilience + +Making this guarantee explicit requires that any changes to the sort algorithm maintain stability. + +## Alternatives considered + +### Providing an `unstableSort()` + +Discussing the *stability* of the current sort naturally brings up the question of providing an alternative sort that is *unstable*. An unstable sort by itself, however, doesn't provide any specific benefit to users — no one is asking for a sort that mixes up equivalent elements! Instead, users could be interested in sort algorithms that have other characteristics, such as using only an array's existing allocation, that are much faster to implement without guaranteeing stability. If and when proposals for those sort algorithms are introduced, the lack of stability can be addressed through documentation and/or API naming, and having the default sort be stable is still valuable for the reasons listed above. + +## Future directions + +There are a variety of other sorting-related improvements that could be interesting to pursue, including key-path or function-based sorting, sorted collection types or protocols, sort descriptors, and more. These ideas can be explored in future pitches and proposals. diff --git a/proposals/0373-vars-without-limits-in-result-builders.md b/proposals/0373-vars-without-limits-in-result-builders.md new file mode 100644 index 0000000000..9beffe1bde --- /dev/null +++ b/proposals/0373-vars-without-limits-in-result-builders.md @@ -0,0 +1,113 @@ +# Lift all limitations on variables in result builders + +* Proposal: [SE-0373](0373-vars-without-limits-in-result-builders.md) +* Author: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#60839](https://github.com/apple/swift/pull/60839) +* Review: ([pitch](https://forums.swift.org/t/pitch-lift-all-limitations-on-variables-in-result-builders/60460)) ([review](https://forums.swift.org/t/se-0373-lift-all-limitations-on-variables-in-result-builders/60592)) ([acceptance](https://forums.swift.org/t/accepted-se-0373-lift-all-limitations-on-variables-in-result-builders/61041)) + +## Introduction + +The implementation of the result builder transform (introduced by [SE-0289](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md)) places a number of limitations on local variable declarations in the transformed function. Specifically, local variables need to have an initializer expression, they cannot be computed, they cannot have observers, and they cannot have attached property wrappers. None of these restrictions were explicit in the SE-0289 proposal, but they are a *de facto* part of the current feature. + +## Motivation + +The result builder proposal [describes how the result builder transform handles each individual component in a function body](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md#the-result-builder-transform). It states that local declarations are unaffected by the transformation, which implies that any declaration allowed in that context should be supported. That is not the case under the current implementation, which requires that local variables declarations must have a simple name, storage, and an initializing expression. + +In certain circumstances, it's useful to be able to declare a local variable that, for example, declares multiple variables, has default initialization, or has an attached property wrapper (with or without an initializer). Let's take a look at a simple example: + +```swift +func compute() -> (String, Error?) { ... } + +func test(@MyBuilder builder: () -> Int?) { + ... +} + +test { + let (result, error) = compute() + + let outcome: Outcome + + if let error { + // error specific logic + outcome = .failure + } else { + // complex computation + outcome = .success + } + + switch outcome { + ... + } +} +``` + +Both declarations are currently rejected because result builders only allow simple (with just one name) stored variables with an explicit initializer expression. + +Local variable declarations with property wrappers (with or without an explicit initializer) can be utilized for a variety of use-cases, including but not limited to: + +* Verification and/or formatting of the user-provided input + +```swift +import SwiftUI + +struct ContentView: View { + var body: some View { + GeometryReader { proxy in + @Clamped(10...100) var width = proxy.size.width + Text("\(width)") + } + } +} +``` + +* Interacting with user defaults + +```swift +import SwiftUI + +struct AppIntroView: View { + var body: some View { + @UserDefault(key: "user_has_ever_interacted") var hasInteracted: Bool + ... + Button("Browse Features") { + ... + hasInteracted = true + } + Button("Create Account") { + ... + hasInteracted = true + } + } +} +``` + + +## Proposed solution + +I propose to treat local variable declarations in functions transformed by result builders as if they appear in an ordinary function without any additional restrictions. + +## Detailed design + +The change is purely semantic, without any new syntax. It allows variables of all of these kinds to be declared in a function that will be transformed by a result builder: + +* uninitialized variables (only if supported by the builder, see below for more details) +* default-initialized variables (e.g. variables with optional type) +* computed variables +* observed variables +* variables with property wrappers +* `lazy` variables + +These variables will be treated just like they are treated in regular functions. All of the ordinary semantic checks to verify their validity will still be performed, and invalid declarations (based on the standard rules) will still be rejected by the compiler. + +There is one notable exception to this general rule. Initializing a variable after its declaration requires writing an assignment to it, and assignments require the result builder to support `Void` results, as described in [SE-0289](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0289-result-builders.md#assignments). If the result builder does not support `Void` results (whether with an explicit `buildExpression` or just by handling them in `buildBlock`), transformed functions will not be allowed to contain uninitialized declarations. + + +## Source compatibility + +This is an additive change which should not affect existing source code. + +## Effect on ABI stability and API resilience + +These changes do not require support from the language runtime or standard library, and they do not change anything about the external interface to the transformed function. diff --git a/proposals/0374-clock-sleep-for.md b/proposals/0374-clock-sleep-for.md new file mode 100644 index 0000000000..f9653cc4e0 --- /dev/null +++ b/proposals/0374-clock-sleep-for.md @@ -0,0 +1,200 @@ +# Add sleep(for:) to Clock + +* Proposal: [SE-0374](0374-clock-sleep-for.md) +* Authors: [Brandon Williams](https://github.com/mbrandonw), [Stephen Celis](https://github.com/stephencelis) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#61222](https://github.com/apple/swift/pull/61222) +* Review: ([pitch](https://forums.swift.org/t/pitch-clock-sleep-for/60376)) ([review](https://forums.swift.org/t/se-0374-add-sleep-for-to-clock/60787)) ([acceptance](https://forums.swift.org/t/accepted-se-0374-add-sleep-for-to-clock/62148)) + +## Introduction + +The `Clock` protocol introduced in Swift 5.7 provides a way to suspend until a future instant, but +does not provide a way to sleep for a duration. This differs from the static `sleep` methods on +`Task`, which provide both a way to sleep until an instant or for a duration. + +This imbalance in APIs might be reason enough to add a `sleep(for:)` method to all clocks, but the +real problem occurs when dealing with `Clock` existentials. Because the `Instant` associated type +is fully erased, and only the `Duration` is preserved via the primary associated type, any API +that deals with instants is inaccessible to an existential. This means one cannot invoke +`sleep(until:)` on an existential clock, and hence you can't really do anything with an existential +clock. + +## Motivation + +Existentials provide a convenient way to inject dependencies into features so that you can use one +kind of dependency in production, and another kind in tests. The most prototypical version of this +is API clients. When you run your feature in production you want the API client to make real life +network requests, but when run in tests you may want it to just return some mock data. + +Due to the current design of `Clock`, it is not possible to inject a clock existential into a +feature so that you can use a `ContinuousClock` in production, but some other kind of controllable +clock in tests. + +For example, suppose you have an observable object for the logic of some feature that wants to show +a welcoming message after waiting 5 seconds. That might look like this: + +```swift +class FeatureModel: ObservableObject { + @Published var message: String? + func onAppear() async { + do { + try await Task.sleep(until: .now.advanced(by: .seconds(5))) + self.message = "Welcome!" + } catch {} + } +} +``` + +If you wrote a test for this, your test suite would have no choice but to wait for 5 real life +seconds to pass before it could make an assertion: + +```swift +let model = FeatureModel() + +XCTAssertEqual(model.message, nil) +await model.onAppear() // Waits for 5 seconds +XCTAssertEqual(model.message, "Welcome!") +``` + +This affects people who don't even write tests. If you put your feature into an Xcode preview, then +you would have to wait for 5 full seconds to pass before you get to see the welcome message. That +means you can't quickly iterate on the styling of that message. + +The solution to these problems is to not reach out to the global, uncontrollable `Task.sleep`, and +instead inject a clock into the feature. And that is typically done using an existential, but +unfortunately that does not work: + +```swift +class FeatureModel: ObservableObject { + @Published var message: String? + let clock: any Clock + + func onAppear() async { + do { + try await self.clock.sleep(until: self.clock.now.advanced(by: .seconds(5))) // 🛑 + self.message = "Welcome!" + } catch {} + } +} +``` + +One cannot invoke `sleep(until:)` on a clock existential because the `Instant` has been fully +erased, and so there is no way to access `.now` and advance it. + +For similar reasons, one cannot invoke `Task.sleep(until:clock:)` with a clock existential: + +```swift +try await Task.sleep(until: self.clock.now.advanced(by: .seconds(5)), clock: self.clock) // 🛑 +``` + +What we need instead is the `sleep(for:)` method on clocks that allow you to sleep for a duration +rather than sleeping until an instant: + +```swift +class FeatureModel: ObservableObject { + @Published var message: String? + let clock: any Clock + + func onAppear() async { + do { + try await self.clock.sleep(for: .seconds(5)) // ✅ + self.message = "Welcome!" + } catch {} + } +} +``` + +Without a `sleep(for:)` method on clocks, one cannot use a clock existential in the feature, and +that forces you to introduce a generic: + +```swift +class FeatureModel>: ObservableObject { + @Published var message: String? + let clock: C + + func onAppear() async { + do { + try await self.clock.sleep(until: self.clock.now.advanced(by: .seconds(5))) + self.message = "Welcome!" + } catch {} + } +} +``` + +But this is problematic. This will force any code that touches `FeatureModel` to also introduce a +generic if you want that code to be testable and controllable. And it's strange that the class +is statically announcing its dependence on a clock when its mostly just an internal detail of the +class. + +By adding a `sleep(for:)` method to `Clock` we can fix all of these problems, and give Swift users +the ability to control time-based asynchrony in their applications. + +## Proposed solution + +A single extension method will be added to the `Clock` protocol: + +```swift +extension Clock { + /// Suspends for the given duration. + /// + /// Prefer to use the `sleep(until:tolerance:)` method on `Clock` if you have access to an + /// absolute instant. + public func sleep( + for duration: Duration, + tolerance: Duration? = nil + ) async throws { + try await self.sleep(until: self.now.advanced(by: duration), tolerance: tolerance) + } +} +``` + +This will allow one to sleep for a duration with a clock rather than sleeping until an instant. + +Further, to make the APIs between `clock.sleep(for:)` and `Task.sleep(for:)` similar, we will also add a `clock` and `tolerance` argument to `Task.sleep(for:)`: + +```swift +extension Task where Success == Never, Failure == Never { + /// Suspends the current task for the given duration on a continuous clock. + /// + /// If the task is cancelled before the time ends, this function throws + /// `CancellationError`. + /// + /// This function doesn't block the underlying thread. + /// + /// try await Task.sleep(for: .seconds(3)) + /// + /// - Parameter duration: The duration to wait. + public static func sleep( + for duration: C.Duration, + tolerance: C.Duration? = nil, + clock: C = ContinuousClock() + ) async throws { + try await sleep(until: clock.now.advanced(by: duration), tolerance: tolerance, clock: clock) + } +} +``` + +And we will add a default value for the `clock` argument of `Task.sleep(until:)`: + +```swift +extension Task where Success == Never, Failure == Never { + public static func sleep( + until deadline: C.Instant, + tolerance: C.Instant.Duration? = nil, + clock: C = ContinuousClock() + ) async throws { + try await clock.sleep(until: deadline, tolerance: tolerance) + } +} +``` + +## Source compatibility, effect on ABI stability, effect on API resilience + +As this is an additive change, it should not have any compatibility, stability or resilience +problems. + +## Alternatives considered + +We could leave things as is, and not add this method to the standard library, as it is possible for +people to define it themselves. diff --git a/proposals/0375-opening-existential-optional.md b/proposals/0375-opening-existential-optional.md new file mode 100644 index 0000000000..e2e5272d39 --- /dev/null +++ b/proposals/0375-opening-existential-optional.md @@ -0,0 +1,66 @@ +# Opening existential arguments to optional parameters + +* Proposal: [SE-0375](0375-opening-existential-optional.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift#61321](https://github.com/apple/swift/pull/61321) +* Review: ([pitch](https://forums.swift.org/t/mini-pitch-for-se-0352-amendment-allow-opening-an-existential-argument-to-an-optional-parameter/60501)) ([review](https://forums.swift.org/t/se-0375-opening-existential-arguments-to-optional-parameters/60802)) ([acceptance](https://forums.swift.org/t/accepted-se-0375-opening-existential-arguments-to-optional-parameters/61045)) + +## Introduction + +[SE-0352 "Implicitly Opened Existentials"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md) has a limitation that prevents the opening of an existential argument when the corresponding parameter is optional. This proposal changes that behavior, so that such a call will succeed when a (non-optional) existential argument is passed to a parameter of optional type: + +```swift +func acceptOptional(_ x: T?) { } +func test(p: any P, pOpt: (any P)?) { + acceptOptional(p) // SE-0352 does not open "p"; this proposal will open "p" and bind "T" to its underlying type + acceptOptional(pOpt) // does not open "pOpt", because there is no "T" to bind to when "pOpt" is "nil" +} +``` + +The rationale for not opening the existential `p` in the first call was to ensure consistent behavior with the second call, in an effort to avoid confusion. SE-0352 says: + +> The case of optionals is somewhat interesting. It's clear that the call `cannotOpen6(pOpt)` cannot work because `pOpt` could be `nil`, in which case there is no type to bind `T` to. We *could* choose to allow opening a non-optional existential argument when the parameter is optional, e.g., +> +> ``` +> cannotOpen6(p1) // we *could* open here, binding T to the underlying type of p1, but choose not to +> ``` +> +> but this proposal doesn't allow this because it would be odd to allow this call but not the `cannotOpen6(pOpt)` call. + +However, experience with implicitly-opened existentials has shown that opening an existential argument in the first case is important, because many functions accept optional parameters. It is possible to work around this limitation, but doing so requires a bit of boilerplate, using a generic function that takes a non-optional parameter as a trampoline to the one that takes an optional parameter: + +```swift +func acceptNonOptionalThunk(_ x: T) { + acceptOptional(x) +} + +func test(p: any P) { + acceptNonOptionalThunk(p) // workaround for SE-0352 to get a call to acceptOptional with opened existential +} +``` + +## Proposed solution + +Allow an argument of (non-optional) existential type to be opened to be passed to an optional parameter: + +```swift +func openOptional(_ value: T?) { } + +func testOpenToOptional(p: any P) { + openOptional(p) // okay, opens 'p' and binds 'T' to its underlying type +} +``` + +## Source compatibility + +Generally speaking, opening an existential argument in one more case will make code that would have been rejected by the compiler (e.g., with an error like "`P` does not conform to `P`") into code that is accepted, because the existential is opened. This can change the behavior of overload resolution, in the same manner as was [discussed in SE-0352](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md#source-compatibility). Experience with SE-0352's integration into Swift 5.7 implies that the practical effect of these changes is quite small. + +## Effect on ABI stability + +This proposal changes the type system but has no ABI impact whatsoever. + +## Effect on API resilience + +This proposal changes the use of APIs, but not the APIs themselves, so it doesn't impact API resilience per se. diff --git a/proposals/0376-function-back-deployment.md b/proposals/0376-function-back-deployment.md new file mode 100644 index 0000000000..396dcc93da --- /dev/null +++ b/proposals/0376-function-back-deployment.md @@ -0,0 +1,213 @@ +# Function Back Deployment + +* Proposal: [SE-0376](0376-function-back-deployment.md) +* Author: [Allan Shortlidge](https://github.com/tshortli) +* Implementation: [apple/swift#41271](https://github.com/apple/swift/pull/41271), [apple/swift#41348](https://github.com/apple/swift/pull/41348), [apple/swift#41416](https://github.com/apple/swift/pull/41416), [apple/swift#41612](https://github.com/apple/swift/pull/41612) as the underscored attribute `@_backDeploy` +* Review Manager: [Frederick Kellison-Linn](https://github.com/jumhyn) +* Review: ([pitch](https://forums.swift.org/t/pitch-function-back-deployment/55769)) ([review](https://forums.swift.org/t/se-0376-function-back-deployment/61015)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0376-function-back-deployment/61507)) ([second review](https://forums.swift.org/t/se-0376-second-review-function-back-deployment/61671)) ([returned for revision (second review)](https://forums.swift.org/t/returned-for-revision-se-0376-second-review-function-back-deployment/62374))([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0376-function-back-deployment/62905)) +* Status: **Implemented (Swift 5.8)** + +## Introduction + +This proposal introduces a `@backDeployed` attribute to allow ABI-stable libraries to make their own public APIs available on older OSes. When a `@backDeployed` API isn't present in the library that ships with an older OS, a client running on that OS can still use the API because a fallback copy of its implementation has been emitted into the client. + +With `@backDeployed`, a function may be emitted into clients as a fallback copy of _itself_. Note that the attribute doesn't mark a function as a fallback implementation of some _other_ function, and therefore it doesn't help one module to extend the availability of APIs declared in some _other_ module. + +## Motivation + +Resilient Swift libraries, such as the ones present in the SDKs for Apple's platforms, are distributed as dynamic libraries. Authors of these libraries use `@available` annotations to indicate the operating system version that a declaration was introduced in. For example, suppose this were the interface of ToastKit, a library that is part of the toasterOS SDK: + +```swift +@available(toasterOS 1.0, *) +public struct BreadSlice { ... } + +@available(toasterOS 1.0, *) +public struct Toast { ... } + +@available(toasterOS 1.0, *) +public struct Toaster { + public func makeToast(_ slice: BreadSlice) -> Toast +} +``` + +In response to developer feedback, the ToastKit authors enhance `Toaster` in toasterOS 2.0 with the capability to make toast in batches: + +```swift +extension Toaster { + @available(toasterOS 2.0, *) + public func makeBatchOfToast(_ slices: [BreadSlice]) -> [Toast] { + var toast: [Toast] = [] + for slice in slices { + toast.append(makeToast(slice)) + } + return toast + } +} +``` + +Unfortunately, developers who wish to both distribute an app compatible with toasterOS 1.0 and also adopt `makeBatchOfToast(_:)` must call the API conditionally to account for its potential unavailability: + +```swift +let slices: [BreadSlice] = ... +if #available(toasterOS 2.0, *) { + let toast = toaster.makeBatchOfToast(slices) + // ... +} else { + // ... do something else, like reimplement makeBatchOfToast(_:) +} +``` + +Considering that the implementation of `makeBatchOfToast(_:)` is self contained and could run unmodified on toasterOS 1.0, it would be ideal if the ToastKit authors had the option to back deploy this new API to older OSes and allow clients to adopt it unconditionally. + +The `@_alwaysEmitIntoClient` attribute is an unofficial Swift language feature that can be used to solve this problem. The bodies of functions with this attribute are emitted into the library's `.swiftinterface` (similarly to `@inlinable` functions) and the compiler makes a local copy of the annotated function in the client module. References to these functions _always_ resolve to a copy in the same module so the function is effectively not a part of the library's ABI. + +While `@_alwaysEmitIntoClient` can be used to back deploy APIs, there are some drawbacks to using it. Since a copy of the function is always emitted, there is code size overhead for every client even if the client's deployment target is new enough that the library API would always be available at runtime. Additionally, if the implementation of the API were to change in order to improve performance, fix a bug, or close a security hole then the client would need to be recompiled against a new SDK before users benefit from those changes. An attribute designed specifically to support back deployment should avoid these drawbacks by ensuring that: + +1. The API implementation from the original library is preferred at runtime when it is available. +2. Fallback copies of the API implementation are absent from clients binaries when they would never be used. + +## Proposed solution + +Add a `@backDeployed(before: ...)` attribute to Swift that can be used to indicate that a copy of the function should be emitted into the client to be used at runtime when executing on an OS prior to the version identified with the `before:` argument. The attribute can be adopted by ToastKit's authors like this: + +```swift +extension Toaster { + @available(toasterOS 1.0, *) + @backDeployed(before: toasterOS 2.0) + public func makeBatchOfToast(_ breadSlices: [BreadSlice]) -> [Toast] { ... } +} +``` + +The API is now available on toasterOS 1.0 and later so clients may now reference `makeBatchOfToast(_:)` unconditionally. The compiler detects applications of `makeBatchOfToast(_:)` and generates code to automatically handle the potentially runtime unavailability of the API. + +## Detailed design + +The `@backDeployed` attribute may apply to functions, methods, and subscripts. Properties may also have the attribute as long as the they do not have storage. The attribute takes a comma separated list of one or more platform versions, so declarations that are available on more than one platform can be back deployed on multiple platforms with a single attribute. The following are examples of legal uses of the attribute: + +```swift +extension Temperature { + @available(toasterOS 1.0, ovenOS 1.0, *) + @backDeployed(before: toasterOS 2.0, ovenOS 2.0) + public var degreesFahrenheit: Double { + return (degreesCelsius * 9 / 5) + 32 + } +} + +extension Toaster { + /// Returns whether the slot at the given index can fit a bagel. + @available(toasterOS 1.0, *) + @backDeployed(before: toasterOS 2.0) + public subscript(fitsBagelsAt index: Int) -> Bool { + get { return index < 2 } + } +} +``` + +### Behavior of back deployed APIs + +When the compiler encounters a call to a back deployed function, it generates and calls a thunk instead that forwards the arguments to either the library copy of the function or a fallback copy of the function. For instance, suppose the client's code looks like this: + +```swift +let toast = toaster.makeBatchOfToast(slices) +``` + +The transformation done by the compiler would effectively result in this: + +```swift +let toast = toaster.makeBatchOfToast_thunk(slices) + +// Compiler generated +extension Toaster { + func makeBatchOfToast_thunk(_ breadSlices: [BreadSlice]) -> [Toast] { + if #available(toasterOS 2.0, *) { + return makeBatchOfToast(breadSlices) // call the original + } else { + return makeBatchOfToast_fallback(breadSlices) // call local copy + } + } + + func makeBatchOfToast_fallback(_ breadSlices: [BreadSlice]) -> [Toast] { + // ... copy of function body from ToastKit + } +} +``` + +When the deployment target of the client app is at least toasterOS 2.0, the compiler can eliminate the branch in `makeBatchOfToast_thunk(_:)` and therefore make `makeBatchOfToast_fallback(_:)` an unused function, which reduces the unnecessary bloat that could otherwise result from referencing a back deployed API. + +### Restrictions on declarations that may be back deployed + +There are rules that limit which declarations may have a `@backDeployed` attribute: + +* The declaration must be `public` or `@usableFromInline` since it only makes sense to offer back deployment for declarations that would be used by other modules. +* Only functions that can be invoked with static dispatch are eligible to back deploy, so back deployed instance and class methods must be `final`. The `@objc` attribute also implies dynamic dispatch and therefore is incompatible with `@backDeployed`. +* The declaration should be available earlier than the platform versions specified in `@backDeployed` (otherwise the fallback functions would never be called). +* The `@_alwaysEmitIntoClient` and `@_transparent` attributes are incompatible with `@backDeployed` because they require the function body to always be emitted into the client, defeating the purpose of `@backDeployed`. +* Declarations with `@inlinable` _may_ use `@backDeployed`. As usual with `@inlinable`, the bodies of these functions may be emitted into the client at the discretion of the optimizer. The copy of the function in the client may therefore be used even when a copy of the function is available in the library. + +### Requirements for the bodies of back deployed functions + +The restrictions on the bodies of back deployed functions are the same as `@inlinable` functions. The body may only reference declarations that are accessible to the client, such as `public` and `@usableFromInline` declarations. Similarly, those referenced declarations must also be at least as available the back deployed function, or `if #available` must be used to handle potential unavailability. Type checking in `@backDeployed` function bodies must ignore the library's deployment target since the body will be copied into clients with unknown deployment targets. + +## Source compatibility + +The introduction of this attribute to the language is an additive change and therefore doesn't affect existing Swift code. + +## Effect on ABI stability + +The `@backDeployed` attribute has no effect on the ABI of Swift libraries. A Swift function with and without a `@backDeployed` attribute has the same ABI; the attribute simply controls whether the compiler automatically generates additional logic in the client module. The thunk and fallback functions that are emitted into the client do have a special mangling to disambiguate them from the original function in the library, but these symbols are never referenced across separately compiled modules. + +## Effect on API resilience + +By itself, adding a `@backDeployed` attribute to a declaration does not affect source compatibility for clients of a library, and neither does removing the attribute. However, adding a `@backDeployed` attribute would typically be done simultaneously with expanding the availability of the declaration by lowering the `introduced:` version in the `@available` attribute. Expansion of the availability of an API is source compatible for clients, but reversing that expansion would not be. + +## Alternatives considered + +### Use a different argument label name + +A few alternative spellings of the argument label `before:` were considered including `upTo:`, `until:`, and `implemented:`. The choice of label is significant because it influences the reader's intuitive understanding of the semantics of the attribute. The label should ideally make the directionality of the effect clear as well as the exclusivity of the OS version range. It also helps if the attribute as a whole reads fluently when expanded into an English sentence like this: + +> The function is back deployed for all minimum deployment targets _before_ iOS 13. + +Reviewers did not consistently agree that any of the labels that were considered successfully clarified the directionality of the effect or the exclusivity of the range but the label `before:` was ultimately deemed the clearest option. + +### Use a different attribute name + +One way to frame the proposed attribute is that it indicates which OS versions the function became ABI stable in. From that perspective, naming the attribute something like `@abi(introduced:)` could make sense. However, by default every public function in an SDK library is already implicitly ABI stable at the `introduced:` version of its availability so it would be reasonable to ask what distinction this attribute is making and why it is not present on every API that is ABI stable. This naming choice would obfuscate the essential effect of the attribute, requiring unfamiliar readers to read the documentation to learn that the purpose of the attribute is to extend the function's availability to earlier deployment targets. + +### Extend @available + +Another possible design for this feature would be to augment the existing `@available` attribute instead of introducing a new attribute. In the following example, a `backDeployBefore:` label is added to the `@available` attribute: + +```swift +extension Toaster { + @available(toasterOS, introduced: 1.0, backDeployBefore: 2.0) + public func makeBatchOfToast(_ breadSlices: [BreadSlice]) -> [Toast] +} +``` + +This design has the advantage of grouping the introduction and back deployment versions together in a single attribute, which may be easier to understand for library authors who want to adopt this capability. However, there are drawbacks: + +- The `@available` attribute's existing responsibilities relate to constraining the contexts in which a declaration can be used. The version in which the declaration became ABI is not an availability constraint, but rather information that the library author provides to the compiler in order to give the declaration extended availability. A client of the library does not need this information in order to understand where the API may be used. It seems wise to avoid further complicating the already complex `@available` attribute with additional responsibilities that do not relate to its core purpose. +- This design would require library authors to use the long form of `@available`, which would lead to increased verbosity for APIs that are available on many different OSes. + +A variant of this alternative design would be to add a `backDeployTo:` label instead and change the meaning of the `introduced:` label to indicate the version of OS that the declaration became ABI stable: + +```swift +extension Toaster { + @available(toasterOS, backDeployTo: 1.0, introduced: 2.0) + public func makeBatchOfToast(_ breadSlices: [BreadSlice]) -> [Toast] +} +``` + +This has the same drawbacks documented above and also further contradicts the principle of progressive disclosure by making it necessary to learn about back deployment as a concept in order to understand where an API declaration may be used. + + +## Future directions + +### Back deployment for other kinds of declarations + +It would also be useful to be able to back deploy the implementations of other types of declarations, such as entire enums, structs, or even protocol conformances. Exploring the feasibility of such a feature is out of scope for this proposal, but whether or not the design can accommodate being extended to other kinds of declarations is important to consider. + +## Acknowledgments + +Thank you to Alexis Laferriere, Ben Cohen, and Xi Ge for their help designing the feature and to Slava Pestov for his assistance with SILGen. diff --git a/proposals/0377-parameter-ownership-modifiers.md b/proposals/0377-parameter-ownership-modifiers.md new file mode 100644 index 0000000000..136860120e --- /dev/null +++ b/proposals/0377-parameter-ownership-modifiers.md @@ -0,0 +1,677 @@ +# `borrowing` and `consuming` parameter ownership modifiers + +* Proposal: [SE-0377](0377-parameter-ownership-modifiers.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm), [Joe Groff](https://github.com/jckarter) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9)** +* Implementation: in main branch of compiler +* Review: ([first pitch](https://forums.swift.org/t/pitch-formally-defining-consuming-and-nonconsuming-argument-type-modifiers/54313)) ([second pitch](https://forums.swift.org/t/borrow-and-take-parameter-ownership-modifiers/59581)) ([first review](https://forums.swift.org/t/se-0377-borrow-and-take-parameter-ownership-modifiers/61020)) ([second review](https://forums.swift.org/t/combined-se-0366-third-review-and-se-0377-second-review-rename-take-taking-to-consume-consuming/61904)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0377-borrowing-and-consuming-parameter-ownership-modifiers/62759)) ([revision and third review](https://forums.swift.org/t/se-0377-revision-make-borrowing-and-consuming-parameters-require-explicit-copying-with-the-copy-operator/64996)) ([revision acceptance](https://forums.swift.org/t/accepted-se-0377-revision-make-borrowing-and-consuming-parameters-require-explicit-copying-with-the-copy-operator/65293)) +* Previous Revisions: ([as of first review](https://github.com/swiftlang/swift-evolution/blob/3f984e6183ce832307bb73ec72c842f6cb0aab86/proposals/0377-parameter-ownership-modifiers.md)) ([as of second review](https://github.com/swiftlang/swift-evolution/blob/7e1d16316e5f68eb94546df9241aa6b4cacb9411/proposals/0377-parameter-ownership-modifiers.md)) + +## Introduction + +We propose new `borrowing` and `consuming` parameter modifiers to allow developers to +explicitly choose the ownership convention that a function uses to receive +immutable parameters. Applying one of these modifiers to a parameter causes +that parameter binding to no longer be implicitly copyable, and potential +copies need to be marked with the new `copy x` operator. This allows for +fine-tuning of performance by reducing the number of ARC calls or copies needed +to call a function, and provides a necessary prerequisite feature for +noncopyable types to specify whether a function consumes a noncopyable value or +not. + +## Motivation + +Swift uses automatic reference counting to manage the lifetimes of reference- +counted objects. There are two broad conventions that the compiler uses to +maintain memory safety when passing an object by value from a caller to a +callee in a function call: + +* The callee can **borrow** the parameter. The caller + guarantees that its argument object will stay alive for the duration of the + call, and the callee does not need to release it (except to balance any + additional retains it performs itself). +* The callee can **consume** the parameter. The callee + becomes responsible for either releasing the parameter or passing ownership + of it along somewhere else. If a caller doesn't want to give up its own + ownership of its argument, it must retain the argument so that the callee + can consume the extra reference count. + +These two conventions generalize to value types, where a "retain" +becomes an independent copy of the value, and "release" the destruction and +deallocation of the copy. By default Swift chooses which convention to use +based on some rules informed by the typical behavior of Swift code: +initializers and property setters are more likely to use their parameters to +construct or update another value, so it is likely more efficient for them to +*consume* their parameters and forward ownership to the new value they construct. +Other functions default to *borrowing* their parameters, since we have found +this to be more efficient in most situations. + +These choices typically work well, but aren't always optimal. +Although the optimizer supports "function signature optimization" that can +change the convention used by a function when it sees an opportunity to reduce +overall ARC traffic, the circumstances in which we can automate this are +limited. The ownership convention becomes part of the ABI for public API, so +cannot be changed once established for ABI-stable libraries. The optimizer +also does not try to optimize polymorphic interfaces, such as non-final class +methods or protocol requirements. If a programmer wants behavior different +from the default in these circumstances, there is currently no way to do so. + +[SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md) +introduces noncopyable types into Swift. Since noncopyable types do not have +the ability to be copied, the distinction between these two conventions becomes +an important part of the API contract: functions that *borrow* noncopyable +values make temporary use of the value and leave it valid for further use, like +reading from a file handle, whereas functions that *consume* a noncopyable +value consume it and prevent its further use, like closing a file handle. +Relying on implicit selection of the parameter convention will not suffice for +these types. + +## Proposed solution + +We give developers direct control over the ownership convention of +parameters by introducing two new parameter modifiers, `borrowing` and +`consuming`. + +## Detailed design + +### Syntax of parameter ownership modifiers + +`borrowing` and `consuming` become contextual keywords inside parameter type +declarations. They can appear in the same places as the `inout` modifier, and +are mutually exclusive with each other and with `inout`. In a `func`, +`subscript`, or `init` declaration, they appear as follows: + +```swift +func foo(_: borrowing Foo) +func foo(_: consuming Foo) +func foo(_: inout Foo) +``` + +In a closure: + +```swift +bar { (a: borrowing Foo) in a.foo() } +bar { (a: consuming Foo) in a.foo() } +bar { (a: inout Foo) in a.foo() } +``` + +In a function type: + +```swift +let f: (borrowing Foo) -> Void = { a in a.foo() } +let f: (consuming Foo) -> Void = { a in a.foo() } +let f: (inout Foo) -> Void = { a in a.foo() } +``` + +Methods can also use the `consuming` or `borrowing` modifier to indicate +respectively that they consume ownership of their `self` parameter or that they +borrow it. These modifiers are mutually exclusive with each other and with the +existing `mutating` modifier: + +```swift +struct Foo { + consuming func foo() // `consuming` self + borrowing func foo() // `borrowing` self + mutating func foo() // modify self with `inout` semantics +} +``` + +`consuming` cannot be applied to parameters of nonescaping closure type, which by +their nature are always borrowed: + +```swift +// ERROR: cannot `consume` a nonescaping closure +func foo(f: consuming () -> ()) { +} +``` + +`consuming` or `borrowing` on a parameter do not affect the caller-side syntax for +passing an argument to the affected declaration, nor do `consuming` or +`borrowing` affect the application of `self` in a method call. For typical +Swift code, adding, removing, or changing these modifiers does not have any +source-breaking effects. (See "related directions" below for interactions with +other language features being considered currently or in the near future which +might interact with these modifiers in ways that cause them to break source.) + +### Ownership convention conversions in protocols and function types + +Protocol requirements can also use `consuming` and `borrowing`, and the modifiers will +affect the convention used by the generic interface to call the requirement. +The requirement may still be satisfied by an implementation that uses different +conventions for parameters of copyable types: + +```swift +protocol P { + func foo(x: consuming Foo, y: borrowing Foo) +} + +// These are valid conformances: + +struct A: P { + func foo(x: Foo, y: Foo) +} + +struct B: P { + func foo(x: borrowing Foo, y: consuming Foo) +} + +struct C: P { + func foo(x: consuming Foo, y: borrowing Foo) +} +``` + +Function values can also be implicitly converted to function types that change +the convention of parameters of copyable types among unspecified, `borrowing`, +or `consuming`: + +```swift +let f = { (a: Foo) in print(a) } + +let g: (borrowing Foo) -> Void = f +let h: (consuming Foo) -> Void = f + +let f2: (Foo) -> Void = h +``` + +These implicit conversions for protocol conformances and function values +are not available for parameter types that are noncopyable, in which case +the convention must match exactly. + +### Using parameter bindings with ownership modifiers + +Inside of a function or closure body, `consuming` parameters may be mutated, as can +the `self` parameter of a `consuming func` method. These +mutations are performed on the value that the function itself took ownership of, +and will not be evident in any copies of the value that might still exist in +the caller. This makes it easy to take advantage of the uniqueness of values +after ownership transfer to do efficient local mutations of the value: + +```swift +extension String { + // Append `self` to another String, using in-place modification if + // possible + consuming func plus(_ other: String) -> String { + // Modify our owned copy of `self` in-place, taking advantage of + // uniqueness if possible + self += other + return self + } +} + +// This is amortized O(n) instead of O(n^2)! +let helloWorld = "hello ".plus("cruel ").plus("world") +``` + +`borrowing` and `consuming` parameter values are also **not implicitly copyable** +inside of the function or closure body: + +```swift +func foo(x: borrowing String) -> (String, String) { + return (x, x) // ERROR: needs to copy `x` +} +func bar(x: consuming String) -> (String, String) { + return (x, x) // ERROR: needs to copy `x` +} +``` + +And so is the `self` parameter within a method that has the method-level +`borrowing` or `consuming` modifier: + +```swift +extension String { + borrowing func foo() -> (String, String) { + return (self, self) // ERROR: needs to copy `self` + } + consuming func bar() -> (String, String) { + return (self, self) // ERROR: needs to copy `self` + } +} +``` + +A value would need to be implicitly copied if: + +- a *consuming operation* is applied to a `borrowing` binding, or +- a *consuming operation* is applied to a `consuming` binding after it has + already been consumed, or while a *borrowing* or *mutating operation* is simultaneously + being performed on the same binding + +where *consuming*, *borrowing*, and *mutating operations* are as described for +values of noncopyable type in +[SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md#using-noncopyable-values). +In essence, disabling implicit copying for a binding makes the binding behave +as if it were of some noncopyable type. + +To allow a copy to occur, the `copy x` operator may be used: + +```swift +func dup(_ x: borrowing String) -> (String, String) { + return (copy x, copy x) // OK, copies explicitly allowed here +} +``` + +`copy x` is a *borrowing operation* on `x` that returns an independently +owned copy of the current value of `x`. The copy may then be independently +consumed or modified without affecting the original `x`. Note that, while +`copy` allows for a copy to occur, it is not a strict +obligation for the compiler to do so; the copy may still be optimized away +if it is deemed semantically unnecessary. + +`copy` is a contextual keyword, parsed as an operator if it is immediately +followed by an identifier on the same line, like the `consume x` operator before +it. In all other cases, `copy` is still treated as a reference to a +declaration named `copy`, as it would have been prior to this proposal. + +The constraint on implicit copies only affects the parameter binding itself. +The value of the parameter may be passed to other functions, or assigned to +other variables (if the convention allows), at which point the value may +be implicitly copied through those other parameter or variable bindings. + +```swift +func foo(x: borrowing String) { + let y = x // ERROR: attempt to copy `x` + bar(z: x) // OK, invoking `bar(z:)` does not require copying `x` +} + +func bar(z: String) { + let w = z // OK, z is implicitly copyable here +} + +func baz(a: consuming String) { + // let aa = (a, a) // ERROR: attempt to copy `a` + + let b = a + let bb = (b, b) // OK, b is implicitly copyable +} +``` + +To clarify the boundary within which the no-implicit-copy constraint applies, a +parameter binding's value *is* noncopyable as part of the *call expression* in +the caller, so if forming the call requires copying, that will raise an error, +even if the parameter would be implicitly copyable in the callee. The function +body serves as the boundary for the no-implicit-copy constraint: + +```swift +struct Bar { + var a: String + var b: String + init(ab: String) { + // OK, ab is implicitly copyable here + a = ab + b = ab + } +} + +func foo(x: borrowing String) { + _ = Bar(ab: x) // ERROR: would need to copy `x` to let `Bar.init` consume it +} +``` + +## Source compatibility + +Adding `consuming` or `borrowing` to a parameter in the language today does not +affect source compatibility with existing code outside of that function. +Callers can continue to call the function as normal, and the function body can +use the parameter as it already does. A method with `consuming` or `borrowing` +modifiers on its parameters can still be used to satisfy a protocol requirement +with different modifiers. Although `consuming` parameter bindings become +mutable, and parameters with either of the `borrowing` or `consuming` modifiers +are not implicitly copyable, the effects are localized to the function +adopting the modifiers. This allows for API authors to use +`consuming` and `borrowing` annotations to fine-tune the copying behavior of +their implementations, without forcing clients to be aware of ownership to use +the annotated APIs. Source-only packages can add, remove, or adjust these +annotations on copyable types over time without breaking their clients. + +Changing parameter modifiers from `borrowing` to `consuming` may however break +source of any client code that also adopts those parameter modifiers, since the +change may affect where copies need to occur in the caller. Going from +`consuming` to `borrowing` however should generally not be source-breaking +for a copyable type. A change in either direction is source-breaking if the +parameter type is noncopyable. + +## Effect on ABI stability + +`consuming` or `borrowing` affects the ABI-level calling convention and cannot be +changed without breaking ABI-stable libraries (except on "trivial types" +for which copying is equivalent to `memcpy` and destroying is a no-op; however, +`consuming` or `borrowing` also has no practical effect on parameters of trivial type). + +## Effect on API resilience + +`consuming` or `borrowing` break ABI for ABI-stable libraries, but are intended to have +minimal impact on source-level API. When using copyable types, adding or +changing these annotations to an API should not affect its existing clients, +except where those clients have also adopted the not-implicitly-copyable +conventions. + +## Alternatives considered + +### Leaving `consuming` parameter bindings immutable inside the callee + +We propose that `consuming` parameters should be mutable inside of the callee, +because it is likely that the callee will want to perform mutations using +the value it has ownership of. There is a concern that some users may find this +behavior unintuitive, since those mutations would not be visible in copies +of the value in the caller. This was the motivation behind +[SE-0003](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0003-remove-var-parameters.md), +which explicitly removed the former ability to declare parameters as `var` +because of this potential for confusion. However, whereas `var` and `inout` +both suggest mutability, and `var` does not provide explicit directionality as +to where mutations become visible, `consuming` on the other hand does not +suggest any kind of mutability to the caller, and it explicitly states the +directionality of ownership transfer. Furthermore, with noncopyable types, the +chance for confusion is moot, because the transfer of ownership means the +caller cannot even use the value after the callee takes ownership anyway. + +Another argument for `consuming` parameters to remain immutable is to serve the +proposal's stated goal of minimizing the source-breaking impact of +parameter ownership modifiers. When `consuming` parameters are mutable, +changing a `consuming` parameter to `borrowing`, or removing the +`consuming` annotation altogether, is potentially source-breaking. However, +any such breakage is purely localized to the callee; callers are still +unaffected (as long as copyable arguments are involved). If a developer wants +to change a `consuming` parameter back into a `borrowing`, they can still assign the +borrowed value to a local variable and use that local variable for local +mutation. + +### Naming + +We have considered several alternative naming schemes for these modifiers: + +- The current implementation in the compiler uses `__shared` and `__owned`, + and we could remove the underscores to make these simply `shared` and + `owned`. These names refer to the way a borrowed parameter receives a + "shared" borrow (as opposed to the "exclusive" borrow on an `inout` + parameter), whereas a consumed parameter becomes "owned" by the callee. + found that the "shared" versus "exclusive" language for discussing borrows, + while technically correct, is unnecessarily confusing for explaining the + model. +- A previous pitch used the names `nonconsuming` and `consuming`. The current + implementation also uses `__consuming func` to notate a method that takes + ownership of its `self` parameter. We think it is better to describe + `borrowing` in terms of what it means, rather than as the opposite of + the other convention. +- The first reviewed revision used `take` instead of `consume`. Along with + `borrow`, `take` arose during [the first review of +SE-0366](https://forums.swift.org/t/se-0366-move-function-use-after-move-diagnostic/59202). + These names also work well as names for operators that explicitly + transfer ownership of a variable or borrow it in place. However, + reviewers observed that `take` is possibly confusing, since it conflicts with + colloquial discussion of function calls "taking their arguments". `consume` + reads about as well while being more specific. +- Reviewers offered `use`, `own`, or `sink` as alternatives to `consume`. + +We think it is helpful to align the naming of these parameter modifiers with +the corresponding `consume` and `borrow` operators (discussed below under +Future Directions), since it helps reinforce the relationship between the +calling conventions and the expression operators: to explicitly transfer +ownership of an argument in a call site to a parameter in a function, use +`foo(consuming x)` at the call site, and use `func foo(_: consuming T)` in the +function declaration. Similarly, to explicitly pass an argument by borrow +without copying, use `foo(borrow x)` at the call site, and `func foo(_: borrowing T)` +in the function declaration. + +### `@noImplicitCopy` attribute + +Instead of having no-implicit-copy behavior be tied to the ownership-related +binding forms and parameter modifiers, we could have an attribute that can +be applied to any binding to say that it should not be implicitly copyable: + +```swift +@noImplicitCopy(self) +func foo(x: @noImplicitCopy String) { + @noImplicitCopy let y = copy x +} +``` + +We had [pitched this possibility](https://forums.swift.org/t/pitch-noimplicitcopy-attribute-for-local-variables-and-function-parameters/61506), +but community feedback rightly pointed out the syntactic weight and noise +of this approach, as well as the fact that, as an attribute, it makes the +ability to control copies feel like an afterthought not well integrated +with the rest of the language. We've decided not to continue in this direction, +since we think that attaching no-implicit-copy behavior to the ownership +modifiers themselves leads to a more coherent design. + +### `copy` as a regular function + +Unlike the `consume x` or `borrow x` operator, copying doesn't have any specific +semantic needs that couldn't be done by a regular function. Instead of an +operator, `copy` could be defined as a regular standard library function: + +```swift +func copy(_ value: T) -> T { + return value +} +``` + +We propose `copy x` as an operator, because it makes the relation to +`consume x` and `borrow x`, and it avoids the issues of polluting the +global identifier namespace and occasionally needing to be qualified as +`Swift.copy` if it was a standard library function. + +### Transitive no-implicit-copy constraint + +The no-implicit-copy constraint for a `borrowing` and `consuming` parameter +only applies to that binding, and is not carried over to other variables +or function call arguments receiving the binding's value. We could also +say that the parameter can only be passed as an argument to another function +if that function's parameter uses the `borrowing` or `consuming` modifier to +keep implicit copies suppressed, or that it cannot be bound to `let` or `var` +bindings and must be bound using one of the borrowing bindings once we have +those. However, we think those additional restrictions would only make the +`borrowing` and `consuming` modifiers harder to adopt, since developers would +only be able to use them in cases where they can introduce them bottom-up from +leaf functions. + +The transitivity restriction also would not really improve +local reasoning; since the restriction is only on *implicit* copies, but +explicit copies are still possible, calling into another function may lead +to that other function performing copies, whether they're implicit or not. +The only way to be sure would be to inspect the callee's implementation. +One of the goals of SE-0377 is to introduce the parameter ownership modifiers +in a way that minimizes disruption to the the rest of a codebase, allowing +for the modifiers to be easily adopted in spots where the added control is +necessary, and a transitivity requirement would interfere with that goal for +little benefit. + +## Related directions + +#### `consume` operator + +[SE-0366](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0366-move-function.md) +introduced an operator that explicitly ends the lifetime of a +variable before the end of its scope. This allows the compiler to reliably +destroy the value of the variable, or transfer ownership, at the point of its +last use, without depending on optimization and vague ARC optimizer rules. +When the lifetime of the variable ends in an argument to a `consume` parameter, +then we can transfer ownership to the callee without any copies: + +```swift +func consume(x: consuming Foo) + +func produce() { + let x = Foo() + consume(x: consume x) + doOtherStuffNotInvolvingX() +} +``` + +#### `borrow` operator + +Relatedly, there are circumstances where the compiler defaults to copying +when it is theoretically possible to borrow, particularly when working with +shared mutable state such as global or static variables, escaped closure +captures, and class stored properties. The compiler does +this to avoid running afoul of the law of exclusivity with mutations. In +the example below, if `callUseFoo()` passed `global` to `useFoo` by borrow +instead of passing a copy, then the mutation of `global` inside of `useFoo` +would trigger a dynamic exclusivity failure (or UB if exclusivity checks +are disabled): + +```swift +var global = Foo() + +func useFoo(x: borrowing Foo) { + // We need exclusive access to `global` here + global = Foo() +} + +func callUseFoo() { + // callUseFoo doesn't know whether `useFoo` accesses global, + // so we want to avoid imposing shared access to it for longer + // than necessary, and we'll pass a copy of the value. This: + useFoo(x: global) + + // will compile more like: + + /* + let globalCopy = copy(global) + useFoo(x: globalCopy) + destroy(globalCopy) + */ +} +``` + +It is difficult for the compiler to conclusively prove that there aren't +potential interfering writes to shared mutable state, so although it may +in theory eliminate the defensive copy if it proves that `useFoo`, it is +unlikely to do so in practice. The developer may know that the program will +not attempt to modify the same object or global variable during a call, +and want to suppress this copy. An explicit `borrow` operator could allow for +this: + +```swift +var global = Foo() + +func useFooWithoutTouchingGlobal(x: borrowing Foo) { + /* global not used here */ +} + +func callUseFoo() { + // The programmer knows that `useFooWithoutTouchingGlobal` won't + // touch `global`, so we'd like to pass it without copying + useFooWithoutTouchingGlobal(x: borrow global) +} +``` + +If `useFooWithoutTouchingGlobal` did in fact attempt to mutate `global` +while the caller is borrowing it, an exclusivity failure would be raised. + +#### Noncopyable types + +The `consuming` versus `borrowing` distinction becomes much more important and +prominent for values that cannot be implicitly copied. +[SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md) +introduces noncopyable types, whose values are never copyable, as well as +attributes that suppress the compiler's implicit copying behavior selectively +for particular variables or scopes. Operations that borrow +a value allow the same value to continue being used, whereas operations that +consume a value destroy it and prevent its continued use. This makes the +convention used for noncopyable parameters a much more important part of their +API contract, since it directly affects whether the value is still available +after the operation: + +```swift +struct FileHandle: ~Copyable { ... } + +// Operations that open a file handle return new FileHandle values +func open(path: FilePath) throws -> FileHandle + +// Operations that operate on an open file handle and leave it open +// borrow the FileHandle +func read(from: borrowing FileHandle) throws -> Data + +// Operations that close the file handle and make it unusable consume +// the FileHandle +func close(file: consuming FileHandle) + +func hackPasswords() throws -> HackedPasswords { + let fd = try open(path: "/etc/passwd") + // `read` borrows fd, so we can continue using it after + let contents = try read(from: fd) + // `close` consumes fd, so we can't use it again + close(fd) + + let moreContents = try read(from: fd) // compiler error: use after consume + + return hackPasswordData(contents) +} +``` + +As such, SE-0390 requires parameters of noncopyable type to explicitly state +whether they are `borrowing` or `consuming`, since there isn't a clear +default that is always safe to assume. + +### `set`/`out` parameter convention + +By making the `borrowing` and `consuming` conventions explicit, we mostly round out +the set of possibilities for how to handle a parameter. `inout` parameters get +**exclusive access** to their argument, allowing them to mutate or replace the +current value without concern for other code. By contrast, `borrowing` parameters +get **shared access** to their argument, allowing multiple pieces of code to +share the same value without copying, so long as none of them mutate the +shared value. A `consuming` parameter consumes a value, leaving nothing behind, but +there still isn't a parameter analog to the opposite convention, which would +be to take an uninitialized argument and populate it with a new value. Many +languages, including C# and Objective-C when used with the "Distributed +Objects" feature, have `out` parameter conventions for this, and the Val +programming language calls this `set`. + +In Swift up to this point, return values have been the preferred mechanism for +functions to pass values back to their callers. This proposal does not propose +to add some kind of `out` parameter, but a future proposal could. + +### `borrowing`, `mutating`, and `consuming` local variables + +Swift currently lacks the ability to form local bindings to part of an +aggregate without copying that part, other than by passing the part as +an argument to a function call. We plan to introduce [`borrow` and `inout` +bindings](https://forums.swift.org/t/pitch-borrow-and-inout-declaration-keywords/62366) +that will provide this functionality, with the same no-implicit-copy constraint +described by this proposal applied to these bindings. + +### Consistency for `inout` parameters and the `self` parameter of `mutating` methods + +`inout` parameters and `mutating` methods have been part of Swift since before +version 1.0, and their existing behavior allows for implicit copying of the +current value of the binding. We can't change the existing language +behavior in Swift 5, but accepting this proposal would leave `inout` parameters +and `mutating self` inconsistent with the new modifiers. There are a few things +we could potentially do about that: + +- We could change the behavior of `inout` and `mutating self` parameters to + make them not implicitly copyable in Swift 6 language mode. +- `inout` is also conspicuous now in not following the `-ing` convention we've + settled on for `consuming`/`borrowing`/`mutating` modifiers. We could introduce + `mutating` as a new parameter modifier spelling, with no-implicit-copy + behavior. + +One consideration is that, whereas `borrowing` and `consuming` are strictly +optional for code that works only with copyable types, and is OK with letting +the compiler manage copies automatically, there is no way to get in-place +mutation through function parameters except via `inout`. Tying +no-implicit-copy behavior to mutating parameters could be seen as a violation +of the "progressive disclosure" goal of these ownership features, since +developers would not be able to avoid interacting with the ownership model when +using `inout` parameters anymore. + +## Acknowledgments + +Thanks to Robert Widmann for the original underscored implementation of +`__owned` and `__shared`: [https://forums.swift.org/t/ownership-annotations/11276](https://forums.swift.org/t/ownership-annotations/11276). + +## Revision history + +The [first reviewed revision](https://github.com/swiftlang/swift-evolution/blob/3f984e6183ce832307bb73ec72c842f6cb0aab86/proposals/0377-parameter-ownership-modifiers.md) +of this proposal used `take` and `taking` as the name of the callee-destroy convention. + +The [second reviewed revision](https://github.com/swiftlang/swift-evolution/blob/e3966645cf07d6103561454574ab3e2cc2b48ee9/proposals/0377-parameter-ownership-modifiers.md) +used the imperative forms, `consume` and `borrow`, as parameter modifiers, +which were changed to the gerunds `consuming` and `borrowing` in review. The +proposal was originally accepted after these revisions. + +The current revision alters the originally-accepted proposal to make it so that +`borrowing` and `consuming` parameter bindings are not implicitly copyable, +and introduces a `copy x` operator that can be used to explicitly allow copies +where needed. diff --git a/proposals/0378-package-registry-auth.md b/proposals/0378-package-registry-auth.md new file mode 100644 index 0000000000..7a9f9f825f --- /dev/null +++ b/proposals/0378-package-registry-auth.md @@ -0,0 +1,449 @@ +# Package Registry Authentication + +* Proposal: [SE-0378](0378-package-registry-auth.md) +* Author: [Yim Lee](https://github.com/yim-lee) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.8)** +* Implementation: [apple/swift-package-manager#5838](https://github.com/apple/swift-package-manager/pull/5838) +* Review: ([pitch](https://forums.swift.org/t/pitch-package-registry-authentication/61047)), ([review](https://forums.swift.org/t/se-0378-swift-package-registry-authentication/61436)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0378-swift-package-registry-authentication/62556)) + +## Introduction + +A package registry may require authentication for some or all of +its API in order to identify user performing the action and authorize +the request accordingly. + +## Motivation + +Common authentication methods used by web services include basic +authentication, access token, and OAuth. SwiftPM supports only basic +authentication today, which limits its abilities to interact with +package registry services. + +## Proposed solution + +We propose to modify the `swift package-registry` command and registry +configuration to add token authentication support. The changes should also +ensure there is flexibility to add other authentication methods in the future. + +The design draws inspiration from [`docker login`](https://docs.docker.com/engine/reference/commandline/login/) and [`npm login`](https://docs.npmjs.com/cli/v8/commands/npm-adduser), +in that there will be a single command for user to verify and persist +registry credentials. + +## Detailed design + +### Changes to `swift package-registry` command + +Instead of the `swift package-registry set` subcommand and the `--login` +and `--password` options as proposed in [SE-0292](0292-package-registry-service.md) originally, +we propose the new `login` and `logout` subcommands for adding/removing +registry credentials. + +#### New `login` subcommand + +Log in to a package registry. SwiftPM will verify the credentials using +the registry service's [`login` API](#login-api). If it returns a successful +response, credentials will be persisted to the operating system's +credential store if supported, or the user-level netrc file otherwise. +The user-level configuration file located at `~/.swiftpm/configuration/registries.json` +will also be updated. + +```manpage +SYNOPSIS + swift package-registry login [options] +OPTIONS: + --username Username + --password Password + + --token Access token + + --no-confirm Allow writing to netrc file without confirmation + --netrc-file Specify the netrc file path + --netrc Use netrc file even in cases where other credential stores are preferred +``` + +`url` should be the registry's base URL (e.g., `https://example-registry.com`). +In case the location of the `login` API is something other than `/login` +(e.g., `https://example-registry.com/api/v1/login`), provide the full URL. + +The URL must be HTTPS. + +The table below shows the supported authentication types and their +required option(s): + +| Authentication Method | Required Option(s) | +| --------------------- | -------------------------- | +| Basic | `--username`, `--password` | +| Token | `--token` | + +The tool will analyze the provided options to determine the authentication +type and prompt (i.e., interactive mode) for the password/token if it +is missing. For example, if only `--username` is present, the tool +assumes basic authentication and prompts for the password. + +For non-interactive mode, simply provide the `--password` or `--token` +option as required or make sure the secret is present in credential storage. + +If the operating system's credential store is not supported, the +tool will prompt user for confirmation before writing credentials +to the less secured netrc file. Use `--no-confirm` to disable +this confirmation. + +To force usage of netrc file instead of the operating system's +credential store, pass the `--netrc` flag. + +##### Example: basic authentication (macOS, interactive) + +```console +> swift package-registry login https://example-registry.com \ + --username jappleseed +Enter password for 'jappleseed': + +Login successful. +Credentials have been saved to the operating system's secure credential store. +``` + +An entry for `example-registry.com` would be added to Keychain. + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic" + }, + ... + }, + ... +} +``` + +##### Example: basic authentication (operating system's credential store not supported, interactive) + +```console +> swift package-registry login https://example-registry.com \ + --username jappleseed +Enter password for 'jappleseed': + +Login successful. + +WARNING: Secure credential store is not supported on this platform. +Your credentials will be written out to netrc file. +Continue? (Yes/No): Yes + +Credentials have been saved to netrc file. +``` + +An entry for `example-registry.com` would be added to the netrc file: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic" + }, + ... + }, + ... +} +``` + +##### Example: basic authentication (use netrc file instead of operating system's credential store, interactive) + +```console +> swift package-registry login https://example-registry.com \ + --username jappleseed + --netrc +Enter password for 'jappleseed': + +Login successful. + +WARNING: You choose to use netrc file instead of the operating system's secure credential store. +Your credentials will be written out to netrc file. +Continue? (Yes/No): Yes + +Credentials have been saved to netrc file. +``` + +An entry for `example-registry.com` would be added to the netrc file: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic" + }, + ... + }, + ... +} +``` + +##### Example: basic authentication (operating system's credential store not supported, non-interactive) + +```console +> swift package-registry login https://example-registry.com \ + --username jappleseed \ + --password alpine \ + --no-confirm + +Login successful. +Credentials have been saved to netrc file. +``` + +An entry for `example-registry.com` would be added to the netrc file: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic" + }, + ... + }, + ... +} +``` + +
+ +##### Example: basic authentication (operating system's credential store not supported, non-interactive, non-default `login` URL) + +```console +> swift package-registry login https://example-registry.com/api/v1/login \ + --username jappleseed \ + --password alpine \ + --no-confirm + +Login successful. +Credentials have been saved to netrc file. +``` + +An entry for `example-registry.com` would be added to the netrc file: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires basic authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "basic", + "loginAPIPath": "/api/v1/login" + }, + ... + }, + ... +} +``` + +##### Example: token authentication + +```console +> swift package-registry login https://example-registry.com \ + --token jappleseedstoken +``` + +An entry for `example-registry.com` would be added to the operating +system's credential store if supported, or the user-level netrc +file otherwise: + +``` +machine example-registry.com +login token +password jappleseedstoken +``` + +`registries.json` would be updated to indicate that `example-registry.com` +requires token authentication: + +```json +{ + "authentication": { + "example-registry.com": { + "type": "token" + }, + ... + }, + ... +} +``` + +#### New `logout` subcommand + +Log out from a registry. Credentials are removed from the operating system's +credential store if supported, and the user-level configuration file +(`registries.json`). + +To avoid accidental removal of sensitive data, netrc file needs to be +updated manually by the user. + +```manpage +SYNOPSIS + swift package-registry logout +``` + +### Changes to registry configuration + +We will introduce a new `authentication` key to the user-level +`registries.json` file, which by default is located at +`~/.swiftpm/configuration/registries.json`. Any package +registry that requires authentication must have a corresponding +entry in this dictionary. + +```json +{ + "registries": { + "[default]": { + "url": "https://example-registry.com" + } + }, + "authentication": { + "example-registry.com": { + "type": , // One of: "basic", "token" + "loginAPIPath": // Optional. Overrides the default API path (i.e., /login). + } + }, + "version": 1 +} +``` + +`type` must be one of the following: +* `basic`: username and password +* `token`: access token + +Credentials are to be specified in the native credential store +of the operating system if supported, otherwise in the user-level +netrc file. (Only macOS Keychain will be supported in the +initial feature release; more might be added in the future.) + +See [credential storage](#credential-storage) for more details on configuring +credentials for each authentication type. + +### Credential storage + +SwiftPM will always use the most secure way to handle credentials +on the platform. In general, this would mean using the operating +system's credential store (e.g., Keychain on macOS). It falls +back to netrc file only if there is no other solution available. + +#### Basic Authentication + +##### macOS Keychain + +Registry credentials should be stored as "Internet password" +items in the macOS Keychain. The "item name" should be the +registry URL, including `https://` (e.g., `https://example-registry.com`). + +##### netrc file (if operating system's credential store is not supported) + +A netrc entry for basic authentication looks as follows: + +``` +machine example-registry.com +login jappleseed +password alpine +``` + +By default, SwiftPM looks for netrc file in the user's +home directory. A custom netrc file can be specified using +the `--netrc-file` option. + +#### Token Authentication + +User can configure access token for a registry as similarly +done for basic authentication, but with `token` as the login/username +and the access token as the password. + +For example, a netrc entry would look like: + +``` +machine example-registry.com +login token +password jappleseedstoken +``` + +### Additional changes in SwiftPM + +1. Only the user-level netrc file will be used. Project-level netrc file will not be supported. +2. SwiftPM will perform lookups in one credential store only. For macOS, it will be Keychain. For all other platforms, it will be the user-level netrc file. +3. The `--disable-keychain` and `--disable-netrc` options will be removed. + +### New package registry service API + +A package registry that requires authentication must implement +the new API endpoint(s) covered in this section. + +#### `login` API + +SwiftPM will send a HTTP `POST` request to this API to validate +user credentials provided by the `login` subcommand. + +The default API path is `/login`, but this [can be overridden](#override-login-url) +by providing the full API URL to the `login` subcommand. + +The API request will include an `Authorization` HTTP header +constructed as follows: + +* Basic authentication: `Authorization: Basic ` +* Token authentication: `Authorization: Bearer ` + +The registry service must return HTTP status `200` in the +response if login is successful, and `401` otherwise. + +In case the registry service does not support an authentication method, +it should return HTTP status `501`. + +SwiftPM will persist user credentials to local credential store +if login is successful. + +## Security + +This proposal moves SwiftPM to use operating system's native credential +store (e.g., macOS Keychain) on supported platforms, which should yield +better security. + +We are also eliminating the use of project-level netrc file. This should +prevent accidental checkin of netrc file and thus leakage of sensitive +information. + +## Impact on existing packages + +This proposal eliminates the project-level netrc file. There should be +no other impact on existing packages. + diff --git a/proposals/0379-opt-in-reflection-metadata.md b/proposals/0379-opt-in-reflection-metadata.md new file mode 100644 index 0000000000..ddbcb87a6c --- /dev/null +++ b/proposals/0379-opt-in-reflection-metadata.md @@ -0,0 +1,244 @@ +# Swift Opt-In Reflection Metadata + +* Proposal: [SE-0379](0379-opt-in-reflection-metadata.md) +* Authors: [Max Ovtsin](https://github.com/maxovtsin) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Returned for revision** +* Implementation: [apple/swift#34199](https://github.com/apple/swift/pull/34199) +* Review: ([first pitch](https://forums.swift.org/t/proposal-opt-in-reflection-metadata/40981)) ([second pitch](https://forums.swift.org/t/pitch-2-opt-in-reflection-metadata/41696)) ([third pitch](https://forums.swift.org/t/pitch-3-opt-in-reflection-metadata/58852)) ([review](https://forums.swift.org/t/se-0379-opt-in-reflection-metadata/61714)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0379-opt-in-reflection-metadata/62390)) + +## Introduction + +This proposal seeks to increase the safety, efficiency, and secrecy of Swift Reflection Metadata by improving the existing mechanism and providing the opportunity to express a requirement on Reflection Metadata in APIs that consume it. + + +## Motivation + +There are two main kinds of Swift metadata emitted by the compiler: + +1. Core Metadata (type metadata records, nominal type descriptors, etc). +2. Reflection metadata (reflection metadata field descriptors). + +Core metadata must constantly be emitted and may only be stripped if provenly not used. (This kind of metadata isn't affected by this proposal.) +On the other hand, reflection metadata contains optional information about declaration fields - their names and references to their types. +The language's runtime features don't use this metadata, and the emission may be skipped if such types aren't passed to reflection-consuming APIs. + + +Currently, there is no way to selectively enable the emission of reflectable metadata for a type or understand if an API consumes reflection metadata under the hood. +Moreover, compiler's flags exist that allow to completely disable emission. + +A developer has two ways right now - either +1. To enable Reflection in full just in case. +2. To try to guess which used APIs consume Reflection, and enable it only for modules that use such APIs. + +Both of those options have flaws. The first one leads to excessive contribution of reflection metadata to binary size and might affects the secrecy of generated code. +The second one isn't safe because many APIs are black boxes. If the developer's guess is wrong, an app might behave not as expected at runtime. + +Furthermore, APIs can use Reflection Metadata differently. Some like `print`, `debugPrint`, and `dump` will still work with disabled reflection, but the output will be limited. +Others, like SwiftUI, rely on it and won't work correctly if the reflection metadata is missing. +While the former can benefit as well, the main focus of this proposal is on the latter. + +A developer can mistakenly turn off Reflection Metadata for a Swift module and won't be warned at compile-time if APIs that consume reflection are used by that module. +An app with such a module won't behave as expected at runtime which may be challenging to notice and track down such bugs back to Reflection. +For instance, SwiftUI implementation uses reflection metadata from user modules to trigger the re-rendering of the view hierarchy when a state has changed. +If for some reason a user module was compiled with metadata generation disabled, changing the state won't trigger that behavior and will cause inconsistency +between state and representation which will make such API less safe since it becomes a runtime issue rather than a compile-time one. + +On the other hand, excessive Reflection metadata may be preserved in a binary even if not used, because there is currently no way to statically determine its usage. +There was an attempt to limit the amount of unused reflection metadata by improving the Dead Code Elimination LLVM pass, but in many cases +it’s still preserved in the binary because it’s referenced by Full Type Metadata. This prevents Reflection Metadata from being stripped. +This unnecessarily increases the binary size and may simplify reverse-engineering. + +Introducing a static compilation check can help to solve both of mentioned issues by adding to the language a way to express the requirement to have Reflection metadata available at runtime. + + +## Proposed solution + +Teaching the Type-checker and IRGen to ensure Reflection metadata is preserved in a binary if reflection-consuming APIs are used, will help to move the problem from runtime to compile time. + +To achieve that, a new marker protocol `Reflectable` will be introduced. Firstly, APIs developers will gain an opportunity to express a dependency on Reflection Metadata through a generic requirement of their functions, which will make such APIs safer. +Secondly, during IRGen, the compiler will be able to selectively emit Reflection symbols for the types that explicitly conform to the `Reflectable` protocol, which will reduce the overhead from reflection symbols for cases when reflection is emitted but not consumed. + +### Case Study 1: + +SwiftUI: +```swift +protocol SwiftUI.View: Reflectable {} +class NSHostingView where Content : View { + init(rootView: Content) { ... } +} +``` +User module: +```swift +import SwiftUI + +struct SomeModel {} + +struct SomeView: SwiftUI.View { + var body: some View { + Text("Hello, World!") + .frame(...) + } +} + +window.contentView = NSHostingView(rootView: SomeView()) +``` +Reflection metadata for `SomeView` will be emitted because it implicitly conforms to `Reflectable` protocol, while for `SomeModel` Reflection metadata won't be emitted. If the user module gets compiled with the reflection metadata disabled, the compiler will emit an error. + + +### Case Study 2: + +Framework: +```swift +public func foo(_ t: T) { ... } +``` +User module: +```swift +struct Bar: Reflectable {} +foo(Bar()) +``` +Reflection metadata for `Bar` will be emitted because it explicitly conforms to Reflectable protocol. Without conformance to Reflectable, an instance of type Bar can't be used on function `foo`. If the user module gets compiled with the reflection metadata disabled, the compiler will emit an error. + + +### Conditional and Force casts (`as? Reflectable`, `as! Reflectable`, `is Reflectable`) + +We also propose to allow conditional and force casts to the `Reflectable` protocol, which would succeed only if Reflection Metadata related to a type is available at runtime. This would allow developers to explicitly check if reflection metadata is present and based on that fact branch the code accordingly. + +```swift +public func conditionalUse(_ t: T) { + if let _t = t as? Reflectable { // Consume Reflection metadata + } else { // Back to default implementation } +} + +public func forceUse(_ t: T) { + debugPrint(t as! Reflectable) // Will crash if reflection metadata isn't available +} + +public func testIsReflectable(_ t: T) -> Bool { + return t is Reflectable // returns True if reflection is available +} +``` + +### Behavior change for Swift 6 + +Starting with Swift 6, we propose to enable Opt-In mode by default, to make the user experience consistent and safe. +However, if full reflection isn't enabled with a new flag (`-enable-full-reflection-metadata`), the emission of reflection metadata will be skipped for all types that don't conform to the `Reflectable` protocol. +This may cause changes in the behavior of the code that wasn't audited to conform to Reflectable and uses reflection-consuming APIs. + +For instance, stdlib's APIs like `dump`, `debugPrint`, `String(describing:)` will be returning limited output. +Library authors will have to prepare their APIs for Swift 6 and introduce generic requirements on `Reflectable` in their APIs. + +We also propose to deprecate the compiler's options that can lead to missing reflection - `-reflection-metadata-for-debugger-only` and `-disable-reflection-metadata` and starting with Swift 6, ignore these arguments in favor of the default opt-in mode. + + +### No stdlib behavior changes + +In Swift `Mirror(reflecting:)` is the only official way to access Reflection metadata, all other APIs are using it under the hood. +We intentionally do not propose adding a Reflectable constraint on Mirror type, because it would impose restrictions on those developers who still don't want to require it and consume Reflection optionally. +If the presence of reflection metadata is mandatory, the requirement on Reflectable protocol should be expressed in the signatures of calling functions. + + +## Detailed design + +To decide when to emit reflection metadata IRGen will check the conformance of a type to the `Reflectable` protocol and if the type conforms, IRGen will emit reflection symbols. + +Conformance to Reflectable should be only allowed at type declarations level, to avoid confusing behavior, when a developer adds conformance on an imported from another module type that doesn't have reflection enabled. + +Transitive conformance to Reflectable should be allowed to give API authors an opportunity to hide reflection logic from APIs users as implementation details. + +```swift +// Library +public protocol Foo: Reflectable {} +public func consume(_ t: T) {} + +// User +struct Bar: Foo {} // Reflection is emitted +consume(Bar()) +``` + +### Changes for debugging + +Since Reflection metadata might be used by the debugger, we propose to always keep that metadata +if full emission of debugging information is enabled (with `-gdwarf-types` or `-g` flags). +However, such Reflection metadata won't be accessible through the nominal type descriptor +which will avoid inconsistencies in API behavior between Release and Debug modes. + +### Changes in flags + +To handle behavior change between Swift pre-6 and 6, we can introduce a new upcoming feature, +which will allow to enable Opt-In mode explicitly for Swift pre-6 with `-enable-upcoming-feature OptInReflection` and will set this mode by default in Swift 6. + +A new flag `-enable-full-reflection-metadata` will also have to be introduced to allow developers to enable reflection in full if they desire in Swift 6 and later. + +For Swift 6, flags `-disable-reflection-metadata` and `-emit-reflection-for-debugger` will be a no-op, to ensure the reflection metadata is always available when needed. + +1. Reflection Disabled (`-disable-reflection-metadata` and `-reflection-metadata-for-debugger-only`) +- For Swift pre-6 emit Reflection metadata only if full debugging information is enabled. +- If there is a type in a module conforming to `Reflectable`, the compiler will emit an error. +- A no-op in Swift 6 and later (Opt-in mode is enabled by default). + +2. Opt-In Reflection (`-enable-upcoming-feature OptInReflection`) +- If debugging is disabled, emit only for types that conform to `Reflectable`. +- Will emit reflection in full for all types if debugging is enabled. +- For Swift pre-6 will require an explicit flag, for Swift 6 will be enabled by default. + +3. Fully enabled (`-enable-full-reflection-metadata`) +- Always emit reflection metadata for all types regardless of debugging information level. +- Conformance to Reflectable will be synthesized for all types to allow usage of reflection-consuming APIs. +- Current default level for Swift pre-6. + +Introducing a new flag to control the feature will allow us to safely roll it out and avoid breakages of the existing code. For those modules that get compiled with fully enabled metadata, nothing will change (all symbols will stay present). For modules that have the metadata disabled, but are consumers of reflectable API, the compiler will emit the error enforcing the guarantee. + +### Casts implementation + +Casting might be a good way to improve the feature's ergonomics because currently there is no way to check if reflection is available at runtime. (`Mirror.children.count` doesn't really help because it doesn't distinguish between the absence of reflection metadata and the absence of fields on a type) + +To implement this feature, we propose to introduce a new runtime function `swift_reflectableCast`, and emit a call to it instead of `swift_dynamicCast`during IRGen if Reflectable is a target type. + +Because of the fact that the compiler emits a call to that function at compile-time, all casts must be statically visible. +All other cases like implicit conversion to `Reflectable` must be banned. +This could be done at CSSimplify, when a new conversion constraint is introduced between a type variable and `Reflectable` type, the compiler will emit an error. + +```swift +func cast(_ x: U) -> T { + return x as! T +} +let a = cast(1) as Reflectable // expression can't be implicitly converted to Reflectable; use 'as? Reflectable' or 'as! Reflectable' instead +let b: Reflectable = cast(1) // expression can't be implicitly converted to Reflectable; use 'as? Reflectable' or 'as! Reflectable' instead +``` +Some diagnostics and optimizations will also have to be disabled even if conformance is statically visible to the compiler because all casts will have to go through the runtime call. + +**Availability checks** +Since reflectable casting will require a new runtime function, it should be gated by availability checks. If a deployment target is lower than supported, an error will be emitted. However, it might be possible to embed new runtime functions into a compatibility library for back deployment. + + +## Source compatibility + +The change won’t break source compatibility in versions prior to Swift 6, because of the gating by the new flag. +If as proposed, it’s enabled by default in Swift 6, the code with types that has not been audited to conform to the `Reflectable` protocol will fail to compile if used with APIs that consume the reflection metadata. + + +## Effect on ABI stability + +`Reflectable` is a marker protocol, which doesn't have a runtime representation, has no requirements and doesn't affect ABI. + +## Effect on API resilience + +This proposal has no effect on API resilience. + +## Alternatives considered + +Dead Code Elimination and linker optimisations were also considered as a way to reduce the amount of present Reflection metadata in release builds. +The optimiser could use a conformance to a `Reflectable` protocol as a hint about what reflection metadata should be preserved. +However, turned out it was quite challenging to statically determine all usages of Reflection metadata even with hints. + +It was also considered to use an attribute `@reflectable` on nominal type declaration to express the requirement to have reflection metadata, however, a lot of logic had to be re-implemented outside of type-checker to ensure all guarantees are fulfilled. + +## Future directions + +Currently, there is only one kind of Reflection Metadata - Field Descriptor Metadata. In the future, it is possible that other kinds will be added (e.g methods, computed properties, etc) `Reflectable` should be able to cover all of them. +If this proposal is approved, it will become easier and more native to migrate Codable to the usage of Reflection metadata for encoding/decoding logic instead of autogenerating code at compile time. + +## Acknowledgments + +Thanks to [Joe Groff](https://github.com/jckarter) for various useful pieces of advice, general help along the way, and for suggesting several useful features like Reflectable casts! diff --git a/proposals/0380-if-switch-expressions.md b/proposals/0380-if-switch-expressions.md new file mode 100644 index 0000000000..a4ad2fd3db --- /dev/null +++ b/proposals/0380-if-switch-expressions.md @@ -0,0 +1,478 @@ +# `if` and `switch` expressions + +* Proposal: [SE-0380](0380-if-switch-expressions.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift), [Hamish Knight](https://github.com/hamishknight) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#612178](https://github.com/apple/swift/pull/62178), including a downloadable toolchain. +* Review: ([pitch](https://forums.swift.org/t/pitch-if-and-switch-expressions/61149)), ([review](https://forums.swift.org/t/se-0380-if-and-switch-expressions/61899)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0380-if-and-switch-expressions/62695)) + +## Introduction + +This proposal introduces the ability to use `if` and `switch` statements as expressions, for the purpose of: +- Returning values from functions, properties, and closures; +- Assigning values to variables; and +- Declaring variables. + +## Motivation + +Swift has always had a terse but readable syntax for closures, which allows the `return` to be omitted when the body is a single expression. [SE-0255: Implicit Returns from Single-Expression Functions](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0255-omit-return.md) extended this to functions and properties with a single expression as their body. + +This omission of the `return` keyword is in keeping with Swift's low-ceremony approach, and is in common with many of Swift's peer "modern" languages. However, Swift differs from its peers in the lack of support for `if` and `switch` expressions. + +In some cases, this causes ceremony to make a return (ahem), for example: + +```swift +public static func width(_ x: Unicode.Scalar) -> Int { + switch x.value { + case 0..<0x80: return 1 + case 0x80..<0x0800: return 2 + case 0x0800..<0x1_0000: return 3 + default: return 4 + } +} +``` + +In other cases, a user might be tempted to lean heavily on the harder-to-read ternary syntax: + +```swift +let bullet = isRoot && (count == 0 || !willExpand) ? "" + : count == 0 ? "- " + : maxDepth <= 0 ? "▹ " : "▿ " +``` + +Opinions vary on this kind of code, from enthusiasm to horror, but it's accepted that it's reasonable to find this syntax _too_ terse. + +Another option is to use Swift's definite initialization feature: + +```swift +let bullet: String +if isRoot && (count == 0 || !willExpand) { bullet = "" } +else if count == 0 { bullet = "- " } +else if maxDepth <= 0 { bullet = "▹ " } +else { bullet = "▿ " } +``` + +Not only does this add the ceremony of explicit typing and assignment on each branch (perhaps tempting overly terse variable names), it is only practical when the type is easily known. It cannot be used with opaque types, and is very inconvenient and ugly if the type is a complex nested generic. + +Programmers less familiar with Swift might not know this technique, so they may be tempted to take the approach of `var bullet = ""`. This is more bug prone where the default value may not be desired in _any_ circumstances, but definitive initialization won't ensure that it's overridden. + +Finally, a closure can be used to simulate an `if` expression: + +```swift +let bullet = { + if isRoot && (count == 0 || !willExpand) { return "" } + else if count == 0 { return "- " } + else if maxDepth <= 0 { return "▹ " } + else { return "▿ " } +}() +``` + +This also requires `return`s, plus some closure ceremony. But here the `return`s are more than ceremony – they require extra cognitive load to understand they are returning from a closure, not the outer function. + +This proposal introduces a new syntax that avoids all of these problems: + +```swift +let bullet = + if isRoot && (count == 0 || !willExpand) { "" } + else if count == 0 { "- " } + else if maxDepth <= 0 { "▹ " } + else { "▿ " } +``` + +Similarly, the `return` ceremony could be dropped from the earlier example: + +```swift +public static func width(_ x: Unicode.Scalar) -> Int { + switch x.value { + case 0..<0x80: 1 + case 0x80..<0x0800: 2 + case 0x0800..<0x1_0000: 3 + default: 4 + } +} +``` + +Both these examples come from posts by [Nate Cook](https://forums.swift.org/t/if-else-expressions/22366/48) and [Michael Ilseman](https://forums.swift.org/t/omitting-returns-in-string-case-study-of-se-0255/24283), documenting many more examples where the standard library code would be much improved by this feature. + + +## Detailed Design + +`if` and `switch` statements will be usable as expressions, for the purpose of: + +- Returning values from functions, properties, and closures (either with implicit or explicit `return`); +- Assigning values to variables; and +- Declaring variables. + +There are of course many other places where an expression can appear, including as a sub-expression, or as an argument to a function. This is not being proposed at this time, and is discussed in the future directions section. + +For an `if` or `switch` to be used as an expression, it would need to meet these criteria: + +**Each branch of the `if`, or each `case` of the `switch`, must be a single expression.** + +Each of these expressions becomes the value of the overall expression if the branch is chosen. + +This does have the downside of requiring fallback to the existing techniques when, for example, a single expression has a log line above it. This is in keeping with the current behavior of `return` omission. + +An exception to this rule is if a branch either explicitly throws, or terminates the program (e.g. with `fatalError`), in which case no value for the overall expression needs to be produced. In these cases, multiple expressions could be executed on that branch prior to that point. + +In the case where a branch throws, either because a call in the expression throws (which requires a `try`) or with an explicit `throw`, there is no requirement to mark the overall expression with an additional `try` (e.g. before the `if`). + +Within a branch, further `if` or `switch` expressions may be nested. + +**Each of those expressions, when type checked independently, must produce the same type.** + +This has two benefits: it dramatically simplifies the compiler's work in type checking the expression, and it makes it easier to reason about both individual branches and the overall expression. + +It has the effect of requiring more type context in ambiguous cases. The following code would _not_ compile: + +```swift +let x = if p { 0 } else { 1.0 } +``` + +since when type checked individually, `0` is of type `Int`, and `1.0` is of type `Double`. The fix would be to disambiguate each branch. In this case, either by rewriting `0` as `0.0`, or by providing type context e.g. `0 as Double`. + +This can be resolved by providing type context to each of the branches: + +```swift +let y: Float = switch x.value { + case 0..<0x80: 1 + case 0x80..<0x0800: 2.0 + case 0x0800..<0x1_0000: 3.0 + default: 4.5 +} +``` + +This decision is in keeping with other recent proposals such as [SE-0244: Opaque Result Types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md): + +```swift +// Error: Function declares an opaque return type 'some Numeric', but the +// return statements in its body do not have matching underlying types +func f() -> some Numeric { + if Bool.random() { + return 0 + } else { + return 1.0 + } +} +``` + +This rule will require explicit type context for declarations in order to determine the type of `nil` literals: + +```swift +// invalid: +let x = if p { nil } else { 2.0 } +// valid with required type context: +let x: Double? = if p { nil } else { 2.0 } +``` + +Of course, when returning from a function or assigning to an existing variable, this type context is always provided. + +It is also in keeping with [SE-0326: Enable multi-statement closure parameter/result type inference]( https://github.com/swiftlang/swift-evolution/blob/main/proposals/0326-extending-multi-statement-closure-inference.md): + +```swift +func test(_: (Int?) -> T) {} + +// invalid +test { x in + guard let x { return nil } + return x +} + +// valid with required type context: +test { x -> Int? in + guard let x { return nil } + return x +} +``` + +It differs from the behavior of the ternary operator (`let x = p ? 0 : 1.0` compiles, with `x: Double`). + +However, the impact of bidirectional inference on the performance of the type checker would likely prohibit this feature from being implemented today, even if it were considered preferable. This is especially true in cases where there are many branches. This decision could be revisited in future: switching to full bidirectional type inference may be source breaking in theory, but probably not in practice (the proposal authors can't think of any examples where it would be). + +Bidirectional inference also makes it very difficult to reason about each of the branches individually, leading to sometimes unexpected results: + +```swift +let x = if p { + [1] +} else { + [1].lazy.map(expensiveOperation) +} +``` + +With full bidirectional inference, the `Array` in the `if` branch would force the `.lazy.map` in the `else` branch to be unexpectedly eager. + +The one exception to this rule is that some branches could produce a `Never` type. This would be allowed, so long as all non-`Never` branches are of the same type: + +```swift +// x is of type Int, discounting the type of the second branch +let x = if .random() { + 1 +} else { + fatalError() +} +``` + +**In the case of `if` statements, the branches must include an `else`** + +This rule is consistent with the current rules for definitive initialization and return statements with `if` e.g. + +```swift +func f() -> String { + let b = Bool.random() + if b == true { + return "true" + } else if b == false { // } else { here would compile + return "false" + } +} // Missing return in global function expected to return 'String' +``` + +This could be revisited in the future across the board (to DI, return values, and `if` expressions) if logic similar to that of exhaustive switches were applied, but this would be a separate proposal. + +**The expression is not part of a result builder expression** + +`if` and `switch` statements are already expressions when used in the context of a result builder, via the `buildEither` function. This proposal does not change this feature. + +The variable declaration form of an `if` will be allowed in result builders. + +**Pattern matching bindings may occur within an `if` or `case`** + +For example, returns could be dropped from + +```swift + private func balance() -> Tree { + switch self { + case let .node(.B, .node(.R, .node(.R, a, x, b), y, c), z, d): + .node(.R, .node(.B,a,x,b),y,.node(.B,c,z,d)) + case let .node(.B, .node(.R, a, x, .node(.R, b, y, c)), z, d): + .node(.R, .node(.B,a,x,b),y,.node(.B,c,z,d)) + case let .node(.B, a, x, .node(.R, .node(.R, b, y, c), z, d)): + .node(.R, .node(.B,a,x,b),y,.node(.B,c,z,d)) + case let .node(.B, a, x, .node(.R, b, y, .node(.R, c, z, d))): + .node(.R, .node(.B,a,x,b),y,.node(.B,c,z,d)) + default: + self + } + } +``` + +and optional unwrapping could be used with `if let`: + +```swift +// equivalent to let x = foo.map(process) ?? someDefaultValue +let x = if let foo { process(foo) } else { someDefaultValue } +``` + +## Future Directions + +This proposal chooses a narrow path of only enabling expressions in the 3 cases laid out at the start. This is intended to cover the vast majority of use cases, but could be followed up by expanded functionality covering many other use cases. Further cases could be added in later proposals once the community has had a chance to use this feature in practice – including source breaking versions introduced under a language variant. + +### Full Expressions + +A feel for the kind of expressions this could produce can be found in [this commit](https://github.com/apple/swift/compare/main...hamishknight:express-yourself#diff-7db38bc4b6f7872e5a631989c2925f5fac21199e221aa9112afbbc9aae66a2de) which adds this functionality to the parser. + +Full expressions would include various fairly conventional examples not proposed here: + +```swift +let x = 1 + if .random() { 3 } else { 4 } +``` + +but also some pretty strange ones such as + +```swift +for b in [true] where switch b { case true: true case false: false } {} +``` + +The strange examples can mostly be considered "weird but harmless" but there are some source breaking edge cases, in particular in result builders: + +```swift +var body: some View { + VStack { + if showButton { + Button("Click me!", action: clicked) + } else { + Text("No button") + } + .someStaticProperty + } +} +``` + +In this case, if `if` expressions were allowed to have postfix member expressions (which they aren't today, even in result builders), it would be ambiguous whether this should be parsed as a modifier on the `if` expression, or as a new expression. This could only be an issue for result builders, but the parser does not have the ability to specialize behavior for result builders. Note, this issue can happen today (and is why `One` exists for Regex Builders) but could introduce a new ambiguity for code that works this way today. + +### `do` Expressions + +`do` blocks could similarly be transformed into expressions, for example: + +```swift +let foo: String = do { + try bar() +} catch { + "Error \(error)" +} +``` + +### Guard + +Often enthusiasm for `guard` leads to requests for `guard` to have parity with `if`. Returning a value from a `guard`'s else is very common, and could potentially be sugared as + +```swift +guard hasNativeStorage else { nil } +``` + +This is appealing, but is really a different proposal, of allowing omission `return` in `guard` statements. + +### Multi-statement branches + +The requirement that every branch be just a single expression leads to an unfortunate usability cliff: + +```swift +let decoded = + if isFastUTF8 { + Log("Taking the fast path") + withFastUTF8 { _decodeScalar($0, startingAt: i) } + } else + Log("Running error-correcting slow-path") + foreignErrorCorrectedScalar( + startingAt: String.Index(_encodedOffset: i)) + } +``` + +This is consistent with other cases, like multi-statement closures. But unlike in that case, where all that is needed is a `return` from the closure, this requires the user refactor the code back to the old mechanisms. + +The trouble is, there is no great solution here. The approach taken by some other languages such as rust is to allow a bare expression at the end of the scope to be the expression value for that scope. There are stylistic preferences for and against this. More importantly, this would be a fairly radical new direction for Swift, and if proposed should probably be considered for all such cases (like function and closure return values too). + +Alternatively, a new keyword could be introduced to make explicit that an expression value is being used as the value for this branch (Java uses `yield` for this in `switch` expressions). + +### Either + +As mentioned above, in result builders an `if` can be used to construct an `Either` type, which means the expressions in the branches could be of different types. + +This could be done with `if` expressions outside result builders too, and would be a powerful new feature for Swift. However, it is large in scope (including the introduction of a language-integrated `Either` type) and should be considered in a separate proposal, probably after the community has adjusted to the more vanilla version proposed here. + +## Alternatives Considered + +### Sticking with the Status Quo + +The list of [commonly rejected proposals](https://github.com/swiftlang/swift-evolution/blob/main/commonly_proposed.md) includes the subject of this proposal: + +> **if/else and switch as expressions**: These are conceptually interesting things to support, but many of the problems solved by making these into expressions are already solved in Swift in other ways. Making them expressions introduces significant tradeoffs, and on balance, we haven't found a design that is clearly better than what we have so far. + +The motivation section above outlines why the alternatives that exist today fall short. One of the reasons this proposal is narrow in scope is to bring the majority of value while avoiding resolving some of these more difficult trade-offs. + +The lack of this feature puts Swift's [claim](https://www.swift.org/about/) to be a modern programming language under some strain. It is one of the few modern languages (Go being the other notable exception) not to support something along these lines. + +### Alternative syntax + +Instead of extending the current implicit return mechanism, where a single expression is treated as the returned value, this proposal could introduce a new syntax for expression versions of `if`/`switch`. For example, in Java: + +```java +var response = switch (utterance) { + case "thank you" -> "you’re welcome"; + case "atchoo" -> "gesundheit"; + case "fire!" -> { + log.warn("fire detected"); + yield "everybody out!"; // yield = value of multi-statement branch + }; + default -> { + throw new IllegalStateException(utterance); + }; +}; +``` + +A similar suggestion was made during [SE-0255: Implicit Returns from Single-Expression Functions](https://forums.swift.org/t/se-0255-implicit-returns-from-single-expression-functions/), where an alternate syntax for single-expression functions was discussed e.g. `func sum() -> Element = reduce(0, +)`. In that case, the core team did not consider introduction of a separate syntax for functions to be sufficiently motivated. + +The main benefit to the alternate `->` syntax is to make it more explicit, but comes at the cost of needing to know about two different kinds of switch syntax. Note that this is orthogonal to, and does not solve, the separate goal of providing a way of explicitly "yielding" an expression value in the case of multi-statement branches (also shown here in this java example) versus taking the "last expression" approach. + +Java did not introduce this syntax for `if` expressions. Since this is a goal for Swift, this implies: + +```swift +let x = + if .random() -> 1 + else -> fatalError() +``` + +However, this then poses an issue when evolving to multi-statement branches. Unlike with `switch`, these would require introducing braces, leaving a combination of both braces _and_ a "this is an expression" sigil: + +```swift +let x = + if .random() -> { + let y = someComputation() + y * 2 + } else -> fatalError() +``` + +Unlike Java and C, this "braces for 2+ arguments" style of `if` is out of keeping in Swift. + +It is also not clear if the `->` would work well if expression status is brought to more kinds of statement e.g. + +```swift +let foo: String = + do -> + try bar() + catch ns as NSError -> + "Error \(error)" +``` + +or mixed branches with expressions and a return: + +```swift +let x = + if .random() -> 1 + else -> return 2 +``` + +If a future direction of full expressions is considered, the `->` form may not work so well, especially when single-line expressions are desired e.g. + +```swift +// is this (p ? 1 : 2) + 3 +// or p ? 1 : (2 + 3) +let x = if p -> 1 else -> 2 + 3 +``` + +### Support for control flow + +An earlier version of this proposal allowed use of `return` in a branch. Similar to `return`, statements that `break` or `continue` to a label, were considered a future direction. + +Allowing new control flow out of expressions could be unexpected and error-prone. Swift currently only allows control flow out of expressions through thrown errors, which must be explicitly marked with `try` (or, in the case of `if` or `switch` branches, with `throw`) as an indication of the control flow to the programmer. Allowing other control flow out of expressions would undermine this principle. The control flow impact of nested return statements would become more difficult to reason about if we extend SE-0380 to support multiple-statement branches in the future. The use-cases for this functionality presented in the review thread were also fairly niche. Given the weak motivation and the potential problems introduced, the Language Workgroup accepts SE-0380 without this functionality. + +## Source compatibility + +As proposed, this addition has one source incompatibility, related to unreachable code. The following currently compiles, albeit with a warning that the `if` statement is unreachable (and the values in the branches unused): + +```swift +func foo() { + return + if .random() { 0 } else { 0 } +} +``` + +but under this proposal, it would fail to compile with an error of "Unexpected non-void return value in void function" because it now parses as returning the `Int` expression from the `if`. This could be fixed in various ways (with `return;` or `return ()` or by `#if`-ing out the dead code explicitly). + +Another similar case can occur if constant evaluation leads the compiler to ignore dead code: + +```swift +func foo() -> Int { + switch true { + case true: + return 0 + case false: + print("unreachable") + } +} +``` + +This currently _doesn't_ warn that the `false` case is unreachable (though probably should), but without special handling would after this proposal result in a type error that `()` does not match expected type `Int`. + +Given these examples all require dead code, it seems reasonable to accept this source break rather than gate this change under a language version or add special handling to avoid the break. + +## Effect on ABI stability + +This proposal has no impact on ABI stability. + +## Acknowledgments + +Much of this implementation layers on top of ground work done by [Pavel Yaskevich](https://github.com/xedin), particularly the work done to allow [multi-statement closure type inference](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0326-extending-multi-statement-closure-inference.md). + +Both [Nate Cook](https://forums.swift.org/t/if-else-expressions/22366/48) and [Michael Ilseman](https://forums.swift.org/t/omitting-returns-in-string-case-study-of-se-0255/24283) provided analysis of use cases in the standard library and elsewhere. Many community members have made a strong case for this change, most recently [Dave Abrahams](https://forums.swift.org/t/if-else-expressions/22366). diff --git a/proposals/0381-task-group-discard-results.md b/proposals/0381-task-group-discard-results.md new file mode 100644 index 0000000000..27ec7607b0 --- /dev/null +++ b/proposals/0381-task-group-discard-results.md @@ -0,0 +1,267 @@ +# DiscardingTaskGroups + +* Proposal: [SE-0381](0381-task-group-discard-results.md) +* Authors: [Cory Benfield](https://github.com/Lukasa), [Konrad Malawski](https://github.com/ktoso) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#62361](https://github.com/apple/swift/pull/62361) +* Review: ([pitch](https://forums.swift.org/t/pitch-task-pools/61703)) ([review](https://forums.swift.org/t/se-0381-discardresults-for-taskgroups/62072)) ([acceptance](https://forums.swift.org/t/accepted-se-0381-discardingtaskgroups/62615)) + +### Introduction + +We propose to introduce a new type of structured concurrency task group: `Discarding[Throwing]TaskGroup`. This type of group is similar to `TaskGroup` however it discards results of its child tasks immediately. It is specialized for potentially never-ending task groups, such as top-level loops of http or other kinds of rpc servers. + +## Motivation + +Task groups are the building block of structured concurrency, allowing for the Swift runtime to relate groups of tasks together. This enables powerful features such as automatic cancellation propagation, correctly propagating errors, and ensuring well-defined lifetimes, as well as providing diagnostic information to programming tools. + +The version of Task Groups introduced in [SE-0304](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md) provides all of these features. However, it also provides the ability to propagate return values to the user of the task group. This capability provides an unexpected limitation in some use-cases. + +As users of Task Groups are able to retrieve the return values of child tasks, it implicitly follows that the Task Group preserves at least the `Result` of any completed child task. As a practical matter, the task group actually preserves the entire `Task` object. This data is preserved until the user consumes it via one of the Task Group consumption APIs, whether that is `next()` or by iterating the Task Group. + +The result of this is that Task Groups are ill-suited to running for a potentially unbounded amount of time. An example of such a use-case is managing connections accepted from a listening socket. A simplified example of such a workload might be: + +```swift +try await withThrowingTaskGroup(of: Void.self) { group in + while let newConnection = try await listeningSocket.accept() { + group.addTask { + handleConnection(newConnection) + } + } +} +``` + +As written, this task group will leak all the child `Task` objects until the listening socket either terminates or throws. If this was written for a long-running server, it is entirely possible for this Task Group to survive for a period of days, leaking thousands of Task objects. For stable servers, this will eventually drive the process into memory exhaustion, forcing it to be killed by the OS. + +The current implementation of Task Groups do not provide a practical way to avoid this issue. Task Groups are (correctly) not `Sendable`, so neither the consumption of completed `Task` results nor the submission of new work can be moved to a separate `Task`. + +The most natural attempt to avoid this unbounded memory consumption would be to attempt to occasionally purge the completed task results. An example might be: + +```swift +try await withThrowingTaskGroup(of: Void.self) { group in + while let newConnection = try await listeningSocket.accept() { + group.addTask { + handleConnection(newConnection) + } + try await group.next() + } +} +``` + +Unfortunately, all of the methods for attempting to pop the queue of completed `Task`s will suspend if all currently live child `Task`s are executing. This means that the above pattern (or any similar pattern) is at risk of occasional livelocks, where pending connections could be accepted, but the `Task` is blocked waiting for existing work to complete. + +There is only one design pattern to avoid this issue, which involves forcibly bounding the maximum concurrency of the Task Group. This pattern looks something like the below: + +```swift +try await withThrowingTaskGroup(of: Void.self) { group in + // Fill the task group up to maxConcurrency + for _ in 0..( + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout DiscardingTaskGroup) async -> GroupResult +) async -> GroupResult { ... } + +public func withThrowingDiscardingTaskGroup( + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout ThrowingDiscardingTaskGroup) async throws -> GroupResult +) async throws -> GroupResult { ... } +``` + +And the types themselves, mostly mirroring the APIs of `TaskGroup`, except that they're missing `next()` and related functionality: + +```swift +public struct DiscardingTaskGroup { + + public mutating func addTask( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Void + ) + + public mutating func addTaskUnlessCancelled( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Void + ) -> Bool + + public var isEmpty: Bool + + public func cancelAll() + public var isCancelled: Bool +} +@available(*, unavailable) +extension DiscardingTaskGroup: Sendable { } + +public struct ThrowingDiscardingTaskGroup { + + public mutating func addTask( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Void + ) + + public mutating func addTaskUnlessCancelled( + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Void + ) -> Bool + + public var isEmpty: Bool + + public func cancelAll() + public var isCancelled: Bool +} +@available(*, unavailable) +extension DiscardingThrowingTaskGroup: Sendable { } +``` + +## Detailed Design + +### Discarding results + +As indicated by the name a `[Throwing]DiscardingTaskGroup` will discard results of its child tasks _immediately_ and release the child task that produced the result. This allows for efficient and "running forever" request accepting loops such as HTTP or RPC servers. + +Specifically, the first example shown in the Motivation section of this proposal, _is_ safe to be expressed using a discarding task group, as follows: + +```swift +// GOOD, no leaks! +try await withThrowingDiscardingTaskGroup() { group in + while let newConnection = try await listeningSocket.accept() { + group.addTask { + handleConnection(newConnection) + } + } +} +``` + +This code–unlike the `withThrowingTaskGroup` version shown earlier–does not leak tasks and therefore is safe and the recommended way to express such handler loops. + +### Error propagation and group cancellation + +Throwing task groups rely on the `next()` (or `waitForAll()`) being throwing and end users consuming the child tasks this way in order to surface any error that the child tasks may have thrown. It is possible for a `ThrowingTaskGroup` to explicitly collect results (and failures), and react to them, like this: + +```swift +try await withThrowingTaskGroup(of: Void.self) { group in + group.addTask { try boom() } + group.addTask { try boom() } + group.addTask { try boom() } + + try await group.next() // re-throws whichever error happened first +} // since body threw, the group and remaining tasks are immediately cancelled +``` + +The above snippet illustrates a simple case of the error propagation out of a child task, through `try await next()` (or `try await group.waitForAll()`) out of the `withThrowingTaskGroup` closure body. As soon as an error is thrown out of the closure body, the group cancels itself and all remaining tasks implicitly, finally proceeding to await all the pending tasks. + +This pattern is not possible with `ThrowingDiscardingTaskGroup` because the the results collecting methods are not available on discarding groups. In order to properly support the common use-case of discarding groups, the failure of a single task, should implicitly and _immediately_ cancel the group and all of its siblings. + +This can be seen as the implicit immediate consumption of the child tasks inspecting the task for failures, and "re-throwing" the failure automatically. The error is then also re-thrown out of the `withThrowingDiscardingTaskGroup` method, like this: + +```swift +try await withThrowingDiscardingTaskGroup() { group in + group.addTask { try boom(1) } + group.addTask { try boom(2) } + group.addTask { try boom(3) } + // whichever failure happened first, is collected, stored, and re-thrown out of the method when exiting. +} +``` + +In other words, discarding task groups follow the "one for all, and all for one" pattern for failure handling. A failure of a single child task, _immediately_ causes cancellation of the group and its siblings. + +Preventing this behavior can be done in two ways: + +- using `withDiscardingTaskGroup`, since the child tasks won't be allowed to throw, and must handle their errors in some other way, +- including normal `do {} catch {}` error handling logic inside the child-tasks, which only re-throws. + +We feel this is the right approach for this structured concurrency primitive, as we should be leaning on normal swift code patterns, rather than introduce special one-off ways to handle and deal with errors. Although, if it were necessary, we could introduce a "failure reducer" in the future. + +## Alternatives Considered + +### Introducing new "TaskPool" type (initial pitch) + +The [original pitch](https://forums.swift.org/t/pitch-task-pools/61703) introduced two new types, `TaskPool` and `ThrowingTaskPool`. These types were introduced in order to expose at the type system level the inability to iterate the pool for new tasks. This would avoid the `next()` behaviour introduced in this pitch, where `next()` always returns `nil`. This was judged a worthwhile change to justify introducing new types. + +Several reviewers of the pitch felt that this was not a sufficiently useful capability to justify the introduction of the new types, and that the pitched behaviour more properly belonged as a "mode" of operation on `TaskGroup`. In line with that feedback, this proposal has moved to using the `discardResults` option. + +### Extending [Throwing]TaskGroup with discardResults flag + +After feedback on the the initial pitch, we attempted to avoid introducing a new type, and instead handle it using a `discardResults: Bool` flag on `with[Throwing]TaskGroup()` this was fairly problematic because: + +- the group would have the `next()` method as well as `AsyncSequence` conformance present, but non-functional, i.e. always returning `nil` from `next()` which could lead to subtle bugs and confusion. +- we'd end up constraining this new option only to child task result types of `Void`, making access to this functionality a bit hard to discover + +The group would also have very different implicit cancellation behavior, ultimately leading us to conclude during the Swift Evolution review that these two behaviors should not be conflated into one type. + +### Alternate Error throwing behaviour + +The pitch proposes that `ThrowingDiscardingTaskGroup` will throw only the _first_ error thrown by a child `Task`. This means that all subsequent errors will be discarded, which is an unfortunate loss of information. Two alternative behaviours could be chosen: we could not provide `ThrowingDiscardingTaskGroup` at all, or we could throw an aggregate error that contains *all* errors thrown by the child `Task`s. + +Not allowing offering `ThrowingDiscardingTaskGroup` at all is a substantial ergonomic headache. Automatic error propagation is one of the great features of structured concurrency, and not being able to use it in servers or other long-running processes is an unnecessary limitation, especially as it's not particularly technically challenging to propagate errors. For this reason, we do not think it's wise to omit `discardResults` on `ThrowingDiscardingTaskGroup`. + +The other alternative is to throw an aggregate error. This would require that `ThrowingDiscardingTaskGroup` persist all (or almost all) errors thrown by child tasks and merge them together into a single error `struct` that is thrown. This idea is a mixed bag. + +The main advantage of throwing an aggregate error is that no information is lost. Programs can compute on all errors that were thrown, and at the very least can log or provide other metrics based on those errors. Avoiding data loss in this way is valuable, and gives programmers more flexibility. + +Throwing an aggregate error has two principal disadvantages. The first is that aggregate errors do not behave gracefully in `catch` statements. If a child task has thrown `MyModuleError`, programmers would like to write `catch MyModuleError` in order to handle it. Aggregate errors break this situation, even if only one error is thrown: programmers have to write `catch let error = error as? MyAggregateError where error.baseErrors.contains(where: { $0 is MyModuleError })`, or something else equally painful. + +The other main disadvantage is the storage bloat from `CancellationError`. The first thrown error will auto-cancel all child `Task`s. This is great, but that cancellation will likely manifest in as series of `CancellationError`s, which will presumably bubble to the top and be handled by the `ThrowingDiscardingTaskGroup`. This means that a `ThrowingDiscardingTaskGroup` will likely store a substantial collection of errors, where all but the first are `CancellationError`. This is a substantial regression in convenience for the mainline case, with additional costs in storage, without providing any more meaningful information. + +For these reasons we've chosen the middle behaviour, where only one error is thrown. We think there is merit in throwing an aggregate error, however, and we'd like community feedback on this alternative. + +### Child Task for reaping + +An alternative would be to have Task Group spin up a child `Task` that can be used to consume tasks from the group. The API surface would look something like this: + +```swift +withTaskGroupWithChildTask(of: Void.self) { group in + group.addTask { + handleConnection(newConnection) + } +} +consumer: { group in + for task in group { } +} +``` + +The advantage of this variant is that it is substantially more flexible, and allows non-`Void`-returning tasks. The downside of this variant is that it muddies the water on the question of whether Task Groups are `Sendable` (requiring a specific-exemption for this use-case) and forces users to understand the lifetime of a pair of different closures. + +## Future Directions + +### Error Handling + +A number of concerns were raised during the pitch process that the "throw the first error only" pattern may be insufficiently flexible. Community members were particularly interested in having some sort of error filter function that could be used to filter, accumulate, or discard errors as needed. + +The proposers feel that introducing this API surface in the first version of this feature adds significant complexity to this type. This requires us to be confident that the API surface proposed is going to serve the necessary use-cases, without adding unnecessary cognitive load. It's also not entirely clear where the line is between features that can be handled using `try`/`catch` and features that require a new error filter function. + +As a result, the proposal authors have elected to defer implementing anything here until there are real-world examples to generalise from. Having some sort of error filter is likely to be valuable, and the implementation will preserve the capability to implement such a function, but for now the proposal is going to be kept relatively small. diff --git a/proposals/0382-expression-macros.md b/proposals/0382-expression-macros.md new file mode 100644 index 0000000000..e3a33eb558 --- /dev/null +++ b/proposals/0382-expression-macros.md @@ -0,0 +1,602 @@ +# Expression Macros + +* Proposal: [SE-0382](0382-expression-macros.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.9)** +* Implementation: Partial implementation is available in `main` under the experimental feature flag `Macros`. An [example macro repository](https://github.com/DougGregor/swift-macro-examples) provides a way to experiment with this feature. +* Review: ([pitch](https://forums.swift.org/t/pitch-expression-macros/61499)) ([pitch #2](https://forums.swift.org/t/pitch-2-expression-macros/61861)) ([review #1](https://forums.swift.org/t/se-0382-expression-macros/62090)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0382-expression-macros/62898)) ([pitch #3](https://forums.swift.org/t/se-0382-expression-macros-mini-pitch-for-updates/62810)) ([review #2](https://forums.swift.org/t/se-0382-second-review-expression-macros/63064)) ([acceptance](https://forums.swift.org/t/accepted-se-0382-expression-macros/63495)) ([post-acceptance update](https://forums.swift.org/t/update-on-se-0382-and-se-0389-expression-macros-and-attached-macros/74094)) + +## Introduction + +Expression macros provide a way to extend Swift with new kinds of expressions, which can perform arbitrary syntactic transformations on their arguments to produce new code. Expression macros make it possible to extend Swift in ways that were only previously possible by introducing new language features, helping developers build more expressive libraries and eliminate extraneous boilerplate. + +## Motivation + +Expression macros are one part of the [vision for macros in Swift](https://github.com/swiftlang/swift-evolution/pull/1927), which lays out general motivation for introducing macros into the language. Expressions in particular are an area where the language already provides decent abstractions for factoring out runtime behavior, because one can create a function that you call as an expression from anywhere. However, with a few hard-coded exceptions like `#file` and `#line`, an expression cannot reason about or modify the source code of the program being compiled. Such use cases will require external source-generating tools, which don't often integrate cleanly with other tooling. + +## Proposed solution + +This proposal introduces the notion of expression macros, which are used as expressions in the source code (marked with `#`) and are expanded into expressions. Expression macros can have parameters and a result type, much like a function, which describes the effect of the macro expansion on the expression without requiring the macro to actually be expanded first. + +The actual macro expansion is implemented with source-to-source translation on the syntax tree: the expression macro is provided with the syntax tree for the macro expansion itself (e.g., starting with the `#` and ending with the last argument), which it can rewrite into the expanded syntax tree. That expanded syntax tree will be type-checked against the result type of the macro. + +As a simple example, let's consider a `stringify` macro that takes a single argument as input and produces a tuple containing both the original argument and also a string literal containing the source code for that argument. This macro could be used in source code as, for example: + +```swift +#stringify(x + y) +``` + +and would be expanded into + +```swift +(x + y, "x + y") +``` + +The type signature of a macro is part of its declaration, which looks a lot like a function: + +```swift +@freestanding(expression) macro stringify(_: T) -> (T, String) +``` + +### Type-checked macro arguments and results + +Macro arguments are type-checked against the parameter types of the macro prior to instantiating the macro. For example, the macro argument `x + y` will be type-checked; if it is ill-formed (for example, if `x` is an `Int` and `y` is a `String`), the macro will never be expanded. If it is well-formed, the generic parameter `T` will be inferred to the result of `x + y`, and that type is carried through to the result type of the macro. There are several benefits to this type-checked model: + +* Macro implementations are guaranteed to have well-typed arguments as inputs, so they don't need to be concerned about incorrect code being passed into the macro. +* Tools can treat macros much like functions, providing the same affordances for code completion, syntax highlighting, and so on, because the macro arguments follow the same rules as other Swift code. +* A macro expansion expression can be partially type-checked without having to expand the macro. This allows tools to still have reasonable results without performing macro expansion, as well as improving compile-time performance because the same macro will not be expanded repeatedly during type inference. + +When the macro is expanded, the expanded syntax tree is type-checked against the result type of the macro. In the `#stringify(x + y)` case, this means that if `x + y` had type `Int`, the expanded syntax tree (`(x + y, "x + y")`) is type-checked with a contextual type of `(Int, String)`. + +The type checking of macro expressions is similar to type-checking a call, allowing type inference information to flow from the macro arguments to the result type and vice-versa. For example, given: + +```swift +let (a, b): (Double, String) = #stringify(1 + 2) +``` + +the integer literals `1` and `2` would be assigned the type `Double`. + +### Syntactic translation + +Macro expansion is a syntactic operation, which takes as input a well-formed syntax tree consisting of the full macro expansion expression (e.g., `#stringify(x + y)`) and produces a syntax tree as output. The resulting syntax tree is then type-checked based on the macro result type. + +Syntactic translation has a number of benefits over more structured approaches such as direct manipulation of a compiler's Abstract Syntax Tree (AST) or internal representation (IR): + +* A macro expansion can use the full Swift language to express its effect. If it can be written as Swift source code at that position in the grammar, a macro can expand to it. +* Swift programmers understand Swift source code, so they can reason about the output of a macro when applied to their source code. This helps both when authoring and using macros. +* Source code that uses macros can be "expanded" to eliminate the use of the macro, for example to make it easier to reason about or debug, or make it work with an older Swift compiler that doesn't support macros. +* The compiler's AST and internal representation need not be exposed to clients, which would limit the ability of the compiler to evolve and improve due to backward-compatibility concerns. + +On the other hand, purely syntactic translations have a number of downsides, too: + +* Syntactic macro expansions are prone to compile-time failures, because we're effectively working with source code as strings, and it's easy to introduce (e.g.) syntax errors or type errors in the macro implementation. +* Syntactic macro expansions are re-parsed and re-type-checked, which incurs more compile-time overhead than an approach that (say) manipulated the AST or IR directly. +* Syntactic macros are not *hygienic*, meaning that the way in which a macro expansion is processed depends on the environment in which it is expanded, and can affect that environment. + +The proposed macro design attempts to mitigate these problems, but they are somewhat fundamental to the use of syntactic macros. On balance, the ease-of-use and easy-of-interpretation of syntactic macros outweighs these problems. + +### Macros defined as separate programs + +Macro definitions operate on syntax trees. Broadly speaking, there are two different ways in which a macro's expansion operation can be defined: + +* *A declarative set of transformations*: this involves extending the language with a special syntax that allows macros to define how the macro is expanded given the macro inputs, and the compiler applies those rules for each macro expansion. The C preprocessor employs a simplistic form of this, but Racket's [pattern-based macros](https://docs.racket-lang.org/guide/pattern-macros.html) and Rust's [declarative macros](https://doc.rust-lang.org/book/ch19-06-macros.html#declarative-macros-with-macro_rules-for-general-metaprogramming) offer more advanced rules that match the macro arguments to a pattern and then perform a rewrite to new syntax as described in the macro. For Swift to adopt this approach, we would likely need to invent a pattern language for matching and rewriting syntax trees. +* *An executable program that transforms the source*: this involves running a program that manipulates the program syntax directly. How the program is executed depends a lot on the environment: [Scala 3 macros](https://docs.scala-lang.org/scala3/guides/macros/macros.html) benefit from the use of the JVM so they can intertwine target code (the program being generated) and host code (where the compiler is running), whereas Rust's [procedural macros](https://doc.rust-lang.org/reference/procedural-macros.html) are built into a separate crate that the compiler interacts with. For Swift to adopt this approach, we would either need to build a complete interpreter for Swift code in the compiler, or take an approach similar to Rust's and build macro definitions as separate programs that the compiler can interact with. + +We propose the latter approach, where a macro definition is a separate program that operates on Swift syntax trees using the [swift-syntax](https://github.com/apple/swift-syntax/) package. Expression macros are defined as types that conform to the `ExpressionMacro` protocol: + +```swift +public protocol ExpressionMacro: FreestandingMacro { + /// Expand a macro described by the given freestanding macro expansion + /// within the given context to produce a replacement expression. + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> ExprSyntax +} +``` + +The `expansion(of:in:)` method takes as arguments the syntax node for the macro expansion expression (e.g., `#stringify(x + y)`) and a "context" that provides more information about the compilation context in which the macro is being expanded. It produces a macro result that includes the rewritten syntax tree. + +The specifics of `Macro`, `ExpressionMacro`, and `MacroExpansionContext` will follow in the Detailed Design section. + +### The `stringify` macro implementation + +Let's continue with the implementation of the `stringify` macro. It's a new type `StringifyMacro` that conforms to `ExpressionMacro`: + +```swift +import SwiftSyntax +import SwiftSyntaxBuilder +import SwiftSyntaxMacros + +public struct StringifyMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + guard let argument = node.argumentList.first?.expression else { + fatalError("compiler bug: the macro does not have any arguments") + } + + return "(\(argument), \(literal: argument.description))" + } +} +``` + +The `expansion(of:in:)` function is fairly small, because the `stringify` macro is relatively simple. It extracts the macro argument from the syntax tree (the `x + y` in `#stringify(x + y)`) and then forms the resulting tuple expression by interpolating in the original argument both as a value and then as source code in [a string literal](https://github.com/apple/swift-syntax/blob/main/Sources/SwiftSyntaxBuilder/ConvenienceInitializers.swift#L259-L265). That string is then parsed as an expression (producing an `ExprSyntax` node) and returned as the result of the macro expansion. This is a simple form of quasi-quoting provided by the `SwiftSyntaxBuilder` module, implemented by making the major syntax nodes (`ExprSyntax` in this case, for expressions) conform to [`ExpressibleByStringInterpolation`](https://developer.apple.com/documentation/swift/expressiblebystringinterpolation), where existing syntax nodes can be interpolated into string literals containing the expanded Swift code. + +The `StringifyMacro` struct is the implementation for the `stringify` macro declared earlier. We will need to tie these together in the source code via some mechanism. We propose to provide a builtin macro that names the module and the `ExpressionMacro` type name within the macro declaration following an `=`, e.g., + +```swift +@freestanding(expression) +macro stringify(_: T) -> (T, String) = + #externalMacro(module: "ExampleMacros", type: "StringifyMacro") +``` + +## Detailed design + +There are two major pieces to the macro design: how macros are declared and expanded within a program, and how they are implemented as separate programs. The following sections provide additional details. + +### Macro declarations + +A macro declaration is described by the following grammar: + +``` +declaration -> macro-declaration + +macro-declaration -> macro-head identifier generic-parameter-clause[opt] macro-signature macro-definition[opt] generic-where-clause[opt] + +macro-head -> attributes[opt] declaration-modifiers[opt] 'macro' + +macro-signature -> parameter-clause macro-function-signature-result[opt] + +macro-function-signature-result -> '->' type + +macro-definition -> '=' expression +``` + +The `@freestanding(expression)` attribute applies only to macros. It indicates that the macro is an expression macro. The "freestanding" terminology comes from the [macros vision document](https://github.com/swiftlang/swift-evolution/pull/1927), and is used to describe macros that are expanded with the leading `#` syntax. + +Macro signatures are function-like, with a parameter clause (that may be empty) and an optional result type. + +Macros can only be declared at file scope. They can be overloaded in the same way as functions, so long as the argument labels, parameter types, or result type differ. + +The `macro-definition` provides the implementation used to expand the macro. It is parsed as a general expression, but must always be a `macro-expansion-expression`, so all non-builtin macros are defined in terms of other macros, terminating in a builtin macro whose definition is provided by the compiler. The arguments provided within the `macro-expansion-expression` of the macro definition must either be direct references to the parameters of the enclosing macro or must be literals. The `macro-expansion-expression` is type-checked (to ensure that the argument and result types make sense), but no expansion is performed at the time of definition. Rather, expansion of the macro referenced by the `macro-definition` occurs when the macro being declared is expanded. See the following section on macro expansion for more information. + +Macro parameters may have default arguments, but those default arguments can only consist of literal expressions and other macro expansions. + +Macros can have opaque result types. The rules for uniqueness of opaque result types for macros are somewhat different from opaque result types of functions, because each macro expansion can easily produce a different type. Therefore, each macro expansion producing an opaque result type will be considered to have a distinct type, e.g., the following is ill-formed: + +```swift +@freestanding(expression) macro someMacroWithOpaqueResult() -> some Collection + +var a = #someMacroWithOpaqueResult +a = #someMacroWithOpaqueResult // cannot assign value with type of macro expansion here to opaque type from macro expansion above +``` + +### Macro expansion + +A macro expansion expression is described by the following grammar: + +``` +primary-expression -> macro-expansion-expression +macro-expansion-expression -> '#' identifier generic-argument-clause[opt] function-call-argument-clause[opt] trailing-closures[opt] +``` + +The `#` syntax for macro expansion expressions was specifically chosen because Swift already contains a number of a `#`-prefixed expressions that are macro-like in nature, some of which could be implemented directly as expression macros. The macro referenced by the `identifier` must be an expression macro, as indicated by `@freestanding(expression)` on the corresponding macro declaration. + +Both `function-call-argument-clause` and `trailing-closures` are optional. When both are omitted, the macro is expanded as-if the empty argument list `()` was provided. Macros are not first-class entities in the way functions are, so they cannot be passed around as values and do not need an "unapplied macro" syntax. This allows `#line` et al to be macros without requiring them to be written as `#line()`. There is some precedent for this with property wrappers, which will also be used for attached macros. + +When a macro expansion is encountered in the source code, its expansion occurs in two phases. The first phase is the type-check phase, where the arguments to the macro are type-checked against the parameters of the named macro, and the result type of the named macro is checked against the context in which the macro expansion occurs. This type-checking is equivalent to that performed for a function call, and does not involve the macro definition. + +The second phase is the macro expansion phase, during which the syntax of the macro arguments is provided to the macro definition. For builtin-macro definitions, the behavior at this point depends on the semantics of the macro, e.g., the `externalMacro` macro invokes the external program and provides it with the source code of the macro expansion. For other macros, the arguments are substituted into the `macro-expansion-expression` of the definition. For example: + +```swift +@freestanding(expression) macro prohibitBinaryOperators(_ value: T, operators: [String]) -> T = + #externalMacro(module: "ExampleMacros", type: "ProhibitBinaryOperators") +@freestanding(expression) macro addBlocker(_ value: T) -> T = #prohibitBinaryOperators(value, operators: ["+"]) + +#addBlocker(x + y * z) +``` + +Here, the macro expansion of `#addBlocker(x + y * z)` will first expand to `#prohibitBinaryOperators(x + y * z, operators: ["+"])`. Then that expansion will be processed by the `ExampleMacros.ProhibitBinaryOperators`, which would be defined as a struct conforming to `ExpressionMacro`. + +Macro expansion produces new source code (in a syntax tree), which is then type-checked using the original macro result type as its contextual type. For example, the `stringify` example macro returned a `(T, String)`, so when given an argument of type `Int`, the result of expanding the macro would be type-checked as if it were on the right-hand side of + +```swift +let _: (Int, String) = +``` + +Macro expansion expressions can occur within the arguments to a macro. For example, consider: + +```swift +#addBlocker(#stringify(1 + 2)) +``` + +The first phase of the macro type-check does not perform any macro expansion: the macro expansion expression `#stringify(1 + 2)` will infer that its `T` is `Int`, and will produce a value of type `(Int, String)`. The `addBlocker` macro expansion expression will infer that its `T` is `(Int, String)`, and the result is the same. + +The second phase of macro expansions occurs outside-in. First, the `addBlocker` macro is expanded, to `#prohibitBinaryOperators(#stringify(1 + 2), operators: ["+"])`. Then, the `prohibitBinaryOperators` macro is expanded given those (textual) arguments. The expansion result it produces will be type-checked, which will end up type-checking `#stringify(1 + 2)` again and, finally, expanding `#stringify(1 + 2)`. + +From an implementation perspective, the compiler reserves the right to avoid performing repeated type checking of the same macro arguments. For example, we type-checked `#stringify(1 + 2)` in the first phase of the expansion of `prohibitBinaryOperators`, and then again on the expanded result. When the compiler recognizes that the same syntax node is being re-used unmodified, it can re-use the types computed in the first phase. This is an important performance optimization for the type checker. + +Macro expansion cannot be recursive: if the expansion of a given macro produces source code that expands that same macro, the program is ill-formed. This prevents unbounded macro expansion. + +With the exception of the built-in macro declarations for source locations (e.g., `#fileID`, `#line`), a macro cannot be used as the default argument of a parameter. The existing features for source locations have special behavior when they appear as a default argument, wherein they are expanded by the caller using the source-location information at the call site rather than in the function declaration where they appear. This is useful, existing behavior that we cannot change, but it might not make sense for all macros, and could be surprising. Therefore, we prohibit such default argument that are (non-built-in) macros to avoid confusion, and are open to revisiting this restriction in the future. + +### Macro implementation library + +Macro definitions will make use of the [swift-syntax](https://github.com/apple/swift-syntax) package, which provides the Swift syntax tree manipulation and parsing capabilities for Swift tools. The `SwiftSyntaxMacros` module will provide the functionality required to define macros. + +#### `Macro` protocols + +The `Macro` protocol is the root protocol for all kinds of macro definitions. At present, it does not have any requirements: + +```swift +public protocol Macro { } +``` + +All "freestanding" macros conform to the `FreestandingMacro` protocol: + +```swift +public protocol FreestandingMacro: Macro { } +``` + +The `ExpressionMacro` protocol is used to describe expression macros, and is a form of freestanding macro: + +```swift +public protocol ExpressionMacro: FreestandingMacro { + /// Expand a macro described by the given freestanding macro expansion syntax node + /// within the given context to produce a replacement expression. + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> ExprSyntax +} +``` + +The `FreestandingMacroExpansionSyntax` protocol is the `swift-syntax` node describing the `macro-expansion-expression` grammar term from above, so it carries the complete syntax tree (including all whitespace and comments) of the macro expansion as it appears in the source code. + +Macro definitions should conform to the `ExpressionMacro` protocol and implement their syntactic transformation via `expansion(of:in:)`, returning the new expression as a syntax node. + +If the macro expansion cannot proceed for some reason, the `expansion(of:in:)` operation can throw an error rather than try to produce a new syntax node. The compiler will then report the error to the user. More detailed diagnostics can be provided via the macro expansion context. + +#### `MacroExpansionContext` + +The macro expansion context provides additional information about the environment in which the macro is being expanded. This context can be queried as part of the macro expansion: + +```swift +/// Protocol whose conforming types provide information about the context in +/// which a given macro is being expanded. +public protocol MacroExpansionContext: AnyObject { + /// Generate a unique name for use in the macro. + public func makeUniqueName(_ name: String) -> TokenSyntax + + /// Emit a diagnostic (i.e., warning or error) that indicates a problem with the macro + /// expansion. + public func diagnose(_ diagnostic: Diagnostic) + + /// Retrieve a source location for the given syntax node. + /// + /// - Parameters: + /// - node: The syntax node whose source location to produce. + /// - position: The position within the syntax node for the resulting + /// location. + /// - filePathMode: How the file name contained in the source location is + /// formed. + /// + /// - Returns: the source location within the given node, or `nil` if the + /// given syntax node is not rooted in a source file that the macro + /// expansion context knows about. + func location( + of node: some SyntaxProtocol, + at position: PositionInSyntaxNode, + filePathMode: SourceLocationFilePathMode + ) -> AbstractSourceLocation? +} +``` + +The `makeUniqueName()` function allows one to create new, unique names so that the macro expansion can produce new declarations that won't conflict with any other declarations in the same scope. It produces an identifier token containing the unique name, which will also incorporate the `name` identifier for better debuggability. This allows macros to be more hygienic, by not introducing new names that could affect the way that the code provided via macro expansion arguments is type-checked. + +It is intended that `MacroExpansionContext` will grow over time to include more information about the build environment in which the macro is being expanded. For example, information about the target platform (such as OS, architecture, and deployment version) and any compile-time definitions passed via `-D`, should be included as part of the context. + +The `diagnose` method allows a macro implementation to provide diagnostics as part of macro expansion. The [`Diagnostic`](https://github.com/apple/swift-syntax/blob/main/Sources/SwiftDiagnostics/Diagnostic.swift) type used in the parameter is part of the swift-syntax library, and its form is likely to change over time, but it is able to express the different kinds of diagnostics a compiler or other tool might produce, such as warnings and errors, along with range highlights, Fix-Its, and attached notes to provide more clarity. A macro definition can introduce diagnostics if, for example, the macro argument successfully type-checked but used some Swift syntax that the macro implementation does not understand. The diagnostics will be presented by whatever tool is expanding the macro, such as the compiler. A macro that emits diagnostics is still expected to produce an expansion result unless it also throws an error, in which case both emitted diagnostics and the error will be reported. + +The `location` operation allows one to determine source location information for a syntax node. The resulting source location contains the file, line, and column for the corresponding syntax node. The `position` and `filePathMode` can be used to customize the resulting output, e.g., which part of the syntax node to point at and how to render the file name. + +```swift +/// Describe the position within a syntax node that can be used to compute +/// source locations. +public enum PositionInSyntaxNode { + /// Refers to the start of the syntax node's leading trivia, which is + /// the first source location covered by the syntax node. + case beforeLeadingTrivia + + /// Refers to the start of the syntax node's first token, which + /// immediately follows the leading trivia. + case afterLeadingTrivia + + /// Refers to the end of the syntax node's last token, right before the + /// trailing trivia. + case beforeTrailingTrivia + + /// Refers just past the end of the source text that is covered by the + /// syntax node, after all trailing trivia. + case afterTrailingTrivia +} + +/// Describes how a source location file path will be formed. +public enum SourceLocationFilePathMode { + /// A file ID consisting of the module name and file name (without full path), + /// as would be generated by the macro expansion `#fileID`. + case fileID + + /// A full path name as would be generated by the macro expansion `#filePath`, + /// e.g., `/home/taylor/alison.swift`. + case filePath +} +``` + +Source locations are described in an abstract form that can be interpolated into source code (they are expressions) in places that expect a string literal (for the file name) or integer literal (for line and column). As with `makeUniqueName` returning a `TokenSyntax` rather than a `String`, this abstraction allows the compiler to introduce a different kind of syntax node (that might not even be expressible in normal Swift) to represent these values. + +```swift +/// Abstractly represents a source location in the macro. +public struct AbstractSourceLocation { + /// A primary expression that represents the file and is `ExpressibleByStringLiteral`. + public let file: ExprSyntax + + /// A primary expression that represents the line and is `ExpressibleByIntegerLiteral`. + public let line: ExprSyntax + + /// A primary expression that represents the column and is `ExpressibleByIntegerLiteral`. + public let column: ExprSyntax +} +``` + +### Macros in the Standard Library + +#### `externalMacro` definition + +The builtin `externalMacro` macro is declared as follows: + +```swift +macro externalMacro(module: String, type: String) -> T +``` + +The arguments identify the module name and type name of the type that provides an external macro definition. Note that the `externalMacro` macro is special in that it can only be expanded to define another macro. It is an error to use it anywhere else, which is why it does not include an `@freestanding(expression)` attribute. + +#### Builtin macro declarations + +As previously noted, expression macros use the same leading `#` syntax as a number of built-in expressions like `#line`. With the introduction of expression macros, we propose to subsume those built-in expressions into macros that come as part of the Swift standard library. The actual macro implementations are provided by the compiler, and may even involve things that aren't necessarily implementable with the pure syntactic macro. However, by providing macro declarations we remove special cases from the language and benefit from all of the tooling affordances provided for macros. + +We propose to introduce a number of macro declarations into the Swift standard library. There are several different kinds of such macros. + +##### Source-location macros + +```swift +// File and path-related information +@freestanding(expression) macro fileID() -> T +@freestanding(expression) macro file() -> T +@freestanding(expression) macro filePath() -> T + +// Current function +@freestanding(expression) macro function() -> T + +// Source-location information +@freestanding(expression) macro line() -> T +@freestanding(expression) macro column() -> T + +// Current shared object handle. +@freestanding(expression) macro dsohandle() -> UnsafeRawPointer +``` + +The operations that provide information about the current location in source code are mostly implementable as `ExpressionMacro`-conforming types, using the `location` operation on the `MacroExpansionContext`. The exceptions are `#file`, which would need an extension to `MacroExpansionContext` to determine whether we are in a compilation mode where `#file` behaves like `#fileID` vs. behaving like [`#filePath`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0285-ease-pound-file-transition.md); `dsohandle`, which requires specific compiler support; and `#function`, which would require contextual information that is not available in the `MacroExpansionContext`. + +The type signatures of these macros capture most of the type system behavior of the existing `#file`, `#line`, etc., because they are treated like literals and therefore can pick up any contextual type that implements the proper `ExpressibleBy*` protocol. However, the implementations above would fail to type-check code like this: + +```swift +let x = #file +``` + +with an error such as + +``` +error: generic parameter 'T' could not be inferred +``` + +To match the existing behavior of the built-in `#file`, `#line`, etc. would require a defaulting rule that matches what we get for literal types. At present, this requires special handling in the compiler, but a future extension to the language to enable default generic arguments would likely allow us to express this notion directly in the type system. + +##### Objective-C helper macros + +The Swift `#selector` and `#keyPath` expressions can have their syntax and type-checking behavior expressed in terms of macro declarations: + +```swift +@freestanding(expression) macro selector(_ method: T) -> Selector +@freestanding(expression) macro selector(getter property: T) -> Selector +@freestanding(expression) macro selector(setter property: T) -> Selector +@freestanding(expression) macro keyPath(_ property: T) -> String +``` + +These macros cannot be implemented in terms of `ExpressionMacro` based on the facilities in this proposal, because one would need to determine which declarations are referenced within the argument of a macro expansion such as `#selector(getter: Person.name)`. However, providing them with macro declarations that have built-in implementations makes them less special, removing some special cases from more of the language. + +##### Object literals + +```swift +@freestanding(expression) macro colorLiteral(red: Float, green: Float, blue: Float, alpha: Float) -> T +@freestanding(expression) macro imageLiteral(resourceName: String) -> T +@freestanding(expression) macro fileLiteral(resourceName: String) -> T +``` + +The object literals allow one to reference a resource in a program of various kinds. The three kinds of object literals (color, image, and file) can be described as expression macros. The type signatures provided above are not exactly how type checking currently works for object literals, because they aren't necessarily generic. Rather, when they are used, the compiler currently looks for a specially-named type (e.g., `_ColorLiteralType`) in the current module and uses that as the type of the corresponding color literal. To maintain that behavior, we propose to type-check macro expansions for object literals by performing the same lookup that is done today (e.g., for `_ColorLiteralType`) and then using that type as the generic argument for the corresponding macro. That way, the type checking behavior is unchanged when moving from special object literal expressions in the language to macro declarations with built-in implementations. + +### Sandboxing macro implementations + +The details of how macro implementation modules are built and provided to the compiler will be left to a separate proposal. However, it's important to call out here that macro implementations will be executed in a sandbox [like SwiftPM plugins](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md#security), preventing file system and network access. This is both a security precaution and a practical way of encouraging macros to not depend on any state other than the specific macro expansion node they are given to expand and its child nodes (but not its parent nodes), and the information specifically provided by the macro expansion context. If in the future macros need access to other information, this will be accomplished by extending the macro expansion context, which also provides a mechanism for the compiler to track what information the macro actually queried. + +## Tools for using and developing macros + +One of the primary concerns with macros is their ease of use and development: how do we know what a macro does to a program? How does one develop and debug a new macro? + +With the right tool support, the syntactic model of macro expansion makes it easy to answer the first question. The tools will need to be able to show the developer what the expansion of any use of a macro is. At a minimum, this should include flags that can be passed to the compiler to expand macros (the prototype provides `-Xfrontend -dump-macro-expansions` for this), and possibly include a mode to write out a "macro-expanded" source file akin to how C compilers can emit a preprocessed source file. Other tools such as IDEs should be able to show the expansion of a given use of a macro so that developers can inspect what a macro is doing. Because the result is always Swift source code, one can reason about it more easily than (say) inspecting the implementation of a macro that manipulates an AST or IR. + +The fact that macro implementations are separate programs actually makes it easier to develop macros. One can write unit tests for a macro implementation that provides the input source code for the macro (say, `#stringify(x + y)`), expands that macro using facilities from swift-syntax, and verifies that the resulting code is free of syntax errors and matches the expected result. Most of the "builtin" macro examples were developed this way in the [syntax macro test file](https://github.com/apple/swift-syntax/blob/main/Tests/SwiftSyntaxMacrosTest/MacroSystemTests.swift). + +## Example expression macros + +There are many uses for expression macros beyond what has been presented here. This section will collect several examples of macro implementations based on existing built-in `#` expressions as well as ones that come from the Swift forums and other sources of inspiration. Prototype implementations of a number of these macros are [available in the swift-syntax repository](https://github.com/apple/swift-syntax/blob/main/Tests/SwiftSyntaxMacrosTest/MacroSystemTests.swift). + +* The `#colorLiteral` macro provides syntax for declaring a color with a given red, green, blue, and alpha values. It can be declared and implemented as follows + + ```swift + // Declaration of #colorLiteral + @freestanding(expression) macro colorLiteral(red: Float, green: Float, blue: Float, alpha: Float) -> _ColorLiteralType + = SwiftBuiltinMacros.ColorLiteralMacro + + // Implementation of #colorLiteral + struct ColorLiteralMacro: ExpressionMacro { + /// Replace the label of the first element in the tuple with the given + /// new label. + func replaceFirstLabel( + of tuple: TupleExprElementListSyntax, with newLabel: String + ) -> TupleExprElementListSyntax{ + guard let firstElement = tuple.first else { + return tuple + } + + return tuple.replacing( + childAt: 0, with: firstElement.withLabel(.identifier(newLabel))) + } + + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + let argList = replaceFirstLabel( + of: node.argumentList, with: "_colorLiteralRed" + ) + let initSyntax: ExprSyntax = ".init(\(argList))" + if let leadingTrivia = node.leadingTrivia { + return MacroResult(initSyntax.withLeadingTrivia(leadingTrivia)) + } + return initSyntax + } + } + ``` + + The same approach can be used for file and image literals. + +* [Power assertions](https://forums.swift.org/t/a-possible-vision-for-macros-in-swift/60900/87) by Kishikawa Katsumi: this assertion macro captures intermediate values within the assertion expression so that when the assertion fails, those values are displayed. The results are exciting! + + ```swift + #powerAssert(mike.isTeenager && john.age < mike.age) + | | | | | | | | + | true | | 42 | | 13 + | | | | Person(name: "Mike", age: 13) + | | | false + | | Person(name: "John", age: 42) + | false + Person(name: "Mike", age: 13) + ``` + +## Source compatibility + +Macros are a pure extension to the language, utilizing new syntax, so they don't have an impact on source compatibility. + +## Effect on ABI stability + +Macros are a source-to-source transformation tool that have no ABI impact. + +## Effect on API resilience + +Macros are a source-to-source transformation tool that have no effect on API resilience. + +## Future Directions + +There are a number of potential directions one could take macros, both by providing additional information to the macro implementations themselves and expanding the scope of macros. + +### Macro argument type information + +The arguments to a macro are fully type-checked before the macro implementation is invoked. However, information produced while performing that type-check is not provided to the macro, which only gets the original source code. In some cases, it would be useful to also have information determined during type checking, such as the types of the arguments and their subexpressions, the full names of the declarations referenced within those expressions, and any implicit conversions performed as part of type checking. For example, consider a use of a macro like the [power assertions](https://forums.swift.org/t/a-possible-vision-for-macros-in-swift/60900/87) mentioned earlier: + +```swift +#assert(Color(parsing: "red") == .red) +``` + +The implementation would likely want to separate the two operands to `==` into local variables (with fresh names generated by `createUniqueName`) to capture the values, so they can be printed later. For example, the assertion could be translated into code like the following: + +```swift +{ + let _unique1 = Color(parsing: "red") + let _unique2 = .red + if !(_unique1 == _unique2) { + fatalError("assertion failed: \(_unique1) != \(_unique2)") + } +}() +``` + +Note, however, that this code will not type check, because initializer for `_unique2` requires context information to determine how to resolve `.red`. If the macro implementation were provided with the types of the two subexpressions, `Color(parsing: "red")` and `.red`, it could have been translated into a something that will type-check properly: + +```swift +{ + let _unique1: Color = Color(parsing: "red") + let _unique2: Color = .red + if !(_unique1 == _unique2) { + fatalError("assertion failed: \(_unique1) != \(_unique2)") + } +}() +``` + +The macro expansion context could be extended with an operation to produce the type of a given syntax node, e.g., + +```swift +extension MacroExpansionContext { + func type(of node: ExprSyntax) -> Type? +} +``` + +When given one of the expression syntax nodes that is part of the macro expansion expression, this operation would produce a representation of the type of that expression. The `Type` would need to be able to represent the breadth of the Swift type system, including structural types like tuple and function types, and nominal types like struct, enum, actor, and protocol names. + +Additional information could be provided about the actual resolved declarations. For example, the syntax node for `.red` could be queried to produce a full declaration name `Color.red`, and the syntax node for `==` could resolve to the full name of the declaration of the `==` operator that compares two `Color` values. A macro could then distinguish between different `==` operator implementations. + +The main complexity of this future direction is in defining the APIs to be used by macro implementations to describe the Swift type system and related information. It would likely be a simplified form of a type checker's internal representation of types, but would need to remain stable. Therefore, while we feel that the addition of type information is a highly valuable extension for expression macros, the scope of the addition means it would best be introduced as a follow-on proposal. + +### Additional kinds of macros + +Expressions are just one place in the language where macros could be valuable. Other places could include function or closure bodies (e.g., to add tracing or logging), within type or extension definitions (e.g., to add new members), or on protocol conformances (e.g., to synthesize a protocol conformance). A number of potential ideas are presented in the [vision for macros in Swift](https://forums.swift.org/t/a-possible-vision-for-macros-in-swift/60900). For each of them, we assume that the basic `macro` declaration will stay roughly the same, but the contexts in which the macro can be used would be different, as might the spelling of the expansion (e.g., `@` might be more appropriate if the macro expansion occurs on a declaration), there would be an attribute on the `macro` declaration that indicates what type of macro it is, and there would be a corresponding protocol that inherits from `Macro` in the `SwiftSyntaxMacros` module. + +## Revision History + +* Revision after acceptance: + * Make the `ExpressionMacro.expansion(of:in:)` requirement non-`async`. +* Revisions based on review feedback: + * Switch `@expression` to `@freestanding(expression)` to align with the other macros proposals and vision document. + * Make the `ExpressionMacro.expansion(of:in:)` requirement `async`. + * Allow macro declarations to have opaque result types, and define the uniqueness rules. + * Simplify the grammar of macro declarations to be more function-like: they always require a parameter list, and if they have a return value, its type is specified following `->`. To account for macros that take no arguments, omitting both an argument list and trailing closures from a macro expansion expression will implicitly add `()`. + * Make `MacroExpansionContext` a class-bound protocol, because the state involving diagnostics and unique names needs to be shared, and the implementations could vary significantly between (e.g.) the compiler and a test harness. + * Introduce a general `location` operation on `MacroExpansionContext` to get the source location of any syntax node from a macro input. Remove the `moduleName` and `fileName`, which were always too limited to be useful. + * Allow macro parameters to have default arguments, with restrictions on what can occur within a default argument. + * Clarify that macro expansion cannot be recursive. + * Rename `createUniqueLocalName` to `makeUniqueName`; the names might not always be local in scope. Also add a parameter to it so developers can provide a partial name that will show up in the unique name. + * Prohibit the use of non-builtin macros as default arguments of parameters. +* Revisions from the second pitch: + * Moved SwiftPM manifest changes to a separate proposal that can explore the building of macros in depth. This proposal will focus only on the language aspects. + * Simplified the type signature of the `#externalMacro` built-in macro. + * Added `@expression` to the macro to distinguish it from other kinds of macros that could come in the future. + * Make `expansion(of:in:)` throwing, and have that error be reported back to the user. + * Expand on how the various builtin standard library macros will work. +* Revisions from the first pitch: + * Rename `MacroEvaluationContext` to `MacroExpansionContext`. + * Remove `MacroResult` and instead allow macros to emit diagnostics via the macro expansion context. + * Remove `sourceLocationConverter` from the macro expansion context; it provides access to the whole source file, which interferes with incremental builds. + * Rename `ExpressionMacro.apply` to `expansion(of:in)` to make it clear that it's producing the expansion of a syntax node within a given context. + * Remove the implementations of `#column`, as well as the implication that things like `#line` can be implemented with macros. Based on the above changes, they cannot. + * Introduce a new section providing declarations of macros for the various `#` expressions that exist in the language, but will be replaced with (built-in) macros. + * Replace the `external-macro-name` production for defining macros with the more-general `macro-expansion-expression`, and a builtin macro `externalMacro` that makes it far more explicit that we're dealing with external types that are looked up by name. This also provides additional capabilities for defining macros in terms of other macros. + * Add much more detail about how macro expansion works in practice. + * Introduce SwiftPM manifest extensions to define macro plugins. + * Added some future directions and alternatives considered. + + +## Acknowledgments + +Richard Wei implemented the compiler plugin mechanism on which the prototype implementation depends, as well as helping identify and explore additional use cases for macros. John McCall and Becca Royal-Gordon provided numerous insights into the design and practical implementation of macros. Tony Allevato provided additional feedback on building and sandboxing. diff --git a/proposals/0383-deprecate-uiapplicationmain-and-nsapplicationmain.md b/proposals/0383-deprecate-uiapplicationmain-and-nsapplicationmain.md new file mode 100644 index 0000000000..a2ea5c6da7 --- /dev/null +++ b/proposals/0383-deprecate-uiapplicationmain-and-nsapplicationmain.md @@ -0,0 +1,117 @@ +# Deprecate @UIApplicationMain and @NSApplicationMain + +* Proposal: [SE-0383](0383-deprecate-uiapplicationmain-and-nsapplicationmain.md) +* Authors: [Robert Widmann](https://github.com/codafi) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.10)** +* Upcoming Feature Flag: `DeprecateApplicationMain` +* Implementation: [PR 62151](https://github.com/apple/swift/pull/62151) +* Review: ([pitch](https://forums.swift.org/t/deprecate-uiapplicationmain-and-nsapplicationmain/61493)) ([review](https://forums.swift.org/t/se-0383-deprecate-uiapplicationmain-and-nsapplicationmain/62375)) ([acceptance](https://forums.swift.org/t/accepted-se-0383-deprecate-uiapplicationmain-and-nsapplicationmain/62645)) + +## Introduction + +`@UIApplicationMain` and `@NSApplicationMain` used to be the standard way for +iOS and macOS apps respectively to declare a synthesized platform-specific +entrypoint for an app. These functions have since been obsoleted by +[SE-0281](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0281-main-attribute.md)'s +introduction of the `@main` attribute, and they now represent a confusing bit of +duplication in the language. This proposal seeks to deprecate these alternative +entrypoint attributes in favor of `@main` in pre-Swift 6, and it makes their use +in Swift 6 a hard error. + +## Motivation + +UIKit and AppKit have fully embraced the `@main` attribute and have made +adoption by applications as simple as conforming to the `UIApplicationDelegate` +and `NSApplicationDelegate` protocols. This now means that an author of an +application is presented with two different, but ultimately needless, choices +for an entrypoint: + +* use one of the hard coded framework-specific attributes `@UIApplicationMain` or `@NSApplicationMain`, or +* use the more general `@main` attribute. + +At runtime, the behavior of the `@main` attribute on classes that conform to +one of the application delegate protocols above is identical to the corresponding +framework-specific attribute. Having two functionally identical ways to express the +concept of an app-specific entrypoint is clutter at best and confusing at worst. +This proposal seeks to complete the migration work implied by +[SE-0281](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0281-main-attribute.md) +by having the compiler push Swift authors towards the more general, unified +solution. + +## Proposed solution + +Using either `@UIApplicationMain` and `@NSApplicationMain` in a pre-Swift 6 +language mode will unconditionally warn and offer to replace these attributes +with the appropriate conformances. In Swift 6 language mode (and later), using +these attributes will result in a hard error. + +## Detailed design + +> Because `@UIApplicationMain` and `@NSApplicationMain` are used in identical +> ways, this portion of the document will only discuss `@UIApplicationMain`. +> The design for `@NSApplicationMain` follows the exact same pattern. + +Framework-specific attributes were added to the language to automate the +boilerplate involved in declaring a standard application entrypoint. In UIKit +code, the entrypoint always ends with a call to `UIApplicationMain`. The last +parameter of this call is the name of a subclass of `UIApplicationDelegate`. +UIKit will search for and instantiate this delegate class so it can issue +application lifecycle callbacks. Swift, therefore, requires this attribute to +appear on a class that conforms to the `UIApplicationDelegate` protocol so it +can provide the name of that class to UIKit. + +But a conformance to `UIApplicationDelegate` comes with more than just lifecycle +callbacks. A default implementation of a `main` entrypoint is [provided for +free](https://developer.apple.com/documentation/uikit/uiapplicationdelegate/3656306-main) +to a conforming type, but the `@UIApplicationMain` attribute suppresses it. This +fact is key to the migration path for existing users of the framework-specific +attribute. + +Under this proposal, when the compiler sees a use of `@UIApplicationMain`, it +will emit a diagnostic including a suggestion to replace the attribute with +`@main`. In Swift 6 and later language modes, this diagnostic will be an error; +otherwise it will be a warning. + +```swift +@UIApplicationMain // warning: '@UIApplicationMain' is deprecated in Swift 5 + // fixit: Change `@UIApplicationMain` to `@main` +final class MyApplication: UIResponder, UIApplicationDelegate { + /**/ +} +``` + +Once the fixit has been applied, the result will be + +```swift +@main +final class MyApplication: UIResponder, UIApplicationDelegate { + /**/ +} +``` + +This simple migration causes the compiler to select the `main` entrypoint +inherited by the conformance to `UIApplicationDelegate`. No further source +changes are required. + +## Source compatibility + +Current Swift libraries will continue to build because they compile under +pre-Swift 6 language modes. Under such language modes this proposal adds only an +unconditional warning when framework-specific entrypoints are used, and provides +diagnostics to avoid the warning by automatically migrating user code. + +In Swift 6 and later modes, this proposal is intentionally source-breaking as the +compiler will issue an unconditional error upon encountering a framework-specific +attribute. This source break will occur primarily in older application code, as +most libraries and packages do not use framework-specific attributes to define a main +entrypoint. Newer code, including templates for applications provided by Xcode 14 +and later, already use the `@main` attribute. + +## Effect on ABI stability + +This proposal has no impact on ABI. + +## Effect on API resilience + +None. diff --git a/proposals/0384-importing-forward-declared-objc-interfaces-and-protocols.md b/proposals/0384-importing-forward-declared-objc-interfaces-and-protocols.md new file mode 100644 index 0000000000..e7da438da2 --- /dev/null +++ b/proposals/0384-importing-forward-declared-objc-interfaces-and-protocols.md @@ -0,0 +1,244 @@ +# Importing Forward Declared Objective-C Interfaces and Protocols + +* Proposal: [SE-0384](0384-importing-forward-declared-objc-interfaces-and-protocols.md) +* Author: [Nuri Amari](https://github.com/NuriAmari) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 5.9)** +* Implementation:[apple/swift#61606]( https://github.com/apple/swift/pull/61606) +* Upcoming Feature Flag: `ImportObjcForwardDeclarations` +* Review: ([pitch](https://forums.swift.org/t/pitch-importing-forward-declared-objective-c-classes-and-protocols/61926)) ([review](https://forums.swift.org/t/se-0384-importing-forward-declared-objective-c-interfaces-and-protocols/62392)) ([acceptance](https://forums.swift.org/t/accepted-se-0384-importing-forward-declared-objective-c-interfaces-and-protocols/62670)) + +## Introduction + +This proposal seeks to improve the usability of existing Objective-C libraries from Swift by reducing the negative +impact forward declarations have on API visibility from Swift. We wish to start synthesizing placeholder types to +represent forward declared Objective-C interfaces and protocols in Swift. + +## Motivation + +Forward declarations are very common in many existing Objective-C code bases, used often to break cyclic dependencies or to improve build performance. +Unfortunately, when it comes to "importability" into Swift they are quite detrimental. + +As it stands, the ClangImporter will fail to import any declaration that references a forward declared type in many common cases. This means a single +forward declared type can render larger portions of an Objective-C API unusable from Swift. For example, the following Objective-C API from the implementation PR +is empty from the Swift perspective (Swift textual interface generated via swift-ide-test): + +_Objective-C_ +```objective-c +#import + +@class ForwardDeclaredInterface; +@protocol ForwardDeclaredProtocol; + +@interface IncompleteTypeConsumer1 : NSObject +@property id propertyUsingAForwardDeclaredProtocol1; +@property ForwardDeclaredInterface *propertyUsingAForwardDeclaredInterface1; +- (id)init; +- (NSObject *)methodReturningForwardDeclaredProtocol1; +- (ForwardDeclaredInterface *)methodReturningForwardDeclaredInterface1; +- (void)methodTakingAForwardDeclaredProtocol1: + (id)param; +- (void)methodTakingAForwardDeclaredInterface1: + (ForwardDeclaredInterface *)param; +@end + +ForwardDeclaredInterface *CFunctionReturningAForwardDeclaredInterface1(); +void CFunctionTakingAForwardDeclaredInterface1( + ForwardDeclaredInterface *param); + +NSObject *CFunctionReturningAForwardDeclaredProtocol1(); +void CFunctionTakingAForwardDeclaredProtocol1( + id param); +``` + +_Swift_ +```swift +class IncompleteTypeConsumer1 : NSObject { + init!() +} +``` +It is possible to fix such issues by importing the definition of the type, either in the Objective-C header or in the consuming Swift file. However, such manual changes make consuming existing libraries more difficult and needlessly increase build times. We wish to make the experience of consuming an Objective-C API with forward declarations from Swift more consistent with the experience of consuming the API from Objective-C. + +We have done some work in the past to help diagnose such failures in the ClangImporter: + +https://forums.swift.org/t/pitch-improved-clangimporter-diagnostics/52687/17 + +https://forums.swift.org/t/pitch-lazy-clangimporter-diagnostics-enabled-by-default/54651 + +We want to build on this work, specifically with respect to forward declarations, this time +fixing a shortcoming of the ClangImporter, not just diagnosing it. + +## Proposed solution + +We propose the following representation for forward declared Objective-C interfaces and protocols in Swift: + +```swift +// @class Foo turns into +@available(*, unavailable, message: “This Objective-C class has only been forward declared; import its owning module to use it”) +class Foo : NSObject {} + +// @protocol Bar turns into +@available(*, unavailable, message: “This Objective-C protocol has only been forward declared; import its owning module to use it”) +protocol Bar : NSObjectProtocol {} +``` + +The idea is to introduce the minimal change that will make Objective-C APIs usable in a predictable safe manner. + +The aforementioned Objective-C API with this change looks like this from Swift: + +```swift +@available(*, unavailable, message: "This Objective-C class has only been forward-declared; import its owning module to use it") +class ForwardDeclaredInterface { +} +@available(*, unavailable, message: "This Objective-C protocol has only been forward-declared; import its owning module to use it") +protocol ForwardDeclaredProtocol : NSObjectProtocol { +} +class IncompleteTypeConsumer1 : NSObject { + var propertyUsingAForwardDeclaredProtocol1: ForwardDeclaredProtocol! + var propertyUsingAForwardDeclaredInterface1: ForwardDeclaredInterface! + init!() + func methodReturningForwardDeclaredProtocol1() -> ForwardDeclaredProtocol! + func methodReturningForwardDeclaredInterface1() -> ForwardDeclaredInterface! + func methodTakingAForwardDeclaredProtocol1(_ param: ForwardDeclaredProtocol!) + func methodTakingAForwardDeclaredInterface1(_ param: ForwardDeclaredInterface!) +} +func CFunctionReturningAForwardDeclaredInterface1() -> ForwardDeclaredInterface! +func CFunctionTakingAForwardDeclaredInterface1(_ param: ForwardDeclaredInterface!) +func CFunctionReturningAForwardDeclaredProtocol1() -> ForwardDeclaredProtocol! +func CFunctionTakingAForwardDeclaredProtocol1(_ param: ForwardDeclaredProtocol!) +``` + +More usage examples can be found in these tests introduced here: https://github.com/apple/swift/pull/61606 + +## Detailed design + +Modifications lie almost exclusively in `ClangImporter` -- specifically in `SwiftDeclConverter`. If asked to convert a +`clang::ObjCInterfaceDecl` or `clang::ObjCProtocolDecl` with no definition `SwiftDeclConverter` will now return a placeholder type instead of +bailing. These placeholder types are as described: + +```swift +// @class Foo turns into +@available(*, unavailable, message: “This Objective-C class has only been forward declared; import its owning module to use it”) +class Foo : NSObject {} + +// @protocol Bar turns into +@available(*, unavailable, message: “This Objective-C protocol has only been forward declared; import its owning module to use it”) +protocol Bar : NSObjectProtocol {} +``` + +Permitted usages of these types are intentionally limited. You will be able to use Objective-C and C declarations that refer to these types without issue. +You will be able to pass around instances of these incomplete types from Swift to Objective-C and vice versa. + +You’ll also be able to call the set of methods provided by `NSObject` or `NSObjectProtocol` on instances. Notably this assumes +that the backing Objective-C implementation does indeed inherit from / conform to `NSObject`. We are under the impression that +inheriting from `NSObject` is a requirement for Swift - Objective-C interop, but are not sure about protocols conforming to `NSObject`. +We could use some input on these assumptions, and will remove this feature if they turn out to be unsound. + +Using the type itself directly in Swift is forbidden by the attached unavailable attribute. This is to keep the impact of the change small and to prevent unsound declarations, +such as declaring a new Swift class that inherits from or conforms to such a type. You will also not be able to create new instances of these types in Swift. + +To make these limitations more intuitive to users, a new diagnostic has been added. This diagnostic helps guide users when they attempt to access a member on the incomplete synthesized +type, expecting the complete type. The new diagnostics are shown in the last 4 lines of this example: + +```swift +swift-client-fwd-declared.swift:6:7: error: value of type 'Foo' has no member 'sayHello' +myFoo.sayHello() +~~~~~ ^~~~~~~~ +:0: note: class 'Foo' is a placeholder for a forward declared Objective-C interface and may be missing members; import the definition to access the complete interface +foo-bar-consumer.h:3:1: note: interface 'Foo' forward declared here +@class Foo; +^ +``` + +In Swift 5.x, the feature is gated behind the upcoming feature flag `ImportObjcForwardDeclarations`. This flag is on by default for Swift version 6 and onwards. + +The flag is always disabled in the REPL, as allowing it currently leads to confusing behavior. In the REPL, the environment in terms of whether a complete definition of some type Foo is available can change during execution. These examples show some of the confusing behavior: + +```swift +import IncompleteFoo +let foo = FunctionReturningFoo() +FunctingTakingFoo(foo) +import CompleteFoo // This has no effect as far as Swift's view of type 'Foo' is concerned +let completeFoo = Foo() // error type 'IncompleteFoo.Foo' has no initializer +foo.methodOnCompleteFoo() // error no member 'methodOnCompleteFoo' on type 'IncompleteFoo.Foo' +``` + +```swift +import IncompleteFoo +import CompleteFoo +let completeFoo = Foo() // This time it works because the cache was not populated until after CompleteFoo was imported +foo.methodOnCompleteFoo() // Works as well +``` + +This happens because `ClangImporter` is lazy and caches imports, once the placeholder is synthesized (when only the incomplete definition is +available), the complete definition will never be imported. Without an understanding of `ClangImporter`, it is impossible to understand why +the statements between the two imports have an effect on those after. We have made attempts at solving these issues, but no satisfactory solution +was found. The details of these attempts are listed in the alternatives considered section. + +The [existing ClangImporter diagnostics](https://forums.swift.org/t/pitch-improved-clangimporter-diagnostics/52687) will still properly warn users about forward declaration limitations, with or with out the feature enabled. + +## Source compatibility + +This change does have the potential to break source compatibility, since +`ClangImporter` will now import declarations that were previously dropped. +That said, the impact should rare. The current implementation already passes +the source compatibility suite and the source kit stress test. + +As mentioned, to address source compatibility issues, the feature is gated behind +a flag for Swift 5 and before, and enabled by default in Swift 6. + +## Effect on ABI stability + +This change has no affect on ABI. + +## Effect on API resilience + +This change does not influence the ABI characteristics +of public APIs. + +## Alternatives considered + +The unavailable attribute attached to synthesized declarations might be more restrictive than necessary. +We considered the introduction of a new attribute to denote an incomplete type. This attribute would be used to prevent +operations that are not suitable for an incomplete type (ie. declaring a Swift type that conforms to an incomplete protocol). + +Allowing more use of the type however increases the complexity of the problem, and the likelihood of source compatibility issues. +For example, if the synthesized type is not file private, issues could arise with different files having different representations +of the type depending on which modules they import. + +We also attempted to make this implementation usable from the REPL. As mentioned, we had issues with `ClangImporter` caching +imports. We must invalidate the cache, at the very least to allow the new definition of type 'Foo' to be imported. This creates issues +where two incompatible representations of the type (`IncompleteFoo.Foo` and `CompleteFoo.Foo`) are in context at the same time. +To solve these issues we attempted the following: + +1. Invalidate all direct or indirect references to `IncompleteFoo.Foo` and replace them with `CompleteFoo.Foo`: + +Issues: + +- We do not know how to safely patch the type of existing instances at runtime (ex: local variable `foo` in the snippet above) +- Any imported declarations that depended upon a type, whose definition was incomplete at the time of import but is now complete, need reimporting + - We need new machinery in `ClangImporter` to support smart cache invalidation like this + - For all cached declarations, we need to keep track of the "completeness" of all types we depend on at the time of caching + - A simple cache invalidation heuristic such as "don't cache anything that depends upon an incomplete type" won't work either + - Besides the significant performance cost, the typechecker compares type declarations using pointer equality, expansion of the typechecker to recognize two imports of the same Clang declaration as equivalent would be required + +2. Convince the typechecker that `IncompleteFoo.Foo` and `CompleteFoo.Foo` are the same, *in all usages*: + +Issues: + +- It seems achievable to teach the typechecker that `IncompleteFoo.Foo` and `CompleteFoo.Foo` can be implicitly converted between one another, but that is not enough + - Any typechecking constraint that `CompleteFoo.Foo` satisfies must also be satisfied by `IncompleteFoo.Foo`. + - It might be possible to "inject" an extension into the REPL's context that adds any required methods, computed properties and protocol conformances to `IncompleteFoo.Foo`. However, it is not possible to add new inheritances. + +We believe that given these issues, it is better to disable the feature in the REPL entirely rather than provide a confusing experience. We believe that this +proposal should advance in some form regardless. Forward declarations are a huge problem for consumers of large Objective-C codebases looking to transition to Swift. +The LLDB / REPL experience already diverges from the compiled experience, is relatively niche, and we are not making the REPL experience any worse. Furthermore, +unlike in large scale projects, the existing solution of "import the definition" is perfectly reasonable for small REPL sessions. If this divergence between the +REPL and compiled experience is unacceptable, we think it could be reasonable to gate the feature behind a flag. This flag could potentially also be enabled in +the REPL, but with appropriate warnings and "at your own risk". + +That said, it would be great if this feature could come to the REPL eventually, but the REPL should not stand in the way of progress in the compiled experience. + +## Acknowledgments + +Thank you to @drodriguez and [@rmaz](https://github.com/rmaz) for helping develop the idea. diff --git a/proposals/0385-custom-reflection-metadata.md b/proposals/0385-custom-reflection-metadata.md new file mode 100644 index 0000000000..8c7d571cf9 --- /dev/null +++ b/proposals/0385-custom-reflection-metadata.md @@ -0,0 +1,443 @@ +# Custom Reflection Metadata + +* Proposal: [SE-0385](0385-custom-reflection-metadata.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin), [Holly Borla](https://github.com/hborla), [Alejandro Alonso](https://github.com/Azoy), [Stuart Montgomery](https://github.com/stmontgomery) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Returned for revision** ([Rationale](https://forums.swift.org/t/returned-for-revision-se-0385-custom-reflection-metadata/63758)) +* Implementation: [PR#1](https://github.com/apple/swift/pull/62426), [PR#2](https://github.com/apple/swift/pull/62738), [PR#3](https://github.com/apple/swift/pull/62818), [PR#4](https://github.com/apple/swift/pull/62850), [PR#5](https://github.com/apple/swift/pull/62920), [PR#6](https://github.com/apple/swift/pull/63057) + +## Introduction + +In Swift, declarations are annotated with attributes to opt into both built-in language features (e.g. `@available`) and library functionality (e.g. `@RegexComponentBuilder`). This proposal introduces the ability to attach library-defined reflection metadata to declarations using custom attributes, which can then be queried by the library to opt client code into library functionality. + +Previous Swift Forum discussions + +* [Custom attributes](https://forums.swift.org/t/custom-attributes/13976) +* [Pitch: introduce custom attributes](https://forums.swift.org/t/pitch-introduce-custom-attributes/21335) + +## Motivation + +There are some problem domains in which it can be beneficial for a library author to let a client annotate within their own code certain declarations that the library should be made aware of, since requiring the client call an explicit API instead would be too onerous, repetitive, or easy to forget. + +One classic example is **testing**: there is a common pattern in unit testing libraries where users define a type that extends one of the library's types, they annotate some of its methods as tests, and the library locates, initializes, and runs all tests automatically. There is no official mechanism in Swift to implement this test discovery pattern today, however XCTest—the language's current de-facto standard test library—has longstanding workarounds: + +* On Apple platforms, XCTest relies on the Objective-C runtime to enumerate all subclasses of a known base class and their methods, and it considers all instance methods with a supported signature and name prefixed with "test" a test method. +* On other platforms, XCTest is typically used from a package, and the Swift Package Manager has special logic to introspect build-time indexer data, locate test methods, and explicitly pass the list of discovered tests to XCTest to run. + +XCTest's current approach has some drawbacks and limitations: + +* Users must adhere to a strict naming convention by prefixing all test methods with the word "test". This prefix can be redundant since all tests include it, and may surprise users if they accidentally use that prefix on a non-test method since the behavior is implicit. +* Since tests are declared implicitly, there is no way for a user to provide additional details about an individual test or group of tests. It would be useful to have a way to indicate whether a test is enabled, its requirements, or other metadata, for example, so that the testing library could use this information to inform how it executes tests and offer more powerful features. +* The lack of a built-in runtime discovery mechanism means that related tools (such as Swift Package Manager) require specialized discovery logic for each test library they support. This makes adding support for alternate test libraries to those tools very difficult and increases their implementation complexity. + +Registering code to be discovered by a framework is a common pattern across Swift programs. For example, a program that uses a plugin architecture commonly uses a protocol for the interface of the plugin, which is then implemented on concrete types in clients. This pattern imposes error-prone registration boilerplate, where clients must explicitly supply a list of concrete plugin types or explicitly register individual plugin types to be used by the framework before the framework needs them. + +More generally, annotating parts of a program with metadata to be used by other parts of the program has use-cases beyond registration patterns. Consider the `Persisted` property wrapper from the [Realm](https://github.com/realm/realm-swift) package: + +```swift +@propertyWrapper +public struct Persisted { ... } + +class Dog: Object { + @Persisted var name: String + @Persisted var age: Int +} +``` + +To support [advanced schema customization](https://forums.swift.org/t/se-0385-custom-reflection-metadata/62777/19), the property wrapper could store a string that provides a custom name for the underlying database column, specified in the attribute arguments, e.g. `@Persisted(named: "CustomName")`. However, storing this metadata in the property wrapper requires additional storage for each _instance_ of the containing type, even though the metadata value is fixed for the declaration the property wrapper is attached to. In addition to higher memory overload, the metadata values are evaluated eagerly, and for each instantiation of the containing type, rendering property-wrapper instance metadata too expensive for this use case. + + +## Proposed solution + +* A new built-in attribute `@reflectionMetadata` that can be applied to structs, enums, classes, and actors. +* Types annotated with this built-in attribute can be used as custom attributes on declarations that can be used as values. + * The custom attribute can have additional arguments; the custom attribute application will turn into an initializer call on the attribute type, passing in the declaration value as the first argument. +* A reflection API that can gather all declarations with a given custom attribute attached, which lazily constructs the metadata values when invoked. + +Combined with [attached macros](https://forums.swift.org/t/pitch-attached-macros/62812), the `@Persisted` property wrapper in Realm can evolve into a macro attached to persistent types, combined with custom metadata attributes that provide schema customization for specific declarations: + +```swift +@reflectionMetadata +struct Named { + let name: String + + init(attachedTo: T.Type, _ name: String) { + self.name = name + } +} + +@Persisted +class Dog: Object { + var name: String + @Named("CustomName") var age: Int +} +``` + +This approach completely eliminates initialization overhead of using property wrappers, provides separate storage of custom metadata values, and enables lazy initialization of metadata values that is only invoked when the framework requests the metadata. + +## Detailed design + +### Declaring reflection metadata attributes + +Reflection metadata custom attributes are declared by attaching the built-in `@reflectionMetadata` attribute to a nominal type, i.e. a struct, enum, class, or actor: + +```swift +@reflectionMetadata +struct Example { ... } +``` + +A reflection metadata type must have a synchronous initializer of the form `init(attachedTo:)`. The type of the `attachedTo:` parameter dictates which types of declarations the custom attribute can be applied to, as described in the following section. + +### Applications of reflection metadata types + +Reflection metadata custom attributes can be applied to any declaration that can be used as a first-class value in Swift, including: + +* Types +* Global functions +* Static methods +* Instance methods, both non-mutating and mutating +* Instance properties + +Reflection metadata types opt into which kinds of declarations are supported based on their initializer overloads which begin with a parameter labeled `attachedTo:`. For an application of a reflection metadata attribute to be well-formed, the reflection metadata type must declare an initializer that accepts the appropriate value as the first argument. Applications of a reflection metadata type to a declaration will synthesize an initializer call with the attribute arguments, and the declaration value passed as the first initializer argument: + +* Types will pass a metatype. +* Global functions will pass an unapplied function reference. +* Static methods on a type `T` will pass a function which calls the method on the metatype `T.Type` passed as the first parameter. +* Instance methods on a type `T` will pass a function which calls the method on an instance `T` passed as the first parameter. The function will support `mutating` instance methods when the first parameter is declared `inout T`. +* Instance properties will pass a key-path. + +```swift +@reflectionMetadata +struct Flag { + // Initializer that accepts a metatype of a nominal type + init(attachedTo: T.Type) { + // ... + } + + // Initializer that accepts an unapplied reference to a global function + init(attachedTo: (Args) -> Result) { + // ... + } + + // Initializer that accepts a function which calls a static method + init(attachedTo: (T.Type, Args) -> Result) { + // ... + } + + // Initializer that accepts a function which calls an instance method + init(attachedTo: (T, Args) -> Result) { + // ... + } + + // Initializer that accepts a function which calls a mutating instance method + init(attachedTo: (inout T, Args) -> Result) { + // ... + } + + // Initializer that accepts a reference to an instance property + init(attachedTo: KeyPath, custom: Int) { + // ... + } +} + +// The compiler will synthesize the following initializer call +// -> Flag.init(attachedTo: doSomething) +@Flag func doSomething(_: Int, other: String) {} + +// The compiler will synthesize the following initializer call +// -> Flag.init(attachedTo: Test.self) +@Flag +struct Test { + // The compiler will synthesize the following initializer call + // -> Flag.init(attachedTo: { metatype in metatype.computeStateless() }) + @Flag static func computeStateless() {} + + // The compiler will synthesize the following initializer call + // -> Flag.init(attachedTo: { instance, values in instance.compute(values: values) }) + @Flag func compute(values: [Int]) {} + + var state = 1 + + // The compiler will synthesize the following initializer call + // -> Flag.init(attachedTo: { (instance: inout Test) in instance.incrementState() }) + @Flag mutating func incrementState() { + state += 1 + } + + // The compiler will synthesize the following initializer call + // -> Flag.init(attachedTo: \Test.answer, custom: 42) + @Flag(custom: 42) var answer: Int = 42 +} +``` + +#### Restrictions on custom reflection metadata application + +A given declaration can have multiple reflection metadata attributes as long as a given reflection metadata type only appears once: + +```swift +@Flag @Ignore func ignored() { 🟢 + // ... +} + +@Flag @Flag func specialFunction() { 🔴 + ^ error: duplicate reflection metadata attribute + // ... +} +``` + +Reflection metadata attributes must be applied at either the primary declaration of a type or in an unavailable unconstrained extension of the type within the same module as the type’s primary declaration. Unavailable extensions are supported to allow API implementers a way to opt-out from an attribute. Applying the attribute to a type in an available/constrained extension or in extension outside its module is prohibited to prevent the same type from having multiple reflection metadata annotations of the same type. + +```swift +@available(*, unavailable) +@Flag extension MyType { 🟢 if extension is in the same module +} +``` + +```swift +@Flag extension MyType { 🔴 + ^ error: cannot associate reflection metadata @Flag with MyType in extension +} +``` + +```swift +@Flag extension MyType where ... { 🔴 + ^ error: cannot associate reflection metadata @Flag with MyType in constrained extension +} +``` + +Declarations with custom reflection metadata attributes must be fully concrete: + +```swift +struct GenericType { + @Flag + var genericValue: T 🔴 + ^ error +} + +extension GenericType where T == Int { + @Flag + var concreteValue: Int // okay +} +``` + +Generic declarations cannot be discovered through the Reflection query that gathers all instances of reflection metadata, because generic values cannot be represented in a higher-kinded way in Swift; generic values must always have substitutions at runtime. Generic declarations could be supported in the future by adding reflection queries for the other direction, e.g. a query to return the custom reflection metadata for a given key-path `\Generic.value`. + + +### Inference of reflection metadata attributes + +A reflection metadata attribute can be applied to a protocol: + +```swift +@EditorCommandRecord +protocol EditorCommand { /* ... */ } +``` + +Conceptually, the reflection metadata attribute is applied to the generic `Self` type that represents the concrete conforming type. When a protocol conformance is written at the primary declaration of a concrete type, the reflection metadata attribute is inferred: + +```swift +// @EditorCommandRecord is inferred +struct SelectWordCommand: EditorCommand { /* ... */ } +``` + +If the protocol conformance is written in an extension on the conforming type, attribute inference is prohibited. A reflection metadata attribute applied to a protocol is a form of requirement, so such conformances declared in extensions are invalid unless the primary declaration already has the explicit reflection metadata attribute: + +```swift +// Error unless the primary declaration of 'SelectWordCommand' has '@EditorCommandRecord' +extension SelectWordCommand : EditorCommand { 🔴 + // ... +} +``` + +Reflection metadata attributes applied to protocols cannot have additional attribute arguments; attribute arguments must be explicitly written on the conforming type. + +A type which conforms to a protocol that has a reflection metadata attribute may specify the attribute explicitly. This can be useful if the reflection metadata type includes additional parameters in its `init(attachedTo: ...)` overload, since it allows the conforming type to pass arguments for those parameters: + +```swift +// Overrides the inferred `@EditorCommandRecord` attribute from `EditorCommand` +@EditorCommandRecord(keyboardShortcut: "j", modifier: .command) +struct SelectWordCommand: EditorCommand { /* ... */ } +``` + +### Accessing metadata through Reflection + +With the introduction of the new [Reflection](https://forums.swift.org/t/pitch-reflection/61438) module, we feel a natural place to reflectively retrieve these attributes is there. The following Reflection APIs provide the runtime query for custom reflection metadata: + +```swift +/// Get all the instances of a custom reflection attribute wherever it's attached to. +/// +/// - Parameters: +/// - type: The type of the attribute that is attached to various sources. +/// - Returns: A sequence of attribute instances of `type` in no particular +/// order. +public enum Attribute { + public static func allInstances(of type: T.Type) -> AttributeInstances +} + +/// A sequence wrapper over some runtime attribute instances. +/// +/// Instances of `AttributeInstances` are created with the +/// `Attribute.allInstances(of:)` function. +public struct AttributeInstances {} + +extension AttributeInstances: IteratorProtocol { + @inlinable + public mutating func next() -> T? +} + +extension AttributeInstances: Sequence {} +``` + +This API will retrieve all of the instances of your reflection attribute across all modules. Instances of metadata types are initialized in the Reflection query to gather the metadata. Attributes who are not available in the current running OS, i.e. because the `attachedTo` declaration is not available as described in the following section, will be excluded from the results. + +### Magic literals in custom reflection metadata attributes + +When custom reflection metadata type is accessed through the Reflection APIs, magic literals - `#function`, `#file`, `#line`, and `#column` associated with `init(attachedTo:)` would behave in a special way. Even though in such cases `init(attachedTo:)` is called from a special generator function `#function` literal is still going to point to the declaration attribute is attached to, and `#file`, `#line`, and `#column` are going to point to the attribute itself at the point of use or to the declaration if the attribute has been inferred. + +**test.swift** + +```swift + 1: @reflectionMetadata + 2: struct Flag { + 3: init(attachedTo: T.Type, + 4: func: String = #function, + 5: file: String = #file, + 6: line: Int = #line, + 7: column: Int = #column) {} + 8: + 9: init(attachedTo: KeyPath, +10: func: String = #function, +11: file: String = #file, +12: line: Int = #line, +13: column: Int = #column) {} +14: } +15: +16: struct Test { +17: @Flag var value: Int = 42 +18: } +19: +20: @Flag +21: protocol Flagged {} +22: +23: struct InferredTest : Flagged {} +``` + +**other.swift** + +```swift +1: let flags = Attribute.allInstances(of: Flag.self) +``` + +`Flag.init(attachedTo:)` associated with `Test.value` in this case is going to receive the following information: + +* `#function` = `"value"` +* `#file` = `"test.swift"` +* `#line` = `17` +* `#column` = `4` + +`Flag.init(attachedTo:)` for implicitly inferred attribute on `InferredTest` in this case is going to receive the following information: + +* `#function` = `"InferredTest"` +* `#file` = `"test.swift"` +* `#line` = `23` +* `#column` = `1` + +We think that this behavior provides the most benefit to the users because it preserves all of the information about attribute locations. + + +### API Availability + +Custom metadata attributes can be attached to declarations with limited availability. The Reflection query for an individual instance of the metadata attribute type will be gated on a matching availability condition and will return `nil` for instances which are unavailable at runtime. For example: + +```swift +@available(macOS 12, *) +@Flag +struct NewType { /* ... */ } +``` + +The Reflection query that produces the `Flag` instance attached to `NewType` will effectively execute the following code: + +```swift +if #available(macOS 12, *) { + return Flag(attachedTo: NewType.self) +} else { + return nil +} +``` + +and if `nil` is returned, there will not be a `Flag` instance representing `NewType` included in the collection returned by `Attribute.allInstances(of:)`. + +## Alternatives considered + +### Extend other language features + +Some reviewers of the original pitch suggested that the motivating use cases could be addressed through a combination of improved Reflection capabilities and enhancing existing language features. For example: + +* We could use existing protocol conformance metadata to allow discovering all types conforming to a protocol. +* We could allow property wrappers to be used to discover properties via reflection. + +These suggestions have some notable downsides, however. Supporting discovery of all types that conform to *any* protocol would be very expensive, and the majority of protocols do not need this reflection capability. Opting-in to this capability via an attribute on protocols which require it is an intentional aspect of this feature’s design intended to mitigate this cost. + +It’s also important to note that a reflection API which *only* allows discovering types that conform to a protocol would be insufficient to satisfy some of the use cases which motivate this feature because it would not allow including additional, custom values in the reflection metadata. For example, the `@EditorCommandRecord(keyboardShortcut: "j", modifier: .command)` example shown above includes custom values on a type conforming to a protocol, and the design of this feature includes a way for the reflection query to retrieve these values in addition to the declaration the attribute was attached to. For types conforming to a protocol, similar functionality could be provided through protocol requirements, but this strategy does not generalize to enable providing custom metadata on functions or computed properties. + +Regarding the use of property wrappers to represent metadata on properties: We feel that property wrappers are not an ideal tool for reflection metadata because they require an instance of the backing property to be stored for each instance, even though the wrapper is constant per-declaration. Property wrappers that are *only* used for reflection metadata don’t need to introduce any access indirection of the wrapped value, either. The value itself can simply be stored inline in the type, rather than synthesizing computed properties. + +### Using reflection types in the `init(attachedTo:)` signature + +We considered using types from the [Reflection](https://forums.swift.org/t/pitch-reflection/61438) module to represent declarations which have reflection attributes. For example, Reflection’s `Field` could be used as the type of the first parameter in `init(attachedTo:)` when a reflection attribute is attached to a property declaration. + +But this design would not allow constraining the types of the declaration(s) the reflection attribute can be attached to using techniques like generic requirements or additional parameters after `attachedTo:` in an initializer, since Reflection types do not expose the interface type of the declaration they represent. For example, `Field` is not parameterized on the field’s type, which would prevent compile-time enforcement of requirements. + +### Use static methods instead of `init(attachedTo:)` overloads + +We considered using static methods such as `buildMetadata(attachedTo:)` instead of overloads of `init(attachedTo:)` on reflection metadata types to generate metadata instances. This could potentially allow the overloads of `buildMetadata` to return a different type than `Self`, or even an associated type from some protocol. For example: + +```swift +// Defined in either the standard library or Reflection +protocol Attribute { + associatedtype Metadata +} + +// Example usage +@reflectionMetadata +struct Flag: Attribute { + static func buildMetadata(attachedTo: ...) -> Metadata { /* ... */ } +} +``` + +This alternative has a potential advantage of making it easier for `@propertyWrapper` types to also act as `@reflectionMetadata` types, because it would mean that the storage for any additional, custom values used for metadata purposes only (which are constant for every instance of the declared property) could be stored separately rather than having those values be stored redundantly in every instance of a property wrapper. + +### Alternative attribute names + +We considered several alternative spellings of the attribute used to declare a reflection metadata type: + +* `@runtimeMetadata` +* `@dynamicMetadata` +* `@metadata` +* `@runtimeAnnotation` +* `@runtimeAttribute` +* `@reflectionAnnotation` + +### Bespoke `@test` attribute + +A previous Swift Evolution discussion suggested [adding a built-in `@test` attribute](https://forums.swift.org/t/rfc-in-line-tests/12111) to the language. However, registration is a general code pattern that is also used outside of testing, so allowing libraries to declare their own domain-specific attributes is a more general approach that supports a wider set of use cases. + +## Acknowledgments + +Thank you to [Thomas Goyne](https://forums.swift.org/u/tgoyne) for surfacing use cases in the Realm Swift project and insights into alternative design directions. + +## Revision history + +### Changes after first pitch + +* Changed the proposed function signature for reflection metadata type initializer overloads for instance methods to accept `T` as the first parameter, instead of an unapplied function reference, and allow `inout T` to support `mutating` instance methods. +* Changed the proposed function signature for reflection metadata type initializer overloads for static methods to accept `T.Type` as the first parameter, instead of an unapplied function reference. +* Changed the spelling of the proposed attribute from `@runtimeMetadata` to `@reflectionMetadata`. +* Added `@reflectionAnnotation` (suggested by @xedin) to the list of alternative attribute spellings considered. +* Updated the list of supported use cases in the "Applications of reflection metadata types" section by separating global functions and static methods into separate bullets, to describe their differing type signatures. In particular, the function parameter for static methods now has type `(T.Type, Args) -> Result`. +* Clarified paragraph describing where reflection metadata attribute can be applied, to mention it is allowed in extensions of a type within the same module as the type's primary declaration, just not in extensions outside the module. +* Mentioned the ability to explicitly specify a reflection attribute on a type conforming to a protocol with that attribute, and described how that can be useful for specifying additional custom values. Added a code example of this. +* Changed the proposed name and return type of the Reflection API to `func allInstances(of type: T.Type) -> AttributeInstances`, returning a custom `Sequence` type whose type is `T`. Clarified that the returned sequence will omit values which do not satisfy the API availability conditions at runtime, rather than including `nil` values for them. +* Added discussion of some alternatives that were considered involving extending Reflection capabilities and other existing language features. +* Added discussion of an alternative that was considered about using Reflection types as the parameters to `init(attachedTo:)`. +* Added discussion of an alternative that was considered about using static methods instead of `init(attachedTo:)` overloads. +* Clarified interaction between extensions and custom reflection metadata attributes. diff --git a/proposals/0386-package-access-modifier.md b/proposals/0386-package-access-modifier.md new file mode 100644 index 0000000000..da88d20016 --- /dev/null +++ b/proposals/0386-package-access-modifier.md @@ -0,0 +1,334 @@ +# New access modifier: `package` + +* Proposal: [SE-0386](0386-package-access-modifier.md) +* Authors: [Ellie Shin](https://github.com/elsh), [Alexis Laferriere](https://github.com/xymus) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#61546](https://github.com/apple/swift/pull/62700), [apple/swift#62704](https://github.com/apple/swift/pull/62704), [apple/swift#62652](https://github.com/apple/swift/pull/62652), [apple/swift#62652](https://github.com/apple/swift/pull/62652) +* Review: ([pitch](https://forums.swift.org/t/new-access-modifier-package/61459)) ([first review](https://forums.swift.org/t/se-0386-package-access-modifier/62808)) ([second review](https://forums.swift.org/t/second-review-se-0386-package-access-modifier/64086)) ([acceptance](https://forums.swift.org/t/accepted-se-0386-package-access-modifier/64904)) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/28fd2fb9b7258117f912cec5e5f7eb178520fbf2/proposals/NNNN-package-access-modifier.md), [2](https://github.com/swiftlang/swift-evolution/blob/32e51946296f67be79a58a8c23eb9d7460a06232/proposals/0386-package-access-modifier.md), [3](https://github.com/swiftlang/swift-evolution/blob/4a3a11b18037526cf8d83a9d10b22b94890727e8/proposals/0386-package-access-modifier.md) + +## Introduction + +This proposal introduces `package` as a new access modifier. Currently, to access a symbol in another module, that symbol needs to be declared `public`. However, a symbol being `public` allows it to be accessed from any module at all, both within a package and from outside of a package, which is sometimes undesirable. We need a new access modifier to enable more control over the visibility scope of such symbols. + +## Motivation + +At the most basic level, every Swift program is just a collection of declarations: functions, types, variables, and so on. In principle, every level of organization above this is arbitrary; all of those declarations could be piled into a single file, compiled, and run. In reality, Swift programs are organized into separate files, directories, libraries, and so on. At each level, this organization reflects programmer judgment about relationships, both in the code and in how it is developed. + +As a language, Swift recognizes some of these levels. Modules are the smallest unit of library structure, with an independent interface and non-cyclic dependencies, and it makes sense for Swift to recognize that in both namespacing and access control. Files are the smallest grouping beneath that and are often used to collect tightly-related declarations, so they also make sense to respect in access control. + +Packages, as expressed by the Swift Package Manager, are a unit of code distribution. Some packages contain just a single module, but it's frequently useful to split a package's code into multiple modules. For example, when a module contains some `internal` helper APIs, those APIs can be split out into a utility module and maybe reused by other modules or packages. + +However, because Swift does not recognize organizations of code above the module level, it is not possible to create APIs like this that are purely internal to the package. To be usable from other modules within the package, the API must be public, but this means it can also be used outside of the package. This allows clients to form unwanted source dependencies on the API. It also means the built module has to export the API, which has negative implications for code size and performance. + +For example, here’s a scenario where a client has access to a utility API from a package it depends on. The client `App` could be an executable or an Xcode project. It depends on a package called `gamePkg`, which contains two modules, `Game` and `Engine`. + + +Module `Engine` in `gamePkg`: +```swift +public struct MainEngine { + public init() { ... } + // Intended to be public + public var stats: String { ... } + // A helper function made public only to be accessed by Game + public func run() { ... } +} +``` + +Module `Game` in `gamePkg`: +```swift +import Engine + +public func play() { + MainEngine().run() // Can access `run` as intended since it's within the same package +} +``` + +Client `App` in `appPkg`: +```swift +import Game +import Engine + +let engine = MainEngine() +engine.run() // Can access `run` from App even if it's not an intended behavior +Game.play() +print(engine.stats) // Can access `stats` as intended +``` + +In the above scenario, `App` can import `Engine` (a utility module in `gamePkg`) and access its helper API directly, even though the API is not intended to be used outside of its package. + +Allowing this kind of unintended public access to package APIs is especially bad because packages are a unit of code distribution. Swift wants to encourage programs to be divided into modules with well-defined interfaces, so it enforces the boundaries between modules with access control. Despite being divided this way, it's not uncommon for closely-related modules to be written by closely-related (or even the same) people. Access control between such modules still serves a purpose — it promotes the separation of concerns — but if a module's interface needs to be fixed, that's usually easy to coordinate, maybe even as simple as a single commit. However, packages allow code to be shared much more broadly than a single small organization. The boundaries between packages often represent significant differences between programmers, making coordination around API changes much more difficult. For example, the developers of an open source package generally don't know most of their clients, and the standard recommendation is for such packages to only ever remove existing APIs in major-version releases. It's therefore particularly important to allow programmers to enforce these boundaries between packages. + +## Proposed solution + +Our goal is to introduce a mechanism to Swift to recognize a package as a unit in the aspect of access control. We propose to do so by introducing a new access modifier called `package`. The `package` access modifier allows symbols to be accessed from outside of their defining module, but only from other modules in the same package. This helps to set clear boundaries between packages. + +## Detailed design + +### `package` Keyword + +`package` is introduced as an access modifier. It cannot be combined with other access modifiers. +`package` is a contextual keyword, so existing declarations named `package` will continue to work. This follows the precedent of `open`, which was also added as a contextual keyword. For example, the following is allowed: + +```swift +package var package: String {...} +``` + +### Declaration Site + +The `package` keyword is added at the declaration site. Using the scenario above, the helper API `run` can be declared with the new access modifier like so: + +Module `Engine`: +```swift +public struct MainEngine { + public init() { ... } + public var stats: String { ... } + package func run() { ... } +} +``` + +The `package` access modifier can be used anywhere that the existing access modifiers can be used, e.g. `class`, `struct`, `enum`, `func`, `var`, `protocol`, etc. + +Swift requires that the declarations used in certain places (such as the signature of a function) be at least as accessible as the containing declaration. For the purposes of this rule, `package` is less accessible than `open` and `public` and more accessible than `internal`, `fileprivate`, and `private`. For example, a `public` function cannot use a `package` type in its parameters or return type, and a `package` function cannot use an `internal` type in its parameters or return type. Similarly, an `@inlinable` `public` function cannot use a `package` declaration in its implementation, and an `@inlinable` `package` function cannot use an `internal` declaration in its implementation. + +### Use Site + +The `Game` module can access the helper API `run` since it is in the same package as `Engine`. + +Module `Game`: +```swift +import Engine + +public func play() { + MainEngine().run() // Can access `run` as it is a package symbol in the same package +} +``` + +However, if a client outside of the package tries to access the helper API, it will not be allowed. + +Client `App`: +```swift +import Game +import Engine + +let engine = MainEngine() +engine.run() // Error: cannot find `run` in scope +``` + +### Package Names + +Swift as a language leaves it up to the build system to define the boundaries of a package. The compiler considers two modules to belong to the same package if they were built with the same package name, which is just a Unicode string. The package name is not exposed in the source language, so its exact contents are not significant as long as it is unique to a "package". + +A new flag `-package-name` is passed down to a commandline invocation, as follows. + +```sh +swiftc -module-name Engine -package-name gamePkg ... +swiftc -module-name Game -package-name gamePkg ... +swiftc -module-name App -package-name appPkg ... +``` + +When building the `Engine` module, the package name `gamePkg` is recorded in the built interface to the module. When building `Game`, its package name `gamePkg` is compared with the package name recorded in `Engine`'s built interface; since they match, `Game` is allowed to access `Engine`'s `package` declarations. When building `App`, its package name `appPkg` is different from `gamePkg`, so it is not allowed to access `package` symbols in either `Engine` or `Game`, which is what we want. + +If `-package-name` is not given, the `package` access modifier is disallowed. Swift code that does not use `package` access will continue to build without needing to pass in `-package-name`. Modules built without a package name are never considered to be in the same package as any other module. + +The build system should make a best effort to ensure that package names are unique. The Swift Package Manager already has a concept of a package identity string for every package. This string is verified to be unique, and it already works as a package name, so SwiftPM will pass it down automatically. Other build systems such as Bazel may need to introduce a new build setting for a package name. Since it needs to be unique, a reverse-DNS name may be used to avoid clashing. + +If a target needs to be excluded from the package boundary, that can be done with a new `packageAccess` setting in the manifest, like so: + +```swift + .target(name: "Game", dependencies: ["Engine"], packageAccess: false) +``` + +The `packageAccess` setting is set to `true` by default, and the target is built with `-package-name PACKAGE_ID` where `PACKAGE_ID` is the manifest's package identifier. If `packageAccess` is set to `false`, `-package-name` is not passed when building the target, thus the target has no access to any package symbols; it essentially acts as if it's a client outside of the package. This would be useful for an example app or a black-box test target in the package. + +### Package Symbols Distribution + +When the Swift frontend builds a `.swiftmodule` file directly from source, the file will include the package name and all of the `package` declarations in the module. When the Swift frontend builds a `.swiftinterface` file from source, the file will include the package name, but it will put `package` declarations in a secondary `.package.swiftinterface` file. When the Swift frontend builds a `.swiftmodule` file from a `.swiftinterface` file that includes a package name, but it does not have the corresponding `.package.swiftinterface` file, it will record this in the `.swiftmodule`, and it will prevent this file from being used to build other modules in the same package. + +### Package Symbols and `@inlinable` + +`package` types can be made `@inlinable`. Just as with `@inlinable public`, not all symbols are usable within the body of `@inlinable package`: they must be `open`, `public`, or `@usableFromInline`. The `@usableFromInline` attribute can be applied to `package` besides `internal` declarations. These attributed symbols are allowed in the bodies of `@inlinable public` or `@inlinable package` declarations (that are defined anywhere in the same package). Just as with `internal` symbols, the `package` declarations with `@usableFromInline` or `@inlinable` are stored in the public `.swiftinterface` for a module. + +Here's an example. + +```swift +func internalFuncA() {} +@usableFromInline func internalFuncB() {} + +package func packageFuncA() {} +@usableFromInline package func packageFuncB() {} + +public func publicFunc() {} + +@inlinable package func pkgUse() { + internalFuncA() // Error + internalFuncB() // OK + packageFuncA() // Error + packageFuncB() // OK + publicFunc() // OK +} + +@inlinable public func publicUse() { + internalFuncA() // Error + internalFuncB() // OK + packageFuncA() // Error + packageFuncB() // OK + publicFunc() // OK +} +``` + +### Subclassing and Overrides + +Access control in Swift usually doesn't distinguish between different kinds of use. If a program has access to a type, for example, that gives the programmer a broad set of privileges: the type name can be used in most places, values of the type can be borrowed, copied, and destroyed, members of the type can be accessed (up to the limits of their own access control), and so on. This is because access control is a tool for enforcing encapsulation and allowing the future evolution of code. Broad privileges are granted because restricting them more precisely usually doesn't serve that goal. + +However, there are two exceptions. The first is that Swift allows `var` and `subscript` to restrict mutating accesses more tightly than read-only accesses; this is done by writing a separate access modifier for the setter, e.g. `private(set)`. The second is that Swift allows classes and class members to restrict subclassing and overriding more tightly than normal references; this is done by writing `public` instead of `open`. Allowing these privileges to be separately restricted serves the goals of promoting encapsulation and evolution. + +Because setter access levels are controlled by writing a separate modifier from the primary access, the syntax naturally extends to allow `package(set)`. However, subclassing and overriding are controlled by choosing a specific keyword (`public` or `open`) as the primary access modifier, so the syntax does not extend to `package` the same way. This proposal has to decide what `package` by itself means for classes and class members. It also has to decide whether to support the options not covered by `package` alone or to leave them as a possible future direction. + +Here is a matrix showing where symbols with each current access level can be used or overridden: + + + + + + + + + + + + + + + + + + + + + + + + + + +
Accessible AnywhereAccessible in Module
Subclassable Anywhereopen(illegal)
Subclassable in Modulepublicinternal
Subclassable Nowherepublic finalinternal final
+ +With `package` as a new access modifier, the matrix is modified like so: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Accessible AnywhereAccessible in PackageAccessible in Module
Subclassable Anywhereopen(illegal)(illegal)
Subclassable in Package?(a)?(b)(illegal)
Subclassable in Modulepublicpackageinternal
Subclassable Nowherepublic finalpackage finalinternal final
+ + +This proposal takes the position that `package` alone should not allow subclassing or overriding outside of the defining module. This is consistent with the behavior of `public` and makes `package` fit into a simple continuum of ever-expanding privileges. It also allows the normal optimization model of `public` classes and methods to still be applied to `package` classes and methods, implicitly making them `final` when they aren't subclassed or overridden, without requiring a new "whole package optimization" build mode. + +However, this choice leaves no way to spell the two combinations marked in the table above with `?`. These are more complicated to design and implement and are discussed in Future Directions. + +## Future Directions + +### Subclassing and Overrides + +The entities marked with `?(a)` and `?(b)` from the matrix above both require accessing and subclassing cross-modules in a package (`open` within a package). The only difference is that (b) hides the symbol from outside of the package and (a) makes it visible outside. Use cases involving (a) should be rare but its underlying flow should be the same as (b) except its symbol visibility. + +Potential solutions include introducing new keywords for specific access combinations (e.g. `packageopen`), allowing `open` to be access-qualified (e.g. `open(package)`), and allowing access modifiers to be qualified with specific purposes (e.g. `package(override)`). + +### Package-Private Modules + +Sometimes entire modules are meant to be private to the package that provides them. Allowing this to be expressed directly would allow these utility modules to be completely hidden outside of the package, avoiding unwanted dependencies on the existence of the module. It would also allow the build system to automatically namespace the module within the package, reducing the need for [explicit module aliases](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0339-module-aliasing-for-disambiguation.md) when utility modules of different packages share a name (such as `Utility`) or when multiple versions of a package need to be built into the same program. + +### Grouping Within A Package + +The basic language design of this proposal can work for any group of related modules, but the application of that design in SPM allows only a single such group per SPM package. Developers with complex SPM packages sometimes find that they have multiple architectural "layers" within a single package and may wish to make `package` apply only within a layer. Logically, it makes some sense to put each layer in its own package. Pragmatically, because different SPM packages must currently live in separate repositories and be independently versioned, splitting a package that way introduces a huge amount of extra complexity to the development process, and it is not something that should be done casually. + +There are several reasonable ways that SPM could evolve to support multiple layers within a single package repository. One would be to allow targets to be grouped within a manifest, such as by adding a `group` parameter to `.target`. An earlier version of this proposal suggested this and even designed the `packageAccess:` exclusion feature around it. However, this would tend to lead to large, complex manifests that mingled the details of all the layers together. A very different approach would be to allow the creation of sub-packages within a repository, each with its own manifest. SPM would treat these sub-packages as logically separate units that happen to share a single repository and version. Because they would be described in independent manifests, they would feel like different packages, and it would make sense for `package` access to be scoped within them. + +### Optimizations + +* A package can be treated as a resilience domain, even with library evolution enabled which makes modules resilient. The Swift frontend will assume that modules defined in the same package will always be rebuilt together and do not require a resilient ABI boundary between them. This removes the need for performance and code size overhead introduced by ABI-resilient code generation, and it also eliminates language requirements such as `@unknown default` for a non-`frozen enum`. + +* By default, `package` symbols are exported in the final libraries/executables. It would be useful to introduce a build setting that allows users to hide package symbols for statically linked libraries; this would help with code size and build time optimizations. + +## Source Compatibility + +The new `package` access modifier is a contextual keyword. Existing symbols that are named `package` should not require renaming, and existing source code should continue to work. + +## Effect on ABI stability + +Boundaries between separately-built modules within a package are still potentially ABI boundaries. The ABI for package symbols is not different from the ABI for public symbols, although it might be considered in the future to add an option to not export package symbols that can be resolved within an image. + +## Alternatives considered + +### `@_spi` + +One workaround for the scenario in the Motivation would be to use the `@_spi(groupName)` attribute, which allows part of the API of a module to be hidden unless it is imported in a special way that explicitly requests access to it. This is an unsatisfying alternative to package-level access control because it is designed around a very different situation. An SPI is a "hole" in the normal public interface, one meant for the use of a specific client. That client is typically outside of the module's normal code-distribution boundary, but the module authors still have a cooperative working relationship. This relationship is reflected in the design of `@_spi` in multiple ways: + +* First, access to the SPI is granted to a specific client by name. This is a clear and unavoidable communication of intent about who is meant to be using the SPI. Other clients can still pose as this client and use the SPI, but that would be a clear breach of trust with predictable consequences. + +* Second, clients must explicitly request the SPI by name. This means that clients must opt in to using the SPI in every file, which works to limit its accidental over-use even by the intended client. It also means that SPI use is obvious in the code, which code reviewers can see and raise questions about, and which SPI authors can easily find with a code search. + +The level of care implied by these properties is appropriate for a carefully-targeted hole in an API that must cross a code-distribution boundary and will therefore require equal amounts of care to ever modify or close. That rarely applies to two modules within the same package, where a package-level interface can ideally be changed with just a quick edit to a few different parts of a repository. The `@_spi` attribute is intentionally designed to not be as lightweight as a package-local change should be. + +* `@_spi` would also not be easy to optimize. By design, clients of an SPI can be anywhere, making it effectively part of the public ABI of a module. To avoid exporting an SPI, the build system would have to know about that specific SPI group and promise the compiler that it was only used in the current built image. Recognizing that all of the modules in a package are being linked into the same image and can be optimized together is comparatively easy for a build system and so is a much more feasible future direction. + +### `@_implementationOnly` + +Another workaround for the scenario in the Motivation is to use the `@_implementationOnly` attribute on the import of a module. This attribute causes the module to be imported only for the use of the current module; clients of the current module don't implicitly transitively import the target module, and the symbols of the target module are restricted from appearing in the `public` API of the current module. This would prevent clients from accidentally using APIs from the target module. However, this is a very incomplete workaround for the lack of package-level access control. For one, it doesn't actually prevent access to the module, which can still be explicitly imported and used. For another, it only works on an entire module at a time, so a module cannot restrict some of its APIs to the package while making others available publicly. Taming transitive import would be a good future direction for Swift, but it does not solve the problems of package-level APIs. + +### Other Workarounds + +There are a few other workarounds to the absence of package-level access control, such as using `@testable` or the `-disable-access-control` flag. These are hacky subversions of Swift's language design, and they severely undermine the use of module boundaries for encapsulation. `-disable-access-control` is also an unstable and unsupported feature that can introduce build failures by causing symbol name collisions. + +### Introduce Submodules + +Instead of adding a new package access level above modules, we could allow modules to contain other modules as components. This is an idea often called "submodules". Packages would then define an "umbrella" module that contains the package's modules as components. However, there are several weaknesses in this approach: + +* It doesn't actually solve the problem by itself. Submodule APIs would still need to be able to declare whether they're usable outside of the umbrella or not, and that would require an access modifier. It might be written in a more general way, like `internal(MyPackage)`, but that generality would also make it more verbose. + +* Submodule structure would be part of the source language, so it would naturally be source- and ABI-affecting. For example, programmers could use the parent module's name to qualify identifiers, and symbols exported by a submodule would include the parent module's name. This means that splitting a module into submodules or adding an umbrella parent module would be much more impactful than desired; ideally, those changes would be purely internal and not change a module's public interface. It also means that these changes would end up permanently encoding package structure. + +* The "umbrella" submodule structure doesn't work for all packages. Some packages include multiple "top-level" modules which share common dependencies. Forcing these to share a common umbrella in order to use package-private dependencies is not desirable. + +* In a few cases, the ABI and source impact above would be desirable. For example, many packages contain internal Utility modules; if these were declared as submodules, they would naturally be namespaced to the containing package, eliminating spurious collisions. However, such modules are generally not meant to be usable outside of the package at all. It is a reasonable future direction to allow whole modules to be made package-private, which would also make it reasonable to automatically namespace them. + +### `@usableFromPackageInline` + +An earlier version of this proposal included a new attribute `@usableFromPackageInline`, which would have allowed an `internal` declaration to be used in the body of an `@inlinable package` declaration, but not in an `@inlinable public` declaration. Under the logic of this proposal, there is no good reason to make a declaration `@usableFromPackageInline internal` instead of simply `package`: the uses of the latter will be restricted to the package and therefore by assumption can still be easily found and reviewed. Furthermore, it is a goal of the Swift project to not require extensive `@inlinable` annotations just to enable basic optimizations between modules: there should be little reason in the long run to have an `@inlinable package` declaration at all. Therefore this attribute has been removed from the proposal. + +## Acknowledgments + +Doug Gregor, Becca Royal-Gordon, Allan Shortlidge, Artem Chikin, and Xi Ge provided helpful feedback and analysis as well as code reviews on the implementation. diff --git a/proposals/0387-cross-compilation-destinations.md b/proposals/0387-cross-compilation-destinations.md new file mode 100644 index 0000000000..40367a6464 --- /dev/null +++ b/proposals/0387-cross-compilation-destinations.md @@ -0,0 +1,605 @@ +# Swift SDKs for Cross-Compilation + +* Proposal: [SE-0387](0387-cross-compilation-destinations.md) +* Authors: [Max Desiatov](https://github.com/MaxDesiatov), [Saleem Abdulrasool](https://github.com/compnerd), [Evan Wilde](https://github.com/etcwilde) +* Review Manager: [Mishal Shah](https://github.com/shahmishal) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift-package-manager#5911](https://github.com/apple/swift-package-manager/pull/5911), + [apple/swift-package-manager#5922](https://github.com/apple/swift-package-manager/pull/5922), + [apple/swift-package-manager#6023](https://github.com/apple/swift-package-manager/pull/6023), + [apple/swift-package-manager#6186](https://github.com/apple/swift-package-manager/pull/6186) +* Review: ([pitch](https://forums.swift.org/t/pitch-cross-compilation-destination-bundles/61777)) + ([first review](https://forums.swift.org/t/se-0387-cross-compilation-destination-bundles/62875)) + ([second review](https://forums.swift.org/t/second-review-se-0387-cross-compilation-destination-bundles/64660)) + +## Table of Contents + +- [Introduction](#introduction) +- [Motivation](#motivation) +- [Proposed Solution](#proposed-solution) +- [Detailed Design](#detailed-design) + - [Swift SDK Bundles](#swift-sdk-bundles) + - [`toolset.json` Files](#toolsetjson-files) + - [`swift-sdk.json` Files](#swift-sdkjson-files) + - [Swift SDK Installation and Configuration](#swift-sdk-installation-and-configuration) + - [Using a Swift SDK](#using-a-swift-sdk) + - [Swift SDK Bundle Generation](#swift-sdk-bundle-generation) +- [Security](#security) +- [Impact on Existing Packages](#impact-on-existing-packages) +- [Prior Art](#prior-art) + - [Rust](#rust) + - [Go](#go) +- [Alternatives Considered](#alternatives-considered) + - [Extensions Other Than `.artifactbundle`](#extensions-other-than-artifactbundle) + - [Building Applications in Docker Containers](#building-applications-in-docker-containers) + - [Alternative Bundle Formats](#alternative-bundle-formats) +- [Making Swift SDK Bundles Fully Self-Contained](#making-swift-sdk-bundles-fully-self-contained) +- [Future Directions](#future-directions) + - [Identifying Platforms with Dictionaries of Properties](#identifying-platforms-with-dictionaries-of-properties) + - [SwiftPM Plugins for Remote Running, Testing, Deployment, and Debugging](#swiftpm-plugins-for-remote-running-testing-deployment-and-debugging) + - [`swift sdk select` Subcommand](#swift-sdk-select-subcommand) + - [SwiftPM and SourceKit-LSP Improvements](#swiftpm-and-sourcekit-lsp-improvements) + - [Source-Based Swift SDKs](#source-based-swift-sdks) + - [Swift SDK Bundles and Package Registries](#swift-sdk-bundles-and-package-registries) + +## Introduction + +Cross-compilation is a common development use case. When cross-compiling, we need to refer to these concepts: + +- a **toolchain** is a set of tools used to build an application or a library; +- a **triple** describes features of a given machine such as CPU architecture, vendor, OS etc, corresponding to LLVM's + triple; +- a **host triple** describes a machine where application or library code is built; +- a **target triple** describes a machine where application or library code is running; +- an **SDK** is a set of dynamic and/or static libraries, headers, and other resources required to generate code for the + target triple. + +When a triple of a machine on which the toolchain is built is different from the host triple, we'll call it a **build triple**. +The cross-compilation configuration itself that involves three different triples is called +[the Canadian Cross](https://en.wikipedia.org/wiki/Cross_compiler#Canadian_Cross). + +Let’s call a Swift toolchain and an SDK bundled together in an artifact bundle a **Swift SDK**. + +## Motivation + +In Swift 5.8 and earlier versions users can cross-compile their code with so called "destination files" passed to +SwiftPM invocations. These destination files are produced on an ad-hoc basis for different combinations of +host and target triples. For example, scripts that produce macOS → Linux destinations were created by both +[the Swift +team](https://github.com/apple/swift-package-manager/blob/swift-5.8-RELEASE/Utilities/build_ubuntu_cross_compilation_toolchain) +and [the Swift community](https://github.com/SPMDestinations/homebrew-tap). At the same time, the distribution process +of assets required for cross-compiling is cumbersome. After building a destination tree on the file system, required +metadata files rely on hardcoded absolute paths. Adding support for relative paths in destination's metadata and +providing a unified way to distribute and install required assets as archives would clearly be an improvement for the +multi-platform Swift ecosystem. + +The primary audience of this pitch are people who cross-compile from macOS to Linux. When deploying to single-board +computers supporting Linux (e.g. Raspberry Pi), building on such hardware may be too slow or run out of available +memory. Quite naturally, users would prefer to cross-compile on a different machine when developing for these platforms. + +In other cases, building in a Docker container is not always the best solution for certain development workflows. For +example, when working with Swift AWS Lambda Runtime, some developers may find that installing Docker just for building a +project is a daunting step that shouldn’t be required. + +The solution described below is general enough to scale for any host/target triple combination. + +## Proposed Solution + +Since a Swift SDK is a collection of binaries arranged in a certain directory hierarchy, it makes sense to distribute +it as an archive. We'd like to build on top of +[SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md) and +extend the `.artifactbundle` format to support this. + +Additionally, we propose introducing a new `swift sdk` CLI command for installation and removal of Swift SDKs on the +local filesystem. + +We introduce a notion of a top-level toolchain, which is the toolchain that handles user’s `swift sdk` +invocations. Parts of this top-level toolchain (linker, C/C++ compilers, and even the Swift compiler) can be overridden +with tools supplied in `.artifactbundle` s installed by `swift sdk` invocations. + +When the user runs `swift build` with the selected Swift SDK, the overriding tools from the corresponding bundle +are invoked by `swift build` instead of tools from the top-level toolchain. + +The proposal is intentionally limited in scope to build-time experience and specifies only configuration metadata, basic +directory layout for proposed artifact bundles, and some CLI helpers to operate on those. + +## Detailed Design + +### Swift SDK Bundles + +As a quick reminder for a concept introduced in +[SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md), an +**artifact bundle** is a directory that has the filename suffix `.artifactbundle` and has a predefined structure with +`.json` manifest files provided as metadata. + +The proposed structure of artifact bundles containing Swift SDKs looks like: + +``` +.artifactbundle +├ info.json +├ +│ ├ swift-sdk.json +│ ├ toolset.json +│ └ +├ +│ ├ swift-sdk.json +│ ├ toolset.json +│ └ +├ +┆ └┄ +``` + +For example, a Swift SDK bundle allowing to cross-compile Swift 5.9 source code to recent versions of Ubuntu from +macOS would look like this: + +``` +swift-5.9_ubuntu.artifactbundle +├ info.json +├ toolset.json +├ ubuntu_jammy +│ ├ swift-sdk.json +│ ├ toolset.json +│ ├ +│ ├ aarch64-unknown-linux-gnu +│ │ ├ toolset.json +│ │ └ +│ └ x86_64-unknown-linux-gnu +│ ├ toolset.json +│ └ +├ ubuntu_focal +│ ├ swift-sdk.json +│ └ x86_64-unknown-linux-gnu +│ ├ toolset.json +│ └ +├ ubuntu_bionic +┆ └┄ +``` + +Here each artifact directory is dedicated to a specific Swift SDK, while files specific to each triple are placed +in `aarch64-unknown-linux-gnu` and `x86_64-unknown-linux-gnu` subdirectories. + +`info.json` bundle manifests at the root of artifact bundles should specify `"type": "swiftSDK"` for +corresponding artifacts. Artifact identifiers in this manifest file uniquely identify a Swift SDK, and +`supportedTriples` property in `info.json` should contain host triples that a given Swift SDK supports. The rest +of the properties of bundle manifests introduced in SE-0305 are preserved. + +Here's how `info.json` file could look like for `swift-5.9_ubuntu.artifactbundle` introduced in the example +above: + +```json5 +{ + "artifacts" : { + "swift-5.9_ubuntu22.04" : { + "type" : "swiftSDK", + "version" : "0.0.1", + "variants" : [ + { + "path" : "ubuntu_jammy", + "supportedTriples" : [ + "arm64-apple-darwin", + "x86_64-apple-darwin" + ] + } + ] + }, + "swift-5.9_ubuntu20.04" : { + "type" : "swiftSDK", + "version" : "0.0.1", + "variants" : [ + { + "path" : "ubuntu_focal", + "supportedTriples" : [ + "arm64-apple-darwin", + "x86_64-apple-darwin" + ] + } + ] + } + }, + "schemaVersion" : "1.0" +} +``` + +### `toolset.json` Files + +We find that properties dedicated to tools configuration are useful outside of the cross-compilation context. Due to +that, separate toolset configuration files are introduced: + +```json5 +{ + "schemaVersion": "1.0", + "rootPath": "optional path to a root directory containing toolchain executables", + // If `rootPath` is specified, all relative paths below will be resolved relative to `rootPath`. + "swiftCompiler": { + "path": "", + "extraCLIOptions": [""] + }, + "cCompiler": { + "path": "", + "extraCLIOptions": [""] + }, + "cxxCompiler": { + "path": "", + "extraCLIOptions": [""] + }, + "linker": { + "path": "", + "extraCLIOptions": [""] + }, + "librarian": { + "path": "", + "extraCLIOptions": [""] + }, + "debugger": { + "path": "", + "extraCLIOptions": [""] + }, + "testRunner": { + "path": "", + "extraCLIOptions": [""] + }, +} +``` + +More types of tools may be enabled in toolset files in the future in addition to those listed above. + +Users familiar with CMake can draw an analogy between toolset files and CMake toolchain files. Toolset files are +designed to supplant previous ad-hoc ways of specifying paths and flags in SwiftPM, such as `SWIFT_EXEC` and `CC` +environment variables, which were applied in use cases unrelated to cross-compilation. We propose that +users also should be able to pass `--toolset ` option to `swift build`, `swift test`, and +`swift run`. + +We'd like to allow using multiple toolset files at once. With this users can "assemble" toolchains on the fly out of +tools that in certain scenarios may even come from different vendors. A toolset file can have an arbitrary name, and +each file should be passed with a separate `--toolset` option, i.e. `swift build --toolset t1.json --toolset t2.json`. + +All of the properties related to names of the tools are optional, which allows merging configuration from multiple +toolset files. For example, consider `toolset1.json`: + +```json5 +{ + "schemaVersion": "1.0", + "swiftCompiler": { + "path": "/usr/bin/swiftc", + "extraCLIOptions": ["-Xfrontend", "-enable-cxx-interop"] + }, + "cCompiler": { + "path": "/usr/bin/clang", + "extraCLIOptions": ["-pedantic"] + } +} +``` + +and `toolset2.json`: + +```json5 +{ + "schemaVersion": "1.0", + "swiftCompiler": { + "path": "/custom/swiftc" + } +} +``` + +With multiple `--toolset` options, passing both of those files will merge them into a single configuration. Tools passed +in subsequent `--toolset` options will shadow tools from previous options with the same names. That is, +`swift build --toolset toolset1.json --toolset toolset2.json` will build with `/custom/swiftc` and no extra flags, as +specified in `toolset2.json`, but `/usr/bin/clang -pedantic` from `toolset1.json` will still be used. + +Tools not specified in any of the supplied toolset files will be looked up in existing implied search paths that are +used without toolsets, even when `rootPath` is present. We'd like toolsets to be explicit in this regard: if a +tool would like to participate in toolset path lookups, it must provide either a relative or an absolute path in a +toolset. + +Tools that don't have `path` property but have `extraCLIOptions` present will append options from that property to a +tool with the same name specified in a preceding toolset file. If no other toolset files were provided, these options +will be appended to the default tool invocation. For example `pedanticCCompiler.json` that looks like this + +```json5 +{ + "schemaVersion": "1.0", + "cCompiler": { + "extraCLIOptions": ["-pedantic"] + } +} +``` + +in `swift build --toolset pedanticCCompiler.json` will pass `-pedantic` to the C compiler located at a default path. + +When cross-compiling, paths in `toolset.json` files supplied in Swift SDK bundles should be self-contained: +no absolute paths and no escaping symlinks are allowed. Users are still able to provide their own `toolset.json` files +outside of artifact bundles to specify additional developer tools for which no relative "non-escaping" path can be +provided within the bundle. + +### `swift-sdk.json` Files + +Note the presence of `swift-sdk.json` files in each `` subdirectory. These files should contain +a JSON dictionary with an evolved version of the schema of [existing `destination.json` files that SwiftPM already +supports](https://github.com/apple/swift-package-manager/pull/1098) and `destination.json` files presented in the pitch +version of this proposal, hence `"schemaVersion": "4.0"`. We'll keep parsing `"version": 1`, `"version": 2`, +and `"version": "3.0"` for backward compatibility, but for consistency with `info.json` this field is renamed to +`"schemaVersion"`. Here's an informally defined schema for these files: + +```json5 +{ + "schemaVersion": "4.0", + "targetTriples": { + "": { + "sdkRootPath": "", + // all of the properties listed below are optional: + "swiftResourcesPath": "", + "swiftStaticResourcesPath": "", + "includeSearchPaths": [""], + "librarySearchPaths": [""], + "toolsetPaths": [""] + }, + // a Swift SDK can support more than one target triple: + "": { + "sdkRootPath": "", + // all of the properties listed below are optional: + "swiftResourcesPath": "", + "swiftStaticResourcesPath": "", + "includeSearchPaths": [""], + "librarySearchPaths": [""], + "toolsetPaths": [""] + } + // more triples can be supported by a single Swift SDK if needed, primarily for sharing files between them. + } +} +``` + +We propose that all relative paths in `swift-sdk.json` files should be validated not to "escape" the Swift SDK +bundle for security reasons, in the same way that `toolset.json` files are validated when contained in Swift SDK +bundles. That is, `../` components, if present in paths, will not be allowed to reference files and +directories outside of a corresponding Swift SDK bundle. Symlinks will also be validated to prevent them from escaping +out of the bundle. + +If `sdkRootPath` is specified and `swiftResourcesPath` is not, the latter is inferred to be +`"\(sdkRootPath)/usr/lib/swift"` when linking the Swift standard library dynamically, `"swiftStaticResourcesPath"` is +inferred to be `"\(sdkRootPath)/usr/lib/swift_static"` when linking it statically. Similarly, `includeSearchPaths` is +inferred as `["\(sdkRootPath)/usr/include"]`, `librarySearchPaths` as `["\(sdkRootPath)/usr/lib"]`. + +Here's `swift-sdk.json` file for the `ubuntu_jammy` artifact previously introduced as an example: + +```json5 +{ + "schemaVersion": "4.0", + "targetTriples": { + "aarch64-unknown-linux-gnu": { + "sdkRootPath": "aarch64-unknown-linux-gnu/ubuntu-jammy.sdk", + "toolsetPaths": ["aarch64-unknown-linux-gnu/toolset.json"] + }, + "x86_64-unknown-linux-gnu": { + "sdkRootPath": "x86_64-unknown-linux-gnu/ubuntu-jammy.sdk", + "toolsetPaths": ["x86_64-unknown-linux-gnu/toolset.json"] + } + } +} +``` + +Since not all platforms can support self-contained Swift SDK bundles, users will be able to provide their own +additional paths on the filesystem outside of bundles after a Swift SDK is installed. The exact options for specifying +paths are proposed in a subsequent section for a newly introduced `swift sdk configure` command. + +### Swift SDK Installation and Configuration + +To manage Swift SDKs, we'd like to introduce a new `swift sdk` command with three subcommands: + +- `swift sdk install `, which downloads a given bundle if needed and + installs it in a location discoverable by SwiftPM. For Swift SDKs installed from remote URLs an additional + `--checksum` option is required, through which users of a Swift SDK can specify a checksum provided by a publisher of + the SDK. The latter can produce a checksum by running `swift package compute-checksum` command (introduced in + [SE-0272](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md)) with the + Swift SDK bundle archive as an argument. + + If a Swift SDK with a given artifact ID has already been installed and its version is equal or higher to a version + of a new Swift SDK, an error message will be printed. If the new version is higher, users should invoke the + `install` subcommand with `--update` flag to allow updating an already installed Swift SDK artifact to a new + version. +- `swift sdk list`, which prints a list of already installed Swift SDKs with their identifiers. +- `swift sdk configure `, which allows users to provide additional search paths and toolsets to be +used subsequently when building with a given Swift SDK. Specifically, multiple `--swift-resources-path`, +`--include-search-path`, `--library-search-path`, and `--toolset` options with corresponding paths can be provided, +which then will be stored as configuration for this Swift SDK. +`swift sdk configure --show-configuration` will print currently set paths, while +`swift sdk configure --reset` will reset all of those at once. +- `swift sdk remove ` will remove a given Swift SDK from the filesystem. + +### Using a Swift SDK + +After a Swift SDK is installed, users can refer to it via its identifier passed to the `--swift-sdk` option, e.g. + +``` +swift build --swift-sdk ubuntu_focal +``` + +We'd also like to make `--swift-sdk` option flexible enough to recognize target triples when there's only a single +Swift SDK installed for such triple: + +``` +swift build --swift-sdk x86_64-unknown-linux-gnu +``` + +When multiple Swift SDKs support the same triple, an error message will be printed listing these Swift SDKs and +asking the user to select a single one via its identifier instead. + +### Swift SDK Bundle Generation + +Swift SDKs can be generated quite differently, depending on host and target triple combinations and user's +needs. We intentionally don't specify in this proposal how exactly Swift SDK bundles should be generated. + +Authors of this document intend to publish source code for a macOS → Linux Swift SDK generator, which community is +welcome to fork and reuse for their specific needs. As a configurable option, this generator will use Docker for setting +up the build environment locally before copying it to a Swift SDK tree. Relying on Docker in this generator makes it +easier to reuse and customize existing build environments. Important to clarify, that Docker is only used for bundle +generation, and users of Swift SDK bundles do not need to have Docker installed on their machine to use these bundles. + +As an example, Swift SDK publishers looking to add a library to an Ubuntu 22.04 target environment would modify a +`Dockerfile` similar to this one in their Swift SDK generator source code: + +```dockerfile +FROM swift:5.9-jammy + +apt-get install -y \ + # PostgreSQL library provided as an example. + libpq-dev + # Add more libraries as arguments to `apt-get install`. +``` + +Then to generate a new Swift SDK, a generator executable delegates to Docker for downloading and installing +required tools and libraries, including the newly added ones. After a Docker image with Swift SDK environment is +ready, the generator copies files from the image to a corresponding `.artifactbundle` Swift SDK tree. + +## Security + +The proposed `--checksum` flag provides basic means of verifying Swift SDK bundle's validity. As a future direction, +we'd like to consider sandboxed and codesigned toolchains included in Swift SDKs running on macOS. + +## Impact on Existing Packages + +This is an additive change with no impact on existing packages. + +## Prior Art + +### Rust + +In the Rust ecosystem, its toolchain and standard library built for a target triple are managed by [the `rustup` +tool](https://github.com/rust-lang/rustup). For example, artifacts required for cross-compilation to +`aarch64-linux-unknown-gnu` are installed with +[`rustup target add aarch64-linux-unknown-gnu`](https://rust-lang.github.io/rustup/cross-compilation.html). Then +building for this target with Rust’s package manager looks like `cargo build --target=aarch64-linux-unknown-gnu` . + +Mainstream Rust tools don’t provide an easy way to create your own targets. You’re only limited to the list +of targets provided by Rust maintainers. This likely isn’t a big problem per se for Rust users, as Rust doesn’t provide +C/C++ interop on the same level as Swift. It means that Rust packages much more rarely than Swift expect certain +system-provided packages to be available in the same way that SwiftPM allows with `systemLibrary`. + +Currently, Rust doesn’t supply all of the required tools when running `rustup target add`. It’s left to a user to +specify paths to a linker that’s suitable for their host/target triple combination manually in a config file. We +feel that this should be unnecessary, which is why Swift SDK bundles proposed for Swift can provide their own tools +via toolset configuration files. + +### Go + +Go’s standard library is famously self-contained and has no dependencies on C or C++ standard libraries. Because of this +there’s no need to install artifacts. Cross-compiling in Go works out of the box by passing `GOARCH` and `GOOS` +environment variables with chosen values, an example of this is `GOARCH=arm64 GOOS=linux go build` invocation. + +This would be a great experience for Swift, but it isn’t easily achievable as long as Swift standard library depends on +C and C++ standard libraries. Any code interoperating with C and/or C++ would have to link with those libraries as well. +When compared to Go, our proposed solution allows both dynamic and, at least on Linux when Musl is supported, full +static linking. We’d like Swift to allow as much customization as needed for users to prepare their own Swift SDK +bundles. + +## Alternatives Considered + +### Extensions Other Than `.artifactbundle` + +Some members of the community suggested that Swift SDK bundles should use a more specific filepath extension. Since +we're relying on the existing `.artifactbundle` format and extension, which is already used for binary targets, we think +a specialized extension only for Swift SDKs would introduce an inconsistency. On the other hand, we think that +specific extensions could make sense with a change applied at once. For example, we could consider `.binarytarget` and +`.swiftsdk` extensions for respective artifact types. But that would require a migration strategy for existing +`.artifactbundle`s containing binary targets. + +### Building Applications in Docker Containers + +Instead of coming up with a specialized bundle format for Swift SDKs, users of Swift on macOS building for Linux could +continue to use Docker. But, as discussed in the [Motivation](#motivation) section, building applications in Docker +doesn’t cover all of the possible use cases and complicates onboarding for new users. It also only supports Linux, while +we’re looking for a solution that can be generalized for all possible platforms. + +### Alternative Bundle Formats + +One alternative is to allow only a single host/target combination per bundle, but this may complicate +distribution of Swift SDK bundles in some scenarios. The existing `.artifactbundle` format is flexible enough to +support bundles with a single or multiple combinations. + +Different formats of Swift SDK bundles can be considered, but we don't think those would be significantly different +from the proposed one. If they were different, this would complicate bundle distribution scenarios for users who want to +publish their own artifact bundles with executables, as defined in SE-0305. + +### Triples nomenclature + +Authors of the proposal considered alternative nomenclature to the established "build/host/target platform" naming convention, +but felt that preserving consistency with other ecosystems is more important. + +While "target" already has a different meaning within the build systems nomenclature, users are most likely to stumble upon +targets when working with SwiftPM package manifests. To avoid this ambiguity, as a future direction SwiftPM can consider renaming +`target` declarations used in `Package.swift` to a different unambiguous term. + +### Making Swift SDK Bundles Fully Self-Contained + +Some users expressed interest in self-contained Swift SDK bundles that ignore the value of `PATH` environment variable +and prevent launching any executables from outside of a bundle. So far in our practice we haven't seen any problems +caused by the use of executables from `PATH`. Quite the opposite, we think most Swift SDKs would want to reuse as many +tools from `PATH` as possible, which would allow making Swift SDK bundles much smaller. For example as of Swift 5.7, +on macOS `clang-13` binary takes ~360 MB, `clangd` ~150 MB, and `swift-frontend` ~420 MB. Keeping copies of these +binaries in every Swift SDK bundle seems quite redundant when existing binaries from `PATH` can be easily reused. +Additionally, we find that preventing tools from being launched from arbitrary paths can't be technically enforced +without sandboxing, and there's no cross-platform sandboxing solution available for SwiftPM. Until such sandboxing +solution is available, we'd like to keep the existing approach where setting `PATH` environment variable behaves in a +predictable way and is consistent with established CLI conventions. + +## Future Directions + +### Identifying Platforms with Dictionaries of Properties + +Platform triples are not specific enough in certain cases. For example, `aarch64-unknown-linux` host triple can’t +prevent a user from installing a Swift SDK bundle on an unsupported Linux distribution. In the future we could +deprecate `supportedTriples` and `targetTriples` JSON properties in favor of dictionaries with keys and values that +describe aspects of platforms that are important for Swift SDKs. Such dictionaries could look like this: + +```json5 +"platform": { + "kernel": "Linux", + "libcFlavor": "Glibc", + "libcMinVersion": "2.36", + "cpuArchitecture": "aarch64" + // more platform capabilities defined here... +} +``` + +A toolchain providing this information could allow users to refer to these properties in their code for conditional +compilation and potentially even runtime checks. + +### SwiftPM Plugins for Remote Running, Testing, Deployment, and Debugging + +After an application is built with a Swift SDK, there are other development workflow steps to be improved. We could +introduce new types of plugins invoked by `swift run` and `swift test` for purposes of remote running, debugging, and +testing. For Linux target triples, these plugins could delegate to Docker for running produced executables. + +### `swift sdk select` Subcommand + +While `swift sdk select` subcommand or a similar one make sense for selecting a Swift SDK instead of +passing `--swift-sdk` to `swift build` every time, users will expect `swift run` and `swift test` to also work for any +Swift SDK previously passed to `swift sdk select`. That’s out of scope for this proposal on its own and +depends on making plugins (from the previous subsection) or some other remote running and testing implementation to +fully work. + +### SwiftPM and SourceKit-LSP Improvements + +It is a known issue that SwiftPM can’t run multiple concurrent builds for different target triples. This may cause +issues when SourceKit-LSP is building a project for indexing purposes (for a host platform by default), while a user may +be trying to build for a target for testing, for example. One of these build processes will fail due to the +process locking the build database. A potential solution would be to maintain separate build databases per platform. + +Another issue related to SourceKit-LSP is that [it always build and indexes source code for the host +platform](https://github.com/apple/sourcekit-lsp/issues/601). Ideally, we want it to maintain indices for multiple +platforms at the same time. Users should be able to select target triples and corresponding indices to enable +semantic syntax highlighting, auto-complete, and other features for areas of code that are conditionally compiled with +`#if` directives. + +### Source-Based Swift SDKs + +One interesting solution is distributing source code of a minimal base SDK, as explored by [Zig programming +language](https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html). In this scenario, Swift SDK +binaries are produced on the fly when needed. We don't consider this option to be mutually exclusive with solutions +proposed in this document, and so it could be explored in the future for Swift as well. However, this requires reducing +the number of dependencies that Swift runtime and core libraries have. + +### Swift SDK Bundles and Package Registries + +Since `info.json` manifest files contained within bundles contain versions, it would make sense to host Swift SDK +bundles at package registries. Although, it remains to be seen whether it makes sense for an arbitrary SwiftPM package +to specify a Swift SDK bundle within its list of dependencies. diff --git a/proposals/0388-async-stream-factory.md b/proposals/0388-async-stream-factory.md new file mode 100644 index 0000000000..cda799d450 --- /dev/null +++ b/proposals/0388-async-stream-factory.md @@ -0,0 +1,165 @@ +# Convenience Async[Throwing]Stream.makeStream methods + +* Proposal: [SE-0388](0388-async-stream-factory.md) +* Authors: [Franz Busch](https://github.com/FranzBusch) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#62968](https://github.com/apple/swift/pull/62968) +* Review: ([pitch](https://forums.swift.org/t/pitch-convenience-async-throwing-stream-makestream-methods/61030)) ([review](https://forums.swift.org/t/se-0388-convenience-async-throwing-stream-makestream-methods/63139)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0388-convenience-async-throwing-stream-makestream-methods/63568)) + +## Introduction + +We propose introducing helper methods for creating `AsyncStream` and `AsyncThrowingStream` +instances which make the stream's continuation easier to access. + +## Motivation + +With [SE-0314](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0314-async-stream.md) +we introduced `AsyncStream` and `AsyncThrowingStream` which act as a root +`AsyncSequence` that the standard library offers. + +After having used `Async[Throwing]Stream` for some time, a common usage +is to pass the continuation and the `Async[Throwing]Stream` to different places. +This requires escaping the `Async[Throwing]Stream.Continuation` out of +the closure that is passed to the initialiser. +Escaping the continuation is slightly inconvenient since it requires a dance +around an implicitly unwrapped optional. Furthermore, the closure implies +that the continuation lifetime is scoped to the closure which it isn't. This is how +an example usage of the current `AsyncStream` API looks like. + +```swift +var cont: AsyncStream.Continuation! +let stream = AsyncStream { cont = $0 } +// We have to assign the continuation to a let to avoid sendability warnings +let continuation = cont + +await withTaskGroup(of: Void.self) { group in + group.addTask { + for i in 0...9 { + continuation.yield(i) + } + continuation.finish() + } + + group.addTask { + for await i in stream { + print(i) + } + } +} +``` + +## Proposed solution + +In order to fill this gap, I propose to add a new static method `makeStream` on +`AsyncStream` and `AsyncThrowingStream` that returns both the stream +and the continuation. An example of using the new proposed convenience methods looks like this: + +```swift +let (stream, continuation) = AsyncStream.makeStream(of: Int.self) + +await withTaskGroup(of: Void.self) { group in + group.addTask { + for i in 0...9 { + continuation.yield(i) + } + continuation.finish() + } + + group.addTask { + for await i in stream { + print(i) + } + } +} +``` + +## Detailed design + +I propose to add the following code to `AsyncStream` and `AsyncThrowingStream` +respectively. These methods are also marked as backdeployed to previous Swift versions. + +```swift +@available(SwiftStdlib 5.1, *) +extension AsyncStream { + /// Initializes a new ``AsyncStream`` and an ``AsyncStream/Continuation``. + /// + /// - Parameters: + /// - elementType: The element type of the stream. + /// - limit: The buffering policy that the stream should use. + /// - Returns: A tuple containing the stream and its continuation. The continuation should be passed to the + /// producer while the stream should be passed to the consumer. + @backDeployed(before: SwiftStdlib 5.9) + public static func makeStream( + of elementType: Element.Type = Element.self, + bufferingPolicy limit: Continuation.BufferingPolicy = .unbounded + ) -> (stream: AsyncStream, continuation: AsyncStream.Continuation) { + var continuation: AsyncStream.Continuation! + let stream = AsyncStream(bufferingPolicy: limit) { continuation = $0 } + return (stream: stream, continuation: continuation!) + } +} + +@available(SwiftStdlib 5.1, *) +extension AsyncThrowingStream { + /// Initializes a new ``AsyncThrowingStream`` and an ``AsyncThrowingStream/Continuation``. + /// + /// - Parameters: + /// - elementType: The element type of the stream. + /// - failureType: The failure type of the stream. + /// - limit: The buffering policy that the stream should use. + /// - Returns: A tuple containing the stream and its continuation. The continuation should be passed to the + /// producer while the stream should be passed to the consumer. + @backDeployed(before: SwiftStdlib 5.9) + public static func makeStream( + of elementType: Element.Type = Element.self, + throwing failureType: Failure.Type = Failure.self, + bufferingPolicy limit: Continuation.BufferingPolicy = .unbounded + ) -> (stream: AsyncThrowingStream, continuation: AsyncThrowingStream.Continuation) where Failure == Error { + var continuation: AsyncThrowingStream.Continuation! + let stream = AsyncThrowingStream(bufferingPolicy: limit) { continuation = $0 } + return (stream: stream, continuation: continuation!) + } +} +``` + +## Source compatibility +This change is additive and does not affect source compatibility. + +## Effect on ABI stability +This change introduces new concurrency library ABI in the form of the `makeStream` methods, but it does not affect the ABI of existing declarations. + +## Effect on API resilience +None; adding static methods is permitted by the existing resilience model. + +## Alternatives considered + +### Return a concrete type instead of a tuple +My initial pitch was using a tuple as the result type of the factory; +however, I walked back on it before the review since I think we can provide better documentation on +the concrete type. However during the review the majority of the feedback was leaning towards the tuple based approach. +After comparing the two approaches, I agree with the review feedback. The tuple based approach has two major benefits: + +1. It nudges the user to destructure the returned typed which we want since the continuation and stream should be retained by the +producer and consumer respectively. +2. It allows us to back deploy the method. + +### Pass a continuation to the `AsyncStream.init()` +During the pitch it was brought up that we could let users pass a continuation to the +`AsyncStream.init()`; however, this opens up a few problems: +1. A continuation could be passed to multiple streams +2. A continuation which is not passed to a stream is useless + +In the end, the `AsyncStream.Continuation` is deeply coupled to one instance of an +`AsyncStream` hence we should create an API that conveys this coupling and prevents +users from misuse. + +### Do nothing alternative +We could just leave the current creation of `Async[Throwing]Stream` as is; +however, since it is part of the standard library we should provide +a better method to create a stream and its continuation. + +## Revision history + +- After review: Changed the return type from a concrete type to a tuple + diff --git a/proposals/0389-attached-macros.md b/proposals/0389-attached-macros.md new file mode 100644 index 0000000000..fe01f3495f --- /dev/null +++ b/proposals/0389-attached-macros.md @@ -0,0 +1,795 @@ +# Attached Macros + +* Proposal: [SE-0389](0389-attached-macros.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Holly Borla](https://github.com/hborla), [Richard Wei](https://github.com/rxwei) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 5.9)** +* Implementation: Implemented on GitHub `main` behind the experimental flag `Macros`. See the [example repository](https://github.com/DougGregor/swift-macro-examples) for more macros. +* Review: ([pitch #1, under the name "declaration macros"](https://forums.swift.org/t/pitch-declaration-macros/62373)) ([pitch #2](https://forums.swift.org/t/pitch-attached-macros/62812)) ([review](https://forums.swift.org/t/se-0389-attached-macros/63165)) ([acceptance](https://forums.swift.org/t/accepted-se-0389-attached-macros/63593)) ([post-acceptance update](https://forums.swift.org/t/update-on-se-0382-and-se-0389-expression-macros-and-attached-macros/74094)) + +## Introduction + +Attached macros provide a way to extend Swift by creating and extending declarations based on arbitrary syntactic transformations on their arguments. They make it possible to extend Swift in ways that were only previously possible by introducing new language features, helping developers build more expressive libraries and eliminate extraneous boilerplate. + +Attached macros are one part of the [vision for macros in Swift](https://github.com/swiftlang/swift-evolution/pull/1927), which lays out general motivation for introducing macros into the language. They build on the ideas and motivation of [SE-0382 "Expression macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) to cover a large new set of use cases; we will refer to that proposal for the basic model of how macros integrate into the language. While expression macros are designed as standalone entities introduced by `#`, attached macros are associated with a specific declaration in the program that they can augment and extend. This supports many new use cases, greatly expanding the expressiveness of the macro system: + +* Creating trampoline or wrapper functions, such as automatically creating a completion-handler version of an `async` function or vice-versa. +* Creating members of a type based on its definition, such as forming an [`OptionSet`](https://developer.apple.com/documentation/swift/optionset) from an enum containing flags and conforming it to the `OptionSet` protocol or adding a memberwise initializer. +* Creating accessors for a stored property or subscript, subsuming some of the behavior of [SE-0258 "Property Wrappers"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md). +* Augmenting members of a type with a new attribute, such as applying a property wrapper to all stored properties of a type. + +There is an [example repository](https://github.com/DougGregor/swift-macro-examples) containing a number of macros that have been implemented using the prototype of this feature. + +## Proposed solution + +The proposal adds *attached macros*, so-called because they are attached to a particular declaration. They are written using the custom attribute syntax (e.g., `@AddCompletionHandler`) that already provides extensibility for declarations through property wrappers, result builders, and global actors. Attached macros can reason about the declaration to which they are attached, and provide additions and changes based on one or more different macro *roles*. Each role has a specific purpose, such as adding members, creating accessors, or adding peers alongside the declaration. A given attached macro can inhabit several different roles, and as such will be expanded multiple times corresponding to the different roles, which allows the various roles to be composed. For example, an attached macro emulating property wrappers might inhabit both the "peer" and "accessor" roles, allowing it to introduce a backing storage property and also synthesize a getter/setter that go through that backing storage property. Composition of macro roles will be discussed in more depth once the basic macro roles have been established. + +As with expression macros, attached declaration macros are declared with `macro`, and have [type-checked macro arguments](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#type-checked-macro-arguments-and-results) that allow their behavior to be customized. Attached macros are identified with the `@attached` attribute, which also provides the specific role as well as any names they introduce. For example, the aforemented macro to add a completion handler would be declared as follows: + +```swift +@attached(peer, names: overloaded) +macro AddCompletionHandler(parameterName: String = "completionHandler") +``` + +The macro can be used as follows: + +```swift +@AddCompletionHandler(parameterName: "onCompletion") +func fetchAvatar(_ username: String) async -> Image? { ... } +``` + +The use of the macro is attached to `fetchAvatar`, and generates a *peer* declaration alongside `fetchAvatar` whose name is "overloaded" with `fetchAvatar`. The generated declaration is: + +```swift +/// Expansion of the macro produces the following. +func fetchAvatar(_ username: String, onCompletion: @escaping (Image?) -> Void) { + Task.detached { + onCompletion(await fetchAvatar(username)) + } +} +``` + +### Implementing attached macros + +All attached macros are implemented as types that conform to one of the protocols that inherits from the `AttachedMacro` protocol. Like the [`Macro` protocol](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-protocols), the `AttachedMacro` protocol has no requirements, but is used to organize macro implementations. Each attached macro role will have its own protocol that inherits `AttachedMacro`. + +```swift +public protocol AttachedMacro: Macro { } +``` + +The biggest difference from expression macros is that there is more than one relevant piece of syntax for the macro to consider: each attached macro implementation receives the syntax node for both the attribute (e.g., `@AddCompletionHandler(parameterName: "onCompletion")`) and the declaration to which the macro is attached (`func fetchAvatar`...), and can return new code that's appropriate to the macro role: peer macros return new declarations, accessor macros return getters/getters, and so on. For example, `PeerMacro` is defined as follows: + +```swift +public PeerMacro: AttachedMacro { + /// Expand a macro described by the given attribute to + /// produce "peer" declarations of the declaration to which it + /// is attached. + /// + /// The macro expansion can introduce "peer" declarations that + /// go alongside the given declaration. + static func expansion( + of node: AttributeSyntax, + providingPeersOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] +} +``` + +### Naming macro-produced declarations + +Unlike expression macros, attached macros can introduce new declarations. These declarations can have an impact on code elsewhere in the program, for example if a macro provides a declaration of a function named `hello` and that function is called from another source file. Our design requires macros to document which names they can introduce: this provides more information up-front to developers and tools alike to understand the impact that a macro can have on the surrounding program. For developers, this can mean fewer surprises; for tools, this can be used to improve compilation times by avoiding unnecessary macro expansions. + +The `@AddCompletionHandler` macro notes that it introduces an *overloaded* name, meaning that it produces a declaration with the same base name as the declaration to which it is attached. A macro that emulated a property wrapper would specify the storage name via `prefixed(_)`, meaning that `_` will be added as a prefix to the name of the declaration to which that macro is attached. Other ways in which macro-generated names are communicated are discussed in the Detailed Design. + +### Kinds of attached macros + +#### Peer macros + +Peer macros produce new declarations alongside the declaration to which they are attached. The `AddCompletionHandler` macro from earlier was a peer macro. Peer macros are implemented via types that conform to the `PeerMacro` protocol shown earlier. The implementation of `AddCompletionHandlerMacro` looks like the following: + +```swift +public struct AddCompletionHandlerMacro: PeerDeclarationMacro { + public static func expansion( + of node: CustomAttributeSyntax, + providingPeersOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] { + // make sure we have an async function to start with + // form a new function "completionHandlerFunc" by starting with that async function and + // - remove async + // - remove result type + // - add a completion-handler parameter + // - add a body that forwards arguments + // return the new peer function + return [completionHandlerFunc] + } +} +``` + +The details of the implementation are left to an Appendix, with a complete version in the [example repository](https://github.com/DougGregor/swift-macro-examples). + +#### Member macros + +Member macros allow one to introduce new members into the type or extension to which the macro is attached. For example, we can write a macro that defines static members to ease the definition of an [`OptionSet`](https://developer.apple.com/documentation/swift/optionset). Given: + +```swift +@OptionSetMembers +struct MyOptions: OptionSet { + enum Option: Int { + case a + case b + case c + } +} +``` + +This struct should be expanded to contain both a `rawValue` field and static properties for each of the options, e.g., + +```swift +// Expands to... +struct MyOptions: OptionSet { + enum Option: Int { + case a + case b + case c + } + + // Synthesized code below... + var rawValue: Int = 0 + + static var a = MyOptions(rawValue: 1 << Option.a.rawValue) + static var b = MyOptions(rawValue: 1 << Option.b.rawValue) + static var c = MyOptions(rawValue: 1 << Option.c.rawValue) +} +``` + +The macro itself will be declared as a member macro that defines an arbitrary set of members: + +```swift +/// Create the necessary members to turn a struct into an option set. +@attached(member, names: named(rawValue), arbitrary) macro OptionSetMembers() +``` + +The `member` role specifies that this macro will be defining new members of the declaration to which it is attached. In this case, while the macro knows it will define a member named `rawValue`, there is no way for the macro to predict the names of the static properties it is defining, so it also specifies `arbitrary` to indicate that it will introduce members with arbitrarily-determined names. + +Member macros are implemented with types that conform to the `MemberMacro` protocol: + +```swift +protocol MemberMacro: AttachedMacro { + /// Expand a macro described by the given attribute to + /// produce additional members of the given declaration to which + /// the attribute is attached. + static func expansion( + of node: AttributeSyntax, + providingMembersOf declaration: some DeclGroupSyntax, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] +} +``` + +#### Accessor macros + +Accessor macros allow a macro to add accessors to a property, turning a stored property into a computed property. For example, consider a macro that can be applied to a stored property to instead access a dictionary keyed by the property name. Such a macro could be used like this: + +```swift +struct MyStruct { + var storage: [AnyHashable: Any] = [:] + + @DictionaryStorage + var name: String + + @DictionaryStorage(key: "birth_date") + var birthDate: Date? +} +``` + +The `DictionaryStorage` attached macro would alter the properties of `MyStruct` as follows, adding accessors to each: + +```swift +struct MyStruct { + var storage: [String: Any] = [:] + + var name: String { + get { + storage["name"]! as! String + } + + set { + storage["name"] = newValue + } + } + + var birthDate: Date? { + get { + storage["birth_date"] as Date? + } + + set { + if let newValue { + storage["birth_date"] = newValue + } else { + storage.removeValue(forKey: "birth_date") + } + } + } +} +``` + +The macro can be declared as follows: + +```swift +@attached(accessor) macro DictionaryStorage(key: String? = nil) +``` + +Implementations of accessor macros conform to the `AccessorMacro` protocol, which is defined as follows: + +```swift +protocol AccessorMacro: AttachedMacro { + /// Expand a macro described by the given attribute to + /// produce accessors for the given declaration to which + /// the attribute is attached. + static func expansion( + of node: AttributeSyntax, + providingAccessorsOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [AccessorDeclSyntax] +} +``` + +The implementation of the `DictionaryStorage` macro would create the accessor declarations shown above, using either the `key` argument (if present) or deriving the key name from the property name. The effect of this macro isn't something that can be done with a property wrapper, because the property wrapper wouldn't have access to `self.storage`. + +An accessor macro can specify that it produces observers by listing at least one of `willSet` or `didSet ` in the names, e.g., `@attached(accessors, names: named(willSet))`. Such a macro can only produce observers; it cannot change a stored property into a computed property. + +The expansion of an accessor macro that does not specify one of `willSet` or `didSet` in its list of names must result in a computed property. A side effect of the expansion is to remove any initializer from the stored property itself; it is up to the implementation of the accessor macro to either diagnose the presence of the initializer (if it cannot be used) or incorporate it in the result. + +#### Member attribute macros + +Member declaration macros allow one to introduce new member declarations within the type or extension to which they apply. In contrast, member *attribute* macros allow one to modify the member declarations that were explicitly written within the type or extension by adding new attributes to them. Those new attributes could then refer to other macros, such as peer or accessor macros, or builtin attributes. As such, they are primarily a means of *composition*, since they have fairly little effect on their own. + +Member attribute macros allow a macro to provide similar behavior to how many built-in attributes work, where declaring the attribute on a type or extension will apply that attribute to each of the members. For example, a global actor `@MainActor` written on an extension applies to each of the members of that extension (unless they specify another global actor or `nonisolated`), an access specifier on an extension applies to each of the members of that extension, and so on. + +As an example, we'll define a macro that partially mimics the behavior of the `@objcMembers` attributes introduced long ago in [SE-0160](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0160-objc-inference.md#re-enabling-objc-inference-within-a-class-hierarchy). Our `myObjCMembers` macro is a member-attribute macro: + +```swift +@attached(memberAttribute) +macro MyObjCMembers() +``` + +The implementation of this macro will add the `@objc` attribute to every member of the type or extension, unless that member either already has an `@objc` macro or has `@nonobjc` on it. For example, this: + +```swift +@MyObjCMembers extension MyClass { + func f() { } + + var answer: Int { 42 } + + @objc(doG) func g() { } + + @nonobjc func h() { } +} +``` + +would result in: + +```swift +extension MyClass { + @objc func f() { } + + @objc var answer: Int { 42 } + + @objc(doG) func g() { } + + @nonobjc func h() { } +} +``` + +Member-attribute macro implementations should conform to the `MemberAttributeMacro` protocol: + +```swift +protocol MemberAttributeMacro: AttachedMacro { + /// Expand a macro described by the given custom attribute to + /// produce additional attributes for the members of the type. + static func expansion( + of node: AttributeSyntax, + attachedTo declaration: some DeclGroupSyntax, + providingAttributesOf member: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [AttributeSyntax] +} +``` + +The `expansion` operation accepts the attribute syntax `node` for the spelling of the member attribute macro and the declaration to which that attribute is attached (i.e., the type or extension). The implementation of the macro can walk the members of the `declaration` to determine which members require additional attributes. The returned dictionary will map from those members to the additional attributes that should be added to each of the members. + +#### Conformance macros + +Conformance macros allow one to introduce new protocol conformances to a type. This would often be paired with other macros whose purpose is to help satisfy the protocol conformance. For example, one could imagine an extended version of the `OptionSetMembers` attributed shown earlier that also adds the `OptionSet` conformance. With it, the minimal implementation of an option set could be: + +```swift +@OptionSet +struct MyOptions { + enum Option: Int { + case a + case b + case c + } +} +``` + +where `OptionSet` is both a member and a conformance macro, providing members (as in `OptionSetMembers`) and the conformance to `OptionSet`. The macro would be declared as, e.g., + +```swift +/// Create the necessary members to turn a struct into an option set. +@attached(member, names: named(rawValue), arbitrary) +@attached(conformance) +macro OptionSet() +``` + +Conformance macro implementations should conform to the `ConformanceMacro` protocol: + +```swift +/// Describes a macro that can add conformances to the declaration it's +/// attached to. +public protocol ConformanceMacro: AttachedMacro { + /// Expand an attached conformance macro to produce a set of conformances. + /// + /// - Parameters: + /// - node: The custom attribute describing the attached macro. + /// - declaration: The declaration the macro attribute is attached to. + /// - context: The context in which to perform the macro expansion. + /// + /// - Returns: the set of `(type, where-clause?)` pairs that each provide the + /// protocol type to which the declared type conforms, along with + /// an optional where clause. + static func expansion( + of node: AttributeSyntax, + providingConformancesOf declaration: some DeclGroupSyntax, + in context: some MacroExpansionContext + ) throws -> [(TypeSyntax, WhereClauseSyntax?)] +} +``` + +The returned array contains the conformances. The `TypeSyntax` describes the protocol to which the enclosing type conforms, and the optional `where` clause provides any additional constraints that would make the conformance conditional. + +## Detailed design + +### Composing macro roles + +A given macro can have several different roles, allowing the various macro features to be composed. Each of the roles is considered independently, so a single use of a macro in source code can result in different macro expansion functions being called. These calls are independent, and could even happen concurrently. As an example, let's define a macro that emulates property wrappers fairly closely. The property wrappers proposal has an example for a [clamping property wrapper](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md#clamping-a-value-within-bounds): + +```swift +@propertyWrapper +struct Clamping { + var value: V + let min: V + let max: V + + init(wrappedValue: V, min: V, max: V) { + value = wrappedValue + self.min = min + self.max = max + assert(value >= min && value <= max) + } + + var wrappedValue: V { + get { return value } + set { + if newValue < min { + value = min + } else if newValue > max { + value = max + } else { + value = newValue + } + } + } +} + +struct Color { + @Clamping(min: 0, max: 255) var red: Int = 127 + @Clamping(min: 0, max: 255) var green: Int = 127 + @Clamping(min: 0, max: 255) var blue: Int = 127 + @Clamping(min: 0, max: 255) var alpha: Int = 255 +} +``` + +Instead, let's implement this as a macro: + +```swift +@attached(peer, prefixed(_)) +@attached(accessor) +macro Clamping(min: T, max: T) = #externalMacro(module: "MyMacros", type: "ClampingMacro") +``` + +The usage syntax is the same in both cases. As a macro, `Clamping` both defines a peer (a backing storage property with an `_` prefix) and also defines accessors (to check min/max). The peer declaration macro is responsible for defining the backing storage, e.g., + +```swift +private var _red: Int = { + let newValue = 127 + let minValue = 0 + let maxValue = 255 + if newValue < minValue { + return minValue + } + if newValue > maxValue { + return maxValue + } + return newValue +}() +``` + +Which is implemented by having `ClampingMacro` conform to `PeerMacro`: + +```swift +enum ClampingMacro { } + +extension ClampingMacro: PeerDeclarationMacro { + static func expansion( + of node: CustomAttributeSyntax, + providingPeersOf declaration: DeclSyntax, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] { + // create a new variable declaration that is the same as the original, but... + // - prepend an underscore to the name + // - make it private + } +} +``` + +And introduces accessors such as + +```swift + get { return _red } + + set(__newValue) { + let __minValue = 0 + let __maxValue = 255 + if __newValue < __minValue { + _red = __minValue + } else if __newValue > __maxValue { + _red = __maxValue + } else { + _red = __newValue + } + } +``` + +by conforming to `AccessorMacro`: + +```swift +extension ClampingMacro: AccessorMacro { + static func expansion( + of node: CustomAttributeSyntax, + providingAccessorsOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [AccessorDeclSyntax] { + let originalName = /* get from declaration */, + minValue = /* get from custom attribute node */, + maxValue = /* get from custom attribute node */ + let storageName = "_\(originalName)", + newValueName = context.getUniqueName(), + maxValueName = context.getUniqueName(), + minValueName = context.getUniqueName() + return [ + """ + get { \(storageName) } + """, + """ + set(\(newValueName)) { + let \(minValueName) = \(minValue) + let \(maxValueName) = \(maxValue) + if \(newValueName) < \(minValueName) { + \(storageName) = \(minValueName) + } else if \(newValueName) > maxValue { + \(storageName) = \(maxValueName) + } else { + \(storageName) = \(newValueName) + } + } + """ + ] + } +} +``` + +This formulation of `@Clamping` offers some benefits over the property-wrapper version: we don't need to store the min and max values as part of the backing storage (so the presence of `@Clamping` doesn't add any storage), nor do we need to define a new type. More importantly, it demonstrates how the composition of different macro roles together can produce interesting effects. + +Not every macro role applies to every kind of declaration: an `accessor` macro doesn't make sense of a function, nor does a `member` member make sense on a property. If a macro is attached to a declaration, the macro will only be expanded for those roles that are applicable to a declaration. For example, if the `Clamping` macro is applied to a function, it will only be expanded as a peer macro; it's role as an accessor macro is ignored. If a macro is attached to a declaration and none of its roles apply to that kind of declaration, the program is ill-formed. + +### Specifying newly-introduced names + +Whenever a macro produces declarations that are visible to other Swift code, it is required to declare the names in advance. All of the names need to be specified within the attribute declaring the macro role, using the following forms: + +- Declarations with a specific fixed name: `named()`. +- Declarations whose names cannot be described statically, for example because they are derived from other inputs: `arbitrary`. + +* Declarations that have the same base name as the declaration to which the macro is attached, and are therefore overloaded with it: `overloaded`. +* Declarations whose name is formed by adding a prefix to the name of the declaration to which the macro is attached: `prefixed(_)`. As a special consideration, `$` is permissible as a prefix, allowing macros to produce names with a leading `$` that are derived from the name of the declaration to which the macro is attached. This carve-out enables macros that behavior similarly to property wrappers that introduce a projected value. +* Declarations whose name is formed by adding a suffix to the name of the declaration to which the macro is attached: `suffixed(_docinfo)`. + +A macro can only introduce new declarations whose names are covered by the kinds provided, or have their names generated via `MacroExpansionContext.createUniqueName`. This ensures that, in most cases (where `arbitrary` is not specified), the Swift compiler and related tools can reason about the set of names that will be introduced by a given use of a macro without having to expand the macro, which can reduce the compile-time cost of macros and improve incremental builds. For example, when the compiler performs name lookup for a name `_x`, it can avoid expanding any macros that are unable to produce a declaration named `_x`. Macros that can produce arbitrary names must always be expanded, as would a macro with `prefixed(_)` that is attached to a declaration named `x`. + +The macro is not required to provide a declaration for every name it describes: for example, `OptionSetMembers` will state that it produces a declaration named `rawValue`, but the implementation may opt not to do so if it sees that such a property already exists. + +### Visibility of names used and introduced by macros + +Macros can introduce new names into the program. Whether and when those names are visible to other parts of the program, and in particular other macros, is a subtle design problem that has a number of interesting constraints. + +First, the arguments to a macro are type-checked before it can be determined which macro is being expanded, so they cannot depend on the expansion of that macro. For example, consider an attached macro use spelled `@myMacro(x)`: if it introduced a declaration named `x`, it would potentially invalidate the type-check of its own argument, or cause that macro expansion to choose a different `myMacro` that doesn't produce an `x`! + +Second, if the output of one macro expansion (say, `@myMacro1(x)`) introduces a name (say, `y`) that is then used in the argument to another macro expansion (say, `@myMacro2(y)`), then the order in which the macros are expanded can affect the semantics of the resulting program. It is not generally possible to introduce an ordering on macro expansions, nor would it be desirable, because Swift generally tries not to have order dependencies among declarations in a program. Incidental orderings based on the names introduced by the macro don't help, either, because names of declaration macros can be specified as `arbitrary` and therefore cannot be predicated. + +Third, macro expansion is a relatively expensive compile-time operation, involving serialization overhead to transfer parts of the program from the compiler/IDE to another program that expands the macro. Therefore, macros are [only expanded once per use site](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion), so their expansions cannot participate in the type checking of their arguments or of other surrounding statements or expressions. For example, consider a this (intentionally obtuse) Swift code: + +```swift +@attached(peer) macro myMacro(_: Int) +@attached(peer, names: named(y)) macro myMacro(_: Double) + +func f(_: Int, body: (Int) -> Void) +func f(_: Double, body: (Double) -> Void) + +f(1) { x in + @myMacro(x) func g() { } + print(y) +} +``` + +The macro use `@myMacro(x)` provides different names depending on whether the argument is an `Int` or ` Double`. From the perspective of the call to `f`, both overloads of `f` are possible, and the only way to check beyond that macro use to the `print` expression that follows is to try to expand the macro... for the first `myMacro`, the `print(y)` expression will fail to type-check (there is no `y`) whereas the second would find the `y` generated by the second `myMacro` and will succeed. However, this approach does not scale, because it could involve expanding macros a large number of times during type checking. Moreover, the macro expansions might end up getting incomplete information if, for example, macros are someday provided with type information and that type information isn't known yet (because the expansion is happening during type inference). + +To address these issues, a name introduced by a macro expansion is not visible within macro arguments for macros the same scope as the macro expansion or any of its enclosing scopes. This is conceptually similar to [outside-in expansion of expression macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion), where one can imagine that all of the macros at a particular scope are type-checked and expanded simultaneously at the top-level scope. Then, macros within each type definition and its extensions are type-checked and expanded simultaneously: they can see the names introduced by top-level macro expansions, but not names introduced at other levels. The following annotated code example shows which names are visible: + +```swift +import OtherModule // declares names x, y, z + +@macro1(x) // uses OtherModule.x, introduces name y +func f() { } + +@macro2(y) // uses OtherModule.y, introduce name x +func g() { } + +struct S1 { + @macro3(x) // uses the name x introduced by #@acro2, introduces the name z + func h() +} + +extension S1 { + @macro4(z) // uses OtherModule.z + func g() { } + + func f() { + print(z) // uses z introduced by @macro3 + } +} +``` + +These name lookup rules, while important for providing a reasonable scheme for macros to be expanded without interfering with each other, do introduce a new kind of name shadowing to the language. If `@macro2` were manually expanded into source code, the declaration `x` it produces would then become visible to the macro argument in `@macro1(x)`, changing the behavior of the program. The compiler might be able to detect such shadowing in practice, when a macro-introduced name at a particular scope would change the meaning of a lookup from that particular scope. + +Within function and closure bodies, the fact that names introduced by the macro expansion are not visible within the current scope means that our earlier example will never find a declaration `y` introduced by `@myMacro`: + +```swift +f(1) { x in + @myMacro(x) func g() { } + print(y) // does not consider any names introduced by `@myMacro` +} +``` + +Therefore, a macro used within a closure or function body can only introduce declarations using names produced by `createUniqueName`. This maintains the [two-phase of checking macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion) where type checking and inference is performed without expanding the macro, then the macro is expanded and its result type-checked independently, with no ability to influence type inference further. + +### Restrictions on `arbitrary` names + +Attached macros that specify `arbitrary` names require expanding the macro in order to determine the exact set of names that the macro generates. This means that name lookup must expand all macros that can produce `arbitrary` names in the given scope in order to determine name lookup results. This is a problem for macros that introduce names at global scope, because any unqualified or module qualified lookup must expand those macros. However, because macro attributes can have arguments that are type-checked prior to macro expansion, type checking those arguments may require unqualified or module qualified name lookup that requires expanding that same macro, which is fundamentally circular. Though macro arguments do not have visibility of macro-generated names in the same scope, macro argument type checking can invoke type checking of other declarations that can use macro-generated names. For example: + +```swift +@attached(peer, names: arbitrary) +macro IntroduceArbitraryPeers(_: T) = #externalMacro(...) + +@IntroduceArbitraryPeers(MyType().x) +struct S {} + +struct MyType { + var x: AnotherType +} +``` + +Resolving the macro attribute `@IntroduceArbitraryPeers(MyType().x)` can invoke type-checking of the `var x: AnotherType` property of the `MyType` struct. Unqualified lookup of `AnotherType` must expand `@IntroduceArbitraryPeers` because the macro can introduce arbitrary names; it might introduce a top-level type called `AnotherType`. Resolving the `@IntroduceArbitraryPeers` attribute depends on type checking the `var x: AnotherType` property, and type checking the property depends on expanding the `@IntroduceArbitraryPeers` macro, which results in a circular reference error. + +Because `arbitrary` names introduced at global scope are extremely prone to circularity, peer macros attached to top-level declarations cannot introduce `arbitrary` names. + + +### Ordering of macro expansions + +When multiple macros are applied to a single declaration, the order in which macros are expanded could have a significant impact on the resulting program if the the outputs of one macro expansion are treated as inputs to the others. This form of macro composition could be fairly powerful, but it also can have a number of surprising side effects: + +1. The order in which attributes are written on a declaration could affect the result. +2. There is no obvious single source of truth for what the inputs to the macro would be. +3. Independently-developed macros could conflict in ways that break the program. + +The fine-grained nature of attached macros means that many attached macros have independent effects, e.g., macros can generate distinct members, conformances, and peers. We consider this independence of different macros applying to the same definition to be a feature, because it makes the application of macros more predictable as well as enabling more efficient implementations. + +To ensure the independence of macro invocations, each macro expansion is provided with syntax nodes as they were originally written, and not updated with the effects of other macros. For example, consider the `OptionSet` macro earlier: + +```swift +@OptionSet +struct MyOptions { + enum Option: Int { + case a + case b + case c + } +} +``` + +The `OptionSet` macro is both a member macro (which generates the instance property `rawValue` as well as static properties `a`, `b`, and `c`) and a conformance macro (to add the `OptionSet` conformance). Two different `expansion` functions will be called on the option set macro implementation: `expansion(of:providingMembersOf:in:)` and `expansion(of:providingConformancesOf:in:)`, respectively. In both cases, the syntax tree for `MyOptions` is passed as the middle argument, and both functions provide a result that augments the definition of `MyOptions`, either by adding members or by adding a conformance. + +In both cases, the expansion operation is provided with the original definition of `MyOptions`, without the new conformance or added members. That way, each expansion operation operates independently of the other---whether it's different roles for the same macro, or different macros entirely---and the order of expansion does not matter. + +### Permitted declaration kinds + +A macro can expand to any declaration that is syntactically and semantically well-formed within the context where the macro is expanded, with a few notable exceptions: + +* `import` declarations can never be produced by a macro. Swift tooling depends on the ability to resolve import declarations based on a simple scan of the original source files. Allowing a macro expansion to introduce an import declaration would complicate import resolution considerably. +* A type with the `@main` attribute cannot be produced by a macro. Swift tooling should be able to determine the presence of a main entry point in a source file based on a syntactic scan of the source file without expanding macros. +* `extension` declarations can never be produced by a macro. The effect of an extension declaration is wide-ranging, with the ability to add conformances, members, and so on. These capabilities are meant to be introduced in a more fine-grained manner. +* `operator` and `precedencegroup` declarations can never be produced by a macro, because they could allow one to reshape the precedence graph for existing code causing subtle differences in the semantics of code that sees the macro expansion vs. code that does not. +* `macro` declarations can never be produced by a macro, because allowing this would allow a macro to trivially produce infinitely recursive macro expansion. +* Top-level default literal type overrides, including `IntegerLiteralType`, + `FloatLiteralType`, `BooleanLiteralType`, + `ExtendedGraphemeClusterLiteralType`, `UnicodeScalarLiteralType`, and + `StringLiteralType`, can never be produced by a macro. + +## Source compatibility + +Attached declaration macros use the same syntax introduced for custom attributes (such as property wrappers), and therefore do not have an impact on source compatibility. + +## Effect on ABI stability + +Macros are a source-to-source transformation tool that have no ABI impact. + +## Effect on API resilience + +Macros are a source-to-source transformation tool that have no effect on API resilience. + +## Alternatives considered + +### Mutating declarations rather than augmenting them + +Attached declaration macro implementations are provided with information about the macro expansion itself and the declaration to which they are attached. The implementation cannot directly make changes to the syntax of the declaration to which it is attached; rather, it must specify the additions or changes by packaging them in `AttachedDeclarationExpansion`. + +An alternative approach would be to allow the macro implementation to directly alter the declaration to which the macro is attached. This would provide a macro implementation with greater flexibility to affect the declarations it is attached to. However, it means that the resulting declaration could vary drastically from what the user wrote, and would preventing the compiler from making any determinations about what the declaration means before expanding the macro. This could have detrimental effects on compile-time performance (one cannot determine anything about a declaration until the macros have been run on it) and might also prevent macros from accessing information about the actual declaration in the future, such as the types of parameters. + +It might be possible to provide a macro implementation API that is expressed in terms of mutation on the original declaration, but restrict the permissible changes to those that can be handled by the implementation. For example, one could "diff" the syntax tree provided to the macro implementation and the syntax tree produced by the macro implementation to identify changes, and return those changes that are acceptable to the compiler. + +## Revision History + +* Revision after acceptance: + * Make the `expansion` requirements non-`async`. +* After the first pitch: + * Added conformance macros, to produce conformances + * Moved the discussion of macro-introduced names from the freestanding macros proposal here. + * Added a carve-out to allow a `$` prefix on names generated from macros, allowing them to match the behavior of property wrappers. + * Revisited the design around the ordering of macro expansions, forcing them to be independent. + * Clarify when accessor macros need to produce observers for a stored property vs. turning it into a computed property. + * Moved the discussion of "permitted declaration kinds" from the freestanding macros proposal here. +* Originally pitched as "declaration macros"; attached macros were separated into their own proposal after the initial discussion. + +## Future directions + +### Additional attached macro roles + +There are numerous ways in which this proposal could be extended to provide new macro roles. Each new macro role would introduce a new role kind to the `@attached` attribute, along with a corresponding protocol. The macro vision document has a number of such suggestions. + +## Appendix + +### Implementation of `AddCompletionHandler` + +```swift +public struct AddCompletionHandler: PeerDeclarationMacro { + public static func expansion( + of node: CustomAttributeSyntax, + providingPeersOf declaration: some DeclSyntaxProtocol, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] { + // Only on functions at the moment. We could handle initializers as well + // with a little bit of work. + guard let funcDecl = declaration.as(FunctionDeclSyntax.self) else { + throw CustomError.message("@AddCompletionHandler only works on functions") + } + + // This only makes sense for async functions. + if funcDecl.signature.asyncOrReasyncKeyword == nil { + throw CustomError.message( + "@AddCompletionHandler requires an async function" + ) + } + + // Form the completion handler parameter. + let resultType: TypeSyntax? = funcDecl.signature.output?.returnType.withoutTrivia() + + let completionHandlerParam = + FunctionParameterSyntax( + firstName: .identifier("completionHandler"), + colon: .colonToken(trailingTrivia: .space), + type: "(\(resultType ?? "")) -> Void" as TypeSyntax + ) + + // Add the completion handler parameter to the parameter list. + let parameterList = funcDecl.signature.input.parameterList + let newParameterList: FunctionParameterListSyntax + if let lastParam = parameterList.last { + // We need to add a trailing comma to the preceding list. + newParameterList = parameterList.removingLast() + .appending( + lastParam.withTrailingComma( + .commaToken(trailingTrivia: .space) + ) + ) + .appending(completionHandlerParam) + } else { + newParameterList = parameterList.appending(completionHandlerParam) + } + + let callArguments: [String] = try parameterList.map { param in + guard let argName = param.secondName ?? param.firstName else { + throw CustomError.message( + "@AddCompletionHandler argument must have a name" + ) + } + + if let paramName = param.firstName, paramName.text != "_" { + return "\(paramName.withoutTrivia()): \(argName.withoutTrivia())" + } + + return "\(argName.withoutTrivia())" + } + + let call: ExprSyntax = + "\(funcDecl.identifier)(\(raw: callArguments.joined(separator: ", ")))" + + // FIXME: We should make CodeBlockSyntax ExpressibleByStringInterpolation, + // so that the full body could go here. + let newBody: ExprSyntax = + """ + + Task.detached { + completionHandler(await \(call)) + } + + """ + + // Drop the @addCompletionHandler attribute from the new declaration. + let newAttributeList = AttributeListSyntax( + funcDecl.attributes?.filter { + guard case let .customAttribute(customAttr) = $0 else { + return true + } + + return customAttr != node + } ?? [] + ) + + let newFunc = + funcDecl + .withSignature( + funcDecl.signature + .withAsyncOrReasyncKeyword(nil) // drop async + .withOutput(nil) // drop result type + .withInput( // add completion handler parameter + funcDecl.signature.input.withParameterList(newParameterList) + .withoutTrailingTrivia() + ) + ) + .withBody( + CodeBlockSyntax( + leftBrace: .leftBraceToken(leadingTrivia: .space), + statements: CodeBlockItemListSyntax( + [CodeBlockItemSyntax(item: .expr(newBody))] + ), + rightBrace: .rightBraceToken(leadingTrivia: .newline) + ) + ) + .withAttributes(newAttributeList) + .withLeadingTrivia(.newlines(2)) + + return [DeclSyntax(newFunc)] + } +} +``` diff --git a/proposals/0390-noncopyable-structs-and-enums.md b/proposals/0390-noncopyable-structs-and-enums.md new file mode 100644 index 0000000000..79549cd230 --- /dev/null +++ b/proposals/0390-noncopyable-structs-and-enums.md @@ -0,0 +1,1906 @@ +# Noncopyable structs and enums + +* Proposal: [SE-0390](0390-noncopyable-structs-and-enums.md) +* Authors: [Joe Groff](https://github.com/jckarter), [Michael Gottesman](https://github.com/gottesmm), [Andrew Trick](https://github.com/atrick), [Kavon Farvardin](https://github.com/kavon) +* Review Manager: [Stephen Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 5.9)** +* Implementation: in main branch of compiler +* Review: ([pitch](https://forums.swift.org/t/pitch-noncopyable-or-move-only-structs-and-enums/61903)) ([first review](https://forums.swift.org/t/se-0390-noncopyable-structs-and-enums/63258)) ([second review](https://forums.swift.org/t/second-review-se-0390-noncopyable-structs-and-enums/63866)) ([acceptance](https://forums.swift.org/t/accepted-se-0390-noncopyable-structs-and-enums/65157)) +* Previous Revisions: [1](https://github.com/swiftlang/swift-evolution/blob/5d075b86d57e3436b223199bd314b2642e30045f/proposals/0390-noncopyable-structs-and-enums.md) + +## Introduction + +This proposal introduces the concept of **noncopyable** types (also known +as "move-only" types). An instance of a noncopyable type always has unique +ownership, unlike normal Swift types which can be freely copied. + +## Motivation + +All currently existing types in Swift are **copyable**, meaning it is possible +to create multiple identical, interchangeable representations of any value of +the type. However, copyable structs and enums are not a great model for +unique resources. Classes by contrast *can* represent a unique resource, +since an object has a unique identity once initialized, and only references to +that unique object get copied. However, because the references to the object are +still copyable, classes always demand *shared ownership* of the resource. This +imposes overhead in the form of heap allocation (since the overall lifetime of +the object is indefinite) and reference counting (to keep track of the number +of co-owners currently accessing the object), and shared access often +complicates or introduces unsafety or additional overhead into an object's +APIs. Swift does not yet have a mechanism for defining types that +represent unique resources with *unique ownership*. + +## Proposed solution + +We propose to allow for `struct` and `enum` types to declare themselves as +*noncopyable*, using a new syntax for suppressing implied generic constraints, +`~Copyable`. Values of noncopyable type always have unique ownership, and +can never be copied (at least, not using Swift's implicit copy mechanism). +Since values of noncopyable structs and enums have unique identities, they can +also have `deinit` declarations, like classes, which run automatically at the +end of the unique instance's lifetime. + +For example, a basic file descriptor type could be defined as: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + init(fd: Int32) { self.fd = fd } + + func write(buffer: Data) { + buffer.withUnsafeBytes { + write(fd, $0.baseAddress!, $0.count) + } + } + + deinit { + close(fd) + } +} +``` + +Like a class, instances of this type can provide managed access to a file +handle, automatically closing the handle once the value's lifetime ends. Unlike +a class, no object needs to be allocated; only a simple struct containing the +file descriptor ID needs to be stored in the stack frame or aggregate value +that uniquely owns the instance. + +## Detailed design + +### The `Copyable` generic constraint + +Before this proposal, almost every type in Swift was automatically copyable. +The standard library provides a new generic constraint `Copyable` to make +this capability explicit. All existing first-class types (excluding nonescaping +closures) implicitly satisfy this constraint, and all generic type parameters, +existential types, protocols, and associated type requirements implicitly +require it. Types may explicitly declare that they are `Copyable`, and generic +types may explicitly require `Copyable`, but this currently has no effect. + +```swift +struct Foo: Copyable {} +``` + +### Declaring noncopyable types + +A `struct` or `enum` type can be declared as noncopyable by suppressing the +`Copyable` requirement on their declaration, by combining the new `Copyable` +constraint with the new requirement suppression syntax `~Copyable`: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 +} +``` + +If a `struct` has a stored property of noncopyable type, or an `enum` has +a case with an associated value of noncopyable type, then the containing type +must also suppress its `Copyable` capability: + +```swift +struct SocketPair: ~Copyable { + var in, out: FileDescriptor +} + +enum FileOrMemory: ~Copyable { + // write to an OS file + case file(FileDescriptor) + // write to an array in memory + case memory([UInt8]) +} + +// ERROR: copyable value type cannot contain noncopyable members +struct FileWithPath { + var file: FileDescriptor + var path: String +} +``` + +Classes, on the other hand, may contain noncopyable stored properties without +themselves becoming noncopyable: + +```swift +class SharedFile { + var file: FileDescriptor +} +``` + +A class type declaration may not use `~Copyable`; all class types remain copyable +by retaining and releasing references to the object. + +```swift +// ERROR: classes must be `Copyable` +class SharedFile: ~Copyable { + var file: FileDescriptor +} +``` + +It is also not yet allowed to suppress the `Copyable` requirement on generic +parameters, associated type requirements in protocols, or the `Self` type +in a protocol declaration, or in extensions: + +```swift +// ERROR: generic parameter types must be `Copyable` +func foo(x: T) {} + +// ERROR: types that conform to protocols must be `Copyable` +protocol Foo where Self: ~Copyable { + // ERROR: associated type requirements must be `Copyable` + associatedtype Bar: ~Copyable +} + +// ERROR: cannot suppress `Copyable` in extension of `FileWithPath` +extension FileWithPath: ~Copyable {} +``` + +`Copyable` also cannot be suppressed in existential type declarations: + +```swift +// ERROR: `any` types must be `Copyable` +let foo: any ~Copyable = FileDescriptor() +``` + +### Restrictions on use in generics + +Noncopyable types may have generic type parameters: + +```swift +// A type that reads from a file descriptor consisting of binary values of type T +// in sequence. +struct TypedFile: ~Copyable { + var rawFile: FileDescriptor + + func read() -> T { ... } +} + +let byteFile: TypedFile // OK +``` + +At this time, as noted above, generic types are still always required to be +`Copyable`, so noncopyable types themselves are not allowed to be used as a +generic type argument. This means a noncopyable type _cannot_: + +- conform to any protocols, except for `Sendable`. +- serve as a type witness for an `associatedtype` requirement. +- be used as a type argument when instantiating generic types or calling generic functions. +- be cast to (or from) `Any` or any other existential. +- be accessed through reflection. +- appear in a tuple. + +The reasons for these restrictions and ways of lifting them are discussed under +Future Directions. The key implication of these restrictions is that a +noncopyable struct or enum is only a subtype of itself, because all other types +it might be compatible with for conversion would also permit copying. + +#### The `Sendable` exception + +The need for preventing noncopyable types from conforming to +protocols is rooted in the fact that all existing constrained generic types +(like `some P` types) and existentials (`any P` types) are assumed to be +copyable. Recording any conformances to these protocols would be invalid for +noncopyable types. + +But, an exception is made where noncopyable types can conform to `Sendable`. +Unlike other protocols, the `Sendable` marker protocol leaves no conformance +record in the output program. Thus, there will be no ABI impact if a future +noncopyable version of the `Sendable` protocol is created. + +The big benefit of allowing `Sendable` conformances is that noncopyable types +are compatible with concurrency. Keep in mind that despite their ability to +conform to `Sendable`, noncopyable structs and enums are still only a subtype +of themselves. That means when the noncopyable type conforms to `Sendable`, you +still cannot convert it to `any Sendable`, because copying that existential +would copy its underlying value: + +```swift +extension FileDescriptor: Sendable {} // OK + +struct RefHolder: ~Copyable, Sendable { + var ref: Ref // ERROR: stored property 'ref' of 'Sendable'-conforming struct 'RefHolder' has non-sendable type 'Ref' +} + +func openAsync(_ path: String) async throws -> FileDescriptor {/* ... */} +func sendToSpace(_ s: some Sendable) {/* ... */} + +@MainActor func example() async throws { + // OK. FileDescriptor can cross actors because it is Sendable + var fd: FileDescriptor = try await openAsync("/dev/null") + + // ERROR: noncopyable types cannot be conditionally cast + // WARNING: cast from 'FileDescriptor' to unrelated type 'any Sendable' always fails + if let sendy: Sendable = fd as? Sendable { + + // ERROR: noncopyable types cannot be conditionally cast + // WARNING: cast from 'any Sendable' to unrelated type 'FileDescriptor' always fails + fd = sendy as! FileDescriptor + } + + // ERROR: noncopyable type 'FileDescriptor' cannot be used with generics + sendToSpace(fd) +} +``` + +#### Working around the generics restrictions + +Since a good portion of Swift's standard library rely on generics, there are a +a number of common types and functions that will not work with today's +noncopyable types: + +```swift +// ERROR: Cannot use noncopyable type FileDescriptor in generic type Optional +let x = Optional(FileDescriptor(open("/etc/passwd", O_RDONLY))) + +// ERROR: Cannot use noncopyable type FileDescriptor in generic type Array +let fds: [FileDescriptor] = [] + +// ERROR: Cannot use noncopyable type FileDescriptor in generic type Any +print(FileDescriptor(-1)) + +// ERROR: Noncopyable struct SocketEvent cannot conform to Error +enum SocketEvent: ~Copyable, Error { + case requestedDisconnect(SocketPair) +} +``` + +For example, the `print` function expects to be able to convert its argument to +`Any`, which is a copyable value. Internally, it also relies on either +reflection or conformance to `CustomStringConvertible`. Since a noncopyable type +can't do any of those, a suggested workaround is to explicitly define a +conversion to `String`: + +```swift +extension FileDescriptor /*: CustomStringConvertible */ { + var description: String { + return "file descriptor #\(fd)" + } +} + +let fd = FileDescriptor(-1) +print(fd.description) +``` + +A more general kind of workaround to mix generics and noncopyable types +is to wrap the value in an ordinary class instance, which itself can participate +in generics. To transfer the noncopyable value in or out of the wrapper class +instance, using `Optional` for the class's field would be +ideal. But until that is supported, a concrete noncopyable enum can represent +the case where the value of interest was taken out of the instance: + +```swift +enum MaybeFileDescriptor: ~Copyable { + case some(FileDescriptor) + case none + + // Returns this MaybeFileDescriptor by consuming it + // and leaving .none in its place. + mutating func take() -> MaybeFileDescriptor { + let old = self // consume self + self = .none // reinitialize self + return old + } +} + +class WrappedFile { + var file: MaybeFileDescriptor + + enum Err: Error { case noFile } + + init(_ fd: consuming FileDescriptor) { + file = .some(fd) + } + + func consume() throws -> FileDescriptor { + if case let .some(fd) = file.take() { + return fd + } + throw Err.noFile + } +} + +func example(_ fd1: consuming FileDescriptor, + _ fd2: consuming FileDescriptor) -> [WrappedFile] { + // create an array of descriptors + return [WrappedFile(fd1), WrappedFile(fd2)] +} +``` + +All of this boilerplate melts away once noncopyable types support generics. +Even before then, one major improvement would be to eliminate the need to define +types like `MaybeFileDescriptor` through a noncopyable `Optional` +(see Future Directions). + + +### Using noncopyable values + +As the name suggests, values of noncopyable type cannot be copied, a major break +from most other types in Swift. Many operations are currently defined as +working as pass-by-value, and use copying as an implementation technique +to give that semantics, but these operations now need to be defined more +precisely in terms of how they *borrow* or *consume* their operands in order to +define their effects on values that cannot be copied. + +We use the term **consume** to refer to an operation +that invalidates the value that it operates on. It may do this by directly +destroying the value, freeing its resources such as memory and file handles, +or forwarding ownership of the value to yet another owner who takes +responsibility for keeping it alive. Performing a consuming operation on +a noncopyable value generally requires having ownership of the value to begin +with, and invalidates the value the operation was performed on after it is +completed. + +We use the term **borrow** to refer to +a shared borrow of a single instance of a value; the operation that borrows +the value allows other operations to borrow the same value simultaneously, and +it does not take ownership of the value away from its current owner. This +generally means that borrowers are not allowed to mutate the value, since doing +so would invalidate the value as seen by the owner or other simultaneous +borrowers. Borrowers also cannot *consume* the value. They can, however, +initiate arbitrarily many additional borrowing operations on all or part of +the value they borrow. + +Both of these conventions stand in contrast to **mutating** (or **inout**) +operations, which take an *exclusive* borrow of their operands. The behavior +of mutating operations on noncopyable values is much the same as `inout` +parameters of copyable type today, which are already subject to the +"law of exclusivity". A mutating operation has exclusive access to its operand +for the duration of the operation, allowing it to freely mutate the value +without concern for aliasing or data races, since not even the owner may +access the value simultaneously. A mutating operation may pass its operand +to another mutating operation, but transfers exclusivity to that other operation +until it completes. A mutating operation may also pass its operand to +any number of borrowing operations, but cannot assume exclusivity while those +borrows are enacted; when the borrowing operations complete, the mutating +operation may assume exclusivity again. Unlike having true ownership of a +value, mutating operations give ownership back to the owner at the end of an +operation. A mutating operation therefore may consume the current value of its +operand, but if it does, it must replace it with a new value before completing. + +For copyable types, the distinction between borrowing and consuming operations +is largely hidden from the programmer, since Swift will implicitly insert +copies as needed to maintain the apparent value semantics of operations; a +consuming operation can be turned into a borrowing one by copying the value and +giving the operation the copy to consume, allowing the program to continue +using the original. This of course becomes impossible for values that cannot +be copied, forcing the distinction. + +Many code patterns that are allowed for copyable types also become errors for +noncopyable values because they would lead to conflicting uses of the same +value, without the ability to insert copies to avoid the conflict. For example, +a copyable value can normally be passed as an argument to the same function +multiple times, even to a `borrowing` and `consuming` parameter of the same +call, and the compiler will copy as necessary to make all of the function's +parameters valid according to their ownership specifiers: + +```swift +func borrow(_: borrowing Value, and _: borrowing Value) {} +func consume(_: consuming Value, butBorrow _: borrowing Value) {} +let x = Value() +borrow(x, and: x) // this is fine, multiple borrows can share +consume(x, butBorrow: x) // also fine, we'll copy x to let a copy be consumed + // while the other is borrowed +``` + +By contrast, a noncopyable value *must* be passed by borrow or consumed, +without copying. This makes the second call above impossible for a noncopyable +`x`, since attempting to consume `x` would end the binding's lifetime while +it also needs to be borrowed: + +```swift +func borrow(_: borrowing FileDescriptor, and _: borrowing FileDescriptor) {} +func consume(_: consuming FileDescriptor, butBorrow _: borrowing FileDescriptor) {} +let x = FileDescriptor() +borrow(x, and: x) // still OK to borrow multiple times +consume(x, butBorrow: x) // ERROR: consuming use of `x` would end its lifetime + // while being borrowed +``` + +A similar effect happens when `inout` parameters take noncopyable arguments. +Swift will copy the value of a variable if it is passed both by value and +`inout`, so that the by-value parameter receives a copy of the current value +while leaving the original binding available for the `inout` parameter to +exclusively access: + +```swift +func update(_: inout Value, butBorrow _: borrow Value) {} +func update(_: inout Value, butConsume _: consume Value) {} +var x = Value() +update(&x, butBorrow: x) // this is fine, we'll copy `x` in the second parameter +update(&x, butConsume: x) // also fine, we'll also copy +``` + +But again, for a noncopyable value, this implicit copy is impossible, so +these sorts of calls become exclusivity errors: + +```swift +func update(_: inout FileDescriptor, butBorrow _: borrow FileDescriptor) {} +func update(_: inout FileDescriptor, butConsume _: consume FileDescriptor) {} + +var y = FileDescriptor() +update(&y, butBorrow: y) // ERROR: cannot borrow `y` while exclusively accessed +update(&y, butConsume: y) // ERROR: cannot consume `y` while exclusively accessed +``` + +The following sections attempt to classify existing language operations +according to what ownership semantics they have when performed on noncopyable +values. + +### Consuming operations + +The following operations are consuming: + +- assigning a value to a new `let` or `var` binding, or setting an existing + variable or property to the binding: + + ```swift + let x = FileDescriptor() + let y = x + use(x) // ERROR: x consumed by assignment to `y` + ``` + + ```swift + var y = FileDescriptor() + let x = FileDescriptor() + y = x + use(x) // ERROR: x consumed by assignment to `y` + ``` + + ```swift + class C { + var property = FileDescriptor() + } + let c = C() + let x = FileDescriptor() + c.property = x + use(x) // ERROR: x consumed by assignment to `c.property` + ``` + + The one exception is assigning to the "black hole" `_ = x`, which is + a borrowing operation, as noted below. This allows for the + `_ = x` idiom to still be used to prevent warnings about a borrowed + binding that is otherwise unused. + +- passing an argument to a `consuming` parameter of a function: + + ```swift + func consume(_: consuming FileDescriptor) {} + let x1 = FileDescriptor() + consume(x1) + use(x1) // ERROR: x1 consumed by call to `consume` + ``` + +- passing an argument to an `init` parameter that is not explicitly + `borrowing`: + + ```swift + struct S: ~Copyable { + var x: FileDescriptor, y: Int + } + let x = FileDescriptor() + let s = S(x: x, y: 219) + use(x) // ERROR: x consumed by `init` of struct `S` + ``` + +- invoking a `consuming` method on a value, or accessing a property of the + value through a `consuming get` or `consuming set` accessor: + + ```swift + extension FileDescriptor { + consuming func consume() {} + } + let x = FileDescriptor() + x.consume() + use(x) // ERROR: x consumed by method `consume` + ``` + +- explicitly consuming a value with the `consume` operator: + + ```swift + let x = FileDescriptor() + _ = consume x + use(x) // ERROR: x consumed by explicit `consume` + ``` + +- `return`-ing a value; + +- pattern-matching a value with `switch`, `if let`, or `if case`: + + ```swift + let x: Optional = getValue() + if let y = consume x { ... } + use(x) // ERROR: x consumed by `if let` + + enum FileDescriptorOrBuffer: ~Copyable { + case file(FileDescriptor) + case buffer(String) + } + + let x = FileDescriptorOrBuffer.file(FileDescriptor()) + + switch consume x { + case .file(let f): + break + case .buffer(let b): + break + } + + use(x) // ERROR: x consumed by `switch` + ``` + + In order to allow for borrowing pattern matching to potentially become + the default later, when it's supported, the operand to `switch` or + the right-hand side of a `case` condition in an `if` or `while` must + use the `consume` operator in order to indicate that it is consumed. + We may want `switch x` to borrow by default in the future. + +- iterating a `Sequence` with a `for` loop: + + ```swift + let xs = [1, 2, 3] + for x in consume xs {} + use(xs) // ERROR: xs consumed by `for` loop + ``` + +(Although noncopyable types are not currently allowed to conform to +protocols, preventing them from implementing the `Sequence` protocol, +and cannot be used as generic parameters, preventing the formation of +`Optional` noncopyable types, these last two cases are listed for completeness, +since they would affect the behavior of other language features that +suppress implicit copying when applied to copyable types.) + +The `consume` operator can always transfer ownership of its operand when the +`consume` expression is itself the operand of a consuming operation. + +Consuming is flow-sensitive, so if one branch of an `if` or other control flow +consumes a noncopyable value, then other branches where the value +is not consumed may continue using it: + +```swift +let x = FileDescriptor() +guard let condition = getCondition() else { + consume(x) + return +} +// We can continue using x here, since only the exit branch of the guard +// consumed it +use(x) +``` + +For the purposes of the following discussion, a closure is considered nonescaping +in the following cases: + +- if the closure literal appears as an argument to a function parameter of + non-`@escaping` function type, or +- if the closure literal is assigned to a local `let` variable, that does not + itself get captured by an escaping closure. + +These cases correspond to the cases where a closure is allowed to capture an +`inout` parameter from its surrounding scope, before this proposal. + +### Borrowing operations + +The following operations are borrowing: + +- Passing an argument to a `func` or `subscript` parameter that does not + have an ownership modifier, or an argument to any `func`, `subscript`, or + `init` parameter which is explicitly marked `borrow`. The + argument is borrowed for the duration of the callee's execution. +- Borrowing a stored property of a struct or tuple borrows the struct or tuple + for the duration of the access to the stored property. This means that one + field of a struct cannot be borrowed while another is being mutated, as in + `call(struc.fieldA, &struc.fieldB)`. Allowing for fine-grained subelement + borrows in some circumstances is discussed as a Future Direction below. +- A stored property of a class may be borrowed using a dynamic exclusivity + check, to assert that there are no aliasing mutations attempted during the + borrow, as discussed under "Noncopyable stored properties in classes" below. +- Invoking a `borrowing` method on a value, or a method which is not annotated + as any of `borrowing`, `consuming` or `mutating`, borrows the `self` parameter + for the duration of the callee's execution. +- Accessing a computed property or subscript through `borrowing` or + `nonmutating` getter or setter borrows the `self` parameter for the duration + of the accessor's execution. +- Capturing an immutable local binding into a nonescaping closure borrows the + binding for the duration of the callee that receives the nonescaping closure. +- Assigning into the "black hole" `_ = x` borrows the right-hand side of the + assignment. + +### Mutating operations + +The following operations are mutating uses: + +- Passing an argument to a `func` parameter that is `inout`. The argument is + exclusively accessed for the duration of the call. +- Projecting a stored property of a struct for mutation is a mutating use of + the entire struct. +- A stored property of a class may be mutated using a dynamic exclusivity + check, to assert that there are no aliasing mutations, as happens today. + For noncopyable properties, the assertion also enforces that no borrows + are attempted during the mutation, as discussed under "Noncopyable stored + properties in classes" below. +- Invoking a `mutating` method on a value is a mutating use of the `self` + parameter for the duration of the callee's execution. +- Accessing a computed property or subscript through a `mutating` getter and/or + setter is a mutating use of `self` for the duration of the accessor's + execution. +- Capturing a mutable local binding into a nonescaping closure is a mutating + use of the binding for the duration of the callee that receives the + nonescaping closure. + +### Declaring functions and methods with noncopyable parameters + +When noncopyable types are used as function parameters, the ownership +convention becomes a much more important part of the API contract. +As such, when a function parameter is declared with a noncopyable type, it +**must** declare whether the parameter uses the `borrowing`, `consuming`, or +`inout` convention: + +```swift +// Redirect a file descriptor +// Require exclusive access to the FileDescriptor to replace it +func redirect(_ file: inout FileDescriptor, to otherFile: borrowing FileDescriptor) { + dup2(otherFile.fd, file.fd) +} + +// Write to a file descriptor +// Only needs shared access +func write(_ data: [UInt8], to file: borrowing FileDescriptor) { + data.withUnsafeBytes { + write(file.fd, $0.baseAddress, $0.count) + } +} + +// Close a file descriptor +// Consumes the file descriptor +func close(file: consuming FileDescriptor) { + close(file.fd) +} +``` + +Methods of the noncopyable type are considered to be `borrowing` unless +declared `mutating` or `consuming`: + +```swift +extension FileDescriptor { + mutating func replace(with otherFile: borrowing FileDescriptor) { + dup2(otherFile.fd, self.fd) + } + + // borrowing by default + func write(_ data: [UInt8]) { + data.withUnsafeBytes { + write(file.fd, $0.baseAddress, $0.count) + } + } + + consuming func close() { + close(fd) + } +} +``` + +Static casts or coercions of function types that change the ownership modifier +of a noncopyable parameter are currently invalid. One reason is that it is +impossible to convert a function with a noncopyable `consuming` parameter, into +one where that parameter is `borrowed`, without inducing a copy of the borrowed +parameter. See Future Directions for details. + +### Declaring properties of noncopyable type + +A class or noncopyable struct may declare stored `let` or `var` properties of +noncopyable type. A noncopyable `let` stored property may only be borrowed, +whereas a `var` stored property may be both borrowed and mutated. Stored +properties cannot generally be consumed because doing so would leave the +containing aggregate in an invalid state. + +Any type may also declare computed properties of noncopyable type. The `get` +accessor returns an owned value that the caller may consume, like a function +would. The `set` accessor receives its `newValue` as a `consuming` parameter, +so the setter may consume the parameter value to update the containing +aggregate. + +Accessors may use the `consuming` and `borrowing` declaration modifiers to +affect the ownership of `self` while the accessor executes. `consuming get` +is particularly useful as a way of forwarding ownership of part of an aggregate, +such as to take ownership away from a wrapper type: + +```swift +struct FileDescriptorWrapper: ~Copyable { + private var _value: FileDescriptor + + var value: FileDescriptor { + consuming get { return _value } + } +} +``` + +However, a `consuming get` cannot be paired with a setter when the containing +type is `~Copyable`, because invoking the getter consumes the aggregate, +leaving nothing to write a modified value back to. + +Because getters return owned values, non-`consuming` getters generally cannot +be used to wrap noncopyable stored properties, since doing so would require +copying the value out of the aggregate: + +```swift +class File { + private var _descriptor: FileDescriptor + + var descriptor: FileDescriptor { + return _descriptor // ERROR: attempt to copy `_descriptor` + } +} +``` + +These limitations could be addressed in the future by exposing the ability for +computed properties to also provide "read" and "modify" coroutines, which would +have the ability to yield borrowing or mutating access to properties without +copying them. + +### Using stored properties and enum cases of noncopyable type + +When classes or noncopyable types contain members that are of noncopyable +type, then the container is the unique owner of the member value. Outside of +the type's definition, client code cannot perform consuming operations on +the value, since it would need to take away the container's ownership to do +so: + +```swift +struct Inner: ~Copyable {} + +struct Outer: ~Copyable { + var inner = Inner() +} + +let outer = Outer() +let i = outer.inner // ERROR: can't take `inner` away from `outer` +``` + +However, when code has the ability to mutate the member, it may freely modify, +reassign, or replace the value in the field: + +```swift +var outer = Outer() +let newInner = Inner() +// OK, transfers ownership of `newInner` to `outer`, destroying its previous +// value +outer.inner = newInner +``` + +Note that, as currently defined, `switch` to pattern-match an `enum` is a +consuming operation, so it can only be performed inside `consuming` methods +on the type's original definition: + +```swift +enum OuterEnum: ~Copyable { + case inner(Inner) + case file(FileDescriptor) +} + +// Error, can't partially consume a value outside of its definition +let enum = OuterEnum.inner(Inner()) +switch enum { +case .inner(let inner): + break +default: + break +} +``` + +Being able to borrow in pattern matches would address this shortcoming. + +### Noncopyable stored properties in classes + +Since objects may have any number of simultaneous references, Swift uses +dynamic exclusivity checking to prevent simultaneous writes of the same +stored property. This dynamic checking extends to borrows of noncopyable +stored properties; the compiler will attempt to diagnose obvious borrowing +failures, as it will for local variables and value types, but a runtime error +will occur if an uncaught exclusivity error occurs, such as an attempt to mutate +an object's stored property while it is being borrowed: + +```swift +class Foo { + var fd: FileDescriptor + + init(fd: FileDescriptor) { self.fd = fd } +} + +func update(_: inout FileDescriptor, butBorrow _: borrow FileDescriptor) {} + +func updateFoo(_ a: Foo, butBorrowFoo b: Foo) { + update(&a.fd, butBorrow: b.fd) +} + +let foo = Foo(fd: FileDescriptor()) + +// Will trap at runtime when foo.fd is borrowed and mutated at the same time +updateFoo(foo, butBorrowFoo: foo) +``` + +`let` properties do not allow mutating accesses, and this continues to hold for +noncopyable types. The value of a `let` property in a class therefore does not +need dynamic checking, even if the value is noncopyable; the value behaves as +if it is always borrowed, since there may potentially be a borrow through +some reference to the object at any point in the program. Such values can +thus never be consumed or mutated. + +The dynamic borrow state of properties is tracked independently for every +stored property in the class, so it is safe to mutate one property while other +properties of the same object are also being mutated or borrowed: + +```swift +class SocketTriple { + var in, middle, out: FileDescriptor +} + +func update(_: inout FileDescriptor, and _: inout FileDescriptor, + whileBorrowing _: borrowing FileDescriptor) {} + +// This is OK +let object = SocketTriple(...) +update(&object.in, and: &object.out, whileBorrowing: object.middle) +``` + +This dynamic tracking, however, cannot track accesses at finer resolution +than properties, so in circumstances where we might otherwise eventually be +able to support independent borrowing of fields in structs, tuples, and enums, +that support will not extend to fields within class properties, since the +entire property must be in the borrowing or mutating state. + +Dynamic borrowing or mutating accesses require that the enclosing object be +kept alive for the duration of the assertion of the access. Normally, this +is transparent to the developer, as the compiler will keep a copy of a +reference to the object retained while these accesses occur. However, if +we introduce noncopyable bindings to class references, such as [the `borrow` +and `inout` bindings](https://forums.swift.org/t/pitch-borrow-and-inout-declaration-keywords/62366) +currently being pitched, this would manifest as a borrow of the noncopyable +reference, preventing mutation or consumption of the reference during +dynamically-asserted accesses to its properties: + +```swift +class SocketTriple { + var in, middle, out: FileDescriptor +} + +func borrow(_: borrowing FileDescriptor, + whileReplacingObject _: inout SocketTriple) {} + +var object = SocketTriple(...) + +// This is OK, since ARC will keep a copy of the `object` reference retained +// while `object.in` is borrowed +borrow(object.in, whileReplacingObject: &object) + +inout objectAlias = &object + +// This is an error, since we aren't allowed to implicitly copy through +// an `inout` binding, and replacing `objectAlias` without keeping a copy +// retained might invalidate the object while we're accessing it. +borrow(objectAlias.in, whileReplacingObject: &objectAlias) +``` + +### Noncopyable variables captured by escaping closures + +Nonescaping closures have scoped lifetimes, so they can borrow their captures, +as noted in the "borrowing operations" and "consuming operations" sections +above. Escaping closures, on the other hand, have indefinite lifetimes, since +they can be copied and passed around arbitrarily, and multiple escaping closures +can capture and access the same local variables alongside the local context +from which those captures were taken. Variables captured by escaping closures +thus behave like class properties; immutable captures are treated as always +borrowed both inside the closure body and in the capture's original context. + +```swift +func escape(_: @escaping () -> ()) {...} + +func borrow(_: borrowing FileDescriptor) {} +func consume(_: consuming FileDescriptor) {} + +func foo() { + let x = FileDescriptor() + + // ERROR: cannot consume variable before it's been captured + consume(x) + + escape { + borrow(x) // OK + consume(x) // ERROR: cannot consume captured variable + } + + // OK + borrow(x) + + // ERROR: cannot consume variable after it's been captured by an escaping + // closure + consume(x) +} +``` + +Mutable captures are subject to dynamic exclusivity checking like class +properties are, and similarly cannot be consumed and reinitialized. When +a closure escapes, the compiler isn't able to statically know when the closure +is invoked, and it may even be invoked multiple overlapping times, or +simultaneously on different threads if the closure is `@Sendable`, so the +captures must always remain in a valid state for memory safety, and exclusivity +of mutations can only be enforced dynamically. + +```swift +var escapedClosure: (@escaping (inout FileDescriptor) -> ())? + +func foo() { + var x = FileDescriptor() + + // ERROR: cannot consume variable before it's been captured. + // (We could potentially support local consumption before the variable + // capture occurs as a future direction.) + consume(x) + x = FileDescriptor() + + escapedClosure = { _ in borrow(x) } + + // Runtime error when exclusive access to `x` dynamically conflicts + // with attempted borrow of `x` during `escapedClosure`'s execution + escapedClosure!(&x) +} +``` + +### Deinitializers + +A noncopyable struct or enum may declare a `deinit`, which will run +implicitly when the lifetime of the value ends (unless explicitly suppressed +with `discard` as explained below): + +```swift +struct File: ~Copyable { + var descriptor: Int32 + + func write(_ values: S) { /*..*/ } + + consuming func close() { + print("closing file") + } + + deinit { + print("deinitializing file") + closeFile(rawDescriptor: descriptor) + } +} +``` + +Like a class `deinit`, a struct or enum `deinit` may not propagate any +uncaught errors. Within the body of the `deinit`, `self` behaves as in +a `borrowing` method; it may not be modified or consumed inside the +`deinit`. (Allowing for mutation and partial invalidation inside a +`deinit` is explored as a future direction.) + +A value's lifetime ends, and its `deinit` runs if present, in the following +circumstances: + +- For a local `var` or `let` binding, or `consuming` function parameter, that is + not itself consumed, `deinit` runs at the end of the binding's lexical + scope. If, on the other hand, the binding is consumed, then responsibility + for deinitialization gets forwarded to the consumer (which may in turn forward + it somewhere else). As explained later, a `_ = consume` operator with no + destination immediately runs the `deinit`. + + ```swift + do { + let file = File(descriptor: 42) + file.close() // consuming use + // file's deinit runs inside `close` + print("done writing") + } + // Output: + // closing file + // deinitializing file + // done writing + + do { + let file = File(descriptor: 42) + file.write([1,2,3]) // borrowing use + print("done writing") + // file's deinit runs here + } + // Output: + // done writing + // deinitializing file + ``` + + If a noncopyable value is conditionally consumed, then the deinitializer + runs as late as possible on any nonconsumed paths: + + ```swift + let condition = false + do { + let file = File(descriptor: 42) + file.write([1,2,3]) // borrowing use + if condition { + file.close() + } else { + print("not closed") + // file's deinit runs here + } + print("done writing") + } + // Output: + // not closed + // deinitializing file + // done writing + ``` + +- When a struct, enum, or class contains a member of noncopyable type, the member is destroyed, and its deinit is +run, after the container's deinit runs. For example: + +```swift +struct Inner: ~Copyable { + deinit { print("destroying inner") } +} + +struct Outer: ~Copyable { + var inner = Inner() + deinit { print("destroying outer") } +} + +do { + _ = Outer() +} +``` + +will print: +``` +destroying outer +destroying inner +``` + +### Suppressing `deinit` in a `consuming` method + +It is often useful for noncopyable types to provide alternative ways to consume +the resource represented by the value besides `deinit`. However, +under normal circumstances, a `consuming` method will still invoke the type's +`deinit` after the last use of `self`, which is undesirable when the method's +own logic already invalidates the value: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + deinit { + close(fd) + } + + consuming func close() { + close(fd) + + // The lifetime of `self` ends here, triggering `deinit` (and another call to `close`)! + } +} +``` + +In the above example, the double-close could be avoided by having the +`close()` method do nothing on its own and just allow the `deinit` to +implicitly run. However, we may want the method to have different behavior +from the deinit; for example, it could raise an error (which a normal `deinit` +is unable to do) if the `close` system call triggers an OS error : + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + consuming func close() throws { + // POSIX close may raise an error (which leaves the file descriptor in an + // unspecified state, so we can't really try to close it again, but the + // error may nonetheless indicate a condition worth handling) + if close(fd) != 0 { + throw CloseError(errno) + } + + // We don't want to trigger another close here! + } +} +``` + +or it could be useful to take manual control of the file descriptor back from +the type, such as to pass to a C API that will take care of closing it: + +```swift +struct FileDescriptor: ~Copyable { + // Take ownership of the C file descriptor away from this type, + // returning the file descriptor without closing it + consuming func take() -> Int32 { + return fd + + // We don't want to trigger close here! + } +} +``` + +We propose to introduce a special operator, `discard self`, which ends the +lifetime of `self` without running its `deinit`: + +```swift +struct FileDescriptor: ~Copyable { + // Take ownership of the C file descriptor away from this type, + // returning the file descriptor without closing it + consuming func take() -> Int32 { + let fd = self.fd + discard self + return fd + } +} +``` + +`discard self` can only be applied to `self`, in a consuming method +defined in the same file as the type's original definition. (This is in +contrast to Rust's similar special function, +[`mem::forget`](https://doc.rust-lang.org/std/mem/fn.forget.html), which is a +standalone function that can be applied to any value, anywhere. Although the +Rust documentation notes that this operation is "safe" on the principle that +destructors may not run at all, due to reference cycles, process termination, +etc., in practice the ability to forget arbitrary values creates semantic +issues for many Rust APIs, particularly when there are destructors on types +with lifetime dependence on each other like `Mutex` and `LockGuard`. As such, +we think it is safer to restrict the ability to suppress the standard `deinit` +for a value to the core API of its type. We can relax this restriction if +experience shows a need to.) + +For the extent of this proposal, we also propose that `discard self` can only +be applied in types whose components include no reference-counted, generic, +or existential fields, nor do they include any types that transitively include +any fields of those types or that have `deinit`s defined of their own. (Such +a type might be called "POD" or "trivial" following C++ terminology). We explore +lifting this restriction as a future direction. + + +Even with the ability to `discard self`, care would still need be taken when +writing destructive operations to avoid triggering the deinit on alternative +exit paths, such as early `return`s, `throw`s, or implicit propagation of +errors from `try` operations. For instance, if we write: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + consuming func close() throws { + // POSIX close may raise an error (which still invalidates the + // file descriptor, but may indicate a condition worth handling) + if close(fd) != 0 { + throw CloseError(errno) + // !!! Oops, we didn't suppress deinit on this path, so we'll double close! + } + + // We don't need to deinit self anymore + discard self + } +} +``` + +then the `throw` path exits the method without `discard`, and +`deinit` will still execute if an error occurs. To avoid this mistake, we +propose that if any path through a method uses `discard self`, then +**every** path must choose either to `discard` or to explicitly `consume self`, +which triggers the standard `deinit`. This will make the above code an error, +alerting that the code should be rewritten to ensure `discard self` +always executes: + +```swift +struct FileDescriptor: ~Copyable { + private var fd: Int32 + + consuming func close() throws { + // Save the file descriptor and give up ownership of it + let fd = self.fd + discard self + + // We can now use `fd` below without worrying about `deinit`: + + // POSIX close may raise an error (which still invalidates the + // file descriptor, but may indicate a condition worth handling) + if close(fd) != 0 { + throw CloseError(errno) + } + } +} +``` + +The [consume operator](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0377-parameter-ownership-modifiers.md) +must be used to explicitly end the value's lifetime using its `deinit` if +`discard` is used to conditionally destroy the value on other paths +through the method. + +```swift +struct MemoryBuffer: ~Copyable { + private var address: UnsafeRawPointer + + init(size: Int) throws { + guard let address = malloc(size) else { + throw MallocError() + } + self.address = address + } + + deinit { + free(address) + } + + consuming func takeOwnership(if condition: Bool) -> UnsafeRawPointer? { + if condition { + // Save the memory buffer and give it to the caller, who + // is promising to free it when they're done. + let address = self.address + discard self + return address + } else { + // We still want to free the memory if we aren't giving it away. + _ = consume self + return nil + } + } +} +``` + +## Source compatibility + +For existing Swift code, this proposal is additive. + +## Effect on ABI stability + +### Adding or removing `Copyable` breaks ABI + +An existing copyable struct or enum cannot have its `Copyable` capability +taken away without breaking ABI, since existing clients may copy values of the +type. + +Ideally, we would allow noncopyable types to become `Copyable` without breaking +ABI; however, we cannot promise this, due to existing implementation choices we +have made in the ABI that cause the copyability of a type to have unavoidable +knock-on effects. In particular, when properties are declared in classes, +protocols, or public non-`@frozen` structs, we define the property's ABI to use +accessors even if the property is stored, with the idea that it should be +possible to change a property's implementation to change it from a stored to +computed property, or vice versa, without breaking ABI. + +The accessors used as ABI today are the traditional `get` and `set` +computed accessors, as well as a `_modify` coroutine which can optimize `inout` +operations and projections into stored properties. `_modify` and `set` are +not problematic for noncopyable types. However, `get` behaves like a +function, producing the property's value by returning it like a function would, +and returning requires *consuming* the return value to transfer it to the +caller. This is not possible for noncopyable stored properties, since the +value of the property cannot be copied in order to return a copy without +invalidating the entire containing struct or object. + +Therefore, properties of noncopyable type need a different ABI in order to +properly abstract them. In particular, instead of exposing a `get` accessor +through abstract interfaces, they must use a `_read` coroutine, which is the +read-only analog to `_modify`, allowing the implementation to yield a borrow of +the property value in-place instead of returning by value. This allows for +noncopyable stored properties to be exposed while still being abstracted enough +that they can be replaced by a computed implementation, since a `get`-based +implementation could still work underneath the `read` coroutine by evaluating +the getter, yielding a borrow of the returned value, then disposing of the +temporary value. + +As such, we cannot simply say that making a noncopyable type copyable is an +ABI-safe change, since doing so will have knock-on effects on the ABI of any +properties of the type. We could potentially provide a "born noncopyable" +attribute to indicate that a copyable type should use the noncopyable ABI +for any properties, as a way to enable the evolution into a copyable type +while preserving existing ABI. However, it also seems unlikely to us that many +types would need to evolve between being copyable or not frequently. + +### Adding, removing, or changing `deinit` in a struct or enum + +An noncopyable type that is not `@frozen` can add or remove its deinit without +affecting the type's ABI. If `@frozen`, a deinit cannot be added or removed, +but the deinit implementation may change (if the deinit is not additionally +`@inlinable`). + +### Adding noncopyable fields to classes + +A class may add fields of noncopyable type without changing ABI. + +## Effect on API resilience + +Introducing new APIs using noncopyable types is an additive change. APIs that +adopt noncopyable types have some notable restrictions on how they can further +evolve while maintaining source compatibility. + +A noncopyable type can be made copyable while generally maintaining source +compatibility. Values in client source would acquire normal ARC lifetime +semantics instead of eager-move semantics when those clients are recompiled +with the type as copyable, and that could affect the observable order of +destruction and cleanup. Since copyable value types cannot directly define +`deinit`s, being able to observe these order differences is unlikely, but not +impossible when references to classes are involved. + +A `consuming` parameter of noncopyable type can be changed into a `borrowing` +parameter without breaking source for clients (and likewise, a `consuming` +method can be made `borrowing`). Conversely, changing +a `borrowing` parameter to `consuming` may break client source. (Either direction +is an ABI breaking change.) This is because a consuming use is required to +be the final use of a noncopyable value, whereas a borrowing use may or may not +be. + +Adding or removing a `deinit` to a noncopyable type does not affect source +for clients. + +## Alternatives considered + +### Naming the attribute "move-only" + +We have frequently referred to these types as "move-only types" in various +vision documents. However, as we've evolved related proposals like the +`consume` operator and parameter modifiers, the community has drifted away +from exposing the term "move" in the language elsewhere. When explaining these +types to potential users, we've also found that the name "move-only" incorrectly +suggests that being noncopyable is a new capability of types, and that there +should be generic functions that only operate on "move-only" types, when really +the opposite is the case: all existing types in Swift today conform to +effectively an implicit "Copyable" requirement, and what this feature does is +allow types not to fulfill that requirement. When generics grow support for +move-only types, then generic functions and types that accept noncopyable +type parameters will also work with copyable types, since copyable types +are strictly more capable. This proposal prefers the term "noncopyable" to make +the relationship to an eventual `Copyable` constraint, and the fact that annotated +types lack the ability to satisfy this constraint, more explicit. + +### Spelling as a generic constraint + +It's a reasonable question why declaring a type as noncopyable isn't spelled +like a regular protocol constraint, instead of as the removal of an existing +constraint: + +```swift +struct Foo: NonCopyable {} +``` + +As noted in the previous discussion, an issue with this notation is that it +implies that `NonCopyable` is a new capability or requirement, rather than +really being the lack of a `Copyable` capability. For an example of why +this might be misleading, consider what would happen if we expand +standard library collection types to support noncopyable elements. Value types +like `Array` and `Dictionary` would become copyable only when the elements they +contain are copyable. However, we cannot write this in terms of `NonCopyable` +conditional requirements, since if we write: + +```swift +extension Dictionary: NonCopyable where Key: NonCopyable, Value: NonCopyable {} +``` + +this says that the dictionary is noncopyable only when both the key and value +are noncopyable, which is wrong because we can't copy the dictionary even if +only the keys or only the values are noncopyable. If we flip the constraint to +`Copyable`, the correct thing would fall out naturally: + +```swift +extension Dictionary: Copyable where Key: Copyable, Value: Copyable {} +``` + +However, for progressive disclosure and source compatibility reasons, we still +want the majority of types to be `Copyable` by default without making them +explicitly declare it; noncopyable types are likely to remain the exception +rather than the rule, with automatic lifetime management via ARC by the +compiler being sufficient for most code like it is today. + +### English language bikeshedding + +Some dictionaries specify that "copiable" is the standard spelling for "able to +copy", although the Oxford English Dictionary and Merriam-Webster both also +list "copyable" as an accepted alternative. We prefer the more regular "copyable" +spelling. + +## Future directions + +### Noncopyable tuples + +It should be possible for a tuple to contain noncopyable elements, rendering +the tuple noncopyable if any of its elements are. Since tuples' structure is +always known, it would be reasonable to allow for the elements within a tuple +to be independently borrowed, mutated, and consumed, as the language allows +today for the elements of a tuple to be independently mutated via `inout` +accesses. (Due to the limitations of dynamic exclusivity checking, this would +not be possible for class properties, globals, and escaping closure captures.) + +### Noncopyable `Optional` + +This proposal initiates support for noncopyable types without any support for +generics at all, which precludes their use in most standard library types, +including `Optional`. We expect the lack of `Optional` support in particular +to be extremely limiting, since `Optional` can be used to manage dynamic +consumption of noncopyable values in situations where the language's static +rules cannot soundly support consumption. For instance, the static rules above +state that a stored property of a class can never be consumed, because it is +not knowable if other references to an object exist that expect the property +to be inhabited. This could be avoided using `Optional` with `mutating` +operation that forwards ownership of the `Optional` value's payload, if any, +writing `nil` back. Eventually this could be written as an extension method +on `Optional`: + +```swift +extension Optional where Self: ~Copyable { + mutating func take() -> Wrapped { + switch self { + case .some(let wrapped): + self = nil + return wrapped + case .none: + fatalError("trying to take from an Optional that's already empty") + } + } +} + +class Foo { + var fd: FileDescriptor? + + func close() { + // We normally would not be able to close `fd` except via the + // object's `deinit` destroying the stored property. But using + // `Optional` assignment, we can dynamically end the value's lifetime + // here. + fd = nil + } + + func takeFD() -> FileDescriptor { + // We normally would not be able to forward `fd`'s ownership to + // anyone else. But using + // `Optional.take`, we can dynamically end the value's lifetime + // here. + return fd.take() + } +} +``` + +Without `Optional` support, the alternative would be for every noncopyable type +to provide its own ad-hoc `nil`-like state, which would be very unfortunate, +and go against Swift's general desire to encourage structural code correctness +by making invalid states unrepresentable. Therefore, `Optional` is likely to +be worth considering as a special case for noncopyable support, ahead of full +generics support for noncopyable types. + +### Generics support for noncopyable types + +This proposal comes with an admittedly severe restriction that noncopyable types +cannot conform to protocols or be used at all as type arguments to generic +functions or types, including common standard library types like `Optional` +and `Array`. All generic parameters in Swift today carry an implicit assumption +that the type is copyable, and it is another large language design project to +integrate the concept of noncopyable types into the generics system. Full +integration will very likely also involve changes to the Swift runtime and +standard library to accommodate noncopyable types in APIs that weren't +originally designed for them, and this integration might then have backward +deployment restrictions. We believe that, even with these restrictions, +noncopyable types are a useful self-contained addition to the language for +safely and efficiently modeling unique resources, and this subset of the feature +also has the benefit of being adoptable without additional runtime requirements, +so developers can begin making use of the feature without giving up backward +compatibility with existing Swift runtime deployments. + +### Conditionally copyable types + +This proposal states that a type, including one with generic parameters, is +currently always copyable or always noncopyable. However, some types may +eventually be generic over copyable and non-copyable types, with the ability +to be copyable for some generic arguments but not all. A simple case might be +a tuple-like `Pair` struct: + +```swift +struct Pair: ~Copyable { + var first: T + var second: U +} +``` + +We will need a way to express this conditional copyability, perhaps using +conditional conformance style declarations: + +```swift +extension Pair: Copyable where T: Copyable, U: Copyable {} +``` + +### Suppressing implicitly derived conformances with `~Constraint` + +There are situations where a type's conformance to a protocol is implicitly +derived because of aspects of its declaration or usage. For instance, enums that +don't have any associated values are implicitly made `Hashable` (and, +by refinement, `Equatable`): + +```swift +enum Foo { + case a, b, c +} + +// OK to compare with `==` because `Foo` is automatically Equatable, +// through an implementation of `==` synthesized by the compiler for you. +print(Foo.a == Foo.b) +``` + +and internal structs and enums are implicitly `Sendable` if all of their +components are `Sendable`: + +```swift +struct Bar { + var x: Int, y: Int +} + +func foo() async { + let x = Bar(x: 17, y: 38) + + // OK to use x in an async task because it's implicitly Sendable + async let y = x +} +``` + +However, this isn't always desirable; an enum may want to reserve the right to +add associated values in the future that aren't `Equatable`, or a type may be +made up of `Sendable` components that represent resources that are not safe +to share across threads. There is currently no direct way to suppress these +automatically derived conformances. We propose to introduce the `~Constraint` +syntax as a way to explicitly suppress automatic derivation of a conformance +that would otherwise be performed for a declaration: + +```swift +enum Candy: ~Equatable { + case redVimes, twisslers, smickers +} + +// ERROR: `Candy` does not conform to `Equatable` +print(Candy.redVimes == Candy.twisslers) + +struct ThreadUnsafeHandle: ~Sendable { + // although this is an integer, it represents a system resource that + // can only be accessed from a specific thread, and should not be shared + // across threads + var handle: Int32 +} + +func foo(handle: ThreadUnsafeHandle) async { + // ERROR: `ThreadUnsafeHandle` is not `Sendable` + async let y = handle +} +``` + +It is important to note that `~Constraint` only avoids the implicit, automatic +derivation of conformance. It does **not** mean that the type strictly does +not conform to the protocol. Extensions may add the conformance back separately, +possibly conditionally: + +```swift +struct ResourceHandle: ~Sendable { + // although this is an integer, it represents a system resource that + // gives access to values of type `T`, which may not be thread safe + // across threads + var handle: Int32 +} + +// It is safe to share the handle when the resource type is thread safe +extension ResourceHandle: Sendable where T: Sendable {} + +// Suppress the implicit Equatable (and Hashable) derivation... +enum Candy: ~Equatable { + case redVimes, twisslers, smickers +} + +// ... and still add an Equatable conformance. +extension Candy: Equatable { + static func ==(a: Candy, b: Candy) -> Bool { + switch (a, b) { + // RedVimes are considered equal to Twisslers + case (.redVimes, .redVimes), (.twisslers, .twisslers), + (.smickers, .smickers), (.twisslers, .redVimes) + (.redVimes, .twisslers): + return true + default: + return false + } + } +} +``` + +Keep in mind that `~Constraint` is not required to suppress Swift's synthesized implementations of protocol requirements. For example, if you only want to +provide your own implementation of `==` for an enum, but are fine with Equatable +(and Hashable, etc) being derived for you, then the derivation of `Equatable` +already will use your version of `==`. + +```swift +enum Soda { + case mxPepper, drPibb, doogh + + // This is used instead of a synthesized `==` when + // implicitly deriving the Equatable conformance + static func ==(a: Soda, b: Soda) -> Bool { + switch (a, b) { + case (.doogh, .doogh): return true + case (_, .doogh), (.doogh, _): return false + default: return true + } + } +} +``` + +### Allowing `deinit` to mutate or consume `self`, while avoiding accidental recursion + +During destruction, `deinit` formally has sole ownership of `self`, so it +is possible to allow `deinit` to mutate or consume `self` as part of +deinitialization. However, inside of other `mutating` or `consuming` methods, +it's easy to inadvertently trigger implicit destruction of the value and +reenter `deinit` again: + +```swift +struct Foo: ~Copyable { + init() { ... } + + consuming func consumingHelper() { + // If a consuming method does nothing else, it will run `deinit` + } + + mutating func mutatingHelper() { + // A mutating method may consume and reassign self, indirectly triggering + // an implicit deinit + consumingHelper() + self = .init() + } + + deinit { + // mutatingHelper calls consumingHelper, which calls deinit again, leading to an infinite loop + mutatingHelper() + } +} +``` + +Since this is an easy trap to fall into, before we allow `deinit` to mutate +or consume `self`, it's worth considering whether there are any constraints we +could impose to make it less likely to get into an infinite +`deinit` loop situation when doing so. Some possibilities include: + +* We could say that the value remains immutable during `deinit`. Many types + don't need to modify their internal state for cleanup, especially if they + only store a pointer or handle to some resource. This seems overly + restrictive for other kinds of types that have direct ownership of resources, + though. +* We could say that individual *fields* of the value inside of `deinit` are + mutable and consumable, but that the value as a whole is not. This would + allow for `deinit` to individually mutate and/or forward ownership of + elements of the value, but not pass off the entire value to be mutated or + consumed (and potentially re-deinited). This would allow for `deinit`s to + implement logic that modifies or consumes part of the value, but they + wouldn't be allowed to use any methods of the type, other than maybe + `borrowing` methods, to share implementation logic with other members of the + type. +* Since `deinit` must be declared as part of the original type declaration, any + nongeneric methods that it can possibly call on the type must be defined in + the same module as the `deinit`, so we could potentially do some local + analysis of those methods. We could raise a warning or error if a method + called from the deinit either visibly contains any implicit deinit calls + itself, or cannot be analyzed because it's generic, from a protocol + extension, etc. +* We could do nothing and leave it in developers' hands to understand why + deinit loops happen when they do. + +### Finer-grained destructuring in `consuming` methods and `deinit` + +As currently specified, noncopyable types are (outside of `init` implementations) +always either fully initialized or fully destroyed, without any support +for incremental destruction even inside of `consuming` methods or deinits. A +`deinit` may modify, but not invalidate, `self`, and a `consuming` method may +`discard self`, forward ownership of all of `self`, or destroy `self`, but +cannot yet partially consume parts of `self`. This would be particularly useful +for types that contain other noncopyable types, which may want to relinquish +ownership of some or all of the resources owned by those members. In the +current proposal, this isn't possible without allowing for an intermediate +invalid state: + +```swift +struct SocketPair: ~Copyable { + let input, output: FileDescriptor + + // Gives up ownership of the output end, closing the input end + consuming func takeOutput() -> FileDescriptor { + // We would like to do something like this, taking ownership of + // `self.output` while leaving `self.input` to be destroyed. + // However, we can't do this without being able to either copy + // `self.output` or partially invalidate `self`. + return self.output + } +} +``` + +Analogously to how `init` implementations use a "definite initialization" +pass to allow the value to initialized field-by-field, we can implement the +inverse dataflow pass to allow `deinit` implementations to partially +invalidate `self`. This analysis would also enable `consuming` methods to +partially invalidate `self` in cases where either the type has no `deinit` or, +as discussed in the following section, `discard self` is used to disable the +`deinit` in cases when the value is partially invalidated. + +### Generalizing `discard self` for types with component cleanups + +The current proposal limits the use of `discard self` to types that don't have +any fields that require additional cleanup, meaning that it cannot be used in +a type that has class, generic, existential, or other noncopyable type fields. +Allowing this would be an obvious generalization; however, allowing it requires +answering some design questions: + +- When `self` is discarded, are its fields still destroyed? +- Is access to `self`'s fields still allowed after `discard self`? In other + words, does `discard self` immediately consume all of `self`, running + the cleanups for its elements at the point where the `discard` is executed, + or does it only disable the `deinit` on `self`, allowing the fields to + still be individually borrowed, mutated, and/or consumed, and leaving them + to be cleaned up when their individual lifetimes end? + +Although Rust's `mem::forget` completely leaks its operand, including its fields, +the authors of this proposal generally believe that is undesirable, so we expect +that `discard self` should only disable the type's own `deinit` while still +leaving the components of `self` to be cleaned up. + +The choice of what effect `discard` has on the lifetime of the fields affects +the observed order in which field deinits occurs, but also affects how code +would be expressed that performs destructuring or partial invalidation: + +```swift +struct SocketPair: ~Copyable { + let input, output: FileDescriptor + + deinit { ... } + + enum End { case input, output } + + // Give up ownership of one end and closes the other end + consuming func takeOneEnd(which: End) -> FileDescriptor { + // If a consuming method could partially invalidate self, would it do it + // like this... +#if discard_immediately_consumes_whats_left_of_self + switch which { + case .input: + // Move out the field we want + let result = self.input + // Destroy the rest of self + discard self + return result + + case .output: + let result = self.output + discard self + return result + } + + // ...or like this +#elseif discard_only_disables_deinit + // Disable deinit on self, which subsequently allows individual consumption + // of its fields + discard self + + switch which { + case .input: + return self.input + case .output: + return self.output + } +#endif + } +} +``` + +### `read` and `modify` accessor coroutines for computed properties + +The current computed property model allows for properties to provide a getter, +which returns the value of the property on read to the caller as an owned value, +and optionally a setter, which receives the `newValue` of the property as +a parameter with which to update the containing type's state. This is +sometimes inefficient for value types, since the get/set pattern requires +returning a copy, modifying the copy, then passing the copy back to the setter +in order to model an in-place update, but it also limits what computed +properties can express for noncopyable types. Because a getter has to return +by value, it cannot pass along the value of a stored noncopyable property +without also destroying the enclosing aggregate, so `get`/`set` cannot be used +to wrap logic around access to a stored noncopyable property. + +The Swift stable ABI for properties internally uses **accessor coroutines** +to allow for efficient access to stored properties, while still providing +abstraction that allows library evolution to change stored properties into +computed and back. These coroutines **yield** access to a value in-place for +borrowing or mutating, instead of passing copies of values back and forth. +We can expose the ability for code to implement these coroutines directly, +which is a good optimization for copyable value types, but also allows for +more expressivity with noncopyable properties. + +### Static casts of functions with ownership modifiers + +The rule for casting function values via `as` or some other static, implicit +coercion is that a noncopyable parameter's ownership modifier must remain the +same. But there are some cases where static conversions of functions +with noncopyable parameters are safe. It's not safe in general to do any dynamic +casts of function values, so `as?` and `as!` are excluded. + +One reason behind the currently restrictive rule for static casts is a matter of +scope for this proposal. There may be a broader demand to support such casts +even for copyable types. For example, it should be safe to allow a cast to +change a `borrowing` parameter into one that is `inout`, as it only adds a +capability (mutation) that is not actually used by the underlying function: +```swift +// This could be possible, but currently is not. +{ (x: borrowing SomeType) in () } as (inout SomeType) -> () +``` +The second reason is that some casts are _only_ valid for copyable types. +In particular, a cast that changes a `consuming` parameter into one that is +`borrowing` is only valid for copyable types, because a copy of the borrowed +value is required to provide a non-borrowed value to the underlying function. +```swift +// String is copyable, so both are OK and currently permitted. +{ (x: borrowing String) in () } as (consuming String) -> () +{ (x: consuming String) in () } as (borrowing String) -> () +// FileDescriptor is noncopyable, so it cannot go from consuming to borrowing: +{ (x: consuming FileDescriptor) in () } as (borrowing String) -> () +// but the reverse could be permitted in the future: +{ (x: borrowing FileDescriptor) in () } as (consuming String) -> () +``` + +## Revision history + +This revision makes the following changes from the [second reviewed revision](https://github.com/swiftlang/swift-evolution/blob/a9e21e3a4eb9526f998915c6554c7c72e5885a91/proposals/0390-noncopyable-structs-and-enums.md) +in response to Language Steering Group review and implementation experience: + +- `_ = x` is now a borrowing operation. +- `switch` and `if/while case` require the subject of a pattern match to use + the `consume x` operator. The fact that they are consuming operations now + is an artifact of the implementation, and with further development, we may + want to make the default semantics of `switch x` without explicit consumption + to be borrowing. +- Escaped closure captures are constrained from being consumed for their + entire lifetime, even before the closure that escapes it is formed. This + analysis was not practical to implement using our current analysis, and the + added expressivity is unlikely to be worth the implementation complexity. +- `self` in a deinit is currently constrained to be immutable, since there + is [ongoing discussion](https://forums.swift.org/t/se-0390-noncopyable-type-deinit-s-mutation-and-accidental-recursion/64767) + about how best to manage mutation or consumption during deinits while + managing the possibility to accidentally cause recursion into `deinit` + by implicit destruction. + +The [second reviewed revision](https://github.com/swiftlang/swift-evolution/blob/a9e21e3a4eb9526f998915c6554c7c72e5885a91/proposals/0390-noncopyable-structs-and-enums.md) +of the proposal made the following changes from the +[first reviewed revision](https://github.com/swiftlang/swift-evolution/blob/5d075b86d57e3436b223199bd314b2642e30045f/proposals/0390-noncopyable-structs-and-enums.md): + +- The original revision did not provide a `Copyable` generic constraint, and + declared types as noncopyable using a `@noncopyable` attribute. The + language workgroup believes that it is a good idea to build toward a future + where noncopyable types are integrated with the language's generics system, + and that the syntax for suppressing generic constraints is a good general + notation to have for suppressing implicit conformances or assumptions about + generic capabilities we may take away in the future, so it makes sense to + provide a syntax that allows for growth in those directions. + +- The original revision suppressed implicit `deinit` within methods using the + spelling `forget self`. Although the term `forget` has a precedent in Rust, + the behavior of `mem::forget` in Rust doesn't correspond to the semantics of + the operation proposed here, and the language workgroup doesn't find the + term clear enough on its own. This revision of the proposal chooses the + `discard` as a starting point for further review discussion. Furthermore, + we limit its use to types whose contents are otherwise trivial, in order to + avoid committing to interactions with elementwise consumption of fields + that we may want to refine later. + +- The original revision allowed for a `consuming` method declared anywhere in + the type's original module to suppress `deinit`. This revision narrows the + capability to only methods declared in the same file as the type, for + consistency with other language features that depend on having visibility into + a type's entire layout, such as implicit `Sendable` inference. diff --git a/proposals/0391-package-registry-publish.md b/proposals/0391-package-registry-publish.md new file mode 100644 index 0000000000..6aba3942c2 --- /dev/null +++ b/proposals/0391-package-registry-publish.md @@ -0,0 +1,705 @@ +# Package Registry Publish + +* Proposal: [SE-0391](0391-package-registry-publish.md) +* Author: [Yim Lee](https://github.com/yim-lee) +* Review Manager: [Tom Doron](https://github.com/tomerd) +* Status: **Implemented (Swift 5.9)** +* Implementation: + * [apple/swift-package-manager#6101](https://github.com/apple/swift-package-manager/pull/6101) + * [apple/swift-package-manager#6146](https://github.com/apple/swift-package-manager/pull/6146) + * [apple/swift-package-manager#6159](https://github.com/apple/swift-package-manager/pull/6159) + * [apple/swift-package-manager#6169](https://github.com/apple/swift-package-manager/pull/6169) + * [apple/swift-package-manager#6188](https://github.com/apple/swift-package-manager/pull/6188) + * [apple/swift-package-manager#6189](https://github.com/apple/swift-package-manager/pull/6189) + * [apple/swift-package-manager#6215](https://github.com/apple/swift-package-manager/pull/6215) + * [apple/swift-package-manager#6217](https://github.com/apple/swift-package-manager/pull/6217) + * [apple/swift-package-manager#6220](https://github.com/apple/swift-package-manager/pull/6220) + * [apple/swift-package-manager#6229](https://github.com/apple/swift-package-manager/pull/6229) + * [apple/swift-package-manager#6237](https://github.com/apple/swift-package-manager/pull/6237) +* Review: ([pitch](https://forums.swift.org/t/pitch-package-registry-publish/62828)), ([review](https://forums.swift.org/t/se-0391-package-registry-publish/63405)), ([acceptance](https://forums.swift.org/t/accepted-se-0391-swift-package-registry-authentication/64088)) + +## Introduction + +A package registry makes packages available to consumers. Starting with Swift 5.7, +SwiftPM supports dependency resolution and package download using any registry that +implements the [service specification](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md) proposed alongside with [SE-0292](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0292-package-registry-service.md). +SwiftPM does not yet provide any tooling for publishing packages, so package authors +must manually prepare the contents (e.g., source archive) and interact +with the registry on their own to publish a package release. This proposal +aims to standardize package publishing such that SwiftPM can offer a complete and +well-rounded experience for using package registries. + +## Motivation + +Publishing package release to a Swift package registry generally involves these steps: + 1. Gather package release metadata. + 1. Prepare package source archive by using the [`swift package archive-source` subcommand](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0292-package-registry-service.md#archive-source-subcommand). + 1. Sign the metadata and archive (if needed). + 1. [Authenticate](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0378-package-registry-auth.md) (if required by the registry). + 1. Send the archive and metadata (and their signatures if any) by calling the ["create a package release" API](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-6). + 1. Check registry server response to determine if publication has succeeded or failed (if the registry processes request synchronously), or is pending (if the registry processes request asynchronously). + +SwiftPM can streamline the workflow by combining all of these steps into a single +`publish` command. + +## Proposed solution + +We propose to introduce a new `swift package-registry publish` subcommand to SwiftPM +as well as standardization on package release metadata and package signing to ensure a +consistent user experience for publishing packages. + +## Detailed design + +### Package release metadata + +Typically a package release has metadata associated with it, such as URL of the source +code repository, license, etc. In general, metadata gets set when a package release is +being published, but a registry service may allow modifications of the metadata afterwards. + +The current [registry service specification](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md) states that: + - A client (e.g., package author, publishing tool) may provide metadata for a package release by including it in the ["create a package release" request](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#462-package-release-metadata). The registry server will store the metadata and include it in the ["fetch information about a package release" response](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-2). + - If a client does not include metadata, the registry server may populate it unless the client specifies otherwise (i.e., by sending an empty JSON object `{}` in the "create a package release" request). + +It does not, however, define any requirements or server-client API contract on the +metadata contents. We would like to change that by proposing the following: + - Package release metadata will continue to be sent as a JSON object. + - Package release metadata must adhere to the [schema](#package-release-metadata-standards). + - Package release metadata will continue to be included in the "create a package release" request as a multipart section named `metadata` in the request body. + - Registry server may allow and/or populate additional metadata by expanding the schema, but it must not alter any of the predefined properties. + - Registry server may make any properties in the schema and additional metadata it defines required. Registry server may fail the "create a package release" request if any required metadata is missing. + - Client cannot change how registry server handles package release metadata. In other words, client will no longer be able to instruct registry server not to populate metadata by sending an empty JSON object `{}`. + - Registry server will continue to include metadata in the "fetch information about a package release" response. + +#### Package release metadata standards + +Package release metadata submitted to a registry must be a JSON object of type +[`PackageRelease`](#packagerelease-type), the schema of which is defined below. + +
+ +Expand to view JSON schema + +```json +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md", + "title": "Package Release Metadata", + "description": "Metadata of a package release.", + "type": "object", + "properties": { + "author": { + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "Name of the author." + }, + "email": { + "type": "string", + "description": "Email address of the author." + }, + "description": { + "type": "string", + "description": "A description of the author." + }, + "organization": { + "type": "object", + "properties": { + "name": { + "type": "string", + "description": "Name of the organization." + }, + "email": { + "type": "string", + "description": "Email address of the organization." + }, + "description": { + "type": "string", + "description": "A description of the organization." + }, + "url": { + "type": "string", + "description": "URL of the organization." + }, + }, + "required": ["name"] + }, + "url": { + "type": "string", + "description": "URL of the author." + }, + }, + "required": ["name"] + }, + "description": { + "type": "string", + "description": "A description of the package release." + }, + "licenseURL": { + "type": "string", + "description": "URL of the package release's license document." + }, + "readmeURL": { + "type": "string", + "description": "URL of the README specifically for the package release or broadly for the package." + }, + "repositoryURLs": { + "type": "array", + "description": "Code repository URL(s) of the package release.", + "items": { + "type": "string", + "description": "Code repository URL." + } + } + } +} +``` + +
+ +##### `PackageRelease` type + +| Property | Type | Description | Required | +| ----------------- | :-----------------: | ------------------------------------------------ | :------: | +| `author` | [Author](#author-type) | Author of the package release. | | +| `description` | String | A description of the package release. | | +| `licenseURL` | String | URL of the package release's license document. | | +| `readmeURL` | String | URL of the README specifically for the package release or broadly for the package. | | +| `repositoryURLs` | Array | Code repository URL(s) of the package. It is recommended to include all URL variations (e.g., SSH, HTTPS) for the same repository. This can be an empty array if the package does not have source control representation.
Setting this property is one way through which a registry can obtain repository URL to package identifier mappings for the ["lookup package identifiers registered for a URL" API](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#45-lookup-package-identifiers-registered-for-a-url). A registry may choose other mechanism(s) for package authors to specify such mappings. | | + +##### `Author` type + +| Property | Type | Description | Required | +| ----------------- | :-----------------: | ------------------------------------------------ | :------: | +| `name` | String | Name of the author. | ✓ | +| `email` | String | Email address of the author. | | +| `description` | String | A description of the author. | | +| `organization` | [Organization](#organization-type) | Organization that the author belongs to. | | +| `url` | String | URL of the author. | | + +##### `Organization` type + +| Property | Type | Description | Required | +| ----------------- | :-----------------: | ------------------------------------------------ | :------: | +| `name` | String | Name of the organization. | ✓ | +| `email` | String | Email address of the organization. | | +| `description` | String | A description of the organization. | | +| `url` | String | URL of the organization. | | + +### Package signing + +A registry may require packages to be signed. In order for SwiftPM to be able to +download and handle signed packages from a registry, we propose to standardize +package signature format and establish server-client API contract on package +signing. + +#### Package signature + +Package signature format will be identified by the underlying standard/technology +(e.g., Cryptographic Message Syntax (CMS), JSON Web Signature (JWS), etc.) and +version number. In the initial release, all signatures will be in [CMS](https://www.rfc-editor.org/rfc/rfc5652.html). + +| Signature format ID | Description | +| ------------------- | --------------------------------------------------------- | +| `cms-1.0.0` | Version 1.0.0 of package signature in CMS | + +##### Package signature format `cms-1.0.0` + +Package signature format `cms-1.0.0` uses CMS. + +| CMS Attribute | Details | +| ------------------------------ | --------------------------------------------------------- | +| Content type | [Signed-Data](https://www.rfc-editor.org/rfc/rfc5652.html#section-5) | +| Encapsulated data | The content being signed ([`EncapsulatedContentInfo.eContent`](https://www.rfc-editor.org/rfc/rfc5652.html#section-5.2)) is omitted since we are constructing an external signature. | +| Message digest algorithm | SHA-256, computed on the package source archive. | +| Signature algorithm | ECDSA P-256 | +| Number of signatures | 1 | +| Certificate | Certificate that contains the signing key. It is up to the registry to define the certificate policy (e.g., trusted root(s)). | + +The signature, represented in CMS, will be included as part +of the "create a package release" API request. + +A registry receiving such signed package will: + - Check if the signature format (`cms-1.0.0`) is accepted. + - Validate the signature is well-formed according to the signature format. + - Validate the certificate chain meets registry policy. + - Extract public key from the certificate and use it to verify the signature. + +Then the registry will process the package and save it for client downloads if publishing is successful. + +The registry must include signature information in the "fetch information about a package release" API +response to indicate the package is signed and the signature format (`cms-1.0.0`). + +After downloading a signed package SwiftPM will: + - Check if the signature format (`cms-1.0.0`) is supported. + - Validate the signature is well-formed according to the signature format. + - Validate that the signed package complies with the locally-configured signing policy. + - Extract public key from the certificate and use it to verify the signature. + +#### Server-side requirements for package signing + +A registry that requires package signing should provide documentations +on the signing requirements (e.g., any requirements for certificates +used in signing). + +A registry must also modify the ["create package release" API](#create-package-release-api) to allow +signature in the request, as well as the response for the ["fetch package release metadata"](#fetch-package-release-metadata-api) +and ["download package source archive"](#download-package-source-archive-api) API to include signature information. + +#### SwiftPM's handling of registry packages + +##### SwiftPM configuration + +Users will be able to configure how SwiftPM handles packages downloaded from a +registry. In the user-level `registries.json` file, which by default is located at +`~/.swiftpm/configuration/registries.json`, we will introduce a new `security` key: + +```json5 +{ + "security": { + "default": { + "signing": { + "onUnsigned": "prompt", // One of: "error", "prompt", "warn", "silentAllow" + "onUntrustedCertificate": "prompt", // One of: "error", "prompt", "warn", "silentAllow" + "trustedRootCertificatesPath": "~/.swiftpm/security/trusted-root-certs/", + "includeDefaultTrustedRootCertificates": true, + "validationChecks": { + "certificateExpiration": "disabled", // One of: "enabled", "disabled" + "certificateRevocation": "disabled" // One of: "strict", "allowSoftFail", "disabled" + } + } + }, + "registryOverrides": { + // The example shows all configuration overridable at registry level + "packages.example.com": { + "signing": { + "onUnsigned": "warn", + "onUntrustedCertificate": "warn", + "trustedRootCertificatesPath": , + "includeDefaultTrustedRootCertificates": , + "validationChecks": { + "certificateExpiration": "enabled", + "certificateRevocation": "allowSoftFail" + } + } + } + }, + "scopeOverrides": { + // The example shows all configuration overridable at scope level + "mona": { + "signing": { + "trustedRootCertificatesPath": , + "includeDefaultTrustedRootCertificates": + } + } + }, + "packageOverrides": { + // The example shows all configuration overridable at package level + "mona.LinkedList": { + "signing": { + "trustedRootCertificatesPath": , + "includeDefaultTrustedRootCertificates": + } + } + } + }, + ... +} +``` + +Security configuration for a package is computed using values from +the following (in descending precedence): +1. `packageOverrides` (if any) +1. `scopeOverrides` (if any) +1. `registryOverrides` (if any) +1. `default` + +The `default` JSON object contains all configurable security options +and their default value when there is no override. + +- `signing.onUnsigned`: Indicates how SwiftPM will handle an unsigned package. + + | Option | Description | + | ------------- | --------------------------------------------------------- | + | `error` | SwiftPM will reject the package and fail the build. | + | `prompt` | SwiftPM will prompt user to see if the unsigned package should be allowed.
  • If no, SwiftPM will reject the package and fail the build.
  • If yes and the package has never been downloaded, its checksum will be stored for [local TOFU](#local-tofu). Otherwise, if the package has been downloaded before, its checksum must match the previous value or else SwiftPM will reject the package and fail the build.
SwiftPM will record user's response to prevent repetitive prompting. | + | `warn` | SwiftPM will not prompt user but will emit a warning before proceeding. | + | `silentAllow` | SwiftPM will allow the unsigned package without prompting user or emitting warning. | + +- `signing.onUntrustedCertificate`: Indicates how SwiftPM will handle a package signed with an [untrusted certificate](#trusted-vs-untrusted-certificate). + + | Option | Description | + | ------------- | --------------------------------------------------------- | + | `error` | SwiftPM will reject the package and fail the build. | + | `prompt` | SwiftPM will prompt user to see if the package signed with an untrusted certificate should be allowed.
  • If no, SwiftPM will reject the package and fail the build.
  • If yes, SwiftPM will proceed with the package as if it were an unsigned package.
SwiftPM will record user's response to prevent repetitive prompting. | + | `warn` | SwiftPM will not prompt user but will emit a warning before proceeding. | + | `silentAllow` | SwiftPM will allow the package signed with an untrusted certificate without prompting user or emitting warning. | + +- `signing.trustedRootCertificatesPath`: Absolute path to the directory containing custom trusted roots. SwiftPM will include these roots in its [trust store](#trusted-vs-untrusted-certificate), and certificates used for package signing must chain to roots found in this store. This configuration allows override at the package, scope, and registry levels. +- `signing.includeDefaultTrustedRootCertificates`: Indicates if SwiftPM should include default trusted roots in its [trust store](#trusted-vs-untrusted-certificate). This configuration allows override at the package, scope, and registry levels. +- `signing.validationChecks`: Validation check settings for the package signature. + + | Validation | Description | + | ------------------------ | --------------------------------------------------------------- | + | `certificateExpiration` |
  • `enabled`: SwiftPM will check that the current timestamp when downloading falls within the signing certificate's validity period. If it doesn't, SwiftPM will reject the package and fail the build.
  • `disabled`: SwiftPM will not perform this check.
| + | `certificateRevocation` | With the exception of `disabled`, SwiftPM will check revocation status of the signing certificate. SwiftPM will only support revocation check done through [OCSP](https://www.rfc-editor.org/rfc/rfc6960) in the first feature release.
  • `strict`: Revocation check must complete successfully and the certificate must be in good status. SwiftPM will reject the package and fail the build if the revocation status is revoked or unknown (including revocation check not supported or failed).
  • `allowSoftFail`: SwiftPM will reject the package and fail the build iff the certificate has been revoked. SwiftPM will allow the certificate's revocation status to be unknown (including revocation check not supported or failed).
  • `disabled`: SwiftPM will not perform this check.
| + +##### Trusted vs. untrusted certificate + +A certificate is **trusted** if it is chained to any roots in SwiftPM's +trust store, which is a combination of: + - SwiftPM's default trust store, if `signing.includeDefaultTrustedRootCertificates` is `true`. + - Custom root(s) in the configured trusted roots directory at `signing.trustedRootCertificatesPath`. + +Otherwise, a certificate is **untrusted** and handled according to the `signing.onUntrustedCertificate` setting. + +Both `signing.includeDefaultTrustedRootCertificates` and `signing.trustedRootCertificatesPath` +support multiple levels of overrides. SwiftPM will choose the configuration value that has the +highest specificity. + + +For example, when evaluating the value of `signing.includeDefaultTrustedRootCertificates` +or `signing.trustedRootCertificatesPath` for package `mona.LinkedList`: + 1. SwiftPM will use the package override from `packageOverrides` (i.e., `packageOverrides["mona.LinkedList"]`), if any. + 1. Otherwise, SwiftPM will use the scope override from `scopeOverrides` (i.e., `scopeOverrides["mona"]`), if any. + 1. Next, depending on the registry the package is downloaded from, SwiftPM will look for and use the registry override in `registryOverrides`, if any. + 1. Finally, if no override is found, SwiftPM will use the value from `default`. + +##### Local TOFU + +When SwiftPM downloads a package release from registry via the +["download source archive" API](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-4), it will: + 1. Search local fingerprints storage, which by default is located at `~/.swiftpm/security/fingerprints/`, to see if the package release has been downloaded before and its recorded checksum. The checksum of the downloaded source archive must match the previous value or else [trust on first use (TOFU)](https://en.wikipedia.org/wiki/Trust_on_first_use) check would fail. + 1. Fetch package release metadata from the registry to get: +
    +
  • Checksum for TOFU if the package release is downloaded for the first time.
  • +
  • Signature information if the package release is signed.
  • +
+ 1. Retrieve security settings from the user-level `registries.json`. + 1. Check if the package is allowed based on security settings. + 1. Validate the signature according to the signature format if package is signed. + 1. Some certificates allow SwiftPM to extract additional information that can drive additional security features. For packages signed with these certificates, SwiftPM will apply additional, publisher-level TOFU by extracting signing identity from the certificate and enforcing the same signing identity across all signed versions of a package. + +### New `package-registry publish` subcommand + +The new `package-registry publish` subcommand will create a package +source archive, sign it if needed, and publish it to a registry. + +```manpage +> swift package-registry publish --help +OVERVIEW: Publish a package release to registry + +USAGE: package-registry publish + +ARGUMENTS: + The package identifier. + The package release version being created. + +OPTIONS: + --url The registry URL. + --scratch-directory The path of the directory where working file(s) will be written. + + --metadata-path The path to the package metadata JSON file if it's not 'package-metadata.json' in the package directory. + + --signing-identity The label of the signing identity to be retrieved from the system's secrets store if supported. + + --private-key-path The path to the certificate's PKCS#8 private key (DER-encoded). + --cert-chain-paths Path(s) to the signing certificate (DER-encoded) and optionally the rest of the certificate chain. The signing certificate must be listed first. + + --dry-run Dry run only; prepare the archive and sign it but do not publish to the registry. +``` + +- `id`: The package identifier in the `.` notation as defined in [SE-0292](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0292-package-registry-service.md#package-identity). It is the package author's responsibility to register the package identifier with the registry beforehand. +- `version`: The package release version in [SemVer 2.0](https://semver.org) notation. +- `url`: The URL of the registry to publish to. SwiftPM will try to determine the registry URL by searching for a scope-to-registry mapping or use the `[default]` URL in `registries.json`. The command will fail if this value is missing. +- `scratch-directory`: The path of the working directory. SwiftPM will write to the package directory by default. + +The following may be required depending on registry support and/or requirements: + - `metadata-path`: The path to the JSON file containing [package release metadata](#package-release-metadata). By default, SwiftPM will look for a file named `package-metadata.json` in the package directory if this is not specified. SwiftPM will include the content of the metadata file in the request body if present. If the package source archive is being signed, the metadata will be signed as well. + - `signing-identity`: The label that identifies the signing identity to use for package signing in the system's secrets store if supported. + - `private-key-path`: Required for package signing unless `signing-identity` is specified, this is the path to the private key used for signing. + - `cert-chain-paths`: Required for package signing unless `signing-identity` is specified, this is the signing certificate chain. + +A signing identity encompasses a private key and a certificate. On +systems where it is supported, SwiftPM can look for a signing identity +using the query string given via the `--signing-identity` option. This +feature will be available on macOS through Keychain in the initial +release, so a certificate and its private key can be located by the +certificate label alone. + +Otherwise, both `--private-key-path` and `--cert-chain-paths` must be +provided to locate the signing key and certificate chain. + +SwiftPM will sign the package source archive and package release metadata if `signing-identity` +or both `private-key-path` and `cert-chain-paths` are set. + +All signatures in the initial release will be in the [`cms-1.0.0`](#package-signature-format-cms-100) format. + +Using these inputs, SwiftPM will: + - Generate source archive for the package release. + - Sign the source archive and metadata if the required parameters are provided. + - Make HTTP request to the "create a package release" API. + - Check server response for any errors. + +Prerequisites: +- Run [`swift package-registry login`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0378-package-registry-auth.md#new-login-subcommand) to authenticate registry user if needed. +- The user has the necessary permissions to call the ["create a package release" API](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-6) for the package identifier. + +### Changes to the registry service specification + +#### Create package release API + +A registry must update [this existing endpoint](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-6) to handle package release +metadata as described in a [previous section](#package-release-metadata) of this document. + +If the package being published is signed, the client must identify the signature format +in the `X-Swift-Package-Signature-Format` HTTP request header so that the +server can process the signature accordingly. + +Signatures of the source-archive and metadata are sent as part of the request body +(`source-archive-signature` and `metadata-signature`, respectively): + +``` +PUT /mona/LinkedList?version=1.1.1 HTTP/1.1 +Host: packages.example.com +Accept: application/vnd.swift.registry.v1+json +Content-Type: multipart/form-data;boundary="boundary" +Content-Length: 336 +Expect: 100-continue +X-Swift-Package-Signature-Format: cms-1.0.0 + +--boundary +Content-Disposition: form-data; name="source-archive" +Content-Type: application/zip +Content-Length: 32 +Content-Transfer-Encoding: base64 + +gHUFBgAAAAAAAAAAAAAAAAAAAAAAAA== + +--boundary +Content-Disposition: form-data; name="source-archive-signature" +Content-Type: application/octet-stream +Content-Length: 88 +Content-Transfer-Encoding: base64 + +l1TdTeIuGdNsO1FQ0ptD64F5nSSOsQ5WzhM6/7KsHRuLHfTsggnyIWr0DxMcBj5F40zfplwntXAgS0ynlqvlFw== + +--boundary +Content-Disposition: form-data; name="metadata" +Content-Type: application/json +Content-Transfer-Encoding: quoted-printable +Content-Length: 25 + +{ "repositoryURLs": [] } + +--boundary +Content-Disposition: form-data; name="metadata-signature" +Content-Type: application/octet-stream +Content-Length: 88 +Content-Transfer-Encoding: base64 + +M6TdTeIuGdNsO1FQ0ptD64F5nSSOsQ5WzhM6/7KsHRuLHfTsggnyIWr0DxMcBj5F40zfplwntXAgS0ynlqvlFw== +``` + +#### Fetch package release metadata API + +A registry may update [this existing endpoint](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-2) for the [metadata changes](#package-release-metadata) +described in this document. + +If the package release is signed, the registry must include a `signing` JSON +object in the response: + +```json5 +{ + "id": "mona.LinkedList", + "version": "1.1.1", + "resources": [ + { + "name": "source-archive", + "type": "application/zip", + "checksum": "a2ac54cf25fbc1ad0028f03f0aa4b96833b83bb05a14e510892bb27dea4dc812", + "signing": { + "signatureBase64Encoded": "l1TdTeIuGdNsO1FQ0ptD64F5nSSOsQ5WzhM6/7KsHRuLHfTsggnyIWr0DxMcBj5F40zfplwntXAgS0ynlqvlFw==", + "signatureFormat": "cms-1.0.0" + } + } + ], + "metadata": { ... } +} +``` + +#### Download package source archive API + +If a registry supports signing, it must update [this existing endpoint](https://github.com/apple/swift-package-manager/blob/main/Documentation/PackageRegistry/Registry.md#endpoint-4) +to include the `X-Swift-Package-Signature-Format` and `X-Swift-Package-Signature` headers in +the HTTP response for a signed package source archive. + +``` +HTTP/1.1 200 OK +Accept-Ranges: bytes +Cache-Control: public, immutable +Content-Type: application/zip +Content-Disposition: attachment; filename="LinkedList-1.1.1.zip" +Content-Length: 2048 +Content-Version: 1 +Digest: sha-256=oqxUzyX7wa0AKPA/CqS5aDO4O7BaFOUQiSuyfepNyBI= +Link: ; rel=duplicate; geo=jp; pri=10; type="application/zip" +X-Swift-Package-Signature-Format: cms-1.0.0 +X-Swift-Package-Signature: l1TdTeIuGdNsO1FQ0ptD64F5nSSOsQ5WzhM6/7KsHRuLHfTsggnyIWr0DxMcBj5F40zfplwntXAgS0ynlqvlFw== +``` + +## Security + +This proposal introduces the framework for package signing, allowing package +authors the ability to provide additional authenticity guarantees by signing their +source archives before publishing them to the registry. Package users will be +able to control the kind(s) of packages they trust by specifying a local validation +policy. This can include a trust on first use approach, or by validating against a +pre-configured set of trusted roots. + +While this proposal introduces the package signature format, it does not validate +that a package is published by a specific entity. Instead, it validates that a package +is published by an entity who can obtain a signing certificate that meets the +requirements defined by the registry, which could be anybody. As such, it does +not provide any protection against malware, and it would be wrong to assumed that +signed packages can be trusted unconditionally. + +In this proposal, package signing is primarily intended to provide additional security +controls for package registries. By requiring packages be signed, and by the +registry limiting what keys or identities are allowed to publish packages, a registry +can provide additional security in the event the package author's registry credentials +are compromised. + +Although this proposal introduces policy controls for package users, they are limited +in scope, and do not yet allow SwiftPM to validate that multiple package versions are +from the same entity--recording signing identities for [TOFU](#local-tofu) provides some +protection against a compromised registry, but it is not for all packages and +[more work needs to be done](#local-signing-identity-checks) before it can be so. As such, SwiftPM continues to trust +the registry to provide authentic packages and accurate information about the +signature status of the package. + +### Privacy implications of certificate revocation check + +Revocation checking via OCSP implicitly discloses to the certificate +authority and anyone on the network the packages that a user may be +downloading. If this is a concern, revocation check can be disabled +in [SwiftPM configuration](#swiftpm-configuration). + +## Impact on existing packages + +Current packages won't be affected by changes in this proposal. + +## Alternatives considered + +### Signing package source archive vs. manifest + +A package manifest is a reference list of files that are present in the +source archive. We considered an approach where SwiftPM would produce +such manifest, sign the manifest instead of the source archive, +then create a new archive containing the source archive, manifest, and +signature file. This way the archive and its signature can be distributed +by the registry as a single file. + +However, given the potential complications with extracting files from the +archive and verifying manifest contents, moreover there is no restriction +that would require single-file download (i.e., SwiftPM can download the +source archive and signature separately), we have decided to take the approach +covered in previous sections of this proposal. + +### Use key in certificate as signing identity for local publisher-level TOFU + +We considered using the key in a certificate as signing identity for +[local publisher-level TOFU](#local-tofu) (i.e., different versions of a package must +have the same signing identity). However, since key can change easily +(e.g., lost key, key rotation, etc.), all users of the package must reset data +used for local TOFU each time or else TOFU check would fail, which can introduce +significant overhead and confusion. + +## Future directions + +### Support encrypted private keys + +Private keys are encrypted typically. SwiftPM commands that have private key +as input, such as `package sign` and `package-registry publish`, should support +reading encrypted private key. This could mean modifying the command to prompt +user for the passphrase if needed, and adding a `--private-key-passphrase` +option to the command for non-interactive/automation use-cases. + +### Auto-populate package release metadata + +Parts of the [package release metadata](#package-release-metadata-standards) can be populated by SwiftPM using +information found in the package directory. The auto-generated metadata can +serve as a default or starting point which package authors may optionally edit, +and ensure every package release to have metadata. + +### Support additional certificate revocation checks + +SwiftPM may support alternative mechanisms to check revocation besides OCSP. + +### Local signing identity checks + +In the current proposal, signing identity is left for the package registry to define, implement, and +enforce at publication time. However, this requires SwiftPM to rely on the package +registry to correctly implement these checks, and a compromise of the registry, or +SwiftPM's connection to the registry, would potentially allow for unauthorized packages +to be published. Performing additional checks in SwiftPM can mitigate this risk, but +requires defining a consistent identity that can be extracted and relied upon, and +determining how those identities are provisioned and authorized. + +A future Swift evolution proposal can provide specification of a certificate +from which signing identity can be extracted, such that more certificates can +be used for [local publisher-level TOFU](#local-tofu), which provides an extra layer of +trust on top of checksum TOFU done at the package release level. + +### Timestamping and Countersignatures + +In the proposed implementation, the signing certificate associated with +the package may expire, and this can prevent SwiftPM from validating +the information and revocation status of the certificate. Using +approaches such as [Time Stamping Authority](https://www.rfc-editor.org/rfc/rfc3161) or having the registry +itself perform a [countersignature](https://www.rfc-editor.org/rfc/rfc5652#section-11.4), information about when a +package was first published can be provided, +even after the signing certificate has expired. This can avoid the need +for package authors to re-sign packages when the signing certificate +expires. + +### Transitive trust + +SwiftPM's TOFU mitigation could be further improved by +including checksum and signing identity in `Package.resolved` +(or another similar file), which then gets included in the package content. +Including such security metadata would allow distributing information about +direct and transitive dependencies across the ecosystem much faster than a +local-only TOFU without requiring a centralized database/service to vend +this information. + +```json5 +{ + "pins": [ + { + "identity": "mona.LinkedList", + "kind": "registry", + "location": "https://packages.example.com/mona/LinkedList", + "state": { + "checksum": "ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d", + "version": "0.12.0" + }, + "signingBy": { + "identityType": , + "name": , + ... + } + }, + { + "identity": "Foo", + "kind": "remoteSourceControl", + "location": "https://github.com/something/Foo.git", + "state": { + "revision": "90a9574276f0fd17f02f58979423c3fd4d73b59e", + "version": "1.0.2", + } + } + ], + "version": 2 +} +``` diff --git a/proposals/0392-custom-actor-executors.md b/proposals/0392-custom-actor-executors.md new file mode 100644 index 0000000000..13497cff8d --- /dev/null +++ b/proposals/0392-custom-actor-executors.md @@ -0,0 +1,993 @@ +# Custom Actor Executors + +* Proposal: [SE-0392](0392-custom-actor-executors.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [John McCall](https://github.com/rjmccall), [Kavon Farvardin](https://github.com/kavon) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 5.9)** +* Previous threads: + - Original pitch thread from around Swift 5.5: [Support custom executors in Swift Concurrency](https://forums.swift.org/t/support-custom-executors-in-swift-concurrency/44425) + - Original "assume..." proposal which was subsumed into this proposal, as it relates closely to asserting on executors: [Pitch: Unsafe Assume on MainActor](https://forums.swift.org/t/pitch-unsafe-assume-on-mainactor/63074/) +* Reviews: + - First review thread: https://forums.swift.org/t/returned-for-revision-se-0392-custom-actor-executors/64172 + - Revisions: + - Rename `Job` to `ExecutorJob`, making it less likely to conflict with existing type names, and typealias `UnownedJob` with `UnownedExecutorJob` (however the old type remains for backwards compatibility). + - Move assert/precondition/assume APIs to extensions on actor types, e.g. `Actor/assertIsolated`, `DistributedActor/preconditionIsolated`, `MainActor/assumeIsolated { ... }` + - Distributed actor executor customization `unownedExecutor` invoked on a remote distributed actor, to return an executor that fatal errors only once attempts are made to enqueue work onto it, rather than crashing immediately upon attempting to obtain the executor. + +## Table of Contents + +* [Introduction](#introduction) +* [Motivation](#motivation) +* [Proposed solution](#proposed-solution) +* [Detailed design](#detailed-design) + + [A low-level design](#a-low-level-design) + + [Executors](#executors) + + [Serial Executors](#serial-executors) + + [ExecutorJobs](#executorjobs) + + [Actors with custom SerialExecutors](#actors-with-custom-serialexecutors) + + [Asserting on executors](#asserting-on-executors) + + [Assuming actor executors](#asserting-actor-executors) + + [Default Swift Runtime Executors](#default-swift-runtime-executors) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#effect-on-abi-stability) +* [Effect on API resilience](#effect-on-api-resilience) +* [Alternatives considered](#alternatives-considered) +* [Future Directions](#future-directions) + + [Overriding the MainActor Executor](#overriding-the-mainactor-executor) + + [Executor Switching Optimizations](#executor-switching) + + [Specifying Task executors](#specifying-task-executors) + + [DelegateActor property](#delegateactor-property) + +## Introduction + +As Swift Concurrency continues to mature it is becoming increasingly important to offer adopters tighter control over where exactly asynchronous work is actually executed. + +This proposal introduces a basic mechanism for customizing actor executors. By providing an instance of an executor, actors can influence "where" they will be executing any task they are running, while upholding the mutual exclusion and actor isolation guaranteed by the actor model. + +> **Note:** This proposal defines only a set of APIs to customize actor executors, and other kinds of executor control is out of scope for this specific proposal. + +## Motivation + +Swift's concurrency design is intentionally vague about the details of how code is actually run. Most code does not rely on specific properties of the execution environment, such as being run to a specific operating system thread, and instead needs only high-level semantic properties, expressed in terms of actor isolation, such as that no other code will be accessing certain variables concurrently. Maintaining flexibility about how work is scheduled onto threads allows Swift to avoid certain performance pitfalls by default. + +Nonetheless, it is sometimes useful to more finely control how code is executed: + +- The code may need to cooperate with an existing system that expects to run code in a certain way. + + For example, the system might expect certain kinds of work to be scheduled in special ways, like how some platforms require UI code to be run on the main thread, or single-threaded event-loop based runtimes assume all calls will be made from the same thread that is owned and managed by the runtime itself. + + For another example, a project might have a large amount of existing code which protects some state with a shared queue. In principle, this is the actor pattern, and the code could be rewritten to use Swift's actor support. However, it may be impossible to do that, or at least impractical to do it immediately. Using the existing queue as the executor for an actor allows code to adopt actors more incrementally. + +- The code may depend on being run on a specific system thread. + + For example, some libraries maintain state in thread-local variables, and running code on the wrong thread will lead to broken assumptions in the library. + + For another example, not all execution environments are homogeneous; some threads may be pinned to processors with extra capabilities. + +- The code's performance may benefit from the programmer being more explicit about where code should run. + + For example, if one actor frequently makes requests of another, and the actors rarely benefit from running concurrently, configuring them to use the same executor may decrease the runtime costs of switching between them. + + For another example, if an asynchronous function makes many calls to the same actor without any intervening suspensions, running the function explicitly on that actor's executor may eventually allow Swift to avoid a lot of switching overhead (or may even be necessary to perform those calls "atomically"). + +This is the first proposal discussing custom executors and customization points in the Swift Concurrency runtime, and while it introduces only the most basic customization points, we are certain that it already provides significant value to users seeking tighter control over their actor's execution semantics. + +Along with introducing ways to customize where code executes, this proposal also introduces ways to assert and assume the appropriate executor is used. This allows for more confidence when migrating away from other concurrency models to Swift Concurrency. + +## Proposed solution + +We propose to give developers the ability to implement simple serial executors, which then can be used with actors in order to ensure that any code executoring on such "actor with custom serial executor" runs on the appropriate thread or context. Implementing a naive executor takes the shape of: + +```swift +final class SpecificThreadExecutor: SerialExecutor { + let someThread: SomeThread // simplified handle to some specific thread + + func enqueue(_ job: consuming ExecutorJob) { + let unownedJob = UnownedExecutorJob(job) // in order to escape it to the run{} closure + someThread.run { + unownedJob.runSynchronously(on: self) + } + } + + func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(ordinary: self) + } +} + +extension SpecificThreadExecutor { + static var sharedUnownedExecutor: UnownedSerialExecutor { + // ... use some shared configured instance and return it ... + } +} +``` + +Such executor can then be used with an actor declaration by implementing its `unownedExecutor` property: + +```swift +actor Worker { + nonisolated var unownedExecutor: UnownedSerialExecutor { + // use the shared specific thread executor mentioned above. + // alternatively, we can pass specific executors to this actors init() and store and use them this way. + SpecificThreadExecutor.sharedUnownedExecutor + } +} +``` + +And lastly, in order to increase the confidence during moves from other concurrency models to Swift Concurrency with custom executors, we also provide ways to assert that a piece of code is executing on the appropriate executor. These methods should be used only if there is not better way to express the requirement statically. For example by expressing the code as a method on a specific actor, or annotating it with a `@GlobalActor`, should be preferred to asserting when possible, however sometimes this is not possible due to the old code fulfilling synchronous protocol requirements that still have these threading requirements. + +Asserting the appropriate executor is used in a synchronous piece of code looks like this: + +````swift +func synchronousButNeedsMainActorContext() { + // check if we're executing on the main actor context (or crash if we're not) + MainActor.preconditionIsolated() + + // same as precondition, however only in DEBUG builds + MainActor.assertIsolated() +} +```` + +Furthermore, we also offer a new API to safely "assume" an actor's execution context. For example, a synchronous function may know that it always will be invoked by the `MainActor` however for some reason it cannot be marked using `@MainActor`, this new API allows to assume (or crash if called from another execution context) the appropriate execution context, including the safety of synchronously accessing any state protected by the main actor executor: + +```swift +@MainActor func example() {} + +func alwaysOnMainActor() /* must be synchronous! */ { + MainActor.assumeIsolated { // will crash if NOT invoked from the MainActor's executor + example() // ok to safely, synchronously, call + } +} + +// Always prefer annotating the method using a global actor, rather than assuming it though. +@MainActor func alwaysOnMainActor() /* must be synchronous! */ { } // better, but not always possible +``` + +## Detailed design + +### A low-level design + +The API design of executors is intended to support high-performance implementations, with an expectation that custom executors will be primarily implemented by experts. Therefore, the following design heavily prioritizes the reliable elimination of abstraction costs over most other conceivable goals. In particular, the primitive operations specified by protocols are generally expressed in terms of opaque, unsafe types which implementations are required to use correctly. These operations are then used to implement more convenient APIs as well as the high-level language operations of Swift Concurrency. + +### Executors + +First, we introduce an `Executor` protocol, that serves as the parent protocol of all the specific kinds of executors we'll discuss next. It is the simplest kind of executor that does not provide any ordering guarantees about the submitted work. It could decide to run the submitted jobs in parallel, or sequentially. + +This protocol has existed in Swift ever since the introduction of Swift Concurrency, however, in this proposal we revise its API to make use of the newly introduced move-only capabilities in the language. The existing `UnownedExecutorJob` API will be deprecated in favor of one accepting a move-only `ExecutorJob`. The `UnownedExecutorJob` type remains available (and equally unsafe), because today still some usage patterns are not supported by the initial revision of move-only types. + +The concurrency runtime uses the `enqueue(_:)` method of an executor to schedule some work onto given executor. + +```swift +/// A service that can execute jobs. +public protocol Executor: AnyObject, Sendable { + + // This requirement is repeated here as a non-override so that we + // get a redundant witness-table entry for it. This allows us to + // avoid drilling down to the base conformance just for the basic + // work-scheduling operation. + func enqueue(_ job: consuming ExecutorJob) + + @available(*, deprecated, message: "Implement the enqueue(_:ExecutorJob) method instead") + func enqueue(_ job: UnownedExecutorJob) +} +``` + +In order to aid this transition, the compiler will offer assistance similar to how the transition from `Hashable.hashValue` to `Hashable.hash(into:)` was handled. Existing executor implementations which implemented `enqueue(UnownedExecutorJob)` will still work, but print a deprecation warning: + +```swift +final class MyOldExecutor: SerialExecutor { + // WARNING: 'Executor.enqueue(UnownedExecutorJob)' is deprecated as a protocol requirement; + // conform type 'MyOldExecutor' to 'Executor' by implementing 'enqueue(ExecutorJob)' instead + func enqueue(_ job: UnownedExecutorJob) { + // ... + } +} +``` + +Executors are required to follow certain ordering rules when executing their jobs: + +- The call to `ExecutorJob.runSynchronously(on:)` must happen-after the call to `enqueue(_:)`. +- If the executor is a serial executor, then the execution of all jobs must be *totally ordered*: for any two different jobs *A* and *B* submitted to the same executor with `enqueue(_:)`, it must be true that either all events in *A* happen-before all events in *B* or all events in *B* happen-before all events in *A*. + - Do note that this allows the executor to reorder `A` and `B`–for example, if one job had a higher priority than the other–however they each independently must run to completion before the other one is allowed to run. + + +### Serial Executors + +We also define a `SerialExecutor` protocol, which is what actors use to guarantee their serial execution of tasks (jobs). + +```swift +/// A service that executes jobs one-by-one, and specifically, +/// guarantees mutual exclusion between job executions. +/// +/// A serial executor can be provided to an actor (or distributed actor), +/// to guarantee all work performed on that actor should be enqueued to this executor. +/// +/// Serial executors do not, in general, guarantee specific run-order of jobs, +/// and are free to re-order them e.g. using task priority, or any other mechanism. +public protocol SerialExecutor: Executor { + /// Convert this executor value to the optimized form of borrowed + /// executor references. + func asUnownedSerialExecutor() -> UnownedSerialExecutor + + // Discussed in depth in "Details of 'same executor' checking" of this proposal. + func isSameExclusiveExecutionContext(other executor: Self) -> Bool +} + +extension SerialExecutor { + // default implementation is sufficient for most implementations + func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(ordinary: self) + } + + func isSameExclusiveExecutionContext(other: Self) -> Bool { + self === other + } +} +``` + +A `SerialExecutor` does not introduce new API, other than the wrapping itself in an `UnownedSerialExecutor` which is used by the Swift runtime to pass executors without incurring reference counting overhead. + +```swift +/// An unowned reference to a serial executor (a `SerialExecutor` +/// value). +/// +/// This is an optimized type used internally by the core scheduling +/// operations. It is an unowned reference to avoid unnecessary +/// reference-counting work even when working with actors abstractly. +/// Generally there are extra constraints imposed on core operations +/// in order to allow this. For example, keeping an actor alive must +/// also keep the actor's associated executor alive; if they are +/// different objects, the executor must be referenced strongly by the +/// actor. +public struct UnownedSerialExecutor: Sendable { + /// The default and ordinary way to expose an unowned serial executor. + public init(ordinary executor: E) + + /// Discussed in depth in "Details of same-executor checking" of this proposal. + public init(complexEquality executor: E) +} +``` + +`SerialExecutors` will potentially be extended to support "switching" which can lessen the amount of thread switches incured when using custom executors. Please refer to the Future Directions for a discussion of this extension. + +### ExecutorJobs + +An `ExecutorJob` is a representation of a chunk of of work that an executor should execute. For example, a `Task` effectively consists of a series of jobs that are enqueued onto executors, in order to run them. The name "job" was selected because we do not want to constrain this API to just "partial tasks", or tie them too closely to tasks, even though the most common type of job created by Swift concurrency are "partial tasks". + +Whenever the Swift Concurrency runtime needs to execute some piece of work, it enqueues an `UnownedExecutorJob`s on a specific executor the job should be executed on. The `UnownedExecutorJob` type is an opaque wrapper around Swift's low-level representation of such job. It cannot be meaningfully inspected, copied and must never be executed more than once. + +```swift +@noncopyable +public struct ExecutorJob: Sendable { + /// The priority of this job. + public var priority: JobPriority { get } +} +``` + +```swift +/// The priority of this job. +/// +/// The executor determines how priority information affects the way tasks are scheduled. +/// The behavior varies depending on the executor currently being used. +/// Typically, executors attempt to run tasks with a higher priority +/// before tasks with a lower priority. +/// However, the semantics of how priority is treated are left up to each +/// platform and `Executor` implementation. +/// +/// A ExecutorJob's priority is roughly equivalent to a `TaskPriority`, +/// however, since not all jobs are tasks, represented as separate type. +/// +/// Conversions between the two priorities are available as initializers on the respective types. +public struct JobPriority { + public typealias RawValue = UInt8 + + /// The raw priority value. + public var rawValue: RawValue +} + +extension TaskPriority { + /// Convert a job priority to a task priority. + /// + /// Most values are directly interchangeable, but this initializer reserves the right to fail for certain values. + public init?(_ p: JobPriority) { ... } +} +``` + +Because move-only types in the first early iteration of this language feature still have a number of limitations, we also offer an `UnownedExecutorJob` type, that is an unsafe "unowned" version of a `ExecutorJob`. One reason one might need to reach for an `UnownedExecutorJob` is whenever a `ExecutorJob` were to be used in a generic context, because in the initial version of move-only types that is available today, such types cannot appear in a generic context. For example, a naive queue implementation using an `[ExecutorJob]` would be rejected by the compiler, but it is possible to express using an `UnownedExecutorJob` (i.e.`[UnownedExecutorJob]`). + +```swift +public struct UnownedExecutorJob: Sendable, CustomStringConvertible { + + /// Create an unsafe, unowned, job by consuming a move-only ExecutorJob. + /// + /// This may be necessary currently when intending to store a job in collections, + /// or otherwise intreracting with generics due to initial implementation + /// limitations of move-only types. + @usableFromInline + internal init(_ job: consuming ExecutorJob) { ... } + + public var priority: JobPriority { ... } + + public var description: String { ... } +} +``` + +A job's description includes its job or task ID, that can be used to correlate it with task dumps as well as task lists in Instruments and other debugging tools (e.g. `swift-inspect`'s ). A task ID is an unique number assigned to a task, and can be useful when debugging scheduling issues, this is the same ID that is currently exposed in tools like Instruments when inspecting tasks, allowing to correlate debug logs with observations from profiling tools. + +Eventually, an executor will want to actually run a job. It may do so right away when it is enqueued, or on some different thread, this is entirely left up to the executor to decide. Running a job is done by calling the `runSynchronously` on a `ExecutorJob` which consumes it. The same method is provided on the `UnownedExecutorJob` type, however that API is not as safe, since it cannot consume the job, and is open to running the same job multiple times accidentally, which is undefined behavior. Generally, we urge developers to stick to using `ExecutorJob` APIs whenever possible, and only move to the unowned API if the noncopyable `ExecutorJob`s restrictions prove too strong to do the necessary operations on it. + +```swift +extension ExecutorJob { + /// Run the job synchronously. + /// + /// This operation consumes the job. + public consuming func runSynchronously(on executor: UnownedSerialExecutor) { + _swiftJobRun(UnownedExecutorJob(job), executor) + } +} + +extension UnownedExecutorJob { + /// Run the job synchronously. + /// + /// A job can only be run *once*. Accessing the job after it has been run is undefined behavior. + public func runSynchronously(on executor: UnownedSerialExecutor) { + _swiftJobRun(job, executor) + } +} +``` + +### Actors with custom SerialExecutors + +All actors implicitly conform to the `Actor` (or `DistributedActor`) protocols, and those protocols include the customization point for the executor they are required to run on in form of the the `unownedExecutor` property. + +An actor's executor must conform to the `SerialExecutor` protocol, which refines the Executor protocol, and provides enough guarantees to implement the actor's mutual exclusion guarantees. In the future, `SerialExecutors` may also be extended to support "switching", which is a technique to avoid thread-switching in calls between actors whose executors are compatible to "lending" each other the currently running thread. This proposal does not cover switching semantics. + +Actors select which serial executor they should use to run jobs by implementing the `unownedExecutor` protocol requirement on the `Actor` and `DistributedActor` protocols: + +```swift +public protocol Actor: AnyActor { + /// Retrieve the executor for this actor as an optimized, unowned + /// reference. + /// + /// This property must always evaluate to the same executor for a + /// given actor instance, and holding on to the actor must keep the + /// executor alive. + /// + /// This property will be implicitly accessed when work needs to be + /// scheduled onto this actor. These accesses may be merged, + /// eliminated, and rearranged with other work, and they may even + /// be introduced when not strictly required. Visible side effects + /// are therefore strongly discouraged within this property. + nonisolated var unownedExecutor: UnownedSerialExecutor { get } +} + +public protocol DistributedActor: AnyActor { + /// Retrieve the executor for this distributed actor as an optimized, + /// unowned reference. This API is equivalent to ``Actor/unownedExecutor``. + /// + /// ## Executor of remote distributed actor reference + /// + /// The default implementation of the `unownedExecutor` uses a special "crash if enqueued on" + /// executor, that can be obtained using `buildDefaultDistributedRemoteActorExecutor(any DistributedActor)` + /// method. If implementing a custom executor of a distributed actor, the implementation may derive + /// its executor value from the `nonisolated var id` every actor possesses (e.g. by means of the `ID` + /// indicating some "executor preference"), however if the actor is remote, the implementation SHOULD + /// return the default remote distributed actor executor, same as the default implementation does. + /// + /// Even if a remote distributed actor reference were to return some shared executor, + /// the Swift runtime will never actively make use of it, because code in this process + /// never runs methods which can be called cross-actor isolated "on" such distributed actor, + /// but merely delegates to the ``DistributedActorSystem/remoteCall` to perform the remote call. + /// This call is performed on the actor system, and is not isolated to the actor. + /// + /// Returning a shared executor for a remote distributed actor reference will not "trick" the + /// swift runtime into wrongly allowing one to `assumeIsolated()` and run code isolated on a + /// remote actor, because a remote actor reference cannot ever be `isolated` with. + /// + /// ## Availability + /// + /// Distributed actors can only use custom executors if their availability requires + /// a platform with Swift 5.9 (or higher) present. On platforms without availability + /// annotations, a distributed actor may always + /// + /// ## Custom implementation requirements + /// + /// This property must always evaluate to the same executor for a + /// given actor instance, and holding on to the actor must keep the + /// executor alive. + /// + /// This property will be implicitly accessed when work needs to be + /// scheduled onto this actor. These accesses may be merged, + /// eliminated, and rearranged with other work, and they may even + /// be introduced when not strictly required. Visible side effects + /// are therefore strongly discouraged within this property. + nonisolated var unownedExecutor: UnownedSerialExecutor { get } +} +``` + +> Note: It is not possible to express this protocol requirement on `AnyActor` directly because `AnyActor` is a "marker protocol" which are not present at runtime, and cannot have protocol requirements. + +The compiler synthesizes an implementation for this requirement for every `(distributed) actor` declaration, unless an explicit implementation is provided. The default implementation synthesized by the compiler uses the default `SerialExecutor`, which uses the appropriate mechanism for the platform (e.g. Dispatch). Actors using this default synthesized implementation are referred to as "Default Actors", i.e. actors using the default serial executor implementation. + +Developers can customize the executor used by an actor on a declaration-by-declaration basis, by implementing this protocol requirement in an actor. For example, thanks to the `sharedUnownedExecutor` static property on `MainActor` it is possible to declare other actors which are also guaranteed to use the same serial executor (i.e. "the main thread"). + +```swift +(distributed) actor MainActorsBestFriend { + nonisolated var unownedExecutor: UnownedSerialExecutor { + MainActor.sharedUnownedExecutor + } + func greet() { + print("Main-friendly...") + try? await Task.sleep(for: .seconds(3)) + } +} + +@MainActor +func mainGreet() { + print("Main hello!") +} + +func test() { + Task { await mainGreet() } + Task { await MainActorsBestFriend().greet() } +} +``` + +The snippet above illustrates that while the `MainActor` and the `MainActorsBestFriend` are different actors, and thus are generally allowed to execute concurrently, because they *share* the same main actor serial executor, they will never execute concurrently. A serial executor can only run one task at any given time, which enforces the mutual exclusive execution of those two actors. + +It is also possible for libraries to offer protocols where a default, library specific, executor is already defined, like this: + +```swift +protocol WithSpecifiedExecutor: Actor { + nonisolated var executor: LibrarySpecificExecutor { get } +} + +protocol LibrarySpecificExecutor: SerialExecutor {} + +extension LibrarySpecificActor { + /// Establishes the WithSpecifiedExecutorExecutor as the serial + /// executor that will coordinate execution for the actor. + nonisolated var unownedExecutor: UnownedSerialExecutor { + executor.asUnownedSerialExecutor() + } +} + +/// A naive "run on calling thread" job executor. +/// Generally executors should enqueue and process the job on another thread instead. +/// Ways to efficiently avoid hops when not necessary, will be offered as part of the +/// "executor switching" feature, that is not part of this proposal. +final class InlineExecutor: SpecifiedExecutor, CustomStringConvertible { + public func enqueue(_ job: __owned ExecutorJob) { + runJobSynchronously(job) + } +} +``` + +Which ensures that users of such library implementing such actors provide the library specific `SpecificExecutor` for their actors: + +```swift +actor MyActor: WithSpecifiedExecutor { + + nonisolated let executor: SpecifiedExecutor + + init(executor: SpecifiedExecutor) { + self.executor = executor + } +} +``` + +A library could also provide a default implementation of such executor as well. + +### Asserting on executors + +A common pattern in event-loop heavy code–not yet using Swift Concurrency–is to ensure/verify that a synchronous piece of code is executed on the exected event-loop. Since one of the goals of making executors customizable is to allow such libraries to adopt Swift Concurrency by making such event-loops conform to `SerialExecutor`, it is useful to allow the checking if code is indeed executing on the appropriate executor, for the library to gain confidence while it is moving towards fully embracing actors and Swift concurrency. + +For example, SwiftNIO intentionally avoids synchronization checks in some synchronous methods, in order to avoid the overhead of doing so, however in DEBUG mode it performs assertions that given code is running on the expected event-loop: + +```swift +// SwiftNIO +private var _channel: Channel +internal var channel: Channel { + self.eventLoop.assertInEventLoop() + assert(self._channel != nil || self.destroyed) + return self._channel ?? DeadChannel(pipeline: self) +} +``` + +Dispatch based systems also have similar functionality, with the `dispatchPrecondition` API: + +```swift +// Dispatch +func checkIfMainQueue() { + dispatchPrecondition(condition: .onQueue(DispatchQueue.main)) +} +``` + +While, generally, in Swift Concurrency such preconditions are not necessary, because we can *statically* ensure to be on the right execution context by putting methods on specific actors, or using global actor annotations: + +```swift +@MainActor +func definitelyOnMainActor() {} + +actor Worker {} +extension Worker { + func definitelyOnWorker() {} +} +``` + +Sometimes, especially when porting existing codebases _to_ Swift Concurrency we recognize the ability to assert in synchronous code if it is running on the expected executor can bring developers more confidence during their migration to Swift Concurrency. In order to support these migrations, we propose the following method: + +```swift +extension SerialExecutor { + /// Checks if the current task is running on the expected executor. + /// + /// Do note that if multiple actors share the same serial executor, + /// this assertion checks for the executor, not specific actor instance. + /// + /// Generally, Swift programs should be constructed such that it is statically + /// known that a specific executor is used, for example by using global actors or + /// custom executors. However, in some APIs it may be useful to provide an + /// additional runtime check for this, especially when moving towards Swift + /// concurrency from other runtimes which frequently use such assertions. + public func preconditionIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} + +extension Actor { + public nonisolated func preconditionIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} + +extension DistributedActor { + public nonisolated func preconditionIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} +``` + +as well as an `assert...` version of this API, which triggers only in `debug` builds: + +```swift +extension SerialExecutor { + // Same as ``SerialExecutor/preconditionIsolated(_:file:line)`` however only in DEBUG mode. + public func assertIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} + +extension Actor { + // Same as ``Actor/preconditionIsolated(_:file:line)`` however only in DEBUG mode. + public nonisolated func assertIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} + +extension DistributedActor { + // Same as ``DistributedActor/preconditionIsolated(_:file:line)`` however only in DEBUG mode. + public nonisolated func assertIsolated( + _ message: @autoclosure () -> String = "", + file: String = #fileID, line: UInt = #line) +} +``` + +The versions of the APIs offered on `Actor` and `DistributedActor` offer better diagnostics than would be possible to implement using a plain `precondition()` implemented by developers using some `precondition(isOnExpectedExecutor(someExecutor))` because they offer a description of the actually active executor when mismatched: + +````swift +MainActor.preconditionIsolated() +// Precondition failed: Incorrect actor executor assumption; Expected 'MainActorExecutor' executor, but was executing on 'Sample.InlineExecutor'. +```` + +It should be noted that this API will return true whenever two actors share an executor. Semantically sharing a serial executor means running in the same isolation domain, however this is only known dynamically and `await`s are still necessary for calls between such actors: + +```swift +actor A { + nonisolated var unownedExecutor: UnownedSerialExecutor { MainActor.sharedUnownedExecutor } + + func test() {} +} + +actor B { + nonisolated var unownedExecutor: UnownedSerialExecutor { MainActor.sharedUnownedExecutor } + + func test(a: A) { + await a.test() // await is necessary, since we do not statically know about them being on the same executor + } +} +``` + +Potential future work could enable static checking where a relationship between actors is expressed statically (a specific instance of `B` declaring that it is on the same serial executor as a specific instance of `A`), and therefore awaits would not be necessary between such two specific actor instances. Such work is not within the scope of this initial proposal though, and only the dynamic aspect is proposed right now. + +At this point, similar to Dispatch, these APIs only offer an "assert" / "precondition" version. And currently the way to dynamically get a boolean answer about being on a specific executor is not exposed. + +### Assuming actor executors + +> Note: This API was initially pitched separately from custom executors, but as we worked on the feature we realized how closely it is related to custom executors and asserting on executors. The initial pitch thread is located here: [Pitch: Unsafe Assume on MainActor](https://forums.swift.org/t/pitch-unsafe-assume-on-mainactor/63074/). + +This revision of the proposal introduces the `MainActor.assumeIsolated(_:)` method, which allows synchronous code to safely assume that they are called within the context of the main actor's executor. This is only available in synchronous functions, because the right way to spell this requirement in asynchronous code is to annotate the function using `@MainActor` which statically ensures this requirement. + +Synchronous code can assume that it is running on the main actor executor by using this assume method: + +```swift +extension MainActor { + /// A safe way to synchronously assume that the current execution context belongs to the MainActor. + /// + /// This API should only be used as last resort, when it is not possible to express the current + /// execution context definitely belongs to the main actor in other ways. E.g. one may need to use + /// this in a delegate style API, where a synchronous method is guaranteed to be called by the + /// main actor, however it is not possible to annotate this legacy API with `@MainActor`. + /// + /// This method cannot be used in an asynchronous context. Instead, prefer implementing + /// a method annotated with `@MainActor` and calling it from your asynchronous context. + /// + /// - Warning: If the current executor is *not* the MainActor's serial executor, this function will crash. + /// + /// Note that this check is performed against the MainActor's serial executor, meaning that + /// if another actor uses the same serial executor--by using ``MainActor/sharedUnownedExecutor`` + /// as its own ``Actor/unownedExecutor``--this check will succeed, as from a concurrency safety + /// perspective, the serial executor guarantees mutual exclusion of those two actors. + @available(*, noasync) + func assumeIsolated( + _ operation: @MainActor () throws -> T, + file: StaticString = #fileID, line: UInt = #line + ) rethrows -> T +} +``` + +Similarly to the `preconditionIsolated` API, the executor check is performed against the target actor's executor, so if multiple actors are run on the same executor, this check will succeed in synchronous code invoked by such actors as well. In other words, the following code is also correct: + +```swift +func check(values: MainActorValues) /* synchronous! */ { + // values.get("any") // error: main actor isolated, cannot perform async call here + MainActor.assumeIsolated { + values.get("any") // correct & safe + } +} + +actor Friend { + var unownedExecutor: UnownedSerialExecutor { + MainActor.sharedUnownedExecutor + } + + func callCheck(values: MainActorValues) { + check(values) // correct + } +} + +actor Unknown { + func callCheck(values: MainActorValues) { + check(values) // will crash, we're not on the MainActor executor + } +} + +@MainActor +final class MainActorValues { + func get(_: String) -> String { ... } +} +``` + +> Note: Because it is not possible to abstract over the `@SomeGlobalActor () -> T` function type's global actor isolation, we currently do not offer a version of this API for _any_ global actor, however it would be possible to implement such API today using macros, which could be expored in a follow-up proposal if seen as important enough. Such API would have to be spelled `SomeGlobalActor.assumeIsolated() { @SomeGlobalActor in ... }`. + +In addition to the `MainActor` specialized API, the same shape of API is offered for instance actors and allows obtaining an `isolated` actor reference if we are guaranteed to be executing on the same serial executor as the given actor, and thus no concurrent access violations are possible. + +```swift +extension Actor { + /// A safe way to synchronously assume that the current execution context belongs to the passed in `actor`. + /// + /// If currently executing in the context of the actor's serial executor, safely execute the `operation` + /// isolated to the actor. Otherwise, crash reporting the difference in expected and actual executor. + /// + /// This method cannot be used in an asynchronous context. Instead, prefer implementing + /// a method on the actor and calling it from your asynchronous context. + /// + /// This API should only be used as last resort, when it is not possible to express the current + /// execution context definitely belongs to the main actor in other ways. E.g. one may need to use + /// this in a delegate style API, where a synchronous method is guaranteed to be called by the + /// main actor, however it is not possible to move some function implementation onto the target + /// `actor` for some reason. + /// + /// - Warning: If the current executor is *not* the actor's serial executor this function will crash. + /// + /// - Parameters: + /// - operation: the operation that will run if the executor checks pass + /// - Returns: the result of the operation + /// - Throws: the error the operation has thrown + @available(*, noasync) + func assumeIsolated( + _ operation: (isolated Self) throws -> T, + file: StaticString = #fileID, line: UInt = #line + ) rethrows -> T +} +``` + +These assume methods have the same semantics as the just explained `MainActor.assumeIsolated` in the sense that the check is performed about the actor's _executor_ and not specific instance. In other words, if many instance actors share the same serial executor, this check would pass for each of them, as long as the same executor is found to be the current one. + +The same method is offered for distributed actors, where code can only ever be isolated to an instance if the reference is to a _local_ distributed actor, as well as the same serial executor as the checked actor is running the current task: + +```swift +extension DistributedActor { + /// A safe way to synchronously assume that the current execution context belongs to the passed in `actor`. + /// + /// If currently executing in the context of the actor's serial executor, safely execute the `operation` + /// isolated to the actor. If the actor is local, or the current and expected executors are not compatible, + /// crash reporting the difference in expected and actual executor. + /// + /// This method cannot be used in an asynchronous context. Instead, prefer implementing + /// a method on the distributed actor and calling it from your asynchronous context. + /// + /// The actor must be a local distributed actor reference, as isolating execution to a remote reference + /// would not be memory safe, since a distributed remote actor reference is allowed to not allocate any + /// memory for its storage, and thus, any attempts to access it are illegal. If the actor is remote, + /// this method will terminate with a fatal error. + /// + /// This API should only be used as last resort, when it is not possible to express the current + /// execution context definitely belongs to the main actor in other ways. E.g. one may need to use + /// this in a delegate style API, where a synchronous method is guaranteed to be called by the + /// main actor, however it is not possible to move some function implementation onto the target + /// `distributed actor` for some reason. + /// + /// - Warning: If the current executor is *not* compatible with the expected serial executor, + /// or the distributed actor is a remote reference, this function will crash. + /// + /// - Parameters: + /// - operation: the operation that will run if the executor checks pass + /// - Returns: the result of the operation + /// - Throws: the error the operation has thrown + @available(*, noasync) + func assumeIsolated( + _ operation: (isolated Self) throws -> T, + file: StaticString = #fileID, line: UInt = #line + ) rethrows -> T +} +``` + +### Details of "same executor" checking + +The previous two sections described the various `assert`, `precondition` and `assume` APIs all of which depend on the notion of "the same serial execution context". By default, every actor gets its own serial executor instance, and each such instance is unique. Therefore without sharing executors, each actor's serial executor is unique to itself, and thus the `precondition` APIs would effectively check "are we on this _specific_ actor" even though the check is performed against the executor identity. + +#### Unique executors delegating to the same SerialExecutor + +There are two cases of checking "the same executor" that we'd like to discuss in this proposal. Firstly, even though some actors may want to share the a serial executor, sometimes developers may not want to receive this "different actors on same serial executor are in the same execution context" semantic for the various precondition checks. + +The solution here is in the way an executor may be implemented, and specifically, it is always possible to provide a _wrapper_ executor around another existing executor. This way we are able to assign unique executor identities, even if they would end up scheduling onto the same serial executor. As an example, this might look like this: + +```swift +final class SpecificThreadExecutor: SerialExecutor { ... } + +final class UniqueSpecificThreadExecutor: SerialExecutor { + let delegate: SpecificThreadExecutor + init(delegate: SpecificThreadExecutor) { + self.delegate = delegate + } + + func enqueue(_ job: consuming ExecutorJob) { + delegate.enqueue(job) + } + + func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(ordinary: self) + } +} + +actor Worker { + let unownedExecutor: UnownedSerialExecutor + + init(executor: SpecificThreadExecutor) { + let uniqueExecutor = UniqueSpecificThreadExecutor(delegate: executor) + self.unownedExecutor = uniqueExecutor.asUnownedSerialExecutor() + } + + func test(other: Worker) { + assert(self !== other) + assertOnActorExecutor(other) // expected crash. + // `other` has different unique executor, + // even through they both eventually delegate to the same + } +} +``` + +#### Different executors offering the same execution context + +We also introduce an optional extension to serial executor identity compatibility checking, which allows an executor to _participate_ in the check. This is in order to handle the inverse situation to what we just discussed: when different executors _are_ in fact the same exclusive serial execution context and _want to_ inform Swift runtime about this for the purpose of these assertion APIs. + +One example of an executor which may have different unique instances of executors, however they should behave as the same exclusive serial execution context are dispatch queues which have the ability to "target" a different queue. In other words, it is possible to have a dispatch queue `Q1` and `Q2` target the same queue `Qx` (or even the "main" dispatch queue). + +In order to facilitate this capability, when exposing the `UnownedSerialExecutor` for itself, the executor must use the `init(complexEquality:)` initializer: + +```swift +extension MyQueueExecutor { + public func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(complexEquality: self) + } +} +``` + +The unique initializer keeps the current semantics of "*if the executor pointers are the same, it is the same executor and exclusive execution context*" fast path of executor equality checking, however it adds a "deep check" code-path if the equality has failed. + +> The word "complex" was selected due to its meaning "consisting of many different and connected parts", which describes this feature very well. The various executors are able to form a complex network that may be necessary to be inspected in order to answer the "*is this the same context?*" question. + +When performing the "is this the same (or a compatible) serial execution context" checks, the Swift runtime first compares the raw pointers to the executor objects. If those are not equal and the executors in question have `complexEquality`, following some additional type-checks, the following `isSameExclusiveExecutionContext(other:)` method will be invoked: + +```swift +protocol SerialExecutor { + + // ... previously discussed protocol requirements ... + + /// If this executor has complex equality semantics, and the runtime needs to compare + /// two executors, it will first attempt the usual pointer-based equality check, + /// and if it fails it will compare the types of both executors, if they are the same, + /// it will finally invoke this method, in an attempt to let the executor itself decide + /// if this and the `other` executor represent the same serial, exclusive, isolation context. + /// + /// This method must be implemented with great care, as wrongly returning `true` would allow + /// code from a different execution context (e.g. thread) to execute code which was intended + /// to be isolated by another actor. + /// + /// This check is not used when performing executor switching. + /// + /// This check is used when performing `preconditionTaskOnActorExecutor`, `preconditionTaskOnActorExecutor`, + /// `assumeOnActorExecutor` and similar APIs which assert about the same "exclusive serial execution context". + /// + /// - Parameter other: + /// - Returns: true, if `self` and the `other` executor actually are mutually exclusive and it is safe–from a concurrency perspective–to execute code assuming one on the other. + func isSameExclusiveExecutionContext(other: Self) -> Bool +} + +extension SerialExecutor { + func isSameExclusiveExecutionContext(other: Self) -> Bool { + self === other + } +} +``` + +This API allows for executor, like for example dispatch queues in the future, to perform the "deep" check and e.g. return true if both executors are actually targeting the same thread or queue, and therefore guaranteeing a properly isolated mutually exclusive serial execution context. + +The API explicitly enforces that both executors must be of the same type, in order to avoid comparing completely unrelated executors using this rather expensive call into user code. The concrete logic for comparing executors for the purpose of the above described APIs is as follows: + +We inspect at the type of the executor (the bit we store in the ExecutorRef, specifically in the Implementation / witness table field), and if both are: + +- "**ordinary**" (or otherwise known as "unique"), which can be be thought of as definitely a "root" executor + - creation: + - using today's `UnownedSerialExecutor.init(ordinary:)` + - comparison: + - compare the two executors pointers directly + - return the result +- **complexEquality**, may be thought of as "**inner**" executor, i.e. one that's exact identity may need deeper introspection + - creation: + - `UnownedSerialExecutor(complexEquality:)` which sets specific bits that the runtime can recognize and enter the complex comparison code-path when necessary + - comparison: + - compare the two executor pointers directly, + - if they are the same, return true (same as in the "ordinary" case) + - check if the *target* executor has `complexEquality`, we check if the current executors have compatible witness tables + - if not, we return false + - invoke the executor implemented comparison the `currentExecutor.isSameExclusiveExecutionContext(expectedExecutor)` + - return the result + +These checks are likely *not* enough to to completely optimize task switching, and other mechanisms will be provided for optimized task switching in the future (see Future Directions). + +### Default Swift Runtime Executors + +Swift Concurrency provides a number of default executors already, such as: + +- the main actor executor, which services any code annotated using @MainActor, and +- the default global concurrent executor, which all (default) actors target by their own per-actor instantiated serial executor instances. + +The `MainActor`'s executor is available via the `sharedUnownedExecutor` static property on the `MainActor`: + +```swift +@globalActor public final actor MainActor: GlobalActor { + public nonisolated var unownedExecutor: UnownedSerialExecutor { get { ... } } + public static var sharedUnownedExecutor: UnownedSerialExecutor { get { ... } } +} +``` + +So putting other actors onto the same executor as the MainActor is executing on, is possible using the following pattern: + +```swift +actor Friend { + nonisolated var unownedExecutor: UnownedSerialExecutor { + MainActor.sharedUnownedExecutor + } +} +``` + +Note that the raw type of the MainActor executor is never exposed, but we merely get unowned wrappers for it. This allows the Swift runtime to pick various specific implementations depending on the runtime environment. + +The default global concurrent executor is not accessible directly from code, however it is the executor that handles all the tasks which do not have a specific executor requirement, or are explicitly required to run on that executor, e.g. like top-level async functions. + +## Source compatibility + +Many of these APIs are existing public types since the first introduction of Swift Concurrency (and are included in back-deployment libraries). As all types and pieces of this proposal are designed in a way that allows to keep source and behavioral compatibility with already existing executor APIs. + +Special affordances are taken to introduce the move-only ExecutorJob based enqueue API in an source compatible way, while deprecating the previously existing ("unowned") API. + +## Effect on ABI stability + +Swift's concurrency runtime has already been using executors, jobs and tasks since its first introduction, as such, this proposal remains ABI compatible with all existing runtime entry points and types. + +The design of `SerialExecutor` currently does not support non-reentrant actors, and it does not support executors for which dispatch is always synchronous (e.g. that just acquire a traditional mutex). + +Some of the APIs discussed in this proposal existed from the first introduction of Swift Concurrency, so making any breaking changes to them is not possible. Some APIs were carefully renamed and polished up though. We encourage discussion of all the types and methods present in this proposal, however changing some of them may prove to be challenging or impossible due to ABI impact. + +## Effect on API resilience + +While some APIs may depend on being executed on particular executors, this proposal makes no effort to formalize that in interfaces, as opposed to being an implementation detail of implementations, and so has no API resilience implications. + +If this is extended in the future to automatic, declaration-driven executor switching, as actors do, that would have API resilience implications. + +## Alternatives considered + +The proposed ways for actors to opt in to custom executors are brittle, in the sense that a typo or some similar error could accidentally leave the actor using the default executor. This could be fully mitigated by requiring actors to explicitly opt in to using the default executor; however, that would be an unacceptable burden on the common case. Short of that, it would be possible to have a modifier that marks a declaration as having special significance, and then complain if the compiler doesn't recognize that significance. However, there are a number of existing features that use a name-sensitive design like this, such as dynamic member lookup ([SE-0195](https://github.com/rjmccall/swift-evolution/blob/custom-executors/proposals/0195-dynamic-member-lookup.md)). A "special significance" modifier should be designed and considered more holistically. + +## Future Directions + +### Overriding the MainActor executor + +Because of the special semantics of `MainActor` as well as its interaction with an asynchronous `main` function, customizing its serial executor is slightly more tricky than customizing any other executor. We must both guarantee that the main function of a program runs on the main thread, and that any `MainActor` code also gets to run on the main thread. This also introduces interesting complications with the main function actually returning an exit code. + +It should be possible to override the serial executor used by the the asynchronous `main` method, as well as the `MainActor`. While the exact semantics remain to be designed, we envision an API that allows replacing the main executor before any asynchronous work has happened, and this way uphold the serial execution guarantees expected from the main actor. + +```swift +// DRAFT; Names of protocols or exact shape of such replacement API are non-final. + +protocol MainActorSerialExecutor: [...]SerialExecutor { ... } +func setMainActorExecutor(_ executor: some MainActorSerialExecutor) { ... } + +@main struct Boot { + func main() async { + // + + // The following call must be made: + // - before any suspension point is encountered + // - before + setMainActorExecutor(...) + + // + await hello() // give control of the "raw" main thread to the RunLoopSerialExecutor + // still main thread, but executing on the selected MainActorExecutor + } +} + +@MainActor +func hello() { + // guaranteed to be MainActor (main thread), + // executed on the selected main actor executor + print("Hello") +} +``` + +### Executor Switching + +Executor switching is the capability to avoid un-necessary thread hops, when attempting to hop between actors/executors, where the target executor is compatible with "taking over" the calling thread. This allows Swift to optimize for less thread hops and scheduling calls. E.g. if actors are scheduled on the same executor identity and they are compatible with switching, it is possible to avoid thread-hops entirely and execution can "follow the Task" through multiple executors. + +The early sketch of switching focused around adding the following methods to the executor protocols: + +```swift +// DRAFT; Names and specific APIs mentioned in this snippet are non-final. + +protocol SerialExecutor: Executor { + // .... existing APIs ... + + /// Is it possible for this executor to give up the current thread + /// and allow it to start running a different actor? + var canGiveUpThread: Bool { get } + + /// Given that canGiveUpThread() previously returned true, give up + /// the current thread. + func giveUpThread() + + /// Attempt to start running a task on the current actor. Returns + /// true if this succeeds. + func tryClaimThread() -> Bool +} +``` + +We will consider adding these, or similar, APIs to enable custom executors to participate in efficient switching, when we are certain these API shapes are "enough" to support all potential use-cases for this feature. + +### Specifying Task executors + +Specifying executors to tasks has a surprising number of tricky questions it has to answer, so for the time being we are not introducing such capability. Specifically, passing an executor to `Task(startingOn: someExecutor) { ... }` would make the Task _start_ on the specified executor, but detailed semantics about if the _all_ of this Task's body is expected to execute on `someExecutor` (i.e. we have to hop-back to it every time after an `await`), or if it is enough to just start on it and then continue avoiding scheduling more jobs if possible (i.e. allow for aggressive switching). + +### DelegateActor property + +The previous pitch of custom executors included a concept of a `delegateActor` which allowed an actor to declare a `delegateActor: Actor` property which would allow given actor to execute on the same executor as another actor instance. At the same time, this would provide enough information to the compiler at compile time, that both actors can be assumed to be within the same isolation domain, and `await`s between those actors could be skipped (!). A property that with custom executors holds dynamically, would this way be reinforced statically by the compiler and type-system. diff --git a/proposals/0393-parameter-packs.md b/proposals/0393-parameter-packs.md new file mode 100644 index 0000000000..3102304563 --- /dev/null +++ b/proposals/0393-parameter-packs.md @@ -0,0 +1,878 @@ +# Value and Type Parameter Packs + +* Proposal: [SE-0393](0393-parameter-packs.md) +* Authors: [Holly Borla](https://github.com/hborla), [John McCall](https://github.com/rjmccall), [Slava Pestov](https://github.com/slavapestov) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.9)** +* Review: ([pitch 1](https://forums.swift.org/t/pitch-parameter-packs/60543)) ([pitch 2](https://forums.swift.org/t/pitch-2-value-and-type-parameter-packs/60830)) ([review](https://forums.swift.org/t/se-0393-value-and-type-parameter-packs/63859)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0393-value-and-type-parameter-packs/64382)) + +## Introduction + +Many modern Swift libraries include ad-hoc variadic APIs with an arbitrary upper bound, typically achieved with overloads that each have a different fixed number of type parameters and corresponding arguments. Without variadic generic programming support in the language, these ad-hoc variadic APIs have a significant cost on library maintenance and the developer experience of using these APIs. + +This proposal adds _type parameter packs_ and _value parameter packs_ to enable abstracting over the number of types and values with distinct type. This is the first step toward variadic generics in Swift. + +## Contents + +- [Value and Type Parameter Packs](#value-and-type-parameter-packs) + - [Introduction](#introduction) + - [Contents](#contents) + - [Motivation](#motivation) + - [Proposed solution](#proposed-solution) + - [Detailed design](#detailed-design) + - [Type parameter packs](#type-parameter-packs) + - [Pack expansion type](#pack-expansion-type) + - [Type substitution](#type-substitution) + - [Single-element pack substitution](#single-element-pack-substitution) + - [Type matching](#type-matching) + - [Label matching](#label-matching) + - [Trailing closure matching](#trailing-closure-matching) + - [Type list matching](#type-list-matching) + - [Member type parameter packs](#member-type-parameter-packs) + - [Generic requirements](#generic-requirements) + - [Same-shape requirements](#same-shape-requirements) + - [Restrictions on same-shape requirements](#restrictions-on-same-shape-requirements) + - [Value parameter packs](#value-parameter-packs) + - [Overload resolution](#overload-resolution) + - [Effect on ABI stability](#effect-on-abi-stability) + - [Alternatives considered](#alternatives-considered) + - [Modeling packs as tuples with abstract elements](#modeling-packs-as-tuples-with-abstract-elements) + - [Syntax alternatives to `repeat each`](#syntax-alternatives-to-repeat-each) + - [The `...` operator](#the--operator) + - [Another operator](#another-operator) + - [Magic builtin `map` method](#magic-builtin-map-method) + - [Future directions](#future-directions) + - [Variadic generic types](#variadic-generic-types) + - [Local value packs](#local-value-packs) + - [Explicit type pack syntax](#explicit-type-pack-syntax) + - [Pack iteration](#pack-iteration) + - [Pack element projection](#pack-element-projection) + - [Dynamic pack indexing with `Int`](#dynamic-pack-indexing-with-int) + - [Typed pack element projection using key-paths](#typed-pack-element-projection-using-key-paths) + - [Value expansion operator](#value-expansion-operator) + - [Pack destructuring operations](#pack-destructuring-operations) + - [Tuple conformances](#tuple-conformances) + - [Revision history](#revision-history) + - [Acknowledgments](#acknowledgments) + +## Motivation + +Generic functions currently require a fixed number of type parameters. It is not possible to write a generic function that accepts an arbitrary number of arguments with distinct types, instead requiring one of the following workarounds: + +* Erasing all of the types involved, e.g. using `Any...` +* Using a single tuple type argument instead of separate type arguments +* Overloading for each argument length with an artificial limit + +One example in the Swift Standard Library is the 6 overloads for each tuple comparison operator: + +```swift +func < (lhs: (), rhs: ()) -> Bool + +func < (lhs: (A, B), rhs: (A, B)) -> Bool where A: Comparable, B: Comparable + +func < (lhs: (A, B, C), rhs: (A, B, C)) -> Bool where A: Comparable, B: Comparable, C: Comparable + +// and so on, up to 6-element tuples +``` + +With language support for a variable number of type parameters, this API could be expressed more naturally and concisely as a single function declaration: + +```swift +func < (lhs: (repeat each Element), rhs: (repeat each Element)) -> Bool +``` + +## Proposed solution + +This proposal adds support for generic functions which abstract over a variable number of type parameters. While this proposal is useful on its own, there are many future directions that build upon this concept. This is the first step toward equipping Swift programmers with a set of tools that enable variadic generic programming. + +Parameter packs are the core concept that facilitates abstracting over a variable number of parameters. A pack is a new kind of type-level and value-level entity that represents a list of types or values, and it has an abstract length. A type parameter pack stores a list of zero or more type parameters, and a value parameter pack stores a list of zero or more value parameters. A type parameter pack is declared in angle brackets using the `each` contextual keyword: + +```swift +// 'S' is a type parameter pack where each pack element conforms to 'Sequence'. +func zip(...) +``` + +A parameter pack itself is not a first-class value or type, but the elements of a parameter pack can be used anywhere that naturally accepts a list of values or types using _pack expansions_, including top-level expressions. + +A pack expansion consists of the `repeat` keyword followed by a type or an expression. The type or expression that `repeat` is applied to is called the _repetition pattern_. The repetition pattern must contain at least one pack reference, spelled with the `each` keyword. At runtime, the pattern is repeated for each element in the substituted pack, and the resulting types or values are _expanded_ into the list provided by the surrounding context. + +Similarly, pack references can only appear inside repetition patterns and generic requirements: + +```swift +func zip(_ sequence: repeat each S) where repeat each S: Sequence +``` + +Given a concrete pack substitution, the pattern is repeated for each element in the substituted pack. If `S` is substituted with `Array, Set`, then `repeat Optional` will repeat the pattern `Optional` for each element in the substitution to produce `Optional>, Optional>`. + +Here are the key concepts introduced by this proposal: + +- Under the new model, all existing types and values in the language become _scalar types_ and _scalar values_. +- A _type pack_ is a new kind of type-level entity which represents a list of scalar types. Type packs do not have syntax in the surface language, but we will write them as `{T1, ..., Tn}` where each `Ti` is a scalar type. Type packs cannot be nested; type substitution is defined to always flatten type packs. +- A _type parameter pack_ is a list of zero or more scalar type parameters. These are declared in a generic parameter list using the syntax `each T`, and referenced with `each T`. +- A _value pack_ is a list of scalar values. The type of a value pack is a type pack, where each element of the type pack is the scalar type of the corresponding scalar value. Value packs do not have syntax in the surface language, but we will write them as `{x1, ..., xn}` where each `xi` is a scalar value. Value packs cannot be nested; evaluation is always defined to flatten value packs. +- A _value parameter pack_ is a list of zero or more scalar function or macro parameters. +- A _pack expansion_ is a new kind of type-level and value-level construct that expands a type or value pack into a list of types or values, respectively. Written as `repeat P`, where `P` is the _repetition pattern_ that captures at least one type parameter pack (spelled with the `each` keyword). At runtime, the pattern is repeated for each element in the substituted pack. + +The following example demonstrates these concepts together: + +```swift +struct Pair { + init(_ first: First, _ second: Second) +} + +func makePairs( + firsts first: repeat each First, + seconds second: repeat each Second +) -> (repeat Pair) { + return (repeat Pair(each first, each second)) +} + +let pairs = makePairs(firsts: 1, "hello", seconds: true, 1.0) +// 'pairs' is '(Pair(1, true), Pair("hello", 2.0))' +``` + +The `makePairs` function declares two type parameter packs, `First` and `Second`. The value parameter packs `first` and `second` have the pack expansion types `repeat each First` and `repeat each Second`, respectively. The return type `(repeat Pair)` is a tuple type where each element is a `Pair` of elements from the `First` and `Second` parameter packs at the given tuple position. + +Inside the body of `makePairs()`, `repeat Pair(each first, each second)` is a pack expansion expression referencing the value parameter packs `first` and `second`. + +The call to `makePairs()` substitutes the type pack `{Int, Bool}` for `First`, and the type pack `{String, Double}` for `Second`. These substitutions are deduced by the _type matching rules_, described below. The function is called with four arguments; `first` is the value pack `{1, "hello"}`, and `second` is the value pack `{true, 2.0}`. + +The substituted return type is the tuple type with two elements `(Pair, Pair)`, and the returned value is the tuple value with two elements `(Pair(1, true), Pair("hello", 2.0))`. + +## Detailed design + +**Note:** While this proposal talks about "generic functions", everything also applies to initializers and subscripts nested inside types. With closure expressions, the situation is slightly more limited. Closure expressions support value parameter packs, however since closure expressions do not have polymorphic types in Swift, they're limited to referencing type parameter packs from outer scopes and cannot declare type parameter packs of their own. Also, the value parameter packs of closures cannot have argument labels, because as usual only named declarations have argument labels in Swift. + +### Type parameter packs + +The generic parameter list of a generic function can contain one or more _type parameter pack declarations_, written as an identifier preceded by `each`: + +```swift +func variadic() {} +``` + +When referenced from type context, this identifier resolves to a _type parameter pack_. References to type parameter packs can only appear in the following positions: + +* The base type of a member type parameter pack, which is again subject to these rules +* The pattern type of a pack expansion type, where it stands for the corresponding scalar element type +* The pattern expression of a pack expansion expression, where it stands for the metatype of the corresponding scalar element type and can be used like any other scalar metatype, e.g. to call a static method, call an initializer, or reify the metatype value +* The subject type of a conformance, superclass, layout, or same-type requirement +* The constraint type of a same-type requirement + +### Pack expansion type + +A pack expansion type, written as `repeat P`, has a *pattern type* `P` and a non-empty set of _captured_ type parameter packs spelled with the `each` keyword. For example, the pack expansion type `repeat Array` has a pattern type `Array` that captures the type parameter pack `T`. + +**Syntactic validity:** Pack expansion types can appear in the following positions: + +* The type of a parameter in a function declaration, e.g. `func foo(values: repeat each T) -> Bool` +* The type of a parameter in a function type, e.g. `(repeat each T) -> Bool` +* The type of an unlabeled element in a tuple type, e.g. `(repeat each T)` + +Because pack expansions can only appear in positions that accept a list of types or values, pack expansion patterns are naturally delimited by a comma, the next statement in top-level code, or an end-of-list delimiter, e.g. `)` for call argument lists or `>` for generic argument lists. + +The restriction where only unlabeled elements of a tuple type may have a pack expansion type is motivated by ergonomics. If you could write `(t: repeat each T)`, then after a substitution `T := {Int, String}`, the substituted type would be `(t: Int, String)`. This would be strange, because projecting the member `t` would only produce the first element. When an unlabeled element has a pack expansion type, like `(repeat each T)`, then after the above substitution you would get `(Int, String)`. You can still write `0` to project the first element, but this is less surprising to the Swift programmer. + +**Capture:** A type _captures_ a type parameter pack if the type parameter pack appears inside the pattern type, without any intervening pack expansion type. For example, if `T` and `U` are type parameter packs, then `repeat Array<(each T) -> each U>` captures both `T` and `U`. However, `repeat Array<(each T) -> (repeat each U)>` captures `T`, but *not* `U`. Only the inner pack expansion type `repeat each U` captures `U`. (Indeed, in a valid program, every reference to a type parameter pack is captured by exactly one pack expansion type.) + +The captures of the pattern type are a subset of the captures of the pack expansion type itself. In some situations (described in the next section), the pack expansion type might capture a type parameter pack that does not appear in the pattern type. + +**Typing rules:** A pack expansion type is _well-typed_ if replacing the captured type parameter packs in the pattern type with scalar type parameters of the same constraints produces a well-typed scalar type. + +For example, if `each T` is a type parameter pack subject to the conformance requirement `each T: Hashable`, then `repeat Set` is well-typed, because `Set` is well-typed given `T: Hashable`. + +However, if `each T` were not subject to this conformance requirement, then `repeat Set` would not be well-typed; the user might substitute `T` with a type pack containing types that do not conform to `Hashable`, like `T := {AnyObject, Int}`, and the expanded substitution `Set, Set` is not well-typed because `Set` is not well-typed. + +### Type substitution + +Recall that a reference to a generic function from expression context always provides an implicit list of *generic arguments* which map each of the function's type parameters to a *replacement type*. The type of the expression referencing a generic declaration is derived by substituting each type parameter in the declaration's type with the corresponding replacement type. + +The replacement type of a type parameter pack is always a type pack. Since type parameter packs always occur inside the pattern type of a pack expansion type, we need to define what it means to perform a substitution on a type that contains pack expansion types. + +Recall that pack expansion types appear in function parameter types and tuple types. Substitution replaces each pack expansion type with an expanded type list, which is flattened into the outer type list. + +**Intuition:** The substituted type list is formed by replacing the captured type parameter pack references with the corresponding elements of each replacement type pack. + +For example, consider the declaration: + +```swift +func variadic( + t: repeat each T, + u: repeat each U +) -> (Int, repeat ((each T) -> each U)) +``` + +Suppose we reference it with the following substitutions: + +```swift +T := {String, repeat each V, Float} +U := {NSObject, repeat Array, NSString} +``` + +The substituted return type of `variadic` becomes a tuple type with 4 elements: + +```swift +(Int, (String) -> NSObject, repeat ((each V) -> Array), (Float) -> NSString) +``` + +**Formal algorithm:** Suppose `repeat P` is a pack expansion type with pattern type `P`, that captures a list of type parameter packs `Ti`, and let `S[Ti]` be the replacement type pack for `Ti`. We require that each `S[Ti]` has the same length; call this length `N`. If the lengths do not match, the substitution is malformed. Let `S[Ti][j]` be the `j`th element of `S[Ti]`, where `0 ≤ j < N`. + +The `j`th element of the replacement type list is derived as follows: + +1. If each `S[Ti][j]` is a scalar type, the element type is obtained by substituting each `Ti` with `S[Ti][j]` in the pattern type `P`. +2. If each `S[Ti][j]` is a pack expansion type, then `S[Ti][j]` = `repeat Pij` for some pattern type `Pij`. The element type is the pack expansion type `repeat Qij`, where `Qij` is obtained by substituting each `Ti` with `Pij` in the pattern type `P`. +3. Any other combination means the substitution is malformed. + +When the lengths or structure of the replacement type packs do not match, the substitution is malformed. This situation is diagnosed with an error by checking generic requirements, as discussed below. + +For example, the following substitutions are malformed because the lengths do not match: + +```swift +T := {String, Float} +U := {NSObject} +``` + +The following substitutions are malformed because the replacement type packs have incompatible structure, hitting Case 3 above: + +```swift +T := {repeat each V, Float} +U := {NSObject, repeat each W} +``` + +To clarify what it means for a type to capture a type parameter pack, consider the following: + +```swift +func variadic(t: repeat each T, u: repeat each U) -> (repeat (each T) -> (repeat each U)) +``` + +The pack expansion type `repeat (each T) -> (repeat each U)` captures `T`, but not `U`. If we apply the following substitutions: + +```swift +T := {Int, String} +U := {Float, Double, Character} +``` + +Then the substituted return type becomes a pair of function types: + +```swift +((Int) -> (Float, Double, Character), (String) -> (Float, Double, Character)) +``` + +Note that the entire replacement type pack for `U` was flattened in each repetition of the pattern type; we did not expand "across" `U`. + +**Concrete pattern type:** It is possible to construct an expression with a pack expansion type whose pattern type does not capture any type parameter packs. This is called a pack expansion type with a _concrete_ pattern type. For example, consider this declaration: + +```swift +func counts(_ t: repeat each T) { + let x = (repeat (each t).count) +} +``` + +The `count` property on the `Collection` protocol returns `Int`, so the type of the expression `(repeat (each t).count)` is written as the one-element tuple type `(repeat Int)` whose element is the pack expansion type `repeat Int`. While the pattern type `Int` does not capture any type parameter packs, the pack expansion type must still capture `T` to represent the fact that after expansion, the resulting tuple type has the same length as `T`. This kind of pack expansion type can arise during type inference, but it cannot be written in source. + +#### Single-element pack substitution + +If a parameter pack `each T` is substituted with a single element, the parenthesis around `(repeat each T)` are unwrapped to produce the element type as a scalar instead of a one-element tuple type. + +For example, the following substitutions both produce the element type `Int`: +- Substituting `each T := {Int}` into `(repeat each T)`. +- Substituting `each T := {}` into `(Int, repeat each T)`. + +Though unwrapping single-element tuples complicates type matching, surfacing single-element tuples in the programming model would increase the surface area of the language. One-element tuples would need to be manually unwrapped with `.0` or pattern matching in order to make use of their contents. This unwrapping would clutter up code. + + +### Type matching + +Recall that the substitutions for a reference to a generic function are derived from the types of call argument expressions together with the contextual return type of the call, and are not explicitly written in source. This necessitates introducing new rules for _matching_ types containing pack expansions. + +There are two separate rules: + +- For call expressions where the callee is a named function declaration, _label matching_ is performed. +- For everything else, _type list matching_ is performed. + +#### Label matching + +Here, we use the same rule as the "legacy" variadic parameters that exist today. If a function declaration parameter has a pack expansion type, the parameter must either be the last parameter, or followed by a parameter with a label. A diagnostic is produced if the function declaration violates this rule. + +Given a function declaration that is well-formed under this rule, type matching then uses the labels to delimit type packs. For example, the following is valid: + + ```swift + func concat(t: repeat each T, u: repeat each U) -> (repeat each T, repeat each U) + + // T := {Int, Double} + // U := {String, Array} + concat(t: 1, 2.0, u: "hi", [3]) + + // substituted return type is (Int, Double, String, Array) + ``` + + while the following is not: + + ```swift + func bad(t: repeat each T, repeat each U) -> (repeat each T, repeat each U) + // error: 'repeat each T' followed by an unlabeled parameter + + bad(1, 2.0, "hi", [3]) // ambiguous; where does 'each T' end and 'each U' start? + ``` + +#### Trailing closure matching + +Argument-to-parameter matching for parameter pack always uses a forward-scan for trailing closures. For example, the following code is valid: + +```swift +func trailing(t: repeat each T, u: repeat each U) {} + +// T := {() -> Int} +// U := {} +trailing { 0 } +``` + +while the following produces an error: + +```swift +func trailing(t: repeat each T, u: repeat each U) {} + +// error: type '() -> Int' cannot conform to 'Sequence' +trailing { 0 } +``` + +#### Type list matching + +In all other cases, we're matching two comma-separated lists of types. If either list contains two or more pack expansion types, the match remains _unsolved_, and the type checker attempts to derive substitutions by matching other types before giving up. (This allows a call to `concat()` as defined above to succeed, for example; the match between the contextual return type and `(repeat each T, repeat each U)` remains unsolved, but we are able to derive the substitutions for `T` and `U` from the call argument expressions.) + +Otherwise, we match the common prefix and suffix as long as no pack expansion types appear on either side. After this has been done, there are three possibilities: + +1. Left hand side contains a single pack expansion type, right hand size contains zero or more types. +2. Left hand side contains zero or more types, right hand side contains a single pack expansion type. +3. Any other combination, in which case the match fails. + +For example: + +```swift +func variadic(_: repeat each T) -> (Int, repeat each T, String) {} + +let fn = { x, y in variadic(x, y) as (Int, Double, Float, String) } +``` + +Case 3 covers the case where one of the lists has a pack expansion, but the other one is too short; for example, matching `(Int, repeat each T, String, Float)` against `(Int, Float)` leaves you with `(repeat each T, String)` vs `()`, which is invalid. + +If neither side contains a pack expansion type, Case 3 also subsumes the current behavior as implemented without this proposal, where type list matching always requires the two type lists to have the same length. For example, when matching `(Int, String)` against `(Int, Float, String)`, we end up with `()` vs `(Float)`, which is invalid. + +The type checker derives the replacement type for `T` in the call to `variadic()` by matching the contextual return type `(Int, Double, Float, String)` against the declared return type `(Int, repeat each T, String)`. The common prefix `Int` and common suffix `String` successfully match. What remains is the pack expansion type `repeat each T` and the type list `Double, Float`. This successfully matches, deriving the substitution `T := {Double, Float}`. + +While type list matching is positional, the type lists may still contain labels if we're matching two tuple types. We require the labels to match exactly when dropping the common prefix and suffix, and then we only allow Case 1 and 2 to succeed if the remaining type lists do not contain any labels. + +For example, matching `(x: Int, repeat each T, z: String)` against `(x: Int, Double, y: Float, z: String)` drops the common prefix and suffix, and leaves you with the pack expansion type `repeat each T` vs the type list `Double, y: Float`, which fails because `Double: y: Float` contains a label. + +However, matching `(x: Int, repeat each T, z: String)` against `(x: Int, Double, Float, z: String)` leaves you with `repeat each T` vs `Double, Float`, which succeeds with `T := {Double, Float}`, because the labels match exactly in the common prefix and suffix, and no labels remain once we get to Case 1 above. + +### Member type parameter packs + +If a type parameter pack `each T` is subject to a protocol conformance requirement `P`, and `P` declares an associated type `A`, then `(each T).A` is a valid pattern type for a pack expansion type, called a _member type parameter pack_. + +Under substitution, a member type parameter pack projects the associated type from each element of the replacement type pack. + +For example: + +```swift +func variadic(_: repeat each T) -> (repeat (each T).Element) +``` + +After the substitution `T := {Array, Set}`, the substituted return type of this function becomes the tuple type `(Int, String)`. + +We will refer to `each T` as the _root type parameter pack_ of the member type parameter packs `(each T).A` and `(each T).A.B`. + +### Generic requirements + +All existing kinds of generic requirements can be used inside _requirement expansions_, which represent a list of zero or more requirements. Requirement expansions are spelled with the `repeat` keyword followed by a generic requirement pattern that captures at least one type parameter pack reference spelled with the `each` keyword. Same-type requirements generalize in multiple different ways, depending on whether one or both sides involve a type parameter pack. + +1. Conformance, superclass, and layout requirements where the subject type is a type parameter pack are interpreted as constraining each element of the replacement type pack: + + ```swift + func variadic(_: repeat each S) where repeat each S: Sequence { ... } + ``` + + A valid substitution for the above might replace `S` with `{Array, Set}`. Expanding the substitution into the requirement `each S: Sequence` conceptually produces the following conformance requirements: `Array: Sequence, Set: Sequence`. + +1. A same-type requirement where one side is a type parameter pack and the other type is a scalar type that does not capture any type parameter packs is interpreted as constraining each element of the replacement type pack to _the same_ scalar type: + + ```swift + func variadic(_: repeat each S) where repeat (each S).Element == T {} + ``` + + This is called a _same-element requirement_. + + A valid substitution for the above might replace `S` with `{Array, Set}`, and `T` with `Int`. + + +3. A same-type requirement where each side is a pattern type that captures at least one type parameter pack is interpreted as expanding the type packs on each side of the requirement, equating each element pair-wise. + + ```swift + func variadic(_: repeat each S) where repeat (each S).Element == Array {} + ``` + + This is called a _same-type-pack requirement_. + + A valid substitution for the above might replace `S` with `{Array>, Set>}`, and `T` with `{Int, String}`. Expanding `(each S).Element == Array` will produce the following list of same-type requirements: `Array.Element == Array, Set>.Element == String`. + +There is an additional kind of requirement called a _same-shape requirement_. There is no surface syntax for spelling a same-shape requirement; they are always inferred, as described in the next section. + +**Symmetry:** Recall that same-type requirements are symmetrical, so `T == U` is equivalent to `U == T`. Therefore some of the possible cases above are not listed, but the behavior can be understood by first transposing the same-type requirement. + +**Constrained protocol types:** A conformance requirement where the right hand side is a constrained protocol type `P` may reference type parameter packs from the generic arguments `Ti` of the constrained protocol type. In this case, the semantics are defined in terms of the standard desugaring. Independent of the presence of type parameter packs, a conformance requirement to a constrained protocol type is equivalent to a conformance requirement to `P` together with one or more same-type requirements that constrain the primary associated types of `P` to the corresponding generic arguments `Ti`. After this desugaring step, the induced same-type requirements can then be understood by Case 2, 3, 4 or 5 above. + +#### Same-shape requirements + +A same-shape requirement states that two type parameter packs have the same number of elements, with pack expansion types occurring at identical positions. + +This proposal does not include a spelling for same-shape requirements in the surface language; same-shape requirements are always inferred, and an explicit same-shape requirement syntax is a future direction. However, we will use the notation `shape(T) == shape(U)` to denote same-shape requirements in this proposal. + +A same-shape requirement always relates two root type parameter packs. Member types always have the same shape as the root type parameter pack, so `shape(T.A) == shape(U.B)` reduces to `shape(T) == shape(U)`. + +**Inference:** Same-shape requirements are inferred in the following ways: + +1. A same-type-pack requirement implies a same-shape requirement between all type parameter packs captured by the pattern types on each side of the requirement. + + For example, given the parameter packs ``, the same-type-pack requirement `Pair == (each S).Element` implies `shape(First) == shape(Second), shape(First) == shape(S), shape(Second) == shape(S)`. + +2. A same-shape requirement is inferred between each pair of type parameter packs captured by a pack expansion type appearing in the following positions + * all types appearing in the requirements of a trailing `where` clause of a generic function + * the parameter types and the return type of a generic function + +Recall that if the pattern of a pack expansion type contains more than one type parameter pack, all type parameter packs must be known to have the same shape, as outlined in the [Type substitution](#type-substitution) section. Same-shape requirement inference ensures that these invariants are satisfied when the pack expansion type occurs in one of the two above positions. + +If a pack expansion type appears in any other position, all type parameter packs captured by the pattern type must already be known to have the same shape, otherwise an error is diagnosed. + +For example, `zip` is a generic function, and the return type `(repeat (each T, each U))` is a pack expansion type, therefore the same-shape requirement `shape(T) == shape(U)` is automatically inferred: + +```swift +// Return type infers 'where length(T) == length(U)' +func zip(firsts: repeat each T, seconds: repeat each U) -> (repeat (each T, each U)) { + return (repeat (each firsts, each seconds)) +} + +zip(firsts: 1, 2, seconds: "hi", "bye") // okay +zip(firsts: 1, 2, seconds: "hi") // error; length requirement is unsatisfied +``` + +Here is an example where the same-shape requirement is not inferred: + +```swift +func foo(t: repeat each T, u: repeat each U) { + let tup: (repeat (each T, each U)) = /* whatever */ +} +``` + +The type annotation of `tup` contains a pack expansion type `repeat (each T, each U)`, which is malformed because the requirement `shape(T) == shape(U)` is unsatisfied. This pack expansion type is not subject to requirement inference because it does not occur in one of the above positions. + +#### Restrictions on same-shape requirements + +Type packs cannot be written directly, but requirements involving pack expansions where both sides are concrete types are desugared using the type matching algorithm. This means it is possible to write down a requirement that constrains a type parameter pack to a concrete type pack, unless some kind of restriction is imposed: + +```swift +func constrain(_: repeat each S) where (repeat (each S).Element) == (Int, String) {} +``` + +Furthermore, since the same-type requirement implies a same-shape requirement, we've implicitly constrained `S` to having a length of 2 elements, without knowing what those elements are. + +This introduces theoretical complications. In the general case, same-type requirements on type parameter packs allows encoding arbitrary systems of integer linear equations: + +```swift +// shape(Q) = 2 * shape(R) + 1 +// shape(Q) = shape(S) + 2 +func solve(q: repeat each Q, r: repeat each R, s: repeat each S) + where (repeat each Q) == (Int, repeat each R, repeat each R), + (repeat each Q) == (repeat each S, String, Bool) { } +``` + +While type-level linear algebra is interesting, we may not ever want to allow this in the language to avoid significant implementation complexity, and we definitely want to disallow this expressivity in this proposal. + +To impose restrictions on same-shape and same-type requirements, we will formalize the concept of the “shape” of a pack, where a shape is one of: + +* A single scalar type element; all scalar types have a singleton ``scalar shape'' +* An abstract shape that is specific to a pack parameter +* A concrete shape that is composed of the scalar shape and abstract shapes + +For example, the pack `{Int, repeat each T, U}` has a concrete shape that consists of two single elements and one abstract shape. + +This proposal only enables abstract shapes. Each type parameter pack has an abstract shape, and same-shape requirements merge equivalence classes of abstract shapes. Any same-type requirement that imposes a concrete shape on a type parameter pack will be diagnosed as a *conflict*, much like other conflicting requirements such as `where T == Int, T == String` today. + +This aspect of the language can evolve in a forward-compatible manner. Over time, some restrictions can be lifted, while others remain, as different use-cases for type parameter packs are revealed. + +### Value parameter packs + +A _value parameter pack_ represents zero or more function or macro parameters, and it is declared with a function parameter that has a pack expansion type. In the following declaration, the function parameter `value` is a value parameter pack that receives a _value pack_ consisting of zero or more argument values from the call site: + +```swift +func tuplify(_ value: repeat each T) -> (repeat each T) + +_ = tuplify() // T := {}, value := {} +_ = tuplify(1) // T := {Int}, value := {1} +_ = tuplify(1, "hello", [Foo()]) // T := {Int, String, [Foo]}, value := {1, "hello", [Foo()]} +``` + +**Syntactic validity:** A value parameter pack can only be referenced from a pack expansion expression. A pack expansion expression is written as `repeat expr`, where `expr` is an expression containing one or more value parameter packs or type parameter packs spelled with the `each` keyword. Pack expansion expressions can appear in any position that naturally accepts a list of expressions, including comma-separated lists and top-level expressions. This includes the following: + +* Call arguments, e.g. `generic(repeat each value)` +* Subscript arguments, e.g. `subscriptable[repeat each index]` +* The elements of a tuple value, e.g. `(repeat each value)` +* The elements of an array literal, e.g. `[repeat each value]` + +Pack expansion expressions can also appear in an expression statement at the top level of a brace statement. In this case, the semantics are the same as scalar expression statements; the expression is evaluated for its side effect and the results discarded. + +**Capture:** A pack expansion expression _captures_ a value (or type) pack parameter the value (or type) pack parameter appears as a sub-expression without any intervening pack expansion expression. + +Furthermore, a pack expansion expression also captures all type parameter packs captured by the types of its captured value parameter packs. + +For example, say that `x` and `y` are both value parameter packs and `T` is a type parameter pack, and consider the pack expansion expression `repeat foo(each x, (each T).self, (repeat each y))`. This expression captures both the value parameter pack `x` and type parameter pack `T`, but it does not capture `y`, because `y` is captured by the inner pack expansion expression `repeat each y`. Additionally, if `x` has the type `Foo`, then our expression captures `U`, but not `V`, because again, `V` is captured by the inner pack expansion type `repeat each V`. + +**Typing rules:** For a pack expansion expression to be well-typed, two conditions must hold: + +1. The types of all value parameter packs captured by a pack expansion expression must be related via same-shape requirements. + +2. After replacing all value parameter packs with non-pack parameters that have equivalent types, the pattern expression must be well-typed. + +Assuming the above hold, the type of a pack expansion expression is defined to be the pack expansion type whose pattern type is the type of the pattern expression. + +**Evaluation semantics:** At runtime, each value (or type) parameter pack receives a value (or type) pack, which is a concrete list of values (or types). The same-shape requirements guarantee that all value (and type) packs have the same length, call it `N`. The evaluation semantics are that for each successive `i` such that `0 ≤ i < N`, the pattern expression is evaluated after substituting each occurrence of a value (or type) parameter pack with the `i`th element of the value (or type) pack. The evaluation proceeds from left to right according to the usual evaluation order, and the list of results from each evaluation forms the argument list for the parent expression. + +For example, pack expansion expressions can be used to forward value parameter packs to other functions: + +```swift +func tuplify(_ t: repeat each T) -> (repeat each T) { + return (repeat each t) +} + +func forward(u: repeat each U) { + let _ = tuplify(repeat each u) // T := {repeat each U} + let _ = tuplify(repeat each u, 10) // T := {repeat each U, Int} + let _ = tuplify(repeat each u, repeat each u) // T := {repeat each U, repeat each U} + let _ = tuplify(repeat [each u]) // T := {repeat Array} +} +``` + +### Overload resolution + +Generic functions can be overloaded by the "pack-ness" of their type parameters. For example, a function can have two overloads where one accepts a scalar type parameter and the other accepts a type parameter pack: + +```swift +func overload(_: T) {} +func overload(_: repeat each T) {} +``` + +If the parameters of the scalar overload have the same or refined requirements as the parameter pack overload, the scalar overload is considered a subtype of the parameter pack overload, because the parameters of the scalar overload can be forwarded to the parameter pack overload. Currently, if a function call successfully type checks with two different overloads, the subtype is preferred. This overload ranking rule generalizes to overloads with parameter packs, which effectively means that scalar overloads are preferred over parameter pack overloads when the scalar requirements meet the requirements of the parameter pack: + +```swift +func overload() {} +func overload(_: T) {} +func overload(_: repeat each T) {} + +overload() // calls the no-parameter overload + +overload(1) // calls the scalar overload + +overload(1, "") // calls the parameter pack overload +``` + +The general overload subtype ranking rule applies after localized ranking, such as implicit conversions and optional promotions. That remains unchanged with this proposal. For example: + +```swift +func overload(_: T, _: Any) {} +func overload(_: repeat each T) {} + +overload(1, "") // prefers the parameter pack overload because the scalar overload would require an existential conversion +``` + +More complex scenarios can still result in ambiguities. For example, if multiple overloads match a function call, but each parameter list can be forwarded to the other, the call is ambiguous: + +```swift +func overload(_: repeat each T) {} +func overload(vals: repeat each T) {} + +overload() // error: ambiguous +``` + +Similarly, if neither overload can forward their parameter lists to the other, the call is ambiguous: + +```swift +func overload(str: String, _: repeat each T) {} +func overload(str: repeat each U) {} + +func test(_ z: Z) { + overload(str: "Hello, world!", z, z) // error: ambiguous +} +``` + +Generalizing the existing overload resolution ranking rules to parameter packs enables library authors to introduce new function overloads using parameter packs that generalize existing fixed-arity overloads while preserving the overload resolution behavior of existing code. + +## Effect on ABI stability + +This is still an area of open discussion, but we anticipate that generic functions with type parameter packs will not require runtime support, and thus will backward deploy. As work proceeds on the implementation, the above is subject to change. + +## Alternatives considered + +### Modeling packs as tuples with abstract elements + +Under this alternative design, packs are just tuples with abstract elements. This model is attractive because it adds expressivity to all tuple types, but there are some significant disadvantages that make packs hard to work with: + +* There is a fundamental ambiguity between forwarding a tuple with its elements flattened and passing a tuple as a single tuple value. This could be resolved by requiring a splat operator to forward the flattened elements, but it would still be valid in many cases to pass the tuple without flattening the elements. This may become a footgun, because you can easily forget to splat a tuple, which will become more problematic when tuples can conform to protocols. +* Because of the above issue, there is no clear way to zip tuples. This could be solved by having an explicit builtin to treat a tuple as a pack, which leads us back to needing a distinction between packs and tuples. + +The pack parameter design where packs are distinct from tuples also does not preclude adding flexibility to all tuple types. Converting tuples to packs and expanding tuple values are both useful features and are detailed in the future directions. + +### Syntax alternatives to `repeat each` + +The `repeat each` syntax produces fairly verbose variadic generic code. However, the `repeat` keyword is explicit signal that the pattern is repeated under substitution, and requiring the `each` keyword for pack references indicates which types or values will be substituted in the expansion. This syntax design helps enforce the mental model that pack expansions result in iteration over each element in the parameter pack at runtime. + +The following syntax alternatives were also considered. + +#### The `...` operator + +A previous version of this proposal used `...` as the pack expansion operator with no explicit syntax for pack elements in pattern types. This syntax choice follows precedent from C++ variadic templates and non-pack variadic parameters in Swift. However, there are some serious downsides of this choice, because ... is already a postfix unary operator in the Swift standard library that is commonly used across existing Swift code bases, which lead to the following ambiguities: + +1. **Pack expansion vs non-pack variadic parameter.** Using `...` for pack expansions in parameter lists introduces an ambiguity with the use of `...` to indicate a non-pack variadic parameter. This ambiguity can arise when expanding a type parameter pack into the parameter list of a function type. For example: + +```swift +struct X { } + +struct Ambiguity { + struct Inner { + typealias A = X<((T...) -> U)...> + } +} +``` + +Here, the `...` within the function type `(T...) -> U` could mean one of two things: + +* The `...` defines a (non-pack) variadic parameter, so for each element `Ti` in the parameter pack, the function type has a single (non-pack) variadic parameter of type `Ti`, i.e., `(Ti...) -> Ui`. So, `Ambiguity.Inner.A` would be equivalent to `X<(String...) -> Float, (Character...) -> Double>`. +* The `...` expands the parameter pack `T` into individual parameters for the function type, and no pack parameters remain after expansion. Only `U` is expanded by the outer `...`. So, `Ambiguity.Inner.A` would be equivalent to `X<(String, Character) -> Float, (String, Character) -> Double>`. + +2. **Pack expansion vs postfix closed-range operator.** Using `...` as the value expansion operator introduces an ambiguity with the postfix closed-range operator. This ambiguity can arise when `...` is applied to a value pack in the pattern of a value pack expansion, and the values in the pack are known to have a postfix closed-range operator, such as in the following code which passes a list of tuple arguments to `acceptAnything`: + +```swift +func acceptAnything(_: T...) {} + +func ranges(values: T..., otherValues: U...) where T: Comparable, shape(T...) == shape(U...) { + acceptAnything((values..., otherValues)...) +} +``` + +In the above code, `values...` in the expansion pattern could mean either: + +* The postfix `...` operator is called on each element in `values`, and the result is expanded pairwise with `otherValues` such that each argument has type `(PartialRangeFrom, U)` +* `values` is expanded into each tuple passed to `acceptAnything`, with each element of `otherValues` appended onto the end of the tuple, and each argument has type `(T... U)` + + +3. **Pack expansion vs operator `...>`.** Another ambiguity arises when a pack expansion type `T...` appears as the final generic argument in the generic argument list of a generic type in expression context: + +```swift +let foo = Foo() +``` + +Here, the ambiguous parse is with the token `...>`, which would necessitate changing the grammar so that `...>` is no longer considered as a single token, and instead parses as the token `...` followed by the token `>`. + + +#### Another operator + +One alternative is to use a different operator, such as `*`, instead of `...` + +```swift +func zip(firsts: T*, seconds: U*) -> ((T, U)*) { + return ((firsts, seconds)*) +} +``` + +The downsides to postfix `*` include: + +* `*` is extremely subtle +* `*` evokes pointer types / a dereferencing operator to programmers familiar with other languages including C/C++, Go, Rust, etc. +* Choosing another operator does not alleviate the ambiguities in expressions, because values could also have a postfix `*` operator or any other operator symbol, leading to the same ambiguity. + +#### Magic builtin `map` method + +The prevalence of `map` and `zip` in Swift makes this syntax an attractive option for variadic generics: + +```swift +func wrap(_ values: repeat each T) -> (repeat Wrapped) { + return values.map { Wrapped($0) } +} +``` + +The downsides of a magic `map` method are: + +* `.map` isn't only used for mapping elements to a new element, it's also used for direct forwarding, which is very different to the way `.map` is used today. An old design exploration for variadic generics used a map-style builtin, but allowed exact forwarding to omit the `.map { $0 }`. Privileging exact forwarding would be pretty frustrating, because you would need to add `.map { $0 }` as soon as you want to append other elements to either side of the pack expansion, and it wouldn't work for other values that you might want to turn into packs such as tuples or arrays. +* There are two very different models for working with packs; the same conceptual expansion has very different spellings at the type and value level, `repeat Wrapped` vs `values.map { Wrapped($0) }`. +* Magic map can only be applied to one pack at a time, leaving no clear way to zip packs without adding other builtins. A `zip` builtin would also be misleading, because expanding two packs in parallel does not need to iterate over the packs twice, but using `zip(t, u).map { ... }` looks that way. +* The closure-like syntax is misleading because it’s not a normal closure that you can write in the language. This operation is also very complex over packs with any structure, including concrete types, because the compiler either needs to infer a common generic signature for the closure that works for all elements, or it needs to separately type check the closure once for each element type. +* `map` would still need to be resolved via overload resolution amongst the existing overloads, so this approach doesn't help much with the type checking issues that `...` has. + +## Future directions + +### Variadic generic types + +This proposal only supports type parameter packs on functions. A complementary proposal will describe type parameter packs on generic structs, enums and classes. + +### Local value packs + +This proposal only supports value packs for function parameters. The notion of a value parameter pack readily generalizes to a local variable of pack expansion type, for example: + +```swift +func variadic(t: repeat each T) { + let tt: repeat each T = repeat each t +} +``` + +References to `tt` have the same semantics as references to `t`, and must only appear inside other pack expansion expressions. + +### Explicit type pack syntax + +In this proposal, type packs do not have an explicit syntax, and a type pack is always inferred through the type matching rules. However, we could explore adding an explicit pack syntax in the future: + +```swift +struct Variadic {} + +extension Variadic where T == {Int, String} {} // {Int, String} is a concrete pack +``` + +### Pack iteration + +All list operations can be expressed using pack expansion expressions by factoring code involving statements into a function or closure. However, this approach does not allow for short-circuiting, because the pattern expression will always be evaluated once for every element in the pack. Further, requiring a function or closure for code involving statements is unnatural. Allowing `for-in` loops to iterate over packs solves both of these problems. + +Value packs could be expanded into the source of a `for-in` loop, allowing you to iterate over each element in the pack and bind each value to a local variable: + +```swift +func allEmpty(_ array: repeat [each T]) -> Bool { + for a in repeat each array { + guard a.isEmpty else { return false } + } + + return true +} +``` + +The type of the local variable `a` in the above example is an `Array` of an opaque element type with the requirements that are written on `each T`. For the *i*th iteration, the element type is the *i*th type parameter in the type parameter pack `T`. + +### Pack element projection + +Use cases for variadic generics that break up pack iteration across function calls, require random access, or operate over concrete packs can be supported in the future by projecting individual elements out from a parameter pack. Because elements of the pack have different types, there are two approaches to pack element projection; using an `Int` index which will return the dynamic type of the element, and using a statically typed index which is parameterized over the requested pack element type. + +#### Dynamic pack indexing with `Int` + +Dynamic pack indexing is useful when the specific type of the element is not known, or when all indices must have the same type, such as for index manipulation or storing an index value. Packs could support subscript calls with an `Int` index, which would return the dynamic type of the pack element directly as the opened underlying type that can be assigned to a local variable with opaque type. Values of this type need to be erased or cast to another type to return an element value from the function: + +```swift +func element(at index: Int, in t: repeat each T) -> any P { + // The subscript returns 'some P', which is erased to 'any P' + // based on the function return type. + let value: some P = t[index] + return value +} +``` + +#### Typed pack element projection using key-paths + +Some use cases for pack element projection know upfront which type within the pack will be projected, and can use a statically typed pack index. A statically typed pack index could be represented with `KeyPath` or a new `PackIndex` type, which is parameterized over the base type for access (i.e. the pack), and the resulting value type (i.e. the element within the pack to project). Pack element projection via key-paths falls out of 1) positional tuple key-paths, and 2) expanding packs into tuple values: + +```swift +struct Tuple { + var elements: (repeat each Elements) + + subscript(keyPath: KeyPath<(repeat each Elements), Value>) -> Value { + return elements[keyPath: keyPath] + } +} +``` + +The same positional key-path application could be supported directly on value packs. + +### Value expansion operator + +This proposal only supports the expansion operator on type parameter packs and value parameter packs, but there are other values that represent a list of zero or more values that the expansion operator would be useful for, including tuples and arrays. It would be desirable to introduce a new kind of expression that receives a scalar value and produces a value of pack expansion type. + +Here, we use the straw-man syntax `.element` for accessing tuple elements as a pack: + +```swift +func foo( + _ t: repeat each T, + _ u: repeat each U +) {} + +func bar1( + t: (repeat each T), + u: (repeat each U) +) { + repeat foo(each t.element, each u.element) +} + +func bar2( + t: (repeat each T), + u: (repeat each U) +) { + repeat foo(each t.element, repeat each u.element) +} +``` + +Here, `bar1(t: (1, 2), u: ("a", "b"))` will evaluate: + +```swift +foo(1, "a") +foo(2, "b") +``` + +While `bar2(t: (1, 2), u: ("a", "b"))` will evaluate: + +```swift +foo(1, "a", "b") +foo(2, "a", "b") +``` + +The distinction can be understood in terms of our notion of _captures_ in pack expansion expressions. + +### Pack destructuring operations + +The statically-known shape of a pack can enable destructing packs with concrete shape into the component elements: + +```swift +struct List { + let element: repeat each Element +} + +extension List { + func firstRemoved() -> List where (repeat each Element) == (First, repeat each Rest) { + let (first, rest) = (repeat each element) + return List(repeat each rest) + } +} + +let list = List(1, "Hello", true) +let firstRemoved = list.firstRemoved() // 'List("Hello", true)' +``` + +The body of `firstRemoved` decomposes `Element` into the components of its shape -- one value of type `First` and a value pack of type `repeat each Rest` -- effectively removing the first element from the list. + +### Tuple conformances + +Parameter packs, the above future directions, and a syntax for declaring tuple conformances based on [parameterized extensions](https://github.com/apple/swift/blob/main/docs/GenericsManifesto.md#parameterized-extensions) over non-nominal types enable implementing custom tuple conformances: + +```swift +extension (repeat each T): Equatable { + public static func ==(lhs: Self, rhs: Self) -> Bool { + for (l, r) in repeat (each lhs.element, each rhs.element) { + guard l == r else { return false } + } + return true + } +} +``` + +## Revision history + +Changes to the [first reviewed revision](https://github.com/swiftlang/swift-evolution/blob/b6ca38b9eee79650dce925e7aa8443a6a9e5e6ea/proposals/0393-parameter-packs.md): + +* The `repeat` keyword is required for generic requirement expansions to distinguish requirement expansions from single requirements on an individual pack element nested inside of a pack expansion expression. +* Overload resolution prefers scalar overloads when the scalar overload is considered a subtype of a parameter pack overload. + + +## Acknowledgments + +Thank you to Robert Widmann for exploring the design space of modeling packs as tuples, and to everyone who participated in earlier design discussions about variadic generics in Swift. Thank you to the many engineers who contributed to the implementation, including Sophia Poirier, Pavel Yaskevich, Nate Chandler, Hamish Knight, and Adrian Prantl. diff --git a/proposals/0394-swiftpm-expression-macros.md b/proposals/0394-swiftpm-expression-macros.md new file mode 100644 index 0000000000..1698611842 --- /dev/null +++ b/proposals/0394-swiftpm-expression-macros.md @@ -0,0 +1,203 @@ +# Package Manager Support for Custom Macros + +* Proposal: [SE-0394](0394-swiftpm-expression-macros.md) +* Authors: [Boris Buegling](https://github.com/neonichu), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 5.9)** +* Implementation: **Available behind pre-release tools-version** ([apple/swift-package-manager#6185](https://github.com/apple/swift-package-manager/pull/6185), [apple/swift-package-manager#6200](https://github.com/apple/swift-package-manager/pull/6200)) +* Review: ([pitch 1](https://forums.swift.org/t/pitch-package-manager-support-for-custom-macros/63482)) ([pitch 2](https://forums.swift.org/t/pitch-2-package-manager-support-for-custom-macros/63868)) ([review](https://forums.swift.org/t/se-0394-package-manager-support-for-custom-macros/64170)) ([acceptance](https://forums.swift.org/t/accepted-se-0394-package-manager-support-for-custom-macros/64589)) + +## Introduction + +Macros provide a way to extend Swift by performing arbitrary syntactic transformations on input source code to produce new code. One example for this are expression macros which were previously proposed in [SE-0382](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md). This proposal covers how custom macros are defined, built and distributed as part of a Swift package. + +## Motivation + +[SE-0382](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) and [A Possible Vision for Macros in Swift](https://gist.github.com/DougGregor/4f3ba5f4eadac474ae62eae836328b71) covered the motivation for macros themselves, defining them as part of a package will offer a straightforward way to reuse and distribute macros as source code. + +## Proposed solution + +Macros implemented in an external program can be declared as part of a package via a new macro target type, defined in +the `CompilerPluginSupport` library: + +```swift +public extension Target { + /// Creates a macro target. + /// + /// - Parameters: + /// - name: The name of the macro. + /// - dependencies: The macro's dependencies. + /// - path: The path of the macro, relative to the package root. + /// - exclude: The paths to source and resource files you want to exclude from the macro. + /// - sources: The source files in the macro. + /// - swiftSettings: The Swift settings for this macro. + /// - linkerSettings: The linker settings for this macro. + /// - plugins: The plugins used by this macro. + static func macro( + name: String, + dependencies: [Dependency] = [], + path: String? = nil, + exclude: [String] = [], + sources: [String]? = nil, + swiftSettings: [SwiftSetting]? = nil, + linkerSettings: [LinkerSetting]? = nil, + plugins: [PluginUsage]? = nil + ) -> Target { ... } +} +``` + +Similar to package plugins ([SE-0303 "Package Manager Extensible Build Tools"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md)), macro plugins are built as executables for the host (i.e, where the compiler is run). The compiler receives the paths to these executables from the build system and will run them on demand as part of the compilation process. Macro executables are automatically available for any target that transitively depends on them via the package manifest. + +A minimal package containing the implementation, definition and client of a macro would look like this: + +```swift +import PackageDescription +import CompilerPluginSupport + +let package = Package( + name: "MacroPackage", + dependencies: [ + .package(url: "https://github.com/apple/swift-syntax", from: "509.0.0"), + ], + targets: [ + .macro(name: "MacroImpl", + dependencies: [ + .product(name: "SwiftSyntaxMacros", package: "swift-syntax"), + .product(name: "SwiftCompilerPlugin", package: "swift-syntax") + ]), + .target(name: "MacroDef", dependencies: ["MacroImpl"]), + .executableTarget(name: "MacroClient", dependencies: ["MacroDef"]), + .testTarget(name: "MacroTests", dependencies: ["MacroImpl"]), + ] +) +``` + +Macro implementations will be executed in a sandbox [similar to package plugins](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md#security), preventing file system and network access. This is a practical way of encouraging macros to not depend on any state other than the specific macro expansion node they are given to expand and its child nodes (but not its parent nodes), and the information specifically provided by the macro expansion context. If in the future macros need access to other information, this will be accomplished by extending the macro expansion context, which also provides a mechanism for the compiler to track what information the macro actually queried. + +Any code from macro implementations can be tested by declaring a dependency on the macro target from a test, this works similarly to the [testing of executable targets](https://github.com/apple/swift-package-manager/pull/3316). + +## Detailed Design + +SwiftPM builds each macro as an executable for the host platform, applying certain additional compiler flags. Macros are expected to depend on SwiftSyntax using a versioned dependency that corresponds to a particular major Swift release. Note that SwiftPM's dependency resolution is workspace-wide, so all macros (and potentially other clients) will end up consolidating on one particular version of SwiftSyntax. Each target that transitively depends on a macro will have access to it, concretely this happens by SwiftPM passing `-load-plugin-executable` to the compiler to specify which executable contains the implementation of a certain macro module (e.g. `-load-plugin-executable /path/to/package/.build/debug/MacroImpl#MacroImpl` where the argument after the hash symbol is a comma separated list of module names which can be referenced by the `module` parameter of external macro declarations). The macro definition refers to the module and concrete type via an `#externalMacro` declaration which allows any dependency of the defining target to have access to the concrete macro. If any target of a library product depends on a macro, clients of said library will also get access to any public macros. Macros can have dependencies like any other target, but product dependencies of macros need to be statically linked, so explicitly dynamic library products cannot be used by a macro target. + +Concretely, the code for the macro package shown earlier would contain a macro implementation looking like this: + +```swift +import SwiftSyntax +import SwiftCompilerPlugin +import SwiftSyntaxBuilder +import SwiftSyntaxMacros + +@main +struct MyPlugin: CompilerPlugin { + var providingMacros: [Macro.Type] = [FontLiteralMacro.self] +} + +/// Implementation of the `#fontLiteral` macro, which is similar in spirit +/// to the built-in expressions `#colorLiteral`, `#imageLiteral`, etc., but in +/// a small macro. +public struct FontLiteralMacro: ExpressionMacro { + public static func expansion( + of macro: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + let argList = replaceFirstLabel( + of: macro.argumentList, + with: "fontLiteralName" + ) + let initSyntax: ExprSyntax = ".init(\(argList))" + if let leadingTrivia = macro.leadingTrivia { + return initSyntax.with(\.leadingTrivia, leadingTrivia) + } + return initSyntax + } +} + +/// Replace the label of the first element in the tuple with the given +/// new label. +private func replaceFirstLabel( + of tuple: TupleExprElementListSyntax, + with newLabel: String +) -> TupleExprElementListSyntax { + guard let firstElement = tuple.first else { + return tuple + } + + return tuple.replacing( + childAt: 0, + with: firstElement.with(\.label, .identifier(newLabel)) + ) +} +``` + +The macro definition would look like this: + +```swift +public enum FontWeight { + case thin + case normal + case medium + case semiBold + case bold +} + +public protocol ExpressibleByFontLiteral { + init(fontLiteralName: String, size: Int, weight: FontWeight) +} + +/// Font literal similar to, e.g., #colorLiteral. +@freestanding(expression) public macro fontLiteral(name: String, size: Int, weight: FontWeight) -> T = #externalMacro(module: "MacroImpl", type: "FontLiteralMacro") + where T: ExpressibleByFontLiteral +``` + +And the client of the macro would look like this: + +```swift +import MacroDef + +struct Font: ExpressibleByFontLiteral { + init(fontLiteralName: String, size: Int, weight: MacroDef.FontWeight) { + } +} + +let _: Font = #fontLiteral(name: "Comic Sans", size: 14, weight: .thin) +``` + +SwiftSyntax's versioning scheme is based on Swift major versions (e.g. 509.0.0 for Swift 5.9). + +If a package depends on two macros using the `from` version dependency and minor versions of a macro use different versions of SwiftSyntax, users should automatically get a version that's compatible with all macros. For example consider the following where a package depends on both Macro 1 and Macro 2 using `from: "1.0.0"` + +``` +Macro 1 SwiftSyntax Macro 2 + +1.0 --------------> 509.0.0 <-------------- 1.0 + 509.0.1 <-------------- 1.1 + 510.0.0 <-------------- 1.2 +``` + +In this case, SwiftPM would choose version 1.0 for Macro 1, version 1.1 for Macro 2 and end up with version 509.0.1 for SwiftSyntax. We're going to monitor how the versioning story plays out in practice and may take further action in SwiftSyntax or SwiftPM's dependency resolution if the concrete need arises. + + +## Impact on existing packages + +Since macro plugins are entirely additive, there's no impact on existing packages. + +## Alternatives considered + +### Package plugins + +The original pitch of expression macros considered declaring macros by introducing a new capability to [package plugins](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md), but since the execution model is significantly different and the APIs used for macros are external to SwiftPM, this idea was discarded. + +### `.macroTarget()` + +We're (slowly) trying to move away from having the target suffix since it is implied by the context. This is already the case for plugin targets and eventually we'd like to have e.g. `.test()` as well. This would also make the target APIs be more in line with the product ones, where e.g. we don't use `.libraryProduct()`. + +### Dependencies on macro targets + +In [SE-0303](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0303-swiftpm-extensible-build-tools.md), we introduced the `plugins` parameter for build order dependencies on plugins, so it could have make sense to use a `macros` parameter for dependencies on macros. However, introducing bespoke API for each type of host-side content used during the build does not seem scalable. We also already have precedence of executables being part of `dependencies` even though that dependency is strictly for build ordering (with the exception of tests, which also applies to macros). Because of this, dependencies on macros are declared via the `dependencies` parameter, however it could be interesting to revisit a separation of build order and linked dependencies in the future. + +## Future Directions + +### Generalized support for additional manifest API + +The macro target type is provided by a new library `CompilerPluginSupport` as a starting point for making package manifests themselves more extensible. Support for product and target type plugins should eventually be generalized to allow other types of externally defined specialized target types, such as, for example, a Windows application. diff --git a/proposals/0395-observability.md b/proposals/0395-observability.md new file mode 100644 index 0000000000..1a3acd1c9f --- /dev/null +++ b/proposals/0395-observability.md @@ -0,0 +1,523 @@ +# Observation + +* Proposal: [SE-0395](0395-observability.md) +* Authors: [Philippe Hausler](https://github.com/phausler), [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 5.9)** +* Review: ([pitch](https://forums.swift.org/t/pitch-observation-revised/63757)), ([first review](https://forums.swift.org/t/se-0395-observability/64342/)), ([second review](https://forums.swift.org/t/second-review-se-0395-observability/65261/)), ([acceptance](https://forums.swift.org/t/accepted-with-revision-se-0395-observability/66760)) + +#### Changes + +* Version 1: [Initial pitch](https://forums.swift.org/t/pitch-observation/62051) +* Version 2: Previously Observation registered observers directly to `Observable`, the new approach registers observers to an `Observable` via a `ObservationTransactionModel`. These models control the "edge" of where the change is emitted. They are the responsible component for notifying the observers of events. This allows the observers to focus on just the event and not worry about "leading" or "trailing" (will/did) "edges" of the signal. Additionally the pitch was shifted from the type wrapper feature over to the more appropriate macro features. +* Version 3: The `Observer` protocol and `addObserver(_:)` method are gone in favor of providing async sequences of changes and transactions. +* Version 4: In order to support observation for subclasses and to provide space to address design question around the asynchronous `values(for:)` and `changes(for:)` methods, the proposal now focuses on an `Observable` marker protocol and the `withTracking(_:changes:)` function. + +#### Suggested Reading + +* [Expression Macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) +* [Attached Macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md) + +## Introduction + +Making responsive apps often requires the ability to update the presentation when underlying data changes. The _observer pattern_ allows a subject to maintain a list of observers and notify them of specific or general state changes. This has the advantages of not directly coupling objects together and allowing implicit distribution of updates across potential multiple observers. An observable object needs no specific information about its observers. + +This design pattern is a well-traveled path by many languages, and Swift has an opportunity to provide a robust, type-safe, and performant implementation. This proposal defines what an observable reference is, what an observer needs to conform to, and the connection between a type and its observers. + +## Motivation + +There are already a few mechanisms for observation in Swift. These include key-value observing (KVO) and `ObservableObject`, but each of those have limitations. KVO can only be used with `NSObject` descendants, and `ObservableObject` requires using Combine, which is restricted to Darwin platforms and does not use current Swift concurrency features. By taking experience from those existing systems, we can build a more generally useful feature that applies to all Swift reference types, not just those that inherit from `NSObject`, and have it work cross-platform with the advantages from language features like `async`/`await`. + +The existing systems get a number of behaviors and characteristics right. However, there are a number of areas that can provide a better balance of safety, performance, and expressiveness. For example, grouping dependent changes into an independent transaction is a common task, but this is complex when using Combine and unsupported when using KVO. In practice, observers want access to transactions, with the ability to specify how transactions are interpreted. + +Annotations clarify what is observable, but can also be cumbersome. For example, Combine requires not just that a type conform to `ObservableObject`, but also requires each property that is being observed to be marked as `@Published`. Furthermore, computed properties cannot be directly observed. In reality, having non-observed fields in a type that is observable is uncommon. + +Throughout this document, references to both KVO and Combine will illustrate what capabilities are benefits and can be incorporated into the new approach, and what drawbacks are possible to solve in a more robust manner. + +### Prior Art + +#### KVO + +Key-value observing has served the Cocoa/Objective-C programming model well, but is limited to class hierarchies that inherit from `NSObject`. The APIs only offer the intercepting of events, meaning that the notification of changes is between the `willSet` and `didSet` events. KVO has great flexibility with granularity of events, but lacks in composability. KVO observers must also inherit from `NSObject`, and rely on the Objective-C runtime to track the changes that occur. Even though the interface for KVO has been updated to utilize the more modern Swift strongly-typed key paths, under the hood its events are still stringly typed. + +#### Combine + +Combine's `ObservableObject` produces changes at the beginning of a change event, so all values are delivered before the new value is set. While this serves SwiftUI well, it is restrictive for non-SwiftUI usage and can be surprising to developers first encountering that behavior. `ObservableObject` also requires all observed properties to be marked as `@Published` to interact with change events. In most cases, this requirement is applied to every single property and becomes redundant to the developer; folks writing an `ObservableObject` conforming type must repeatedly (with little to no true gained clarity) annotate each property. In the end, this results in meaning fatigue of what is or isn't a participating item. + +## Proposed solution + +A formalized observer pattern needs to support the following capabilities: + +* Marking a type as observable +* Tracking changes within an instance of an observable type +* Observing and utilizing those changes from somewhere else + +In addition, the design and implementation should meet these criteria: + +* Observable types are easy to annotate (without fatigue of meaning) +* Access control should be respected +* Adopting the features for observability should require minimal effort to get started +* Using advanced features should progressively disclose to more complex systems +* Observation should be able to handle more than one observed member at once +* Observation should be able to work with computed properties that reference other properties +* Observation should be able to work with computed properties that store their values in external storage +* Integration of observation should work in transactions of graphs and not just singular objects + +We propose a new standard library module named `Observation` that includes the required functionality to implement such a pattern. + +Primarily, a type can declare itself as observable simply by using the `@Observable` macro annotation: + +```swift +@Observable class Car { + var name: String + var awards: [Award] +} +``` + +The `@Observable` macro implements conformance to the `Observable` marker protocol and tracking for each stored property. Unlike `ObservableObject` and `@Published`, the properties of an `@Observable` type do not need to be individually marked as observable. Instead, all stored properties are implicitly observable. + +The `Observation` module also provides the top-level function `withObservationTracking`, which detects accesses to tracked properties within a specific scope. Once those properties are identified, any changes to the tracked properties triggers a call to the provided `onChange` closure. + +```swift +let cars: [Car] = ... + +@MainActor +func renderCars() { + withObservationTracking { + for car in cars { + print(car.name) + } + } onChange: { + Task { @MainActor in + renderCars() + } + } +} +``` + +In the example above, the `render` function accesses each car's `name` property. When any of the cars change `name`, the `onChange` closure is then called on the first change. However, if a car has an award added, the `onChange` call won't happen. This design supports uses that require implicit observation tracking, ensuring that updates are only performed in response to relevant changes. + +## Detailed Design + +The `Observable` protocol, `@Observable` macro, and a handful of supporting types comprise the `Observation` module. As described below, this design allows adopters to use a straightforward syntax for simple cases, while allowing full control over the details the implementation when necessary. + +### `Observable` protocol + +Observable types conform to the `Observable` marker protocol. While the `Observable` protocol doesn't have formal requirements, it includes a semantic requirement that conforming types must implement tracking for each stored property using an `ObservationRegistrar`. Most types can meet that requirement simply by using the `@Observable` macro: + +```swift +@Observable public final class MyObject { + public var someProperty = "" + public var someOtherProperty = 0 + fileprivate var somePrivateProperty = 1 +} +``` + +### `@Observable` Macro + +In order to make implementation as simple as possible, the `@Observable` macro automatically synthesizes conformance to the `Observable` protocol, transforming annotated types into a type that can be observed. When fully expanded, the `@Observable` macro does the following: + +- declares conformance to the `Observable` protocol, +- adds a property for the registrar, +- and adds internal helper methods for tracking accesses and mutations. + +Additionally, for each stored property, the macro: + +- annotates each stored property with the `@ObservationTracked` macro, +- converts each stored property to a computed property, +- and adds an underscored, `@ObservationIgnored` version of each stored property. + +Since all of the code generated by the macro could be manually written, developers can write or customize their own implementation when they need more fine-grained control. + +As an example of the `@Observable` macro expansion, consider the following `Model` type: + +```swift +@Observable class Model { + var order: Order? + var account: Account? + + var alternateIconsUnlocked: Bool = false + var allRecipesUnlocked: Bool = false + + func purchase(alternateIcons: Bool, allRecipes: Bool) { + alternateIconsUnlocked = alternateIcons + allRecipesUnlocked = allRecipes + } +} +``` + +Expanding the `@Observable` macro, as well as the generated macros, results in the following declaration: + +```swift +class Model: Observable { + internal let _$observationRegistrar = ObservationRegistrar() + + internal func access( + keyPath: KeyPath + ) { + _$observationRegistrar.access(self, keyPath: keyPath) + } + + internal func withMutation( + keyPath: KeyPath, + _ mutation: () throws -> T + ) rethrows -> T { + try _$observationRegistrar.withMutation(of: self, keyPath: keyPath, mutation) + } + + var order: Order? { + get { + self.access(keyPath: \.order) + return _order + } + set { + self.withMutation(keyPath: \.order) { + _order = newValue + } + } + } + + var account: Account? { + get { + self.access(keyPath: \.account) + return _account + } + set { + self.withMutation(keyPath: \.account) { + _account = newValue + } + } + } + + var alternateIconsUnlocked: Bool { + get { + self.access(keyPath: \.alternateIconsUnlocked) + return _alternateIconsUnlocked + } + set { + self.withMutation(keyPath: \.alternateIconsUnlocked) { + _alternateIconsUnlocked = newValue + } + } + } + + var allRecipesUnlocked: Bool { + get { + self.access(keyPath: \.allRecipesUnlocked) + return _allRecipesUnlocked + } + set { + self.withMutation(keyPath: \.allRecipesUnlocked) { + _allRecipesUnlocked = newValue + } + } + } + + var _order: Order? + var _account: Account? + + var _alternateIconsUnlocked: Bool = false + var _allRecipesUnlocked: Bool = false +} +``` + +### `@ObservationTracked` and `@ObservationIgnored` macros + +The `Observation` module includes two additional macros that can annotate properties of observable types. The `@ObservationTracked` macro is added to stored properties by the `@Observable` macro expansion, and, when expanded, converts a stored property to a computed one with access and mutation tracking. Developers generally won't use `@ObservationTracked` themselves. + +The `@ObservationIgnored` macro, on the other hand, doesn't add anything to a source file when expanded. Instead, it acts as a marker for properties that shouldn't be tracked. The `@Observable` macro expansion adds `@ObservationIgnored` to the underscored stored properties it creates. Developers can also apply `@ObservationIgnored` to stored properties that shouldn't be included in observation tracking. + +### Computed properties + +Computed properties that derive their values from stored properties are automatically tracked due to their reliance on tracked properties. Computed properties that source their value from remote storage or via indirection, however, must manually add tracking using the generated `access(keyPath:)` and `withMutation(keyPath:)` methods. + +For example, consider the `AtomicModel` in the following code sample. `AtomicModel` stores a score in an `AtomicInt`, with a computed property providing an `Int` interface. The atomic property is annotated with the `@ObservationIgnored` macro because it isn't useful to track the constant value for observation. For the computed `score` property, which is the public interface of the type, the getter and setter include manually-written calls to track accesses and mutations. + +```swift +@Observable +public class AtomicModel { + @ObservationIgnored + fileprivate let _scoreStorage = AtomicInt(initialValue: 0) + + public var score: Int { + get { + self.access(keyPath: \.score) + return _scoreStorage.value + } + set { + self.withMutation(keyPath: \.score) { + _scoreStorage.value = newValue + } + } + } +} +``` + +### `willSet`/`didSet` + +Observation is supported for properties with `willSet` and `didSet` property observers. For example, the `@Observable` macro on the `PropertyExample` type here: + +```swift +@Observable class PropertyExample { + var a = 0 { + willSet { print("will set triggered") } + didSet { print("did set triggered") } + } + var b = 0 + var c = "" +} +``` + +...transforms the `a` property as follows, preserving the `willSet` and `didSet` behavior: + +```swift +var a: Int { + get { + self.access(keyPath: \.a) + return _a + } + set { + self.withMutation(keyPath: \.a) { + _a = newValue + } + } +} + +var _a = 0 { + willSet { print("will set triggered") } + didSet { print("did set triggered") } +} +``` + +### Initializers + +Because observable types generally use the implicitly generated initializers, the `@Observable` macro requires that all stored properties have a default value. This guarantees definitive initialization, so that additional initializers can be added to observable types in an extension. + +The default value requirement could be relaxed in a future version; see the Future Directions section for more. + +### Subclasses + +Developers can create `Observable` subclasses of either observable or non-observable types. Only the properties of a type that implements the `Observable` tracking requirements will be observed. That is, when working with an observable subclass of a non-observable type, the superclass's stored properties will not be tracked under observation. + +### `withObservationTracking(_:onChange:)` + +In order to provide automatically scoped observation, the `ObservationModule` provides a function to capture accesses to properties within a given scope, and then call out upon the first change to any of those properties. This can be used by user interface libraries, such as SwiftUI, to provide updates to the specific properties which are accessed within a particular scope, limiting interface updates or renders to only the relevant changes. For more detail, see the SDK Impact section below. + +```swift +public func withObservationTracking( + _ apply: () -> T, + onChange: @autoclosure () -> @Sendable () -> Void +) -> T +``` + +The `withObservationTracking` function takes two closures. Any access to a tracked property within the `apply` closure will flag the property; any change to a flagged property will trigger a call to the `onChange` closure. + +Accesses are recognized for: +- tracked properties on observable objects +- tracked properties of properties that have observable type +- properties that are accessed via computed property accesses + +For example, this `Person` class has multiple tracked properties, some of which are internal: + +```swift +@Observable public class Person: Sendable { + internal var firstName = "" + internal var lastName = "" + public var age: Int? + + public var fullName: String { + "\(firstName) \(lastName)" + } + + public var friends: [Person] = [] +} +``` + +Accessing the `fullName` and `friends` properties will result in the `firstName`, `lastName`, and `friends` properties being tracked for changes: + +```swift +@MainActor +func renderPerson(_ person: Person) { + withObservationTracking { + print("\(person.fullName) has \(person.friends.count) friends.") + } onChange: { + Task { @MainActor in + renderPerson(person) + } + } +} +``` + +Whenever the person's `firstName` or `lastName` properties are updated, the `onChange` closure will be called, even though those properties are internal, since their accesses are linked to a public computed property. Mutations to the `friends` array will also cause a call to `onChange`, though changes to individual members of the array are not tracked. + +### `ObservationRegistrar` + +`ObservationRegistrar` is the required storage for tracking accesses and mutations. The `@Observable` macro synthesizes a registrar to handle these mechanisms as a generalized feature. By default, the registrar is thread safe and must be as `Sendable` as containers could potentially be; therefore it must be designed to handle independent isolation for all actions. + +```swift +public struct ObservationRegistrar: Sendable { + public init() + + public func access( + _ subject: Subject, + keyPath: KeyPath + ) + + public func willSet( + _ subject: Subject, + keyPath: KeyPath + ) + + public func didSet( + _ subject: Subject, + keyPath: KeyPath + ) + + public func withMutation( + of subject: Subject, + keyPath: KeyPath, + _ mutation: () throws -> T + ) rethrows -> T +} +``` + +The `access` and `withMutation` methods identify transactional accesses. These methods register access to the underlying tracking system for access and identify mutations to the transactions registered for observers. + +## SDK Impact (a preview of SwiftUI interaction) + +When using the existing `ObservableObject`-based observation, there are a number of edge cases that can be surprising unless developers have an in-depth understanding of SwiftUI. Formalizing observation can make these edge cases considerably more approachable by reducing the complexity of the different systems needed to be understood. + +The following is adapted from the [Fruta sample app](https://developer.apple.com/documentation/swiftui/fruta_building_a_feature-rich_app_with_swiftui), modified for clarity: + +```swift +class Model: ObservableObject { + @Published var order: Order? + @Published var account: Account? + + var hasAccount: Bool { + return userCredential != nil && account != nil + } + + @Published var favoriteSmoothieIDs = Set() + @Published var selectedSmoothieID: Smoothie.ID? + + @Published var searchString = "" + + @Published var isApplePayEnabled = true + @Published var allRecipesUnlocked = false + @Published var unlockAllRecipesProduct: Product? +} + +struct SmoothieList: View { + var smoothies: [Smoothie] + @ObservedObject var model: Model + + var listedSmoothies: [Smoothie] { + smoothies + .filter { $0.matches(model.searchString) } + .sorted(by: { $0.title.localizedCompare($1.title) == .orderedAscending }) + } + + var body: some View { + List(listedSmoothies) { smoothie in + ... + } + } +} +``` + +The `@Published` attribute identifies each field that participates in changes in the object, but it does not provide any differentiation or distinction as to the source of changes. This unfortunately results in additional layouts, rendering, and updates. + +The proposed API not only reduces the `@Published` repetition, but also simplifies the SwiftUI view code too! With the proposed `@Observable` macro, the previous example can instead be written as the following: + +```swift +@Observable class Model { + var order: Order? + var account: Account? + + var hasAccount: Bool { + userCredential != nil && account != nil + } + + var favoriteSmoothieIDs: Set = [] + var selectedSmoothieID: Smoothie.ID? + + var searchString = "" + + var isApplePayEnabled = true + var allRecipesUnlocked = false + var unlockAllRecipesProduct: Product? +} + +struct SmoothieList: View { + var smoothies: [Smoothie] + var model: Model + + var listedSmoothies: [Smoothie] { + smoothies + .filter { $0.matches(model.searchString) } + .sorted(by: { $0.title.localizedCompare($1.title) == .orderedAscending }) + } + + var body: some View { + List(listedSmoothies) { smoothie in + ... + } + } +} +``` + +There are some other interesting differences that follow from using the proposed observation system. For example, tracking observation of access within a view can be applied to an array, an optional, or even a custom type. This opens up new and interesting ways that developers can utilize SwiftUI more easily. + +This is a potential future direction for SwiftUI, but is not part of this proposal. + +## Source compatibility + +This proposal is additive and provides no impact to existing source code. + +## Effect on ABI stability + +This proposal is additive and no impact is made upon existing ABI stability. This does have implication to the marking of inline to functions and back-porting of this feature. In the cases where it is determined to be performance critical to the distribution of change events the methods will be marked as inlineable. + +Changing a type from not observable to `@Observable` has the same ABI impact as changing a property from stored to computed (which is not ABI breaking). Removing `@Observable` not only transitions from computed to stored properties but also removes a conformance (which is ABI breaking). + +## Effect on API resilience + +This proposal is additive and no impact is made upon existing API resilience. The types that adopt `@Observable` cannot remove it without breaking API contract. + +## Location of API + +This API will be housed in a module that is part of the Swift language but outside of the standard library. To use this module `import Observation` must be used (and provisionally using the preview `import _Observation`). + +## Future Directions + +The requirement that all stored properties of an observable type have initial values could be relaxed in the future, if language features are added that would support that. For example, property wrappers have a feature that allows their underlying wrapped value to be provided in an initializer rather than as a default value. Generalizing that feature to all properties could allow the `@Observable` macro to enable a more typical initialization implementation. + +Another area of focus for future enhancements is support for observable `actor` types. This would require specific handling for key paths that currently does not exist for actors. + +An earlier version of this proposal included asynchronous sequences of coalesced transactions and individual property changes, named `values(for:)` and `changes(for:)`. Similar invariant-preserving asynchronous sequences could be added in a future proposal. + +## Alternatives considered + +An earlier consideration instead of defining transactions used direct will/did events to the observer. This, albeit being more direct, promoted mechanisms that did not offer the correct granularity for supporting the required synchronization between dependent members. It was determined that building transactions are worth the extra complexity to encourage developers using the API to consider what models for transactionality they need, instead of thinking just in terms of will/did events. + +Another design included an `Observer` protocol that could be used to build callback-style observer types. This has been eliminated in favor of the `AsyncSequence` approach. + +The `ObservedChange` type could have the `Sendable` requirement relaxed by making the type only conditionally `Sendable` and then allowing access to the subject in all cases; however this poses some restriction to the internal implementations and may have a hole in the sendable nature of the type. Since it is viewed that accessing values is most commonly by one property the values `AsyncSequence` fills most of that role and for cases where more than one field is needed to be accessed on a given actor the iteration can be done with a weak reference to the observable subject. + +## Acknowledgments + +* [Holly Borla](https://github.com/hborla) - For providing fantastic ideas on how to implement supporting infrastructure to this pitch +* [Pavel Yaskevich](https://github.com/xedin) - For tirelessly iterating on prototypes for supporting compiler features +* Rishi Verma - For bouncing ideas and helping with the design of integrating this idea into other work +* [Kyle Macomber](https://github.com/kylemacomber) - For connecting resources and providing useful feedback +* Matt Ricketson - For helping highlight some of the inner guts of SwiftUI + +## Related systems + +* [Swift `Combine.ObservableObject`](https://developer.apple.com/documentation/combine/observableobject/) +* [Objective-C Key Value Observing](https://developer.apple.com/documentation/objectivec/nsobject/nskeyvalueobserving?language=objc) +* [C# `IObservable`](https://learn.microsoft.com/en-us/dotnet/api/system.iobservable-1?view=net-6.0) +* [Rust `Trait rx::Observable`](https://docs.rs/rx/latest/rx/trait.Observable.html) +* [Java `Observable`](https://docs.oracle.com/javase/7/docs/api/java/util/Observable.html) +* [Kotlin `observable`](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.properties/-delegates/observable.html) diff --git a/proposals/0396-never-codable.md b/proposals/0396-never-codable.md new file mode 100644 index 0000000000..be3ecd86ea --- /dev/null +++ b/proposals/0396-never-codable.md @@ -0,0 +1,59 @@ +# Conform `Never` to `Codable` + +* Proposal: [SE-0396](0396-never-codable.md) +* Author: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#64899](https://github.com/apple/swift/pull/64899) +* Review: ([pitch](https://forums.swift.org/t/pitch-conform-never-to-codable/64056)) ([review](https://forums.swift.org/t/se-0396-conform-never-to-codable/64469)) ([acceptance](https://forums.swift.org/t/accepted-se-0396-conform-never-to-codable/64848)) + +## Introduction + +Extend `Never` so that it conforms to the `Encodable` and `Decodable` protocols, together known as `Codable`. + +## Motivation + +Swift can synthesize `Codable` conformance for any type that has `Codable` members. Generic types often participate in this synthesized conformance by constraining their generic parameters, like this `Either` type: + +```swift +enum Either { + case left(A) + case right(B) +} + +extension Either: Codable where A: Codable, B: Codable {} +``` + +In this way, `Either` instances where both generic parameters are `Codable` are `Codable` themselves, such as an `Either`. However, since `Never` isn't `Codable`, using `Never` as one of the parameters blocks the conditional conformance, even though it would be perfectly fine to encode or decode a type like `Either`. + +## Proposed solution + +The standard library should add `Encodable` and `Decodable` conformance to the `Never` type. + +## Detailed design + +The `Encodable` conformance is simple — since it's impossible to have a `Never` instance, the `encode(to:)` method can simply be empty. + +The `Decodable` protocol requires the `init(from:)` initializer, which clearly can't create a `Never` instance. Because trying to decode invalid input isn't a programmer error, a fatal error would be inappropriate. Instead, the implementation throws a `DecodingError.typeMismatch` error if decoding is attempted. + +## Source compatibility + +If existing code already declares `Codable` conformance, that code will begin to emit a warning: e.g. `Conformance of 'Never' to protocol 'Encodable' was already stated in the type's module 'Swift'`. + +The new conformance shouldn't differ from existing conformances, since it isn't possible to construct an instance of `Never`. + +## ABI compatibility + +The proposed change is additive and does not change any existing ABI. + +## Implications on adoption + +The new conformance will have availability annotations. + +## Future directions + +None. + +## Alternatives considered + +A previous iteration of this proposal used `DecodingError.dataCorrupted` as the error thrown from the `Decodable` initializer. Since that error is typically used for syntactical errors, such as malformed JSON, the `typeMismatch` error is now used instead. A custom error type could be created for this purpose, but as `typeMismatch` already exists as documented API, and provides the necessary information for a developer to understand the error, its use is appropriate here. diff --git a/proposals/0397-freestanding-declaration-macros.md b/proposals/0397-freestanding-declaration-macros.md new file mode 100644 index 0000000000..ab59886f02 --- /dev/null +++ b/proposals/0397-freestanding-declaration-macros.md @@ -0,0 +1,341 @@ +# Freestanding Declaration Macros + +* Proposal: [SE-0397](0397-freestanding-declaration-macros.md) +* Authors: [Doug Gregor](https://github.com/DougGregor), [Richard Wei](https://github.com/rxwei), [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9)** +* Vision: [Macros](https://github.com/swiftlang/swift-evolution/blob/main/visions/macros.md) +* Implementation: On `main` behind the experimental flag `FreestandingMacros` +* Review: ([review](https://forums.swift.org/t/se-0397-freestanding-declaration-macros/64655)) ([partial acceptance and second review](https://forums.swift.org/t/se-0397-second-review-freestanding-declaration-macros/64997)) ([acceptance](https://forums.swift.org/t/accepted-se-0397-freestanding-declaration-macros/65167)) +* Previous revisions: ([1](https://github.com/swiftlang/swift-evolution/blob/c0f1e6729b6ca1a4fc2367efe68612fde175afe4/proposals/0397-freestanding-declaration-macros.md)) + +## Introduction + +[SE-0382 "Expression macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) introduced macros into Swift. The approach involves an explicit syntax for uses of macros (prefixed by `#`), type checking for macro arguments prior to macro expansion, and macro expansion implemented via separate programs that operate on the syntax tree of the arguments. + +This proposal generalizes the `#`-prefixed macro expansion syntax introduced for expression macros to also allow macros to generate declarations, enabling a number of other use cases, including: + +* Generating data structures from a template or other data format (e.g., JSON). +* Subsuming the `#warning` and `#error` directives introduced in [SE-0196](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md) as macros. + +## Proposed solution + +The proposal extends the notion of "freestanding" macros introduced in [SE-0382 "Expression macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) to also allow macros to introduce new declarations. Like expression macros, freestanding declaration macros are expanded using the `#` syntax, and have type-checked macro arguments. However, freestanding declaration macros can be used any place that a declaration is provided, and never produce a value. + +As with other macros, freestanding declaration macros are declared with the `macro` introducer. They will use the `@freestanding` attribute with the new `declaration` role and, optionally, a set of *introduced names* as described in [SE-0389 "Attached macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#specifying-newly-introduced-names). For example, a freestanding declaration macro would have an attribute like this: + +```swift +@freestanding(declaration) +``` + +whereas a freestanding declaration macro that introduced an enum named `CodingKeys` would have an attribute like this: + +```swift +@freestanding(declaration, names: named(CodingKeys)) +``` + +Implementations of freestanding declaration macros are types that conform to the `DeclarationMacro` protocol, which is defined as follows: + +```swift +public protocol DeclarationMacro: FreestandingMacro { + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] +} +``` + +Declaration macros can be used anywhere that a declaration is permitted, e.g., in a function or closure body, at the top level, or within a type definition or extension thereof. Declaration macros produce zero or more declarations. The `warning` directive introduced by [SE-0196](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md) can be described as a freestanding declaration macro as follows: + +```swift +/// Emits the given message as a warning, as in SE-0196. +@freestanding(declaration) +macro warning(_ message: String) = #externalMacro(module: "MyMacros", type: "WarningMacro") +``` + +Given this macro declaration, the syntax + +```swift +#warning("unsupported configuration") +``` + +can be used anywhere a declaration can occur. + +The implementation of a `warning` declaration macro extracts the string literal argument (producing an error if there wasn't one) and emits a warning. It returns an empty list of declarations: + +```swift +public struct WarningMacro: DeclarationMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] { + guard let messageExpr = node.argumentList.first?.expression.as(StringLiteralExprSyntax.self), + messageExpr.segments.count == 1, + let firstSegment = messageExpr.segments.first, + case let .stringSegment(message) = firstSegment else { + throw SimpleError(node, "warning macro requires a non-interpolated string literal") + } + + context.diagnose(Diagnostic(node: Syntax(node), message: SimpleDiagnosticMessage( + message: message.description, + diagnosticID: .init(domain: "test", id: "error"), + severity: .warning))) + return [] + } +} +``` + +## Detailed design + +### Syntax + +The syntactic representation of a freestanding macro expansion site is a macro expansion declaration. A macro expansion declaration is described by the following grammar. It is based on the production rule as a [macro expansion expression](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion), but with the addition of attributes and modifiers: + +``` +declaration -> macro-expansion-declaration +macro-expansion-declaration -> attributes? declaration-modifiers? '#' identifier generic-argument-clause[opt] function-call-argument-clause[opt] trailing-closures[opt] +``` + +At top level and function scope where both expressions and declarations are allowed, a freestanding macro expansion site is first parsed as a macro expansion expression. It will be replaced by a macro expansion declaration later during type checking, if the macro resolves to a declaration macro. It is ill-formed if a macro expansion expression resolves to a declaration macro but isn't the outermost expression. This parsing rule is required in case an expression starts with a macro expansion expression, such as in the following infix expression: + +```swift +#line + 1 +#line as Int? +``` + +#### Attributes and modifiers + +Any attributes and modifiers written on a freestanding macro declaration are implicitly applied to each declaration produced by the macro expansion. For example: + +```swift +@available(toasterOS 2.0, *) +public #gyb( + """ + struct Int${0} { ... } + struct UInt${0} { ... } + """, + [8, 16, 32, 64] +) +``` + +would expand to: + +```swift +@available(toasterOS 2.0, *) +public struct Int8 { ... } + +@available(toasterOS 2.0, *) +public struct UInt8 { ... } + +@available(toasterOS 2.0, *) +public struct Int16 { ... } + +@available(toasterOS 2.0, *) +public struct UInt16 { ... } + +@available(toasterOS 2.0, *) +public struct Int32 { ... } + +@available(toasterOS 2.0, *) +public struct UInt32 { ... } + +@available(toasterOS 2.0, *) +public struct Int64 { ... } + +@available(toasterOS 2.0, *) +public struct UInt64 { ... } +``` + +### Restrictions + +Like attached peer macros, a freestanding declaration macro can expand to any declaration that is syntactically and semantically well-formed within the context where the macro is expanded. It shares the same requirements and restrictions: + +- [**Specifying newly-introduced names**](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#specifying-newly-introduced-names) + - Note that only `named(...)` and `arbitrary` are allowed as macro-introduced names for a declaration macro. `overloaded`, `prefixed`, and `suffixed` do not make sense when there is no declaration from which to derive names. +- [**Visibility of names used and introduced by macros**](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#visibility-of-names-used-and-introduced-by-macros) +- [**Restrictions on `arbitrary` names**](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#restrictions-on-arbitrary-names) +- [**Permitted declaration kinds**](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md#permitted-declaration-kinds) + +One additional restriction is that a macro declaration can have at most one freestanding macro role. This is because top level and function scopes allow a combination of expressions, statements, and declarations, which would be ambiguous to a freestanding macro expansion with multiple roles. + +```swift +@freestanding(expression) +@freestanding(declaration) // error: a macro cannot have multiple freestanding macro roles +macro foo() +``` + +### Examples + +#### SE-0196 `warning` and `error` + +The `#warning` and `#error` directives introduced in [SE-0196](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md): can be implemented directly as declaration macros: + +```swift +/// Emit a warning containing the given message. +@freestanding(declaration) macro warning(_ message: String) + +/// Emit an error containing the given message +@freestanding(declaration) macro error(_ message: String) +``` + +### Template code generation + +The Swift Standard Library makes extensive use of the [gyb](https://github.com/apple/swift/blob/main/utils/gyb.py) tool to generate boilerplate-y Swift code such as [tgmath.swift.gyb](https://github.com/apple/swift/blob/main/stdlib/public/Platform/tgmath.swift.gyb). The template code is written in `.gyb` files, which are processed by the gyb tool separately before Swift compilation. With freestanding declaration macros, one could write a macro to accept a string as a template and a list of replacement values, allowing templates to be defined inline and eliminating the need to set up a separate build phase. + +```swift +@freestanding(declaration, names: arbitrary) +macro gyb(String, [Any]) = #externalMacro(module: "MyMacros", type: "GYBMacro") + +#gyb( + """ + public struct Int${0} { ... } + public struct UInt${0} { ... } + """, + [8, 16, 32, 64] +) +``` + +This expands to: + +```swift + public struct Int8 { ... } + public struct UInt8 { ... } + + public struct Int16 { ... } + public struct UInt16 { ... } + + public struct Int32 { ... } + public struct UInt32 { ... } + + public struct Int64 { ... } + public struct UInt64 { ... } +``` + +### Data model generation + +Declaring a data model for an existing textual serialization may need some amount of eyeballing and is prone to errors. A freestanding declaration macro can be used to analyze a template textual serialization, e.g. JSON, and declare a model data structure against the template. + +```swift +@freestanding(declaration, names: arbitrary) +macro jsonModel(String) = #externalMacro(module: "MyMacros", type: "JSONModelMacro") + +struct JSONValue: Codable { + #jsonModel(""" + "name": "Produce", + "shelves": [ + { + "name": "Discount Produce", + "product": { + "name": "Banana", + "points": 200, + "description": "A banana that's perfectly ripe." + } + } + ] + """) +} +``` + +This expands to: + +```swift +struct JSONValue: Codable { + var name: String + var shelves: [Shelves] + + struct Shelves: Codable { + var name: String + var product: Product + + struct Product: Codable { + var description: String + var name: String + var points: Double + } + } +} +``` + +## Source compatibility + +Freestanding macros use the same syntax introduced for [expression macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md), which were themselves a pure extension without an impact on source compatibility. Because a given macro can only have a single freestanding role, and we retain the parsing rules for macro expansion expressions, this proposal introduces no new ambiguities with SE-0382. + +## Effect on ABI stability + +Macros are a source-to-source transformation tool that have no ABI impact. + +## Effect on API resilience + +Macros are a source-to-source transformation tool that have no effect on API resilience. + +## Alternatives considered + +### Multiple freestanding macro roles on a single macro + +The proposed feature bans declaring a macro as having multiple freestanding macro roles such as being both `@freestanding(expression)` and `@freestanding(declaration)`. But such a scenario could be allowed with proper rules. + +One possible solution would be to expand such a macro based on its expansion context. If it's being expanded where a declaration is allowed, it will always be expanded as a declaration. Otherwise, it's expanded as an expression. + +```swift +@freestanding(expression) +@freestanding(declaration) +macro dualRoleMacro() + +// File scope +#dualRoleMacro // expanded as a declaration + +func foo() { + #dualRoleMacro // expanded as a declaration + + _ = #dualRoleMacro // expanded as an expression + + bar(#dualRoleMacro) // expanded as an expression + + takesClosure { + #dualRoleMacro // expanded as a declaration + } +} +``` + +If a future use case deems this feature necessary, this restriction can be lifted following its own proposal. + +## Revision History + +- Scoped code item macros out as a future direction. + +## Future directions + +### Code item macros + +A code item macro is another kind of freestanding macro that can produce any mix of declarations, statements, and expressions, which are collectively called "code items" in the grammar. Code item macros can be used for top-level code and within the bodies of functions and closures. They are declaration with `@freestanding(codeItem)`. For example, we could declare a macro that logs when we are entering and exiting a function: + +```swift +@freestanding(codeItem) macro logEntryExit(arguments: Any...) +``` + +Code item macros are implemented as types conforming to the `CodeItemMacro` protocol: + +```swift +public protocol CodeItemMacro: FreestandingMacro { + /// Expand a macro described by the given freestanding macro expansion declaration + /// within the given context to produce a set of code items, which can be any mix of + /// expressions, statements, and declarations. + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> [CodeBlockItemSyntax] +} +``` + +The `logEntryExit` macro could introduce code such as: + +```swift +print("- Entering \(#function)(\(arguments))") +defer { + print("- Exiting \(#function)(\(arguments))") +} +``` + +Code item macros can only introduce new declarations that have unique names, created with `makeUniqueName(_:)`. They cannot introduce named declarations, because doing so affects the ability to type-check without repeatedly expanding the macro with potentially complete information. See the section on the visibility of names used and introduced by macros. + +Code item macros are currently under both `FreestandingMacros` and `CodeItemMacros` experimental feature flags. diff --git a/proposals/0398-variadic-types.md b/proposals/0398-variadic-types.md new file mode 100644 index 0000000000..a138841f25 --- /dev/null +++ b/proposals/0398-variadic-types.md @@ -0,0 +1,252 @@ +# Allow Generic Types to Abstract Over Packs + +* Proposal: [SE-0398](0398-variadic-types.md) +* Authors: [Slava Pestov](https://github.com/slavapestov), [Holly Borla](https://github.com/hborla) +* Review Manager: [Frederick Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 5.9)** +* Previous Proposal: [SE-0393](0393-parameter-packs.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-variadic-generic-types-abstracting-over-packs/64377)) ([review](https://forums.swift.org/t/se-0398-allow-generic-types-to-abstract-over-packs/64661)) ([acceptance](https://forums.swift.org/t/accepted-se-0398-allow-generic-types-to-abstract-over-packs/64998)) + +## Introduction + +Previously [SE-0393](0393-parameter-packs.md) introduced type parameter packs and several related concepts, allowing generic function declarations to abstract over a variable number of types. This proposal generalizes these ideas to generic type declarations. + +## Motivation + +Generic type declarations that abstract over a variable number of types arise naturally when attempting to generalize common algorithms on collections. For example, the current `zip` function returns a `Zip2Sequence`, but it's not possible from SE-0393 alone to define an equivalent variadic `zip` function because the return type would need an arbitrary number of type parameters—one for each input sequence: + +```swift +func zip(_ seq: repeat each S) -> ??? + where repeat each S: Sequence +``` + +## Proposed solution + +In the generic parameter list of a generic type, the `each` keyword declares a generic parameter pack, just like it does in the generic parameter list of a generic function. The types of stored properties can contain pack expansion types, as in `let seq` and `var iter` below. + +This lets us define the return type of the variadic `zip` function as follows: + +```swift +struct ZipSequence: Sequence { + typealias Element = (repeat (each S).Element) + + let seq: (repeat each S) + + func makeIterator() -> Iterator { + return Iterator(iter: (repeat (each seq).makeIterator())) + } + + struct Iterator: IteratorProtocol { + typealias Element = (repeat (each S).Element) + + var iter: (repeat (each S).Iterator) + + mutating func next() -> Element? { + return ... + } + } +} + +func zip(_ seq: repeat each S) -> ZipSequence + where repeat each S: Sequence +``` + +## Detailed design + +Swift has the following kinds of generic type declarations: + +- Structs +- Enums +- Classes (and actors) +- Type aliases + +A generic type is _variadic_ if it directly declares a type parameter pack with `each`, or if it is nested inside of another variadic type. In this proposal, structs, classes, actors and type aliases can be variadic. Enums will be addressed in a follow-up proposal. + +### Single parameter + +A generic type is limited to declaring at most one type parameter pack. The following are allowed: + +```swift +struct S1 {} +struct S2 {} +struct S3 {} +``` + +But this is not: + +```swift +struct S4 {} +``` + +However, by virtue of nesting, a variadic type can still abstract over multiple type parameter packs: + +```swift +struct Outer { + struct Inner { + var fn: (repeat each T) -> (repeat each U) + } +} +``` + +### Referencing a variadic type + +When used with a variadic type, the generic argument syntax `S<...>` allows a variable number of arguments to be specified. Since there can only be one generic parameter pack, the non-pack parameters are always specified with a fixed prefix and suffix of the generic argument list. + +```swift +struct S {} + +S.self // T := Int, U := Pack{}, V := Float +S.self // T := Int, U := Pack{Bool}, V := Float +S.self // T := Int, U := Pack{Bool, String}, V := Float +``` + +Note that `S` substitutes U with the empty pack type, which is allowed. The minimum number of generic arguments is equal to the number of non-pack generic parameters. In our above example, the minimum argument count is 2, because `T` and `V` must always be specified: + +```swift +S.self // error: expected at least 2 generic arguments +``` + +If the generic parameter list of a variadic type consists of a single generic parameter pack and nothing else, it is possible to reference it with an empty generic argument list: + +```swift +struct V {} + +V< >.self +``` +Note that `V< >` is not the same as `V`. The former substitutes the generic parameter pack `T` with the empty pack. The latter does not constrain the pack at all and is only permitted in contexts where the generic argument can be inferred (or within the body of `V` or an extension thereof, where it is considered identical to `Self`). + +A placeholder type in the generic argument list of a variadic generic type is always understood as a single pack element. For example: + +```swift +struct V {} + +let x: V<_> = V() // okay +let x: V<_, _> = V() // okay +let x: V<_> = V() // error +``` + +### Stored properties + +In a variadic type, the type of a stored property can contain pack expansion types. The type of a stored property cannot _itself_ be a pack expansion type. Stored properties are limited to having pack expansions nested inside tuple types, function types and other named variadic types: + +```swift +struct S { + var a: (repeat each Array) + var b: (repeat each T) -> (Int) + var c: Other +} +``` +This is in contrast with the parameters of generic function declarations, which can have a pack expansion type. A [future proposal](#future-directions) might lift this restriction and introduce true "stored property packs." + +### Requirements + +The behavior of generic requirements on type parameter packs is mostly unchanged between generic functions and generic types. However, allowing types to abstract over parameter packs introduces _requirement inference_ of [generic requirement expansions](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0393-parameter-packs.md#generic-requirements). Requirement expansion inference follows these rules: + +1. If a generic type that imposes an inferred scalar requirement is applied to a pack element inside a pack expansion, the inferred requirement is a requirement expansion. +2. If a generic type imposes an inferred requirement expansion, the requirement is expanded for each of the concrete generic arguments. +3. If a generic type that imposes an inferred requirement expansion is applied to a pack element inside a pack expansion: + + 1. The inferred requirement is invalid if it contains multiple pack elements captured by expansions at different depths. + 2. Otherwise, the nested requirement expansion is semantically equivalent to the innermost requirement expansion. + + The below code demonstrates each of the above rules: + + ```swift + protocol P { + associatedtype A + } + struct ImposeRequirement where T: P {} + struct ImposeRepeatedRequirement where repeat each T: P {} + struct ImposeRepeatedSameType where repeat T1.A == each T2 {} + + // Infers 'repeat each U: P' + func demonstrate1(_: repeat ImposeRequirement) + + // Infers 'Int: P, V: P, repeat each U: P' + func demonstrate2(_: ImposeRepeatedRequirement) + + // Error. Would attempt to infer 'repeat repeat U'.A == V' which is not a supported requirement in the language + func demonstrate3a(_: repeat ImposeRepeatedSameType)) + + // Infers 'Int: P, repeat each V: P' + func demonstrate3b(_: repeat (each U, ImposeRepeatedRequirement)) +``` + +### Conformances + +Variadic structs, classes and actors can conform to protocols. The associated type requirements of the protocol may be fulfilled by a type alias whose underlying type contains pack expansions. + +### Type aliases + +As with the other variadic types, a variadic type alias either has a generic parameter pack of its own, or can be nested inside of another variadic generic type. + +The underlying type of a variadic type alias can reference pack expansion types in the same manner as the type of a stored property. That is, the pack expansions must appear in nested positions, but not at the top level. + +```swift +typealias Element = (repeat (each S).Element) +typealias Callback = (repeat each S) -> () +typealias Factory = Other +``` + +Like other type aliases, variadic type aliases can be nested inside of generic functions (and like other structs and classes, variadic structs and classes cannot). + +### Classes + +While there are no restrictions on non-final classes adopting type parameter packs, for the time being the proposal restricts such classes from being the superclass of another class. + +An attempt to inherit from a variadic generic class outputs an error. The correct behavior of override checking and initializer inheritance in variadic generic classes will be dealt with in a follow-up proposal: + +```swift +class Base { + func foo(t: repeat each T) {} +} + +// error: cannot inherit from a class with a type parameter pack +class Derived: Base { + override func foo(t: U, _: V) {} +} +``` + +## Source compatibility + +Variadic generic types are a new language feature which does not impact source compatibility with existing code. + +## ABI compatibility + +Variadic type aliases are not part of the binary interface of a module and do not require runtime support. + +All other variadic types make use of new entry points and other behaviors being added to the Swift runtime. Since the runtime support requires extensive changes to the type metadata logic, backward deployment to older runtimes is not supported. + +Replacing a non-variadic generic type with a variadic generic type is **not** binary-compatible in either direction. When adopting variadic generic types, binary-stable frameworks must introduce them as wholly-new symbols. + +## Future directions + +A future proposal will address variadic generic enums, and complete support for variadic generic classes. + +Another possible future direction is stored property packs, which would eliminate the need to wrap a pack expansion in a tuple type in order to store a variable number of values inside of a variadic type: + +```swift +struct S { + var a: repeat each Array +} +``` + +However, there is no expressivity lost in requiring the tuple today, since the contents of a tuple can be converted into a value pack. + +## Alternatives considered + +The one-parameter limitation could be lifted if we introduced labeled generic parameters at the same time. The choice to allow only a single (unlabeled) generic parameter pack does not preclude this possibility from being explored in the future. + +Another alternative is to not enforce the one-parameter limitation at all. There would then exist variadic generic types which cannot be spelled explicitly, but can still be constructed by type inference: + +```swift +struct S { + init(t: repeat each T, u: repeat each U) {} +} + +S(t: 1, "hi", u: false) +``` + +It was felt that in the end, the single-parameter model is the simplest. + +We could require that variadic classes are declared `final`, instead of rejecting subclassing at the point of use. However, since adding or removing `final` on a class is an ABI break, this would preclude the possibility of publishing APIs which work with the existing compiler but can allow subclassing in the future. diff --git a/proposals/0399-tuple-of-value-pack-expansion.md b/proposals/0399-tuple-of-value-pack-expansion.md new file mode 100644 index 0000000000..8918268825 --- /dev/null +++ b/proposals/0399-tuple-of-value-pack-expansion.md @@ -0,0 +1,127 @@ +# Tuple of value pack expansion + +* Proposal: [SE-0399](0399-tuple-of-value-pack-expansion.md) +* Authors: [Sophia Poirier](https://github.com/sophiapoirier), [Holly Borla](https://github.com/hborla) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 5.9)** +* Implementation: On `main` gated behind `-enable-experimental-feature VariadicGenerics` +* Previous Proposals: [SE-0393](0393-parameter-packs.md), [SE-0398](0398-variadic-types.md) +* Review: ([pitch](https://forums.swift.org/t/tuple-of-value-pack-expansion/64269)) ([review](https://forums.swift.org/t/se-0399-tuple-of-value-pack-expansion/65017)) ([acceptance](https://forums.swift.org/t/accepted-se-0399-tuple-of-value-pack-expansion/65271)) + +## Introduction + +Building upon the **Value and Type Parameter Packs** proposal [SE-0393](https://forums.swift.org/t/se-0393-value-and-type-parameter-packs/63859), this proposal enables referencing a tuple value that contains a value pack inside a pack repetition pattern. + +## Motivation + +When a tuple value contains a value pack, there is no way to reference those pack elements or pass them as a value pack function argument. + +Additionally, type parameter packs are only permitted within a function parameter list, tuple element, or generic argument list. This precludes declaring a type parameter pack as a function return type, type alias, or as a local variable type to permit storage. The available solution to these restrictions is to contain the value pack in a tuple, which makes it important to provide full functional parity between value packs and tuple values containing them. This proposal fills that functionality gap by providing a method to reference individual value pack elements contained within a tuple. + +## Proposed solution + +This proposal extends the functionality of pack repetition patterns to values of _abstract tuple type_, which enables an implicit conversion of an _abstract tuple value_ to its contained value pack. An _abstract tuple type_ is a tuple that has an unknown length and elements of unknown types. Its elements are that of a single type parameter pack and no additional elements, and no label. In other words, the elements of the type parameter pack are the elements of the tuple. An _abstract tuple value_ is a value of _abstract tuple type_. This proposal provides methods to individually access the dynamic pack elements of an abstract tuple value inside of a repetition pattern. + +The following example demonstrates how, with this proposal, we can individually reference and make use of the elements in an abstract tuple value that was returned from another function. The example also highlights some constructs that are not permitted under this proposal: + +```swift +func tuplify(_ value: repeat each T) -> (repeat each T) { + return (repeat each value) +} + +func example(_ value: repeat each T) { + let abstractTuple = tuplify(repeat each value) + repeat print(each abstractTuple) // okay as of this proposal + + let concreteTuple = (true, "two", 3) + repeat print(each concreteTuple) // invalid + + let mixedConcreteAndAbstractTuple = (1, repeat each value) + repeat print(each mixedConcreteAndAbstractTuple) // invalid + + let labeledAbstractTuple = (label: repeat each value) + repeat print(each labeledAbstractTuple) // invalid +} +``` + +### Distinction between tuple values and value packs + +The following example demonstrates a pack repetition pattern on a value pack and an abstract tuple value separately first, then together but with the repetition pattern operating only on the value pack, and finally with the repetition pattern operating on both the value pack and the tuple value's contained value pack together interleaved. Note that, because the standard library function `print` does not currently accept parameter packs but instead only a single value parameter, all of the calls to it wrap the value argument pack in a tuple (hence all of those parentheses). + +```swift +func example(packElements value: repeat each T, tuple: (repeat each T)) { + print((repeat each value)) + print((repeat each tuple)) + + print((repeat (each value, tuple))) + + print((repeat (each value, each tuple))) +} + +example(packElements: 1, 2, 3, tuple: (4, 5, 6)) + +// Prints the following output: +// (1, 2, 3) +// (4, 5, 6) +// ((1, (4, 5, 6)), (2, (4, 5, 6)), (3, (4, 5, 6))) +// ((1, 4), (2, 5), (3, 6)) +``` + +## Detailed design + +Pack reference expressions inside a repetition pattern can have abstract tuple type. The outer structure of the tuple is removed, leaving the elements of a value pack: + +```swift +func expand(value: (repeat each T)) -> (repeat (each T)?) { + return (repeat each value) +} +``` + +Applying the pack repetition pattern effectively removes the outer structure of the tuple leaving just the value pack. In the repetition expression, the base tuple is evaluated once before iterating over its elements. + +```swift +repeat each + +// the above is evaluated like this +let tempTuple = +repeat each tempTuple +``` + +## Source compatibility + +There is no source compatibility impact given that this is an additive change. It enables compiling code that previously would not compile. + +## ABI compatibility + +This proposal does not add or affect ABI as its impact is only on expressions. It does not change external declarations or types. It rests atop the ABI introduced in the **Value and Type Parameter Packs** proposal. + +## Implications on adoption + +Given that this change rests atop the ABI introduced in the **Value and Type Parameter Packs** proposal, this shares with it the same runtime back-deployment story. + +## Alternatives considered + +An earlier design required the use of an abstract tuple value expansion operator, in the form of `.element` (effectively a synthesized label for the value pack contained within the abstract tuple value). This proposal already requires a tuple with a single element that is a value pack, so it is unnecessary to explicitly call out that the expansion is occurring on that element. Requiring `.element` could also introduce potential source breakage in the case of existing code that contains a tuple using the label "element". Dropping the `.element` requirement averts the language inconsistency of designating a reserved tuple label that functions differently than any other tuple label. + +## Future directions + +### Repetition patterns for concrete tuples + +It could help unify language features to extend the repetition pattern syntax to tuples of concrete type. + +```swift +func example(_ value: repeat each T) { + let abstractTuple = (repeat each value) + let concreteTuple = (true, "two", 3) + repeat print(each abstractTuple) + repeat print(each concreteTuple) // currently invalid +} +``` + +### Pack repetition patterns for arrays + +If all elements in a type argument pack are the same or share a conformance, then it should be possible to declare an Array value using a value pack. + +### Lift the single value pack with no label restriction + +This would be required to enable pack repetition patterns for a contained value pack amongst arbitrary other tuple elements that could be addressable via their labels. diff --git a/proposals/0400-init-accessors.md b/proposals/0400-init-accessors.md new file mode 100644 index 0000000000..852421b0fd --- /dev/null +++ b/proposals/0400-init-accessors.md @@ -0,0 +1,657 @@ +# Init Accessors + +* Proposal: [SE-0400](0400-init-accessors.md) +* Authors: [Holly Borla](https://github.com/hborla), [Doug Gregor](https://github.com/douggregor) +* Review Manager: [Frederick Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 5.9)** +* Implementation: On `main` behind experimental feature flag `InitAccessors` +* Review: ([pitch](https://forums.swift.org/t/pitch-init-accessors/64881)) ([review](https://forums.swift.org/t/se-0400-init-accessors/65583)) ([acceptance](https://forums.swift.org/t/accepted-se-0400-init-accessors/66212)) + +## Introduction + +Init accessors generalize the out-of-line initialization feature of property wrappers to allow any computed property on types to opt into definite initialization analysis, and subsume initialization of a set of stored properties with custom initialization code. + +## Motivation + +Swift applies [definite initialization analysis](https://en.wikipedia.org/wiki/Definite_assignment_analysis) to stored properties, stored local variables, and variables with property wrappers. Definite initialization ensures that memory is initialized on all paths before it is accessed. A common pattern in Swift code is to use one property as backing storage for one or more computed properties, and abstractions like [property wrappers](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md) and [attached macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md) help facilitate this pattern. Under this pattern, the backing storage is an implementation detail, and most code works with the computed property, including initializers. + +Property wrappers support bespoke definite initialization that allows initializing the backing property wrapper storage via the computed property, always re-writing initialization-via-wrapped-property in the form `self.value = value` to initialization of the backing storage in the form of `_value = Wrapper(wrappedValue: value)`: + +```swift +@propertyWrapper +struct Wrapper { + var wrappedValue: T +} + +struct S { + @Wrapper var value: Int + + init(value: Int) { + self.value = value // Re-written to self._value = Wrapper(wrappedValue: value) + } + + init(other: Int) { + self._value = Wrapper(wrappedValue: other) // Okay, initializes storage '_value' directly + } +} +``` + +The ad-hoc nature of property wrapper initializers mixed with an exact definite initialization pattern prevent property wrappers with additional arguments from being initialized out-of-line. Furthermore, property-wrapper-like macros cannot achieve the same initializer usability, because any backing storage variables added must be initialized directly instead of supporting initialization through computed properties. For example, the [`@Observable` macro](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0395-observability.md) applies a property-wrapper-like transform that turns stored properties into computed properties backed by the observation APIs, but it provides no way to write an initializer using the original property names like the programmer expects: + +```swift +@Observable +struct Proposal { + var title: String + var text: String + + init(title: String, text: String) { + self.title = title // error: 'self' used before all stored properties are initialized + self.text = text // error: 'self' used before all stored properties are initialized + } // error: Return from initializer without initializing all stored properties +} +``` + +## Proposed solution + +This proposal adds _`init` accessors_ to opt computed properties on types into definite initialization that subsumes initialization of a set of zero or more specified stored properties, which allows assigning to computed properties in the body of a type's initializer: + +```swift +struct Angle { + var degrees: Double + var radians: Double { + @storageRestrictions(initializes: degrees) + init(initialValue) { + degrees = initialValue * 180 / .pi + } + + get { degrees * .pi / 180 } + set { degrees = newValue * 180 / .pi } + } + + init(degrees: Double) { + self.degrees = degrees // initializes 'self.degrees' directly + } + + init(radiansParam: Double) { + self.radians = radiansParam // calls init accessor for 'self.radians', passing 'radiansParam' as the argument + } +} +``` + +The signature of an `init` accessor specifies up to two sets of stored properties: the properties that are accessed (via `accesses`) and the properties that are initialized (via `initializes`) by the accessor. `initializes` and `accesses` are side-effects of the `init` accessor. Access effects specify the other stored properties that can be accessed from within the `init` accessor (no other uses of `self` are allowed), and therefore must be initialized before the computed property's `init` accessor is invoked. The `init` accessor must initialize each of the initialized stored properties on all control flow paths. The `radians` property in the example above specifies no access effect, but initializes the `degrees` property, so it specifies only `initializes: degrees`. + +Access effects allow a computed property to be initialized by placing its contents into another stored property: + +```swift +struct ProposalViaDictionary { + private var dictionary: [String: String] + + var title: String { + @storageRestrictions(accesses: dictionary) + init(newValue) { + dictionary["title"] = newValue + } + + get { dictionary["title"]! } + set { dictionary["title"] = newValue } + } + + var text: String { + @storageRestrictions(accesses: dictionary) + init(newValue) { + dictionary["text"] = newValue + } + + get { dictionary["text"]! } + set { dictionary["text"] = newValue } + } + + init(title: String, text: String) { + self.dictionary = [:] // 'dictionary' must be initialized before init accessors access it + self.title = title // calls init accessor to insert title into the dictionary + self.text = text // calls init accessor to insert text into the dictionary + + // it is an error to omit either initialization above + } +} +``` + +Both `init` accessors document that they access `dictionary`, which allows them to insert the new values into the dictionary with the appropriate key as part of initialization. This allows one to fully abstract away the storage mechanism used in the type. + +Finally, computed properties with `init` accessors are privileged in the synthesized member-wise initializer. With this proposal, property wrappers have no bespoke definite and member-wise initialization support. Instead, the desugaring for property wrappers with an `init(wrappedValue:)` includes an `init` accessor for wrapped properties and a member-wise initializer including wrapped values instead of the respective backing storage. The property wrapper code in the Motivation section will desugar to the following code: + +```swift +@propertyWrapper +struct Wrapper { + var wrappedValue: T +} + +struct S { + private var _value: Wrapper + var value: Int { + @storageRestrictions(initializes: _value) + init(newValue) { + self._value = Wrapper(wrappedValue: newValue) + } + + get { _value.wrappedValue } + set { _value.wrappedValue = newValue } + } + + // This initializer is the same as the generated member-wise initializer. + init(value: Int) { + self.value = value // Calls 'init' accessor on 'self.value' + } +} + +S(value: 10) +``` + +This proposal allows macros to model the following property-wrapper-like patterns including out-of-line initialization of the computed property: +* A wrapped property with attribute arguments +* A wrapped property that is backed by an explicit stored property +* A set of wrapped properties that are backed by a single stored property + +## Detailed design + +### Syntax + +The proposal adds a new kind of accessor, an `init` accessor, which can be written in the accessor list of a computed property. Init accessors add the following production rules to the grammar: + +``` +init-accessor -> 'init' init-accessor-parameter[opt] function-body + +init-accessor-parameter -> '(' identifier ')' + +accessor-block -> init-accessor +``` + +The `identifier` in an `init-accessor-parameter`, if provided, is the name of the parameter that contains the initial value. If not provided, a parameter with the name `newValue` is automatically created. The minimal init accessor has no parameter list and no initialization effects: + +```swift +struct Minimal { + var value: Int { + init { + print("init accessor called with \(newValue)") + } + + get { 0 } + } +} +``` + +This proposal also adds a new `storageRestrictions` attribute to describe the storage restrictions for `init` accessor blocks. The attribute can only be used on `init` accessors. The attribute is described by the following production rules in the grammar: + +``` +attribute ::= storage-restrictions-attribute + +storage-restrictions-attribute ::= '@' storageRestrictions '(' storage-restrictions[opt] ')' + +storage-restrictions-initializes ::= 'initializes' ':' identifier-list +storage-restrictions-accesses ::= 'accesses' ':' identifier-list + +storage-restrictions ::= storage-restrictions-accesses +storage-restrictions ::= storage-restrictions-initializes +storage-restrictions ::= storage-restrictions-initializes ',' storage-restrictions-accesses +``` + +The storage restriction attribute can include a list of stored properties that are initialized by this accessor (the identifier list in `storage-restrictions-initializes`), and a list of stored properties that are accessed by this accessor (the identifier list in `storage-restrictions-accesses`), each of which are optional: + +```swift +struct S { + var readMe: String + + var _x: Int + + var x: Int { + @storageRestrictions(initializes: _x, accesses: readMe) + init(newValue) { + print(readMe) + _x = newValue + } + + get { _x } + set { _x = newValue } + } +} +``` + +If the accessor uses the default parameter name `newValue` and neither initializes nor accesses any stored property, the signature is not required. + +Init accessors can subsume the initialization of a set of stored properties. Subsumed stored properties are specified through the `initializes` argument to the attribute. The body of an `init` accessor is required to initialize the subsumed stored properties on all control flow paths. + +Init accessors can also require a set of stored properties to already be initialized when the body is evaluated, which are specified through the `accesses` argument to the attribute. These stored properties can be accessed in the accessor body; no other properties or methods on `self` are available inside the accessor body, nor is `self` available as a whole object (i.e., to call methods on it). + +### Definite initialization of properties on `self` + +The semantics of an assignment inside of a type's initializer depend on whether or not all of `self` is initialized on all paths at the point of assignment. Before all of `self` is initialized, assignment to a computed property with an `init` accessor is re-written to an `init` accessor call; after `self` has been initialized, assignment to a computed property is re-written to a setter call. + +With this proposal, all of `self` is initialized if: +* All stored properties are initialized on all paths, and +* All computed properties with `init` accessors are initialized on all paths. + +An assignment to a computed property with an `init` accessor before all of `self` is initialized will call the computed property's `init` accessor and initialize all of the stored properties specified in its `initializes` clause: + +```swift +struct S { + var x1: Int + var x2: Int + + var computed: Int { + @storageRestrictions(initializes: x1, x2) + init(newValue) { ... } + } + + init() { + self.computed = 1 // initializes 'computed', 'x1', and 'x2'; 'self' is now fully initialized + } +} +``` + +An assignment to a computed property that has not been initialized on all paths will be re-written to an `init` accessor call: + +```swift +struct S { + var x: Int + var y: Int + + var point: (Int, Int) { + @storageRestrictions(initializes: x, y) + init(newValue) { + (self.x, self.y) = newValue + } + get { (x, y) } + set { (x, y) = newValue } + } + + init(x: Int, y: Int) { + if (x == y) { + self.point = (x, x) // calls 'init' accessor + } + + // 'self.point' is not initialized on all paths here + + self.point = (x, y) // calls 'init' accessor + + // 'self.point' is initialized on all paths here + } +} +``` + +An assignment to a stored property before all of `self` is initialized will initialize that stored property. When all of the stored properties listed in the `initializes` clause of a computed property with an `init` accessor have been initialized, that computed property is considered initialized: + +```swift +struct S { + var x1: Int + var x2: Int + var x3: Int + + var computed: Int { + @storageRestrictions(initializes: x1, x2) + init(newValue) { ... } + } + + init() { + self.x1 = 1 // initializes 'x1'; neither 'x2' or 'computed' is initialized + self.x2 = 1 // initializes 'x2' and 'computed' + self.x3 = 1 // initializes 'x3'; 'self' is now fully initialized + } +} +``` + +An assignment to a computed property where at least one of the stored properties listed in `initializes` is initialized, but `self` is not initialized, is an error. This prevents double-initialization of the underlying stored properties: + +```swift +struct S { + var x: Int + var y: Int + + var point: (Int, Int) { + @storageRestrictions(initializes: x, y) + init(newValue) { + (self.x, self.y) = newValue + } + get { (x, y) } + set { (x, y) = newValue } + } + + init(x: Int, y: Int) { + self.x = x // Only initializes 'x' + self.point = (x, y) // error: neither the `init` accessor nor the setter can be called here + } +} +``` + +### Memberwise initializers + +If a struct does not declare its own initializers, it receives an implicit memberwise initializer based on the stored properties of the struct, because the storage is what needs to be initialized. Because many use-cases for `init` accessors are fully abstracting a single computed property to be backed by a single stored property, such as in the property-wrapper use case, an `init` accessor provides a preferred mechanism for initializing storage because the programmer will primarily interact with that storage through the computed property. As such, the memberwise initializer parameter list will include computed properties that have init accessors along with only those stored properties that have not been subsumed by an init accessor. + +```swift +struct S { + var _x: Int + + var x: Int { + @storageRestrictions(initializes: _x) + init(newValue) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} + +S(x: 10, y: 100) +``` + +The above struct `S` receives a synthesized initializer: + +```swift +init(x: Int, y: Int) { + self.x = x + self.y = y +} +``` + +The parameters of the memberwise initializer follow source order. However, if an init accessor `accesses` a stored property that precedes it in the memberwise initializer, then the properties cannot be initialized in the same order as the parameters occur in the memberwise initializer. For example: + +```swift +struct S { + var _x: Int + + var x: Int { + @storageRestrictions(initializes: _x, accesses: y) + init(newValue) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} +``` + +If the memberwise initializer of the above struct were written to initialize the properties in the same order as the parameters, it would produce an error: + +```swift +init(x: Int, y: Int) { + self.x = x // error + self.y = y +} +``` + +Therefore, the compiler will order the initializations in the synthesized memberwise initializer to respect the `accesses` clauses: + +```swift +init(x: Int, y: Int) { + self.y = y + self.x = x +} +``` + +The initial review of this proposal suppressed the memberwise initializer in such cases, based on a concern that out-of-order initialization would cause surprises. However, given the fact that the fields are initialized independently (or have `accessses` relationships that define their relative ordering), and that side effects here are limited to those of the `init` accessors themselves, one has to introduce global side effects during initialization to observe any difference. + +There remain cases where a memberwise initializer cannot be synthesized. For example, if a type contains several computed properties with `init` accessors that initialize the same stored property, it is not clear which computed property should be used within the member-wise initializer. In such cases, a member-wise initializer will not be synthesized. + +### Init accessors on computed properties + +An init accessor can be provided on a computed property, in which case it is used for initialization and as a default argument in the memberwise initializer. For example, given the following: + +```swift +struct Angle { + var degrees: Double + + var radians: Double { + @storageRestrictions(initializes: degrees) + init(initialValue) { + degrees = initialValue * 180 / .pi + } + + get { degrees * .pi / 180 } + set { degrees = newValue * 180 / .pi } + } +} +``` + +The implicit memberwise initializer will contain `radians`, but not the `degrees` stored property that it subsumes: + +```swift +init(radians: Double) { + self.radians = radians // calls init accessor, subsumes initialization of 'degrees' +} +``` + +### Init accessors for read-only properties + +Init accessors can be provided for properties that lack a setter. Such properties act much like a `let` property, able to be initialized (exactly) once and not set thereafter: + +```swift +struct S { + var _x: Int + + var x: Int { + @storageRestrictions(initializes: _x) + init(initialValue) { + self._x = x + } + + get { _x } + } + + init(halfOf y: Int) { + self.x = y / 2 // okay, calls init accessor for x + self.x = y / 2 // error, 'x' cannot be set + } +} + +``` + +### Initial values on properties with an init accessor + +A property with an init accessor can have an initial value, e.g., + +```swift +struct WithInitialValues { + var _x: Int + + var x: Int = 0 { + @storageRestrictions(initializes: _x) + init(initialValue) { + _x = initialValue + } + + get { ... } + set { ... } + } + + var y: Int +} +``` + +The synthesized memberwise initializer will use the initial value as a default argument, so it will look like the following: + +```swift +init(x: Int = 0, y: Int) { + self.x = x // calls init accessor, which initializes _x + self.y = y +} +``` + +In a manually written initializer, the initial value will be used to initialize the property with the init accessor prior to any user-written code: + +```swift +init() { + // implicitly initializes self.x = 0 + self.y = 10 + self.x = 20 // calls setter +} +``` + +### Restrictions + +A property with an `init` accessor can only be declared in the primary +declaration of a type. + +## Source compatibility + +`init` accessors are an additive capability with new syntax; there is no impact on existing source code. + +## ABI compatibility + +`init` accessors are an ABI-additive change; they are at most `internal` but can +be ABI-public. +Calling an `init` accessor from an `inlinable` type initializer requires that +the `init` accessor is ABI-public. + +## Implications on adoption + +Because `init` accessors are always called from within the defining module, adopting `init` accessors is an ABI-compatible change. Adding an `init` accessor to an existing property also cannot have any source compatibility impact outside of the defining module; the only possible source incompatibilities are on the generated memberwise initializer (if new entries are added), or on the type's `init` implementation (if new initialization effects are added). + +## Alternatives considered + +### Syntax for "initializes" and "accesses" + +A number of different syntaxes have been considered for specifying the set of stored properties that are initialized or accessed by a property that has an `init` accessor. The original pitch specified them in the parameter list using special labels: + +```swift +struct S { + var _x: Int + var x: Int { + init(newValue, initializes: _x, accesses: y) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} +``` + +This syntax choice is misleading because the effects look like function parameters, while `initializes` behaves more like the output of an init accessor, and `accesses` are not explicitly provided at the call-site. Conceptually, `initializes` and `accesses` are side effects of an `init` accessor, so the proposal was revised to place these modifiers in the effects clause. + +The first reviewed version of this proposal placed `initializes` and `accesses` along with other *effects*, e.g., + +```swift +struct S { + var _x: Int + var x: Int { + init(newValue) initializes(_x), accesses(y) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} +``` + +However, `initializes` and `effects` don't behave in the same manner as other effects in Swift, such as `throws` and `async`, for several reasons. First, there's no annotation like `try` or `await` at the call site. Second, these aren't part of the type of the entity (e.g., there is not function type that has an `initializes` clause). Therefore, using the effects clause is not a good match for Swift's semantic model. + +The current proposal uses an attribute. With attributes, there is question of whether we can remove the `@` to turn it into a declaration modifier: + +```swift +struct S { + var _x: Int + var x: Int { + storageRestrictions(initializes: _x, accesses: y) + init(newValue) { + _x = newValue + } + + get { _x } + set { _x = newValue } + } + + var y: Int +} +``` + +This is doable within the confines of this proposal's init accessors, but would prevent further extensions of this proposal that would allow the use of `initializes` or `accesses` on arbitrary functions. For example, such an extension might allow the following + +```swift +var _x, _y: Double + +storageRestrictions(initializes: _x, _y) +func initCoordinates(radius: Double, angle: Double) { ... } + +if let (r, theta) = decodeAsPolar() { + initCoordinates(radius: r, angle: theta) +} else { + // ... +} +``` + +However, there is a parsing ambiguity in the above because `storageRestrictions(initializes: _x, _y)` could be a call to a function names `storageRestrictions(initializes:)` or it could be a declaration modifier specifying that `initCoordinates` initializes `_x` and `_y`. + +Other syntax suggestions from pitch reviewers included: + +* Using a capture-list-style clause, e.g. `init { [&x, y] in ... }` +* Using more concise effect names, e.g. `writes` and `reads` instead of `initializes` and `accesses` +* And more! + +However, the current syntax in this proposal, which uses an attribute, most accurately models the semantics of initialization effects. An `init` accessor is a function -- not a closure -- that has side-effects related to initialization. _Only_ the `init` accessor has these effects; though the `set` accessor often contains code that looks the same as the code in the `init` accessor, the effects of these accessors are different. Because `init` accessors are called before all of `self` is initialized, they do not receive a fully-initialized `self` as a parameter like `set` accessors do, and assignments to `initializes` stored properties in `init` accessors have the same semantics as that of a standard initializer, such as suppressing `willSet` and `didSet` observers. + +## Future directions + +### `init` accessors for local variables + +`init` accessors for local variables have different implications on definite initialization, because re-writing assignment to `init` or `set` is not based on the initialization state of `self`. Local variable getters and setters can also capture any other local variables in scope, which raises more challenges for diagnosing escaping uses before initialization during the same pass where assignments may be re-written to `init` or `set`. As such, local variables with `init` accessors are a future direction. + +### Generalization of storage restrictions to other functions + +In the future, the `storageRestrictions` attribute could be be generalized to apply to other functions. For example, this could allow one to implement a common initialization function within a class: + +```swift +class C { + var id: String + var state: State + + @storageRestrictions(initializes: state, accesses: id) + func initState() { + self.state = /* initialization code here */ + } + + init(id: String) { + self.id = id + initState() // okay, accesses id and initializes state + } +} +``` + +The principles are the same as with `init` accessors: a function's implementation can be restricted to only access certain stored properties, and to initialize others along all paths. A call to the function then participates in definite initialization. + +This generalization comes with limitations that were not relevant to `init` accessors, because the functions are more akin to fragments of an initializer. For example, the `initState` function cannot be called after `state` is initialized (because it would re-initialize `state`), nor can it be used as a "first-class" function: + +```swift + init(id: String) { + self.id = id + initState() // okay, accesses id and initializes state + + initState() // error, 'state' is already initialized + let fn = self.initState // error: can't treat it like a function value + } +``` + +These limitations are severe enough that this future direction would require a significant amount of justification on its own to pursue, and therefore is not part of the `init` accessors proposal. + +## Revision history + +* Following the initial review: + * Replaced the "effects" syntax with the `@storageRestrictions` attribute. + * Add section on init accessors for computed properties. + * Add section on init accessors for read-only properties. + * Allow reordering of the initializations in the synthesized memberwise initializer to respect `accesses` restrictions. + * Add a potential future direction for the generalization of storage restrictions to other functions. + * Clarify the behavior of properties that have init accessors and initial values. + +## Acknowledgments + +Thank you to TJ Usiyan, Michel Fortin, and others for suggesting alternative syntax ideas for `init` accessor effects; thank you to Pavel Yaskevich for helping with the implementation. diff --git a/proposals/0401-remove-property-wrapper-isolation.md b/proposals/0401-remove-property-wrapper-isolation.md new file mode 100644 index 0000000000..48ba56235d --- /dev/null +++ b/proposals/0401-remove-property-wrapper-isolation.md @@ -0,0 +1,212 @@ +# Remove Actor Isolation Inference caused by Property Wrappers + +* Proposal: [SE-0401](0401-remove-property-wrapper-isolation.md) +* Authors: [BJ Homer](https://github.com/bjhomer) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#63884](https://github.com/apple/swift/pull/63884) +* Upcoming Feature Flag: `DisableOutwardActorInference` +* Review: ([pitch](https://forums.swift.org/t/pitch-stop-inferring-actor-isolation-based-on-property-wrapper-usage/63262)) ([review](https://forums.swift.org/t/se-0401-remove-actor-isolation-inference-caused-by-property-wrappers/65618)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0401-remove-actor-isolation-inference-caused-by-property-wrappers/66241)) + +## Introduction + +[SE-0316: Global Actors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0316-global-actors.md) introduced annotations like `@MainActor` to isolate a type, function, or property to a particular global actor. It also introduced various rules for how that global actor isolation could be inferred. One of those rules was: + +> Declarations that are not explicitly annotated with either a global actor or `nonisolated` can infer global actor isolation from several different places: +> +> [...] +> +> - A struct or class containing a wrapped instance property with a global actor-qualified wrappedValue infers actor isolation from that property wrapper: +> +> ```swift +> @propertyWrapper +> struct UIUpdating { +> @MainActor var wrappedValue: Wrapped +> } +> +> struct CounterView { // infers @MainActor from use of @UIUpdating +> @UIUpdating var intValue: Int = 0 +> } +> ``` + + +This proposal advocates for **removing this inference rule** when compiling in the Swift 6 language mode. Given the example above, CounterView would no longer infer `@MainActor` isolation in Swift 6. + +## Motivation + +This particular inference rule is surprising and nonobvious to many users. Some developers have trouble understanding the Swift Concurrency model because it's not obvious to them when actor isolation applies. When something is inferred, it is not visible to the user, and that makes it harder to understand. This frequently arises when using the property wrappers introduced by Apple's SwiftUI framework, although it is not limited to that framework. For example: + +### An example using SwiftUI + +```swift +struct MyView: View { + // Note that `StateObject` has a MainActor-isolated `wrappedValue` + @StateObject private var model = Model() + + var body: some View { + Text("Hello, \(model.name)") + .onAppear { viewAppeared() } + } + + // This function is inferred to be `@MainActor` + func viewAppeared() { + updateUI() + } +} + +@MainActor func updateUI() { /* do stuff here */ } +``` + +The above code compiles just fine. But if we change `@StateObject` to `@State`, we get an error: + +```diff +- @StateObject private var model = Model() ++ @State private var model = Model() +``` + +```swift + func viewAppeared() { + // error: Call to main actor-isolated global function + // 'updateUI()' in a synchronous nonisolated context + updateUI() + } +``` + +Changing `@StateObject var model` to `@State var model` caused `viewAppeared()` to stop compiling, even though that function didn't use `model` at all. It feels non-obvious that changing the declaration of one property should cause a _sibling_ function to stop compiling. In fact, we also changed the isolation of the entire `MyView` type by changing one property wrapper. + +### An example not using SwiftUI + +This problem is not isolated to SwiftUI. For example: + + +```swift +// A property wrapper for use with our database library +@propertyWrapper +struct DBParameter { + @DatabaseActor public var wrappedValue: T +} + +// Inferred `@DatabaseActor` isolation because of use of `@DBParameter` +struct DBConnection { + @DBParameter private var connectionID: Int + + func executeQuery(_ query: String) -> [DBRow] { /* implementation here */ } +} + + +// In some other file... + +@DatabaseActor +func fetchOrdersFromDatabase() async -> [Order] { + let connection = DBConnection() + + // No 'await' needed here, because 'connection' is also isolated to `DatabaseActor`. + connection.executeQuery("...") +} +``` + +Removing the property wrapper on `DBConnection.connectionID` would remove the inferred actor isolation of `DBConnection`, which would in turn cause `fetchOrdersFromDatabase` to fail to compile. **It's unprecedented in Swift that changes to a _private_ property should cause compilation errors in some entirely separate file**. Upward inference of actor isolation (from property wrappers to their containing type) means that we can no longer locally reason about the effects of even *private* properties within a type. Instead, we get "spooky action at a distance". + +### Does this cause actual problems? + +This behavior has caused quite a bit of confusion in the community. For example, see [this tweet](https://twitter.com/teilweise/status/1580105376913297409?s=61&t=hwuO4NDJK1aIxSntRwDuZw), [this blog post](https://oleb.net/2022/swiftui-task-mainactor/), and [this entire Swift Forums thread](https://forums.swift.org/t/reconsider-inference-of-global-actor-based-on-property-wrappers/60821). One particular callout comes from [this post](https://forums.swift.org/t/reconsider-inference-of-global-actor-based-on-property-wrappers/60821/6/), where this inference made it hard to adopt Swift Concurrency in some cases, because the actor isolation goes "viral" beyond the intended scope: + +```swift +class MyContainer { + let contained = Contained() // error: Call to main actor-isolated initializer 'init()' in a synchronous nonisolated context +} + +class Contained { + @OnMainThread var i = 1 +} +``` + +The author created an `@OnMainThread` property wrapper, intended to declare that a particular property was isolated to the main thread. However, they cannot enforce that by using `@MainActor` within the property wrapper, because doing so causes the entire contained type to become unexpectedly isolated. + +The [original motivation](https://forums.swift.org/t/se-0401-remove-actor-isolation-inference-caused-by-property-wrappers/65618/10) for this inference rule was to reduce the annotation burden when using property wrappers like SwiftUI's `@ObservedObject`. But it's not clear it actually makes anything significantly easier; it only saves us from writing a single annotation on the type, and the loss of that annotation introduces violations of the [principle of least surprise](https://en.wikipedia.org/wiki/Principle_of_least_astonishment). + + +## Proposed solution + +The proposal is simple: In the Swift 6 language mode, property wrappers used within a type will not affect the type's actor isolation. We simply disable this inference step entirely. + +In the Swift 5 language mode, isolation will continue to be inferred as it currently is. The new behavior can be requested using the **`-enable-upcoming-feature DisableOutwardActorInference`** compiler flag. + +## Detailed design + +[`ActorIsolationRequest.getIsolationFromWrappers()`](https://github.com/apple/swift/blob/85d59d2e55e5e063c552c15f12a8abe933d8438a/lib/Sema/TypeCheckConcurrency.cpp#L3618) implements the actor isolation inference described in this proposal. That function will be adjusted to avoid producing any inference when running in the Swift 6 language mode or when the compiler flag described above is passed. + +## Source compatibility + +This change _does_ introduce potential for source incompatibility, because there may be code which was relying on the inferred actor isolation. That code can be explicitly annotated with the desired global actor in a source-compatible way right now. For example, if a type is currently inferred to have `@MainActor` isolation, you could explicitly declare that isolation on the type right now to avoid source compatibility. (See note about warnings in Alternatives Considered.) + +There may be cases where the source incompatibility could be mitigated by library authors in a source-compatible way. For example, if Apple chose to make SwiftUI's `View` protocol `@MainActor`-isolated, then all conforming types would consistently be isolated to the Main Actor, rather than being inconsistently isolated based on the usage of certain property wrappers. This proposal only notes that this mitigation may be _possible_, but does not make any recommendation as to whether that is necessary. + +### Source compatibility evaluation + +In an effort to determine the practical impact of this change, I used a macOS toolchain containing these changes and evaluated various open-source Swift projects (from the Swift Source Compatibility Library and elsewhere). I found no instances of actual source incompatibility as a result of the proposed changes. Most open-source projects are libraries that use no property wrappers at all, but I tried to specifically seek out a few projects that *do* use property wrappers and may be affected by this change. The results are as follows: + +Project | Outcome | Notes +---|---|--- +[ACNHBrowserUI](https://github.com/Dimillian/ACHNBrowserUI) | Fully Compatible | Uses SwiftUI property wrappers +[AlamoFire](https://github.com/Alamofire/Alamofire) | Fully Compatible | Uses custom property wrappers, but none are actor isolated +[Day One (Mac)](https://dayoneapp.com) | Fully Compatible | Uses SwiftUI property wrappers. (Not open source) +[Eureka](https://github.com/xmartlabs/Eureka) | Fully Compatible | Does not use property wrappers at all +[NetNewsWire](https://github.com/Ranchero-Software/NetNewsWire) | Fully Compatible | Uses SwiftUI property wrappers +[swift-nio](https://github.com/apple/swift-nio) | Fully Compatible | Does not use property wrappers at all +[SwiftyJSON](https://github.com/SwiftyJSON/SwiftyJSON) | Fully Compatible | Does not use property wrappers at all +[XcodesApp](https://github.com/RobotsAndPencils/XcodesApp) | Fully Compatible | Uses SwiftUI property wrappers + +All of the above had a `Swift Concurrency Checking` setting of **Minimal** by default. When I changed the concurrency checking level to **Targeted**, all of the above continued to compile with no errors, both with and without the proposed changes. + +When I changed the concurrency checking level to **Complete**, most of the above projects had compilation errors, _even without the changes proposed here_. The changes proposed here likely contributed a few _additional_ errors under "Complete" checking, but they did not break source compatibility in projects that would have otherwise been source compatible. + +## Effect on ABI stability + +This change is ABI stable, as the actor isolation of a type is not reflected in its runtime calling convention in any way. + +## Effect on API resilience + +This proposal has no effect on API resilience. + +## Alternatives considered + +#### Warn about Property Wrapper-based inference in Swift 5 + +In certain cases, we produce a warning that code will become invalid in a future Swift release. (For example, this has been done with the planned changes to Swift Concurrency in Swift 6.) I considered adding a warning to the Swift 5 language mode along these lines: + +```swift + +// ⚠️ Warning: `MyView` is inferred to use '@MainActor' isolation because +// it uses `@StateObject`. This inference will go away in Swift 6. +// +// Add `@MainActor` to the type to silence this warning. + +struct MyView: View { + @StateObject private var model = Model() + + var body: some View { + Text("Hello") + } +} +``` + +However, I found two problems: + +1. This would produce a _lot_ of warnings, even in code that will not break under the Swift 6 language mode. + +2. There's no way to silence this warning _without_ isolating the type. If I actually _didn't_ want the type to be isolated, there's no way to express that. You can't declare a non-isolated type: + +```swift +nonisolated // 🛑 Error: 'nonisolated' modifier cannot be applied to this declaration +struct MyView: View { + /* ... */ +} +``` + +Given that users cannot silence the warning in a way that matches the new Swift 6 behavior, it seems inappropriate to produce a warning here. + + +## Acknowledgments + +Thanks to Dave DeLong for reviewing this proposal, and to the many members of the Swift community who have engaged in discussion on this topic. diff --git a/proposals/0402-extension-macros.md b/proposals/0402-extension-macros.md new file mode 100644 index 0000000000..80513bae90 --- /dev/null +++ b/proposals/0402-extension-macros.md @@ -0,0 +1,200 @@ +# Generalize `conformance` macros as `extension` macros + +* Proposal: [SE-0402](0402-extension-macros.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9)** +* Implementation: [apple/swift#66967](https://github.com/apple/swift/pull/66967), [apple/swift-syntax#1859](https://github.com/apple/swift-syntax/pull/1859) +* Review: ([pitch](https://forums.swift.org/t/pitch-generalize-conformance-macros-as-extension-macros/65653)) ([review](https://forums.swift.org/t/se-0402-generalize-conformance-macros-as-extension-macros/65965)) ([acceptance](https://forums.swift.org/t/accepted-se-0402-generalize-conformance-macros-as-extension-macros/66276)) + +## Introduction + +This proposal generalizes the `conformance` macro role as an `extension` macro role that can add a member list to an extension in addition to a protocol and `where` clause. + +## Motivation + +[SE-0389: Attached Macros](0389-attached-macros.md) introduced conformance macros, which expand to a conformance with a `where` clause written in an extension on the type the macro is attached to: + +```swift +@attached(conformance) +macro AddEquatable() = #externalMacro(...) + +@AddEquatable +struct S {} + +// expands to +extension S: Equatable {} +``` + +However, the `conformance` macro role is extremely limited on its own. A conformance macro _only_ has the ability to return a protocol name and the syntax for a `where` clause. If the protocol conformance requires members --- as most protocol conformances do --- those must be added through a separate `member` macro role. + +More importantly, conformance macros are the only way for a macro to expand to an extension on the annotated type. The inability to add members in an extension of a type rather than the primary declaration is a serious limitation of the macro system, because extensions have several important semantic implications, including (but not limited to): + +* Protocols can only provide default implementations of requirements in extensions +* Initializers added in an extension of a type do not suppress the compiler-synthesized initializers +* Computed properties and methods in protocol and class extensions do not participate in dynamic dispatch + +Extensions also have stylistic benefits. Code inside an extension will share the generic requirements on the extension itself rather than repeating the generic requirement on every method, and implementing conformance requirements in an extension is a common practice in Swift. + +## Proposed solution + +This proposal removes the `conformance` macro role in favor of an `extension` macro role. An `extension` macro role can be used with the `@attached` macro attribute, and it can add a conformance, a `where` clause, and a member list in an extension on the type the macro is attached to: + +```swift +protocol MyProtocol { + func requirement +} + +@attached(extension, conformances: MyProtocol, names: named(requirement)) +macro MyProtocol = #externalMacro(...) + +@MyProtocol +struct S {} + +// expands to + +extension S: MyProtocol where T: MyProtocol { + func requirement() { ... } +} +``` + +The generated extensions of the macro must only extend the type the macro is attached to. Any conformances or members must also be specified upfront by the `@attached(extension)` attribute. + +## Detailed design + +### Specifying macro-introduced protocol conformances and member names + +SE-0389 states that whenever a macro produces declarations that are visible to other Swift code, it is required to declare the names in advance. This rule also applies to extension macros, which must specify: + +* Declarations inside the extension, which can be specified using `named`, `prefixed`, `suffixed`, and `arbitrary`. +* The names of protocols that are listed in the extension's conformance clause. These protocols are specified in the `conformances:` list of the `@attached(conformances:)` attribute. Each name that appears in this list must be a conformance constraint, where a conformance constraint is one of: + * A protocol name + * A typealias whose underlying type is a conformance constraint + * A protocol composition whose entries are each a conformance constraint + +The following restrictions apply to generated conformances and names listed in `@attached(extension)`: + +* An extension macro cannot add a conformance to a protocol that is not covered by the `conformances:` list in `@attached(extension, conformances:)`. +* An extension macro cannot add a member that is not covered by the `names:` list in `@attached(extension, names:)`. +* An extension macro cannot introduce an extension with an attached `peer` macro, because the peer-macro-generated names are not covered by the original `@attached(extension)` attribute. + +### Extension macro application + +Extension macros can only be attached to the primary declaration of a nominal type; they cannot be attached to typealias or extension declarations. + + +Swift only allows `extension` declarations at the top level in a file. Despite this, extension macros can be applied to a nested type: + +```swift +@attached(extension, conformances: MyProtocol, names: named(requirement)) +macro MyProtocol = #externalMacro(...) + +struct Outer { + @MyProtocol + struct Inner {} +} +``` + +In this situation, the macro expansion containing the `extension` is inserted at the top level of the file, instead of immediately where the macro is invoked, where the `extension` would be invalid. The above code expands to: + +```swift +struct Outer { + struct Inner {} +} + +extension Outer.Inner: MyProtocol { + func requirement() { ... } +} +``` + +It is an error to apply an extension macro to a local type, because there is no way to write an extension on a local type in Swift: + +```swift +func test() { + @MyProtocol // error + struct LocalType {} +} +``` + +### Implementing extension macros + +Extension macro implementations should conform to the `ExtensionMacro` protocol: + +```swift +/// Describes a macro that can add extensions of the declaration it's +/// attached to. +public protocol ExtensionMacro: AttachedMacro { + /// Expand an attached extension macro to produce the contents that will + /// create a set of extensions. + /// + /// - Parameters: + /// - node: The custom attribute describing the attached macro. + /// - declaration: The declaration the macro attribute is attached to. + /// - type: The type to provide extensions of. + /// - protocols: The list of protocols to add conformances to. These will + /// always be protocols that `type` does not already state a conformance + /// to. + /// - context: The context in which to perform the macro expansion. + /// + /// - Returns: the set of extension declarations introduced by the macro, + /// which are always inserted at top-level scope. Each extension must extend + /// the `type` parameter. + static func expansion( + of node: AttributeSyntax, + attachedTo declaration: some DeclGroupSyntax, + providingExtensionsOf type: some TypeSyntaxProtocol, + conformingTo protocols: [TypeSyntax], + in context: some MacroExpansionContext + ) throws -> [ExtensionDeclSyntax] +} +``` + +Each `ExtensionDeclSyntax` in the resulting array must use the `providingExtensionsOf` parameter as the extended type, which is a qualified type name. For example, for the following code: + +```swift +struct Outer { + @MyProtocol + struct Inner {} +} +``` + +The type syntax passed to `ExtensionMacro.expansion` for `providingExtensionsOf` is `Outer.Inner`. + +#### Suppressing redundant conformances + +The `conformingTo:` parameter of `ExtensionMacro.expansion` allows extension macros to suppress generating conformances that are already stated in the original source code. The `conformingTo:` argument array will contain only the protocols from the `conformances:` list in `@attached(extension conformances:)` that the type does not already conform to in the original source code, including through implied conformances or class inheritance. + +For example, consider the following code which contains an attached extension macro: + +```swift +protocol Encodable {} +protocol Decodable {} + +typealias Codable = Encodable & Decodable + +@attached(extension, conformances: Codable) +macro MyMacro() = #externalMacro(...) + +@MyMacro +struct S { ... } + +extension S: Encodable { ... } +``` + +The extension macro can add conformances to `Codable`, aka `Encodable & Decodable`. Because the struct `S` already conforms to `Encodable` in the original source, the `ExtensionMacro.expansion` method will receive the argument `[TypeSyntax(Encodable)]` for the `conformingTo:` parameter. Using this information, the macro implementation can decide to only add an extension with a conformance to `Decodable`. + +## Source compatibility + +This proposal removes the `conformance` macro role from SE-0389, which is accepted and implemented in Swift 5.9. If this proposal is accepted after 5.9, the `conformance` macro role will remain in the language as sugar for an `extension` macro that adds only a conformance. + +## ABI compatibility + +Extensions macros are expanded to regular Swift code at compile-time and have no ABI impact. + +## Implications on adoption + +The adoption implications for using extensions macros are the same as writing the expanded code manually in the project. + +## Acknowledgments + +Thank you to Gwendal Roué for inspiring the idea of `extension` macros by suggesting combining `member` macros and `conformance` macros. diff --git a/proposals/0403-swiftpm-mixed-language-targets.md b/proposals/0403-swiftpm-mixed-language-targets.md new file mode 100644 index 0000000000..c339f2e424 --- /dev/null +++ b/proposals/0403-swiftpm-mixed-language-targets.md @@ -0,0 +1,576 @@ +# Package Manager Mixed Language Target Support + +* Proposal: [SE-0403](0403-swiftpm-mixed-language-targets.md) +* Authors: [Nick Cooke](https://github.com/ncooke3) +* Review Manager: [Saleem Abdulrasool](https://github.com/compnerd) +* Status: **Returned for Revision** +* Implementation: [apple/swift-package-manager#5919](https://github.com/apple/swift-package-manager/pull/5919) +* Review: ([pitch](https://forums.swift.org/t/61564)), ([review](https://forums.swift.org/t/66202)), ([returned for revision](https://forums.swift.org/t/66975)) + +## Introduction + +This is a proposal for adding package manager support for targets containing +both Swift and [C based language sources][SE-0038] (henceforth, referred to as +mixed language sources). Currently, a target’s source can be either Swift or a +C based language ([SE-0038]), but not both. + +Swift-evolution thread: [Discussion thread topic for that +proposal](https://forums.swift.org/) + +## Motivation + +This proposal enables Swift Package Manager support for multi-language targets. + +Packages may need to contain mixed language sources for both legacy or +technical reasons. For developers building or maintaining packages with mixed +languages (e.g. Swift and Objective-C), there are two workarounds for doing so +with Swift Package Manager, but they have drawbacks that degrade the developer +experience, and sometimes are not even an option: +- Distribute binary frameworks via binary targets. Drawbacks include that the + package will be less portable as it can only support platforms that the + binaries support, binary dependencies are only available on Apple platforms, + customers cannot view or easily debug the source in their project workspace, + and tooling is required to generate the binaries for release. +- Separate a target’s implementation into sub-targets based on language type, + adding dependencies where necessary. For example, a target `Foo` may have + Swift-only sources that can call into an underlying target `FooObjc` that + contains Clang-only sources. Drawbacks include needing to depend on the + public API surfaces between the targets, increasing the complexity of the + package’s manifest and organization for both maintainers and clients, and + preventing package developers from incrementally migrating internal + implementation from one language to another (e.g. Objective-C to Swift) since + there is still a separation across targets based on language. + +Package manager support for mixed language targets addresses both of the above +drawbacks by enabling developers to mix sources of supported languages within a +single target without complicating their package’s structure or developer +experience. + +## Proposed solution + +Package authors can create a mixed target by mixing language sources in their +target's source directory. When mixing some languages, like C++, authors have +the option of opting in to advanced interoperability features by configuring +the target with an interoperability mode [`SwiftSetting.InteroperabilityMode`]. + +When building a mixed language target, the package manager will build the +public API into a single module for use by clients. + +At a high level, the build process is split into two parts based on +the language of the sources. The Swift sources are built by the Swift compiler +and the C/Objective-C/C++ sources are built by the Clang compiler. + +1. The Swift compiler is made aware of the Clang part of the package when + building the Swift sources into a `swiftmodule`. +1. The Clang part of the package is built with knowledge of the + interoperability Swift header. The contents of this header will vary + depending on if/what language-specific interoperability mode is configured + on the target. The interoperability header is modularized as part of the + mixed target's public interface. + + +The [following example][mixed-package] defines a package containing mixed +language sources. + +``` +MixedPackage +├── Package.swift +├── Sources +│   └── MixedPackage +│ ├── Jedi.swift ⎤-- Swift sources +│ ├── Lightsaber.swift ⎦ +│ ├── Sith.m ⎤-- Implementations & internal headers +│ ├── SithRegistry.h ⎟ +│ ├── SithRegistry.m ⎟ +│   ├── droid_debug.c ⎦ +│   ├── hello_there.txt ]-- Resources +│   └── include ⎤-- Public headers +│   ├── MixedPackage.h ⎟ +│   ├── Sith.h ⎟ +│   └── droid_debug.h ⎦ +└── Tests + └── MixedPackageTests + ├── JediTests.swift ]-- Swift tests + ├── SithTests.m        ]-- Objective-C tests + ├── ObjcTestConstants.h ⎤-- Mixed language test utils + ├── ObjcTestConstants.m ⎟ + └── SwiftTestConstants.swift ⎦ +``` + +The proposed solution would enable the above targets to do the following: +1. Export their public API, if any, from across the mixed language sources. +1. Use C/Objective-C/C++ compatible Swift API from target’s Swift sources + within the target’s C/Objective-C/C++ sources. +1. Use Swift compatible C/Objective-C/C++ API from target’s C/Objective-C/C++ + sources within the target’s Swift sources. +1. Access target resources from Swift and Objective-C contexts. + +### Limitations + +Initial support for targets containing mixed language sources will have the +following limitations: +1. The target must be either a library or test target. Support for other types + of targets is deferred until the use cases become clear. +1. If the target contains a custom module map, it cannot contain a submodule of + the form `$(ModuleName).Swift`. This is because the package manager will + synthesize an _extended_ module map that includes a submodule that + modularizes the generated Swift interop header. + +### Importing a mixed target + +Mixed targets can be imported into a client target in several ways. The +following examples will reference `MixedPackage`, a package containing mixed +language target(s). + +#### Importing within a **Swift** context + +The public API of a mixed target, `MixedPackage`, can be imported into a +**Swift** file via an `import` statement: + +```swift +// MyClientTarget.swift + +import MixedPackage +``` + +Testing targets can import the mixed target via +`@testable import MixedPackage`. As expected, this will expose internal Swift +types within the module. It will not expose any non-public C language types. + +#### Importing within an **C/Objective-C/C++** context + +How a mixed target, `MixedPackage`, is imported into an **C/Objective-C/C++** +file will vary depending on the language it is being imported in. + +When Clang modules are supported, clients can import the module. Textual +imports are also an option. + + +For this example, consider `MixedPackage` being organized as such: + +``` +MixedPackage +├── Package.swift +└── Sources +    ├── NewCar.swift +   └── include ]-- Public headers directory +    ├── OldCar.h + └── MixedPackage-Swift.h ]-- This header is generated + during the build. +``` + + +Like Clang targets, `MixedPackage`'s public headers directory (`include` in the +above example) is added a header search path to client targets. The following +example demonstrates all the possible public headers that can be imported from +`MixedPackage`. + +```objc +// MyClientTarget.m + +// If module imports are supported, the public API (including API in the +// generated Swift header) can be imported via a module import. +@import MixedPackage; +// Imports types defined in `OldCar.h`. +#import "OldCar.h" +// Imports Objective-C compatible Swift types defined in `MixedPackage`. +#import "MixedPackage-Swift.h" +``` + +## Plugin Support + +Package manager plugins should be able to process mixed language source +targets. The following type will be added to the `PackagePlugin` module +to represent a mixed language target in a plugin's context. + +This API was created by joining together the properties of the existing +`SwiftSourceModuleTarget` and `ClangSourceModuleTarget` types +([source][Swift-Clang-SourceModuleTarget]). + +```swift +/// Represents a target consisting of a source code module compiled using both the Clang and Swift compiler. +public struct MixedSourceModuleTarget: SourceModuleTarget { + /// Unique identifier for the target. + public let id: ID + + /// The name of the target, as defined in the package manifest. This name + /// is unique among the targets of the package in which it is defined. + public let name: String + + /// The kind of module, describing whether it contains unit tests, contains + /// the main entry point of an executable, or neither. + public let kind: ModuleKind + + /// The absolute path of the target directory in the local file system. + public let directory: Path + + /// Any other targets on which this target depends, in the same order as + /// they are specified in the package manifest. Conditional dependencies + /// that do not apply have already been filtered out. + public let dependencies: [TargetDependency] + + /// The name of the module produced by the target (derived from the target + /// name, though future SwiftPM versions may allow this to be customized). + public let moduleName: String + + /// The source files that are associated with this target (any files that + /// have been excluded in the manifest have already been filtered out). + public let sourceFiles: FileList + + /// Any custom compilation conditions specified for the target's Swift sources. + public let swiftCompilationConditions: [String] + + /// Any preprocessor definitions specified for the target's Clang sources. + public let clangPreprocessorDefinitions: [String] + + /// Any custom header search paths specified for the Clang target. + public let headerSearchPaths: [String] + + /// The directory containing public C headers, if applicable. This will + /// only be set for targets that have a directory of a public headers. + public let publicHeadersDirectory: Path? + + /// Any custom linked libraries required by the module, as specified in the + /// package manifest. + public let linkedLibraries: [String] + + /// Any custom linked frameworks required by the module, as specified in the + /// package manifest. + public let linkedFrameworks: [String] +} +``` + + +## Detailed design + +### Modeling a mixed language target + +Up until this proposal, when a package was loading, each target was represented +programmatically as either a [`SwiftTarget`] or [`ClangTarget`]. Which of these +types to use was informed by the sources found in the target. For targets with +mixed language sources, an error was thrown and surfaced to the client. During +the build process, each of those types mapped to another type +([`SwiftTargetBuildDescription`] or [`ClangTargetBuildDescription`]) that +described how the target should be built. + +This proposal adds two new types, `MixedTarget` and `MixedTargetDescription`, +that represent targets with mixed language sources during the package loading +and building phases, respectively. + +While an implementation detail, it’s worth noting that in this approach, a +`MixedTarget` is a wrapper type around an underlying `SwiftTarget` and +`ClangTarget`. Initializing a `MixedTarget` will internally initialize a +`SwiftTarget` from the given Swift sources and a `ClangTarget` from the given +Clang sources. This extends to the `MixedTargetDescription` type in that it +wraps a `SwiftTargetDescription` and `ClangTargetDescription`. + +Using this approach allows for greater code-reuse, and reduces the chance of +introducing a regression from changing existing sub-target types like +`SwiftTarget` and `ClangTarget`. + +The role of the `MixedTargetBuildDescription` is to generate auxiliary +artifacts needed for the build and pass specific build flags to the underlying +`SwiftTargetBuildDescription` and `ClangTargetBuildDescription`. + +The following diagram shows the relationship between the various types. +```mermaid +flowchart LR + A>Swift sources] --> B[SwiftTarget] --> C[SwiftTargetBuildDescription] + D>Clang sources] --> E[ClangTarget] --> F[ClangTargetBuildDescription] + + subgraph MixedTarget + SwiftTarget + ClangTarget + end + + subgraph MixedTargetBuildDescription + SwiftTargetBuildDescription + ClangTargetBuildDescription + end + + G>Mixed sources] --> MixedTarget --> MixedTargetBuildDescription +``` + +### Building a mixed language target + + + + + + + +The Swift part of the target is built before the Clang part. This is because +the C language sources may require resolving a textual import of the generated +interop header, and that header is emitted alongside the Swift module when the +Swift part of the target is built. This relationship is enforced in that the +generated interop header is listed as an input to the compilation commands for +the target’s C language sources. This is specified in the llbuild manifest +(`debug.yaml` in the package's `.build` directory). + +##### Additional Swift build flags +The following flags are additionally used when compiling the Swift sub-target: +1. `-import-underlying-module` This flag triggers a partial build of the + underlying C language sources when building the Swift module. This critical + flag enables the Swift sources to use C language types defined in the Clang + part of the target. +1. `-I /path/to/modulemap_dir` The above `-import-underlying-module` flag + will look for a module map in the given header search path. The module + map used here cannot modularize the generated interop header as will be + created from building the Swift sub-target and therefore does not exist + yet. If a custom module map is provided, the public headers directory + will be used as that is where the custom module map is enforced to be + located. It's also enforced that this module map does not expose an + interop header. If a custom module map is _not_ provided, the package + manager will pass the target's build directory as that is where a module + map will be synthesized. This module map will be _un-extended_, in that + it does not modularize the generated interop header. +1. _If a custom module is NOT provided,_ the package manager will synthesize + two module maps. One is _extended_ in that it modualrizes the generated + interop header. The other is _un-extended_ in that it does not modularize + the generated interop header. A VFS Overlay file is created to swap the + extended one (named `module.modulemap`) for the unextended one + (`unextended-module.modulemap`) for the build. +1. `-Xcc -I -Xcc $(TARGET_SRC_PATH)` Adding the target's [path] allows for + importing headers using paths relative to the root of the target. Because + passing `-import-underlying-module` triggers a partial build of the Clang + sources, this is needed for resolving possible header imports. +1. `-Xcc -I -Xcc $(TARGET_PUBLIC_HDRS)` Adding the target's public header's + path allows for importing headers using paths relative to the public + header's directory. Because passing `-import-underlying-module` triggers + a partial build of the Clang sources, this is needed for resolving + possible header imports. + +##### Additional Clang build flags +The following flags are additionally used when compiling the Clang sub-target: +1. `-I $(target’s path)` Adding the target's [path] allows for importing + headers using paths relative to the root of the target. +1. `-I /path/to/generated_swift_header_dir/` The generated Swift header may be + needed when compiling the Clang sources. + +#### Performing the build + +To actually build a package, the package manager creates a llbuild manifest and +passes it to the llbuild system. Adding support for mixed targets involved +modifying [LLBuildManifestBuilder.swift] to convert a +`MixedTargetBuildDescription` into llbuild build nodes. +`MixedTargetBuildDescription` intentionally wraps and configures an underlying +`SwiftTargetBuildDescription` and `ClangTargetBuildDescription`. This means +that creating a llbuild build node for a mixed target is really just creating +build nodes for the its `SwiftTargetBuildDescription` and +`ClangTargetBuildDescription`, respectively. + +#### Build artifacts for client targets + + + + + + + +##### Module Maps + +The client-facing module map’s purpose is to define the public API of the mixed +language module. It has two parts, a primary module declaration and a secondary +submodule declaration. The former of which exposes the public C language +headers and the latter of which exposes the generated interop header. + +There are two cases when creating the client-facing module map: +- If a custom module map exists in the target, its contents are copied and + extended to modularize the generated interop header. These contents are + written to the build directory as `extended-custom-module.modulemap`. + Since the public header directory and build directory are passed as import + paths to the build invocations, a different name is needed for this module + map as the `-import-underlying-module` should only be able to find one + `module.modulemap` file from the given import paths. +- Else, the module map’s contents will be generated via the same + generation rules established in [SE-0038] with an added step to generate the + `.Swift` submodule. This file is called `module.modulemap` and lives in the + build directory. + +Clients will use an _extended_ module map that includes the modularized interop +header. Building the target will use _unextended_ module map. + +> Note: It’s possible that the Clang part of the module exports no public API. +> This could be the case for a target whose public API surface is written in +> Swift but whose implementation is written in Objective-C. In this case, the +> primary module declaration will expose no headers. + +Below is an example of a module map for a target that has an umbrella +header in its public headers directory (`include`). + +``` +// extended-custom-module.modulemap + +// This declaration is either copied from the custom module map or generated +// via the rules from SE-0038. +module MixedTarget { + umbrella header "/Users/crusty/Developer/MixedTarget/Sources/MixedTarget/include/MixedTarget.h" + export * +} +// This is added on by the package manager as part of this proposal. +module MixedTarget.Swift { + header "MixedTarget-Swift.h" + requires objc +} +``` + +##### all-product-headers.yaml + +An `all-product-headers.yaml` VFS overlay file will adjust the public headers +directory to expose the interop header as a relative path, and, if a custom +module map exists, swap it out for the extended one that modualrizes the interop +header. + +In either case, it will be passed alongside the module map as a compilation +argument to clients: +``` +-fmodule-map-file=/Users/crusty/Developer/MixedTarget/Sources/MixedTarget/include/module.modulemap +-ivfsoverlay /Users/crusty/Developer/MixedTarget/.build/.../MixedTarget.build/Product/all-product-headers.yaml +``` + +### Additional changes to the package manager + +It is the goal for mixed language targets to work on all platforms supported +by the package manager. One obstacle to that is that the package manager, +at the time of this proposal, does not invoke the build system with the +flag needed to emit the interoperability header +([code][should-emit-header]). This limitation is outdated and will be +removed as part of this proposal. + +See the related discussion [thread][swift-emit-header-fr] from the initial +formal review. + +### Related change to the Swift compiler + +When the Swift compiler creates the generated interop header (via +`-emit-objc-header`), any Objective-C symbol referenced in the Swift API that +cannot be forward declared (e.g. superclass, protocol, etc.) will attempt to +be imported via an umbrella header. Since the compiler evaluates +the target as a framework (as opposed to an app), the compiler assumes an +umbrella header exists in a subdirectory (named after the module) within +the public headers directory: + + #import <$(ModuleName)/$(ModuleName).h> + +The compiler assumes that the above path can be resolved relative to the public +header directory. Instead of forcing package authors to structure their +packages around that constraint, the Swift compiler's interop header generation +logic will be amended to do the following in such cases where the target +does not have the public headers directory structure of an xcframework: + +- If an umbrella header that is modularized by the Clang module exists, the + interop header emit a reference directly to that umbrella header instead. +- Else, the interop header will import all textual includes from the Clang + module map. + +See the related discussion [thread][swift-compiler-thread-fr] from the initial +formal review. + +### Mixed language Test Targets + +To complement library targets with mixed languages, mixed test targets are +also supported as part of this proposal. + +Using the [example package][mixed-package] from before, consider the following +layout of the package's `Tests` directory. + +``` +MixedPackage +├── ... +└── Tests + └── MixedPackageTests + ├── JediTests.swift ]-- Swift tests + ├── SithTests.m        ]-- Objective-C tests + ├── ObjcTestConstants.h ⎤-- Mixed language test utils + ├── ObjcTestConstants.m ⎟ + └── TestConstants.swift ⎦ +``` + +The types defined in `ObjcTestConstants.h` are visible in `SithTests.m` (via +importing the header). + +The Objective-C compatible types defined in `TestConstants.swift` are visible +in `JediTests.swift` (via importing the header). + +This design should give package authors flexibility in designing test suites +for their mixed targets. + + +### Failure cases + +There are several failure cases that may surface to end users: +- Attempting to build a mixed target using a tools version that does not + include this proposal’s implementation. + ``` + target at '\(path)' contains mixed language source files; feature not supported + ``` +- Attempting to build a mixed target that is neither a library target + or test target. + ``` + Target with mixed sources at '\(path)' is a \(type) target; targets + with mixed language sources are only supported for library and test + targets. + ``` +- Attempting to build a mixed target containing a custom module map + that contains a `$(MixedTargetName).Swift` submodule. + ``` + The target's module map may not contain a Swift submodule for the + module \(target name). + ``` + +## Security + +This has no impact on security, safety, or privacy. + +## Impact on existing packages + +This proposal will not affect the behavior of existing packages. In the +proposed solution, the code path to build a mixed language package is separate +from the existing code paths to build packages with Swift sources and C +Language sources, respectively. + +Additionally, this feature will be gated on a tools minor version update, so +mixed language targets building on older toolchains that do not support this +feature will continue to [throw an error][mixed-target-error]. + +## Future Directions + +- Enable package authors to expose non-public headers to their mixed + target's Swift implementation. +- Extend mixed language target support to currently unsupported types of + targets (e.g. executables). +- Extend this solution so that all targets are mixed language targets by + default. This could simplify the implementation as language-specific types + like `ClangTarget`, `SwiftTarget`, and `MixedTarget` could be consolidated + into a single type. This approach was avoided in the initial implementation + of this feature to reduce the risk of introducing a regression. + + + +[SE-0038]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0038-swiftpm-c-language-targets.md + +[mixed-package]: https://github.com/ncooke3/MixedPackage + +[`SwiftTarget`]: https://github.com/apple/swift-package-manager/blob/ce099264a187759c2f587393bd209d317a0352b4/Sources/PackageModel/Target.swift#L313 + +[`ClangTarget`]: https://github.com/apple/swift-package-manager/blob/ce099264a187759c2f587393bd209d317a0352b4/Sources/PackageModel/Target.swift#L470 + +[`SwiftTargetBuildDescription`]: https://github.com/apple/swift-package-manager/blob/main/Sources/Build/BuildPlan.swift#L549 + +[`ClangTargetBuildDescription`]: https://github.com/apple/swift-package-manager/blob/ce099264a187759c2f587393bd209d317a0352b4/Sources/Build/BuildPlan.swift#L232 + +[path]: https://developer.apple.com/documentation/packagedescription/target/path + +[LLBuildManifestBuilder.swift]: https://github.com/apple/swift-package-manager/blob/14d05ccaa13b768449cd405fff81d630a520e04a/Sources/Build/LLBuildManifestBuilder.swift + +[mixed-target-error]: https://github.com/apple/swift-package-manager/blob/ce099264a187759c2f587393bd209d317a0352b4/Sources/PackageLoading/TargetSourcesBuilder.swift#L183-L189 + +[`SwiftSetting.InteroperabilityMode`]: https://developer.apple.com/documentation/packagedescription/swiftsetting/interoperabilitymode + +[swift-compiler-thread-fr]: https://forums.swift.org/t/se-0403-package-manager-mixed-language-target-support/66202/32 + +[should-emit-header]: https://github.com/apple/swift-package-manager/blob/6478e2724b8bf77856ff358cba5f59a4a62978bf/Sources/Build/BuildDescription/SwiftTargetBuildDescription.swift#L732-L735 + +[swift-emit-header-fr]: https://forums.swift.org/t/se-0403-package-manager-mixed-language-target-support/66202/31 + +[Swift-Clang-SourceModuleTarget]: https://github.com/apple/swift-package-manager/blob/8e512308530f808e9ef0cd149f4f632339c65bc4/Sources/PackagePlugin/PackageModel.swift#L231-L319 diff --git a/proposals/0404-nested-protocols.md b/proposals/0404-nested-protocols.md new file mode 100644 index 0000000000..9722c8b280 --- /dev/null +++ b/proposals/0404-nested-protocols.md @@ -0,0 +1,168 @@ +# Allow Protocols to be Nested in Non-Generic Contexts + +* Proposal: [SE-0404](0404-nested-protocols.md) +* Authors: [Karl Wagner](https://github.com/karwa) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.10)** +* Implementation: [apple/swift#66247](https://github.com/apple/swift/pull/66247) (gated behind flag `-enable-experimental-feature NestedProtocols`) +* Review: ([pitch](https://forums.swift.org/t/pitch-allow-protocols-to-be-nested-in-non-generic-contexts/65285)), ([review](https://forums.swift.org/t/se-0404-allow-protocols-to-be-nested-in-non-generic-contexts/66332)), ([acceptance](https://forums.swift.org/t/accepted-se-0404-allow-protocols-to-be-nested-in-non-generic-contexts/66668)) + +## Introduction + +Allows protocols to be nested in non-generic `struct/class/enum/actor`s, and functions. + +## Motivation + +Nesting nominal types inside other nominal types allows developers to express a natural scope for the inner type -- for example, `String.UTF8View` is `struct UTF8View` nested within `struct String`, and its name clearly communicates its purpose as an interface to the UTF-8 code-units of a String value. + +However, nesting is currently restricted to struct/class/enum/actors within other struct/class/enum/actors; protocols cannot be nested at all, and so must always be top-level types within a module. This is unfortunate, and we should relax this restriction so that developers can express protocols which are naturally scoped to some outer type. + +## Proposed solution + +We would allow nesting protocols within non-generic struct/class/enum/actors, and also within functions that do not belong to a generic context. + +For example, `TableView.Delegate` is naturally a delegate protocol pertaining to table-views. Developers should be declare it as such - nested within their `TableView` class: + +```swift +class TableView { + protocol Delegate: AnyObject { + func tableView(_: TableView, didSelectRowAtIndex: Int) + } +} + +class DelegateConformer: TableView.Delegate { + func tableView(_: TableView, didSelectRowAtIndex: Int) { + // ... + } +} +``` + +Currently, developers resort to giving things compound names, such as `TableViewDelegate`, to express the same natural scoping that could otherwise be expressed via nesting. + +As an additional benefit, within the context of `TableView`, the nested protocol `Delegate` can be referred to by a shorter name (as is the case with all other nested types): + +```swift +class TableView { + weak var delegate: Delegate? + + protocol Delegate { /* ... */ } +} +``` + +Protocols can also be nested within non-generic functions and closures. Admittedly, this is of somewhat limited utility, as all conformances to such protocols must also be within the same function. However, there is also no reason to artificially limit the complexity of the models which developers create within a function. Some codebases (of note, the Swift compiler itself) make use of large closures with nested types, and they benefit from abstractions using protocols. + +```swift +func doSomething() { + + protocol Abstraction { + associatedtype ResultType + func requirement() -> ResultType + } + struct SomeConformance: Abstraction { + func requirement() -> Int { ... } + } + struct AnotherConformance: Abstraction { + func requirement() -> String { ... } + } + + func impl(_ input: T) -> T.ResultType { + // ... + } + + let _: Int = impl(SomeConformance()) + let _: String = impl(AnotherConformance()) +} +``` + +## Detailed design + +Protocols may be nested anywhere that a struct/class/enum/actor may be nested, with the exception of generic contexts. For example, the following remains forbidden: + +```swift +class TableView { + + protocol Delegate { // Error: protocol 'Delegate' cannot be nested within a generic context. + func didSelect(_: Element) + } +} +``` + +The same applies to generic functions: + +```swift +func genericFunc(_: T) { + protocol Abstraction { // Error: protocol 'Abstraction' cannot be nested within a generic context. + } +} +``` + +And to other functions within generic contexts: + +```swift +class TableView { + func doSomething() { + protocol MyProtocol { // Error: protocol 'Abstraction' cannot be nested within a generic context. + } + } +} +``` + +Supporting this would require either: + +- Introducing generic protocols, or +- Mapping generic parameters to associated types. + +Neither is in in-scope for this proposal, but this author feels there is enough benefit here even without supporting generic contexts. Either of these would certainly make for interesting future directions. + +### Associated Type matching + +When nested in a concrete type, protocols do not witness associated type requirements. + +```swift +protocol Widget { + associatedtype Delegate +} + +struct TableWidget: Widget { + // Does NOT witness Widget.Delegate + protocol Delegate { ... } +} +``` + +Associated types associate one concrete type with one conforming type. Protocols are constraint types which many concrete types may conform to, so there is no obvious meaning to having a protocol witness an associated type requirement. + +There have been discussions in the past about whether protocols could gain an "associated protocol" feature, which would allow these networks of constraint types to be expressed. If such a feature were ever introduced, it may be reasonable for associated protocol requirements to be witnessed by nested protocols, the same way associated type requirements can be witnessed by nested concrete types today. + +## Source compatibility + +This feature is additive. + +## ABI compatibility + +This proposal is purely an extension of the language's ABI and does not change any existing features. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints. + +In general, moving a protocol in/out of a parent context is a source-breaking change. However, this breakage can mitigated by providing a `typealias` to the new name. + +As with other nested types, the parent context forms part of the mangled name of a nested protocol. Therefore, moving a protocol in/out of a parent context is an ABI-incompatible change. + +## Future directions + +- Allow nesting other (non-protocol) types in protocols + + Protocols themselves sometimes wish to define types which are naturally scoped to that protocol. For example, the standard library's [`FloatingPointRoundingRule`](https://developer.apple.com/documentation/swift/FloatingPointRoundingRule) enum is used by a [requirement of the `FloatingPoint` protocol](https://developer.apple.com/documentation/swift/floatingpoint/round(_:)) and defined for that purpose. + +- Allow nesting protocols in generic types + + As mentioned in the Detailed Design section, there are potentially strategies that would allow nesting protocols within generic types, and one could certainly imagine ways to use that expressive capability. The community is invited to discuss potential approaches in a separate topic. + +## Alternatives considered + +None. This is a straightforward extension of the language's existing nesting functionality. + +## Acknowledgments + +Thanks to [`@jumhyn`](https://forums.swift.org/u/jumhyn/) for reminding me about this, and to [`@suyashsrijan`](https://forums.swift.org/u/suyashsrijan/). diff --git a/proposals/0405-string-validating-initializers.md b/proposals/0405-string-validating-initializers.md new file mode 100644 index 0000000000..636b4092cc --- /dev/null +++ b/proposals/0405-string-validating-initializers.md @@ -0,0 +1,223 @@ +# String Initializers with Encoding Validation + +* Proposal: [SE-0405](0405-string-validating-initializers.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 6.0)** +* Bugs: rdar://99276048, rdar://99832858 +* Implementation: [Swift PR 68419](https://github.com/apple/swift/pull/68419), [Swift PR 68423](https://github.com/apple/swift/pull/68423) +* Review: ([pitch](https://forums.swift.org/t/66206)), ([review](https://forums.swift.org/t/se-0405-string-initializers-with-encoding-validation/66655)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0405-string-initializers-with-encoding-validation/67134)) +* Previous Revisions: [0](https://gist.github.com/glessard/d1ed79b7968b4ad2115462b3d1eba805), [1](https://github.com/swiftlang/swift-evolution/blob/37531427931a57ff2a76225741c99de8fa8b8c59/proposals/0405-string-validating-initializers.md) + +## Introduction + +We propose adding new `String` failable initializers that validate encoded input, and return `nil` when the input contains any invalid elements. + +## Motivation + +The `String` type guarantees that it represents well-formed Unicode text. When data representing text is received from a file, the network, or some other source, it may be relevant to store it in a `String`, but that data must be validated first. `String` already provides a way to transform data to valid Unicode by repairing invalid elements, but such a transformation is often not desirable, especially when dealing with untrusted sources. For example a JSON decoder cannot transform its input; it must fail if a span representing text contains any invalid UTF-8. + +This functionality has not been available directly from the standard library. It is possible to compose it using existing public API, but only at the cost of extra memory copies and allocations. The standard library is uniquely positioned to implement this functionality in a performant way. + +## Proposed Solution + +We will add a new `String` initializer that can fail, returning `nil`, when its input is found to be invalid according the encoding represented by a type parameter that conforms to `Unicode.Encoding`. + +```swift +extension String { + public init?( + validating codeUnits: some Sequence, + as: Encoding.Type + ) +} +``` + +When processing data obtained from C, it is frequently the case that UTF-8 data is represented by `Int8` (typically as `CChar`) rather than `UInt8`. We will provide a convenience initializer for this use case: + +```swift +extension String { + public init?( + validating codeUnits: some Sequence, + as: Encoding.Type + ) where Encoding.CodeUnit == UInt8 +} +``` + +`String` already features a validating initializer for UTF-8 input, intended for C interoperability. Its argument label does not convey the expectation that its input is a null-terminated C string, and this has caused errors. We propose to change the labels in order to clarify the preconditions: + +```swift +extension String { + public init?(validatingCString nullTerminatedUTF8: UnsafePointer) + + @available(Swift 5.XLIX, deprecated, renamed: "String.init(validatingCString:)") + public init?(validatingUTF8 cString: UnsafePointer) +} +``` + +Note that unlike `String.init?(validatingCString:)`, the `String.init?(validating:as:)` initializers convert their whole input, including any embedded `\0` code units. + +## Detailed Design + +We want these new initializers to be performant. As such, their implementation should minimize the number of memory allocations and copies required. We achieve this performance with `@inlinable` implementations that leverage `withContiguousStorageIfAvailable` to provide a concrete (`internal`) code path for the validation cases. The concrete `internal` initializer itself calls a number of functions internal to the standard library. + +```swift +extension String { + /// Creates a new `String` by copying and validating the sequence of + /// code units passed in, according to the specified encoding. + /// + /// This initializer does not try to repair ill-formed code unit sequences. + /// If any are found, the result of the initializer is `nil`. + /// + /// The following example calls this initializer with the contents of two + /// different arrays---first with a well-formed UTF-8 code unit sequence and + /// then with an ill-formed UTF-16 code unit sequence. + /// + /// let validUTF8: [UInt8] = [67, 97, 0, 102, 195, 169] + /// let valid = String(validating: validUTF8, as: UTF8.self) + /// print(valid) + /// // Prints "Optional("Café")" + /// + /// let invalidUTF16: [UInt16] = [0x41, 0x42, 0xd801] + /// let invalid = String(validating: invalidUTF16, as: UTF16.self) + /// print(invalid) + /// // Prints "nil" + /// + /// - Parameters: + /// - codeUnits: A sequence of code units that encode a `String` + /// - encoding: A conformer to `Unicode.Encoding` to be used + /// to decode `codeUnits`. + @inlinable + public init?( + validating codeUnits: some Sequence, + as encoding: Encoding.Type + ) where Encoding: Unicode.Encoding + + /// Creates a new `String` by copying and validating the sequence of + /// `Int8` passed in, according to the specified encoding. + /// + /// This initializer does not try to repair ill-formed code unit sequences. + /// If any are found, the result of the initializer is `nil`. + /// + /// The following example calls this initializer with the contents of two + /// different arrays---first with a well-formed UTF-8 code unit sequence and + /// then with an ill-formed ASCII code unit sequence. + /// + /// let validUTF8: [Int8] = [67, 97, 0, 102, -61, -87] + /// let valid = String(validating: validUTF8, as: UTF8.self) + /// print(valid) + /// // Prints "Optional("Café")" + /// + /// let invalidASCII: [Int8] = [67, 97, -5] + /// let invalid = String(validating: invalidASCII, as: Unicode.ASCII.self) + /// print(invalid) + /// // Prints "nil" + /// + /// - Parameters: + /// - codeUnits: A sequence of code units that encode a `String` + /// - encoding: A conformer to `Unicode.Encoding` that can decode + /// `codeUnits` as `UInt8` + @inlinable + public init?( + validating codeUnits: some Sequence, + as encoding: Encoding.Type + ) where Encoding: Unicode.Encoding, Encoding.CodeUnit == UInt8 +} +``` + +```swift +extension String { + /// Creates a new string by copying and validating the null-terminated UTF-8 + /// data referenced by the given pointer. + /// + /// This initializer does not try to repair ill-formed UTF-8 code unit + /// sequences. If any are found, the result of the initializer is `nil`. + /// + /// The following example calls this initializer with pointers to the + /// contents of two different `CChar` arrays---first with well-formed + /// UTF-8 code unit sequences and the second with an ill-formed sequence at + /// the end. + /// + /// let validUTF8: [CChar] = [67, 97, 102, -61, -87, 0] + /// validUTF8.withUnsafeBufferPointer { ptr in + /// let s = String(validatingCString: ptr.baseAddress!) + /// print(s) + /// } + /// // Prints "Optional("Café")" + /// + /// let invalidUTF8: [CChar] = [67, 97, 102, -61, 0] + /// invalidUTF8.withUnsafeBufferPointer { ptr in + /// let s = String(validatingCString: ptr.baseAddress!) + /// print(s) + /// } + /// // Prints "nil" + /// + /// - Parameter nullTerminatedUTF8: A pointer to a null-terminated UTF-8 code sequence. + @_silgen_name("sSS14validatingUTF8SSSgSPys4Int8VG_tcfC") + public init?(validatingCString nullTerminatedUTF8: UnsafePointer) + + @available(*, deprecated, renamed: "String.init(validatingCString:)") + @_silgen_name("_swift_stdlib_legacy_String_validatingUTF8") + @_alwaysEmitIntoClient + public init?(validatingUTF8 cString: UnsafePointer) +} +``` + +## Source Compatibility + +This proposal consists mostly of additions, which are by definition source compatible. + +The proposal includes the renaming of one function from `String.init?(validatingUTF8:)` to `String.init?(validatingCString:)`. The existing function name will be deprecated, producing a warning. A fixit will support an easy transition to the renamed version of the function. + +## ABI Compatibility + +This proposal adds new functions to the ABI. + +The renamed function reuses the existing ABI entry point, making the change ABI-compatible. + +## Implications on adoption + +This feature requires a new version of the standard library. + +## Alternatives considered + +#### Initializers specifying the encoding by their argument label + +For convenience and discoverability for the most common case, we originally proposed an initializer that specifies the UTF-8 input encoding as part of its argument label: + +```swift +extension String { + public init?(validatingAsUTF8 codeUnits: some Sequence) +} +``` + +Reviewers and the Language Steering Group believed that this initializer does not carry its weight, and that the discoverability issues it sought to alleviate would best be solved by improved tooling. + +#### Have `String.init?(validating: some Sequence)` take a parameter typed as `some Sequence`, or as a specific `Collection` of `CChar` + +Defining this validating initializer in terms of `some Sequence` would produce a compile-time ambiguity on platforms where `CChar` is typealiased to `UInt8` rather than `Int8`. The reviewed proposal suggested defining it in terms of `UnsafeBufferPointer`, since this parameter type would avoid such a compile-time ambiguity. The actual root of the problem is that `CChar` is a typealias instead of a separate type. Given this, discussions during the review period and by the Language Steering Group led to this initializer to be re-defined using `some Sequence`. This solves the `CChar`-vs-`UInt8` interoperability issue at source-code level, and preserves as much flexibility as possible without ambiguities. + +## Future directions + +#### Throw an error containing information about a validation failure + +When decoding a byte stream, obtaining the details of a validation failure would be useful in order to diagnose issues. We would like to provide this functionality, but the current input validation functionality is not well-suited for it. This is left as a future improvement. + +#### Improve input-repairing initialization + +There is only one initializer in the standard library for input-repairing initialization, and it suffers from a discoverability issue. We can add a more discoverable version specifically for the UTF-8 encoding, similarly to one of the additions proposed here. + +#### Add normalization options + +It is often desirable to normalize strings, but the standard library does not expose public API for doing so. We could add initializers that perform normalization, as well as mutating functions that perform normalization. + +#### Other + +- Add a (non-failable) initializer to create a `String` from `some Sequence`. +- Add API devoted to input validation specifically. + +## Acknowledgements + +Thanks to Michael Ilseman, Tina Liu and Quinn Quinn for discussions about input validation issues. + +[SE-0027](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0027-string-from-code-units.md) by [Zachary Waldowski](https://github.com/zwaldowski) was reviewed in February 2016, covering similar ground. It was rejected at the time because the design of `String` had not been finalized. The name `String.init(validatingCString:)` was suggested as part of SE-0027. Lily Ballard later [pitched](https://forums.swift.org/t/21538) a renaming of `String.init(validatingUTF8:)`, citing consistency with other `String` API involving C strings. + diff --git a/proposals/0406-async-stream-backpressure.md b/proposals/0406-async-stream-backpressure.md new file mode 100644 index 0000000000..58817b3ede --- /dev/null +++ b/proposals/0406-async-stream-backpressure.md @@ -0,0 +1,806 @@ +# Backpressure support for AsyncStream + +* Proposal: [SE-0406](0406-async-stream-backpressure.md) +* Author: [Franz Busch](https://github.com/FranzBusch) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Returned for revision** +* Implementation: [apple/swift#66488](https://github.com/apple/swift/pull/66488) +* Review: ([pitch](https://forums.swift.org/t/pitch-new-apis-for-async-throwing-stream-with-backpressure-support/65449)) ([review](https://forums.swift.org/t/se-0406-backpressure-support-for-asyncstream/66771)) ([return for revision](https://forums.swift.org/t/returned-for-revision-se-0406-backpressure-support-for-asyncstream/67248)) + +## Introduction + +[SE-0314](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0314-async-stream.md) +introduced new `Async[Throwing]Stream` types which act as root asynchronous +sequences. These two types allow bridging from synchronous callbacks such as +delegates to an asynchronous sequence. This proposal adds a new way of +constructing asynchronous streams with the goal to bridge backpressured systems +into an asynchronous sequence. Furthermore, this proposal aims to clarify the +cancellation behaviour both when the consuming task is cancelled and when +the production side indicates termination. + +## Motivation + +After using the `AsyncSequence` protocol and the `Async[Throwing]Stream` types +extensively over the past years, we learned that there are a few important +behavioral details that any `AsyncSequence` implementation needs to support. +These behaviors are: + +1. Backpressure +2. Multi/single consumer support +3. Downstream consumer termination +4. Upstream producer termination + +In general, `AsyncSequence` implementations can be divided into two kinds: Root +asynchronous sequences that are the source of values such as +`Async[Throwing]Stream` and transformational asynchronous sequences such as +`AsyncMapSequence`. Most transformational asynchronous sequences implicitly +fulfill the above behaviors since they forward any demand to a base asynchronous +sequence that should implement the behaviors. On the other hand, root +asynchronous sequences need to make sure that all of the above behaviors are +correctly implemented. Let's look at the current behavior of +`Async[Throwing]Stream` to see if and how it achieves these behaviors. + +### Backpressure + +Root asynchronous sequences need to relay the backpressure to the producing +system. `Async[Throwing]Stream` aims to support backpressure by providing a +configurable buffer and returning +`Async[Throwing]Stream.Continuation.YieldResult` which contains the current +buffer depth from the `yield()` method. However, only providing the current +buffer depth on `yield()` is not enough to bridge a backpressured system into +an asynchronous sequence since this can only be used as a "stop" signal but we +are missing a signal to indicate resuming the production. The only viable +backpressure strategy that can be implemented with the current API is a timed +backoff where we stop producing for some period of time and then speculatively +produce again. This is a very inefficient pattern that produces high latencies +and inefficient use of resources. + +### Multi/single consumer support + +The `AsyncSequence` protocol itself makes no assumptions about whether the +implementation supports multiple consumers or not. This allows the creation of +unicast and multicast asynchronous sequences. The difference between a unicast +and multicast asynchronous sequence is if they allow multiple iterators to be +created. `AsyncStream` does support the creation of multiple iterators and it +does handle multiple consumers correctly. On the other hand, +`AsyncThrowingStream` also supports multiple iterators but does `fatalError` +when more than one iterator has to suspend. The original proposal states: + +> As with any sequence, iterating over an AsyncStream multiple times, or +creating multiple iterators and iterating over them separately, may produce an +unexpected series of values. + +While that statement leaves room for any behavior we learned that a clear distinction +of behavior for root asynchronous sequences is beneficial; especially, when it comes to +how transformation algorithms are applied on top. + +### Downstream consumer termination + +Downstream consumer termination allows the producer to notify the consumer that +no more values are going to be produced. `Async[Throwing]Stream` does support +this by calling the `finish()` or `finish(throwing:)` methods of the +`Async[Throwing]Stream.Continuation`. However, `Async[Throwing]Stream` does not +handle the case that the `Continuation` may be `deinit`ed before one of the +finish methods is called. This currently leads to async streams that never +terminate. The behavior could be changed but it could result in semantically +breaking code. + +### Upstream producer termination + +Upstream producer termination is the inverse of downstream consumer termination +where the producer is notified once the consumption has terminated. Currently, +`Async[Throwing]Stream` does expose the `onTermination` property on the +`Continuation`. The `onTermination` closure is invoked once the consumer has +terminated. The consumer can terminate in four separate cases: + +1. The asynchronous sequence was `deinit`ed and no iterator was created +2. The iterator was `deinit`ed and the asynchronous sequence is unicast +3. The consuming task is canceled +4. The asynchronous sequence returned `nil` or threw + +`Async[Throwing]Stream` currently invokes `onTermination` in all cases; however, +since `Async[Throwing]Stream` supports multiple consumers (as discussed in the +`Multi/single consumer support` section), a single consumer task being canceled +leads to the termination of all consumers. This is not expected from multicast +asynchronous sequences in general. + +## Proposed solution + +The above motivation lays out the expected behaviors from a root asynchronous +sequence and compares them to the behaviors of `Async[Throwing]Stream`. These +are the behaviors where `Async[Throwing]Stream` diverges from the expectations. + +- Backpressure: Doesn't expose a "resumption" signal to the producer +- Multi/single consumer: + - Divergent implementation between throwing and non-throwing variant + - Supports multiple consumers even though proposal positions it as a unicast + asynchronous sequence +- Consumer termination: Doesn't handle the `Continuation` being `deinit`ed +- Producer termination: Happens on first consumer termination + +This section proposes new APIs for `Async[Throwing]Stream` that implement all of +the above-mentioned behaviors. + +### Creating an AsyncStream with backpressure support + +You can create an `Async[Throwing]Stream` instance using the new `makeStream(of: +backpressureStrategy:)` method. This method returns you the stream and the +source. The source can be used to write new values to the asynchronous stream. +The new API specifically provides a multi-producer/single-consumer pattern. + +```swift +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) +``` + +The new proposed APIs offer three different ways to bridge a backpressured +system. The foundation is the multi-step synchronous interface. Below is an +example of how it can be used: + +```swift +do { + let writeResult = try source.write(contentsOf: sequence) + + switch writeResult { + case .produceMore: + // Trigger more production + + case .enqueueCallback(let callbackToken): + source.enqueueCallback(token: callbackToken, onProduceMore: { result in + switch result { + case .success: + // Trigger more production + case .failure(let error): + // Terminate the underlying producer + } + }) + } +} catch { + // `write(contentsOf:)` throws if the asynchronous stream already terminated +} +``` + +The above API offers the most control and highest performance when bridging a +synchronous producer to an asynchronous sequence. First, you have to write +values using the `write(contentsOf:)` which returns a `WriteResult`. The result +either indicates that more values should be produced or that a callback should +be enqueued by calling the `enqueueCallback(callbackToken: onProduceMore:)` +method. This callback is invoked once the backpressure strategy decided that +more values should be produced. This API aims to offer the most flexibility with +the greatest performance. The callback only has to be allocated in the case +where the producer needs to be suspended. + +Additionally, the above API is the building block for some higher-level and +easier-to-use APIs to write values to the asynchronous stream. Below is an +example of the two higher-level APIs. + +```swift +// Writing new values and providing a callback when to produce more +try source.write(contentsOf: sequence, onProduceMore: { result in + switch result { + case .success: + // Trigger more production + case .failure(let error): + // Terminate the underlying producer + } +}) + +// This method suspends until more values should be produced +try await source.write(contentsOf: sequence) +``` + +With the above APIs, we should be able to effectively bridge any system into an +asynchronous stream regardless if the system is callback-based, blocking or +asynchronous. + +### Downstream consumer termination + +> When reading the next two examples around termination behaviour keep in mind +that the newly proposed APIs are providing a strict unicast asynchronous sequence. + +Calling `finish()` terminates the downstream consumer. Below is an example of +this: + +```swift +// Termination through calling finish +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) + +_ = try await source.write(1) +source.finish() + +for await element in stream { + print(element) +} +print("Finished") + +// Prints +// 1 +// Finished +``` + +The other way to terminate the consumer is by deiniting the source. This has the +same effect as calling `finish()` and makes sure that no consumer is stuck +indefinitely. + +```swift +// Termination through deiniting the source +let (stream, _) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) + +for await element in stream { + print(element) +} +print("Finished") + +// Prints +// Finished +``` + +Trying to write more elements after the source has been finish will result in an +error thrown from the write methods. + +### Upstream producer termination + +The producer will get notified about termination through the `onTerminate` +callback. Termination of the producer happens in the following scenarios: + +```swift +// Termination through task cancellation +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) + +let task = Task { + for await element in stream { + + } +} +task.cancel() +``` + +```swift +// Termination through deiniting the sequence +let (_, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) +``` + +```swift +// Termination through deiniting the iterator +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) +_ = stream.makeAsyncIterator() +``` + +```swift +// Termination through calling finish +let (stream, source) = AsyncStream.makeStream( + of: Int.self, + backpressureStrategy: .watermark(low: 2, high: 4) +) + +_ = try source.write(1) +source.finish() + +for await element in stream {} + +// onTerminate will be called after all elements have been consumed +``` + +Similar to the downstream consumer termination, trying to write more elements after the +producer has been terminated will result in an error thrown from the write methods. + +## Detailed design + +All new APIs on `AsyncStream` and `AsyncThrowingStream` are as follows: + +```swift +/// Error that is thrown from the various `write` methods of the +/// ``AsyncStream.Source`` and ``AsyncThrowingStream.Source``. +/// +/// This error is thrown when the asynchronous stream is already finished when +/// trying to write new elements. +public struct AsyncStreamAlreadyFinishedError: Error {} + +extension AsyncStream { + /// A mechanism to interface between producer code and an asynchronous stream. + /// + /// Use this source to provide elements to the stream by calling one of the `write` methods, then terminate the stream normally + /// by calling the `finish()` method. + public struct Source: Sendable { + /// A strategy that handles the backpressure of the asynchronous stream. + public struct BackpressureStrategy: Sendable { + /// When the high watermark is reached producers will be suspended. All producers will be resumed again once + /// the low watermark is reached. + public static func watermark(low: Int, high: Int) -> BackpressureStrategy {} + } + + /// A type that indicates the result of writing elements to the source. + @frozen + public enum WriteResult: Sendable { + /// A token that is returned when the asynchronous stream's backpressure strategy indicated that production should + /// be suspended. Use this token to enqueue a callback by calling the ``enqueueCallback(_:)`` method. + public struct CallbackToken: Sendable {} + + /// Indicates that more elements should be produced and written to the source. + case produceMore + + /// Indicates that a callback should be enqueued. + /// + /// The associated token should be passed to the ``enqueueCallback(_:)`` method. + case enqueueCallback(CallbackToken) + } + + /// A callback to invoke when the stream finished. + /// + /// The stream finishes and calls this closure in the following cases: + /// - No iterator was created and the sequence was deinited + /// - An iterator was created and deinited + /// - After ``finish(throwing:)`` was called and all elements have been consumed + /// - The consuming task got cancelled + public var onTermination: (@Sendable () -> Void)? + + /// Writes new elements to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// - Parameter sequence: The elements to write to the asynchronous stream. + /// - Returns: The result that indicates if more elements should be produced at this time. + public func write(contentsOf sequence: S) throws -> WriteResult where Element == S.Element, S: Sequence {} + + /// Write the element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// - Parameter element: The element to write to the asynchronous stream. + /// - Returns: The result that indicates if more elements should be produced at this time. + public func write(_ element: Element) throws -> WriteResult {} + + /// Enqueues a callback that will be invoked once more elements should be produced. + /// + /// Call this method after ``write(contentsOf:)`` or ``write(_:)`` returned ``WriteResult/enqueueCallback(_:)``. + /// + /// - Important: Enqueueing the same token multiple times is not allowed. + /// + /// - Parameters: + /// - token: The callback token. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. + public func enqueueCallback(token: WriteResult.CallbackToken, onProduceMore: @escaping @Sendable (Result) -> Void) {} + + /// Cancel an enqueued callback. + /// + /// Call this method to cancel a callback enqueued by the ``enqueueCallback(callbackToken:onProduceMore:)`` method. + /// + /// - Note: This method supports being called before ``enqueueCallback(callbackToken:onProduceMore:)`` is called and + /// will mark the passed `token` as cancelled. + /// + /// - Parameter token: The callback token. + public func cancelCallback(token: WriteResult.CallbackToken) {} + + /// Write new elements to the asynchronous stream and provide a callback which will be invoked once more elements should be produced. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then `onProduceMore` will be invoked with + /// a `Result.failure`. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. This callback might be + /// invoked during the call to ``write(contentsOf:onProduceMore:)``. + public func write(contentsOf sequence: S, onProduceMore: @escaping @Sendable (Result) -> Void) where Element == S.Element, S: Sequence {} + + /// Writes the element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then `onProduceMore` will be invoked with + /// a `Result.failure`. + /// + /// - Parameters: + /// - sequence: The element to write to the asynchronous stream. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. This callback might be + /// invoked during the call to ``write(_:onProduceMore:)``. + public func write(_ element: Element, onProduceMore: @escaping @Sendable (Result) -> Void) {} + + /// Write new elements to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// This method returns once more elements should be produced. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + public func write(contentsOf sequence: S) async throws where Element == S.Element, S: Sequence {} + + /// Write new element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// This method returns once more elements should be produced. + /// + /// - Parameters: + /// - sequence: The element to write to the asynchronous stream. + public func write(_ element: Element) async throws {} + + /// Write the elements of the asynchronous sequence to the asynchronous stream. + /// + /// This method returns once the provided asynchronous sequence or the asynchronous stream finished. + /// + /// - Important: This method does not finish the source if consuming the upstream sequence terminated. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + public func write(contentsOf sequence: S) async throws where Element == S.Element, S: AsyncSequence {} + + /// Indicates that the production terminated. + /// + /// After all buffered elements are consumed the next iteration point will return `nil`. + /// + /// Calling this function more than once has no effect. After calling finish, the stream enters a terminal state and doesn't accept + /// new elements. + public func finish() {} + } + + /// Initializes a new ``AsyncStream`` and an ``AsyncStream/Source``. + /// + /// - Parameters: + /// - elementType: The element type of the stream. + /// - backpressureStrategy: The backpressure strategy that the stream should use. + /// - Returns: A tuple containing the stream and its source. The source should be passed to the + /// producer while the stream should be passed to the consumer. + public static func makeStream( + of elementType: Element.Type = Element.self, + backpressureStrategy: Source.BackpressureStrategy + ) -> (`Self`, Source) {} +} + +extension AsyncThrowingStream { + /// A mechanism to interface between producer code and an asynchronous stream. + /// + /// Use this source to provide elements to the stream by calling one of the `write` methods, then terminate the stream normally + /// by calling the `finish()` method. You can also use the source's `finish(throwing:)` method to terminate the stream by + /// throwing an error. + public struct Source: Sendable { + /// A strategy that handles the backpressure of the asynchronous stream. + public struct BackpressureStrategy: Sendable { + /// When the high watermark is reached, producers will be suspended. All producers will be resumed again once + /// the low watermark is reached. + public static func watermark(low: Int, high: Int) -> BackpressureStrategy {} + } + + /// A type that indicates the result of writing elements to the source. + @frozen + public enum WriteResult: Sendable { + /// A token that is returned when the asynchronous stream's backpressure strategy indicated that production should + /// be suspended. Use this token to enqueue a callback by calling the ``enqueueCallback(_:)`` method. + public struct CallbackToken: Sendable {} + + /// Indicates that more elements should be produced and written to the source. + case produceMore + + /// Indicates that a callback should be enqueued. + /// + /// The associated token should be passed to the ``enqueueCallback(_:)`` method. + case enqueueCallback(CallbackToken) + } + + /// A callback to invoke when the stream finished. + /// + /// The stream finishes and calls this closure in the following cases: + /// - No iterator was created and the sequence was deinited + /// - An iterator was created and deinited + /// - After ``finish(throwing:)`` was called and all elements have been consumed + /// - The consuming task got cancelled + public var onTermination: (@Sendable () -> Void)? {} + + /// Writes new elements to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// - Parameter sequence: The elements to write to the asynchronous stream. + /// - Returns: The result that indicates if more elements should be produced at this time. + public func write(contentsOf sequence: S) throws -> WriteResult where Element == S.Element, S: Sequence {} + + /// Write the element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// - Parameter element: The element to write to the asynchronous stream. + /// - Returns: The result that indicates if more elements should be produced at this time. + public func write(_ element: Element) throws -> WriteResult {} + + /// Enqueues a callback that will be invoked once more elements should be produced. + /// + /// Call this method after ``write(contentsOf:)`` or ``write(_:)`` returned ``WriteResult/enqueueCallback(_:)``. + /// + /// - Important: Enqueueing the same token multiple times is not allowed. + /// + /// - Parameters: + /// - token: The callback token. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. + public func enqueueCallback(token: WriteResult.CallbackToken, onProduceMore: @escaping @Sendable (Result) -> Void) {} + + /// Cancel an enqueued callback. + /// + /// Call this method to cancel a callback enqueued by the ``enqueueCallback(callbackToken:onProduceMore:)`` method. + /// + /// - Note: This method supports being called before ``enqueueCallback(callbackToken:onProduceMore:)`` is called and + /// will mark the passed `token` as cancelled. + /// + /// - Parameter token: The callback token. + public func cancelCallback(token: WriteResult.CallbackToken) {} + + /// Write new elements to the asynchronous stream and provide a callback which will be invoked once more elements should be produced. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then `onProduceMore` will be invoked with + /// a `Result.failure`. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. This callback might be + /// invoked during the call to ``write(contentsOf:onProduceMore:)``. + public func write(contentsOf sequence: S, onProduceMore: @escaping @Sendable (Result) -> Void) where Element == S.Element, S: Sequence {} + + /// Writes the element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then `onProduceMore` will be invoked with + /// a `Result.failure`. + /// + /// - Parameters: + /// - sequence: The element to write to the asynchronous stream. + /// - onProduceMore: The callback which gets invoked once more elements should be produced. This callback might be + /// invoked during the call to ``write(_:onProduceMore:)``. + public func write(_ element: Element, onProduceMore: @escaping @Sendable (Result) -> Void) {} + + /// Write new elements to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// first element of the provided sequence. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// This method returns once more elements should be produced. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + public func write(contentsOf sequence: S) async throws where Element == S.Element, S: Sequence {} + + /// Write new element to the asynchronous stream. + /// + /// If there is a task consuming the stream and awaiting the next element then the task will get resumed with the + /// provided element. If the asynchronous stream already terminated then this method will throw an error + /// indicating the failure. + /// + /// This method returns once more elements should be produced. + /// + /// - Parameters: + /// - sequence: The element to write to the asynchronous stream. + public func write(_ element: Element) async throws {} + + /// Write the elements of the asynchronous sequence to the asynchronous stream. + /// + /// This method returns once the provided asynchronous sequence or the the asynchronous stream finished. + /// + /// - Important: This method does not finish the source if consuming the upstream sequence terminated. + /// + /// - Parameters: + /// - sequence: The elements to write to the asynchronous stream. + public func write(contentsOf sequence: S) async throws where Element == S.Element, S: AsyncSequence {} + + /// Indicates that the production terminated. + /// + /// After all buffered elements are consumed the next iteration point will return `nil` or throw an error. + /// + /// Calling this function more than once has no effect. After calling finish, the stream enters a terminal state and doesn't accept + /// new elements. + /// + /// - Parameters: + /// - error: The error to throw, or `nil`, to finish normally. + public func finish(throwing error: Failure?) {} + } + + /// Initializes a new ``AsyncThrowingStream`` and an ``AsyncThrowingStream/Source``. + /// + /// - Parameters: + /// - elementType: The element type of the stream. + /// - failureType: The failure type of the stream. + /// - backpressureStrategy: The backpressure strategy that the stream should use. + /// - Returns: A tuple containing the stream and its source. The source should be passed to the + /// producer while the stream should be passed to the consumer. + public static func makeStream( + of elementType: Element.Type = Element.self, + throwing failureType: Failure.Type = Failure.self, + backpressureStrategy: Source.BackpressureStrategy + ) -> (`Self`, Source) where Failure == Error {} +} +``` + +## Comparison to other root asynchronous sequences + +### swift-async-algorithm: AsyncChannel + +The `AsyncChannel` is a multi-consumer/multi-producer root asynchronous sequence +which can be used to communicate between two tasks. It only offers asynchronous +production APIs and has no internal buffer. This means that any producer will be +suspended until its value has been consumed. `AsyncChannel` can handle multiple +consumers and resumes them in FIFO order. + +### swift-nio: NIOAsyncSequenceProducer + +The NIO team have created their own root asynchronous sequence with the goal to +provide a high performance sequence that can be used to bridge a NIO `Channel` +inbound stream into Concurrency. The `NIOAsyncSequenceProducer` is a highly +generic and fully inlinable type and quite unwieldy to use. This proposal is +heavily inspired by the learnings from this type but tries to create a more +flexible and easier to use API that fits into the standard library. + +## Source compatibility + +This change is additive and does not affect source compatibility. + +## ABI compatibility + +This change is additive and does not affect ABI compatibility. All new methods +are non-inlineable leaving us flexibility to change the implementation in the +future. + +## Future directions + +### Adaptive backpressure strategy + +The high/low watermark strategy is common in networking code; however, there are +other strategies such as an adaptive strategy that we could offer in the future. +An adaptive strategy regulates the backpressure based on the rate of +consumption and production. With the proposed new APIs we can easily add further +strategies. + +### Element size dependent strategy + +When the stream's element is a collection type then the proposed high/low +watermark backpressure strategy might lead to unexpected results since each +element can vary in actual memory size. In the future, we could provide a new +backpressure strategy that supports inspecting the size of the collection. + +### Deprecate `Async[Throwing]Stream.Continuation` + +In the future, we could deprecate the current continuation based APIs since the +new proposed APIs are also capable of bridging non-backpressured producers by +just discarding the `WriteResult`. The only use-case that the new APIs do not +cover is the _anycast_ behaviour of the current `AsyncStream` where one can +create multiple iterators to the stream as long as no two iterators are +consuming the stream at the same time. This can be solved via additional +algorithms such as `broadcast` in the `swift-async-algorithms` package. + +To give developers more time to adopt the new APIs the deprecation of the +current APIs should be deferred to a future version. Especially since those new +APIs are not backdeployed like the current Concurrency runtime. + +### Introduce a `Writer` and an `AsyncWriter` protocol + +The newly introduced `Source` type offers a bunch of different write methods. We +have seen similar types used in other places such as file abstraction or +networking APIs. We could introduce a new `Writer` and `AsyncWriter` protocol in +the future to enable writing generic algorithms on top of writers. The `Source` +type could then conform to these new protocols. + +## Alternatives considered + +### Providing an `Async[Throwing]Stream.Continuation.onConsume` + +We could add a new closure property to the `Async[Throwing]Stream.Continuation` +which is invoked once an element has been consumed to implement a backpressure +strategy; however, this requires the usage of a synchronization mechanism since +the consumption and production often happen on separate threads. The +added complexity and performance impact led to avoiding this approach. + +### Provide a getter for the current buffer depth + +We could provide a getter for the current buffer depth on the +`Async[Throwing]Stream.Continuation`. This could be used to query the buffer +depth at an arbitrary time; however, it wouldn't allow us to implement +backpressure strategies such as high/low watermarks without continuously asking +what the buffer depth is. That would result in a very inefficient +implementation. + +### Extending `Async[Throwing]Stream.Continuation` + +Extending the current APIs to support all expected behaviors is problematic +since it would change the semantics and might lead to currently working code +misbehaving. Furthermore, extending the current APIs to support backpressure +turns out to be problematic without compromising performance or usability. + +### Introducing a new type + +We could introduce a new type such as `AsyncBackpressured[Throwing]Stream`; +however, one of the original intentions of `Async[Throwing]Stream` was to be +able to bridge backpressured systems. Furthermore, `Async[Throwing]Stream` is +the best name. Therefore, this proposal decided to provide new interfaces to +`Async[Throwing]Stream`. + +### Stick with the current `Continuation` and `yield` naming + +The proposal decided against sticking to the current names since the existing +names caused confusion to them being used in multiple places. Continuation was +both used by the `AsyncStream` but also by Swift Concurrency via +`CheckedContinuation` and `UnsafeContinuation`. Similarly, yield was used by +both `AsyncStream.Continuation.yield()`, `Task.yield()` and the `yield` keyword. +Having different names for these different concepts makes it easier to explain +their usage. The currently proposed `write` names were chosen to align with the +future direction of adding an `AsyncWriter` protocol. `Source` is a common name +in flow based systems such as Akka. Other names that were considered: + +- `enqueue` +- `send` + +### Provide the `onTermination` callback to the factory method + +During development of the new APIs, I first tried to provide the `onTermination` +callback in the `makeStream` method. However, that showed significant usability +problems in scenarios where one wants to store the source in a type and +reference `self` in the `onTermination` closure at the same time; hence, I kept +the current pattern of setting the `onTermination` closure on the source. + +### Provide a `onConsumerCancellation` callback + +During the pitch phase, it was raised that we should provide a +`onConsumerCancellation` callback which gets invoked once the asynchronous +stream notices that the consuming task got cancelled. This callback could be +used to customize how cancellation is handled by the stream e.g. one could +imagine writing a few more elements to the stream before finishing it. Right now +the stream immediately returns `nil` or throws a `CancellationError` when it +notices cancellation. This proposal decided to not provide this customization +because it opens up the possibility that asynchronous streams are not terminating +when implemented incorrectly. Additionally, asynchronous sequences are not the +only place where task cancellation leads to an immediate error being thrown i.e. +`Task.sleep()` does the same. Hence, the value of the asynchronous not +terminating immediately brings little value when the next call in the iterating +task might throw. However, the implementation is flexible enough to add this in +the future and we can just default it to the current behaviour. + +### Create a custom type for the `Result` of the `onProduceMore` callback + +The `onProduceMore` callback takes a `Result` which is used to +indicate if the producer should produce more or if the asynchronous stream +finished. We could introduce a new type for this but the proposal decided +against it since it effectively is a result type. + +### Use an initializer instead of factory methods + +Instead of providing a `makeStream` factory method we could use an initializer +approach that takes a closure which gets the `Source` passed into. A similar API +has been offered with the `Continuation` based approach and +[SE-0388](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0388-async-stream-factory.md) +introduced new factory methods to solve some of the usability ergonomics with +the initializer based APIs. + +## Acknowledgements + +- [Johannes Weiss](https://github.com/weissi) - For making me aware how +important this problem is and providing great ideas on how to shape the API. +- [Philippe Hausler](https://github.com/phausler) - For helping me designing the +APIs and continuously providing feedback +- [George Barnett](https://github.com/glbrntt) - For providing extensive code +reviews and testing the implementation. diff --git a/proposals/0407-member-macro-conformances.md b/proposals/0407-member-macro-conformances.md new file mode 100644 index 0000000000..6e31c39d1f --- /dev/null +++ b/proposals/0407-member-macro-conformances.md @@ -0,0 +1,116 @@ +# Member Macro Conformances + +* Proposal: [SE-0407](0407-member-macro-conformances.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 5.9.2)** +* Vision: [Macros](https://github.com/swiftlang/swift-evolution/blob/main/visions/macros.md) +* Implementation: [apple/swift#67758](https://github.com/apple/swift/pull/67758) +* Review: ([pitch](https://forums.swift.org/t/pitch-member-macros-that-know-what-conformances-are-missing/66590)) ([review](https://forums.swift.org/t/se-0407-member-macro-conformances/66951)) ([acceptance](https://forums.swift.org/t/accepted-se-0407-member-macro-conformances/67345)) + +## Introduction + +The move from conformance macros to extension macros in [SE-0402](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0402-extension-macros.md) included the ability for extension macros to learn about which protocols the type already conformed to (e.g., because a superclass conformed or an explicit conformance was stated somewhere), so that the macro could avoid adding declarations and conformances that aren't needed. It also meant that any new declarations added are part of an extension---not the original type definition---which is generally beneficial, because it means that (e.g.) a new initializer doesn't suppress the memberwise initializer. It's also usually considered good form to split protocol conformances out into their own extensions. + +However, there are some times when the member used for the conformance really needs to be part of the original type definition. For example: + +- An initializer in a non-final class needs to be a `required init` to satisfy a protocol requirement. +- An overridable member of a non-final class. +- A stored property or case can only be in the primary type definition. + +For these cases, a member macro can produce the declarations. However, member macros aren't provided with any information about which protocol conformances they should provide members for, so a macro might erroneously try to add conforming members to a type that already conforms to the protocol (e.g., through a superclass). This can make certain macros---such as macros that implement the `Encodable` or `Decodable` protocols---unimplemented. + +## Proposed solution + +To make it possible for a member macro to provide the right members for a conformance, we propose to extend member macros with the same ability that extension macros have to reason about conformances. Specifically: + +* The `attached` attribute specifying a `member` role gains the ability to specify the set of protocol conformances it is interested in, the same way an `extension` macro specifies the conformances it can provide. +* The `expansion` operation for a `MemberMacro` -conforming implementation receives the set of protocols that were stated (as above) and which the type does not already conform to. + +This information allows a macro to reason about which members it should produce to satisfy conformances. Member macros that are interested in conformances are often going to also be extension macros, which work along with the member macro to provide complete conformance information. + +As an example, consider a `Codable` macro that provides the `init(from:)` and `encode(to:)` operations required by the `Decodable` and `Encodable` protocols, respectively. Such a macro could be defined as follows: + +```swift +@attached(member, conformances: Decodable, Encodable, names: named(init(from:), encode(to:))) +@attached(extension, conformances: Decodable, Encodable, names: named(init(from:), encode(to:))) +macro Codable() = #externalMacro(module: "MyMacros", type: "CodableMacro") +``` + +This macro has several important decisions to make about where and how to generate `init(from:)` and `encode(to:)`: + +* For a struct, enum, actor, or final class, `init(from:)` and `encode(to:)` should be emitted into an extension (via the member role) along with the conformance. This is both good style and, for structs, ensures that the initializer doesn't inhibit the memberwise initializer. +* For a non-final class, `init(from:)` and `encode(to:)` should be emitted into the main class definition (via the member role) so that they can be overridden by subclasses. +* For a class that inherits `Encodable` or `Decodable` conformances from a superclass, the implementations of `init(from:)` and `encode(to:)` need to call the superclass's initializer and method, respectively, to decode/encode the entire class hierarchy. + +Given existing syntactic information about the type (including the presence or absence of `final`), and providing both the member and extension roles with information about which conformances the type needs (as proposed here), all of the above decisions can be made in the macro implementation, allowing a flexible implementation of a `Codable` macro that accounts for all manner of types. + +## Detailed design + +The specification of the `conformances` argument for the `@attached(member, ...)` attribute matches that of the corresponding argument for extension macros documented in [SE-0402](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0402-extension-macros.md). + +For macro implementations, the `expansion` requirement in the `MemberMacro` protocol is augmented with a `conformingTo:` argument that receives the same set of protocols as for extension macros. The `MemberMacro` protocol is now defined as follows: + +```swift +protocol MemberMacro: AttachedMacro { + /// Expand an attached declaration macro to produce a set of members. + /// + /// - Parameters: + /// - node: The custom attribute describing the attached macro. + /// - declaration: The declaration the macro attribute is attached to. + /// - missingConformancesTo: The set of protocols that were declared + /// in the set of conformances for the macro and to which the declaration + /// does not explicitly conform. The member macro itself cannot declare + /// conformances to these protocols (only an extension macro can do that), + /// but can provide supporting declarations, such as a required + /// initializer or stored property, that cannot be written in an + /// extension. + /// - context: The context in which to perform the macro expansion. + /// + /// - Returns: the set of member declarations introduced by this macro, which + /// are nested inside the `attachedTo` declaration. + static func expansion( + of node: AttributeSyntax, + providingMembersOf declaration: some DeclGroupSyntax, + conformingTo protocols: [TypeSyntax], + in context: some MacroExpansionContext + ) throws -> [DeclSyntax] +} +``` + +Note that member macro definitions don't provide the conformances themselves; that is still part of the extension macro role. + +## Source compatibility + +This proposal uses the existing syntactic space for the `@attached` attribute and is a pure extension; it does not have any source compatibility impact. + +## ABI compatibility + +As a macro feature, this proposal does not affect ABI in any way. + +## Implications on adoption + +This feature can be freely adopted in source code with no deployment constraints or affecting source or ABI compatibility. Uses of any macro that employs this feature can also be removed from the source code by expanding the macro in place. + +## Alternatives considered + +### Extensions that affect the primary type definition + +A completely different approach to the stated problem would be to introduce a form of extension that adds members to the type as-if they were written directly in the type definition. For example: + +```swift +class MyClass { ... } + +@implementation extension MyClass: Codable { + required init(from decoder: Decoder) throws { ... } + func encode(to coder: Coder) throws { ... } +} +``` + +The members of `@implementation` extensions would follow the same rules as members in the main type definition. For example, stored properties could be defined in the `@implementation` extension, as could `required` initializers, and any overridable methods, properties, or subscripts. The deinitializer and enum cases could also be defined in `@implementation` extensions if that were deemed useful. + +There would be some limitations on `@implementation` extensions: they could only be defined in the same source file as the original type, and these extensions might not be permitted to have any additional generic constraints. Protocols don't have implementations per se, and therefore might not support implementation extensions. + +Given the presence of `@implementation` extensions, the extension to member macros in this proposal would no longer be needed, because one could achieve the desired effect using an extension macro that produces an `@implementation` extension for cases where it needs to extend the implementation itself. + +The primary drawback to this notion of implementation extensions is that it would no longer be possible to look at the primary definition of a type to find its full "shape": its stored properties, designated/required initializers, overridable methods, and so on. Instead, that information could be scattered amongst the original type definition and any implementation extensions, requiring readers to stitch together a view of the whole type. This would have a particularly negative effect on macros that want to reason about the shape of the type, because macros only see a single entity (such as a class or struct definition) and not extensions to that entity. The `Codable` macro discussed in this proposal, for example, would not be able to encode or decode stored properties that are written in an implementation extension, and the [`Observable`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0395-observability.md) macro would silently fail to observe any properties written in an implementation extension. By trying to use implementation extensions to address the shortcoming of macros described in this proposal, we would end up creating a larger problem for those same macros. diff --git a/proposals/0408-pack-iteration.md b/proposals/0408-pack-iteration.md new file mode 100644 index 0000000000..b625affbc9 --- /dev/null +++ b/proposals/0408-pack-iteration.md @@ -0,0 +1,203 @@ +# Pack Iteration + +* Proposal: [SE-0408](0408-pack-iteration.md) +* Authors: [Sima Nerush](https://github.com/simanerush), [Holly Borla](https://github.com/hborla) +* Review Manager: [Doug Gregor](https://github.com/DougGregor/) +* Status: **Implemented (Swift 6.0)** +* Implementation: [apple/swift#67594](https://github.com/apple/swift/pull/67594) +* Review: ([pitch](https://forums.swift.org/t/pitch-enable-pack-iteration/66168), [review](https://forums.swift.org/t/review-se-0408-pack-iteration/67152), [acceptance](https://forums.swift.org/t/accepted-se-0408-pack-iteration/67598)) + +## Introduction + +Building upon the Value and Type Parameter Packs proposal [SE-0393](https://forums.swift.org/t/se-0393-value-and-type-parameter-packs/63859), this proposal enables iterating over each element in a value pack and bind each value to a local variable using a `for-in` syntax. + + +## Motivation + +Currently, it is possible to express list operations on value packs using pack expansion expressions. This approach requires putting code involving statements into a function or closure. For example, limiting repetition patterns to expressions does not allow for short-circuiting with `break` or `continue` statements, so the pattern expression will always be evaluated once for every element in the pack. The only way to stop evaluation would be to mark the function/closure containing the pattern expression throwing, and catch the error in a do/catch block to return, which is unnatural for Swift users. + +The following implementation of `==` over tuples of arbitrary length demonstrates these workarounds: + +```swift +struct NotEqual: Error {} + +func == (lhs: (repeat each Element), rhs: (repeat each Element)) -> Bool { + // Local throwing function for operating over each element of a pack expansion. + func isEqual(_ left: T, _ right: T) throws { + if left == right { + return + } + + throw NotEqual() + } + + // Do-catch statement for short-circuiting as soon as two tuple elements are not equal. + do { + repeat try isEqual(each lhs, each rhs) + } catch { + return false + } + + return true +} +``` + +Here, the programmer can only return `false` when the `NotEqual` error was thrown. The `isEqual` function performs the comparison and throws if the sides are not equal. + +## Proposed Solution + +We propose allowing iteration over value packs using `for-in` loops. With the adoption of pack iteration, the implementation of the standard library methods like `==` operator for tuples of any number of elements will become straightforward. Instead of throwing a `NotEqual` error, the function can simply iterate over each respective element of the tuple and `return false` in the body of the loop if the elements are not equal: + +```swift +func == (lhs: (repeat each Element), rhs: (repeat each Element)) -> Bool { + + for (left, right) in repeat (each lhs, each rhs) { + guard left == right else { return false } + } + return true +} +``` + +The above code iterates pairwise over two tuples `lhs` and `rhs` using a `for-in` loop syntax. At each iteration, the pack elements of `lhs` and `rhs` are bound to the local variables `left` and `right`, respectively. Because `lhs` and `rhs` have the same type, so do `left` and `right`, and the `Equatable` requirement allows comparing `left` and `right` with `==`. + +## Detailed Design + +In addition to expressions that conform to `Sequence`, the source of a `for-in` loop can be a pack expansion expression. + +```swift +func iterate(over element: repeat each Element) { + for element in repeat each element { + + } +} +``` + +On the *i*th iteration, the type of `element` is the *i*th type parameter in the `Element` type parameter pack, and the value of `element` is the *i*th value parameter in the shadowed `element` value parameter pack. Conceptually, the type of `element` is the pattern type of `repeat each element` with each captured type parameter pack replaced with an implicit scalar type parameter with matching requirements. In this case, the pattern type is `each Element`, so the type of `element` is a scalar type parameter with no requirements. Let’s call the scalar type parameter `Element'`. For example, if `iterate` is called with `each Element` bound to the type pack `{Int, String, Bool}` the body of the `for-in` loop will first substitute `Element'` for `Int`, then `String`, then `Bool`. + +If the type parameter packs captured by the pack expansion pattern contain requirements, the scalar type parameter in the loop body will have the same requirements: + +```swift +struct Generic {} + +protocol P {} + +func iterate() { + for x in repeat Generic() { + // the type of 'x' is Generic + } +} +``` + +In the above code, the pattern type of the pack expansion is `Generic` where `each Element: P`, so the type of `x` is a scalar type parameter ` where Element': P`. + +Like regular `for-in` loops, `for-in` loops over pack expansions can pattern match over each element of the value pack: + +```swift +enum E { + case one(T) + case two +} + +func iterate(over element: repeat E) { + for case .one(let value) in repeat each element { + // 'value' has type Element' + } +} +``` +The pattern expression in the source of a `for-in repeat` loop is evaluated once at each iteration, instead of `n` times eagerly where `n` is the length of the packs captured by the pattern. If `p_i` is the pattern expression at the `i`th iteration and control flow exits the loop at iteration `i`, then `p_j` is not evaluated for `i < j < n`. For example: + +```swift +func printAndReturn(_ value: Value) -> Value { + print("Evaluated pack element value \(value)") + return value +} + +func iterate(_ t: repeat each T) { + var i = 0 + for value in repeat printAndReturn(each t) { + print("Evaluating loop iteration \(i)") + if i == 1 { + break + } else { + i += 1 + } + } + + print("Done iterating") +} + +iterate(1, "hello", true) +``` + +The above code has the following output + +``` +Evaluated pack element value 1 +Evaluating loop iteration 0 +Evaluated pack element value "hello" +Evaluating loop iteration 1 +Done iterating +``` + +## Source Compatibility + +There is no source compatibility impact, since this is an additive change. + + +## ABI Compatibility + +This proposal does not affect ABI, since its impact is only in expressions. + + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. + +## Alternatives Considered + +The only alternative to allowing pack iteration would be placing code with statements into functions/closures, however, that approach is unnatural and over-complicated. For example, this is a version of the `==` operator for tuples implementation mentioned earlier. + +## Future directions + +### Enabling `guard let` + +Another familiar pattern to Swift programmers is `guard let`. + +For example, consider the `zip` function from the standard library. `guard let` can be used for enabling this function to support any number of sequences, instead of just 2: + +```swift +public func zip(_ sequences: repeat each S) -> ZipSequence { + .init(sequences: repeat each sequences) +} + +public struct ZipSequence { + let sequences: (repeat each S) +} + +extension ZipSequence: Sequence { + public typealias Element = (repeat (each S).Element) + + public struct Iterator: IteratorProtocol { + var iterators: (repeat (each S).Iterator) + var reachedEnd = false + + public mutating func next() -> Element? { + if reachedEnd { + return nil + } + + // Using guard let for checking that the next element is not nil. + guard let element = repeat (each iterators).next() else { + return nil + } + + return (repeat each element) + } + } + + public func makeIterator() -> Iterator { + return Iterator(iter: (repeat (each sequences).makeIterator())) + } +} +``` + diff --git a/proposals/0409-access-level-on-imports.md b/proposals/0409-access-level-on-imports.md new file mode 100644 index 0000000000..4eec8c9df6 --- /dev/null +++ b/proposals/0409-access-level-on-imports.md @@ -0,0 +1,342 @@ +# Access-level modifiers on import declarations + +* Proposal: [SE-0409](0409-access-level-on-imports.md) +* Author: [Alexis Laferrière](https://github.com/xymus) +* Review Manager: [Frederick Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.0)** +* Implementation: On main and release/5.9 gated behind the frontend flag `-enable-experimental-feature AccessLevelOnImport` +* Upcoming Feature Flag: `InternalImportsByDefault` +* Review: ([pitch](https://forums.swift.org/t/pitch-access-level-on-import-statements/66657)) ([review](https://forums.swift.org/t/se-0409-access-level-modifiers-on-import-declarations/67290)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0409-access-level-modifiers-on-import-declarations/67666)) + +## Introduction + +Declaring the visibility of a dependency with an access-level modifier on import declarations enables enforcing which declarations can reference the imported module. +A dependency can be marked as being visible only to the source file, module, package, or to all clients. +This brings the familiar behavior of the access level of declarations to dependencies and imported declarations. +This feature can hide implementation details from clients and helps to manage dependency creep. + +## Motivation + +Good practices guide us to separate public and internal services to avoid having external clients rely on internal details. +Swift already offers access levels with their respective modifiers to declarations and enforcement during type-checking, +but there is currently no equivalent official feature for dependencies. + +The author of a library may have a different intent for each of the library dependencies; +some are expected to be known to the library clients while others are for implementation details internal to the package, module, or source file. +Without a way to enforce the intended access level of dependencies +it is easy to make a mistake and expose a dependency of the library to the library clients by referencing it from a public declaration even if it's intended to remain an implementation detail. + +All the library dependencies being visible to the library clients also requires the compiler to do more work than necessary. +The compiler must load all of the library dependencies when building a client of the library, +even the dependencies that are not actually required to build the client. + +## Proposed solution + +The core of this proposal consists of extending the current access level logic to support declaring the existing modifiers (excluding `open`) on import declarations and +applying the access level to the imported declarations. + +Here's an example case where a module `DatabaseAdapter` is an implementation detail of the local module. +We don't want to expose it to clients so we mark the import as `internal`. +The compiler then allows references to it from internal functions but diagnoses references from the signature of public functions. +```swift +internal import DatabaseAdapter + +internal func internalFunc() -> DatabaseAdapter.Entry {...} // Ok +public func publicFunc() -> DatabaseAdapter.Entry {...} // error: function cannot be declared public because its result uses an internal type +``` + +Additionally, this proposal uses the access level declared on each import declaration in all source files composing a module to determine when clients of a library need to load the library's dependencies or when they can be skipped. +To balance source compatibility and best practices, an import without explicit access level has an implicit access level of `public` in Swift 5 and Swift 6. It will be `internal` in a future language mode. +The attribute `@usableFromInline` on an import allows references from inlinable code. + +## Detailed design + +In this section we discuss the three main language changes of this proposal: +accept access-level modifiers on import declarations to declare the visibility of the imported module, +apply that information when type-checking the source file, +and determine when indirect clients can skip loading transitive dependencies. +We then cover other concerns addressed by this proposal: +the different default access levels of imports in different language modes, +and the relationship with other attributes on imports. + +### Declaring the access level of an imported module + +The access level is declared in front of the import declaration using some of the +modifiers used for a declaration: `public`, `package`, `internal`, `fileprivate`, and `private`. + +A public dependency can be referenced from any declaration and will be visible to all clients. +It is declared with the `public` modifier. + +```swift +public import PublicDependency +``` + +A dependency visible only to the modules of the same package is declared with the `package` modifier. +Only the signature of `package`, `internal`, `fileprivate` and `private` declarations can reference the imported module. + +```swift +package import PackageDependency +``` + +A dependency internal to the module is declared with the `internal` modifier. +Only the signature of `internal`, `fileprivate` and `private` declarations can reference the imported module. + +```swift +internal import InternalDependency +``` + +A dependency private to this source file is declared with either the `fileprivate` or the `private` modifier. +In both cases the access is scoped to the source file declaring the import. +Only the signature of `fileprivate` and `private` declarations can reference the imported module. + +```swift +fileprivate import DependencyPrivateToThisFile +private import OtherDependencyPrivateToThisFile +``` + +The `open` access-level modifier is rejected on import declarations. + +The `@usableFromInline` attribute can be applied to an import declaration to allow referencing a dependency from inlinable code +while limiting which declarations signatures can reference it. +The attribute `@usableFromInline` can be used only on `package` and `internal` imports. +It marks the dependency as visible to clients. +```swift +@usableFromInline package import UsableFromInlinePackageDependency +@usableFromInline internal import UsableFromInlineInternalDependency +``` + +*Note: Support for @usableFromInline on imports has yet to be implemented.* + +### Type-checking references to imported modules + +Current type-checking enforces that declaration respect their respective access levels. +It reports as errors when a more visible declaration refers to a less visible declaration. +For example, it raises an error if a `public` function signature uses an `internal` type. + +This proposal extends the existing logic by using the access level on the import declaration as an upper bound to the visibility of imported declarations within the source file with the import. +For example, when type-checking a source file with an `internal import SomeModule`, +we consider all declarations imported from `SomeModule` to have an access level of `internal` in the context of the file. +In this case, type-checking will enforce that declarations imported as `internal` are only referenced from `internal` or lower declaration signatures and in regular function bodies. +They cannot appear in public declaration signatures, `@usableFromInline` declaration signatures, or inlinable code. +This will be reported by the familiar diagnostics currently applied to access-level modifiers on declarations and to inlinable code. + +We apply the same logic for `package`, `fileprivate` and `private` import declarations. +In the case of a `public` import, there is no restriction on how the imported declarations can be referenced +beyond the existing restrictions on imported `package` declarations which cannot be referenced from public declaration signatures. + +The attribute `@usableFromInline` on an import takes effect for inlinable code: +`@inlinable` and `@backDeployed` function bodies, default initializers of arguments, and properties of `@frozen` structs. +The `@usableFromInline` imported dependency can be referenced from inlinable code +but doesn't affect type-checking of declaration signatures where only the access level is taken into account. + +Here is an example of the approximate diagnostics produced from type-checking in a typical case with a `fileprivate` import. +```swift +fileprivate import DatabaseAdapter + +fileprivate func fileprivateFunc() -> DatabaseAdapter.Entry { ... } // Ok + +internal func internalFunc() -> DatabaseAdapter.Entry { ... } // error: function cannot be declared internal because its return uses a fileprivate type + +public func publicFunc(entry: DatabaseAdapter.Entry) { ... } // error: function cannot be declared public because its parameter uses a fileprivate type + +public func useInBody() { + DatabaseAdapter.create() // Ok +} + +@inlinable +public func useInInlinableBody() { + DatabaseAdapter.create() // error: global function 'create()' is fileprivate and cannot be referenced from an '@inlinable' function +} +``` + +### Transitive dependency loading + +When using this access level information at the module level, +if a dependency is never imported publicly and other requirements are met, +it becomes possible to hide the dependency from clients. +The clients can then be built without loading the transitive dependency. +This can speed up build times and +avoid the need to distribute modules that are implementation details. + +The same dependency can be imported with different access levels by different files of a same module. +At the module level, we only take into account the most permissive access level. +For example, if a dependency is imported as `package` and `internal` from two different files, +we consider the dependency to be of `package` visibility at the module level. + +The module level information implies different behaviors for transitive clients. +Transitive clients are modules that have an indirect dependency on the module. +For example, in the following scenario, `TransitiveClient` is a transitive client +of `IndirectDependency` via the import of `MiddleModule`. + +``` +module IndirectDependency + ↑ +module MiddleModule + ↑ +module TransitiveClient +``` + +Depending on how the indirect dependency is imported from the middle module, +the transitive client may or may not need to load it at compile time. +There are four factors requiring a transitive dependency to be loaded; +if none of these apply, the dependency can be hidden. + +1. `public` or `@usableFromInline` dependencies must always be loaded by transitive clients. + +2. All dependencies of a non-resilient module must be loaded by transitive clients. + This is because types in the module can use types from those dependencies in their storage, + and the compiler needs complete information about the storage of non-resilient types + in order to emit code correctly. + This restriction is discussed further in the Future Directions section. + +3. `package` dependencies of a module must be loaded by its transitive clients if the module and the transitive client are part of the same package. + This is because `package` declarations in the module may use types from that dependency in their signatures. + We consider two modules to be in the same package when their package name matches, + applying the same logic used for package declarations. + +4. All dependencies of a module must be loaded if the transitive client has a `@testable` import of it. + This is because testable clients can use `internal` declarations, which may rely on dependencies with any level of import visibility. + Even `private` and `fileprivate` dependencies must be loaded. + +In all other cases, the dependency is hidden, and it doesn't have to be loaded by transitive clients. +Note that a dependency hidden on one import path may still need to be loaded because of a different import path. + +The module interface associated with a hidden dependency doesn't need to be distributed to clients. +However, the binary associated to the module still needs to be distributed to execute the resulting program. + +### Default import access level + +The access level of a default import declaration without an explicit access-level modifier depends on the language version. +We list here the implicit access levels and reasoning behind this choice. + +In language modes up to Swift 6, an import is `public` by default. +This choice preserves source compatibility. +The only official import previously available in Swift 5 behaves like the public import proposed in this document. + +In a future language mode, an import will be `internal` by default. +This will align the behavior of imports with declarations where the implicit access level is internal. +It should help limit unintentional dependency creep as marking a dependency public will require an explicit modifier. + +As a result, the following import is `public` in language modes up to Swift 6, but it will be `internal` in a future language mode: +```swift +import ADependency +``` + +The future language change will likely require source changes in code that adopts the new language mode. It will not break source compatibility for code that remains on current language modes. +A migration tool could automatically insert the `public` modifier where required. +Where the tool is unavailable, a simple script can insert a `public` modifier in front of all imports to preserve the Swift 5 behavior. + +The upcoming feature flag `InternalImportsByDefault` will enable the future language behavior even when using Swift 5 or 6. + +### Interactions with other modifiers on imports + +The `@_exported` attribute is a step above a `public` import, +as clients see the imported module declarations is if they were part of the local module. +With this proposal, `@_exported` is accepted only on public import declarations, +both with the modifier or the default `public` visibility in current language modes. + +The `@testable` attribute allows the local module to reference the internal declarations of the imported module. +The current design even allows to use an imported internal or package type in a public declaration. +The access level behavior applies in the same way as a normal import, +all imported declarations have as upper-bound the access level on the import declaration. +In the case of a `@testable` import, even the imported internal declarations are affected by the bound. + +Current uses of `@_implementationOnly import` should be replaced with an internal import or lower. +In comparison, this new feature enables stricter type-checking and shows fewer superfluous warnings. +After replacing with an internal import, the transitive dependency loading requirements will remain the same for resilient modules, +but will change for non-resilient modules where transitive dependencies must always be loaded. +In all cases, updating modules relying on `@_implementationOnly` to instead use internal imports is strongly encouraged. + +The scoped imports feature is independent from the access level declared on the same import. +In the example below, the module `Foo` is a public dependency at the module level and can be referenced from public declaration signatures in the local source file. +The scoped part, `struct Foo.Bar`, limits lookup so only `Bar` can be referenced from this file; it also prioritizes resolving references to this `Bar` if there are other `Bar` declarations in other imports. +Scoped imports cannot be used to restrict the access level of a single declaration. +```swift +public import struct Foo.Bar +``` + +## Source compatibility + +To preserve source compatibility, imports are public by default in current language modes, including Swift 6. +This will preserve the current behavior of imports in Swift 5. +As discussed previously, the future language mode behavior changes the default value and will require code changes. + +## ABI compatibility + +This proposal doesn't affect ABI compatibility, +it is a compile time change enforced by type-checking. + +## Implications on adoption + +Adopting or reverting the adoption of this feature should not affect clients if used with care. + +In the case of adoption in a non-resilient module, the change is in type-checking of the module source files only. +In this case changing the access level of different dependencies won't affect clients. + +For adoption in a resilient module, +marking an existing import as less than public will affect how clients build. +The compiler can build the clients by loading fewer transitive dependencies. +In theory, this shouldn't affect the clients but it may still lead to different compilation behaviors. + +In theory, these transitive dependencies couldn't be used by the clients, +so hiding them doesn't affect the clients. +In practice, there are leaks allowing use of extension members from transitive dependencies. +Adopting this feature may skip loading transitive dependencies and prevent those leaks, +it can break source compatibility in code relying of those behaviors. + +## Future directions + +### Hiding dependencies for non-resilient modules + +Hiding dependencies on non-resilient modules would be possible in theory but requires rethinking a few restrictions in the compilation process. +The main restriction is the need of the compiler to know the memory layout of imported types, which can depend on transitive dependencies. +Resilient modules can provide this information at run time so the transitive module isn't required at build time. +Non-resilient modules do not provide this information at run time, so the compiler must load the transitive dependencies at build time to access it. +Solutions could involve copying the required information in each modules, +or restricting further how a dependency can be referenced. +In all cases, it's a feature in itself and distinct from this proposal. + +## Alternatives considered + +### `@_implementationOnly import` + +The unofficial `@_implementationOnly` attribute offers a similar feature with both type-checking and hiding transitive dependencies. +This attribute has lead to instability and run time crashes when used from a non-resilient module or combined with an `@testable` import. +It applies a slightly different semantic than this proposal and its type-checking isn't as strict as it could be. +It relied on its own type-checking logic to report references to the implementation-only imported module from public declarations. +In contrast, this proposal uses the existing access level checking logic and semantics, +this should make it easier to learn. +Plus this proposal introduces whole new features with `package` imports and file-scoped imports with `private` and `fileprivate`. + +### Use `open import` as an official `@_exported import` + +The access-level modifier `open` remains available for use on imports as this proposal doesn't assign it a specific meaning. +It has been suggested to use it as an official `@_exported`. +That is, mark an import that is visible from all source files of the module and shown to clients as if it was part of the same module. +We usually use `@_exported` for Swift overlays to clang module +where two modules share the same name and the intention is to show them as unified to clients. + +Two main reasons keep me from incorporating this change to this proposal: + +1. A declaration marked as `open` can be overridden from outside the module. + This meaning has no relation with the behavior of `@_exported`. + The other access levels have a corresponding meaning between their use on a declaration and on an import declaration. +2. A motivation for this proposal is to hide implementation details and limit dependency creep. + Encouraging the use of `open import` or `@_exported` goes against this motivation and addresses a different set of problems. + It should be discussed in a distinct proposal with related motivations. + +### Infer the visibility of a dependency from its use in API + +By analyzing a module the compiler could determine which dependencies are used by public declarations and need to be visible to clients. +We could then automatically consider all other dependencies as internal and hide them from indirect clients if the other criteria are met. + +This approach lacks the duplication of information offered by the access-level modifier on the import declaration and the references from declaration signatures. +This duplication enables the type-checking behavior described in this proposal by +allowing the compiler to compare the intent marked on the import with the use in declaration signatures. +This check is important when the dependency is not distributed, +a change from a hidden dependency to a public dependency may break the distributed module on a dependency that is not available to third parties. + +## Acknowledgments + +Becca Royal-Gordon contributed to the design and wrote the pre-pitch of this proposal. + diff --git a/proposals/0410-atomics.md b/proposals/0410-atomics.md new file mode 100644 index 0000000000..21d348b3a2 --- /dev/null +++ b/proposals/0410-atomics.md @@ -0,0 +1,1844 @@ +# Low-Level Atomic Operations ⚛︎ + +* Proposal: [SE-0410](0410-atomics.md) +* Author: [Karoy Lorentey](https://github.com/lorentey), [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Bug: [SR-9144](https://github.com/apple/swift/issues/51640) +* Implementation: [apple/swift#68857](https://github.com/apple/swift/pull/68857) +* Version: 2023-12-04 +* Status: **Implemented (Swift 6.0)** +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/d35d6566fe2297f4782bdfac4d5253e0ca96b353/proposals/0410-atomics.md) +* Decision Notes: [pitch](https://forums.swift.org/t/atomics/67350), [first review](https://forums.swift.org/t/se-0410-atomics/68007), [first return for revision](https://forums.swift.org/t/returned-for-revision-se-0410-atomics/68522), [second review](https://forums.swift.org/t/second-review-se-0410-atomics/68810), [acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0410-atomics/69244) + +## Introduction + +This proposal adds a limited set of low-level atomic operations to the Standard Library, including native spellings for C++-style memory orderings. Our goal is to enable intrepid library authors and developers writing system level code to start building synchronization constructs directly in Swift. + +Previous Swift-evolution thread: [Low-Level Atomic Operations](https://forums.swift.org/t/low-level-atomic-operations/34683) + +New Swift-evolution thread: [Atomics](https://forums.swift.org/t/atomics/67350) + +## Revision History + +- 2020-04-13: Initial proposal version. +- 2020-06-05: Second revision. + - Removed all new APIs; the proposal is now focused solely on C interoperability. +- 2023-09-18: Third revision. + - Introduced new APIs to the standard library. +- 2023-12-04: Fourth revision. + - Response to language steering group [review decision notes](https://forums.swift.org/t/returned-for-revision-se-0410-atomics/68522). + - New APIs are now in a `Synchronization` module instead of the default `Swift` module. + - Declaring a `var` of `Atomic` type is now an error. + +## Table of Contents + + * [Motivation](#motivation) + * [Proposed Solution](#proposed-solution) + * [The Synchronization Module](#the-synchronization-module) + * [Atomic Memory Orderings](#atomic-memory-orderings) + * [The Atomic Protocol Hierarchy](#the-atomic-protocol-hierarchy) + * [Optional Atomics](#optional-atomics) + * [Custom Atomic Types](#custom-atomic-types) + * [Atomic Storage Types](#atomic-storage-types) + * [WordPair](#wordpair) + * [The Atomic type](#the-atomic-type) + * [Basic Atomic Operations](#basic-atomic-operations) + * [Specialized Integer Operations](#specialized-integer-operations) + * [Specialized Boolean Operations](#specialized-boolean-operations) + * [Atomic Lazy References](#atomic-lazy-references) + * [Restricting Ordering Arguments to Compile\-Time Constants](#restricting-ordering-arguments-to-compile-time-constants) + * [Interaction with Existing Language Features](#interaction-with-existing-language-features) + * [Interaction with Swift Concurrency](#interaction-with-swift-concurrency) + * [Detailed Design](#detailed-design) + * [Atomic Memory Orderings](#atomic-memory-orderings-1) + * [Atomic Protocols](#atomic-protocols) + * [AtomicRepresentable](#atomicrepresentable) + * [AtomicOptionalRepresentable](#atomicoptionalrepresentable) + * [WordPair](#wordpair-1) + * [Atomic Types](#atomic-types) + * [Atomic<Value>](#atomicvalue) + * [AtomicLazyReference<Instance>](#atomiclazyreferenceinstance) + * [Source Compatibility](#source-compatibility) + * [Effect on ABI Stability](#effect-on-abi-stability) + * [Effect on API Resilience](#effect-on-api-resilience) + * [Potential Future Directions](#potential-future-directions) + * [Atomic Strong References and The Problem of Memory Reclamation](#atomic-strong-references-and-the-problem-of-memory-reclamation) + * [Additional Low\-Level Atomic Features](#additional-low-level-atomic-features) + * [Alternatives Considered](#alternatives-considered) + * [Default Orderings](#default-orderings) + * [A Truly Universal Generic Atomic Type](#a-truly-universal-generic-atomic-type) + * [Providing a value Property](#providing-a-value-property) + * [Alternative Designs for Memory Orderings](#alternative-designs-for-memory-orderings) + * [Encode Orderings in Method Names](#encode-orderings-in-method-names) + * [Orderings As Generic Type Parameters](#orderings-as-generic-type-parameters) + * [Ordering Views](#ordering-views) + * [Directly bring over `swift-atomics`'s API](#directly-bring-over-swift-atomicss-api) + * [References](#references) + +## Motivation + +In Swift today, application developers use Swift's recently accepted concurrency features including async/await, structured concurrency with Task and TaskGroup, AsyncSequence/AsyncStream, etc. as well as dispatch queues and Foundation's NSLocking protocol to synchronize access to mutable state across concurrent threads of execution. + +However, for Swift to be successful as a systems programming language, it needs to also provide low-level primitives that can be used to implement such synchronization constructs (and many more!) directly within Swift. Such low-level synchronization primitives allow developers more flexible ways to synchronize access to specific properties or storage allowing them to opt their types into Swift conconcurrency by declaring their types `@unchecked Sendable`. Of course these low-level primitives also allow library authors to build more high level synchronization structures that are both easier and safer to use that developers can also utilize to synchronize memory access. + +One such low-level primitive is the concept of an atomic value, which (in the form we propose here) has two equally important roles: + +- First, atomics introduce a limited set of types whose values provide well-defined semantics for certain kinds of concurrent access. This includes explicit support for concurrent mutations -- a concept that Swift never supported before. + +- Second, atomic operations come with explicit memory ordering arguments, which provide guarantees on how/when the effects of earlier or later memory accesses become visible to other threads. Such guarantees are crucial for building higher-level synchronization abstractions. + +These new primitives are intended for people who wish to implement synchronization constructs or concurrent data structures in pure Swift code. Note that this is a hazardous area that is full of pitfalls. While a well-designed atomics facility can help simplify building such tools, the goal here is merely to make it *possible* to build them, not necessarily to make it *easy* to do so. We expect that the higher-level synchronization tools that can be built on top of these atomic primitives will provide a nicer abstraction layer. + +We want to limit this proposal to constructs that satisfy the following requirements: + +1. All atomic operations need to be explicit in Swift source, and it must be possible to easily distinguish them from regular non-atomic operations on the underlying values. + +2. The atomic type we provide must come with a lock-free implementation on every platform that implements them. (Platforms that are unable to provide lock-free implementations must not provide the affected constructs at all.) + +3. Every atomic operation must compile down to the corresponding CPU instruction (when one is available), with minimal overhead. (Ideally even if the code is compiled without optimizations.) Wait-freedom isn't a requirement -- if no direct instruction is available for an operation, then it must still be implemented, e.g. by mapping it to a compare-exchange loop. + +Following the acceptance of [Clarify the Swift memory consistency model (SE-0282)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md), the [swift-atomics package](https://github.com/apple/swift-atomics) was shortly created to experiment and design what a standard atomic API would look like. This proposal is relying heavily on some of the ideas that package has spent years developing and designing. + +## Proposed Solution + +We propose to introduce new low-level atomic APIs to the standard library via a new module. These atomic APIs will serve as the foundation for building higher-level concurrent code directly in Swift. + +As a quick taste, this is how atomics will work: + +```swift +import Synchronization +import Dispatch + +let counter = Atomic(0) + +DispatchQueue.concurrentPerform(iterations: 10) { _ in + for _ in 0 ..< 1_000_000 { + counter.wrappingAdd(1, ordering: .relaxed) + } +} + +print(counter.load(ordering: .relaxed)) +``` + +### The Synchronization Module + +While most Swift programs won't directly use the new atomic primitives, we still consider the new constructs to be an integral part of the core Standard Library. + +That said, it seems highly undesirable to add low-level atomics to the default namespace of every Swift program, so we propose to place the atomic constructs in a new Standard Library module called `Synchronization`. Code that needs to use low-level atomics will need to explicitly import the new module: + +```swift +import Synchronization +``` + +We expect that most Swift projects will use atomic operations only indirectly, through higher-level synchronization constructs. Therefore, importing the `Synchronization` module will be a relatively rare occurrence, mostly limited to projects that implement such tools. + +### Atomic Memory Orderings + +The atomic constructs later in this proposal implement concurrent read/write access by mapping to atomic instructions in the underlying architecture. All accesses of a particular atomic value get serialized into some global sequential timeline, no matter what thread executed them. + +However, this alone does not give us a way to synchronize accesses to regular variables, or between atomic accesses to different memory locations. To support such synchronization, each atomic operation can be configured to also act as a synchronization point for other variable accesses within the same thread, preventing previous accesses from getting executed after the atomic operation, and/or vice versa. Atomic operations on another thread can then synchronize with the same point, establishing a strict (although partial) timeline between accesses performed by both threads. This way, we can reason about the possible ordering of operations across threads, even if we know nothing about how those operations are implemented. (This is how locks or dispatch queues can be used to serialize the execution of arbitrary blocks containing regular accesses to shared variables.) For more details, see \[[C++17], [N2153], [Boehm 2008]]. + +In order to enable atomic synchronization within Swift, we must first introduce memory orderings that will give us control of the timeline of these operations across threads. Luckily, with the acceptance of [Clarify the Swift memory consistency model (SE-0282)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md), Swift already adopts the C/C++ concurrency memory model. In this model, concurrent access to shared state remains undefined behavior unless all such access is forced into a conflict-free timeline through explicit synchronization operations. + +This proposal introduces five distinct memory orderings, organized into three logical groups, from loosest to strictest: + +* `.relaxed` +* `.acquiring`, `.releasing`, `.acquiringAndReleasing` +* `.sequentiallyConsistent` + +These align with select members of the standard `std::memory_order` enumeration in C++, and are intended to carry the same semantic meaning: + +| C++ | Swift | +| :---: | :---: | +| `std::memory_order_relaxed` | `.relaxed` | +| `std::memory_order_consume` | *not yet adopted [[P0735]]* | +| `std::memory_order_acquire` | `.acquiring` | +| `std::memory_order_release` | `.releasing` | +| `std::memory_order_acq_rel` | `.acquiringAndReleasing` | +| `std::memory_order_seq_cst` | `.sequentiallyConsistent` | + +Atomic orderings are grouped into three frozen structs based on the kind of operation to which they are attached, as listed below. By modeling these as separate types, we can ensure that unsupported operation/ordering combinations (such as an atomic "releasing load") will lead to clear compile-time errors: + +```swift +/// Specifies the memory ordering semantics of an atomic load operation. +public struct AtomicLoadOrdering { + public static var relaxed: Self { get } + public static var acquiring: Self { get } + public static var sequentiallyConsistent: Self { get } +} + +/// Specifies the memory ordering semantics of an atomic store operation. +public struct AtomicStoreOrdering { + public static var relaxed: Self { get } + public static var releasing: Self { get } + public static var sequentiallyConsistent: Self { get } +} + +/// Specifies the memory ordering semantics of an atomic read-modify-write +/// operation. +public struct AtomicUpdateOrdering { + public static var relaxed: Self { get } + public static var acquiring: Self { get } + public static var releasing: Self { get } + public static var acquiringAndReleasing: Self { get } + public static var sequentiallyConsistent: Self { get } +} +``` + +These structs behave like non-frozen enums with a known (non-public) raw representation. This allows us to define additional memory orderings in the future (if and when they become necessary, specifically `std::memory_order_consume`) while making use of the known representation to optimize existing cases. (These cannot be frozen enums because that would prevent us from adding more orderings, but regular resilient enums can't freeze their representation, and the layout indirection interferes with guaranteed optimizations, especially in -Onone.) + +Every atomic operation introduced later in this proposal requires an ordering argument. We consider these ordering arguments to be an essential part of these low-level atomic APIs, and we require an explicit `ordering` argument on all atomic operations. The intention here is to force developers to carefully think about what ordering they need to use, each time they use one of these primitives. (Perhaps more importantly, this also makes it obvious to readers of the code what ordering is used -- making it far less likely that an unintended default `.sequentiallyConsistent` ordering slips through code review.) + +Projects that prefer to default to sequentially consistent ordering are welcome to add non-public `Atomic` extensions that implement that. However, we expect that providing an implicit default ordering would be highly undesirable in most production uses of atomics. + +We also provide a top-level function called `atomicMemoryFence` that allows issuing a memory ordering constraint without directly associating it with a particular atomic operation. This corresponds to `std::atomic_thread_fence` in C++ [[C++17]]. + +```swift +/// Establishes a memory ordering without associating it with a +/// particular atomic operation. +/// +/// - A relaxed fence has no effect. +/// - An acquiring fence ties to any preceding atomic operation that +/// reads a value, and synchronizes with any releasing operation whose +/// value was read. +/// - A releasing fence ties to any subsequent atomic operation that +/// modifies a value, and synchronizes with any acquiring operation +/// that reads the result. +/// - An acquiring and releasing fence is a combination of an +/// acquiring and a releasing fence. +/// - A sequentially consistent fence behaves like an acquiring and +/// releasing fence, and ensures that the fence itself is part of +/// the single, total ordering for all sequentially consistent +/// operations. +/// +/// This operation corresponds to `std::atomic_thread_fence` in C++. +/// +/// Be aware that Thread Sanitizer does not support fences and may report +/// false-positive races for data protected by a fence. +public func atomicMemoryFence(ordering: AtomicUpdateOrdering) +``` + +Fences are slightly more powerful (but even more difficult to use) than orderings tied to specific atomic operations [[N2153]]; we expect their use will be limited to the most performance-sensitive synchronization constructs. + +### The Atomic Protocol Hierarchy + +The notion of an atomic type is captured by the `AtomicRepresentable` protocol. + +```swift +/// A type that supports atomic operations through a separate atomic storage +/// representation. +public protocol AtomicRepresentable { + associatedtype AtomicRepresentation + + static func encodeAtomicRepresentation( + _ value: consuming Self + ) -> AtomicRepresentation + + static func decodeAtomicRepresentation( + _ representation: consuming AtomicRepresentation + ) -> Self +} +``` + +The requirements in `AtomicRepresentable` set up a bidirectional mapping between values of the atomic type and an associated storage representation that implements the actual primitive atomic operations. + +`AtomicRepresentation` is intentionally left unconstrained because as you'll see later in the proposal, atomic operations are only available when `AtomicRepresentation` is one of the core atomic storage types found here: [Atomic Storage Types](#atomic-storage-types). + +The full set of standard types implementing `AtomicRepresentable` is listed below: + +```swift +extension Int: AtomicRepresentable {...} +extension Int64: AtomicRepresentable {...} +extension Int32: AtomicRepresentable {...} +extension Int16: AtomicRepresentable {...} +extension Int8: AtomicRepresentable {...} +extension UInt: AtomicRepresentable {...} +extension UInt64: AtomicRepresentable {...} +extension UInt32: AtomicRepresentable {...} +extension UInt16: AtomicRepresentable {...} +extension UInt8: AtomicRepresentable {...} + +extension Bool: AtomicRepresentable {...} + +extension Float16: AtomicRepresentable {...} +extension Float: AtomicRepresentable {...} +extension Double: AtomicRepresentable {...} + +/// New type in the standard library discussed +/// shortly after this. +extension WordPair: AtomicRepresentable {...} + +extension Duration: AtomicRepresentable {...} + +extension Never: AtomicRepresentable {...} + +extension UnsafeRawPointer: AtomicRepresentable {...} +extension UnsafeMutableRawPointer: AtomicRepresentable {...} +extension UnsafePointer: AtomicRepresentable {...} +extension UnsafeMutablePointer: AtomicRepresentable {...} +extension Unmanaged: AtomicRepresentable {...} +extension OpaquePointer: AtomicRepresentable {...} +extension ObjectIdentifier: AtomicRepresentable {...} + +extension UnsafeBufferPointer: AtomicRepresentable {...} +extension UnsafeMutableBufferPointer: AtomicRepresentable {...} +extension UnsafeRawBufferPointer: AtomicRepresentable {...} +extension UnsafeMutableRawBufferPointer: AtomicRepresentable {...} + +extension Optional: AtomicRepresentable where Wrapped: AtomicOptionalRepresentable {...} +``` + +* On 32 bit platforms that do not support double-word atomics, the following conformances are not available: + * `UInt64` + * `Int64` + * `Double` + * `UnsafeBufferPointer` + * `UnsafeMutableBufferPointer` + * `UnsafeRawBufferPointer` + * `UnsafeMutableRawBufferPointer` +* On 64 bit platforms that do not support double-word atomics, the following conformances are not available: + * `Duration` + * `UnsafeBufferPointer` + * `UnsafeMutableBufferPointer` + * `UnsafeRawBufferPointer` + * `UnsafeMutableRawBufferPointer` + +This proposal does not conform `Duration` to `AtomicRepresentable` on any currently supported 32 bit platform. (Not even those where quad-word atomics are technically available, like arm64_32.) + +#### Optional Atomics + +The standard atomic pointer types and unmanaged references also support atomic operations on their optional-wrapped form. To spell out this optional wrapped, we introduce a new protocol: + +```swift +public protocol AtomicOptionalRepresentable: AtomicRepresentable { + associatedtype AtomicOptionalRepresentation + + static func encodeAtomicOptionalRepresentation( + _ value: consuming Self? + ) -> AtomicOptionalRepresentation + + static func decodeAtomicOptionalRepresentation( + _ representation: consuming AtomicOptionalRepresentation + ) -> Self? +} +``` + +Similar to `AtomicRepresentable`, `AtomicOptionalRepresentable`'s requirements create a bidirectional mapping between an optional value of `Self` to some atomic optional storage representation and vice versa. + +`Optional` implements `AtomicRepresentable` through a conditional conformance to this new `AtomicOptionalRepresentable` protocol. + +```swift +extension Optional: AtomicRepresentable where Wrapped: AtomicOptionalRepresentable { + ... +} +``` + +This proposal enables optional-atomics support for the following types: + +```swift +extension UnsafeRawPointer: AtomicOptionalRepresentable {} +extension UnsafeMutableRawPointer: AtomicOptionalRepresentable {} +extension UnsafePointer: AtomicOptionalRepresentable {} +extension UnsafeMutablePointer: AtomicOptionalRepresentable {} +extension Unmanaged: AtomicOptionalRepresentable {} +extension OpaquePointer: AtomicOptionalRepresentable {} +extension ObjectIdentifier: AtomicOptionalRepresentable {} +``` + +Atomic optional pointers and references are helpful when building lock-free data structures. (Although this initial set of reference types considerably limits the scope of what can be built; for more details, see the discussion on the [ABA problem](#wordpair) and [memory reclamation](#atomic-strong-references-and-the-problem-of-memory-reclamation).) + +For example, consider the lock-free, single-consumer stack implementation below. (It supports an arbitrary number of concurrently pushing threads, but it only allows a single pop at a time.) + +```swift +class LockFreeSingleConsumerStack { + struct Node { + let value: Element + var next: UnsafeMutablePointer? + } + typealias NodePtr = UnsafeMutablePointer + + private let _last = Atomic(nil) + private let _consumerCount = Atomic(0) + + deinit { + // Discard remaining nodes + while let _ = pop() {} + } + + // Push the given element to the top of the stack. + // It is okay to concurrently call this in an arbitrary number of threads. + func push(_ value: Element) { + let new = NodePtr.allocate(capacity: 1) + new.initialize(to: Node(value: value, next: nil)) + + var done = false + var current = _last.load(ordering: .relaxed) + while !done { + new.pointee.next = current + (done, current) = _last.compareExchange( + expected: current, + desired: new, + ordering: .releasing + ) + } + } + + // Pop and return the topmost element from the stack. + // This method does not support multiple overlapping concurrent calls. + func pop() -> Element? { + precondition( + _consumerCount.wrappingAdd(1, ordering: .acquiring).oldValue == 0, + "Multiple consumers detected") + defer { _consumerCount.wrappingSubtract(1, ordering: .releasing) } + var done = false + var current = _last.load(ordering: .acquiring) + while let c = current { + (done, current) = _last.compareExchange( + expected: c, + desired: c.pointee.next, + ordering: .acquiring + ) + + if done { + let result = c.move() + c.deallocate() + return result.value + } + } + return nil + } +} +``` + +#### Custom Atomic Types + +To enable a limited set of user-defined atomic types, `AtomicRepresentable` also provides a full set of default implementations for `RawRepresentable` types whose raw value is itself atomic: + +```swift +extension RawRepresentable where Self: AtomicRepresentable, RawValue: AtomicRepresentable { + ... +} +``` + +The default implementations work by forwarding all atomic operations to the raw value's implementation, converting to/from as needed. + +This enables code outside of the Standard Library to add new `AtomicRepresentable` conformances without manually implementing any of the requirements. This is especially handy for trivial raw-representable enumerations, such as in simple atomic state machines: + +```swift +enum MyState: Int, AtomicRepresentable { + case starting + case running + case stopped +} + +let currentState = Atomic(.starting) +... +if currentState.compareExchange( + expected: .starting, + desired: .running, + ordering: .sequentiallyConsistent +).exchanged { + ... +} +... +currentState.store(.stopped, ordering: .sequentiallyConsistent) +``` + +We also support the `AtomicOptionalRepresentable` defaults for `RawRepresentable` as well: + +```swift +extension RawRepresentable where Self: AtomicOptionalRepresentable, RawValue: AtomicOptionalRepresentable { + ... +} +``` + +For example, we can use this to add atomic operations over optionals of types whose raw value is a pointer: + +```swift +struct MyPointer: RawRepresentable, AtomicOptionalRepresentable { + var rawValue: UnsafeRawPointer + + init(rawValue: UnsafeRawPointer) { + self.rawValue = rawValue + } +} + +let myAtomicPointer = Atomic(nil) +... +if myAtomicPointer.compareExchange( + expected: nil, + desired: MyPointer(rawValue: somePointer), + ordering: .relaxed +).exchanged { + ... +} +... +myAtomicPointer.store(nil, ordering: .releasing) +``` + +(This gets you an `AtomicRepresentable` conformance for free as well because `AtomicOptionalRepresentable` refines `AtomicRepresentable`. So this also allows non-optional use with `Atomic`.) + +### Atomic Storage Types + +Fundamental to working with atomics is knowing that CPUs can only do atomic operations on integers. While we could theoretically do atomic operations with our current list of standard library integer types (`Int8`, `Int16`, ...), some platforms don't ensure that these types have the same alignment as their size. For example, `Int64` and `UInt64` have 4 byte alignment on i386. Atomic operations must occur on correctly aligned types. To ensure this, we need to introduce helper types that all atomic operations will be trafficked through. These types will serve as the `AtomicRepresentation` for all of the standard integer types: + +```swift +extension Int8: AtomicRepresentable { + public typealias AtomicRepresentation = ... +} + +... + +extension UInt64: AtomicRepresentable { + public typealias AtomicRepresentation = ... +} + +... +``` + +The actual underlying type is an implementation detail of the standard library. While we generally don't prefer to propose such API, the underlying types themselves are quite useless and only useful for the primitive integers. One can still access the underlying type by using the public name, `Int8.AtomicRepresentation`, for example. An example conformance to `AtomicRepresentable` may look something like the following: + +```swift +struct MyCoolInt { + var x: Int +} + +extension MyCoolInt: AtomicRepresentable { + typealias AtomicRepresentation = Int.AtomicRepresentation + + static func encodeAtomicRepresentation( + _ value: consuming MyCoolInt + ) -> AtomicRepresentation { + Int.encodeAtomicRepresentation(value.x) + } + + static func decodeAtomicRepresentation( + _ representation: consuming AtomicRepresentation + ) -> MyCoolInt { + MyCoolInt( + x:Int.decodeAtomicRepresentation(representation) + ) + } +} +``` + +This works by going through `Int`'s `AtomicRepresentable` conformance and converting our `MyCoolInt` -> `Int` -> `Int.AtomicRepresentation` . + +### `WordPair` + +In their current single-word form, atomic pointer and reference types are susceptible to a class of race condition called the *ABA problem*. A freshly allocated object often happens to be placed at the same memory location as a recently deallocated one. Therefore, two successive `load`s of a simple atomic pointer may return the exact same value, even though the pointer may have received an arbitrary number of updates between the two loads, and the pointee may have been completely replaced. This can be a subtle, but deadly source of race conditions in naive implementations of many concurrent data structures. + +While the single-word atomic primitives introduced in this document are already useful for some applications, it would be helpful to also provide a set of additional atomic operations that operate on two consecutive `Int`-sized values in the same transaction. All currently supported architectures provide direct hardware support for such double-word atomic operations. + +We propose a new separate type that provides an abstraction over the layout of what a double-word is for a platform. + +```swift +public struct WordPair { + public var first: UInt { get } + public var second: UInt { get } + + public init(first: UInt, second: UInt) +} + +// Not a real compilation conditional +#if hasDoubleWideAtomics +extension WordPair: AtomicRepresentable { +// Not a real compilation conditional +#if 64 bit + public typealias AtomicRepresentaton = ... 128 bit 16 aligned storage +#elseif 32 bit + public typealias AtomicRepresentation = ... 64 bit 8 aligned storage +#else +#error("Not a supported platform") +#endif + + ... +} +#endif +``` + +For example, the second word can be used to augment atomic values with a version counter (sometimes called a "stamp" or a "tag"), which can help resolve the ABA problem by allowing code to reliably verify if a value remained unchanged between two successive loads. + +Note that not all CPUs support double-word atomic operations and so if Swift starts supporting such processors, this type's conformance to `AtomicRepresentable` may not always be available. Platforms that cannot support double-word atomics must not make `WordPair`'s `AtomicRepresentable` conformance available for use. + +(If this becomes a real concern, a future proposal could introduce something like a `#if hasDoubleWordAtomics` compile-time condition to let code adapt to more limited environments. However, this is deferred until Swift actually starts supporting such platforms.) + +### The Atomic type + +So far, we've introduced memory orderings, giving us control of memory access around atomic operations; the atomic protocol hierarchy, which give us the initial list of standard types that can be as atomic values; and the `WordPair` type, providing an abstraction over a platform's double-word type. However, we haven't yet introduced a way to actually _use_ atomics. Here we introduce the single Atomic type that exposes atomic operations for us: + +```swift +/// An atomic value. +public struct Atomic: ~Copyable { + public init(_ initialValue: consuming Value) +} +``` + +A value of `Atomic` shares the same layout as `Value.AtomicRepresentation`. + +Now that we know how to create an atomic value, it's time to introduce some actual atomic operations. + +### Basic Atomic Operations + +`Atomic` provides seven basic atomic operations when `Value.AtomicRepresentation` is one of the fundamental atomic storage types on the standard integer types: + +```swift +extension Atomic where Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation { + /// Atomically loads and returns the current value, applying the specified + /// memory ordering. + /// + /// - Parameter ordering: The memory ordering to apply on this operation. + /// - Returns: The current value. + public borrowing func load(ordering: AtomicLoadOrdering) -> Value + + /// Atomically sets the current value to `desired`, applying the specified + /// memory ordering. + /// + /// - Parameter desired: The desired new value. + /// - Parameter ordering: The memory ordering to apply on this operation. + public borrowing func store( + _ desired: consuming Value, + ordering: AtomicStoreOrdering + ) + + /// Atomically sets the current value to `desired` and returns the original + /// value, applying the specified memory ordering. + /// + /// - Parameter desired: The desired new value. + /// - Parameter ordering: The memory ordering to apply on this operation. + /// - Returns: The original value. + public borrowing func exchange( + _ desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> Value + + /// Perform an atomic compare and exchange operation on the current value, + /// applying the specified memory ordering. + /// + /// This operation performs the following algorithm as a single atomic + /// transaction: + /// + /// ``` + /// atomic(self) { currentValue in + /// let original = currentValue + /// guard original == expected else { return (false, original) } + /// currentValue = desired + /// return (true, original) + /// } + /// ``` + /// + /// This method implements a "strong" compare and exchange operation + /// that does not permit spurious failures. + /// + /// - Parameter expected: The expected current value. + /// - Parameter desired: The desired new value. + /// - Parameter ordering: The memory ordering to apply on this operation. + /// - Returns: A tuple `(exchanged, original)`, where `exchanged` is true if + /// the exchange was successful, and `original` is the original value. + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Value) + + /// Perform an atomic compare and exchange operation on the current value, + /// applying the specified success/failure memory orderings. + /// + /// This operation performs the following algorithm as a single atomic + /// transaction: + /// + /// ``` + /// atomic(self) { currentValue in + /// let original = currentValue + /// guard original == expected else { return (false, original) } + /// currentValue = desired + /// return (true, original) + /// } + /// ``` + /// + /// The `successOrdering` argument specifies the memory ordering to use when + /// the operation manages to update the current value, while `failureOrdering` + /// will be used when the operation leaves the value intact. + /// + /// This method implements a "strong" compare and exchange operation + /// that does not permit spurious failures. + /// + /// - Parameter expected: The expected current value. + /// - Parameter desired: The desired new value. + /// - Parameter successOrdering: The memory ordering to apply if this + /// operation performs the exchange. + /// - Parameter failureOrdering: The memory ordering to apply on this + /// operation if it does not perform the exchange. + /// - Returns: A tuple `(exchanged, original)`, where `exchanged` is true if + /// the exchange was successful, and `original` is the original value. + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + successOrdering: AtomicUpdateOrdering, + failureOrdering: AtomicLoadOrdering + ) -> (exchanged: Bool, original: Value) + + /// Perform an atomic weak compare and exchange operation on the current + /// value, applying the memory ordering. This compare-exchange variant is + /// allowed to spuriously fail; it is designed to be called in a loop until + /// it indicates a successful exchange has happened. + /// + /// This operation performs the following algorithm as a single atomic + /// transaction: + /// + /// ``` + /// atomic(self) { currentValue in + /// let original = currentValue + /// guard original == expected else { return (false, original) } + /// currentValue = desired + /// return (true, original) + /// } + /// ``` + /// + /// (In this weak form, transient conditions may cause the `original == + /// expected` check to sometimes return false when the two values are in fact + /// the same.) + /// + /// - Parameter expected: The expected current value. + /// - Parameter desired: The desired new value. + /// - Parameter ordering: The memory ordering to apply on this operation. + /// - Returns: A tuple `(exchanged, original)`, where `exchanged` is true if + /// the exchange was successful, and `original` is the original value. + public borrowing func weakCompareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Value) + + /// Perform an atomic weak compare and exchange operation on the current + /// value, applying the specified success/failure memory orderings. This + /// compare-exchange variant is allowed to spuriously fail; it is designed to + /// be called in a loop until it indicates a successful exchange has happened. + /// + /// This operation performs the following algorithm as a single atomic + /// transaction: + /// + /// ``` + /// atomic(self) { currentValue in + /// let original = currentValue + /// guard original == expected else { return (false, original) } + /// currentValue = desired + /// return (true, original) + /// } + /// ``` + /// + /// (In this weak form, transient conditions may cause the `original == + /// expected` check to sometimes return false when the two values are in fact + /// the same.) + /// + /// The `ordering` argument specifies the memory ordering to use when the + /// operation manages to update the current value, while `failureOrdering` + /// will be used when the operation leaves the value intact. + /// + /// - Parameter expected: The expected current value. + /// - Parameter desired: The desired new value. + /// - Parameter successOrdering: The memory ordering to apply if this + /// operation performs the exchange. + /// - Parameter failureOrdering: The memory ordering to apply on this + /// operation does not perform the exchange. + /// - Returns: A tuple `(exchanged, original)`, where `exchanged` is true if + /// the exchange was successful, and `original` is the original value. + public borrowing func weakCompareExchange( + expected: consuming Value, + desired: consuming Value, + successOrdering: AtomicUpdateOrdering, + failureOrdering: AtomicLoadOrdering + ) -> (exchanged: Bool, original: Value) +} +``` + +Because these are only available when `Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation`, some atomic specializations may not support atomic operations at all. + +The first three operations are relatively simple: + +- `load` returns the current value. +- `store` updates it. +- `exchange` is a combination of `load` and `store`; it updates the + current value and returns the previous one as a single atomic + operation. + +The three `compareExchange` variants are somewhat more complicated: they implement a version of `exchange` that only performs the update if the original value is the same as a supplied expected value. To be specific, they execute the following algorithm as a single atomic transaction: + +```swift + guard currentValue == expected else { + return (exchanged: false, original: currentValue) + } + currentValue = desired + return (exchanged: true, original: expected) +``` + +All four variants implement the same algorithm. The single ordering variants use the same memory ordering whether or not the exchange succeeds, while the others allow callers to specify two distinct memory orderings for the success and failure cases. The two orderings are independent from each other -- all combinations of update/load orderings are supported [[P0418]]. (Of course, the implementation may need to "round up" to the nearest ordering combination that is supported by the underlying code generation layer and the targeted CPU architecture.) + +The `weakCompareExchange` form may sometimes return false even when the original and expected values are equal. (Such failures may happen when some transient condition prevents the underlying operation from succeeding -- such as an incoming interrupt during a load-link/store-conditional instruction sequence.) This variant is designed to be called in a loop that only exits when the exchange is successful; in such loops, using `weakCompareExchange` may lead to a performance improvement by eliminating a nested loop in the regular, "strong", `compareExchange` variants. + +The compare-exchange primitive is special: it is a universal operation that can be used to implement all other atomic operations, and more. For example, here is how we could use `compareExchange` to implement a wrapping add operation over `Atomic` values: + +```swift +extension Atomic where Value == Int { + func wrappingAdd( + _ operand: Int, + ordering: AtomicUpdateOrdering + ) { + var done = false + var current = load(ordering: .relaxed) + while !done { + (done, current) = compareExchange( + expected: current, + desired: current &+ operand, + ordering: ordering + ) + } + } +} +``` + +### Specialized Integer Operations + +Most CPU architectures provide dedicated atomic instructions for certain integer operations, and these are generally more efficient than implementations using `compareExchange`. Therefore, it makes sense to expose a set of dedicated methods for common integer operations so that these will always get compiled into the most efficient implementation available. + +| Method Name | Returns | Implements | +| --- | --- | --- | +| `wrappingAdd(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a &+= b` | +| `wrappingSubtract(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a &-= b` | +| `add(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a += b` (checks for overflow) | +| `subtract(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a -= b` (checks for overflow) | +| `bitwiseAnd(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a &= b` | +| `bitwiseOr(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a \|= b` | +| `bitwiseXor(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a ^= b` | +| `min(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a = Swift.min(a, b)` | +| `max(_: Value, ordering: AtomicUpdateOrdering)` | `(oldValue: Value, newValue: Value)` | `a = Swift.max(a, b)` | + +All operations are also marked as `@discardableResult` in the case where one doesn't care about the old value or new value. The `add` and `subtract` operations explicitly check for overflow and will trap at runtime if one occurs, except in `-Ounchecked` builds. + +While we require all atomic operations to be free of locks, we don't require wait-freedom. Therefore, on architectures that don't provide direct hardware support for some or all of these operations, we still require them to be implemented using `compareExchange` loops like the one for `wrappingAdd` above. + +`Atomic` exposes these operations when `Value` is one of the standard fixed-width integer types. + +```swift +extension Atomic where Value == Int {...} +extension Atomic where Value == UInt8 {...} +... + +let counter = Atomic(0) +counter.wrappingAdd(42, ordering: .relaxed) + +let oldMax = counter.max(82, ordering: .relaxed).oldValue +``` + +### Specialized Boolean Operations + +Similar to the specialized integer operations, we can provide similar ones for booleans: + +| Method Name | Returns | Implements | +| ---------------------------- | --------------------- | ------------ | +| `logicalAnd(_: Bool, ordering: AtomicUpdateOrdering)` | `(oldValue: Bool, newValue: Bool)` | `a = a && b` | +| `logicalOr(_: Bool, ordering: AtomicUpdateOrdering)` | `(oldValue: Bool, newValue: Bool)` | `a = a \|\| b` | +| `logicalXor(_: Bool, ordering: AtomicUpdateOrdering)` | `(oldValue: Bool, newValue: Bool)` | `a = a != b` | + +Like the integer operations, all of these boolean operations are marked as `@discardableResult`. + +`Atomic` exposes these operations when `Value` is `Bool`. + +```swift +extension Atomic where Value == Bool {...} + +let tracker = Atomic(false) +let newOr = tracker.logicalOr(true, ordering: .relaxed).newValue +``` + +### Atomic Lazy References + +The operations provided by `Atomic>` only operate on the unmanaged reference itself. They don't allow us to directly access the referenced object -- we need to manually invoke the methods `Unmanaged` provides for this purpose (usually, `takeUnretainedValue`). + +Note that loading the atomic unmanaged reference and converting it to a strong reference are two distinct operations that won't execute as a single atomic transaction. This can easily lead to race conditions when a thread releases an object while another is busy loading it: + +```swift +// BROKEN CODE. DO NOT EMULATE IN PRODUCTION. +let myAtomicRef = Atomic>(...) + +// Thread A: Load the unmanaged value and then convert it to a regular +// strong reference. +let ref = myAtomicRef.load(ordering: .acquiring).takeUnretainedValue() +... + +// Thread B: Store a new reference in the atomic unmanaged value and +// release the previous reference. +let new = Unmanaged.passRetained(...) +let old = myAtomicRef.exchange(new, ordering: .acquiringAndReleasing) +old.release() // RACE CONDITION +``` + +If thread B happens to release the same object that thread A is in the process of loading, then thread A's `takeUnretainedValue` may attempt to retain a deallocated object. + +Such problems make `Atomic>` exceedingly difficult to use in all but the simplest situations. The section on [*Atomic Strong References*](#atomic-strong-references-and-the-problem-of-memory-reclamation) below describes some new constructs we may introduce in future proposals to assist with this issue. + +For now, we provide the standalone type `AtomicLazyReference`; this is an example of a useful construct that could be built on top of `Atomic>` operations. (Of the atomic constructs introduced in this proposal, only `AtomicLazyReference` represents a regular strong reference to a class instance -- the other pointer/reference types leave memory management entirely up to the user.) + +An `AtomicLazyReference` holds an optional reference that is initially set to `nil`. The value can be set exactly once, but it can be read an arbitrary number of times. Attempts to change the value after the first `storeIfNil` call are ignored, and return the current value instead. + +```swift +/// A lazily initializable atomic strong reference. +/// +/// These values can be set (initialized) exactly once, but read many +/// times. +public struct AtomicLazyReference: ~Copyable { + /// The value logically stored in an atomic lazy reference value. + public typealias Value = Instance? + + /// Initializes a new managed atomic lazy reference with a nil value. + public init() +} + +extension AtomicLazyReference { + /// Atomically initializes this reference if its current value is nil, then + /// returns the initialized value. If this reference is already initialized, + /// then `storeIfNil(_:)` discards its supplied argument and returns + /// the current value without updating it. + /// + /// The following example demonstrates how this can be used to implement a + /// thread-safe lazily initialized reference: + /// + /// ``` + /// class Image { + /// var _histogram: AtomicLazyReference = .init() + /// + /// // This is safe to call concurrently from multiple threads. + /// var atomicLazyHistogram: Histogram { + /// if let histogram = _histogram.load() { return histogram } + /// // Note that code here may run concurrently on + /// // multiple threads, but only one of them will get to + /// // succeed setting the reference. + /// let histogram = ... + /// return _histogram.storeIfNil(histogram) + /// } + /// ``` + /// + /// This operation uses acquiring-and-releasing memory ordering. + public borrowing func storeIfNil( + _ desired: consuming Instance + ) -> Instance + + /// Atomically loads and returns the current value of this reference. + /// + /// The load operation is performed with the memory ordering + /// `AtomicLoadOrdering.acquiring`. + public borrowing func load() -> Instance? +} +``` + +This is the only atomic type in this proposal that doesn't provide the usual `load`/`store`/`exchange`/`compareExchange` operations. + +This construct allows library authors to implement a thread-safe lazy initialization pattern: + +```swift +let _foo: AtomicLazyReference = ... + +// This is safe to call concurrently from multiple threads. +nonisolated var atomicLazyFoo: Foo { + if let foo = _foo.load() { return foo } + // Note: the code here may run concurrently on multiple threads. + // All but one of the resulting values will be discarded. + let foo = Foo() + return _foo.storeIfNil(foo) +} +``` + +The Standard Library has been internally using such a pattern to implement deferred bridging for `Array`, `Dictionary` and `Set`. + +Note that unlike the rest of the atomic types, `load` and `storeIfNil(_:)` do not expose `ordering` parameters. (Internally, they map to acquiring/releasing operations to guarantee correct synchronization.) + +### Restricting Ordering Arguments to Compile-Time Constants + +Modeling orderings as regular function parameters allows us to specify them using syntax that's familiar to all Swift programmers. Unfortunately, it means that in the implementation of atomic operations we're forced to switch over the ordering argument: + +```swift +extension Atomic where Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation { + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Int) { + // Note: This is a simplified version of the actual implementation + let won: Bool + let oldValue: Value + + switch ordering { + case .relaxed: + (oldValue, won) = Builtin.cmpxchg_monotonic_monotonic_IntNN( + address, expected, desired + ) + + case .acquiring: + (oldValue, won) = Builtin.cmpxchg_acquire_acquire_IntNN( + address, expected, desired + ) + + case .releasing: + (oldValue, won) = Builtin.cmpxchg_release_monotonic_IntNN( + address, expected, desired + ) + + case .acquiringAndReleasing: + (oldValue, won) = Builtin.cmpxchg_acqrel_acquire_IntNN( + address, expected, desired + ) + + case .sequentiallyConsistent: + (oldValue, won) = Builtin.cmpxchg_seqcst_seqcst_IntNN( + address, expected, desired + ) + + default: + fatalError("Unknown atomic memory ordering") + } + + return (exchanged: won, original: oldValue) + } +} +``` + +Given our requirement that primitive atomics must always compile down to the actual atomic instructions with minimal additional overhead, we must guarantee that these switch statements always get optimized away into the single case we need; they must never actually be evaluated at runtime. + +Luckily, configuring these special functions to always get force-inlined into all callers guarantees that constant folding will get rid of the switch statement *as long as the supplied ordering is a compile-time constant*. Unfortunately, it's all too easy to accidentally violate this latter requirement, with dire consequences to the expected performance of the atomic operation. + +Consider the following well-meaning attempt at using `compareExchange` to define an atomic integer addition operation that traps on overflow rather than allowing the result to wrap around: + +```swift +extension Atomic where Value == Int { + // Non-inlinable + public func add(_ operand: Int, ordering: AtomicUpdateOrdering) { + var done = false + var current = load(ordering: .relaxed) + + while !done { + (done, current) = compareExchange( + expected: current, + desired: current + operand, // Traps on overflow + ordering: ordering + ) + } + } +} + +// Elsewhere: +counter.add(1, ordering: .relaxed) +``` + +If for whatever reason the Swift compiler isn't able (or willing) to inline the `add` call, then the value of `ordering` won't be known at compile time to the body of the function, so even though `compareExchange` will still get inlined, its switch statement won't be eliminated. This leads to a potentially significant performance regression that could interfere with the scalability of the operation. + +The big issue here is that if `add` is in another module, then callers of this function have no visibility inside this function's body. If callers can't see this function's implementation, then the switch statement will be executed at runtime regardless of the compiler optimization mode. However, another issue is that the ordering argument may still be dynamic in which case the compiler still can't eliminate the switch statement even though the caller may be able to see the entire implementation. + +To prevent the last issue, the memory ordering arguments of all atomic operations must be compile-time constants. Any attempt to pass a dynamic ordering value (such as in the `compareExchange` call above) will result in a compile-time error. + +An ordering expression will be considered constant-evaluable if it's either (1) a direct call to one of the `Atomic*Ordering` factory methods (`.relaxed`, `.acquiring`, etc.), or (2) it is a direct reference to a variable that is in turn constrained to be constant-evaluable. + +## Interaction with Existing Language Features + +Please refer to the [Clarify the Swift memory consistency model (SE-0282)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md#interaction-with-non-instantaneous-accesses) proposal which goes over how atomics interact with the Law of Exclusivity, Non-Instantaneous Accesses, and Implicit Pointer Conversions. + +An additional note with regards to the Law of Exclusivity, atomic values should never be declared with a `var` binding, always prefer a `let` one. Consider the following: + +```swift +class Counter { + var value: Atomic +} +``` + +By declaring this variable as a `var`, we opt into Swift's dynamic exclusivity checking for this property, so all non-exclusive accesses incur a runtime check to see if there is an active exclusive (e.g. mutating) access. This inherently means that atomic operations through such a variable will incur undesirable runtime overhead -- they are no longer purely atomic. (Even if the check never actually triggers a trap.) + +To prevent users from accidentally falling into this trap, `Atomic` (and `AtomicLazyReference`) will not support `var` bindings. It is a compile-time error to have a `var` that has an explicit or inferred type of `Atomic`. + +```swift +// error: variable of type 'Atomic' must be declared with a 'let' +var myAtomic = Atomic(123) +``` + +By making this a compiler error, we can safely assume that atomic accesses will never incur an unexpected dynamic exclusivity check. It is forbidden to create mutable variables of type `struct Atomic`. + +Similarly, it is an error to declare a computed property that returns an `Atomic`, as its getter would need to create and return a new instance each time it is accessed. Instead, you can return the actual value that would be the initial value for the atomic: + +```swift +var computedInt: Int { + 123 +} + +let myAtomic = Atomic(computedInt) +``` + +Alternatively, you can choose to convert the property to a function. This makes it clear that a new instance is being returned every time the function is called: + +```swift +func makeAnAtomic() -> Atomic { + Atomic(123) +} + +let myAtomic = makeAnAtomic() +``` + + + +In the same vein, these types must never be passed as `inout` parameters as that declares that the callee has exclusive access to the atomic, which would make the access no longer atomic. Attempting to create an `inout` binding for an atomic variable is also a compile-time error. Parameters that are used to pass `Atomic` values must either be `borrowing` or `consuming`. (Passing a variable as `consuming` is also an exclusive access, but it's destroying the original variable, so we no longer need to care for its atomicity.) + +```swift +// error: parameter of type 'Atomic' must be declared as either 'borrowing' or 'consuming' +func passAtomic(_: inout Atomic) +``` + +Mutating methods on atomic types are also forbidden, as they introduce `inout` bindings on `self`. For example, trying to extend `Atomic` with our own `mutating` method results in a compile-time error: + +```swift +extension Atomic { + // error: type `Atomic` cannot have mutating function 'greet()' + mutating func greet() { + print("Hello! From: Atomic") + } +} +``` + +These conditions for `Atomic` and `AtomicLazyReference` are important to prevent users from accidentally introducing dynamic exclusivity for these low-level performance sensitive concurrency primitives. + +### Interaction with Swift Concurrency + +The `Atomic` type is `Sendable` where `Value: Sendable`. One can pass a value of an `Atomic` to an actor or any other async context by using a borrow reference. + +```swift +actor Updater { + ... + + func update(_ counter: borrowing Atomic) { + ... + } + + func doOtherWork() {} +} + +func version1() async { + let counter = Atomic(0) + + // |--- There are no suspension points in this function, so + // | this atomic value will be allocated on whatever + // | thread's stack that decides to run this async function. + // | + // v + let updatedCount = counter.load(ordering: .relaxed) +} + +func version2() async { + let counter = Atomic(0) + + let updater = Updater() + await updater.update(counter) // <--------- Potential suspension point that + // uses the atomic value directly. + // The atomic value will get + // promoted to the async stack frame + // meaning it will be available until + // this async function has ended. + + // |----- Atomic value used after suspension point. + // | Because of the fact that we're passing it to a + // | suspension point, it's already been allocated on + // | the async stack frame, and accessing it later will + // | access that same resource, preserving atomicity. + // | + // v + let updatedCount = counter.load(ordering: .relaxed) +} + +func version3() async { + let counter = Atomic(0) + + let updater = Updater() + await updater.doOtherWork() // <--------- Potential suspension point that + // doesn't use the atomic value directly. + + + // |----- Atomic value used after suspension point, so it is + // | promoted to the async stack frame which makes this + // | value's lifetime persist even after the await. + // | The compiler could in theory also reorder the + // | atomic value's initialization after the suspension + // | because it isn't used before nor during meaning it + // | could be allocated on whatever thread's stack frame. + // | + // v + let updatedCount = counter.load(ordering: .relaxed) +} +``` + +Variables of type `struct Atomic` are always located at a single, stable memory location, no matter its nature (be that a stored property in a class type or a noncopyable struct, an associated value in a noncopyable enum, a local variable that got promoted to the heap through a closure capture, or any other kind of variable.) + +Considering these factors, we can safely declare that `struct Atomic` is `Sendable` whenever its value is `Sendable`. By analogue reasoning, `struct AtomicLazyReference` is declared `Sendable` whenever its instance is `Sendable`. + +## Detailed Design + +In the interest of keeping this document (relatively) short, the following API synopsis does not include API documentation, inlinable method bodies, or `@usableFromInline` declarations, and omits most attributes (`@available`, `@inlinable`, etc.). + +To allow atomic operations to compile down to their corresponding CPU instructions, most entry points listed here will be defined `@inlinable`. + +For the full API definition, please refer to the [implementation](https://github.com/apple/swift/pull/68857). + +### Atomic Memory Orderings + +```swift +public struct AtomicLoadOrdering: Equatable, Hashable, CustomStringConvertible { + public static var relaxed: Self { get } + public static var acquiring: Self { get } + public static var sequentiallyConsistent: Self { get } + + public static func ==(left: Self, right: Self) -> Bool + public func hash(into hasher: inout Hasher) + public var description: String { get } +} + +public struct AtomicStoreOrdering: Equatable, Hashable, CustomStringConvertible { + public static var relaxed: Self { get } + public static var releasing: Self { get } + public static var sequentiallyConsistent: Self { get } + + public static func ==(left: Self, right: Self) -> Bool + public func hash(into hasher: inout Hasher) + public var description: String { get } +} + +public struct AtomicUpdateOrdering: Equatable, Hashable, CustomStringConvertible { + public static var relaxed: Self { get } + public static var acquiring: Self { get } + public static var releasing: Self { get } + public static var acquiringAndReleasing: Self { get } + public static var sequentiallyConsistent: Self { get } + + public static func ==(left: Self, right: Self) -> Bool + public func hash(into hasher: inout Hasher) + public var description: String { get } +} + +public func atomicMemoryFence(ordering: AtomicUpdateOrdering) +``` + +### Atomic Protocols + +#### `AtomicRepresentable` + +```swift +public protocol AtomicRepresentable { + associatedtype AtomicRepresentation + + static func encodeAtomicRepresentation( + _ value: consuming Self + ) -> AtomicRepresentation + + static func decodeAtomicRepresentation( + _ representation: consuming AtomicRepresentation + ) -> Self +} +``` + +The requirements set up a bidirectional mapping between values of the atomic type and an associated storage representation that supplies the actual primitive atomic operations. + +Conforming types: + +```swift +extension Int: AtomicRepresentable {...} +extension Int64: AtomicRepresentable {...} +extension Int32: AtomicRepresentable {...} +extension Int16: AtomicRepresentable {...} +extension Int8: AtomicRepresentable {...} +extension UInt: AtomicRepresentable {...} +extension UInt64: AtomicRepresentable {...} +extension UInt32: AtomicRepresentable {...} +extension UInt16: AtomicRepresentable {...} +extension UInt8: AtomicRepresentable {...} + +extension Bool: AtomicRepresentable {...} + +extension Float16: AtomicRepresentable {...} +extension Float: AtomicRepresentable {...} +extension Double: AtomicRepresentable {...} + +extension WordPair: AtomicRepresentable {...} +extension Duration: AtomicRepresentable {...} + +extension Never: AtomicRepresentable {...} + +extension UnsafeRawPointer: AtomicRepresentable {...} +extension UnsafeMutableRawPointer: AtomicRepresentable {...} +extension UnsafePointer: AtomicRepresentable {...} +extension UnsafeMutablePointer: AtomicRepresentable {...} +extension Unmanaged: AtomicRepresentable {...} +extension OpaquePointer: AtomicRepresentable {...} +extension ObjectIdentifier: AtomicRepresentable {...} + +extension UnsafeBufferPointer: AtomicRepresentable {...} +extension UnsafeMutableBufferPointer: AtomicRepresentable {...} +extension UnsafeRawBufferPointer: AtomicRepresentable {...} +extension UnsafeMutableRawBufferPointer: AtomicRepresentable {...} + +extension Optional: AtomicRepresentable where Wrapped: AtomicOptionalRepresentable {...} +``` + +To support custom "atomic-representable" types, `AtomicRepresentable` also comes with default implementations for all its requirements for `RawRepresentable` types whose `RawValue` is also atomic: + +```swift +extension RawRepresentable where Self: AtomicRepresentable, RawValue: AtomicRepresentable { + // Implementations for all requirements. +} +``` + +The default implementations work by converting values to their `rawValue` form, and forwarding all atomic operations to it. + +#### `AtomicOptionalRepresentable` + +```swift +public protocol AtomicOptionalRepresentable: AtomicRepresentable { + associatedtype AtomicOptionalRepresentation + + static func encodeAtomicOptionalRepresentation( + _ value: consuming Self? + ) -> AtomicOptionalRepresentation + + static func decodeAtomicOptionalRepresentation( + _ representation: consuming AtomicOptionalRepresentation + ) -> Self? +} +``` + +Atomic `Optional` operations are currently enabled for the following `Wrapped` types: + +```swift +extension UnsafeRawPointer: AtomicOptionalRepresentable {} +extension UnsafeMutableRawPointer: AtomicOptionalRepresentable {} +extension UnsafePointer: AtomicOptionalRepresentable {} +extension UnsafeMutablePointer: AtomicOptionalRepresentable {} +extension Unmanaged: AtomicOptionalRepresentable {} +extension OpaquePointer: AtomicOptionalRepresentable {} +extension ObjectIdentifier: AtomicOptionalRepresentable {} +``` + +### `WordPair` + +```swift +public struct WordPair { + public var first: UInt { get } + public var second: UInt { get } + + public init(first: UInt, second: UInt) +} + +extension WordPair: AtomicRepresentable {...} +extension WordPair: Equatable {...} +extension WordPair: Hashable {...} + +// NOTE: WordPair is semantically a (UInt, UInt). Tuple comparability +// works based of lexicographical ordering, so WordPair will do +// the same. It will compare 'first' first, and 'second' second. +extension WordPair: Comparable {...} + +extension WordPair: CustomStringConvertible {...} +extension WordPair: CustomDebugStringConvertible {...} +extension WordPair: Sendable {} +``` + +### Atomic Types + +#### `Atomic` + +```swift +public struct Atomic: ~Copyable { + public init(_ initialValue: consuming Value) +} + +extension Atomic where Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation { + // Atomic operations: + + public borrowing func load( + ordering: AtomicLoadOrdering + ) -> Value + + public borrowing func store( + _ desired: consuming Value, + ordering: AtomicStoreOrdering + ) + + public borrowing func exchange( + _ desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> Value + + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Value) + + public borrowing func compareExchange( + expected: consuming Value, + desired: consuming Value, + successOrdering: AtomicUpdateOrdering, + failureOrdering: AtomicLoadOrdering + ) -> (exchanged: Bool, original: Value) + + public borrowing func weakCompareExchange( + expected: consuming Value, + desired: consuming Value, + ordering: AtomicUpdateOrdering + ) -> (exchanged: Bool, original: Value) + + public borrowing func weakCompareExchange( + expected: consuming Value, + desired: consuming Value, + successOrdering: AtomicUpdateOrdering, + failureOrdering: AtomicLoadOrdering + ) -> (exchanged: Bool, original: Value) +} + +extension Atomic: @unchecked Sendable where Value: Sendable {} +``` + +`Atomic` also provides a handful of integer operations for the standard fixed-width integer types. This is implemented via same type requirements: + +```swift +extension Atomic where Value == Int { + @discardableResult + public borrowing func wrappingAdd( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func wrappingSubtract( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func add( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func subtract( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func bitwiseAnd( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func bitwiseOr( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func bitwiseXor( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func min( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func max( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) +} + +extension Atomic where Value == Int8 {...} +... +``` + +as well as providing convenience functions for boolean operations: + +```swift +extension Atomic where Value == Bool { + @discardableResult + public borrowing func logicalAnd( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func logicalOr( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) + + @discardableResult + public borrowing func logicalXor( + _ operand: Value, + ordering: AtomicUpdateOrdering + ) -> (oldValue: Value, newValue: Value) +} +``` + +#### `AtomicLazyReference` + +```swift +public struct AtomicLazyReference: ~Copyable { + public typealias Value = Instance? + + public init(_ initialValue: consuming Instance) + + // Atomic operations: + + public borrowing func storeIfNil( + _ desired: consuming Instance + ) -> Instance + + public borrowing func load() -> Instance? +} + +extension AtomicLazyReference: @unchecked Sendable where Instance: Sendable {} +``` + +## Source Compatibility + +This is a purely additive change with no source compatibility impact. + +## Effect on ABI Stability + +This proposal introduces new entry points to the Standard Library ABI in a standalone `Synchronization` module, but otherwise it has no effect on ABI stability. + +On ABI-stable platforms, the struct types and protocols introduced here will become part of the stdlib's ABI with availability matching the first OS releases that include them. + +Most of the atomic methods introduced in this document will be force-inlined directly into client code at every call site. As such, there is no reason to bake them into the stdlib's ABI -- the stdlib binary will not export symbols for them. + +## Effect on API Resilience + +This is an additive change; it has no effect on the API of existing code. + +For the new constructs introduced here, the proposed design allows us to make the following changes in future versions of the Swift Standard Library: + +- Addition of new atomic types (and higher-level constructs built around them). (These new types would not directly back-deploy to OS versions that predate their introduction.) + +- Addition of new memory orderings. Because all atomic operations compile directly into user code, new memory orderings that we decide to introduce later could potentially back-deploy to any OS release that includes this proposal. + +- Addition of new atomic operations on the types introduced here. These would be also be back deployable. + +- Introducing a default memory ordering for atomic operations (either by adding a default value to `ordering`, or by adding new overloads that lack that parameter). This too would be a back-deployable change. + +- Change the memory ordering model as long as the changes preserve source compatibility. + +(We don't necessarily plan to actually perform any of these changes; we merely leave the door open to doing them.) + +## Potential Future Directions + +### Atomic Strong References and The Problem of Memory Reclamation + +Perhaps counter-intuitively, implementing a high-performance, *lock-free* atomic version of regular everyday strong references is not a trivial task. This proposal doesn't attempt to provide such a construct beyond the limited use-case of `AtomicLazyReference`. + +Under the hood, Swift's strong references have always been using atomic operations to implement reference counting. This allows references to be read (but not mutated) from multiple, concurrent threads of execution, while also ensuring that each object still gets deallocated as soon as its last outstanding reference disappears. However, atomic reference counts on their own do not allow threads to safely share a single *mutable* reference without additional synchronization. + +The difficulty is in the implementation of the atomic load operation, which boils down to two separate sub-operations, both of which need to be part of the *same atomic transaction*: + +1. Load the value of the reference. +2. Increment the reference count of the corresponding object. + +If an intervening store operation were allowed to release the reference between steps 1 and 2, then the loaded reference could already be deallocated by the time `load` tries to increment its refcount. + +Without an efficient way to implement these two steps as a single atomic transaction, the implementation of `store` needs to delay releasing the overwritten value until it can guarantee that every outstanding load operation is completed. Exactly how to implement this is the problem of *memory reclamation* in concurrent data structures. + +There are a variety of approaches to tackle this problem, but the one we think would be the best fit is the implementation of [`AtomicReference`][https://swiftpackageindex.com/apple/swift-atomics/1.2.0/documentation/atomics/atomicreference] in the [swift-atomics package](https://github.com/apple/swift-atomics). + +### Additional Low-Level Atomic Features + +To enable use cases that require even more fine-grained control over atomic operations, it may be useful to introduce additional low-level atomics features: + +* support for additional kinds of atomic values (such as floating-point atomics [[P0020]]), +* new memory orderings, such as a consuming load ordering [[P0750]] or tearable atomics [[P0690]], +* "volatile" atomics that prevent certain compiler optimizations +* and more + +We defer these for future proposals. + +## Alternatives Considered + +### Default Orderings + +We considered defaulting all atomic operations to sequentially consistent ordering. While we concede that doing so would make atomics slightly more approachable, implicit ordering values tend to interfere with highly performance-sensitive use cases of atomics (which is *most* use cases of atomics). Sequential consistency tends to be relatively rarely used in these contexts, and implicitly defaulting to it would allow accidental use to easily slip through code review. + +Users who wish for default orderings are welcome to define their own overloads for atomic operations: + +```swift +extension Atomic where Value.AtomicRepresentation == UInt8.AtomicRepresentation { + func load() -> Value { + load(ordering: .sequentiallyConsistent) + } + + func store(_ desired: consuming Value) { + store(desired, ordering: .sequentiallyConsistent) + } + + func exchange(_ desired: consuming Value) -> Value { + exchange(desired, ordering: .sequentiallyConsistent) + } + + func compareExchange( + expected: consuming Value, + desired: consuming Value + ) -> (exchanged: Bool, original: Value) { + compareExchange( + expected: expected, + desired: desired, + ordering: .sequentiallyConsistent + ) + } + + func weakCompareExchange( + expected: consuming Value, + desired: consuming Value + ) -> (exchanged: Bool, original: Value) { + weakCompareExchange( + expected: expected, + desired: desired, + successOrdering: .sequentiallyConsistent, + failureOrdering: .sequentiallyConsistent + ) + } +} + +... + +extension Atomic where Value == Int { + func wrappingAdd(_ operand: Value) { + wrappingAdd(operand, ordering: .sequentiallyConsistent) + } + + etc. +} + +... +``` + +### A Truly Universal Generic Atomic Type + +While future proposals may add a variety of other atomic types, we do not expect to ever provide a truly universal generic `Atomic` construct. The `Atomic` type is designed to provide high-performance lock-free primitives, and these are heavily constrained by the atomic instruction sets of the CPU architectures Swift targets. + +A universal `Atomic` type that can hold *any* value is unlikely to be implementable without locks, so it is outside the scope of this proposal. We may eventually consider adding such a construct in a future concurrency proposal: + +```swift +struct Serialized: ~Copyable { + private let _lock = UnfairLock() + private var _value: Value + + func withLock(_ body: (inout Value) throws -> T) rethrows -> T { + _lock.lock() + defer { _lock.unlock() } + + return try body(&_value) + } +} +``` + +### Providing a `value` Property + +Our atomic constructs are unusual because even though semantically they behave like containers holding a value, they do not provide direct access to it. Instead of exposing a getter and a setter on a handy `value` property, they expose cumbersome `load` and `store` methods. There are two reasons for this curious inconsistency: + +First, there is the obvious issue that property getter/setters have no room for an ordering parameter. + +Second, there is a deep underlying problem with the property syntax: it encourages silent race conditions. For example, consider the code below: + +```swift +let counter = Atomic(0) +... +counter.value += 1 +``` + +Even though this increment looks like it may be a single atomic operation, it gets executed as two separate atomic transactions: + +```swift +var temp = counter.value // atomic load +temp += 1 +counter.value = temp // atomic store +``` + +If some other thread happens to update the value after the atomic load, then that update gets overwritten by the subsequent store, resulting in data loss. + +To prevent this gotcha, none of the proposed atomic types provide a property for accessing their value, and we don't foresee adding such a property in the future, either. + +(Note that this problem cannot be mitigated by implementing [modify accessors]. Lock-free updates cannot be implemented without the ability to retry the update multiple times, and modify accessors can only yield once.) + +[modify accessors]: https://forums.swift.org/t/modify-accessors/31872 + +### Alternative Designs for Memory Orderings + +Modeling memory orderings with enumeration(-like) values fits well into the Standard Library's existing API design practice, but `ordering` arguments aren't without problems. Most importantly, the quality of code generation depends greatly on the compiler's ability to constant-fold switch statements over these ordering values into a single instruction. This can be fragile -- especially in unoptimized builds. We think [constraining these arguments to compile-time constants](#restricting-ordering-arguments-to-compile-time-constants) strikes a good balance between readability and performance, but it's instructive to look at some of the approaches we considered before settling on this choice. + +#### Encode Orderings in Method Names + +One obvious idea is to put the ordering values directly in the method name for every atomic operation. This would be easy to implement but it leads to practically unusable API names. Consider the two-ordering compare/exchange variant below: + +```swift +flag.sequentiallyConsistentButAcquiringAndReleasingOnFailureCompareExchange( + expected: 0, + desired: 1 +) +``` + +We could find shorter names for the orderings (`Serialized`, `Barrier` etc.), but ultimately the problem is that this approach tries to cram too much information into the method name, and the resulting multitude of similar-but-not-exactly-the-same methods become an ill-structured mess. + +#### Orderings As Generic Type Parameters + +A second idea is model the orderings as generic type parameters on the atomic types themselves. + +```swift +struct Atomic { + ... +} + +let counter = Atomic(0) +counter.wrappingAdd(1) +``` + +This simplifies the typical case where all operations on a certain atomic value use the same "level" of ordering (relaxed, acquire/release, or sequentially consistent). However, there are considerable drawbacks: + +* This design puts the ordering specification far away from the actual operations -- obfuscating their meaning. +* It makes it a lot more difficult to use custom orderings for specific operations (like the speculative relaxed load in the `wrappingAdd` example in the section on [Atomic Operations](#atomic-operations) above). +* We wouldn't be able to provide a default value for a generic type parameter. +* Finally, there is also the risk of unspecialized generics interfering with runtime performance. + +#### Ordering Views + +The most promising alternative idea to represent memory orderings was to model them like `String`'s encoding views: + +```swift +let counter = Atomic(0) + +counter.relaxed.wrappingAdd(1) + +let current = counter.acquiring.load() +``` + +There are some things that we really like about this "ordering view" approach: + +- It eliminates the need to ever switch over orderings, preventing any and all constant folding issues. +- It makes it obvious that memory orderings are supposed to be compile-time parameters. +- The syntax is arguably more attractive. + +However, we ultimately decided against going down this route, for the following reasons: + + - **Composability.** Such ordering views are unwieldy for the variant of `compareExchange` that takes separate success/failure orderings. Ordering views don't nest very well at all: + + ```swift + counter.acquiringAndReleasing.butAcquiringOnFailure.compareExchange(...) + ``` + + - **API surface area and complexity.** Ordering views act like a multiplier for API entry points. In our prototype implementation, introducing ordering views increased the API surface area of atomics by 3×: we went from 6 public structs with 53 public methods to 27 structs with 175 methods. While clever use of protocols and generics could reduce this factor, the increased complexity seems undesirable. (E.g., generic ordering views would reintroduce potential performance problems in the form of unspecialized generics.) + + API surface area is not necessarily the most important statistic, but public methods do have some cost. (In e.g. the size of the stdlib module, API documentation etc.) + + - **Unintuitive syntax.** While the syntax is indeed superficially attractive, it feels backward to put the memory ordering *before* the actual operation. While memory orderings are important, I suspect most people would consider them secondary to the operations themselves. + + - **Limited Reuse.** Implementing ordering views takes a rather large amount of (error-prone) boilerplate-heavy code that is not directly reusable. Every new atomic type would need to implement a new set of ordering views, tailor-fit to its own use-case. + +#### Memory Orderings as Overloads + +Another promising alternative was the idea to model each ordering as a separate type and have overloads for the various atomic operations. + +```swift +struct AtomicMemoryOrdering { + struct Relaxed { + static var relaxed: Self { get } + } + + struct Acquiring { + static var acquiring: Self { get } + } + + ... +} + +extension Atomic where Value.AtomicRepresentation == {U}IntNN.AtomicRepresentation { + func load(ordering: AtomicMemoryOrdering.Relaxed) -> Value {...} + func load(ordering: AtomicMemoryOrdering.Acquiring) -> Value {...} + ... +} +``` + +This approach shares a lot of the same benefits of views, but the biggest reason for this alternative was the fact that the switch statement problem we described earlier just doesn't exist anymore. There is no switch statement! The overload always gets resolved to a single atomic operation + ordering + storage meaning there's no question about what to compile the operation down to. However, this is just another type of flavor of views in that the API surface explodes especially with double ordering operations. + +There are 5 storage types and we define the primitive atomic operations on extensions of all of these. For the constant expression case for single ordering operations that's `5 (storage) * 1 (ordering) = 5` number of overloads and `5 (storage) * 1 (update ordering) * 1 (load ordering) = 5` for the double ordering case. The overload solution is now dependent on the number of orderings supported for a specific operation. So for single ordering loads it's `5 (storage) * 3 (orderings) = 15` different load orderings and for the double ordering compare and exchange it's `5 (storage) * 5 (update orderings) * 3 (load orderings) = 75` overloads. + +| | Overloads | Constant Expressions | +| ----------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| Overload Resolution | Very bad | Not so bad | +| API Documentation | Very bad (but can be fixed!) | Not so bad (but can be fixed!) | +| Custom Atomic Operations | Requires users to define multiple overloads for their operations. | Allows users to define a single entrypoint that takes a constant ordering and passes that to the primitive atomic operations. | +| Back Deployable New Orderings | Almost impossible unless we defined the ordering types in C because types in Swift must come with availability. | Can easily be done because the orderings are static property getters that we can back deploy. | + +The same argument for views creating a very vast API surface can be said about the overloads which helped us determine that the constant expression approach is still superior. + +### Directly bring over `swift-atomics`'s API + +The `swift-atomics` package has many years of experience using their APIs to interface with atomic values and it would be beneficial to simply bring over the same API. However, once we designed the general purpose `Atomic` type, we noticied a few deficiencies with the atomic protocol hierarchy that made using this type awkward for users. We've redesigned these protocols to make using the atomic type easier to use and easier to reason about. + +While there are some API differences between this proposal and the package, most of the atomic operations are the same and should feel very familiar to those who have used the package before. We don't plan on drastically renaming any core atomic operation because we believe `swift-atomics` already got those names correct. + +### A different name for `WordPair` + +Previous revisions of this proposal named this type `DoubleWord`. This is a good name and is in fact the name used in the `swift-atomics` package. We felt the prefix `Double*` could cause confusion with the pre-existing type in the standard library `Double`. The name `WordPair` has a couple of advantages: + +1. Code completion. Because this name starts with an less common letter in the English alphabet, the likelihood of seeing this type at the top level in code completion is very unlikely and not generally a type used for newer programmers of Swift. +2. Directly conveys the semantic meaning of the type. This type is not semantically equivalent to something like `{U}Int128` (on 64 bit platforms). While although its layout shares the same size, the meaning we want to drive home with this type is quite simply that it's a pair of `UInt` words. If and when the standard library proposes a `{U}Int128` type, that will add a conformance to `AtomicRepresentable` on 64 bit platforms who support double-words as well. That itself wouldn't deprecate uses of `WordPair` however, because it's much easier to grab both words independently with `WordPair` as well as being a portable name for such semantics on both 32 bit and 64 bit platforms. + +### A different name for the `Synchronization` module + +In its [notes returning the initial version of this proposal for revision](https://forums.swift.org/t/returned-for-revision-se-0410-atomics/68522), the Swift Language Steering Group suggested the strawman name `Atomics` as a for this module. I think this name is far too restrictive because it prevents other similar low-level concurrency primitives or somewhat related features like volatile loads/stores from sharing a module. It would also be extremely source breaking for folks that upgrade their Swift SDK to a version that may include this proposed new module while depending on the existing [swift-atomics](https://github.com/apple/swift-atomics) whose module is also named `Atomics`. + +We shouldn't be afraid of conflicting module names causing spurious source breaks when introducing a new module to the standard libraries; however, in this case, the direct precursor is prominently using this name, and reusing the same module name would cause significant breakage. We expect this package will need to remain in active use for a number of years, as it will be able to provide reimplementations of the constructs proposed here without the ABI availability constraints that come with Standard Library additions. + +## References + +[Clarify the Swift memory consistency model (SE-0282)]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md +**\[Clarify the Swift memory consistency model (SE-0282)]** Karoy Lorenty. "Clarify the Swift memory consistency model."*Swift Evolution Proposal*, 2020. https://github.com/swiftlang/swift-evolution/blob/main/proposals/0282-atomics.md + +[C++17]: https://isocpp.org/std/the-standard +**\[C++17]** ISO/IEC. *ISO International Standard ISO/IEC 14882:2017(E) – Programming Language C++.* 2017. + https://isocpp.org/std/the-standard + +[Boehm 2008]: https://doi.org/10.1145/1375581.1375591 +**\[Boehm 2008]** Hans-J. Boehm, Sarita V. Adve. "Foundations of the C++ Concurrency Memory Model." In *PLDI '08: Proc. of the 29th ACM SIGPLAN Conf. on Programming Language Design and Implementation*, pages 68–78, June 2008. + https://doi.org/10.1145/1375581.1375591 + +[N2153]: http://wg21.link/N2153 +**\[N2153]** Raúl Silvera, Michael Wong, Paul McKenney, Bob Blainey. *A simple and efficient memory model for weakly-ordered architectures.* WG21/N2153, January 12, 2007. http://wg21.link/N2153 + +[P0020]: http://wg21.link/P0020 +**\[P0020]** H. Carter Edwards, Hans Boehm, Olivier Giroux, JF Bastien, James Reus. *Floating Point Atomic.* WG21/P0020r6, November 10, 2017. http://wg21.link/P0020 + +[P0418]: http://wg21.link/P0418 +**\[P0418]** JF Bastien, Hans-J. Boehm. *Fail or succeed: there is no atomic lattice.* WG21/P0417r2, November 9, 2016. http://wg21.link/P0418 + +[P0690]: http://wg21.link/P0690 +**\[P0690]** JF Bastien, Billy Robert O'Neal III, Andrew Hunter. *Tearable Atomics.* WG21/P0690, February 10, 2018. http://wg21.link/P0690 + +[P0735]: http://wg21.link/P0735 +**\[P0735]**: Will Deacon, Jade Alglave. *Interaction of `memory_order_consume` with release sequences.* WG21/P0735r1, June 17, 2019. http://wg21.link/P0735 + +[P0750]: http://wg21.link/P0750 +**\[P0750]** JF Bastien, Paul E. McKinney. *Consume*. WG21/P0750, February 11, 2018. http://wg21.link/P0750 + +⚛︎︎ + + + + + + + + diff --git a/proposals/0411-isolated-default-values.md b/proposals/0411-isolated-default-values.md new file mode 100644 index 0000000000..02280a41be --- /dev/null +++ b/proposals/0411-isolated-default-values.md @@ -0,0 +1,330 @@ +# Isolated default value expressions + +* Proposal: [SE-0411](0411-isolated-default-values.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 5.10)** +* Bug: *if applicable* [apple/swift#58177](https://github.com/apple/swift/issues/58177) +* Implementation: [apple/swift#68794](https://github.com/apple/swift/pull/68794) +* Upcoming Feature Flag: `IsolatedDefaultValues` +* Review: ([pitch](https://forums.swift.org/t/pitch-isolated-default-value-expressions/67714)), ([review](https://forums.swift.org/t/se-0411/68065)), ([acceptance](https://forums.swift.org/t/accepted-se-0411-isolated-default-value-expressions/68806)) + +## Introduction + +Default value expressions are permitted for default arguments and default stored property values. There are several issues with the current actor isolation rules for default value expressions: the rules for stored properties admit data races, the rules for default argument values are overly restrictive, and the rules between the different places you can use default value expressions are inconsistent with each other, making the actor isolation model harder to understand. This proposal unifies the actor isolation rules for default value expressions, eliminates data races, and improves expressivity by safely allowing isolation for default values. + +## Motivation + +The current actor isolation rules for initial values of stored properties admit data races. For example, the following code is currently valid: + +```swift +@MainActor func requiresMainActor() -> Int { ... } +@AnotherActor func requiresAnotherActor() -> Int { ... } + +class C { + @MainActor var x1 = requiresMainActor() + @AnotherActor var x2 = requiresAnotherActor() + + nonisolated init() {} // okay??? +} +``` + +The above code allows any context to initialize an instance of `C()` through a synchronous, nonisolated `init`. The initializer synchronously calls both `requiresMainActor()` and `requiresAnotherActor()`, which are `@MainActor`-isolated and `@AnotherActor`-isolated, respectively. This violates actor isolation checking because `requiresMainActor()` and `requiresAnotherActor()` may run concurrently with other code on their respective global actors. + +The current actor isolation rules for default argument values do not admit data races, but default argument values are always `nonisolated` which is overly restrictive. This rule prohibits programmers from making `@MainActor`-isolated calls in default argument values of `@MainActor`-isolated functions. For example, the following code is not valid even though it is perfectly safe: + +```swift +@MainActor class C {} + +@MainActor func f(c: C = C()) {} // error: Call to main actor-isolated initializer 'init()' in a synchronous nonisolated context + +@MainActor func useFromMainActor() { + f() +} +``` + +## Proposed solution + +I propose allowing default value expressions to have the same isolation as the enclosing function or the corresponding stored property. As usual, if the caller is not already in the isolation domain of the callee, then the call must be made asynchronously and must be explicitly marked with `await`. For isolated default values of stored properties, the implicit initialization only happens in the body of an `init` with the same isolation. + +These rules make the stored property example above invalid at the `nonisolated` initializer: + +```swift +@MainActor func requiresMainActor() -> Int { ... } +@AnotherActor func requiresAnotherActor() -> Int { ... } + +class C { + @MainActor var x1 = requiresMainActor() + @AnotherActor var x2 = requiresAnotherActor() + + nonisolated init() {} // error: 'self.x1' and 'self.x2' are not initialized +} +``` + +Calling `requiresMainActor()` and `requiresAnotherActor()` explicitly with `await` resolves the issue: + +```swift +class C { + @MainActor var x1 = requiresMainActor() + @AnotherActor var x2 = requiresAnotherActor() + + nonisolated init() async { + self.x1 = await requiresMainActor() + self.x2 = await requiresAnotherActor() + } +} +``` + +This rule also makes the default argument example above valid, because the default argument and the enclosing function are both `@MainActor`-isolated. + +## Detailed design + +### Inference of default value isolation requirements + +Default value expressions are always evaluated in a synchronous context, so all calls that are made during the evaluation of the expression must also be synchronous. If the callee is isolated, then the default value expression must already be in the same isolation domain in order to make the call synchronously. So, for a given default value expression, the inferred isolation is the required isolation of its subexpressions. For example: + +```swift +@MainActor func requiresMainActor() -> Int { ... } + +@MainActor func useDefault(value: Int = requiresMainActor()) { ... } +``` + +In the above code, the default argument for `value` requires `@MainActor` isolation, because the default value calls `requiresMainActor()` which is isolated to `@MainActor`. + +#### Closures + +Evaluating a closure literal itself can happen in any isolation domain; the actor isolation of a closure only applies when calling the closure. An actor-isolated closure enables the closure body to make calls within that isolation domain synchronously. For a closure literal in a default value expression that is not explicitly annotated with actor isolation, the inferred isolation of the closure is the union of the isolation of all callees in the closure body for synchronous calls. For example: + +```swift +@MainActor func requiresMainActor() -> Int { ... } + +@MainActor func useDefaultClosure( + closure: () -> Void = { + requiresMainActor() + } +) {} +``` + +The above `useDefaultClosure` function has a default argument value that is a closure literal. The closure body calls a `@MainActor`-isolated function synchronously, therefore the closure itself must be `@MainActor` isolated. + +Note that the only way for a closure literal in a default argument to be isolated to an actor instance is for the isolation to be written explicitly with an isolated parameter. The inference algorithm will never determine the isolation to be an actor instance based on the following two properties: + +1. To be isolated to an actor instance, a closure must either have its own (explicit) isolated parameter or capture an isolated parameter from its enclosing context. +2. Closure literals in default arguments cannot capture values. + +#### Restrictions + +* If a function or type itself has actor isolation, the required isolation of its default value expressions must share the same actor isolation. For example, a `@MainActor`-isolated function cannot have a default argument that is isolated to `@AnotherActor`. Note that it's always okay to mix isolated default values with `nonisolated` default values. +* If a function or type is `nonisolated`, then the required isolation of its default value expressions must be `nonisolated`. + +### Enforcing default value isolation requirements + +#### Default argument values + +Isolation requirements for default argument expressions are enforced at the caller. If the caller is not in the required isolation domain, the default arguments must be evaluated asynchronously and explicitly marked with `await`. For example: + +```swift +@MainActor func requiresMainActor() -> Int { ... } + +@MainActor func useDefault(value: Int = requiresMainActor()) { ... } + +@MainActor func mainActorCaller() { + useDefault() // okay +} + +func nonisolatedCaller() async { + await useDefault() // okay + + useDefault() // error: call is implicitly async and must be marked with 'await' +} +``` + +In the above example, `useDefault` has default arguments that are isolated to `@MainActor`. The default arguments can be evaluated synchronously from a `@MainActor`-isolated caller, but the call must be marked with `await` from outside the `@MainActor`. Note that these rules already fall out of the semantics of calling actor isolated functions. + +#### Argument evaluation + +For a given call, argument evaluation happens in the following order: + +1. Left-to-right evaluation of explicit r-value arguments +2. Left-to-right evaluation of default arguments and formal access arguments + +For example: + +```swift +nonisolated var defaultVal: Int { print("defaultVal"); return 0 } +nonisolated var explicitVal: Int { print("explicitVal"); return 0 } +nonisolated var explicitFormalVal: Int { + get { print("explicitFormalVal"); return 0 } + set {} +} + +func evaluate(x: Int = defaultVal, y: Int = defaultVal, z: inout Int) {} + +evaluate(y: explicitVal, z: &explicitFormalVal) +``` + +The output of the above program is + +``` +explicitVal +defaultVal +explicitFormalVal +``` + +Unlike the explicit argument list, isolated default arguments must be evaluated in the isolation domain of the callee. As such, if any of the argument values require the isolation of the callee, argument evaluation happens in the following order: + +1. Left-to-right evaluation of explicit r-value arguments +2. Left-to-right evaluation of formal access arguments +3. Hop to the callee's isolation domain +4. Left-to-right evaluation of default arguments + +For example: + +```swift +@MainActor var defaultVal: Int { print("defaultVal"); return 0 } +nonisolated var explicitVal: Int { print("explicitVal"); return 0 } +nonisolated var explicitFormalVal: Int { + get { print("explicitFormalVal"); return 0 } + set {} +} + +@MainActor func evaluate(x: Int = defaultVal, y: Int = defaultVal, z: inout Int) {} + +nonisolated func nonisolatedCaller() { + await evaluate(y: explicitVal, z: &explicitFormalVal) +} +``` + +The output of calling `nonisolatedCaller()` is: + +``` +explicitVal +explicitFormalVal +defaultVal +``` + +#### Stored property initial values + +Isolation requirements for default initializer expressions for stored properties apply in the body of initializers. If an `init` does not match the isolation of the initializer expression, the initialization of that stored property is not emitted at the beginning of the `init`. Instead, the stored property must be explicitly initialized in the body of the `init`. For example: + +```swift +@MainActor func requiresMainActor() -> Int { ... } +@AnotherActor func requiresAnotherActor() -> Int { ... } + +class C { + @MainActor var x1: Int = requiresMainActor() + @AnotherActor var x2: Int = requiresAnotherActor() + + nonisolated init() {} // error: 'self.x1' and 'self.x2' aren't initialized + + nonisolated init(x1: Int, x2: Int) { // okay + self.x1 = x1 + self.x2 = x2 + } + + @MainActor init(x2: Int) { // okay + // 'self.x1' gets assigned to the default value 'requiresMainActor()' + self.x2 = x2 + } +} +``` + +In the above example, the no-parameter `nonisolated init()` is invalid, because it does not initialize `self.x1` and `self.x2`. Because the default initializer expressions require different actor isolation, those values are not used in the `nonisolated` initializer. The other two initializers are valid. + +### Stored property isolation in initializers + +#### Initializing isolated stored properties from across isolation boundaries + +It is invalid to initialize an isolated stored property from across isolation boundaries: + +```swift +class NonSendable {} + +class C { + @MainActor var ns: NonSendable + + init(ns: NonSendable) { + self.ns = ns // error: passing non-Sendable value 'ns' to a MainActor-isolated context. + } +} +``` + +The above code violates `Sendable` guarantees because the initialization of the `MainActor`-isolated property `self.ns` from a `nonisolated` context is effectively passing a non-`Sendable` value across isolation boundaries. To prevent this class of data races, this proposal requires that any `init` that initializes a global actor isolated stored property must also be isolated to that global actor. + +Note that this rule is not specific to default values, but it's necessary to specify the behavior of default values in compiler-synthesized initializers. + +#### Default value isolation in synthesized initializers + +For structs, default initializer expressions for stored properties are used as default argument values to the compiler-generated memberwise initializer. For structs and classes that have a compiler-generated no-parameter initializer, the default initializer expressions are also used in the synthesized `init()` body. + +If any of the type's stored properties with non-`Sendable` type are actor isolated, or if any of the isolated default initializer expressions are actor isolated, then the compiler-synthesized initializer(s) must also be actor isolated. For example: + +```swift +class NonSendable {} + +@MainActor struct MyModel { + // @MainActor inferred from annotation on enclosing struct + var value: NonSendable = .init() + + /* compiler-synthesized memberwise init is @MainActor + @MainActor + init(value: NonSendable = .init()) { + self.value = value + } + */ +} +``` + +If none of the type's stored properties are non-`Sendable` and actor isolated, and none of the default initializer expressions require actor isolation, then the compiler-synthesized initializer is `nonisolated`. For example: + +```swift +@MainActor struct MyView { + // @MainActor inferred from annotation on enclosing struct + var value: Int = 0 + + /* compiler-synthesized 'init's are 'nonisolated' + + nonisolated init() { + self.value = 0 + } + + nonisolated init(value: Int = 0) { + self.value = value + } + */ + + // @MainActor inferred from the annotation on the enclosing struct + var body: some View { ... } +} +``` + +These rules ensure that the default value expressions in compiler-synthesized initializers are always valid. If a default value expression requires actor isolation, then the enclosing initializer always shares the same actor isolation. It is an error for two different default values to require different actor isolation, because it's not possible to ever use those default values. Initializing an instance of a type using two different initial value expressions with different actor isolation must be done in an `async` initializer, with suspension points explicitly marked with `await`. + +## Source compatibility + +The actor isolation rules for initial values of stored properties are stricter than what is currently accepted in Swift 5 mode in order to eliminate data races. The isolation rules for stored properties will be staged in under the `IsolatedDefaultValues` upcoming feature identifier. + +## ABI compatibility + +This is a change to actor isolation checking with no impact on ABI. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. + +## Alternatives considered + +### Remove isolation from all default initializer expressions + +SE-0327 originally proposed changing default initializer expressions for stored properties to always be `nonisolated`, matching the current default argument value rules. However, this change was implemented and later reverted because it impacted a lot of code that followed a common pattern: a `@MainActor`-isolated type with stored properties that have default values that call the initializers of other `@MainActor`-isolated types. In some cases, it's possible to make the initializer of a `@MainActor` type `nonisolated`, but many of these cases do access `@MainActor`-isolated properties and functions in the body of the initializer. + +## Acknowledgments + +Thank you to Kavon Farvardin for implementing the default initializer expression rules originally proposed by SE-0327 and discovering the usability issues outlined in this proposal. Thank you to John McCall for the observation that memberwise initializers can and should be `nonisolated` when possible. + +## Revision history + +* Changes from the first pitch + * Require that isolated default arguments share the same isolation as their enclosing function or type. + * Specify the semantic restrictions on initializing actor isolated properties from across isolation boundaries. + * Enable using isolated default arguments from across isolation boundaries by changing the argument evaluation between formal access and default arguments. diff --git a/proposals/0412-strict-concurrency-for-global-variables.md b/proposals/0412-strict-concurrency-for-global-variables.md new file mode 100644 index 0000000000..d045827598 --- /dev/null +++ b/proposals/0412-strict-concurrency-for-global-variables.md @@ -0,0 +1,109 @@ +# Strict concurrency for global variables + +* Proposal: [SE-0412](0412-strict-concurrency-for-global-variables.md) +* Authors: [John McCall](https://github.com/rjmccall), [Sophia Poirier](https://github.com/sophiapoirier) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 5.10)** +* Upcoming Feature Flag: `GlobalConcurrency` (Enabled in Swift 6 language mode) +* Implementation: On `main` gated behind `-enable-experimental-feature GlobalConcurrency` +* Previous Proposals: [SE-0302](0302-concurrent-value-and-concurrent-closures.md), [SE-0306](0306-actors.md), [SE-0316](0316-global-actors.md), [SE-0337](0337-support-incremental-migration-to-concurrency-checking.md), [SE-0343](0343-top-level-concurrency.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-strict-concurrency-for-global-variables/66908)), ([review](https://forums.swift.org/t/se-0412-strict-concurrency-for-global-variables/68352)), ([acceptance](https://forums.swift.org/t/accepted-se-0412-strict-concurrency-for-global-variables/69004)) + +## Introduction + +This proposal defines options for the usage of global variables free of data races. Within this proposal, global variables encompass any storage of static duration: `let`s and stored `var`s that are either declared at global scope or as static member variables. + +## Motivation + +Global state poses a challenge within concurrency because it is memory that can be accessed from any program context. Global variables are of particular concern in data isolation checking because they defy other attempts to enforce isolation. Variables that are local and un-captured can only be accessed from that local context, which implicitly isolates them. Stored properties of value types are already isolated by the exclusivity rules. Stored properties of reference types can be isolated by isolating their containing object with sendability enforcement or using actor restrictions. But global variables can be accessed from anywhere, so these tools do not work. + +```swift +var value = 1 + +func f() { + value = 2 // warning: reference to var 'value' is not concurrency-safe because it involves shared mutable state +} +``` + +## Proposed solution + +Under strict concurrency checking, require every global variable to either be isolated to a global actor or be both: + +1. immutable +2. of `Sendable` type + +Global variables that are immutable and `Sendable` can be safely accessed from any context, and otherwise, isolation is required. + +Top-level global variables are already implicitly isolated to `@MainActor` and therefore automatically meet these proposed requirements. + +## Detailed design + +These requirements can be enforced in the type checker at declaration time. + +Although global variables are lazily initialized, the initialization is already guaranteed to be thread-safe and therefore requires no further specification under strict concurrency checking. + +There may be need in some circumstances to opt out of static checking to enable the developer to rely upon their own data isolation management, such as with an associated global lock serializing data access. The attribute `nonisolated(unsafe)` can be used to annotate the global variable (or any form of storage). Though this will disable static checking of data isolation for the global variable, note that without correct implementation of a synchronization mechanism to achieve data isolation, dynamic run-time analysis from exclusivity enforcement or tools such as Thread Sanitizer could still identify failures. + +```swift +nonisolated(unsafe) var global: String +``` + +The same annotation on a local variable can be used to suppress a static diagnostic being generated when the local variable is referenced asynchronously: + +```swift +func f() async { + nonisolated(unsafe) var value = 1 + let task = Task { + value = 2 + return value + } + print(await task.value) +} +``` + +Because `nonisolated` is a contextual keyword, there is ambiguity when using `nonisolated(unsafe)` on a separate line immediately preceding a top-level variable declaration in script mode as it could also be the invocation of a function named `nonisolated` with argument `unsafe`. This ambiguity can be resolved by favoring the interpretation of `nonisolated` as a keyword if it has a single unlabeled argument of `unsafe` and precedes a variable declaration. + +Importing a module via `@preconcurrency import` suppresses any potential errors resulting from data isolation checking of imported global variables that lack explicit concurrency annotations. Any use of a `@preconcurrency import`ed concurrency-unsafe global variable will produce a warning at the use site. + +Note that imports from other languages are implicitly `@preconcurrency`. There remain tools for enforcing safety for imported global variables from other languages, such as isolating to a global actor using for example `__attribute__((swift_attr("@MainActor")))` in C or Obj-C, or wrapping access within a safer API that declares the correct isolation or locks appropriately. + +## Source compatibility + +Due to the addition of restrictions, this could require changes to some type declaration when strict concurrency checking is in use. Such source changes however would still be backwards compatible to any version of Swift with concurrency features. + +Resolving the ambiguity of `nonisolated(unsafe)` in a top-level variable declaration would break existing top-level script code that invokes a function named `nonisolated` with a single unlabeled argument `unsafe` when immediately preceding a variable declaration by eliminating that function invocation in favor of its interpretation as an isolation specification. + +## ABI compatibility + +This proposal does not add or affect ABI in and of itself, however type declaration changes that it may instigate upon an adopting project could impact that project's ABI. + +## Implications on adoption + +Some global variable types may need to be modified in a project adopting strict concurrency checking. + +## Alternatives considered + +For isolation, rather than requiring a global actor, we could implicitly lock around accesses of the variable. While providing memory safety, this can be problematic for thread safety, because developers can easily write non-atomic use patterns: + +```swift +// value of global may concurrently change between +// the read for the multiplication expression +// and the write for the assignment +global = global * 2 +``` + +Though we could consider implicit locking if we needed to do something source-compatible in old language modes, generally our approach has just been to say that old language modes are concurrency-unsafe. It also would not work for non-`Sendable` types unless we force the value to remain isolated while accessing it. We potentially could accomplish that with the proposed [Safely sending non-Sendable values across isolation domains](https://forums.swift.org/t/pitch-safely-sending-non-sendable-values-across-isolation-domains/66566) feature, but that is probably too advanced a feature to push as a solution for such a basic problem. + +We could default all global variables that require isolation to `@MainActor`. It is arguably better to make developers think about the choice (e.g. perhaps it should just be a `let` constant). + +Access control is theoretically useful here: for example, we could know that a global variable is concurrency-safe because it is private to a file and all of the accesses in that file are from within a single global actor context, or because it is never mutated. That is a more global analysis than we usually want to do in the compiler, though; we would have to check everything in the context, and then it might be hard for the developer to understand why it works. + +## Future directions + +We do not necessarily need to require isolation to a global actor to be _explicit_; there is room for inferring the right global actor. A global mutable variable of global-actor-constrained type could be inferred to be constrained to that global actor (though unnecessary if the variable is immutable, since global-actor-constrained class types are `Sendable`). + +## Revision history + +Post-review changes: +* removed implicit `nonisolated(unsafe)` import of C global variables in favor of `@preconcurrency import` as the mechanism to suppress static isolation checking of global variables +* clarifed `nonisolated(unsafe)` for local variables diff --git a/proposals/0413-typed-throws.md b/proposals/0413-typed-throws.md new file mode 100644 index 0000000000..9ee9502297 --- /dev/null +++ b/proposals/0413-typed-throws.md @@ -0,0 +1,1401 @@ +# Typed throws + +* Proposal: [SE-0413](0413-typed-throws.md) +* Authors: [Jorge Revuelta (@minuscorp)](https://github.com/minuscorp), [Torsten Lehmann](https://github.com/torstenlehmann), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.0)** +* Review: [latest pitch](https://forums.swift.org/t/pitch-n-1-typed-throws/67496), [review](https://forums.swift.org/t/se-0413-typed-throws/68507), [acceptance](https://forums.swift.org/t/accepted-se-0413-typed-throws/69099) + +## Introduction + +Swift's error handling model allows functions and closures marked `throws` to note that they can exit by throwing an error. The error values themselves are always type-erased to `any Error`. This approach encourages errors to be handled generically, and remains a good default for most code. However, there are some places where the type erasure is unfortunate, because it doesn't allow for more precise error typing in narrow places where it is possible and desirable to handle all errors, or where the costs of type erasure are prohibitive. + +This proposal introduces the ability to specify that functions and closures only throw errors of a particular concrete type. + +> Note: the [originally accepted version](https://github.com/swiftlang/swift-evolution/blob/821970ae986219f88eb3f950ed787a55ce31d512/proposals/0413-typed-throws.md) of this proposal included type inference changes intended for Swift 6.0 that were behind the upcoming feature flag `FullTypedThrows`. These type inference changes did not get implemented in Swift 6.0, and have therefore been removed from this proposal and placed into "Future Directions" so they can be revisited once implemented. + +## Table of Contents + +[Typed throws](#typed-throws) + + * [Introduction](#introduction) + * [Motivation](#motivation) + * [Communicates less error information than Result or Task](#communicates-less-error-information-than-result-or-task) + * [Inability to interconvert throws with Result or Task](#inability-to-interconvert-throws-with-result-or-task) + * [Approach 1: Chaining Results](#approach-1-chaining-results) + * [Approach 2: Unwrap/switch/wrap on every chaining/mapping point](#approach-2-unwrapswitchwrap-on-every-chainingmapping-point) + * [Existential error types incur overhead](#existential-error-types-incur-overhead) + * [Proposed solution](#proposed-solution) + * [Specific types in catch blocks](#specific-types-in-catch-blocks) + * [Throwing any Error or Never](#throwing-any-error-or-never) + * [An alternative to rethrows](#an-alternative-to-rethrows) + * [When to use typed throws](#when-to-use-typed-throws) + * [Detailed design](#detailed-design) + * [Syntax adjustments](#syntax-adjustments) + * [Function type](#function-type) + * [Closure expression](#closure-expression) + * [Function, initializer, and accessor declarations](#function-initializer-and-accessor-declarations) + * [Examples](#examples) + * [Throwing and catching with typed throws](#throwing-and-catching-with-typed-throws) + * [Throwing within a function that declares a typed error](#throwing-within-a-function-that-declares-a-typed-error) + * [Catching typed thrown errors](#catching-typed-thrown-errors) + * [rethrows](#rethrows) + * [Opaque thrown error types](#opaque-thrown-error-types) + * [async let](#async-let) + * [Subtyping rules](#subtyping-rules) + * [Function conversions](#function-conversions) + * [Protocol conformance](#protocol-conformance) + * [Override checking](#override-checking) + * [Type inference](#type-inference) + * [Associated type inference](#associated-type-inference) + * [Standard library adoption](#standard-library-adoption) + * [Converting between throws and Result](#converting-between-throws-and-result) + * [Standard library operations that rethrow](#standard-library-operations-that-rethrow) + * [Source compatibility](#source-compatibility) + * [Effect on API resilience](#effect-on-api-resilience) + * [Effect on ABI stability](#effect-on-abi-stability) + * [Future directions](#future-directions) + * [Closure thrown type inference](#closure-thrown-type-inference) + * [Standard library operations that rethrow](#standard-library-operations-that-rethrow) + * [Concurrency library adoption](#concurrency-library-adoption) + * [Specific thrown error types for distributed actors](#specific-thrown-error-types-for-distributed-actors) + * [Alternatives considered](#alternatives-considered) + * [Thrown error type syntax](#thrown-error-type-syntax) + * [Multiple thrown error types](#multiple-thrown-error-types) + * [Treat all uninhabited thrown error types as nonthrowing](#treat-all-uninhabited-thrown-error-types-as-nonthrowing) + * [Typed rethrows](#typed-rethrows) + * [Revision history](#revision-history) + +## Motivation + +Swift is known for being explicit about semantics and using types to communicate constraints that apply to specific APIs. From that perspective, the fact that all thrown errors are of type `any Error` feels like an outlier. However, it reflects the view laid out in the original [error handling rationale](https://github.com/apple/swift/blob/main/docs/ErrorHandlingRationale.md) that errors are generally propagated and rendered, but rarely handled exhaustively, and are prone to changing over time in a way that types are not. + +The desire to provide specific thrown error types has come up repeatedly on the Swift forums. Here are just a few of the forum threads calling for some form of typed throws: + +* [[Pitch N+1] Typed throws](https://forums.swift.org/t/pitch-n-1-typed-throws/67496) +* [Typed throw functions](https://forums.swift.org/t/typed-throw-functions/38860) +* [Status check: typed throws](https://forums.swift.org/t/status-check-typed-throws/66637) +* [Precise error typing in Swift](https://forums.swift.org/t/precise-error-typing-in-swift/52045) +* [Typed throws](https://forums.swift.org/t/typed-throws/6501) +* [[Pitch\] Typed throws](https://forums.swift.org/t/pitch-typed-throws/5233) +* [Type-annotated throws](https://forums.swift.org/t/type-annotated-throws/3875) +* [Proposal: Allow Type Annotations on Throws](https://forums.swift.org/t/proposal-allow-type-annotations-on-throws/1149) +* [Proposal: Allow Type Annotations on Throws](https://forums.swift.org/t/proposal-allow-type-annotations-on-throws/623) +* [Proposal: Typed throws](https://forums.swift.org/t/proposal-typed-throws/268) +* [Type Inferencing For Error Handling (try catch blocks)](https://forums.swift.org/t/type-inferencing-for-error-handling-try-catch-blocks/117) + +In a sense, Swift started down the path toward typed throws with the introduction of the [`Result`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0235-add-result.md) type in the standard library, which captured a specific thrown error type in its `Failure` parameter. That pattern was replicated in the [`Task` type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md) and other concurrency APIs. The loss of information between types like `Result` and `Task` and the language's error-handling system provides partial motivation for the introduction of typed throws, and is discussed further below. + +Typed throws also provides benefits in places where clients need to exhaustively handle errors. For this to make sense, the set of potential failure conditions must be relatively fixed, either because they come from the same module or package as the clients, or because they come from a library that is effectively standalone and unlikely to evolve to (e.g.) pass through an error from another lower-level library. Typed throws also provides benefits in generic code that will propagate errors from its arguments, but never generate errors itself, as a more flexible alternative to the existing `rethrows`. Finally, typed throws also open up the potential for more efficient code, because they avoid the overhead associated with existential types (`any Error`). + +Even with the introduction of typed throws into Swift, the existing (untyped) `throws` remains the better default error-handling mechanism for most Swift code. The section ["When to use typed throws"](#when-to-use-typed-throws) describes the circumstances in which typed throws should be used. + +### Communicates less error information than `Result` or `Task` + +Assume you have this Error type + +```swift +enum CatError: Error { + case sleeps + case sitsAtATree +} +``` + +Compare + +```swift +func callCat() -> Result +``` + +or + +```swift +func callFutureCat() -> Task +``` + +with + +```swift +func callCatOrThrow() throws -> Cat +``` + +`throws` communicates less information about why the cat is not about to come to you. + +### Inability to interconvert `throws` with `Result` or `Task` + +The fact that`throws` carries less information than `Result` or `Task` means that conversions to `throws` loses type information, which can only be recovered by explicit casting: + +```swift +func callAndFeedCat1() -> Result { + do { + return Result.success(try callCatOrThrow()) + } catch { + // won't compile, because error type guarantee is missing in the first place + return Result.failure(error) + } +} +``` + +```swift +func callAndFeedCat2() -> Result { + do { + return Result.success(try callCatOrThrow()) + } catch let error as CatError { + // compiles + return Result.failure(error) + } catch { + // won't compile, because exhaustiveness can't be checked by the compiler + // so what should we return here? + return Result.failure(error) + } +} +``` + +### `Result` is not the go to replacement for `throws` in imperative languages + +Using explicit errors with `Result` has major implications for a code base. Because the exception handling mechanism ("goto catch") is not built into the language (like `throws`), you need to do that on your own, mixing the exception handling mechanism with domain logic. + +#### Approach 1: Chaining Results + +If you use `Result` in a functional (i.e. monadic) way, you need extensive use of `map`, `flatMap` and similar operators. + +Example is taken from [Question/Idea: Improving explicit error handling in Swift (with enum operations) - Using Swift - Swift Forums](https://forums.swift.org/t/question-idea-improving-explicit-error-handling-in-swift-with-enum-operations/35335). + +```swift +struct SimpleError: Error { + let message: String +} + +struct User { + let firstName: String + let lastName: String +} + +func stringResultFromArray(_ array: [String], at index: Int, errorMessage: String) -> Result { + guard array.indices.contains(index) else { return Result.failure(SimpleError(message: errorMessage)) } + return Result.success(array[index]) +} + +func userResultFromStrings(strings: [String]) -> Result { + return stringResultFromArray(strings, at: 0, errorMessage: "Missing first name") + .flatMap { firstName in + stringResultFromArray(strings, at: 1, errorMessage: "Missing last name") + .flatMap { lastName in + return Result.success(User(firstName: firstName, lastName: lastName)) + } + } +} +``` + +That's the functional way of writing exceptions, but Swift does not provide enough functional constructs to handle that comfortably (compare with [Haskell/do notation](https://en.wikibooks.org/wiki/Haskell/do_notation)). + +#### Approach 2: Unwrap/switch/wrap on every chaining/mapping point + +We can also just unwrap every result by switching over it and wrapping the value or error into a result again. + +```swift +func userResultFromStrings(strings: [String]) -> Result { + let firstNameResult = stringResultFromArray(strings, at: 0, errorMessage: "Missing first name") + + switch firstNameResult { + case .success(let firstName): + let lastNameResult = stringResultFromArray(strings, at: 1, errorMessage: "Missing last name") + + switch lastNameResult { + case .success(let lastName): + return Result.success(User(firstName: firstName, lastName: lastName)) + case .failure(let simpleError): + return Result.failure(simpleError) + } + + case .failure(let simpleError): + return Result.failure(simpleError) + } +} +``` + +This is even more boilerplate than the first approach, because now we are writing the implementation of the `flatMap` operator over and over again. + +### Existential error types incur overhead + +Untyped errors have the existential type `any Error`, which incurs some [necessary overhead](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md), in code size, heap allocation overhead, and execution performance, due to the need to support values of unknown type. In constrained environments such as those supported by [Embedded Swift](https://forums.swift.org/t/embedded-swift/67057), existential types may not be permitted due to these overheads, making the existing untyped throws mechanism unusable in those environments. + + +## Proposed solution + +In general, we want to add the possibility of using `throws` with a single, specific error type. + +```swift +func callCat() throws(CatError) -> Cat { + if Int.random(in: 0..<24) < 20 { + throw .sleeps + } + // ... +} +``` + +The function can only throw instances of `CatError`. This provides contextual type information for all throw sites, so we can write `.sleeps` instead of the more verbose `CatError.sleeps` that's needed with untyped throws. Any attempt to throw any other kind of error out of the function will be an error: + +```swift +func callCatBadly() throws(CatError) -> Cat { + throw SimpleError(message: "sleeping") // error: SimpleError cannot be converted to CatError +} +``` + +Maintaining specific error types throughout a function is much easier than when using `Result`, because one can use `try` consistently: + +```swift +func stringFromArray(_ array: [String], at index: Int, errorMessage: String) throws(SimpleError) -> String { + guard array.indices.contains(index) else { throw SimpleError(message: errorMessage) } + return array[index] +} + +func userResultFromStrings(strings: [String]) throws(SimpleError) -> User { + let firstName = try stringFromArray(strings, at: 0, errorMessage: "Missing first name") + let lastName = try stringFromArray(strings, at: 1, errorMessage: "Missing last name") + return User(firstName: firstName, lastName: lastName) +} +``` + +The error handling mechanism is pushed aside and you can see the domain logic more clearly. + +### Specific types in catch blocks + +With typed throws, a throwing function contains the same information about the error type as `Result`, making it easier to convert between the two: + +```swift +func callAndFeedCat1() -> Result { + do { + return Result.success(try callCat()) + } catch { + // would compile now, because error is `CatError` + return Result.failure(error) + } +} +``` + +Note that the implicit `error` variable within the catch block is inferred to the concrete type `CatError`; there is no need for the existential `any Error`. + +When a `do` statement can throw errors with different concrete types, or involves any calls to functions using untyped throws, the `catch` block will receive a thrown error type of an `any Error` type: + +```swift +func callKids() throws(KidError) -> [Kid] { ... } + +do { + try callCat() + try callKids() +} catch { + // error has type 'any Error', as it does today +} +``` + +The caught error type for a `do..catch` statement will be inferred from the various throwing sites within the body of the `do` block. One can explicitly specify this type with a `throws` clause on ` do` block itself, i.e., + +```swift +do throws(CatError) { + if isDaylight && foodBowl.isEmpty { + throw .sleeps // equivalent to CatError.sleeps + } + try callCat() +} catch let myError { + // myError is of type CatError +} +``` + +When one needs to translate errors of one concrete type to another, use a `do...catch` block around each sequence of calls that produce the same kind of error : + +```swift +func firstNameResultFromArray(_ array: [String]) throws(FirstNameError) -> String { + guard array.indices.contains(0) else { throw FirstNameError() } + return array[0] +} + +func userResultFromStrings(strings: [String]) throws(SimpleError) -> User { + do { + let firstName = try firstNameResultFromArray(strings) + return User(firstName: firstName, lastName: "") + } catch { + // error is a `FirstNameError`, map it to a `SimpleError`. + throw SimpleError(message: "Missing first name") + } +} +``` + +### Throwing `any Error` or `Never` + +Typed throws generalizes over both untyped throws and non-throwing functions. A function specified with `any Error` as its thrown type: + +```swift +func throwsAnything() throws(any Error) { ... } +``` + +is equivalent to untyped throws: + +```swift +func throwsAnything() throws { ... } +``` + +Similarly, a function specified with `Never` as its thrown type: + +```swift +func throwsNothing() throws(Never) { ... } +``` + +is equivalent to a non-throwing function: + +```swift +func throwsNothing() { } +``` + +There is a more general subtyping rule here that says that you can loosen the thrown type, i.e., converting a non-throwing function to a throwing one, or a function that throws a concrete type to one that throws `any Error`. + +### An alternative to `rethrows` + +The ability to throw a generic error parameter that might be `Never` allows one to safely express some rethrowing patterns that are otherwise not possible with rethrows. For example, consider a function that semantically rethrows, but needs to do so by going through some code that doesn't throw: + +```swift +/// Count number of nodes in the tree that match a particular predicate +func countNodes(in tree: Node, matching predicate: (Node) throws -> Bool) rethrows -> Int { + class MyNodeVisitor: NodeVisitor { + var error: (any Error)? = nil + var count: Int = 0 + var predicate: (Node) throws -> Bool + + init(predicate: @escaping (Node) throws -> Bool) { + self.predicate = predicate + } + + override func visit(node: Node) { + do { + if try predicate(node) { + count = count + 1 + } + } catch let localError { + error = error ?? localError + } + } + } + + return try withoutActuallyEscaping(predicate) { predicate in + let visitor = MyNodeVisitor(predicate: predicate) + visitor.visitTree(node) + if let error = visitor.error { + throw error // error: is not throwing as a consequence of 'predicate' throwing. + } else { + return visitor.count + } + } +} +``` + +Walking through the code, we can convince ourselves that `MyNodeVisitor.error` will only ever be set as a result of the predicate throwing an error, so this code semantically fulfills the contract of `rethrows`. However, the Swift compiler's rethrows checking cannot perform such an analysis, so it will reject this function. The limitation on `rethrows` has prompted at least [two](https://forums.swift.org/t/pitch-rethrows-unchecked/10078) [pitches](https://forums.swift.org/t/pitch-fix-rethrows-checking-and-add-rethrows-unsafe/44863) to add an "unsafe" or "unchecked" rethrows variant, turning this into a runtime-checked contract. + +Typed throws offer a compelling alternative: one can capture the error type of the closure argument in a generic parameter, and use that consistently throughout. This is immediately useful for maintaining precise typed error information in generic code that only rethrows the error from its closure arguments, like `map`: + +```swift +extension Collection { + func map(body: (Element) throws(E) -> U) throws(E) -> [U] { + var result: [U] = [] + for element in self { + result.append(try body(element)) + } + return result + } +} +``` + +When given a closure that throws `CatError`, this formulation of `map` will throw `CatError`. When given a closure that doesn't throw, `E` will be `Never`, so `map` is non-throwing. + +This approach extends to our `countNodes` example: + +```swift +/// Count number of nodes in the tree that match a particular predicate +func countNodes(in tree: Node, matching predicate: (Node) throws(E) -> Bool) throws(E) -> Int { + class MyNodeVisitor: NodeVisitor { + var error: E? = nil + var count: Int = 0 + var predicate: (Node) throws(E) -> Bool + + init(predicate: @escaping (Node) throws(E) -> Bool) { + self.predicate = predicate + } + + override func visit(node: Node) { + do { + if try predicate(node) { + count = count + 1 + } + } catch let localError { + error = error ?? localError // okay, error has type E?, localError has type E + } + } + } + + return try withoutActuallyEscaping(predicate) { predicate in + let visitor = MyNodeVisitor(predicate: predicate) + visitor.visitTree(node) + if let error = visitor.error { + throw error // okay! error has type E, which can be thrown out of this function + } else { + return visitor.count + } + } +} +``` + +Note that typed throws has elegantly solved our problem, because any throwing site that throws a value of type `E` is accepted. When the closure argument doesn't throw, `E` is inferred to `Never`, and (dynamically) no instance of it will ever be created. + +### When to use typed throws + +Typed throws makes it possible to strictly specify the thrown error type of a function, but doing so constrains the evolution of that function's implementation. Additionally, errors are usually propagated or rendered, but not exhaustively handled, so even with the addition of typed throws to Swift, untyped `throws` is better for most scenarios. Consider typed throws only in the following circumstances: + +1. In code that stays within a module or package where you always want to handle the error, so it's purely an implementation detail and it is plausible to handle the error. +2. In generic code that never produces its own errors, but only passes through errors that come from user components. The standard library contains a number of constructs like this, whether they are `rethrows` functions like `map` or are capturing a `Failure` type like in `Task` or `Result`. +3. In dependency-free code that is meant to be used in a constrained environment (e.g., Embedded Swift) or cannot allocate memory, and will only ever produce its own errors. + +Resist the temptation to use typed throws because there is only a single kind of error that the implementation can throw. For example, consider an operation that loads bytes from a specified file: + +```swift +public func loadBytes(from file: String) async throws(FileSystemError) -> [UInt8] // should use untyped throws +``` + +Internally, it is using some file system library that throws a `FileSystemError`, which it then republishes directly. However, the fact that the error was specified to always be a `FileSystemError` may hamper further evolution of this API: for example, it might be reasonable for this API to start supporting loading bytes from other sources (say, a network connection or database) when the file name matches some other schema. However, errors from those other libraries will not be `FileSystemError` instances, which poses a problem for `loadBytes(from:)`: it either needs to translate the errors from other libraries into `FileSystemError` (if that's even possible), or it needs to break its API contract by adopting a more general error type (or untyped `throws`). + +This section will be added to the [Swift API Design Guidelines](https://www.swift.org/documentation/api-design-guidelines/). + +## Detailed design + +### Syntax adjustments + +The [Swift grammar](https://docs.swift.org/swift-book/ReferenceManual/zzSummaryOfTheGrammar.html) is updated wherever there is either `throws` or `rethrows`, to optionally include a thrown type, e.g., + +``` +throws-clause -> throws thrown-type(opt) + +thrown-type -> '(' type ')' +``` + +#### Function type + +Changing from + +``` +function-type → attributes(opt) function-type-argument-clause async(opt) throws(opt) -> type +``` + +to + +``` +function-type → attributes(opt) function-type-argument-clause async(opt) throws-clause(opt) -> type +``` + +Examples + +```swift +() -> Bool +() throws -> Bool +() throws(CatError) -> Bool +``` + +#### Closure expression + +Changing from + +``` +closure-signature → capture-list(opt) closure-parameter-clause async(opt) throws(opt) function-result opt in +``` + +to + +``` +closure-signature → capture-list(opt) closure-parameter-clause async(opt) throws-clause(opt) function-result opt in +``` + +Examples + +```swift +{ () -> Bool in true } +{ () throws -> Bool in true } +{ () throws(CatError) -> Bool in true } +``` + + +#### Function, initializer, and accessor declarations + +Changing from + +``` +function-signature → parameter-clause async(opt) throws(opt) function-result(opt) +function-signature → parameter-clause async(opt) rethrows(opt) function-result(opt) +initializer-declaration → initializer-head generic-parameter-clause(opt) parameter-clause async(opt) throws(opt) +initializer-declaration → initializer-head generic-parameter-clause(opt) parameter-clause async(opt) throws(opt) +``` + +to + +``` +function-signature → parameter-clause async(opt) throws-clause(opt) function-result(opt) +initializer-declaration → initializer-head generic-parameter-clause(opt) parameter-clause async(opt) throws-clause(opt) +``` + +Note that the current grammar does not account for throwing accessors, although they should receive the same transformation. + +#### `do..catch` blocks + +The syntax of a `do..catch` block is extended with an optional throw clause: + +``` +do-statement → do throws-clause(opt) code-block catch-clauses? +``` + +If a `throws-clause` is present, then there must be at least one `catch-clause`. + +#### Examples + +```swift +func callCat() -> Cat +func callCat() throws -> Cat +func callCat() throws(CatError) -> Cat + +init() +init() throws +init() throws(CatError) + +var value: Success { + get throws(Failure) { ... } +} +``` + +### Throwing and catching with typed throws + +#### Throwing within a function that declares a typed error + +Any function, closure or function type that is marked as `throws` can declare which type the function throws. That type, which is called the *thrown error type*, must conform to the `Error` protocol. + +Every uncaught error that can be thrown from the body of the function must be convertible to the thrown error type. This applies to both explicit `throw` statements and any errors thrown by other calls (as indicated by a `try`). For example: + +```swift +func throwingTypedErrors() throws(CatError) { + throw CatError.asleep // okay, type matches + throw .asleep // okay, can infer contextual type from the thrown error type + throw KidError() // error: KidError is not convertible to CatError + + try callCat() // okay + try callKids() // error: throws KidError, which is not convertible to CatError + + do { + try callKids() // okay, because this error is caught and suppressed below + } catch { + // eat the error + } +} +``` + +Because a value of any `Error`-conforming type implicitly converts to `any Error`, this implies that an function declared with untyped `throws` can throw anything: + +```swift +func untypedThrows() throws { + throw CatError.asleep // okay, CatError converts to any Error + throw KidError() // okay, KidError converts to any Error + try callCat() // okay, thrown CatError converts to any Error + try callKids() // okay, thrown KidError converts to any Error +} +``` + +Therefore, these rules subsume those of untyped throws, and no existing code will change behavior. + +Note that the constraint that the thrown error type must conform to `Error` means that one cannot use an existential type such as `any Error & Codable` as the thrown error type: + +```swift +// error: any Error & Codable does not conform to Error +func remoteCall(function: String) async throws(any Error & Codable) -> String { ... } +``` + +The `any Error` existential has [special semantics](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0235-add-result.md#adding-swifterror-self-conformance) that allow it to conform to the `Error` protocol, introduced along with `Result`. A separate language change would be required to allow other existential types to conform to the `Error` protocol. + +#### Catching typed thrown errors + +A `do...catch` block is used to catch and process thrown errors. With only untyped errors, the type of the error thrown from inside the `do` block is always `any Error`. In the presence of typed throws, the type of the error thrown from inside the `do` block can either be explicitly specified with a `throws` clause following the `do`, or inferred from the specific throwing sites. + +When the `do` block specifies a thrown error type, that error type can be used for inferring the contextual type of `throw` statements. For example: + +```swift +do throws(CatError) { + if isDaytime && foodBowl.isEmpty { + throw .sleep + } +} catch { + // implicit 'error' value has type CatError +} +``` + +As with other uses of untyped throws, `do throws` is equivalent to `do throws(any Error)`. + +When there is no throws clause, the thrown error type is inferred from the body of the `do` block. When all throwing sites within a `do` block produce the same error type (ignoring any that throw `Never`), that error type is used as the type of the thrown error. For example: + +```swift +do /*infers throws(CatError)*/ { + try callCat() // throws CatError + if something { + throw CatError.asleep // throws CatError + } +} catch { + // implicit 'error' value has type CatError + if error == .asleep { + openFoodCan() + } +} +``` + +This also implies that one can use the thrown type context to perform type-specific checks in the catch clauses, e.g., + +```swift +do /*infers throws(CatError)*/ { + try callCat() // throws CatError + if something { + throw CatError.asleep // throws CatError + } +} catch .asleep { + openFoodCan() +} // note: CatError can be thrown out of this do...catch block when the cat isn't asleep +``` + +> **Rationale**: By inferring a concrete result type for the thrown error type, we can entirely avoid having to reason about existential error types within `catch` blocks, leading to a simpler syntax. Additionally, it preserves the notion that a `do...catch` block that has a `catch` site accepting anything (i.e., one with no conditions) can exhaustively suppress all errors. + +When throw sites within the `do` block throw different (non-`Never`) error types, the inferred error type is `any Error`. For example: + +```swift +do /*infers throws(any Error)*/ { + try callCat() // throws CatError + try callKids() // throw KidError +} catch { + // implicit 'error' variable has type 'any Error' +} +``` + +In essence, when there are multiple possible thrown error types, we immediately resolve to the untyped equivalent of `any Error`. We will refer to this notion as a type function `errorUnion(E1, E2, ..., EN)`, which takes `N` different error types (e.g., for throwing sites within a `do` block) and produces the union error type of those types. Our definition and use of `errorUnion` for typed throws subsumes the existing rule for untyped throws, in which every throw site produces an error of type `any Error`. + +> **Rationale**: While it would be possible to compute a more precise "union" type of different error types, doing so is potentially an expensive operation at compile time and run time, as well as being harder for the programmer to reason about. If in the future it becomes important to tighten up the error types, that could be done in a mostly source-compatible manner. + +The semantics specified here are not fully source compatible with existing Swift code. A `do...catch` block that contains `throw` statements of a single concrete type (and no other throwing sites) might depend on the error being caught as `any Error`. Here is a contrived example: + +```swift +do /*infers throws(CatError) in Swift 6 */ { + throw CatError.asleep +} catch { + var e = error // currently has type any Error, will have type CatError + e = KidsError() // currently well-formed, will become an error +} +``` + +To prevent this source compatibility issue, we refine the rule slightly to specify that any `throw` statement always throws a value of type `any Error`. That way, one can only get a caught error type more specific than `any Error` when the both of the `do..catch` contains no `throw` statements and all of the `try` operations are using functions that make use of typed throws. + +Note that the only way to write an exhaustive `do...catch` statement is to have an unconditional `catch` block. The dynamic checking provided by `is` or `as` patterns in the `catch` block cannot be used to make a catch exhaustive, even if the type specified is the same as the type thrown from the body of the `do`: + +```swift +func f() { + do /*infers throws(CatError)*/ { + try callCat() + } catch let ce as CatError { + + } // error: do...catch is not exhaustive, so this code rethrows CatError and is ill-formed +} +``` + +> **Note**: Exhaustiveness checking in the general is expensive at compile time, and the existing language uses the presence of an unconditional `catch` block as the indicator for an exhaustive `do...catch`. See the section on closure thrown type inference for more details about inferring throwing closures. + +#### `rethrows` + +A function marked `rethrows` throws only when one of its closure parameters throws. It is typically used with higher-order functions, such as the `map` operation on a collection: + +```swift +extension Collection { + func map(body: (Element) throws -> U) rethrows -> [U] { + var result: [U] = [] + for element in self { + result.append(try body(element)) + } + return result + } +} +``` + +When provided with a throwing closure, `map` can throw, and it chooses to directly throw the same error as the body. This contract can be more precisely modeled using typed throws: + +```swift +extension Collection { + func map(body: (Element) throws(E) -> U) throws(E) -> [U] { + var result: [U] = [] + for element in self { + result.append(try body(element)) + } + return result + } +} +``` + +Now, when `map` is provided with a closure that throws `E`, it can only throw an `E`. For a non-throwing closure, `E` will be `Never` and `map` is non-throwing. For an untyped throwing closure, `E` will be `any Error` and we get the same type-level behavior as the `rethrows` version of `map`. + +However, because `rethrows` uses untyped errors, `map` would be permitted to substitute a different error type that, for example, provides more information about the failing element: + +```swift +struct MapError: Error { + var failedElement: Element + var underlyingError: any Error +} + +extension Collection { + func map(body: (Element) throws -> U) rethrows -> [U] { + var result: [U] = [] + for element in self { + do { + result.append(try body(element)) + } catch { + // Provide more information about the failure + throw MapError(failedElement: element, underlyingError: error) + } + } + return result + } +} +``` + +Typed throws, as presented here, is not able to express the contract of this function. + +The Swift standard library does not perform error substitution of this form, and its contract for operations like `map` is best expressed by typed throws as shown above. It is likely that many existing `rethrows` functions are better expressed with typed throws. However, not *all* `rethrows` functions can be expressed by typed throws, if they are performing error substitution like this last `map`. + +Therefore, this proposal does not change the primary semantics of `rethrows`: it remains untyped, and it is ill-formed to attempt to provide a thrown error type to a `rethrows` function. The Alternatives Considered section provides several options for `rethrows`, which can become the subject of a future proposal. + +However, there is a small change in the type checking behavior of a `rethrows` function to improve source compatibility in certain cases. Specifically, consider a `rethrows` function that calls into a function with typed throws: + +```swift +extension Collection { + func filter(_ isIncluded: (Element) throws(E) -> Bool) throws(E) -> [Element] { ... } + + func filterOdds(_ isIncluded: (Element) throws -> Bool) rethrows -> [Element { + var onOdd = true + return try filter { element in + defer { onOdd = !onOdd } + return onOdd && isIncluded(element) + } // error: call to filter isn't "rethrows" + } +} +``` + +The standard `rethrows` checking rejects the call to `filter` because, technically, it could throw `any Error` under any circumstances. Unfortunately, this behavior is a source compatibility problem for the standard library's adoption of typed throws, because an existing `rethrows` function calling into something like `map` or `filter` would be rejected once those introduce typed throws. This proposal introduces a small compatibility feature that considers a function that + +1. Has a thrown error type that is a generic parameter (call it `E`) of the function itself, +2. Has no protocol requirements on `E` other than that it conform to the `Error` protocol, and +3. Any parameters of throwing function type throw the specific error type `E`. + +to be a rethrowing function for the purposes of `rethrows` checking in its caller. This compatibility feature introduces a small soundness hole in `rethrows` functions that can only be removed with improvements to type inference behavior. + +#### Opaque thrown error types + +The thrown error type of a function can be specified with an [opaque result type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0244-opaque-result-types.md). For example: + +```swift +func doSomething() throws(some Error) { ... } +``` + +The opaque thrown error type is like a result type, so the concrete type of the error is chosen by the `doSomething` function itself, and could change from one version to the next. The caller only knows that the error type conforms to the `Error` protocol; the concrete type won't be knowable until runtime. + +Opaque result types can be used as an alternative to existentials (`any Error`) when there is a fixed number of potential error types that might be thrown , and we either can't (due to being in an embedded environment) or don't want to (for performance or code-evolution reasons) expose the precise error type. For example, one could use a suitable `Either` type under the hood: + +```swift +func doSomething() throws(some Error) { + do { + try callCat() + } catch { + throw Either.left(error) + } + + do { + try callKids() + } catch { + throw Either.right(error) + } +} +``` + +Due to the contravariance of parameters, an opaque thrown error type that occurs within a function parameter will be an [opaque parameter](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md). This means that the closure argument itself will choose the type, so + +```swift +func map(_ transform: (Element) throws(some Error) -> T) rethrows -> [T] +``` + +is equivalent to + +```swift +func map(_ transform: (Element) throws(E) -> T) rethrows -> [T] +``` + +#### `async let` + +An `async let` initializer can throw an error, and that error is effectively rethrown at any point where one of the variables defined in the `async let` is referenced. For example: + +```swift +async let answer = callCat() +// ... +try await answer // could rethrow the result from the initializer here +``` + +The type thrown by the variables of an `async let` is determined using the same rules as for the `do` part of a `do...catch` block. In the example above, accesses to `answer` can throw an error of type `CatError`. + +### Subtyping rules + +A function type that throws an error of type `A` is a subtype of a function type that differs only in that it throws an error of type `B` when `A` is a subtype of `B`. As previously noted, a `throws` function that does not specify the thrown error type will have a thrown type of `any Error`, and a non-throwing function has a thrown error type of `Never`. For subtyping purposes, `Never` is assumed to be a subtype of all error types. + +The subtyping rule manifests in a number of places, including function conversions, protocol conformance checking and refinements, and override checking, all of which are described below. + +#### Function conversions + +Having related errors and a non-throwing function + +```swift +class BaseError: Error {} +class SubError: BaseError {} + +let f1: () -> Void +``` + +Converting a non-throwing function to a throwing one is allowed + +```swift +let f2: () throws(SubError) -> Void = f1 +``` + +It's also allowed to assign a subtype of a thrown error, though the subtype information is erased and the error of f2 will be casted up. + +```swift +let f3: () throws(BaseError) -> Void = f2 +``` + +Erasing the specific error type is possible + +```swift +let f4: () throws -> Void = f3 +``` + +#### Protocol conformance + +Protocols should have the possibility to conform and refine other protocols containing throwing functions based on the subtype relationship of their functions. This way it would be possible to throw a more specialised error or don't throw an error at all. + +```swift +protocol Throwing { + func f() throws +} + +struct ConcreteNotThrowing: Throwing { + func f() { } // okay, doesn't have to throw +} + +enum SpecificError: Error { ... } + +struct ConcreteThrowingSpecific: Throwing { + func f() throws(SpecificError) { } // okay, throws a specific error +} +``` + +#### Override checking + +A declaration in a subclass that overrides a superclass declaration can be a subtype of the superclass declaration, for example: + +```swift +class BlueError: Error { ... } +class DeepBlueError: BlueError { ... } + +class Superclass { + func f() throws { } + func g() throws(BlueError) { } +} + +class Subclass: Superclass { + override func f() throws(BlueError) { } // okay + override func g() throws(DeepBlueError) { } // okay +} + +class Subsubclass: Subclass { + override func f() { } // okay + override func g() { } // okay +} +``` + +### Type inference + +The type checker can infer thrown error types in a number of different places, making it easier to carry specific thrown type information through a program without additional annotation. This section covers the various ways in which thrown errors interact with type inference. + +#### Associated type inference + +An associated type can be used as the thrown error type in other protocol requirements. For example: + +```swift +protocol CatFeeder { + associatedtype FeedError: Error + + func feedCat() throws(FeedError) -> CatStatus +} +``` + +When a concrete type conforms to such a protocol, the associated type can be inferred from the declarations that satisfy requirements that mention the associated type in a typed throws clause. For the purposes of this inference, a non-throwing function has `Never` as its error type and an untyped `throws` function has `any Error` as its error type. For example: + +```swift +struct Tabby: CatFeeder { + func feedCat() throws(CatError) -> CatStatus { ... } // okay, FeedError is inferred to CatError +} + +struct Sphynx: CatFeeder { + func feedCat() throws -> CatStatus { ... } // okay, FeedError is inferred to any Error +} + +struct Ragdoll: CatFeeder { + func feedCat() -> CatStatus { ... } // okay, FeedError is inferred to Never +} +``` + +#### `Error` requirement inference + +When a function signature uses a generic parameter or associated type as a thrown type, that generic parameter or associated type is implicitly inferred to conform to the `Error` type. For example, given this declaration for `map`: + +```swift +func map(body: (Element) throws(E) -> T) throws(E) { ... } +``` + +the function has an inferred requirement `E: Error`. + +### Standard library adoption + +#### Converting between `throws` and `Result` + +`Result`'s [init(catching:)](https://developer.apple.com/documentation/swift/result/3139399-init) operation translates a throwing closure into a `Result` instance. It's currently defined only when the `Failure` type is `any Error`, i.e., + +```swift +init(catching body: () throws -> Success) where Failure == any Error { ... } +``` + +Replace this with an initializer that uses typed throws: + +```swift +init(catching body: () throws(Failure) -> Success) +``` + +The new initializer is more flexible: in addition to retaining the error type from typed throws, it also supports non-throwing closure arguments by inferring `Failure` to be equal to `Never`. + +Additionally, `Result`'s `get()` operation: + +```swift +func get() throws -> Success +``` + +should use `Failure` as the thrown error type: + +```swift +func get() throws(Failure) -> Success +``` + +#### Standard library operations that `rethrow` + +The standard library contains a large number of operations that `rethrow`. In all cases, the standard library will only throw from a call to one of the closure arguments: it will never substitute a different thrown error. Therefore, each `rethrows` operation in the standard library should be replaced with one that uses typed throws to propagate the same error type. For example, the `Optional.map` operation would change from: + +```swift +public func map( + _ transform: (Wrapped) throws -> U +) rethrows -> U? +``` + +to + +```swift +public func map( + _ transform: (Wrapped) throws(E) -> U +) throws(E) -> U? +``` + +This is a mechanical transformation that is applied throughout the standard library. + +## Source compatibility + +This proposal has called out a few specific places where the introduction of typed throws into the language could affect source compatibility. However, in those places, we have opted for semantics that ensure that existing Swift code that does not change behavior, to make this proposal act as a purely additive change to the language. Once a function adopts typed throws, the effect of typed throws can then ripple to its callers. + +## Effect on API resilience + +An API that uses typed throws cannot make its thrown error type more general (or untyped) without breaking existing clients that depend on the specific thrown error type: + +```swift +// Library +public enum DataLoaderError { + case missing +} + +public class DataLoader { + func load() throws(DataLoaderError) -> Data { ... } +} + +// Client code +func processError(_ error: DataLoaderError) { ... } + +func load(from dataLoader: dataLoader) { + do { + try dataLoader.load() + } catch { + processError(error) + } +} +``` + +Any attempt to generalize the thrown type of `DataLoader.load()` will break the client code, which depends on getting a `DataLoaderError` in the `catch` block. + +Going in the other direction, of making the thrown error type *more* specific than it used to be (or adopting typed throws in an API that previously used untyped throws) can also break clients, but in much more limited cases. For example, let's consider the same API above, but in reverse: + +```swift +// Library +public enum DataLoaderError { + case missing +} + +public class DataLoader { + func load() throws -> Data { ... } +} + +// Client +func processError(_ error: any Error) { ... } + +func load(from dataLoader: dataLoader) { + do { + try dataLoader.load() + } catch { + processError(error) + } +} +``` + +Here, the `DataLoader.load()` function could be updated to throw `DataLoaderError` and this particular client code would still work, because `DataLoaderError` is convertible to `any Error`. Note that clients could still be broken by this kind of change, for example overrides of an `open` function, declarations that satisfy a protocol requirement, or code that relies on the precide error type (say, by overloading). However, such a change is far less likely to break clients of an API than loosening thrown type informance. + +A `rethrows` function can generally be replaced with a function that is generic over the thrown error type of its closure argument and propagates that thrown error. For example, one can replace this API: + +```swift +public func last( + where predicate: (Element) throws -> Bool +) rethrows -> Element? +``` + +with + +```swift +public func last( + where predicate: (Element) throws(E) -> Bool +) throws(E) -> Element? +``` + +When calling this function, the closure argument supplies the thrown error type (`E`), which can also be inferred to `any Error` (for untyped `throws`) or `Never` (for non-throwing functions). Existing clients of this new function therefore see the same behavior as with the `rethrows` version. + +There is one difference between the two functions that could break client code that is referring to such functions without calling them. For example, consider the following code: + +```swift +let primes = [2, 3, 5, 7] +let getLast = primes.last(where:) +``` + +With the `rethrows` formulation of the `last(where:)` function, `getLast` will have the type `((Int) throws -> Bool) throws -> Int?`. With the typed-errors formulation, this code will result in an error because the an argument for the generic parameter `E` cannot be inferred without context. Note that this is only a problem when there is no context type for `getLast`, and can be fixed by providing it with a type: + +```swift +let getLast: ((Int) -> Bool) -> Int? = primes.last(where:) // okay, E is inferred to Never +``` + +Note that one would have to do the same thing with the `rethrows` formulation to produce a non-throwing `getLast`, because `rethrows` is not a part of the formal type system. Given that most `rethrows` operations are already generic in other parameters (unlike `last(where:)`), and most uses of such APIs are either calls or have type context, it is expected that the actual source compatibility impact of replacing `rethrows` with typed errors will be small. + +## Effect on ABI stability + +The ABI between a function with an untyped throws and one that uses typed throws will be different, so that typed throws can benefit from knowing the precise type. + +Replacing a `rethrows` function with one that uses typed throws, as proposed for the standard library, is an ABI-breaking change. However, it can be done in a manner that doesn't break ABI by retaining the `rethrows` function only for binary-compatibility purposes. The existing `rethrows` functions will be renamed at the source level (so they don't conflict with the new ones) and made `@usableFromInline internal`, which retains the ABI while making the function invisible to clients of the standard library: + +```swift +@usableFromInline +@_silgen_name() +internal func _oldRethrowingMap( + _ transform: (Wrapped) throws -> U +) rethrows -> U? +``` + +Then, the new typed-throws version will be introduced with [back-deployment support](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0376-function-back-deployment.md): + +```swift +@backDeploy(...) +public func map( + _ transform: (Wrapped) throws(E) -> U +) throws(E) -> U? +``` + +This way, clients compiled against the updated standard library will always use the typed-throws version. Note that many of these functions are quite small and will be generic, so implementers may opt to use `@_alwaysEmitIntoClient` rather than `@backDeploy`. + +## Future directions + +### Closure thrown type inference + +Function declarations must always explicitly specify whether they throw, optionally providing a specific thrown error type. For closures, whether they throw or not is inferred by the Swift compiler. Specifically, the Swift compiler looks at the structure of body of the closure. If the body of the closure contains a throwing site (either a `throw` statement or a `try` expression) that is not within an exhaustive `do...catch` (i.e., one that has an unconditional `catch` clause), then the closure is inferred to be `throws`. Otherwise, it is non-throwing. Here are some examples: + +```swift +{ throw E() } // throws + +{ try call() } // throws + +{ + do { + try call() + } catch let e as CatError { + // ... + } +} // throws, the do...catch is not exhaustive + +{ + do { + try call() + } catch e {} + // ... + } +} // does not throw, the do...catch is exhaustive +``` + +With typed throws, the closure type could be inferred to have a typed error by considering all of the throwing sites that aren't caught (let each have a thrown type `Ei`) and then inferring the closure's thrown error type to be `errorUnion(E1, E2, ... EN)`. + +This inference rule will change the thrown error types of existing closures that throw concrete types. For example, the following closure: + +```swift +{ + if Int.random(in: 0..<24) < 20 { + throw CatError.asleep + } +} +``` + +will currently be inferred as `throws`. With the rule specified here, it will be inferred as `throws(CatError)`. This could break some code that depends on the precisely inferred type. To prevent this from becoming a source compatibility problem, we apply the same rule as for `do...catch` statements to limit inference: `throw` statements within the closure body are treated as having the type `any Error` in Swift 5. This way, one can only infer a more specific thrown error type in a closure when the `try` operations are calling functions that make use of typed errors. + +Note that one can explicitly specify the thrown error type of a closure to disable this type inference, which has the nice effect of also providing a contextual type for throw statements: + +```swift +{ () throws(CatError) in + if Int.random(in: 0..<24) < 20 { + throw .asleep + } +} +``` + +Such a change would need to be under an upcoming feature flag (e.g., `FullTypedThrows`) and should also involve inference from the actual thrown error type of `throw` statements as well as closing the minor semantic hole introduced for compatibility with `rethrows` functions. + +### Concurrency library adoption + +The concurrency library has a number of places that could benefit from the adoption of typed throws, including `Task` creation and completion, continuations, task cancellation, task groups, and async sequences and streams. + +`Task` is similar to `Result` because it also carries a `Failure` type that could benefit from typed throws. Continuations and task groups could propagate typed throws information from closures to make more of the library usable with precise thrown type information. + +`AsyncSequence`, and the asynchronous `for..in` loop that depends on it, could be improved by using typed throws. Both `AsyncIteratorProtocol` and `AsyncSequence` could be augmented with a `Failure` associated type that is used for the thrown error type of `next()`, and will be used by the asynchronous `for..in` loop to determine whether the sequence can throw. This can be combined with [primary associated types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0346-light-weight-same-type-syntax.md) to make it possible to use existentials such as `any AsyncSequence`: + +```swift +public protocol AsyncIteratorProtocol { + associatedtype Element + associatedtype Failure: Error = any Error + mutating func next() async throws(Failure) -> Element? +} + +public protocol AsyncSequence { + associatedtype AsyncIterator: AsyncIteratorProtocol + associatedtype Element where AsyncIterator.Element == Element + associatedtype Failure where AsyncIterator.Failure == Failure + __consuming func makeAsyncIterator() -> AsyncIterator +} +``` + +The scope of potential changes to the concurrency library to make full use of typed throws is large. Unlike with the standard library, the adoption of typed throws in the concurrency library requires some interesting design. Therefore, we leave it to a follow-on proposal, noting only that whatever form `AsyncSequence` takes with typed throws, the language support for asynchronous `for..in` will need to adjust. + +### Specific thrown error types for distributed actors + +The transport mechanism for [distributed actors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0344-distributed-actor-runtime.md), `DistributedActorSystem`, can throw an error due to transport failures. This error is currently untyped, but it should be possible to adopt typed throws (with a `Failure` associated type in `DistributedActorSystem` and mirrored in `DistributedActor`) so that the distributed actor system can be more specific about the kind of error it throws. Calls to a distributed actor from outside the actor (i.e., that could be on a different node) would then throw `errorUnion(Failure, E)` where the `E` is the type that the function normally throws. + +## Alternatives considered + +### Thrown error type syntax + +There have been several alternatives to the `throws(E)` syntax proposed here. The `throws(E)` syntax was chosen because it is syntactically unambiguous, allows arbitrary types for `E`, and is consistent with the way in which attributes (like property wrappers or macros with arguments) and modifiers (like `unowned(unsafe)`) are written. + +The most commonly proposed syntax omits the parentheses, i.e., `throws E`. However, this syntax introduces some syntactic ambiguities that would need to be addressed and might cause problems for future evolution of the language: + +* The following code is syntactically ambiguous if `E` is parsed with the arbitrary `type` grammar: + + ```swift + func f() throws (E) -> Int { ... } + ``` + + because the error type would parse as either `(E)` or `(E) -> Int`. One could parse a subset of the `type` grammar that doesn't include function types to dodge this ambiguity, with a more complicated grammar. + +* The identifier following `throws` could end up conflicting with a future effect: + + ```swift + func f() throws E { ... } + ``` + + If `E` were an effect name in some later Swift version, then there is an ambiguity between typed throws and that effect that we would need to resolve. Future effect modifiers might require more than one argument (and therefore need parentheses), which would make them inconsistent with `throws E`. + +Another suggestion uses angle brackets around the thrown type, i.e., + +```swift +func f() throws -> Int { ... } +``` + +This follows more closely with generic syntax, and highlights the type nature of the arguments more clearly. It's inconsistent with the use of parentheses in modifiers, but has some precedent in attached macros where one can explicitly specify the generic arguments to the macro, e.g., `@OptionSet`. + +### Multiple thrown error types + +This proposal specifies that a function may throw at most one error type, and if there is any reason to throw more than one error type, one should use `any Error` (or the equivalent untyped `throws` spelling). It would be possible to support multiple error types, e.g., + +```swift +func fetchData() throws(FileSystemError, NetworkError) -> Data +``` + +However, this change would introduce a significant amount of complexity in the type system, because everywhere that deals with thrown errors would have to deal with an arbitrary set of thrown errors. + +A more reasonable direction to support this use case would be to introduce a form of anonymous enum (often called a *sum* type) into the language itself, where the type `A | B` can be either an `A` or ` B`. With such a feature in place, one could express the function above as: + +```swift +func fetchData() throws(FileSystemError | NetworkError) -> Data +``` + +Trying to introduce multiple thrown error types directly into the language would introduce nearly all of the complexity of sum types, but without the generality, so this proposal only considers a single thrown error type. + +### Treat all uninhabited thrown error types as nonthrowing + +This proposal specifies that a function type whose thrown error type is `Never` is equivalent to a function type that does not throw. This rule could be generalized from `Never` to any *uninhabited* type, i.e., any type for which we can structurally determine that there is no runtime value. The simplest uninhabited type is a frozen enum with no cases, which is how `Never` itself is defined: + +```swift +@frozen public enum Never {} +``` + +However, there are other forms of uninhabited type: a `struct` or `class` with a stored property of uninhabited type is uninhabited, as is an enum where all cases have an associated value containing an uninhabited type (a generalization of the "no cases" rule mentioned above). This can happen generically. For example, a simple `Pair` struct: + +```swift +struct Pair { + var first: First + var second Second +} +``` + +will be uninhabited when either `First` or `Second` is uninhabited. The `Either` enum will be uninhabited when both of its generic arguments are uninhabited. `Optional` is never uninhabited, because it's always possible to create a `nil` value. + +It is possible to generalize the rule about non-throwing function types to consider any function type with an uninhabited thrown error type to be equivalent to a non-throwing function type (all other things remaining equal). However, we do not do so due to implementation concerns: the check for a type being uninhabited is nontrivial, requiring one to walk all of the storage of the type, and (in the presence of indirect enum cases and reference types) is recursive, making it a potentially expensive computation. Crucially, this computation will need to be performed at runtime, to produce proper function type metadata within generic functions: + +```swift +func f(_: E.Type)) { + typealias Fn = () throws(E) -> Void + let meta = Fn.self +} + +f(Never.self) // Fn should be equivalent to () -> Void +f(Either.self) // Fn should be equivalent to () -> Void +f(Pair.self) // Fn should be equivalent to () -> Void +``` + +The runtime computation of "uninhabited" therefore carries significant cost in terms of the metadata required (one may need to walk all of the storage of the type) as well as the execution time to evaluate that metadata during runtime type formation. + +The most plausible route here involves the introduction of an `Uninhabited` protocol, which could then be used with conditional conformances to propagate the "uninhabited" type information. For example, `Never` would conform to `Uninhabited`, and one could conditionally conform a generic error type. For example: + +```swift +struct WrappedError: Error { + var wrapped: E +} + +extension WrappedError: Uninhabited where E: Uninhabited { } +``` + +With this, one can express "rethrowing" behavior that wraps the underlying error via typed throws: + +```swift +func translatesError(f: () throws(E) -> Void) throws(WrappedError) { ... } +``` + +Here, when give a non-throwing closure for `f` (which infers `E = Never`), `translatesError` is known not to throw because `WrappedError` is known to be uninhabited (via the conditional conformance). This approach extends to the use of an `Either` type to capture errors: + +```swift +extension Either: Uninhabited when Left: Uninhabited, Right: Uninhabited { } +``` + +However, it breaks down when there are two such generic error parameters for something like `WrappedError`, because having either one of them be `Uninhabited` makes the struct uninhabited, and the generics system does not permit disjunctive constraints like that. + +Extending from `Never` to arbitrary uninhabited types has some benefits, but requires enough additional design work and complexity that it should constitute a separate proposal. Therefore, we stick with the simpler rule where `Never` is the only uninhabited type considered to be special. + +### Typed `rethrows` + +A function marked `rethrows` throws only when one or more of its closure arguments throws. As note previously, typed throws allows one to more precisely express when the function only rethrows exactly the error from its closure, without translation, as demonstrated with `map`: + +```swift +func map(_ transform: (Element) throws(E) -> T) throws(E) -> [T] +``` + +However, it cannot express rethrowing behavior when the function is performing translation of errors. For example, consider the following: + +```swift +func translateErrors( + f: () throws(E1) -> Void, + g: () throws(E2) -> Void +) ??? { + do { + try f() + } catch { + throw SimpleError(message: "E1: \(error)") + } + + do { + try g() + } catch { + throw SimpleError(message: "E2: \(error)") + } +} +``` + +This function will only throw when `f` or `g` throw, and in both cases will translate the errors into `SimpleError`. With this proposal, there are two options for specifying the error-handling behavior of `translateErrors`, neither of which is precise: + +* `rethrows` correctly communicates that this function throws only when the arguments for `f` or `g` do, but the thrown error type is treated as `any Error`. +* `throws(SimpleError)` correctly communicates that this function throws errors of type `SimpleError`, but not that it throws when the argument for `f` or `g` do. + +One way to address this would be to allow `rethrows` to specify the thrown error type, e.g., `rethrows(SimpleError)`, which captures both of the aspects of how this function behaves---when it throws, and what specific error type it `throws`. + +With typed `rethrows`, a bare `rethrows` could be treated as syntactic sugar for `rethrows(any Error)`, similarly to how `throws` is syntactic sugar for `throws(any Error)`. This extension is source-compatible and allows one to express more specific error types with throwing behavior. + +However, this definition of `rethrows` is somewhat unfortunate in a typed-throws world, because it is likely the wrong default. Many use cases for `rethrows` do not involve error translation, and would be better served by using typed throws in the manner that `map` does. If `rethrows` were not already part of the Swift language prior to this proposal, it's likely that we either would not introduce the feature at all, or would treat it as syntactic sugar for typed throws that introduces a generic parameter for the error type that is used for the thrown type of the closure parameters and the function itself. For example: + +```swift +// rethrows could try rethrows as syntactic sugar.. +func map(_ transform: (Element) throws -> T) rethrows -> [T] +// for typed errors: +func map(_ transform: (Element) throws(E) -> T) throws(E) -> [T] +``` + +Removing or changing the semantics of `rethrows` would be a source-incompatible change, so we leave such concerns to a later proposal. + +## Revision history + +* Revision 6 (post-review): + * Closure type inference did not get implemented in Swift 6.0, so this proposal has been "shrunk" down to what actually got implemented in Swift 6.0. +* Revision 5 (first review): + * Add `do throws(MyError)` { ... } syntax to allow explicit specification of the thrown error type within the body of a `do..catch` block, suppressing type inference of the thrown error type. Thank you to Becca Royal-Gordon for the idea! +* Revision 4: + * Update the introduction, motivation, and "when to use typed throws" to be more direct. + * Re-incorporate the replacement of `rethrows` functions in the standard library with generic typed throws into the actual proposal. It's so mechanical and straightforward that it doesn't need a separate proposal. + * Extend the discussion on API resilience to talk through the source compatibility impacts of replacing a `rethrows` function with one that uses typed throws, since it is quite relevant to this proposal. + * Explain that one cannot currently have a thrown error type of `any Error & Codable` or similar because it doesn't conform to `Error`. + * Introduce a compatibility feature to `rethrows` functions to cope with their callees moving to typed throws. +* Revision 3: + * Move the the typed `rethrows` feature out of this proposal, and into Alternatives Considered. Once we gain more experience with typed throws, we can decide what to do with `rethrows`. + * Expand the discussion on allowing all uninhabited error types to mean "non-throwing". + * Provide a better example for inferring `Error` conformance on generic parameters. + * Move the replacement of `rethrows` in the standard library with typed throws into "Future Directions", because it is large enough that it needs a separate proposal. + * Move the concurrency library changes for typed throws into "Future Directions", because it is large enough that it needs a separate proposal. + * Add an extended example of replacing the need for `rethrows(unsafe)` with typed throws. + * Provide a more significant example of opaque thrown errors that makes use of `Either` internally. +* Revision 2: + * Add a short section on when to use typed throws + * Add an Alternatives Considered section for other syntaxes + * Make it clear that only unconditional catches make `do...catch` exhaustive + * Update continuation APIs with typed throws + * Add an example of an existential thrown error type + * Describe semantics of `async let` with respect to thrown errors + * Add updates to task cancellation APIs diff --git a/proposals/0414-region-based-isolation.md b/proposals/0414-region-based-isolation.md new file mode 100644 index 0000000000..715b245ccc --- /dev/null +++ b/proposals/0414-region-based-isolation.md @@ -0,0 +1,2074 @@ +# Region based Isolation + +* Proposal: [SE-0414](0414-region-based-isolation.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm) [Joshua Turcotti](https://github.com/jturcotti) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 6.0)** +* Upcoming Feature Flag: `RegionBasedIsolation` +* Review: ([first pitch](https://forums.swift.org/t/pitch-safely-sending-non-sendable-values-across-isolation-domains/66566)), ([second pitch](https://forums.swift.org/t/pitch-region-based-isolation/67888)), ([first review](https://forums.swift.org/t/se-0414-region-based-isolation/68805)), ([revision](https://forums.swift.org/t/returned-for-revision-se-0414-region-based-isolation/69123)), ([second review](https://forums.swift.org/t/se-0414-second-review-region-based-isolation/69740)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0414-region-based-isolation/70051)) + +## Introduction + +Swift Concurrency assigns values to *isolation domains* determined by actor and +task boundaries. Code running in distinct isolation domains can execute +concurrently, and `Sendable` checking defines away concurrent access to +shared mutable state by preventing non-`Sendable` values from being passed +across isolation boundaries full stop. In practice, this is a significant +semantic restriction, because it forbids natural programming patterns that are +free of data races. + +In this document, we propose loosening these rules by introducing a +new control flow sensitive diagnostic that determines whether a non-`Sendable` +value can safely be transferred over an isolation boundary. This is done by +introducing the concept of *isolation regions* that allows the compiler to +reason conservatively if two values can affect each other. Through the usage of +isolation regions, the language can prove that transferring a non-`Sendable` +value over an isolation boundary cannot result in races because the value (and +any other value that might reference it) is not used in the caller after the +point of transfer. + +## Motivation + +[SE-0302](0302-concurrent-value-and-concurrent-closures.md) states +that non-`Sendable` values cannot be passed across isolation boundaries. The +following code demonstrates a `Sendable` violation when passing a +newly-initialized value into an actor-isolated function: + +```swift +// Not Sendable +class Client { + init(name: String, initialBalance: Double) { ... } +} + +actor ClientStore { + var clients: [Client] = [] + + static let shared = ClientStore() + + func addClient(_ c: Client) { + clients.append(c) + } +} + +func openNewAccount(name: String, initialBalance: Double) async { + let client = Client(name: name, initialBalance: initialBalance) + await ClientStore.shared.addClient(client) // Error! 'Client' is non-`Sendable`! +} +``` + +This is overly conservative; the program is safe because: + +* `client` does not have access to any non-`Sendable` state from its initializer + parameters since Strings and Doubles are `Sendable`. +* `client` just being initialized implies that `client` cannot have any uses + outside of `openNewAccount`. +* `client` is not used within `openNewAccount` beyond `addClient`. + +The simple example above shows the expressivity limitations of Swift's strict +concurrency checking. Programmers are required to use unsafe escape hatches, +such as `@unchecked Sendable` conformances, for common patterns that are already +free of data races. + +## Proposed solution + +We propose the introduction of a new control flow sensitive diagnostic that +enables transferring non-`Sendable` values across isolation boundaries and emits +errors at use sites of non-`Sendable` values that have already been transferred +to a different isolation domain. + +This change makes the motivating example valid code, because the `client` +variable does not have any further uses after it's transferred to the +`ClientStore.shared` actor through the call to `addClient`. If we were to modify +`openNewAccount` to call a method on `client` after the call to `addClient`, the +code would be invalid since a non-`Sendable` value that had already been +transferred from a non-isolated context to an actor-isolated context could be +accessed concurrently: + +```swift +func openNewAccount(name: String, initialBalance: Double) async { + let client = Client(name: name, initialBalance: initialBalance) + await ClientStore.shared.addClient(client) + client.logToAuditStream() // Error! Already transferred into clientStore's isolation domain... this could race! +} +``` + +After the call to `addClient`, any other non-`Sendable` value that is statically +proven to be impossible to reference from `client` can still be used safely. We +can prove this property using the concept of *isolation regions*. An isolation +region is a set of values that can only ever be referenced through other values +within that set. Formally, two values $x$ and $y$ are defined to be within the +same isolation region at a program point $p$ if: + +1. $x$ may alias $y$ at $p$. +2. $x$ or a property of $x$ might be referenceable from $y$ via chained access of $y$'s properties at $p$. + +This definition ensures that non-`Sendable` values in different isolation +regions can be used concurrently, because any code that uses $x$ cannot affect +$y$. Lets consider a further example: + +```swift +let john = Client(name: "John", initialBalance: 0) +let joanna = Client(name: "Joanna", initialBalance: 0) + +await ClientStore.shared.addClient(john) +await ClientStore.shared.addClient(joanna) // (1) +``` + +The above code creates two new `Client` instances. It's impossible for +`john` to reference `joanna` and vice versa, so these two values belong to +different isolation regions. Values in different isolation regions can be +used concurrently, so the use of `joanna` at `(1)`, which may be executing +concurrently with some code inside `ClientStore.shared` that accesses `john`, +is safe from data races. + +In contrast, if we add a `friend` property to `Client` and assign `joanna` to +`john.friend`: + +```swift +let john = Client(name: "John", initialBalance: 0) +let joanna = Client(name: "Joanna", initialBalance: 0) + +john.friend = joanna // (1) + +await ClientStore.shared.addClient(john) +await ClientStore.shared.addClient(joanna) // (2) +``` + +After the assignment at point `(1)`, `joanna` can be referenced through +`john.friend`, so `john` and `joanna` must be in the same isolation region at +`(1)`. The access to `joanna` at point `(2)` can be executing concurrently with +code inside `ClientStore.shared` that accesses `john.friend`. Using `joanna` at +point `(2)` is diagnosed as a potential data race. + +## Detailed Design + +NOTE: While this proposal contains rigorous details that enable the compiler to +prove the absence of data races, programmers will not have to reason about +regions at this level of detail. The compiler will allow transfers of non-`Sendable` values between +isolation domains where it can prove they are safe and will emit diagnostics +when it cannot at potential concurrent access points so that programmers don't +have to reason through the data flow themselves. + +### Isolation Regions + +#### Definitions + +An *isolation region* is a set of non-`Sendable` values that can only be aliased +or reachable from values that are within the isolation region. An isolation +region can be associated with a specific *isolation domain* associated with a +task, protected by an actor instance or a global actor, or disconnected from any +specific isolation domain. As the program executes, each isolation region can be +merged with other isolation regions as new values begin to alias or be reachable +from each other. + +Isolation regions and isolation domains are not concepts that are explicitly +denoted in source code. To help explain the concepts throughout this proposal, +isolation regions and their isolation domains will be written in comments in +the following notation: + +* `[(a)]`: A single disconnected region with a single value. + +* `[{(a), actorInstance}]`: A single region that is isolated to actorInstance. + +* `[(a), {(b), actorInstance}]`: Two values in separate isolation regions. a's + region is disconnected but b's region is assigned to the isolation domain of + the actor instance `actorInstance`. + +* `[{(x, y), @OtherActor}, (z), (w, t)]`: Five values in three separate + isolation regions. `x` and `y` are within one isolation region that is + isolated to the global actor `@OtherActor`. `z` is within its own + disconnected isolation region. `w` and `t` are within the same disconnected + region. + +* `[{(a), Task1}]`: A single region that is part of `Task1`'s + isolation domain. + +#### Rules for Merging Isolation Regions + +Isolation regions are merged together when the program introduces a potential +alias or access path to another value. This can happen through function calls, +and assignments. Many expression forms are sugar for a function application, +including property accesses. + +Given a function $f$ with arguments $a_{i}$ and result that is assigned to +variable $y$: + +$$ +y = f(a_{0}, ..., a_{n}) +$$ + +1. All regions of non-`Sendable` arguments $a_{i}$ are merged into one larger + region after $f$ executes. +2. If any of $a_{i}$ are non-`Sendable` and $y$ is non-`Sendable`, then $y$ is in + the same merged region as $a_{i}$. If all of the $a_{i}$ are `Sendable`, + then $y$ is within a new disconnected region that consists only of $y$. +3. If $y$ is not a new variable, i.e. it's mutable, then + + a) If $y$ was previously captured by reference in a closure, then the assignment + to $y$ merges $y$'s new region into its old region. + + b) If $y$ was not captured by reference, then $y$'s old region is + forgotten. + +The above rules are conservative; without any further annotations, we must assume: +* In the implementation of $f$, any $a_{i}$ could become reachable from $a_{j}$. +* $y$ could be one of the $a_{i}$ values or alias contents of $a_{i}$. +* If $y$ was captured by reference in a closure and then assigned a new value, + calling the closure could reference $y$'s new value. + +See the future directions section for additional annotations that enable more +precise regions. + +##### Examples + +Now lets apply these rules to some specific examples in Swift code: + +* **Initializing a `let` or `var` binding**. ``let y = x, var y = x``. Initializing + a let or var binding `y` with `x` results in `y` being in the same region as + `x`. This follows from rule `(2)` since formally a copy is equivalent to calling a + function that accepts `x` and returns a copy of `x`. + + ```swift + func bindingInitialization() { + let x = NonSendable() + // Regions: [(x)] + let y = x + // Regions: [(x, y)] + let z = consume x + // Regions: [(x, y, z)] + } + ``` + + Note that whether or not `x` is in the region after `consume x` does not + change program semantics. A valid program must still obey the no-reuse + constraints of `consume`. + +* **Assigning a `var` binding**. ``y = x``. Assigning a var binding `y` with `x` + results in `y` being in the same region as `x`. If `y` is not captured by + reference in a closure, then `y`'s previous assigned region is forgotten due + to `(3)(b)`: + + ```swift + func mutableBindingAssignmentSimple() { + var x = NonSendable() + // Regions: [(x)] + let y = NonSendable() + // Regions: [(x), (y)] + x = y + // Regions: [(x, y)] + let z = NonSendable() + // Regions: [(x, y), (z)] + x = z + // Regions: [(y), (x, z)] + } + ``` + + In contrast if `y` was captured in a closure by reference, then `y`'s former + region is merged with the region of `x` due to `(3)(a)`. + + ```swift + // Since we pass x as inout in the closure, the closure has to capture x by + // reference. + func mutableBindingAssignmentClosure() { + var x = NonSendable() + // Regions: [(x)] + let closure = { useInOut(&x) } + // Regions: [(x, closure)] + let y = NonSendable() + // Regions: [(x, closure), (y)] + x = y + // Regions: [(x, closure, y)] + } + ``` + +* **Accessing a non-`Sendable` property of a non-`Sendable` value**. + ``let y = x.f``. Accessing a property `f` on a non-`Sendable` value `x` + results in a value `y` that must be in the same region as `x`. This follows + from `(2)` since formally a property access is equivalent to calling a getter + passing `x` as `self`. Importantly, this property forces all non-`Sendable` + types to form one large region containing their non-`Sendable` state: + + ```swift + func assignFieldToValue() { + let x = NonSendableStruct() + // Regions: [(x)] + let y = x.field + // Regions: [(x, y)] + } + ``` + +* **Setting a non-`Sendable` property of a non-`Sendable` value**. ``y.f = x`` + Assigning `x` into a property `y.f` results in `y` and `y.f` being in the + same region as `x`. This again follows from `(2)`: + + ```swift + func assignValueToField() { + let x = NonSendableStruct() + // Regions: [(x)] + let y = NonSendable() + // Regions: [(x), (y)] + x.field = y + // Regions: [(x, y)] + } + ``` + +* **Capturing non-`Sendable` values by reference in a closure**. ``closure = { + useX(x); useY(y) }``. Capturing non-`Sendable` values `x` and `y` results in + `x` and `y` being in the same region. This is a consequence of `(2)` since + `x` and `y` are formally arguments to the closure formation. This + also means that the closure must be part of that same region: + + ```swift + func captureInClosure() { + let x = NonSendable() + let y = NonSendable() + // Regions: [(x), (y)] + let closure = { print(x); print(y) } + // Regions: [(x, y, closure)] + } + ``` + +* **Function arguments in the body of a function**. Given a function `func + transfer(x: NonSendable, y: NonSendable) async`, in the body of + `transfer`, `x` and `y` are considered to be within the same region. Since + `self` is a function argument to methods, this implies that when `self` is + non-`Sendable` all method arguments must be in the same region as `self`: + + ```swift + func transfer(x: NonSendable, y: NonSendable) { + // Regions: [(x, y)] + let z = NonSendable() + // Regions: [(x, y), (z)] + f(x, z) + // Regions: [(x, y, z)] + } + ``` + +#### Control Flow + +Isolation regions are also affected by control flow. Let $x$ and $y$ +be two values that are used in a control flow statement. After the +control flow statement, the regions of $x$ and $y$ are merged if any +of the blocks within the statement merge the regions of $x$ and $y$. +For example: + +```swift +// Regions: [(x), (y)] +var x: NonSendable? = NonSendable() +var y: NonSendable? = NonSendable() +if ... { + // Regions: [(x), (y)] + x = y + // Regions: [(x, y)] +} else { + // Regions: [(x), (y)] +} + +// Regions: [(x, y)] +``` + +Because the first block of the `if` statement assigns `x` to `y`, causing +their regions to be merged within that block, `x` and `y` are in the +same region after the `if` statement. + +This rule is conservative since it is always safe to consider two values +that are disconnected from each other as if they are isolated together. The +only effect would be the rejection of programs that we otherwise could accept. + +The above description of regions naturally allows the definition of an +optimistic forward dataflow problem that allows us to determine at every point +of the program the isolation region that a value belongs to. We outline this +dataflow in more detail in an [appendix](#isolation-region-dataflow) to this proposal. + +### Transferring Values and Isolation Regions + +As defined above, all non-`Sendable` values in a Swift program belong to some +isolation region. An isolation region is isolated to an actor's isolation +domain, a task's isolation domain, or disconnected from any specific isolation +domain: + +```swift +actor Actor { + // 'field' is in an isolation region that is isolated to the actor instance. + var field: NonSendable + + func method() { + // 'ns' is in a disconnected isolation region. + let ns = NonSendable() + } +} + +func nonisolatedFunction() async { + // 'ns' is in a disconnected isolation region. + let ns = NonSendable() +} + +// 'globalVariable' is in a region that is isolated to @GlobalActor. +@GlobalActor var globalVariable: NonSendable + +// 'x' is isolated to the task that calls taskIsolatedArgument. +func taskIsolatedArgument(_ x: NonSendable) async { ... } +``` + +As the program executes, an isolation region can be passed across isolation +boundaries, but an isolation region can never be accessed by multiple +isolation domains at once. When a region $R_{1}$ is merged into another region +$R_{2}$ that is isolated to an actor, $R_{1}$ becomes protected by +that isolation domain and cannot be passed or accessed across isolation +boundaries again. + +The following code example demonstrates merging a disconnected region into a +region that is `@MainActor` isolated: + +```swift +@MainActor func transferToMainActor(_ t: T) async { ... } + +func assigningIsolationDomainsToIsolationRegions() async { + // Regions: [] + + let x = NonSendable() + // Regions: [(x)] + + let y = x + // Regions: [(x, y)] + + await transferToMainActor(x) + // Regions: [{(x, y), @MainActor}] + + print(y) // Error! +} +``` + +Passing `x` into `transferToMainActor` introduces a potential alias to `x` +from any `@MainActor`-isolated state, because the implementation of +`transferToMainActor` can store `x` into any state within that isolation +domain. So, the region containing `x` must be merged into the `@MainActor`'s +region. Accessing `y` after that merge is an error because `x` and `y` are now +both effectively `@MainActor` isolated, and the access occurs from outside the +`@MainActor`. + +Formally, when we pass a non-`Sendable` value $v$ into a function $f$ and the +call to $f$ crosses an isolation boundary, then we say that $v$ and $v$'s +region are *transferred* into $f$. During the execution of $f$, the only way to +reference $v$ or any value in the same region as $v$ is through the parameter +bound to $v$ in the implementation of $f$. This deep structural isolation +guarantees that values in a region cannot be accessed concurrently. + +In this proposal, we are defining the default convention for passing +non-`Sendable` values across isolation boundaries as being a transfer +operation. This does not apply when calling async functions from within the same +isolation domain. To do so would require an explicit transferring modifier which +is described in the [Future Directions](#transferring-parameters) section below. + +### Taxonomy of Isolation Regions + +There are four types of isolation regions that a non-`Sendable` value can belong +to that determine the rules for transferring value over an isolation boundary. + +#### Disconnected Isolation Regions + +A *disconnected isolation region* is a region that consists only of +non-`Sendable` values and is not associated with a specific isolation +domain. A value in a disconnected region can be transferred to another +isolation domain as long as the value is used uniquely by said isolation +domain and never used later outside of that isolation domain lest we introduce +races: + +```swift +@MainActor func transferToMainActor(_ t: T) async { ... } + +actor Actor { + func method() async { + let x = NonSendable() + // Regions: [(x)] + + await transferToMainActor(x) + // Regions: [{(x), @MainActor}] + + print(x) // Error! x being used outside of @MainActor isolated code. + } +} +``` + +#### Actor Isolated Regions + +An *actor isolated region* is a region that is strongly bound to a specific +actor's isolation domain. Since the region is tied to an actor's isolation +domain, the values of the region can *never* be transferred into another +isolation domain since that would cause the non-`Sendable` value to be used by +code both inside and outside the actor's isolation domain allowing for races: + +```swift +actor Actor { + var nonSendable: NonSendable +} + +@MainActor func actorRegionExample() async { + let a = Actor() + // Regions: [{(a.nonSendable), a}] + + let x = await a.nonSendable // Error! + + await transferToMainActor(a.nonSendable) // Error! +} +``` + +In the above code example, `x` must be in the actor `a`'s region because it +aliases actor-isolated state, making `x` effectively isolated to `a`. The +initialization is invalid, because `x` is not usable from a `@MainActor` +context. Similarly, attempting to transfer actor-isolated state into another +isolation domain is invalid. + +The parameters of an actor method or a global actor isolated function are +considered to be within the actor's region. This is since a caller can pass +actor isolated state as an argument to such a method or function. This implies +that parameters of actor isolated methods and functions can not be transferred +like other values in actor isolation regions. + +The objects that make up an actor region varies depending on the kind of actor: + +* **Actor**. An actor region for an actor contains the actor's non-`Sendable` + fields and any values derived from the actor's fields. + + ```swift + class NonSendableLinkedList { + var next: NonSendableLinkedList? + } + + actor Actor { + var listHead: NonSendableLinkedList + + func method() async { + // Regions: [{(self.listHead, self.listHead.next, ...), self}] + + let x = self.listHead + // Regions: [{(x, self.listHead, self.listHead.next, ...), self}] + + let z = self.listHead.next! + // Regions: [{(x, z, self.listHead, self.listHead.next, ...), self}] + ... + } + } + ``` + + In the above example, `x` is in `self`'s region because it aliases + non-`Sendable` state isolated to `self`, and `z` is in `self`'s region + because the value of `next` is reachable from `self.listHead`. + +* **Global Actor**. An actor region for a global actor contains any global + variables isolated to the global actor, all instances of nominal types + isolated to the global actor, and all values derived from the fields of the + isolated global variable or nominal types. + + ```swift + @GlobalActor var firstList: NonSendableLinkedList + @GlobalActor var secondList: NonSendableLinkedList + + @GlobalActor func useGlobalActor() async { + // Regions: [{(firstList, secondList), @GlobalActor}] + + let x = firstList + // Regions: [{(x, firstList, secondList), @GlobalActor}] + + let y = secondList.listHead.next! + // Regions: [{(x, firstList, secondList, y), @GlobalActor}] + ... + } + ``` + + In the above code example `x` is in `@GlobalActor`'s region because it + aliases `@GlobalActor`-isolated state, and `y` is in `@GlobalActor`'s region + because it aliases a value that's reachable from `@GlobalActor`-isolated + state. + +An operation to disconnect a value from an actor region in order to transfer +it to another isolation domain is out of the scope of this proposal. A +potential extension to enable this is described in the [Future Directions](disconnected-fields-and-the-disconnect-operator). + +#### Task Isolated Regions + +A task isolated isolation region consists of values that are isolated to a +specific task. This can only occur today in the form of the parameters of +nonisolated asynchronous functions since unlike actors, tasks do not have +non-`Sendable` state that can be isolated to them. Similarly to actor isolated +regions, a task isolated region is strongly tied to the task so values within +the task isolated region cannot be transferred out of the task: + +```swift +@MainActor func transferToMainActor(_ x: NonSendable) async { ... } + +func nonIsolatedCallee(_ x: NonSendable) async { ... } + +func nonIsolatedCaller(_ x: NonSendable) async { + // Regions: [{(x), Task1}] + + // Not a transfer! Same Task! + await nonIsolatedCallee(x) + + // Error! + await transferToMainActor(x) +} +``` + +In the example above, `x` is in a task isolated region. Since +`nonIsolatedCallee` will execute on the same task as `nonIsolatedCaller`, they +are in the same isolation domain and a transfer does not occur. In contrast, +`transferToMainActor` is in a different isolation domain so passing `x` to it is +a transfer resulting in an error. + +#### Invalid Isolation Regions + +An invalid isolation region is a region that results from conditional control +flow causing the merging of regions that can never be merged together due to +isolation properties. It is an error to use a value that is in an invalid +isolation region since statically the specific region that the value belongs to +can not be determined: + +```swift +func mergeTwoActorRegions() async { + let a1 = Actor() + // Regions: [{(), a1}] + let a2 = Actor() + // Regions: [{(), a1}, {(), a2}] + let x = NonSendable() + // Regions: [{(), a1}, {(), a2}, (x)] + + if await boolean { + await a1.useNS(x) + // Regions: [{(x), a1}, {(), a2}] + } else { + await a2.useNS(x) + // Regions: [{(), a1}, {(x), a2}] + } + + // Regions: [{(x), invalid}, {(), a1}, {(), a2}] +} +``` + +#### Merging Isolation Regions + +The behavior of merging two isolation regions depends on the kind of each +region. + +* **Disconnected and Disconnected**. Given two non-`Sendable` values in separate + disconnected regions, merging the regions produces one large disconnected + region. + + ```swift + let x = NonSendable() + // Regions: [(x)] + let y = NonSendable() + // Regions: [(x), (y)] + useValue(x, y) + // Regions: [(x, y)] + ``` + +* **Disconnected and Actor Isolated**. Merging a disconnected region and an + actor-isolated region expands the actor-isolated region with the values in + the disconnected region. This forces all values in the disconnected region + to be treated as if they are isolated to the actor. This can only occur when + calling a method on an actor or assigning into an actor's field: + + ```swift + func example1() async { + let x = NonSendable() + // Regions : [(x)] + + let a = Actor() + // Regions: [(x), {(a.field), a}] + + await a.useNonSendable(x) + // Regions: [{(x, a.field), a}] + + useValue(x) // Error! 'x' is effectively isolated to 'a' + + let y = NonSendable() + // Regions: [{(x, a.field), a}, (y)] + + a.field = y + // Regions: [{(x, a.field, y), a}] + + useValue(y) // Error! 'y' is effectively isolated to 'a' + } + ``` + +* **Disconnected and Task isolated**. Merging a disconnected region and a + task-isolated region expands the task-isolated region with the values in the + disconnected region. This forces all values in the disconnected region to be + treated like they are isolated to the task: + + ```swift + func nonIsolated(_ arg: NonSendable) async { + // Regions: [{(arg), Task1}] + let x = NonSendable() + // Regions: [{(arg), Task1}, (x)] + arg.doSomething(x) + // Regions: [{(arg, x), Task1}] + await transferToMainActor(x) // Error! 'x' is isolated to 'Task1' + } + ``` + +* **Actor isolated and Actor isolated**. Merging two actor-isolated regions + results in an invalid region. This can only occur via conditional control flow + since an actor isolated region cannot be transferred into another actor's + isolation region: + + ```swift + func test() async { + let a1 = Actor() + // Regions: [{(), a1}] + let a2 = Actor() + // Regions: [{(), a1}, {(), a2}] + let x = NonSendable() + // Regions: [{(), a1}, {(), a2}, (x)] + + if await boolean { + await a1.useNS(x) + // Regions: [{(x), a1}, {(), a2}] + } else { + await a2.useNS(x) + // Regions: [{(), a1}, {(x), a2}] + } + + // Regions: [{(x), invalid}, {(), a1}, {(), a2}] + } + ``` + + In the above example, `x` cannot be accessed from `test` after the `if` + statement since `x` is now in an invalid isolation domain. + +* **Actor Isolated and Task Isolated**. Merging an actor isolated region and + task isolated region results in an invalid isolation region. This occurs since + an actor isolated region and a task isolated region can run concurrently from + each other. Since values in either type of region cannot be transferred, this + can only occur through conditional control flow: + + ```swift + func nonIsolated(_ arg: NonSendable) async { + // Regions: [{(arg), Task1}] + let a = Actor() + // Regions: [{(), a}, {(arg), Task1}] + let x = NonSendable() + // Regions: [(x), {(), a}, {(arg), Task1}] + + if await boolean { + await a.useNS(x) + // Regions: [{(x), a}, {(arg), Task1}] + } else { + arg.useNS(x) + // Regions: [{(), a}, {(arg, x), Task1}] + } + + // Regions: [{(arg, x), invalid}, {(), a}, {(), Task1}] + } + ``` + +* **Task Isolated and Task Isolated**. Since task isolated isolation regions are + only introduced due to function arguments, it is impossible to have two + separate task isolated regions that could be merged. + +### Weak Transfers, `nonisolated` functions, and disconnected isolation regions + +When we transfer a value over an isolation boundary, the caller according to the +ownership conventions of Swift may still own the value despite it being illegal +for the caller to use the value due to region based isolation: + +```swift +class NonSendable { + deinit { print("deinit was called") } +} + +@MainActor func transferToMainActor(_ t: T) async { } + +actor MyActor { + func example() async { + // Regions: [{(), self}] + let x = NonSendable() + + // Regions: [(x), {(), self}] + await transferToMainActor(x) + // Regions: [{(x), @MainActor}, {(), self}] + + // Error! Since 'x' was transferred to @MainActor, we cannot use 'x' + // directly here. + useValue(x) // (1) + + print("After nonisolated callee") + + // But since example still owns 'x', the lifetime of 'x' ends here. (2) + } +} + +let a = MyActor() +await a.example() +``` + +In the above example, the program will first print out "After nonisolated +callee" and then "deinit was called". This is because even though +`nonIsolatedCallee` is transferred `x`'s region, `x` is still passed to +`nonIsolatedCallee` using Swift's default guaranteed ownership convention. This +implies that the caller from an ownership perspective still owns the memory of +the class implying the lifetime of `x` actually ends at `(1)` despite the caller +not being able to use `x` directly at that point. + +This illustrates how the transfer convention used when passing a value over an +isolation boundary is a *weak transfer* convention. A weak transfer convention +implies that one can still reference a value within the transferred region from +the original isolation domain, but one cannot access the value through the +reference. In contrast, a *strong transfer* convention would require that the +caller isolation domain cannot maintain even references to values in the +transferred isolation region. This would require transferring to always be a +1 +operation since to preserve this property we would always need to pass off +ownership from the caller to the callee to ensure that the callee cleans up the +region as shown in the example above. + +Requiring our transfer convention to be a strong convention would have several +unfortunate side-effects: + +* All async functions would by default take their parameters as owned. This + would be an ABI break and would also have the unfortunate consequence that the + bodies of asynchronous functions could never be marked as readonly or readnone + since they may need to invoke a deinit to end ownership of a value and deinits + may have unknown side-effects. + +* This would hurt the performance of asynchronous functions by increasing the + amount of ARC overhead required since unless we inline, there will be a cross + function call boundary copy that can not be eliminated. This in turn would + cause hits to code-size since to remedy this performance problem the inliner + would need to be more aggressive about inlining code. + +To achieve a *strong transfer* convention, one can use the *transferring* function +parameter annotation. Please see extensions below for more information about +*transferring*. + +Since our transfer convention is weak, a disconnected isolation region that +was transferred into an isolation domain can be used again if the isolation +domain no longer maintains any references to the region. This occurs with +`nonisolated` asynchronous functions. When we transfer a disconnected value into +a `nonisolated` asynchronous functions, the value becomes part of the function's +task isolated isolation domain for the duration of the function's +execution. Once the function finishes executing, we know that the value is no +longer isolated to the function since: + +* A `nonisolated` function does not have any non-temporary isolated state of its + own that the non-`Sendable` value could escape into. + +* Parameters in a task isolated isolation region cannot be transferred into a + different isolation domain that does have persistent isolated state. + +Thus the value in the caller's region again becomes disconnected once more and +thus can be used after the function returns and be transferred again: + +```swift +func nonIsolatedCallee(_ x: NonSendable) async { ... } +func useValue(_ x: NonSendable) { ... } +@MainActor func transferToMainActor(_ t: T) { ... } + +actor MyActor { + var state: NonSendable + + func example() async { + // Regions: [{(), self}] + + let x = NonSendable() + // Regions: [(x), {(), self}] + + // While nonIsolatedCallee executes the regions are: + // Regions: [{(x), Task}, {(), self}] + await nonIsolatedCallee(x) + // Once it has finished executing, 'x' is disconnected again + // Regions: [(x), {(), self}] + + // 'x' can be used since it is disconnected again. + useValue(x) // (1) + + // 'x' can be transferred since it is disconnected again. + await transferToMainActor(x) // (2) + + // Error! After transferring to main actor, permanently + // in main actor, so we can't use it. + useValue(x) // (3) + } +} +``` + +In the example above, we transfer `x` into `nonIsolatedCallee` and while +`nonIsolatedCallee` is executing are not allowed to access `x` in the +caller. Since `nonIsolatedCallee`'s execution ends immediately after it is +called, we are then allowed to use `x` again. + +### non-`Sendable` Closures + +Currently non-`Sendable` closures like other non-`Sendable` values are not +allowed to be passed over isolation boundaries since they may have captured +state from within the isolation domain in which the closure is defined. We would +like to loosen these rules. + +#### Captures + +A non-`Sendable` closure's region is the merge of its non-`Sendable` captured +parameters. As such a nonisolated non-`Sendable` closure that only captures +values that are in disconnected regions must itself be in a disconnected region +and can be transferred: + +```swift +let x = NonSendable() +// Regions: [(x)] +let y = NonSendable() +// Regions: [(x), (y)] +let closure = { useValues(x, y) } +// Regions: [(x, y, closure)] +await transferToMain(closure) // Ok to transfer! +// Regions: [{(x, y, closure), @MainActor}] +``` + +A non-`Sendable` closure that captures an actor-isolated value is considered to +be within the actor-isolated region of the value: + +```swift +actor MyActor { + var ns = NonSendable() + + func doSomething() { + let closure = { print(self.ns) } + // Regions: [{(closure, self.ns), self}] + await transferToMain(closure) // Error! Cannot transfer value in actor region. + } +} +``` + +When a non-`Sendable` value is captured by an actor-isolated non-`Sendable` +closure, we treat the value as being transferred into the actor isolation domain +since the value is now able to merged into actor-isolated state: + +```swift +@MainActor var nonSendableGlobal = NonSendable() + +func globalActorIsolatedClosureTransfersExample() { + let x = NonSendable() + // Regions: [(x), {(nonSendableGlobal), MainActor}] + let closure = { @MainActor in + nonSendableGlobal = x // Error! x is transferred into @MainActor and then accessed later. + } + // Regions: [{(nonSendableGlobal, x, closure), MainActor}] + useValue(x) // Later access is here +} + +actor MyActor { + var field = NonSendable() + + func closureThatCapturesActorIsolatedStateTransfersExample() { + let x = NonSendable() + // Regions: [(x), {(nonSendableGlobal), MainActor}] + let closure = { + self.field.doSomething() + x.doSomething() // Error! x is transferred into @MainActor and then accessed later. + } + // Regions: [{(nonSendableGlobal, x, closure), MainActor}] + useValue(x) // Later access is here + } +} +``` + +Importantly this ensures that APIs like `assumeIsolated` that take an +actor-isolated closure argument cannot introduce races by transferring function +parameters of nonisolated functions into an isolated closure: + +```swift +actor ContainsNonSendable { + var ns: NonSendableType = .init() + + nonisolated func unsafeSet(_ ns: NonSendableType) { + self.assumeIsolated { isolatedSelf in + isolatedSelf.ns = ns // Error! Cannot transfer a parameter! + } + } +} + +func assumeIsolatedError(actor: ContainsNonSendable) async { + let x = NonSendableType() + actor.unsafeSet(x) + useValue(x) // Race is here +} +``` + +Within the body of a non-`Sendable` closure, the closure and its non-`Sendable` +captures are treated as being Task isolated since just like a parameter, both +the closure and the captures may have uses in their caller: + +```swift +var x = NonSendable() +var closure = {} +closure = { + await transferToMain(x) // Error! Cannot transfer Task isolated value! + await transferToMain(closure) // Error! Cannot transfer Task isolated value! +} +``` + +#### Transferring + +A nonisolated non-`Sendable` synchronous or asynchronous closure that is in a +disconnected region can be transferred into another isolation domain if the +closure's region is never used again locally: + +```swift +extension MyActor { + func synchronousNonIsolatedNonSendableClosure() async { + // This is non-Sendable and nonisolated since it does not capture MyActor or + // any field of my actor. + let nonSendable = NonSendable() + let closure: () -> () = { + print("I am in a closure: \(nonSendable.name)") + } + + // We can safely transfer closure. + await transferClosure(closure) + + // If we were to invoke closure again, an error diagnostic would be + // emitted. + closure() // Error! + + // If we were to access nonSendable, an error diagnostic would be + // emitted. + nonSendable.doSomething() // Error! + } +} +``` + +An actor-isolated synchronous non-`Sendable` closure cannot be transferred to a +callsite that expects a synchronous closure. This is because as part of +transferring the closure, we have erased the specific isolation domain that the +closure was isolated to, so we cannot guarantee that we will invoke the value in +the actor's isolation domain: + +```swift +@MainActor func transferClosure(_ f: () -> ()) async { ... } + +extension Actor { + func isolatedClosure() async { + // This closure is isolated to actor since it captures self. + let closure: () -> () = { + self.doSomething() + } + + // When we transfer the closure, we have lost the specific actor that + // the closure belongs to so an error must be emitted! + await transferClosure(closure) // Error! + } +} +``` + +We may be able to accept this code in the future if we allowed for isolated +synchronous closures to propagate around the specific isolation domain that they +belonged to and dynamically swap to it. We discuss *dynamic isolation domains* +as an extension below. + +In contrast, one can transfer an actor-isolated synchronous non-`Sendable` +closure at a call site that expects an asynchronous function argument. This is +because the closure will be wrapped into an asynchronous thunk that will hop +onto the defining isolation domain of the closure: + +```swift +@MainActor func transferClosure(_ f: () async -> ()) async { ... } + +extension Actor { + func isolatedClosure() async { + // This closure is isolated to actor since it captures self. + let closure: () -> () = { + self.doSomething() + } + + // As part of transferring the closure, the closure is wrapped into an + // asynchronous thunk that will hop onto the Actor's executor. + await transferClosure(closure) + } +} +``` + +In the example above, since the closure is wrapped in the asynchronous thunk and +that thunk hops onto the Actor's executor before calling the closure, we know +that isolation to the actor is preserved when we call the synchronous closure. + +An actor-isolated asynchronous non-`Sendable` closure can be transferred since +upon the closure's invocation, we will always hop into the actor's isolation +domain: + +```swift +extension Actor { + func isolatedClosure() async { + // This async closure is isolated to actor since it captures self. + let closure: () async -> () = { + self.doSomething() + } + + // Since the closure is async, we can transfer it as much as we want + // since we will always invoke the closure within the actor's isolation + // domain... + await transferClosure(closure) + + // ... so this is safe as well. + await transferClosure(closure) + } +} +``` + +#### Closures and Global Actors + +If a closure uses values that are isolated from a global actor in any way, we +assume that the closure must also be isolated to that global actor: + +```swift +@MainActor func mainActorUtility() {} + +@MainActor func mainActorIsolatedClosure() async { + let closure = { + mainActorUtility() + } + // Regions: [{(closure), @MainActor}] + await transferToCustomActor(closure) // Error! +} +``` + +If `mainActorUtility` was not called within `closure`'s body then `closure` +would be disconnected and could be transferred: + +```swift +@MainActor func mainActorUtility() {} + +@MainActor func mainActorIsolatedClosure() async { + let closure = { + ... + } + // Regions: [(closure)] + await transferToCustomActor(closure) // Ok! +} +``` + +### KeyPath + +A non-`Sendable` keypath that is not actor-isolated is considered to be +disconnected and can be transferred into an isolation domain as long as the +value's region is not reused again locally: + +```swift +class Person { + var name = "John Smith" +} + +class Wrapper { + var root: Root + init(root: Root) { self.root = root } + func setKeyPath(_ keyPath: ReferenceWritableKeyPath, to value: T) { + root[keyPath: keyPath] = value + } +} + +func useNonIsolatedKeyPath() async { + let nonIsolated = Person() + // Regions: [(nonIsolated)] + let wrapper = Wrapper(root: nonIsolated) + // Regions: [(nonIsolated, wrapper)] + let keyPath = \Person.name + // Regions: [(nonIsolated, wrapper, keyPath)] + await transferToMain(keyPath) // Ok! + await wrapper.setKeyPath(keyPath, to: "Jenny Smith") // Error! +} +``` + +A non-`Sendable` keypath that is actor-isolated is considered to be in the +actor's isolation domain and as such cannot be transferred out of the actor's +isolation domain: + +```swift +@MainActor +final class MainActorIsolatedKlass { + var name = "John Smith" +} + +@MainActor +func useKeyPath() async { + let actorIsolatedKlass = MainActorIsolatedKlass() + // Regions: [{(actorIsolatedKlass.name), @MainActor}] + let wrapper = Wrapper(root: actorIsolatedKlass) + // Regions: [{(actorIsolatedKlass.name), @MainActor}] + let keyPath = \MainActorIsolatedKlass.name + // Regions: [{(actorIsolatedKlass.name, keyPath), @MainActor}] + await wrapper.setKeyPath(keyPath, to: "value") // Error! Cannot pass non-`Sendable` + // keypath out of actor isolated domain. +} +``` + +If a KeyPath captures any values then the KeyPath's region consists of a merge +of the captured values regions combined with the actor-isolation region of the +KeyPath if the KeyPath is isolated to an actor: + +```swift +class NonSendableType { + subscript(_ t: T) -> Bool { ... } +} + +func keyPathInActorIsolatedRegionDueToCapture() async { + let mainActorKlass = MainActorIsolatedKlass() + // Regions: [{(mainActorKlass), @MainActor}] + let keyPath = \NonSendableType.[mainActorKlass] + // Regions: [{(mainActorKlass, keyPath), @MainActor}] + await transferToMainActor(keyPath) // Error! Cannot transfer keypath in actor isolated region! +} + +func keyPathInDisconnectedRegionDueToCapture() async { + let ns = NonSendableType() + // Regions: [(ns)] + let keyPath = \NonSendableType.[ns] + // Regions: [(ns, keyPath)] + await transferToMainActor(ns) + useValue(keyPath) // Error! Use of keyPath after transferring ns +} +``` + +### Async Let + +When an async let binding is initialized with an expression that uses a +disconnected non-`Sendable` value, the value is treated as being transferred +into a `nonisolated` asynchronous callee that additionally allows for the value +to be transferred. If the value is used only by synchronous code and +`nonisolated` asynchronous functions, we allow for the value to be reused again +once the async let binding has been awaited upon: + +```swift +func nonIsolatedCallee(_ x: NonSendable) async -> Int { 5 } + +actor MyActor { + func example() async { + // Regions: [{(), self}] + let x = NonSendable() + // Regions: [(x), {(), self}] + async let value = nonIsolatedCallee(x) + x.integerField + // Regions: [{(x), Task}, {(), self}] + useValue(x) // Error! Illegal to use x here. + await value + // Regions: [(x), {(), self}] + useValue(x) // Ok! x is disconnected again so it can be used... + await transferToMainActor(x) // and even transferred to another actor. + } +} +``` + +If the disconnected value is transferred into an actor region, the value is +treated as if the value was transferred into the actor region at the point where +the async let is declared and is considered transferred even after the async let +has been awaited upon: + +```swift +// Regions: [] +let x = NonSendable() +// Regions: [(x)] +async let y = transferToMainActor(x) // Transferred here. +// Regions: [{(x), @MainActor}] +_ = await y +// Regions: [{(x), @MainActor}] +useValue(x) // Error! x is used after it has been transferred! +``` + +If a disconnected value is reused later in an async let initializer after +transferring it into an actor region, a use after transfer error diagnostic will +be emitted: + +```swift +// Regions: [] +let x = NonSendable() +// Regions: [(x)] +async let y = + transferToMainActorAndReturnInt(x) + + useValueAndReturnInt(x) // Error! Cannot use x after it has been transferred! +``` + +Since a disconnected value can only be transferred into one async let binding at +a time, a use after transfer diagnostic will be emitted if one initializes +multiple async let bindings in one statement with the same non-`Sendable` +disconnected value: + +```swift +// Regions: [] +let x = NonSendable() +// Regions: [(x)] +async let y = x, + z = x // Error! Cannot use x after it has been transferred! +``` + +A non-`Sendable` value that is in an actor isolation region is never allowed to +be used to initialize an async let binding since values in an async let +binding's initializer are allowed to be transferred into further callees: + +```swift +actor MyActor { + var field = NonSendable() + + func example() async { + // Regions: [{(self.field), self}] + async let value = transferToMainActor(field) // Error! Cannot transfer actor + // isolated field to + // @MainActor! + _ = await value + } +} +``` + +### Using transferring to simplify `nonisolated` actor initializers and actor deinitializers + +In [SE-0327](0327-actor-initializers.md), a flow sensitive diagnostic +was introduced to ensure that one can directly access stored properties of `self` +in `nonisolated` actor designated initializers and actor deinitializers despite +the methods not being isolated to self. The diagnostic set out a model where +initially `nonisolated` self is stated to have a weaker form of isolation that +relies on having exclusive access to self. While self is in that state, one is +allowed to access stored properties of self, but once self has escaped that +property is lost and self becomes nonisolated preventing one from accessing its +stored properties without using synchronization. In this proposal, we subsume +that proposal into the region based isolation model and eliminate the need for a +separate flow sensitive diagnostic. + +In Swift's concurrency model, an actor is Sendable since one can only access the +actor's internal state from the actor's executor. If the actor is nonisolated to +the current function this implies one must hop on to the actor's executor to +safely access state. In the case of an initializer or deinitializer with +nonisolated self, this creates a conundrum since we explicitly want to +initialize or deinitialize self's stored fields without synchronizing by hopping +onto the actor's executor. + +In order to implement these semantics, we model self as entering these methods +as a non-`Sendable` value that is strongly transferred into the method. Since +self is strongly transferred, we know that there cannot be any other references +in the program to self when the method begins executing and thus it is safe to +initially access the internal state of the actor directly. Self must initially +be a non-`Sendable` value since if self's storage can be accessed directly, then +passing self to another task could lead to a race on self's storage. To prevent +this possibility, when self escapes self becomes instantaneously +`Sendable`. Once self is `Sendable`, it is no longer safe to access self's +storage directly: + +```swift +actor Actor { + var nonSendableField: NonSendableType + + // self is passed into init using a strongly transferred convention. This means + // that it is unique and safe to access without worrying about concurrency. + init() { + // At this point, self is non-Sendable and we can access its fields directly. + self.nonSendableField = NonSendableType() + + // self is Sendable once callMethod is executed. This includes in callMethod itself. + self.callMethod() + + // Error! Cannot directly access storage of a Sendable actor. + self.nonSendableField.useValue() + } +} +``` + +In the example above, self starts as a unique non-`Sendable` typed value. Thus +it is safe for us to initialize `self.nonSendableField`. When self is passed +into `callMethod`, self becomes `Sendable`. Since self could have been +transferred to another task by callMethod, it is no longer safe to directly +access self's memory and thus we emit an error when we access +`self.nonSendableField`. + +Deinits work just like inits with one additional rule. Just like with initializers, +self is considered initially to be strongly transferred and non-`Sendable`. One +is allowed to access the `Sendable` stored properties of self while self is +non-`Sendable`. One can access the non-`Sendable` fields of self if one knows +statically that the non-`Sendable` fields are uniquely isolated to the self +instance. For the case of actors, this means that since the actor's state is +completely isolated only to that one actor instance we can touch non-`Sendable` +fields. But in the case of global actor isolated classes this is not true since +other global actor isolated class instances could also have a reference to the +same non-`Sendable` value since all global actor isolated instances are part of +the same isolation region: + +```swift +actor Actor { + var mutableNonSendableField: NonSendableType + let immutableNonSendableField: NonSendableType + var mutableSendableField: SendableType + let immutableSendableField: SendableType + + deinit { + _ = self.immutableSendableField // Ok + _ = self.mutableSendableField // Ok + // Safe to access since no other actor instances + _ = self.mutableNonSendableField // Ok + _ = self.immutableNonSendableField // Ok + + escapeSelfIntoNonIsolated(self) + + _ = self.immutableSendableField // Ok + _ = self.mutableSendableField // Error! Must be immutable. + _ = self.mutableNonSendableField // Error! Must be sendable + _ = self.immutableNonSendableField // Error! Must be sendable + } +} + +@MainActor class GlobalActorIsolatedClass { + var mutableNonSendableField: NonSendableType + let immutableNonSendableField: NonSendableType + var mutableSendableField: SendableType + let immutableSendableField: SendableType + + deinit { + _ = self.immutableSendableField // Ok + _ = self.mutableSendableField // Ok + _ = self.mutableNonSendableField // Error! Must be sendable! + _ = self.immutableNonSendableField // Error! Must be sendable! + + escapeSelfIntoNonIsolated(self) + + _ = self.immutableSendableField // Ok + _ = self.mutableSendableField // Error! Must be immutable! + _ = self.mutableNonSendableField // Error! Must be sendable! + _ = self.immutableNonSendableField // Error! Must be sendable! + } +} +``` + +### Using transferring to pass non-Sendable values to async isolated actor initializers + +In [SE-0327](0327-actor-initializers.md), all initializers with non-`Sendable` +arguments were only allowed to be called by delegating initializers: + +```swift +actor MyActor { + var x: NonSendableType + + // Can call this from anywhere. + init(_ arg: SendableType) { + self.init(NonSendableType(arg)) + } + + // Since this has a non-Sendable type, this designated initializer can only + // be called by other initializers like the delegating init above. + init(_ arg: NonSendableType) { + x = arg + } +} + +func constructActor() { + // Error! Cannot call init with non-`Sendable` argument from outside of + // MyActor. + let a = Actor(NonSendableType()) +} +``` + +Using isolation regions we can loosen this restriction and allow for +non-`Sendable` types to be passed to asynchronous initializers since our region +isolation rules guarantee that the caller will have transferred the value into +the initializer due to the isolation boundary: + +```swift +actor MyActor { + var x: NonSendableType + + init(_ arg: NonSendableType) async { + self.x = arg + } +} + +func makeActor() async -> MyActor { + // Regions: [] + let x = NonSendableType() + // Regions: [(x)] + let a = await MyActor(x) // Ok! + // Regions: [{(x), a}] + return a +} +``` + +In the above example, it is safe to pass `x` into `MyActor` despite `x` being +non-`Sendable` since if we were to use `x` afterwards, the compiler would error +since we would be using `x` from multiple isolation domains: + +```swift +func makeActor() async -> MyActor { + // Regions: [] + let x = NonSendableType() + // Regions: [(x)] + let a = await MyActor(x) // Ok! + // Regions: [{(x), a}] + x.doSomething() // Error! 'x' was transferred to a's isolation domain! + return a +} +``` + +Sadly synchronous initializers without additional work can still only take +`Sendable` types since there is not a guarantee that the non-`Sendable` types +that are passed to it is in its own region. In order to pass a non-`Sendable` +type to a synchronous initializer, one must mark the parameter with the +`transferring` function parameter modifier which is described below in [Future +Directions](#transferring-parameters). + +### Regions Merge when assigning to Struct and Tuple type var like bindings + +In this proposal, regions are not computed in a field sensitive manner. This +means that if we assign into a struct with multiple stored fields or a tuple +with multiple fields then assigning to one field affects the region of the +entire struct and requires us to merge into such types rather than assign since +otherwise we would lose the regions associated with the other fields: + +```swift +struct NonSendableBox { + var s1 = NonSendable() + var s2 = NonSendable() +} + +func mergeWhenAssignIntoMultiFieldStructField() async { + var box = NonSendableBox() + // Regions: [(box.s1, box.s1)] + let x = NonSendable() + // Regions: [(box.s1, box.s2), (x)] + let y = NonSendable() + // Regions: [(box.s1, box.s2), (x), (y)] + box.s1 = x + // Regions: [(box.s1, box.s2, x), (y)] + // If we used an assignment operation instead of a merge operation, + // this would cause us to lose that x was still in box.s1 and thus + // in box's region. + box.s2 = y + // Regions: [(box.s1, box.s2, x, y)] +} +``` + +In the above example, if we were to treat ``box.s2 = y`` as an assignment +instead of merge then we would be removing ``x`` from ``box``'s region which +would be unsound since ``x`` and ``box.s1`` still point at the same +reference. Unfortunately this has the affect that when we overwrite an element +of a var like struct, the previous region assigned to that field would have to +remain in the overall struct/tuple's region: + +```swift +func mergeWhenAssignIntoMultiFieldTupleField() async { + var box = (NonSendable(), NonSendable()) + // Regions: [(box.0, box.1)] + let x = NonSendable() + // Regions: [(box.0, box.1), (x)] + let y = NonSendable() + // Regions: [(box.0, box.1), (x), (y)] + box.0 = x + // Regions: [(box.0, box.1, x), (y)] + box.0 = y (1) + // Regions: [(box.0, box.1, x, y)] +} +``` + +In the above, even though we reassign ``box.0`` from ``x`` to ``y``, since we +must perform a merge, we must have that ``x`` is still in ``box``'s region. If +one assigns over the entire box though, one can still get an assign instead of a +region: + +```swift +func mergeWhenAssignIntoMultiFieldTupleField2() async { + var box = (NonSendable(), NonSendable()) + // Regions: [(box.0, box.1)] + let x = NonSendable() + // Regions: [(box.0, box.1), (x)] + let y = NonSendable() + // Regions: [(box.0, box.1), (x), (y)] + box.0 = x + // Regions: [(box.0, box.1, x), (y)] + box = (y, NonSendable()) + // Regions: [(box.0, box.1, y), (x)] +} +``` + +In order to mitigate this, we are able to be stricter with structs and tuples +that store a single field. In such a case, since the struct/tuple does not have +multiple fields updating the single field does not cause us to lose the region +of any other values: + +```swift +func assignWhenAssignIntoSingleFieldStruct() async { + var box = SingleFieldBox() + // Regions: [(box.field)] + let x = NonSendable() + // Regions: [(box.field), (x)] + let y = NonSendable() + // Regions: [(box.field), (x), (y)] + box.field = x + // Regions: [(box.field, x), (y)] + box.field = y + // Regions: [(box.field, y), (x)] +} +``` + +### Accessing `Sendable` fields of non-`Sendable` types after weak transferring + +Given a non-`Sendable` value `x` that has been weakly transferred, a `Sendable` +field `x.f` can be accessed in the caller after `x`'s transferring if the +compiler can statically prove that there cannot be any writes to `x.f` from +another concurrency domain. This is necessary since although `x.f` is +`Sendable`, if code from another concurrency domain can reference `x` in a +manner that allows for `x.f` to be written to, our initial access to `x.f` could +result in a race. Of course once the access is over, we are safe against races +due to the Sendability of `x.f`'s underlying type. The situations where this +occurs varies in between reference types and value types. We go through the +individual cases below. + +#### Classes + +If `x` is a reference type like a class, we only allow for `Sendable` let fields +of `x` to be accessed. This is safe since a let field can never be modified +after initialization implying that we cannot race on assignment to the field +when attempting to read from the field. We cannot allow for `Sendable` var +fields to be accessed due to the aforementioned possible race caused by another +concurrency domain writing to the `Sendable` field as we attempt to access it: + +```swift +class NonSendable { + let letSendable: SendableType + var varSendable: SendableType + let ns: NonSendable +} + +@MainActor func modifyOnMainActor(_ x: NonSendable) async { + x.varSendable = SendableType() +} + +func example() async { + let x = NonSendable() + await modifyOnMainActor(x) + _ = x.letSendable // This is safe. + _ = x.varSendable // Error! Use after transfer of mutable field that could + // race with a write to x.varSendable in modifyOnMainActor. +} +``` + +#### Immutable Bindings to Value Types + +If `x` is an immutable binding (e.x.: let) to a value type (e.x.: struct, tuple, +enum) then we allow for access to all of `x`'s `Sendable` subtypes. This is safe +because: + +1. `x` will be initialized by copying its initial value. This means that even if + `x`'s initial value is a field of a larger value, any modifications to the + other value will not cause `x`'s fields to point to different values. + +2. When `x` is transferred to a callee, `x` will be passed by value. Thus the + callee will receive a completely new value type albeit with copied + fields. This means that if the callee attempts to modify the value, it will + be modifying the new value instead of our caller value implying that we + cannot race against any assignment when accessing the field in our + caller. + + ```swift + struct NonSendableStruct { + let letSendableField: Sendable + var varSendableField: Sendable + let ns: NonSendable + } + + @MainActor func modifyOnMainActor(_ y: consuming NonSendableStruct) async { + // These assignments only affect our parameter, not x in the callee. + y.varSendableField = Sendable() + y = NonSendableStruct() + } + + func letExample() async { + let x = NonSendableStruct() + + await modifyOnMainActor(x) // Transfer x, giving useValueOnMainActor a + // shallow copy of x. + + // We do not race with the assignment in modifyOnMainActor since the + // assignment is to y, not to x. Since the fields are sendable, once + // we avoid the race on accessing the field, we are safe. + print(x.letSendableField) + print(x.varSendableField) + } + ``` + +3. If `x` is captured by reference, since `x` is a let it will be captured + immutably implying that we cannot write to `x.f`. + +#### Mutable Bindings to Value Types + +If `x` is a mutable binding (e.x.: `var`), then we can follow the same logic as +with our immutable bindings except in the case where `x` is captured by +reference. If `x` is captured by reference, it is captured mutably implying that +when accessing `x.f`, we could race against an assignment to `x.f` in the +closure: + +```swift +struct NonSendableStruct { + let letSendableField: Sendable + var varSendableField: Sendable + let ns: NonSendable +} + +@MainActor func invokeOnMain(_ f: () -> ()) async { + f() +} + +func unsafeMutableReferenceCaptureExample() async { + var x = NonSendableStruct() + let closure = { + x = NonSendableStruct(otherInit: ()) + } + await invokeOnMain(closure) + + _ = x.letSendableField // Error! Could race against write in closure! + _ = x.varSendableField // Error! Could race against write in closure! +} +``` + +This also implies that one cannot access `Sendable` computed properties or +functions later since those routines could perform a read like the above +resulting in a race against a write in the closure. + +## Source compatibility + +Region-based isolation opens up a new data-race safety hole when using APIs +change the static isolation in the implementation of a `nonisolated` function, +such as `assumeIsolated`, because values can become referenced by actor-isolated +state without any indication in the function signature: + +```swift +class NonSendable {} + +@MainActor var globalNonSendable: NonSendable = .init() + +nonisolated func stashIntoMainActor(ns: NonSendable) { + MainActor.assumeIsolated { + globalNonSendable = ns + } +} + +func stashAndTransfer() -> NonSendable { + let ns = NonSendable() + stashIntoMainActor(ns) + Task.detached { + print(ns) + } +} + +@MainActor func transfer() async { + let ns = stashAndTransfer() + await sendSomewhereElse(ns) +} +``` + +Without additional restrictions, the above code would be valid under this proposal, +but it risks a runtime data-race because the value returned from `stashAndTransfer` +is stored in `MainActor`-isolated state and send to another isolation domain to +be accessed concurrently. To close this hole, values must be sent into and out of +`assumeIsolated`. The base region-isolation rules accomplish this by treating +captures of isolated closures as a region merge, and the standard library annotates +`assumeIsolated` as requiring the result type `T` to conform to `Sendable`. This +impacts existing uses of `assumeIsolated`, so the change is staged in as warnings +under complete concurrency checking, which enables `RegionBasedIsolation` by default, +and an error in Swift 6 mode. + +## ABI compatibility + +This has no affect on ABI. + +## Future directions + +### Transferring Parameters + +In the above, we mentioned that the transferring of non-`Sendable` values as +discussed above is a callee side property since when analyzing an async callee, +we do not know if the callee's caller is from a different isolation domain or +not. This means that we must be conservative and treat all function parameters as +being in the same region and prevent transferring of function parameters. + +We could introduce a stronger form of transferring that is applied to a function +argument in the callee's signature and forces all callers to transfer the +parameter even if the caller is synchronous or is async but in the same +isolation domain. + +The transferred parameter is guaranteed to be strongly transferred so we know +that once the callee is called there are no other program visible references to +the value outside of the callee's parameter. The implications of this are: + +* Since the value is strongly isolated, it will be within its own disconnected + region separate from the regions of the other parameters: + + ```swift + actor Actor { + func method(_ x: transferring NonSendable, + _ y : NonSendable, + _ z : NonSendable) async { + // Regions: [(x), {(y, z), self}] + // Safe to transfer x since x is marked as transferring. + await transferToMainActor(x) + } + } + ``` + + +* Regardless of if the callee is synchronous or asynchronous, a non-`Sendable` + value that is passed as a transferring parameter cannot be used again locally. + + ```swift + actor Actor { + func transfer(_ t: transferring T) async {} + func method() async { + let a = NonSendable() + + // Pass a into transfer. Even though we are in the same + // isolation domain as transfer... + await transfer(a) + + // Since we transferred a, we are no longer allowed to use a here. Error! + useValue(a) + } + } + ``` + +* Given an asynchronous function, one can safely transfer the non-`Sendable` + parameter to another asynchronous function with a different isolation domain: + + ```swift + @MainActor func transferToMainActor(_ t: T) async {} + + actor Actor { + func method(_ x: transferring NonSendable) async { + // Regions: [(x)] + // Safe to transfer x since x is marked as transferring. + await transferToMainActor(x) + } + } + ``` + +* Given a transferring parameter of a synchronous function, the parameter's + strongly isolated implies that we can transfer it into `Task.init` or + `Task.detach`. + + ```swift + func someSynchronousFunction(_ x: transferring NonSendable) { + Task { + doSomething(x) + } + } + ``` + + if we did not have the strong isolation, then `x` could still be used in the + caller of `someSynchronousFunction`. + +* Due to the isolation of a transferring parameter, it is legal to have a + non-`Sendable` transferring parameter of a synchronous actor designated + initializer: + + ```swift + actor Actor { + var field: NonSendable + + init(_ x: transferring NonSendable) { + self.field = x + } + } + ``` + + Without the transferring argument modifier on `x`, it would not be safe to + store `x` into `self.field` since it may be introducing a value into the + actor's state that could be raced upon. + +#### Returns Isolated + +As discussed above, if a function takes non-`Sendable` parameters and has a +non-`Sendable` result, then the result is part of the merged region of the +function's parameters. This is not always the appropriate semantics since there +are APIs whose results will be in different regions than their parameters. As an +example of this, consider a function that performs control flow based off of +non-`Sendable` state and then returns a result: + +```swift +func example(_ x: NonSendable) async -> NonSendable? { + if x.boolean { + return NonSendable() + } + return nil +} +``` + +In the above, the result of `example` is a newly initialized value that has no +data dependence on the parameter `x`, but as laid out in this proposal, we +cannot express this. We propose the addition of a new function parameter +modifier called `returnsIsolated` that causes callers to treat the result of a +function as being in a disconnected region regardless of the inputs. As part of +this annotation, we would only allow for the callee to return a value that is in +a disconnected region preventing the returning of function arguments or in the +case of an actor any state related internally to the actor: + +```swift +actor Actor { + var field: NonSendableType + + func getValue() -> @returnsIsolated NonSendableType { + // Regions: [{(self.field), self}] + let x = NonSendableType() + // Regions: [(x), {(self.field), self}] + + if await booleanValue { + // Safe to do since 'x' is in a disconnected region. + return x + } + + // Error! Cannot return a value from the actor's region! + return field + } +} +``` + +Since the value returned is always in its own disconnected region, it can be +used in the caller isolation domain without triggering races: + +```swift +func getValueFromActor(_ a: Actor) async { + // Regions: [{(a.field), a}] + + // This is safe since we know that 'x' is independent of the actor. + let x = await a.getValue() + // Regions: [(x), {(a.field), a}] + + // So we could transfer it to another function if we wanted to. + await transferToMainActor(x) +} +``` + +> NOTE: @returnsIsolated is just a strawman syntax introduced for the purpose of +> expositing this extension. It is not an actual proposed or final syntax. + +#### Disconnected Fields and the Disconnect Operator + +Even though we can use `@returnsIsolated` to return a value from the Actor's +isolation domain, we have not specified a manner to safely return non-`Sendable` +values from the internal state of an Actor or GAIT. To do so, we introduce a new +type of field called a *disconnected field*. A disconnected field of an actor is +an actor isolated region that is separate from the normal actor's region. Since +it is separate from the other region of the actor, it cannot be reachable by the +other fields of the actor... but since it is an actor field, it cannot be +escaped from the actor without doing additional work. In order to escape such a +field, we introduce a new `disconnect` operation that consumes the disconnected +field and returns the field's value as a new disconnected region which is safe +to use as a `@returnsIsolated` result: + +```swift +actor MyActor { + disconnected var x: NonSendableType + + /// Reinitialize a field, returning the old value. + func reinitField() -> @returnsIsolated NonSendableType { + let result = disconnect x + x = NonSendableType() + return result + } +} +``` + +In the above example, we disconnect `x`'s value into `result`, reinitialize `x` +with a fresh value, and return the result. + +> NOTE: We may be able to reuse the `consume` operator for this purpose, but for +> the purposes of framing this as an extension, we introduce a new operator for +> simplicity. + +If the author forgets to update the disconnected field with a new value, a +control flow sensitive error will be emitted: + +```swift +actor MyActor { + disconnected var x: NonSendableType + + func reinitField() -> @returnsIsolated NonSendableType { + let result = disconnect x + + if booleanTest { + x = newValue + } else { + ... + } + + return result + } // Error! Must update disconnected field 'x' along all program paths after disconnecting! +} +``` + +In the above example, we emit an error since along the else path we do not +provide a new value for `x`. + +Since a disconnected field can only be initialized with a value from a +disconnected region implying that a field cannot be assigned to by a parameter +of an actor method unless the parameter is transferred: + +```swift +actor MyActor { + disconnected var x: NonSendableType + + /// Update the internal state to use a new value, returning the old value + func updateValue(_ newValue: transferring NonSendableType) -> @returnsIsolated NonSendableType { + let result = disconnect x + x = newValue + return result + } +} +``` + +since the parameter in the above example is transferred, it has a disconnected +region and thus can be assigned into the disconnected region. + +## Alternatives considered + +### Require users to audit all types for sendability + +We could require users to audit all of their non-`Sendable` types for +Sendability. This would create a large annotation burden on users that this +approach avoids. + +### Force weak transferring to be explicitly marked + +We could require transferred arguments to be explicitly marked with an operator +like consume or transfer. This is not needed since the APIs in question are +already explicitly marked as being a point of concurrency via `async`, `await`, +or `Task` implying that whether or not an API can result in transferring is +already explicitly marked. The only information that requiring an additional +explicit marker would provide the user is that the programmer can know without +reading the API surface that a transfer will occur here, information that can +also be ascertained by just reading the source. + +## Acknowledgments + +This proposal is based on work from the PLDI 2022 paper [A Flexible Type System for Fearless Concurrency](https://www.cs.cornell.edu/andru/papers/gallifrey-types/). + +Thanks to Doug Gregor, Kavon Farvardin for early assistance to Joshua during his +internship. + +Thanks to Doug Gregor and Holly Borla for our stimulating discussions and to +Holly for her help with editing! + +## Appendix + +### Isolation Region Dataflow + +The dataflow for computing *isolation regions* is defined as follows: + +1. The lattice of the dataflow consists of graphs where each value is a node and + each edge represents a statement that causes two values to be apart of the + same region. We partially order our lattice by stating that given a graph + `g1` and a graph `g2` then `g1 <= g2` only if `g1 U g2 = g1` where `U` is a + graph union operation. + +2. Control flow merges are defined by unions of graphs meaning that if there is + an edge in between two nodes in any predecessor control flow blocks, there is + an edge in the successor control flow block. + +3. We consider the top of the dataflow to be the empty graph consisting of + values that are all in their own independent regions and the bottom of our + dataflow to be a completely connected graph where all values are in the same + region. + +4. Since the dataflow is a forward optimistic dataflow, we initially treat + backedges as propagating the top graph. + +5. We can prove that our dataflow always converges since our transfer function + can be proven as monotonic since given two sets `g1`, `g2` with `g1 <= g2`, + we know that `F(g1) <= F(g2)` since any edges that we remove from `g1` must + also be removed from `g2` and any edges that we add will be added identically + to `g1` and `g2` since `g1` is a subset of `g2`. diff --git a/proposals/0415-function-body-macros.md b/proposals/0415-function-body-macros.md new file mode 100644 index 0000000000..12797b0a2f --- /dev/null +++ b/proposals/0415-function-body-macros.md @@ -0,0 +1,312 @@ +# Function Body Macros + +* Proposal: [SE-0415](0415-function-body-macros.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 6.0)** +* Feature Flag: `BodyMacros` +* Review: [pitch](https://forums.swift.org/t/function-body-macros/66471), [review](https://forums.swift.org/t/se-0415-function-body-macros/68847), [returned for revision](https://forums.swift.org/t/returned-for-revision-se-0415-function-body-macros/69114), [second review](https://forums.swift.org/t/se-0415-second-review-function-body-macros/71644), [acceptance](https://forums.swift.org/t/accepted-se-0415-function-body-macros/72013) + +## Table of contents + +* [Introduction](#introduction) +* [Proposed solution](#proposed-solution) +* [Detailed design](#detailed-design) + * [Declaring function body macros](#declaring-function-body-macros) + * [Implementing function body macros](#implementing-function-body-macros) + * [Composing function body macros](#composing-function-body-macros) + * [Type checking of functions involving function body macros](#type-checking-of-functions-involving-function-body-macros) +* [Source compatibility](#source-compatibility) +* [Effect on ABI stability](#effect-on-abi-stability) +* [Effect on API resilience](#effect-on-api-resilience) +* [Future directions](#future-directions) + * [Function body macros on closures](#function-body-macros-on-closures) +* [Alternatives considered](#alternatives-considered) + * [Eliminating preamble macros](#eliminating-preamble-macros) + * [Capturing the withSpan pattern in another macro role](#capturing-the-withspan-pattern-in-another-macro-role) + * [Type-checking bodies as they were written](#type-checking-bodies-as-they-were-written) +* [Revision history](#revision-history) + +## Introduction + +Macros augment Swift programs with additional code, which can include new declarations, expressions, and statements. One of the key ways in which one might want to augment code---synthesizing or updating the body of a function---is not currently supported by the macro system. One can create new functions that have their own function bodies, but not provide, augment, or replace function bodies for a function declared by the user. + +This proposal introduces *function body macros*, which do exactly that: allow the wholesale synthesis of function bodies given a declaration, as well as augmenting an existing function body with more functionality. This opens up a number of new use cases for macros, including: + +* Synthesizing function bodies given the function declaration and some metadata, such as automatically synthesizing remote procedure calls that pass along the provided arguments. +* Augmenting function bodies to perform logging/tracing, check preconditions, or establish invariants. +* Replacing function bodies with a new implementation based on the one provided. For example, moving the body into a closure that is executed somewhere else, or treating the body as written as a domain specific language that the macro "lowers" to executable code. + +## Proposed solution + +This proposal introduces *function body macros*, which are [attached macros](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md) that can augment a function (including initializers, deinitializers, and accessors) with a new body. For example, one could introduce a `Remote` macro that packages up arguments for a remote procedure call: + +```swift + @Remote + func f(a: Int, b: String) async throws -> String +``` + +which could expand the function to provide a body, e.g.: + +```swift +func f(a: Int, b: String) async throws -> String { + return try await remoteCall(function: "f", arguments: ["a": a, "b": b]) +} +``` + +One could also use a macro to introduce logging code on entry and exit to a function, expanding the following + +```swift +@Logged +func g(a: Int, b: Int) -> Int { + return a + b +} +``` + +into + +```swift +func g(a: Int, b: Int) -> Int { + log("Entering g(a: \(a), b: \(b))") + defer { + log("Exiting g") + } + return a + b +} +``` + +Or one could provide a macro that makes it easier to assume that a function that cannot be marked as `@MainActor` using [`assumeIsolated`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md): + +```swift +extension MyView: SomeDelegate { + @AssumeMainActor + nonisolated func onSomethingHappened(event: Event) { + myView.title = newTitle(processing: event) + } +} +``` + +which could expand to: + +```swift +extension MyView: SomeDelegate { + nonisolated func onSomethingHappened(event: Event) { + MainActor.assumeIsolated { + myView.title = newTitle(processing: event) + } + } +} +``` + +Function body macros can be applied to accessors as well, in which case they go on the accessor itself, e.g., + +```swift +var area: Double { + @Logged get { + return length * width + } +} +``` + +When using the shorthand syntax for get-only properties, a function body macro can be applied to the property itself: + +```swift +@Logged var area: Double { + return length * width +} +``` + +## Detailed design + +### Declaring function body macros + +Function body macros are declared with the `body` role, which indicate that they can be attached to any kind of function, and can produce the contents of a function body. For example, here are declarations for the macros used above: + +```swift +@attached(body) macro Remote() = #externalMacro(...) + +@attached(body) macro Logged() = #externalMacro(...) + +@attached(body) macro AssumeMainActor() = #externalMacro(...) +``` + +Like other attached macros, function body macros have no return type. + +### Implementing function body macros + +Body macros are implemented with a type that conforms to the `BodyMacro` protocol: + +```swift +/// Describes a macro that can create the body for a function. +public protocol BodyMacro: AttachedMacro { + /// Expand a macro described by the given custom attribute and + /// attached to the given declaration and evaluated within a + /// particular expansion context. + /// + /// The macro expansion introduces code block items that will become the body for the + /// given function. Any existing body will be implicitly ignored. + static func expansion( + of node: AttributeSyntax, + providingBodyFor declaration: some DeclSyntaxProtocol & WithOptionalCodeBlockSyntax, + in context: some MacroExpansionContext + ) throws -> [CodeBlockItemSyntax] +} +``` + +That function may have a function body, which will be replaced by the code items produced from the macro implementation. + +### Composing function body macros + +At most one `body` macro can be applied to a given function. It receives the function declaration to which it is attached as it was written in the source code and produces a new function body. + +### Type checking of functions involving function body macros + +When a function body macro is applied, the macro-expanded function body will need to be type checked when it is incorporated into the program. However, the function might already have a body that was written by the developer, which can be inspected by the macro implementation. The function body as written must be syntactically well-formed (i.e., it must conform to the [Swift grammar](https://docs.swift.org/swift-book/documentation/the-swift-programming-language/summaryofthegrammar/)) but will *not* be type-checked, so it need not be semantically well-formed. + +This approach follows what other attached macros do: they operate on the syntax of the declaration to which they are attached, and the declaration itself need not have been type-checked before the macro is expanded. However, this approach does lend itself to potential abuse. For example, one could create a `SQL` macro that expects the function body to be a SQL statement, then rewrites that into code that executes the query. For example, the input could be: + +```swift +@SQL +func employees(hiredIn year: Int) -> [String] { + SELECT + name + FROM + employees + WHERE + YEAR(hire_date) = year; +} +``` + +However, this would only work for places where the SQL grammar is a subset of the Swift grammar. Collapsing the same function into two lines would produce an error because it is not syntactically well-formed Swift: + +```swift +@SQL +func employees(hiredIn year: Int) -> [String] { + SELECT name FROM employees // error: consecutive statements on a line must be separated by ';' + WHERE YEAR(hire_date) = year; +} +``` + +The requirement for syntactic wellformedness should help rein in the more outlandish uses of function body macros, as well as making sure that existing tools that operate on source code will continue to work well even in the presence of body macros. + +## Source compatibility + +Function body macros introduce a new macro role into the existing attached macro syntax, and therefore does not have an impact on source compatibility. + +## Effect on ABI stability + +Macros are a source-to-source transformation tool that have no ABI impact. + +## Effect on API resilience + +Macros are a source-to-source transformation tool that have no effect on API resilience. + +## Future directions + +### Function body macros on closures + +Function body macros as presented in this proposal are limited to declared functions, initializers, deinitializers, and accessors. In the future, they could be expanded to apply to closures as well, e.g., + +```swift +@Traced(z) { (x, y) in + x + y +} +``` + +This extension would involve extending the `BodyMacro` protocol with another `expansion` method that accepts closure syntax. The primary challenge with applying function body macros to closures is the interaction with type inference, because closures generally occur within an expression and some of the macro arguments themselves might be part of the expression. In the example above, the `z` value could come from an outer scope and be the subject of type inference: + +```swift +f(0) { z in + @Traced(z) { (x, y) in + x + y + } +} +``` + +Macros are designed to avoid [multiply instantiating the same macro](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md#macro-expansion), and have existing limitations in place to prevent the type checker from getting into a position where it is not obvious which macro to expand or the same macro needs to be expanded multiple times. To extend function body macros to closures will require a solution to this type-checking issue, and might be paired with lifting other restrictions on (e.g.) freestanding declaration macros. + +### Preamble macros + +The first reviewed revision of this proposal contained *preamble* macros, which let a macro introduce code at the beginning of a function without changing the rest of the function body. Preamble macros aren't technically necessary, because one could always write a function body macro that injects the preamble code into an existing body. However, preamble macros provide several end-user benefits over function body macros for the cases where they apply: + +* Preamble macros can be composed, whereas function body macros cannot. +* Preamble macros don't change the code as written by the user, so they provide a better user experience (e.g., for diagnostics, code completion, and so on). + +Preamble macros would be expressed as its own attached macro role (`preamble`), implemented with a type that conforms to the `PreambleMacro` protocol. Details are available in [the prior revision](https://github.com/swiftlang/swift-evolution/blob/f1b9da80315578666352a7d6d40a9f6cc936f69a/proposals/0415-function-body-macros.md). + +Preamble macros have been moved out to Future Directions because they represent a possible future, but not an obviously right one: preamble macros might not add sufficient expressivity to cover the cost of the complexity they introduce, and another kind of macro (like the "wrapper" macro below) might provide a more reasonable tradeoff between expressivity and complexity. + +### Wrapper macros + +A number of use cases for body macros involve "wrapping" the existing body in additional logic. For example, consider an alternative formulation of the `Traced` macro (let's call it `@TracedWithSpan`) could make use of the [`withSpan` API](https://swiftpackageindex.com/apple/swift-distributed-tracing/1.0.1/documentation/tracing) such that a function such as: + +```swift +@TracedWithSpan("Doing complicated math") +func h(a: Int, b: Int) -> Int { + return a + b +} +``` + +will expand to: + +```swift +func h(a: Int, b: Int) -> Int { + withSpan("Doing complicated math") { + return a + b + } +} +``` + +This `withSpan` function used here is one instance of a fairly general pattern in Swift, where a function accepts a closure argument and runs it with some extra contextual parameters. As we with the `preamble` macro role mentioned above, we could introduce a special macro role that describes this pattern: the macro would not see the function body that was written by the developer at all, but would instead have a function value representing the body that it could call opaquely. For example, the `TracedWithSpan` example function `h` would expand to: + +```swift +func h(a: Int, b: Int) -> Int { + withSpan("Doing complicated math", body: h-impl) +} +``` + +With this approach, the original function body for `h` would be type-checked prior to macro expansion, and then would be handed off to the macro as an opaque value `h-impl` to be called by `withSpan`. The macro could introduce its own closure wrapping that body as needed, e.g., + +```swift +@TracedWithSpan("Doing complicated math", { span in + span.attributes["operation"] = "addition" +}) +func myMath(a: Int, b: Int) -> Int { + return a + b +} +``` + +could expand to: + +```swift +func myMath(a: Int, b: Int) -> Int { + return withSpan("Doing complicated math") { span in + span.attributes["operation"] = "addition" + return myMath-impl() + } +} +``` + +The advantage of this approach over allowing a `body` macro to replace a body is that we can type-check the function body as it was written, and only need to do so once---then it becomes a value of function type that's passed along to the underlying macro. Also like preamble macros, this approach can compose, because the result of one macro could produce another value of function type that can be passed along to another macro. [Python decorators](https://www.datacamp.com/tutorial/decorators-python) have been successful in that language for customizing the behavior of functions in a similar manner. + +## Alternatives considered + +### Type-checking bodies as they were written + +As noted previously, not checking the body of functions that was written by the user and then replaced by a `body` macro has some down sides. For one, it allows some abuse, where code that wouldn't make sense in Swift is permitted to be written by the user and then significantly altered by the `body` macro. Moreover, wherever the macro is performing some modification that makes ill-formed code into well-formed code (even by something as simple as introducing a `span` variable like `@Traced` does), tools that cannot reason about the macro expansion might be less useful: code completion won't know to provide `span` as a possible completion, nor will it know what type `span` would have. Therefore, the experience of writing code that makes use of `body` macros could be significantly worse than that for normal Swift code. + +On the other hand, type-checking the function bodies before macro expansion has other issues. Type checking is a significant part of compilation time, and having to type-check the body of a function twice---once before macro expansion, once after---could be prohibitively expensive. Type-checking the function body before macro expansion also limits what can be expressed by body macros, including making some use cases (like the `@Traced` macro described earlier) impossible to express without more extensions to the model. + +## Revision history + +* Revision 3: + * Narrowed the focus down to `body` macros. + * Moved preamble macros into Future Directions, added discussion of wrapper macros. +* Revision 2: + * Clarify that preamble macro-introduced local names can shadow names from outer scopes + * Clarify the effect of function body macros on single-expression functions and implicit returns +* Revision 1: + * Allow preamble macros to introduce names. + * Introduce `@AssumeMainActor` example macro for body macros that perform replacement. + * Switch `@Traced` example over to be a preamble macro with push/pop operations, so it can nicely introduce `span`. + * Allow function body macros to be applied to properties that use the shorthand getter syntax. diff --git a/proposals/0416-keypath-function-subtyping.md b/proposals/0416-keypath-function-subtyping.md new file mode 100644 index 0000000000..4816effb16 --- /dev/null +++ b/proposals/0416-keypath-function-subtyping.md @@ -0,0 +1,99 @@ +# Subtyping for keypath literals as functions + +* Proposal: [SE-0416](0416-keypath-function-subtyping.md) +* Authors: [Frederick Kellison-Linn](https://github.com/jumhyn) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.0)** +* Implementation: [apple/swift#39612](https://github.com/apple/swift/pull/39612) +* Review: ([pitch](https://forums.swift.org/t/pitch-generalize-keypath-to-function-conversions/52681)) ([review](https://forums.swift.org/t/se-0416-subtyping-for-keypath-literals-as-functions/68984)) ([acceptance](https://forums.swift.org/t/accepted-se-0416-subtyping-for-keypath-literals-as-functions/69241)) + +## Introduction + +Today, keypath literals can only be narrowly converted to a function which exactly matches the argument and return type. This proposal allows key path literals to partake in the full generality of the conversions we allow between arbitrary function types, so that the following code compiles without error: + +```swift +let _: (String) -> Int? = \.count +``` + +## Motivation + +[SE-0249](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0249-key-path-literal-function-expressions.md) introduced a conversion between key path literals and function types, which allowed users to write code like the following: + +```swift +let strings = ["Hello", "world", "!"] +let counts = strings.map(\.count) // [5, 5, 1] +``` + +However, SE-0249 does not quite live up to its promise of allowing the equivalent key path construction "wherever it allows (Root) -> Value functions." Function types permit conversions that are covariant in the result type and contravariant in the parameter types, but key path literals require exact type matches. This can lead to some potentially confusing behavior from the compiler: + +```swift +struct S { + var x: Int +} + +// All of the following are okay... +let f1: (S) -> Int = \.x +let f2: (S) -> Int? = f1 +let f3: (S) -> Int? = { $0.x } +let f4: (S) -> Int? = { kp in { root in root[keyPath: kp] } }(\S.x) +let f5: (S) -> Int? = \.x as (S) -> Int + +// But the direct conversion fails! +let f6: (S) -> Int? = \.x // <------------------- Error! +``` + +## Proposed solution + +Allow key path literals to be converted freely in the same manner as functions are converted today. This would allow the definition `f6` above to compile without error, in addition to allowing constructions like: + +```swift +class Base { + var derived: Derived { Derived() } +} +class Derived: Base {} + +let g1: (Derived) -> Base = \Base.derived +``` + +## Detailed design + +Rather than permitting a key path literal with root type `Root` and value type `Value` to only be converted to a function type `(Root) -> Value`, key path literals will be permitted to be converted to any function type which `(Root) -> Value` may be converted to. + +The actual key-path-to-function conversion transformation proceeds exactly as before, generating code with the following semantics (adapting an example from SE-0249): + +```swift +// You write this: +let f: (User) -> String? = \User.email + +// The compiler generates something like this: +let f: (User) -> String? = { kp in { root in root[keyPath: kp] } }(\User.email) +``` + +## Source compatibility + +This proposal allows conversions in some situations that were previously impossible. This can affect source compatibility because overloaded function calls may gain new viable overload candidates. + +In typical scenarios, these new candidates will be strictly worse than previous candidates because the new conversion is strictly less favorable. In situations such as: + +```swift +func evil(_: (T) -> U) { print("generic") } +func evil(_ x: (String) -> Bool?) { print("concrete") } + +evil(\String.isEmpty) +``` + +Swift will (without this proposal) prefer to call the generic function because the conversion necessary for the concrete function is invalid. With this proposal, Swift will still prefer to call the generic function because the concrete function requires an extra conversion (not only does the keypath need to be converted to a function, but the 'natural' type of the keypath function is `(String) -> Bool`, which requires another conversion to get to `(String) -> Bool?`). + +However, this is not always true. A newly-viable overload candidate may be disfavored for the key path conversion but favored for other reasons. This should be uncommon, and so the author expects this proposal will have a very small impact in practice, but this will need to be demonstrated as part of landing the proposal in a Swift release. + +## Effect on ABI stability + +N/A + +## Effect on API resilience + +N/A + +## Acknowledgements + +Thanks to [@ChrisOffner](https://forums.swift.org/u/chrisoffner) for kicking off this discussion on the forums to point out the inconsistency here, and to [@jrose](https://forums.swift.org/u/jrose) for assistance in exploring some strange edge cases in the existing behavior of this feature. diff --git a/proposals/0417-task-executor-preference.md b/proposals/0417-task-executor-preference.md new file mode 100644 index 0000000000..9f0d5848fd --- /dev/null +++ b/proposals/0417-task-executor-preference.md @@ -0,0 +1,798 @@ +# Task Executor Preference + +* Proposal: [SE-0417](0417-task-executor-preference.md) +* Author: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [John McCall](https://github.com/rjmccall), [Franz Busch](https://github.com/FranzBusch) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-task-executor-preference/68191)), ([review](https://forums.swift.org/t/se-0417-task-executor-preference/68958)), ([acceptance](https://forums.swift.org/t/accepted-se-0417-task-executor-preference/69705)) + +## Introduction + +Swift Concurrency uses tasks and actors to model concurrency and primarily relies on actor isolation to determine where a specific piece of code shall execute. + +The recent introduction of custom actor executors in [SE-0392](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md) allows specifying a `SerialExecutor` implementation code should be running on while isolated to a specific actor. This allows developers to gain some control over exact threading semantics of actors, by e.g. making sure all work made by a specific actor is made on a dedicated queue or thread. + +Today, the same flexibility is not available to tasks in general, and nonisolated asynchronous functions are always executed on the default global concurrent thread pool managed by Swift concurrency. + +## Motivation + +Custom actor executors allow developers to customize where execution of a task “on” an actor must happen (e.g. on a specific queue or thread, represented by a `SerialExecutor`), the same capability is currently missing for code that is not isolated to an actor. + +Notably, since Swift 5.7’s [SE-0338: Clarify the Execution of Non-Actor-Isolated Async Functions](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0338-clarify-execution-non-actor-async.md), functions which are not isolated to an actor will always hop to the default global concurrent executor, which is great for correctness and understanding the code and avoids “hanging onto” actors longer than necessary. This is also a desirable semantic for code running on the `MainActor` calling into `nonisolated` functions, since it allows the main actor to be quickly freed up to proceed with other work, however it has a detrimental effect on applications which want to *avoid* hops in order to maximize request processing throughput. This is especially common with event-loop based systems, such as network servers or other kinds of tight request handling loops. + +As Swift concurrency is getting adopted in a wider variety of performance sensitive codebases, it has become clear that the lack of control over where nonisolated functions execute is a noticeable problem. +At the same time, the defensive "hop-off" semantics introduced by [SE-0338](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0338-clarify-execution-non-actor-async.md) are still valuable, but sometimes too restrictive and some use-cases might even say that the exact opposite behavior might be desirable instead. + +This proposal acknowledges the different needs of various use-cases, and provides a new flexible mechanism for developers to tune their applications and avoid potentially unnecessary context switching when possible. + +## Proposed solution + +We propose to introduce an additional layer of control over where a task can be executed, and have this executor setting be “sticky” to the task. + +**Currently** the decision where an async function or closure is going to execute is binary: + +``` +// `func` execution semantics before this proposal + +[ func / closure ] - /* where should it execute? */ + | + +--------------+ +==========================+ + +- no - | is isolated? | - yes -> | default (actor) executor | + | +--------------+ +==========================+ + | + | +==========================+ + +-------------------------------> | on global conc. executor | + +==========================+ +``` + +This proposal introduces a way to control hopping off to the global concurrent pool for `nonisolated` functions and closures. This is expressed as **task executor preference** and is sticky to the task and entire *structured task hierarchy* created from a task with a specified preference. This changes the current decision diagram to the following: + +``` +// `func` execution semantics with this proposal + +[ func / closure ] - /* where should it execute? */ + | + +--------------+ +===========================+ + +-------- | is isolated? | - yes -> | actor has unownedExecutor | + | +--------------+ +===========================+ + | | | + | yes no + | | | + | v v + | +=======================+ /* task executor preference? */ + | | on specified executor | | | + | +=======================+ yes no + | | | + | | v + | | +==========================+ + | | | default (actor) executor | + | v +==========================+ + v +==============================+ +/* task executor preference? */ ---- yes ----> | on Task's preferred executor | + | +==============================+ + no + | + v + +===============================+ + | on global concurrent executor | + +===============================+ +``` + +In other words, this proposal introduces the ability to specify where code may execute from a Task, and not just by using a custom actor executor, +and even influence the thread use of default actors. + +With this proposal a **`nonisolated` function** will execute, as follows: + +* if task preference **is not** set: + * it is equivalent to current semantics, and will execute on the global concurrent executor, + +* if a task preference **is** set, + * **(new)** nonisolated functions will execute on the selected executor. + + +The preferred executor also may influence where **actor-isolated code** may execute, specifically: + +- if task preference **is** set: + - **(new)** default actors will use the task's preferred executor + - actors with a custom executor execute on that specified executor (i.e. "preference" has no effect), and are not influenced by the task's preference + +The task executor preference can be specified either at task creation time: + +```swift +Task(executorPreference: executor) { + // starts and runs on the 'executor' + await nonisolatedAsyncFunc() +} + +Task.detached(executorPreference: executor) { + // starts and runs on the 'executor' + await nonisolatedAsyncFunc() +} + +await withDiscardingTaskGroup { group in + group.addTask(executorPreference: executor) { + // starts and runs on the 'executor' + await nonisolatedAsyncFunc() + } +} + +func nonisolatedAsyncFunc() async -> Int { + // if the Task has a specific executor preference, + // runs on that 'executor' rather than on the default global concurrent executor + return 42 +} +``` + +or, for a specific scope using the `withTaskExecutorPreference` method. Notably, the task executor preference is in effect for the entire structured task hierarchy while running in a task or scope where a task executor preference is set. For example, the following snippet illustrates child tasks created inside of a `withTaskExecutorPreference`: + +```swift +await withTaskExecutorPreference(executor) { + // if not already running on specified 'executor' + // the withTaskExecutorPreference would hop to it, and run this closure on it. + + // task groups + await withDiscardingTaskGroup { group in + group.addTask { + // starts and runs on the 'executor' + await nonisolatedAsyncFunc() // also runs on 'executor' + } + } + + // async let + async let number = nonisolatedAsyncFunc() // starts and runs on 'executor' + await number +} +``` + +If a task with such executor preference encounters code which is `isolated` to some specific actor, the isolation properties of the actor still are upheld, however, unless that actor has a custom executor configured, the source of the thread actually running the actor's functions will be from the preferred executor: + +```swift +let capy: Capybara = Capybara() +actor Capybara { func eat() {} } + +Task(executorPreference: executor) { + // starts on 'executor' + try await capy.eat() // execution is isolated to the 'capy' actor, however execution happens on the 'executor' TaskExecutor +} +``` + +In a way, one should think of the `SerialExecutor` of the actor and `TaskExecutor` both being tracked and providing different semantics. +The `SerialExecutor` guarantees mutual exclusion, and the `TaskExecutor` provides a source of threads. + +## Detailed design + +### Setting task executor preference + +A new concept of task executor preference is added to Swift Concurrency tasks. This preference is stored in a task and propagated throughout child tasks (such as ones created by TaskGroups and async let). + +The preference can be set using various APIs that will be discussed in detail in their respective sections. The first of those APIs is `withTaskExecutorPreference` which can be called inside an asynchronous context to both ensure we’re executing on the expected executor, as well as set the task executor preference for the duration of the operation closure: + +```swift +await withTaskExecutorPreference(someExecutor) { + // guaranteed to be executing on someExecutor +} +``` + +Once set, the effect of an executor preference is such that a nonisolated func instead of immediately hopping to the global pool, it may hop to the preferred executor, e.g.: + +```swift +nonisolated func doSomething() async { + // ... +} + +let preferredExecutor: SomeConcreteTaskExecutor = ... +Task(executorPreference: preferredExecutor) { + // executes on 'preferredExecutor' + await doSomething() // doSomething body would execute on 'preferredExecutor' +} + + await doSomething() // doSomething body would execute on 'default global concurrent executor' +``` + +### The `TaskExecutor` protocol + +In order to fulfil the requirement that we'd like default actors to run on a task executor, if it was set, we need to introduce a new kind of executor. + +This stems from the fact that `SerialExecutor` and how a default actor effectively acts as an executor "for itself" in Swift. +A default actor (so an actor which does not use a custom executor), has a "default" serial executor that is created by the Swift runtime and uses the actor is the executor's identity. +This means that the runtime executor tracking necessarily needs to track that some code is executing on a specific serial executor in order for things like `assumeIsolated` or the built-in runtime thread-safety checks can utilize them. + +The new protocol mirrors `Executor` and `SerialExecutor` in API, however it provides different semantics, and is tracked using a different mechanism at runtime -- by obtaining it from a task's executor preference record. + +The `TaskExecutor` is defined as: + +```swift +public protocol TaskExecutor: Executor { + func enqueue(_ job: consuming ExecutorJob) + + func asUnownedTaskExecutor() -> UnownedTaskExecutor +} +``` + +As an intuitive way to think about `TaskExecutor` and `SerialExecutor`, one can think of the prior as being a "source of threads" to execute work on, +and the latter being something that "provides serial isolation" and is a crucial part of Swift actors. The two share similarities, however the task executor has a more varied application space. + +### Task executor preference inheritance in Structured Concurrency + +Task executor preference is inherited by child tasks and actors which do not declare an explicit executor (so-called "default actors"), and is *not* inherited by un-structured tasks. + +Specifically: + +* **Do** inherit task executor preference + * TaskGroup’s `addTask()`, unless overridden with explicit parameter + * `async let` + * methods on default actors (actors which do not use a custom executor) +* **Do not** inherit task executor preference + * Unstructured tasks: `Task {}` and `Task.detached {}` + * methods on actors which **do** use a custom executor (including e.g. the `MainActor`) + +This also means that an entire tree can be made to execute their nonisolated work on a specific executor, just by means of setting the preference on the top-level task. + +### Task executor preference and async let + +Since `async let` are the simplest form of structured concurrency, they do not offer much in the way of customization. + +An async let currently always executes on the global concurrent executor, and with the inclusion of this proposal, it does take into account task executor preference. In other words, if an executor preference is set, it will be used by async let to enqueue its underlying task: + +```swift +func test() async -> Int { + return 42 +} + +await withTaskExecutorPreference(someExecutor) { + async let value = test() // async let's "body" and target function execute on 'someExecutor' + // ... + await value +} +``` + +### Task executor preference and TaskGroups + +A `TaskGroup` and its various friends (`ThrowingTaskGroup`, `DiscardingTaskGroup`, ...) are the most powerful, but also most explicit and verbose API for structured concurrency. A group allows creating multiple child tasks using the `addTask` method, and always awaits all child tasks to complete before returning. + +This proposal adds overloads to the `addTask` method, which changes the executor the child tasks will be enqueued on: + +```swift +extension (Discarding)(Throwing)TaskGroup { + mutating func addTask( + on executor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async (throws) -> Void + ) +} +``` + +Which allows users to require child tasks be enqueued and run on specific executors: + +```swift +Task(executorPreference: specialExecutor) { + _ = await withTaskGroup(of: Int.self) { group in + group.addTask { + // using 'specialExecutor' (inherited preference) + return 12 + } + group.addTask(executorPreference: differentExecutor) { + // using 'differentExecutor', overridden preference + return 42 + } + group.addTask(executorPreference: nil) { + // using 'specialExecutor' (inherited preference) + // + // explicitly documents that this task has "no task executor preference". + // this is semantically equivalent to the addTask() call without specifying + // an executor, and therefore since the surrounding scope has a specialExecutor preference, + // that's the executor used. + return 84 + } + group.addTask(executorPreference: globalConcurrentExecutor) { + // using 'globalConcurrentExecutor', overridden preference + // + // using the global concurrent executor -- effectively overriding + // the task executor preference set by the outer scope back to the + // default semantics of child tasks -- to execute on the global concurrent executor. + return 84 + } + return await group.next()! + } +``` + +This gives developers explicit control over where a task group child task shall be executed. Notably, this gives callers of libraries more control over where work should be performed. Do note that code executing on an actor will always hop to that actor; and task executor preference has no impact on code which *requires* to be running in some specific isolation. + +If a library really wants to ensure that hops to the global concurrent executor *are* made by child tasks it may use the newly introduced `globalConcurrentExecutor` global variable. + +### Task executor preference and Unstructured Tasks + +We propose adding new APIs and necessary runtime changes to allow a Task to be enqueued directly on a specific `Executor`, by using a new `Task(executorPreference:)` initializer: + +```swift +extension Task where Failure == Never { + @discardableResult + public init( + executorPreference taskExecutor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Success + ) + + @discardableResult + static func detached( + executorPreference taskExecutor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async -> Success + ) +} + +extension Task where Failure == Error { + @discardableResult + public init( + executorPreference taskExecutor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Success + ) + + @discardableResult + static func detached( + executorPreference taskExecutor: (any TaskExecutor)?, + priority: TaskPriority? = nil, + operation: @Sendable @escaping () async throws -> Success + ) +} +``` + +Tasks created this way are **immediately enqueued** on given executor. + +It is possible to pass `nil` to all task executor accepting APIs introduced in this proposal. Passing `nil` to an `executorPreference:` parameter means "no preference", and for structured tasks means to inherit the surrounding context's executor preference; and for unstructured tasks (`Task.init`, `Task.detached`) it serves as a way of documenting no specific executor preference was selected for this task. In both cases, passing `nil` is equivalent to calling the methods which do not accept an executor preference. + +By default, serial executors are not task executors, and therefore cannot be directly used with these APIs. +This is because it would cause confusion in the runtime about having two "mutual exclusion" contexts at the same time, which could result in difficult to understand behaviors. + +It is possible however to write a custom `SerialExecutor` and conform to the `TaskExecutor` protocol at the same time, if indeed one intended to use it for both purposes. +The serial executor conformance can be used for purposes of isolation (including the asserting and "assuming" of isolation), and the task executor conformance allows +using a type to provide a hint where tasks should execute although cannot be used to fulfil isolation requirements. + +#### Task executor preference and default actor isolated methods + +It is also worth explaining the interaction with actors which do not use a custom executor -- which is the majority of actors usually defined in a typical codebase. +Such actors are referred to as "default actors" and are the default way of how actors are declared: + +```swift +actor RunsAnywhere { // a "default" actor == without an executor requirement + func hello() { + return "Hello" + } +} +``` + +Such actor has no requirement as to where it wants to execute. This means that if we were to call the `hello()` isolated +actor method from a task that has defined an executor preference -- the hello() method would still execute on a thread owned by that executor (!), +however isolation is still guaranteed by the actor's semantics: + +```swift +let anywhere = RunsAnywhere() +Task { await anywhere.hello() } // runs on "default executor", using a thread from the global pool + +Task(executorPreference: myExecutor) { + // runs on preferred executor, using a thread owned by that executor + await anywhere.hello() +} +``` + +Methods which assert isolation, such as `Actor/assumeIsolated` and similar still function as expected. + +The task executor can be seen as a "source of threads" for the execution, while the actor's serial executor is used to ensure the serial and isolated execution of the code. + +## Inspecting task executor preference + +It is possible to inspect the current preferred task executor of a task, however because doing so is inherently unsafe -- due to lack of guarantees surrounding the lifetime of an executor referred to using an `UnownedTaskExecutor`, +this operation is only exposed on the `UnsafeCurrentTask`. + +Furthermore, the API purposefully does not expose an `any TaskExecutor` because this would risk incurring atomic ref-counting on an executor object that may have been already deallocated. + +This API is intended only for fine-tuning and checking if we are executing on the "expected" task executor, and therefore the `UnownedTaskExecutor` also implements the `Equatable` protocol, +and implements it using pointer equality. This comparison is not strictly safe, in case if an executor was deallocated, and a new executor was allocated in the same memory location, +however for purposes of executors -- especially long-lived ones, we believe this is not going to prove to be a problem in practical uses of task executors. + +An example use of this API might be something like this: + +``` swift +struct MyEventLoopTaskExecutor: TaskExecutor {} + +func test(expected eventLoop: MyEventLoopTaskExecutor) { + withUnsafeCurrentTask { task in + guard let task else { + fatalError("Missing task?") + } + guard let currentTaskExecutor = task.unownedTaskExecutor else { + fatalError("Expected to have task executor") + } + + precondition(currentTaskExecutor == eventLoop.asUnownedTaskExecutor()) + + // perform action that is required to run on the expected executor + } +} +``` + +This may be useful in synchronous functions; however should be used sparingly, and with caution. +Asynchronous functions, or functions on actors should instead rely on the usual ways to statically ensure to be running on an expected executor: +by providing the right annotations or custom executors to their enclosing actors. + +Instead, functions which have strict execution requirements may be better served as declaring them inside of an actor +that has the required specific executor specified (by using custom actor executors), or by using an asynchronous function +and wrapping the code that is required to run on a specific executor in an `withTaskExecutorPreference(eventLoop) { ... }` block. + +Nevertheless, because we understand there may be situations where synchronous code may want to compare task executors, this capability is exposed for advanced use cases. + +### TaskExecutor ownership + +Task executors, unlike serial executors, are explicitly owned by tasks as long as they are running on the given task executor. + +This is achieved in two ways. The `withTaskExecutorPreference` APIs by their construction as `with...`-style APIs, +naturally retain and keep alive the task executor for as long as the `with... { ... }` body is executing. +This also naturally extends to other structured concurrency constructs like `async let` and task groups, which can +rely on the task executor to remain alive while these constructs are running within such `withTaskExecutorPreference(...) { ... }` closure body. + +Unstructured tasks which are started with a task executor preference (e.g. `Task(executorPreference: someTaskExecutor)`), +take ownership of the executor for as long as the task is running. + +In other words, it is safe to rely on a task, structured or not, to keep alive the task executor it may be running on. +This makes it possible to write code like the following snippet, without having to worry about manually keeping the +executor alive until "all tasks which may be executing on it have finished": + +```swift +func computeThings() async { + let eventLoop: any TaskExecutor = MyCoolEventLoop() + defer { eventLoop.shutdown() } + + let computed = withTaskExecutorPreference(eventLoop) { + async let first = computation(1) + async let second = computation(2) + return await first + second + } + + return computed // event loop will be shutdown and the executor destroyed(!) +} + +func computation(_ int: Int) -> Int { + withUnsafeCurrentTask { task in + let unownedExecutor: UnownedTaskExecutor? = task?.unownedTaskExecutor + let eventLoop: MyCoolEventLoop = EventLoops.find(unownedExecutor) + // we need to start an unstructured task for some reason (try to avoid this if possible) + // and we have located the `MyCoolEventLoop` in our "cache". + // + // Since we have a real MyCoolEventLoop reference, this is safe to forward + // to the unstructured task which will retain it. + Task(executorPreference: eventLoop) { + async let something = ... // inherits the executor preference + } + } +} +``` + +Same as with `SerialExecutor`'s `UnownedSerialExecutor` type, the `UnownedTaskExecutor` does _not_ retain the executor, +so you have to be extra careful when relying on unowned task executor references for any kind of operations. If you +were to write some form of "lookup" function, which takes an unowned executor and returns an `any TaskExecutor`, +please make sure that the returned references are alive (i.e. by keeping them alive in the "cache" using strong references). + +## Combining `SerialExecutor` and `TaskExecutor` + +It is possible to declare a single executor type and have it conform to *both* the `SerialExecutor` (introduced in the custom actor executors proposal), +as well as the `TaskExecutor` (introduce in this proposal). + +If declaring an executor which conforms to both protocols, it truly **must** adhere to the `SerialExecutor` +semantics of not running work concurrently, as it may be used as an *isolation context* by an actor. + +```swift +// naive executor for illustration purposes; we'll assert on the dispatch queue and isolation. +final class NaiveQueueExecutor: TaskExecutor, SerialExecutor { + let queue: DispatchQueue + + init(_ queue: DispatchQueue) { + self.queue = queue + } + + public func enqueue(_ _job: consuming ExecutorJob) { + let job = UnownedJob(_job) + queue.async { + job.runSynchronously( + isolatedOn: self.asUnownedSerialExecutor(), + taskExecutor: self.asUnownedTaskExecutor()) + } + } + + @inlinable + public func asUnownedSerialExecutor() -> UnownedSerialExecutor { + UnownedSerialExecutor(ordinary: self) + } + + @inlinable + public func asUnownedTaskExecutor() -> UnownedTaskExecutor { + UnownedTaskExecutor(ordinary: self) + } +} +``` + +Since the enqueue method shares the same signature between the two protocols it is possible to just implement it once. +It is of crucial importance to run the job using the new `runSynchronously(isolatedOn:taskExecutor:)` overload +of the `runSynchronously` method. This will set up all the required thread-local state for both isolation assertions +and task-executor preference semantics to be handled properly. + +Given such an executor, we are able to have it both be used by an actor (thanks to being a `SerialExecutor`), and have +any structured tasks or nonisolated async functions execute on it (thanks to it being a `TaskExecutor`): + +```swift +nonisolated func nonisolatedFunc(expectedExecutor: NaiveQueueExecutor) async { + dispatchPrecondition(condition: .onQueue(expectedExecutor.queue)) + expectedExecutor.assertIsolated() +} + +actor Worker { + let executor: NaiveQueueExecutor + + init(on executor: NaiveQueueExecutor) { + self.executor = executor + } + + func test(_ expectedExecutor: NaiveQueueExecutor) async { + // we are isolated to the serial-executor (!) + dispatchPrecondition(condition: .onQueue(expectedExecutor.queue)) + expectedExecutor.preconditionIsolated() + + // the nonisolated async func properly executes on the task-executor + await nonisolatedFunc(expectedExecutor: expectedExecutor) + + /// the task-executor preference is inherited properly: + async let val = { + dispatchPrecondition(condition: .onQueue(expectedExecutor.queue)) + expectedExecutor.preconditionIsolated() + return 12 + }() + _ = await val + + // as expected not-inheriting + _ = await Task.detached { + dispatchPrecondition(condition: .notOnQueue(expectedExecutor.queue)) + }.value + + // we properly came back to the serial executor, just to make sure + dispatchPrecondition(condition: .onQueue(expectedExecutor.queue)) + expectedExecutor.preconditionIsolated() + } +} +``` + +### The `globalConcurrentExecutor` + +This proposal also introduces a way to obtain a reference to the global concurrent executor which is used by default by all tasks and asynchronous functions unless they require some specific executor. + +The implementation of this executor is not exposed as a type, however it is accessible through the `globalConcurrentExecutor` global variable: + +```swift +nonisolated(unsafe) +public var globalConcurrentExecutor: any _TaskExecutor { get } +``` + +Accessing this global computed property is thread-safe and can be done without additional synchronization. + +At present, it is not possible to customize the returned executor from this property, however customizing it is something we are interested in exploring in the future (as well as the main actor's executor). + +This executor does not introduce new functionality to Swift Concurrency per se, as it was always there since the beginning of the concurrency runtime, however it is the first time it is possible to obtain a reference to the global concurrent executor in pure Swift. Generally just creating tasks and calling nonisolated asynchronous functions would automatically enqueue them onto this underlying global thread-pool. + +This proposal introduces the `globalConcurrentExecutor` variable in order to be able to effectively "disable" a task executor preference, because setting a task's executor preference to the default executor is equivalent to the task having the default behavior, as if no executor preference was set. This matters particularly which child tasks, which do want to execute on the default executor under any circumstances: + +```swift +async let noPreference = computation() // child task executes on the global concurrent executor + +await withTaskExecutorPreference(specific) { + async let compute = computation() // child task executes on 'specific' executor + + await withTaskGroup(of: Int.self) { group in + // child task executes on 'specific' executor + group.addTask { computation() } + + // child task executes on global concurrent executor + group.addTask(executorPreference: globalConcurrentExecutor) { + async let compute = computation() // child task executes on the global concurrent executor + + computation() // executed on the global concurrent executor + } + } +} +``` + +## Execution semantics discussion + +### Not a Golden Hammer + +As with many new capabilities in libraries and languages, one may be tempted to use task executors to solve various problems. + +We advise care when doing so with task executors, because while they do minimize the "hopping off" of executors and the associated context switching, +this is also a behavior that may be entirely _undesirable_ in some situations. For example, over-hanging on the MainActor's executor is one of the main reasons +earlier Swift versions moved to make `nonisolated` asynchronous functions always hop off their calling execution context; and this proposal brings back this behavior +for specific executors. + +Applying task executors to solve a performance problem should be done after thoroughly understanding the problem an application is facing, +and only then determining the right "sticky"-ness behavior and specific pieces of code which might benefit from it. + +Examples of good candidates for task executor usage would be systems which utilize some form of "specific thread" to minimize synchronization overhead, +like for example event-loop based systems (often, network applications), or IO systems which willingly perform blocking operations and need to perform them off the global concurrency pool. + +### Analysis of use-cases and the "sticky" preference semantics + +The semantics explained in this proposal may at first seem tricky, however in reality the rule is quite straightforward: + +- when there is a strict requirement for code to run on some specific executor, *it will* (and therefore disregard the "preference"), +- when there is no requirement where asynchronous code should execute, this proposal allows to specify a preference and therefore avoid hopping and context switches, leading to more efficient programs. + +It is worth discussing how user-control is retained with this proposal. Most notably, we believe this proposal follows Swift's core principle of progressive disclosure. + +When developing an application at first one does not have to optimize for fewer context switches, however if as applications grow performance analysis diagnoses context switching being a problem this proposal gives developers the tools to, selectively, in specific parts of a code-base introduce sticky task executor behavior. + +### Separating blocking code off the global shared pools + +This proposal gives control to developers who know that they'd like to isolate their code off from callers. For example, imagine an IO library which wraps blocking IO primitives like read/write system calls. You may not want to perform those on the width-limited default pool of Swift Concurrency, but instead wrap APIs which will be calling such APIs with the executor preference of some "`DedicatedIOExecutor`" (not part of this proposal): + +```swift +// MyCoolIOLibrary.swift + +func blockingRead() -> Bytes { ... } + +public func callRead() async -> Bytes { + await withTaskExecutorPreference(DedicatedIOExecutor.shared) { // sample executor + blockingRead() // OK, we're on our dedicated thread + } +} + +public func callBulk() async -> Bytes { + // The same executor is used for both public functions + await withTaskExecutorPreference(DedicatedIOExecutor.shared) { // sample executor + await callRead() + await callRead() + } +} +``` + +This way we won't be blocking threads inside the shared pool, and are not risking thread starving the entire application. + +We can call `callRead` from inside `callBulk` and avoid unnecessary context switching as the same thread servicing the IO operation may be used for those asynchronous functions -- and no actual context switch may need to be performed when `callBulk` calls into `callRead` either. + +End-users of this library don't need to worry about any of this, but the author of such a library is in full control over where execution will happen -- be it using task executor preference, or custom actor executors. + +This also works the other way around, when a user of a library notices that it is doing blocking work which they would rather separate out onto a different executor. This is true even if the library has declared asynchronous methods but still is taking too long to yield the thread for some reason, causing issues to the shared pool. + +```swift +// SomeLibrary +nonisolated func slowSlow() async { ... } // causes us issues by blocking +``` + + In such situation, we, as users of given library can notice and work around this issue by wrapping it with an executor preference: + +```swift +// our code +func caller() async { + // on shared global pool... + // let's make sure to run slowSlow on a dedicated IO thread: + await withTaskExecutorPreference(DedicatedIOExecutor.shared) { // sample executor + await slowSlow() // will not hop to global pool, but stay on our IOExecutor + } +} +``` + +In other words, task executor preference gives control to developers when and where care needs to be taken. + +The default of hop-avoiding when a preference is set has the benefit of optimizing for less context switching and can lead to better performance. + +It is possible to effectively restore the default behavior as-if no task executor preference was present, by setting the preference to the `globalConcurrentExecutor` which is the executor used by default actors, tasks, and free async functions when no task executor preference is set: + +```swift +func function() async { + // make sure to ignore caller's task executor preference, + // and always use the global concurrent executor. + await withTaskExecutorPreference(globalConcurrentExecutor) { ... } +} +``` + +## Prior-Art + +It is worth comparing with other concurrency runtimes with similar concepts to make sure if there are some common ideas or something different other projects have researched. + +For example, in Kotlin, a `launch` which equivalent to Swift’s creation of a new task, takes a coroutine context which can contain an executor preference. The official documentation showcases the following example: + +```kotlin +launch { // context of the parent, main runBlocking coroutine + println("main runBlocking : I'm working in thread ${Thread.currentThread().name}") +} +launch(Dispatchers.Unconfined) { // not confined -- will work with main thread + println("Unconfined : I'm working in thread ${Thread.currentThread().name}") +} +launch(Dispatchers.Default) { // will get dispatched to DefaultDispatcher + println("Default : I'm working in thread ${Thread.currentThread().name}") +} +launch(newSingleThreadContext("MyOwnThread")) { // will get its own new thread + println("newSingleThreadContext: I'm working in thread ${Thread.currentThread().name}") +} +``` + +Which is similar to the here proposed semantics of passing a specific executor preference. Notably though, because Swift has the concept of actor `isolation` the executor semantics introduced in this proposal are only a preference and will never override the executor requirements of actually strongly `isolated` code. + +Kotlin jobs also inherit the coroutine context from their parent, which is similar to the here proposed executor inheritance works. + +## Future directions + +### Task executor preference and global actors + +Thanks to the improvements to treating @SomeGlobalActor isolation proposed in [SE-NNNN: Improved control over closure actor isolation](https://github.com/swiftlang/swift-evolution/pull/2174) we would be able to that a Task may prefer to run on a specific global actor’s executor, and shall be isolated to that actor. + +Thanks to the equivalence between `SomeGlobalActor.shared` instance and `@SomeGlobalActor` annotation isolations (introduced in the linked proposal), this does not require a new API, but uses the previously described API that accepts an actor as parameter, to which we can pass a global actor’s `shared` instance. + +```swift +@MainActor +var example: Int = 0 + +Task(executorPreference: MainActor.shared) { + example = 12 // not crossing actor-boundary +} +``` + +It is more efficient to write `Task(executorPreference: MainActor.shared) {}` than it is to `Task { @MainActor in }` because the latter will first launch the task on the inferred context (either enclosing actor, or global concurrent executor), and then hop to the main actor. The `executorPreference: MainActor.shared` spelling allows Swift to immediately enqueue on the actor itself. + +### Static closure isolation + +It would be interesting to allow starting a task on a specific actor's executor, and have this infer the specific isolation. + +Today the proposal does not allow using serial executors, which are strictly associated with actors to start a Task "on" such executor. +We could consider adding some form of such ability, and then be able to infer that the closure of a Task is isolated to the actor passed to `Task(executorPreference: some Actor)`. + +The upcoming [SE-NNNN: Improved control over closure actor isolation](https://github.com/swiftlang/swift-evolution/pull/2174) proposal includes a future direction which would allow isolating a closure to a known other value. + +This could be utilized to spell the `Task` initializer like this: + +```swift +extension Task where ... { + init( + executorPreference target: TargetActor, + // ..., + operation: @isolated(target) () async -> () + ) where TargetActor: Actor +} +``` + +This would allow us to cut down on the noise of passing the isolated-on parameter explicitly and avoid a hop to the global executor before the task eventually hops back to the intended actor. +Today, if we were to allow a default actor's executor to be used as `TaskExecutor`, a similar API could be made that would look like this: + +```swift +actor Worker { func work() {} } +let worker: Worker = Worker() + +Task(executorPreference: worker) { worker in // noisy parameter; though required for isolation purposes + worker.work() +} +``` + +However, it would be noisy in the sense of having to repeat the `worker` parameter for purposes of isolation. + + + +## Alternatives considered + +### Do not provide any control over task executors + +We considered if not introducing this feature could be beneficial and forcing developers to always pass explicit `isolated` parameters instead. We worry that this becomes a) very tedious and b) impossibly ties threading semantics with public API and ABI of methods. We are concerned that the lack of executor “preference” which only affects the nonisolated functions in a task hierarchy would cause developers to defensively and proactively create multiple versions of APIs. It would also only allow passing actors as the executors, because isolation is an actor concept, and therefore we’d only be able to isolate using serial executors, while we may want to isolate using general purpose `Executor` types. + + +## Revisions + +- 1.6 + - introduce the global `var defaultConcurrentExecutor: any TaskExecutor` we we can express a task specifically wanting to run on the default global concurrency pool. +- 1.5 + - document that an executor may be both SerialExecutor and TaskExecutor at the same time +- 1.4 + - added `unownedTaskExecutor` to UnsafeCurrentTask +- 1.3 + - introduce TaskExecutor in order to be able to implement actor isolation properly and still use a different thread for running default actors + - wording cleanups + - removal of the `Task(executorPreference: Actor)` APIs; we could perhaps revisit this if we made default actors' executors somehow aware of being a thread source as well etc. +- 1.2 + - preference also has effect on default actors +- 1.1 + - added future direction about simplifying the isolation of closures without explicit parameter passing + - removed ability to observe current executor preference of a task diff --git a/proposals/0418-inferring-sendable-for-methods.md b/proposals/0418-inferring-sendable-for-methods.md new file mode 100644 index 0000000000..9e030a6872 --- /dev/null +++ b/proposals/0418-inferring-sendable-for-methods.md @@ -0,0 +1,421 @@ +# Inferring `Sendable` for methods and key path literals + +* Proposal: [SE-0418](0418-inferring-sendable-for-methods.md) +* Authors: [Angela Laar](https://github.com/angela-laar), [Kavon Farvardin](https://github.com/kavon), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 6.0)** +* Upcoming Feature Flag: `InferSendableFromCaptures` +* Review: ([pitch](https://forums.swift.org/t/pitch-inferring-sendable-for-methods/66565)) ([review](https://forums.swift.org/t/se-0418-inferring-sendable-for-methods-and-key-path-literals/68999)) ([acceptance](https://forums.swift.org/t/accepted-se-0418-inferring-sendable-for-methods-and-key-path-literals/69242)) + +## Introduction + +This proposal is focused on a few corner cases in the language surrounding functions as values and key path literals when using concurrency. We propose Sendability should be inferred for partial and unapplied methods. We also propose to lift a Sendability restriction placed on key path literals in [SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md#key-path-literals) by allowing the developers to control whether key path literal is Sendable or not. The goal is to improve flexibility, simplicity, and ergonomics without significant changes to Swift. + +## Motivation + +The partial application of methods and other first-class uses of functions have a few rough edges when combined with concurrency. + +Let’s look at partial application on its own before we combine it with concurrency. In Swift, you can create a function-value representing a method by writing an expression that only accesses (but does not call) a method using one of its instances. This access is referred to as a "partial application" of a method to one of its (curried) arguments - the object instance. + +```swift +struct S { + func f() { ... } +} + +let partial: (() -> Void) = S().f +``` + + +When referencing a method *without* partially applying it to the object instance, using the expression NominalType.method, we call it "unapplied." + + +```swift +let unapplied: (S) -> (() -> Void) = S.f +``` + + +Suppose we want to create a generic method that expects an unapplied function method conforming to Sendable as a parameter. We can create a protocol `P` that conforms to the `Sendable` protocol and tell our generic function to expect some generic type that conforms to `P`. We can also use the `@Sendable` attribute, introduced for closures and functions in [SE-302](https://github.com/kavon/swift-evolution/blob/sendable-functions/proposals/0302-concurrent-value-and-concurrent-closures.md), to annotate the closure parameter. + + +```swift +protocol P: Sendable { + init() +} + +func g(_ f: @escaping @Sendable (T) -> (() -> Void)) where T: P { + Task { + let instance = T() + f(instance)() + } +} +``` + +Now let’s call our method and pass our struct type `S` . First we should make `S` conform to Sendable, which we can do by making `S` conform to our new Sendable type `P` . + +This should make `S` and its methods Sendable as well. However, when we pass our unapplied function `S.f` to our generic function `g`, we get a warning that `S.f` is not Sendable as `g()` is expecting. + + +```swift +struct S: P { + func f() { ... } +} + +g(S.f) // Converting non-sendable function value to '@Sendable (S) -> (() -> Void)' may introduce data races +``` + + +We can work around this by wrapping our unapplied function in a Sendable closure. + +```swift +// S.f($0) == S.f() +g({ @Sendable in S.f($0) }) +``` + + +However, this is a lot of churn to get the expected behavior. The compiler should preserve `@Sendable` in the type signature instead. + +**Key Paths** + +[SE-0302](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md#key-path-literals) makes an explicit mention that all key path literals are treated as implicitly `Sendable` which means that they are not allowed to capture any non-`Sendable` values. This behavior is justified when key path values are passed across concurrency domains or otherwise involved in concurrently executed code but is too restrictive for non-concurrency related code. + +```swift +class Info : Hashable { + // some information about the user +} + +public struct Entry {} + +public struct User { + public subscript(info: Info) -> Entry { + // find entry based on the given info + } +} + +let entry: KeyPath = \.[Info()] +``` + +With sendability checking enabled this example is going to produce the following warning: + +``` +warning: cannot form key path that captures non-sendable type 'Info' +let entry: KeyPath = \.[Info()] + ^ +``` + +Use of the key path literal is currently being diagnosed because all key path literals should be Sendable. In actuality, this code is concurrency-safe, there are no data races here because key path doesn’t actually cross any isolation boundary. The compiler should instead verify and diagnose situations when key path is actually passed across an isolation boundary otherwise a warning like that would be confusing for the developers unfamiliar with Swift concurrency, might not always be actionable when type is declared in a different module, and goes against the progressive disclosure principle of the language. + +## Proposed solution + +We propose the compiler should automatically employ `Sendable` on functions and key paths that cannot capture non-Sendable values. This includes partially-applied and unapplied instance methods of `Sendable` types, as well as non-local functions. Additionally, it should be disallowed to utilize `@Sendable` on instance methods of non-`Sendable` types. + +**Functions** + +For a function, the `@Sendable` attribute primarily influences the kinds of values that can be captured by the function. But methods of a nominal type do not capture anything but the object instance itself. Semantically, a method can be thought of as being represented by the following functions: + + +```swift +// Pseudo-code declaration of a Nominal Type: +type NominalType { + func method(ArgType) -> ReturnType { /* body of method */ } +} + +// Can desugar to these two global functions: +func NominalType_method_partiallyAppliedTo(_ obj: NominalType) -> ((ArgType) -> ReturnType) { + let inner = { [obj] (_ arg1: ArgType) -> ReturnType in + return NominalType_method(obj, arg1) + } + return inner +} +// The actual method call +func NominalType_method(_ self: NominalType, _ arg1: ArgType) -> ReturnType { + /* body of method */ +} +``` + +Thus, the only way a partially-applied method can be `@Sendable` is if the `inner` closure were `@Sendable`, which is true if and only if the nominal type conforms to `Sendable`. + + +```swift +type NominalType : Sendable { + func method(ArgType) -> ReturnType { /* body of method */ } +} +``` + +For example, by declaring the following type `Sendable`, the partial and unapplied function values of the type would have implied Sendability and the following code would compile with no errors. + +```swift +struct User : Sendable { + func updatePassword (new: String, old: String) -> Bool { + /* update password*/ + return true + } +} + +let unapplied: @Sendable (User) -> ((String, String) → Bool) = User.updatePassword // no error + +let partial: @Sendable (String, String) -> Bool = User().updatePassword // no error +``` + +**Key paths** + +Key path literals are very similar to functions, their sendability could be influenced by sendability of the values they capture in their arguments and isolation of the referenced properties and subscripts. Instead of requiring key path literals to always be sendable and warning about cases where key path literals capture non-Sendable types, let’s flip that requirement and allow the developers to explicitly state when a key path is required to be Sendable via `& Sendable` type composition and employ type inference to infer sendability in the same fashion as functions when no contextual type is specified. [The key path hierarchy of types is non-Sendable]. + +Let’s extend our original example type `User` with a new property and a subscript to showcase the change in behavior: + +```swift +struct User { + var name: String + + @MainActor var age: Int + + subscript(_ info: Info) -> Entry { ... } +} +``` + +A key path to reference a property `name` does not capture any non-Sendable types which means the type of such key path literal could either be inferred as `WritableKeyPath & Sendable` or stated to have a sendable type via `& Sendable` composition: + +```swift +let name = \User.name // WritableKeyPath **& Sendable** +let name: KeyPath & Sendable = \.name // 🟢 +``` + +It is also allowed to use `@Sendable` function type and `& Sendable` key path interchangeably: + +```swift +let name: @Sendable (User) -> String = \.name 🟢 +``` + +It is important to note that **under the proposed rule all of the declarations that do not explicitly specify a Sendable requirement alongside key path type are treated as non-Sendable** (see Source Compatibility section for further discussion): + +```swift +let name: KeyPath = \.name // 🟢 but key path is **non-Sendable** +``` + +Since Sendable is a marker protocol it should be possible to adjust all declarations where `& Sendable` is desirable without any ABI impact. + +Existing APIs that use key path in their parameter types or default values can add `Sendable` requirement in a non-ABI breaking way by marking existing declarations as @preconcurrency and adding `& Sendable` at appropriate positions: + +```swift +public func getValue(_: KeyPath) { ... } +``` + +becomes + +```swift +@preconcurrency public func getValue(_: KeyPath & Sendable) { ... } +``` + +Explicit sendability annotation does not override sendability checking and it would still be incorrect to state that the key path literal is Sendable when it captures non-Sendable values: + +```swift +let entry: KeyPath & Sendable = \.[Info()] 🔴 Info is a non-Sendable type +``` + +Such `entry` declaration would be diagnosed by the sendability checker: + +```swift +warning: cannot form key path that captures non-sendable type 'Info' +``` + +In the same fashion key path that references `age` (i.e. `\User.age`), which is a global actor isolated property, is non-Sendable. + +## Detailed design + +This proposal includes five changes to `Sendable` behavior. + +The first two are what we just discussed regarding partial and unapplied methods. + +```swift +struct User : Sendable { + var address: String + var password: String + + func changeAddress (new: String, old: String) {/*do work*/ } +} +``` + +1. The inference of `@Sendable` for unapplied references to methods of a Sendable type. + +```swift +let unapplied : @Sendable (User)-> ((String, String) -> Void) = User.changeAddress // no error +``` + +2. The inference of `@Sendable` for partially-applied methods of a Sendable type. + +```swift +let partial : @Sendable (String, String) -> Void = User().changeAddress // no error +``` + + +These two rules include partially applied and unapplied static methods but do not include partially applied or unapplied mutable methods. Unapplied references to mutable methods are not allowed in the language because they can lead to undefined behavior. More details about this can be found in [SE-0042](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0042-flatten-method-types.md). + + +3. A key path literal without non-Sendable type captures and references to actor-isolated properties and/or subscripts is going to be inferred as key path type with a `& Sendable` requirement or a function type with `@Sendable` attribute. + +```swift +extension User { + @MainActor var age: Int { get { 0 } } +} + +let ageKP = \User.age +let infoKP = \User.[Info()] +``` + +The type of `ageKP` is `KeyPath` because `age` is isolated to a global actor. Similarly `infoKP` is a non-Sendable key path because `Info()` argument to a subscript reference has a non-Sendable type. + +Key path types respect all of the existing sub-typing rules related to Sendable protocol which means a key path that is not marked as Sendable cannot be assigned to a value that is Sendable. + +```swift +let name: KeyPath = \.name +let otherName: KeyPath & Sendable = \.name 🔴 +``` + +The conversion between key path and a `@Sendable` function doesn’t actually require the key path itself to be `Sendable` because it’s not captured by the closure but wrapped by it. + +```swift +let name: @Sendable (User) -> String = \.name 🟢 +``` + + The example above is accepted and is transformed by the compiler into: + +```swift +let name: @Sendable (User) -> String = { $0[keyPath: \.name] } +``` + +But any subscript arguments that are non-Sendable would preclude the conversion because they’d be captured by the implicitly synthesized closure which makes the closure non-Sendable: + +```swift +let value: NonSendable = NonSendable() +let _: @Sendable (User) -> String = \.[value] 🔴 +``` + +This is an error because `value` has a non-Sendable type and the compiler synthesized closure that wraps the key path - `{ $0[keyPath: \.[value]] }` is going to be inferred as non-Sendable (because it captures `value`) hence non-convertible to a `@Sendable` function type. + +Similarly if the conversion captures a key path that has a reference to an isolated property or subscript the implicitly generated closure is not inferred to be non-Sendable. + +Key path literals are allowed to infer Sendability requirements from the context i.e. when a key path literal is passed as an argument to a parameter that requires a Sendable type: + +```swift +func getValue(_: KeyPath & Sendable) -> T {} + +getValue(name) // 🟢 both parameter & argument match on sendability requirement +getValue(\.name) // 🟢 use of '& Sendable' by the parameter transfers to the key path literal +getValue(\.[NonSendable()]) // 🔴 This is invalid because key path captures a non-Sendable type + +func filter(_: @Sendable (User) -> T) {} +filter(name) // 🟢 use of @Sendable applies a sendable key path +``` + +Next is: + +4. The inference of `@Sendable` when referencing non-local functions. + +Unlike closures, which retain the captured value, global functions can't capture any variables - because global variables are just referenced by the function without any ownership. With this in mind there is no reason not to make these `Sendable` by default. This change will also include static global functions. + +```swift +func doWork() -> Int { + Int.random(in: 1..<42) +} + +Task.detached(priority: nil, operation: doWork) // Converting non-sendable function value to '@Sendable () async -> Void' may introduce data races +``` + +Currently, trying to start a `Task` with the global function `doWork` will cause an error complaining that the function is not `Sendable`. This should compile with no issue. + +5. Prohibition of marking methods `@Sendable` when the type they belong to is not `@Sendable`. + +```swift +class C { + var random: Int = 0 // random is mutable so `C` can't be checked sendable + + @Sendable func generateN() async -> Int { //error: adding @Sendable to function of non-Senable type prohibited + random = Int.random(in: 1..<100) + return random + } +} + +func test(x: C) { x.generateN() } + +let num = C() +Task.detached { + test(num) +} +test(num) // data-race +``` + +If we move the previous work we wanted to do into a class that stores the random number we generate as a mutable value, we could be introducing a data race by marking the function responsible for this work `@Sendable` . Doing this should be prohibited by the compiler. + +Since `@Sendable` attribute will be automatically determined with this proposal, you will no longer have to explicitly write it on function and method declarations. + +### Extending key path merging functionality to preserve sendability + +Existing Key path API provides a way to join two key paths together via using instance method `appending(...)` . Overloads of this method take key path types of varying mutability as their parameters and produce a new “joined” key path of a desired mutability (read-only, writable, or reference writable). + +Under the proposed semantics all overloads of this method become non-Sendable but it is possible and desirable to alleviate that and support/propagate sendability if both “base” and “appended” key paths are `Sendable`. + +Such could be archived by introducing new overloads to `func appending(...)` that utilize `& Sendable` for their parameter and result in an extension of `Sendable` protocol. For example: + +```swift +extension Sendable where Self: AnyKeyPath { + @inlinable + public func appending( + path: KeyPath & Sendable + ) -> KeyPath & Sendable where Self : KeyPath { + ... + } +} +``` + +This overload would be selected if both “base” key path and the argument are `Sendable` and would produce a new `Sendable` key path: + +```swift +func makeUTF8CountKeyPath(from base: KeyPath & Sendable) -> KeyPath & Sendable { + // Both `base` and `\String.utf8.count` are Sendable key paths, + // so `appending(path:)` returns a Sendable key path too. + return base.appending(path: \.utf8.count) 🟢 +} +``` + +Standard library would have to introduce a variety of new overloads to keep `Sendable` capable `appending(...)` on par with existing non-Sendable functionality. + +## Source compatibility + +As described in the Proposed Solution section, some of the existing property and variable declarations **without explicit types** could change their type but the impact of the inference change should be very limited. For example, it would only be possible to observe it when a function or key path value which is inferred as Sendable is passed to an API which is overloaded on Sendable capability: + +```swift +func callback(_: @Sendable () -> Void) {} +func callback(_: () -> Void) {} + +callback(MyType.f) // if `f` is inferred as @Sendable first `callback` is preferred + +func getValue(_: KeyPath & Sendable) {} +func getValue(_: KeyPath) {} + +getValue(\.utf8.count) // prefers first overload of `getValue` if key path is `& Sendable` +``` + +Such calls to `callback` and `getValue` are currently ambiguous but under the proposed rules the type-checker would pick the first overload of `callback` and `getValue` as a solution if `f` is inferred as `@Sendable` and `\String.utf8.count` would be inferred as having a type of `KeyPath & Sendable` instead of just `KeyPath`. + +## Effect on ABI stability + +When you remove an explicit `@Sendable` from a method, the mangling of that method will change. Since `@Sendable` will now be inferred, if you choose to remove the explicit annotation to "adopt" the inference, you may need to consider the mangling change. + +Adding or removing `& Sendable` from type doesn’t have any ABI impact because `Sendable` is a marker protocol that can be added transparently. + +## Effect on API resilience + +N/A + +## Future Directions + +Accessors are not currently allowed to participate with the `@Sendable` system in this proposal. It would be straightforward to allow getters to do so in a future proposal if there was demand for this. + +## Alternatives Considered + +Swift could forbid explicitly marking function declarations with the `@Sendable` attribute, since under this proposal there’s no longer any reason to do this. + +```swift +/*@Sendable*/ func alwaysSendable() {} +``` + +However, since these attributes are allowed today, this would be a source breaking change. Swift 6 could potentially include fix-its to remove `@Sendable` attributes to ease migration, but it’d still be disruptive. The attributes are harmless under this proposal, and they’re still sometimes useful for code that needs to compile with older tools, so we have chosen not to make this change in this proposal. We can consider deprecation at a later time if we find a good reason to do so. diff --git a/proposals/0419-backtrace-api.md b/proposals/0419-backtrace-api.md new file mode 100644 index 0000000000..d185267a86 --- /dev/null +++ b/proposals/0419-backtrace-api.md @@ -0,0 +1,472 @@ +# Swift Backtrace API + +* Proposal: [SE-0419](0419-backtrace-api.md) +* Authors: [Alastair Houghton](https://github.com/al45tair) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Accepted** +* Implementation: Implemented on main, requires explicit `_Backtracing` import. +* Review: ([pitch](https://forums.swift.org/t/pitch-swift-backtracing-api/62741)) ([review](https://forums.swift.org/t/se-0419-swift-backtracing-api/69595)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0419-swift-backtracing-api/70318)) + +## Introduction + +This year we are improving the usability of Swift for command line and +server-side development by adding first-class support for backtraces +to Swift. + +The backtrace support consists of two parts; the first is the actual +backtracing implementation, and the second is the new API surface in +the Swift standard library. This proposal concerns the latter. + +## Motivation + +In addition to the runtime providing backtraces when programs crash or +terminate abnormally, it is often useful for testing frameworks and +sometimes even library or application code to capture details of the +call stack at a point in time. + +This functionality is somewhat tricky to implement correctly and any +implementation tends, of necessity, to be non-portable. Existing +third-party packages that provide backtrace support have various +downsides, including lack of support for tracing through async frames, +and add additional dependencies to client packages and applications. + +## Proposed solution + +We will add a `Backtrace` struct to the standard library, with methods +to capture a backtrace from the current location, and support for +symbolication and symbol demangling. All of the backtracing types will +exist in a new `Runtime` module. + +Note, importantly, that **the API presented here is not async-signal-safe**, +and **it is not an appropriate tool with which to build a general purpose +crash reporter**. The intended use case for this functionality is the +programmatic capture of backtraces during normal execution. + +## Detailed design + +The `Backtrace` struct will capture an `Array` of `Frame` objects, +each of which will represent a stack frame or a `Task` activation +context. + +```swift +/// Holds a backtrace. +public struct Backtrace: CustomStringConvertible, Codable, Sendable { + /// The type of an address. + /// + /// This is used as an opaque type; if you have some Address, you + /// can ask if it's NULL, and you can attempt to convert it to a + /// FixedWidthInteger. + /// + /// This is intentionally _not_ a pointer, because you shouldn't be + /// dereferencing them; they may refer to some other process, for + /// example. + public struct Address: Comparable, Hashable, Codable, Sendable, + LosslessStringConvertible, + ExpressibleByIntegerLiteral { + var bitWidth: Int { get } + var isNull: Bool { get } + } + + /// The unwind algorithm to use. + public enum UnwindAlgorithm { + /// Choose the most appropriate for the platform. + case auto + + /// Use the fastest viable method. + /// + /// Typically this means walking the frame pointers. + case fast + + /// Use the most precise available method. + /// + /// On Darwin and on ELF platforms, this will use EH unwind + /// information. On Windows, it will use Win32 API functions. + case precise + } + + /// Represents an individual frame in a backtrace. + public enum Frame: CustomStringConvertible, Codable, Sendable { + /// An accurate program counter. + /// + /// This might come from a signal handler, or an exception or some + /// other situation in which we have captured the actual program counter. + case programCounter(Address) + + /// A return address. + /// + /// Corresponds to a call from a normal function. + case returnAddress(Address) + + /// An async resume point. + /// + /// Corresponds to an `await` in an async task. + case asyncResumePoint(Address) + + /// Indicates a discontinuity in the backtrace. + /// + /// This occurs when you set a limit and a minimum number of frames at + /// the top. For example, if you set a limit of 10 frames and a minimum + /// of 4 top frames, but the backtrace generated 100 frames, you will see + /// + /// 0: frame 100 <----- bottom of call stack + /// 1: frame 99 + /// 2: frame 98 + /// 3: frame 97 + /// 4: frame 96 + /// 5: ... <----- omittedFrames(92) + /// 6: frame 3 + /// 7: frame 2 + /// 8: frame 1 + /// 9: frame 0 <----- top of call stack + /// + /// Note that the limit *includes* the discontinuity. + /// + /// This is good for handling cases involving deep recursion. + case omittedFrames(Int) + + /// Indicates a discontinuity of unknown length. + /// + /// This can only be present at the end of a backtrace; in other cases + /// we will know how many frames we have omitted. For instance, + /// + /// 0: frame 100 <----- bottom of call stack + /// 1: frame 99 + /// 2: frame 98 + /// 3: frame 97 + /// 4: frame 96 + /// 5: ... <----- truncated + case truncated + + /// The original program counter, with no adjustment. + /// + /// The value returned from this property is undefined if the frame + /// is a discontinuity. + public var originalProgramCounter: Address { get } + + /// The adjusted program counter to use for symbolication. + /// + /// The value returned from this property is undefined if the frame + /// is a discontinuity. + public var adjustedProgramCounter: Address { get } + + /// A textual description of this frame. + public var description: String { get } + } + + /// Represents an image loaded in the process's address space + public struct Image: CustomStringConvertible, Codable, Identifiable, Sendable { + /// The name of the image (e.g. libswiftCore.dylib). + public var name: String? { get } + + /// The full path to the image (e.g. /usr/lib/swift/libswiftCore.dylib). + public var path: String? { get } + + /// The unique ID of the image, as a byte array (note that the exact number + /// of bytes may vary, and that some images may not have a unique ID). + /// + /// On Darwin systems, this is the LC_UUID value; on Linux this is the + /// build ID, which may take one of a number of forms or may not even + /// be present. + public var uniqueID: [UInt8]? { get } + + /// The base address of the image. + public var baseAddress: Address { get } + + /// The end of the text segment in this image. + public var endOfText: Address { get } + + /// Provide a textual description of an Image. + public var description: String { get } + } + + /// The architecture of the process to which this backtrace refers. + public var architecture: String + + /// A `Sequence` of captured frame information. + /// + /// The underlying storage is intentionally not exposed, because there may + /// be cases where it's desirable to use a more compact form (for instance + /// delta compression). + public var frames: some Sequence { get } + + /// A list of captured images. + /// + /// Some backtracing algorithms may require this information, in which case + /// it will be filled in by the `capture()` method. Other algorithms may + /// not, in which case it will be `nil` and you can capture an image list + /// separately yourself using `captureImages()`. + public var images: [Image]? + + /// Capture a backtrace from the current program location. + /// + /// The `capture()` method itself will not be included in the backtrace; + /// i.e. the first frame will be the one in which `capture()` was called, + /// and its programCounter value will be the return address for the + /// `capture()` method call. + /// + /// @param algorithm Specifies which unwind mechanism to use. If this + /// is set to `.auto`, we will use the platform default. + /// @param limit The backtrace will include at most this number of + /// frames; you can set this to `nil` to remove the + /// limit completely if required. + /// @param offset Says how many frames to skip; this makes it easy to + /// wrap this API without having to inline things and + /// without including unnecessary frames in the backtrace. + /// @param top Sets the minimum number of frames to capture at the + /// top of the stack. + /// + /// @returns A new `Backtrace` struct. + @inline(never) + public static func capture(algorithm: UnwindAlgorithm = .auto, + limit: Int? = 64, + offset: Int = 0, + top: Int = 16) throws -> Backtrace + + /// Capture a list of the images currently mapped into the calling + /// process. + /// + /// @returns A list of `Image`s. + public static func captureImages() -> [Image] + + /// Specifies options for the `symbolicated` method. + public struct SymbolicationOptions: OptionSet { + public let rawValue: Int + + /// Add virtual frames to show inline function calls. + public static let showInlineFrames: SymbolicationOptions + + /// Look up source locations. + /// + /// This may be expensive in some cases; it may be desirable to turn + /// this off e.g. in Kubernetes so that pods restart promptly on crash. + public static let showSourceLocations: SymbolicationOptions + + /// Use a symbol cache, if one is available. + public static let useSymbolCache: SymbolicationOptions + + public static let default: SymbolicationOptions = [.showInlineFrames, + .showSourceLocations, + .useSymbolCache] + } + + /// Return a symbolicated version of the backtrace. + /// + /// @param images Specifies the set of images to use for symbolication. + /// If `nil`, the function will look to see if the `Backtrace` + /// has already captured images. If it has, those will be + /// used; otherwise we will capture images at this point. + /// + /// @param options Symbolication options; see `SymbolicationOptions`. + /// + /// @returns A new `SymbolicatedBacktrace`. + public func symbolicated(with images: [Image]? = nil, + options: SymbolicationOptions = .default) + -> SymbolicatedBacktrace? + + /// Provide a textual version of the backtrace. + public var description: String { get } +} +``` + +We allow `Address` to be converted to a `FixedWidthInteger` by means of an +extension on `FixedWidthInteger`: + +```swift +extension FixedWidthInteger { + /// Convert from a Backtrace.Address. + /// + /// This initializer will return nil if the address width is larger than the + /// type you are attempting to convert into. + /// + /// @param address The `Address` to convert. + init?(_ address: Backtrace.Address) +} +``` + +_Symbolication_, by which we mean the process of looking up the symbols +associated with addresses in a backtrace, is in general an expensive +process, and for efficiency reasons is normally performed for a backtrace +as a whole, rather than for individual frames. It therefore makes sense +to provide a separate `SymbolicatedBacktrace` type and to provide a +method on a `Backtrace` +to symbolicate. + +```swift +/// A symbolicated backtrace +public struct SymbolicatedBacktrace: CustomStringConvertible, Codable, Sendable { + /// The `Backtrace` from which this was constructed + public var backtrace: Backtrace + + /// Represents a location in source code. + /// + /// The information in this structure comes from compiler-generated + /// debug information and may not correspond to the current state of + /// the filesystem --- it might even hold a path that only works + /// from an entirely different machine. + public struct SourceLocation: CustomStringConvertible, Codable, Sendable { + /// The path of the source file. + var path: String { get } + + /// The line number. + var line: Int { get } + + /// The column number. + var column: Int { get } + + /// Provide a textual description. + public var description: String { get } + } + + /// Represents an individual frame in the backtrace. + public struct Frame: CustomStringConvertible, Codable, Sendable { + /// The captured frame from the `Backtrace`. + public var captured: Backtrace.Frame { get } + + /// The result of doing a symbol lookup for this frame. + public var symbolInfo: SymbolInfo? { get } + + /// If `true`, then this frame was inlined. + public var isInline: Bool { get } + + /// `true` if this frame represents a Swift runtime failure. + public var isSwiftRuntimeFailure: Bool { get } + + /// `true` if this frame represents a Swift thunk function. + public var isSwiftThunk: Bool { get } + + /// `true` if this frame is a system frame. + public var isSystem: Bool { get } + + /// A textual description of this frame. + public var description: String { get } + } + + /// Represents a symbol we've located + public struct SymbolInfo: CustomStringConvertible, Codable, Sendable { + /// The image in which the symbol for this address is located. + public var image: Backtrace.Image { get } + + /// The raw symbol name, before demangling. + public var rawName: String { get } + + /// The demangled symbol name. + public var name: String { get } + + /// The offset from the symbol. + public var offset: Int { get } + + /// The source location, if available. + public var sourceLocation: SourceLocation? { get } + + /// True if this symbol represents a Swift runtime failure. + /// + /// These are things that are trapped by Swift itself at runtime, for + /// example divide by zero or arithmetic overflow. + public var isSwiftRuntimeFailure: Bool { get } + + /// True if this symbol is a Swift thunk function. + public var isSwiftThunk: Bool { get } + + /// True if this symbol represents a system function. + /// + /// System frames are generally things that people not involved in + /// compiler or runtime development would not be interested in, for + /// instance runtime initialisation routines that happen before + /// the Swift program is started, or runtime support code for the + /// Swift Concurrency system. + public var isSystem: Bool { get } + + /// Construct a new Symbol. + public init(image: Backtrace.Image, rawName: String, offset: Int, + sourceLocation: SourceLocation?) + + /// A textual description of this symbol. + public var description: String { get } + } + + /// A list of captured frame information. + public var frames: some Sequence { get } + + /// A list of images found in the process. + public var images: [Backtrace.Image] + + /// True if this backtrace is a Swift runtime failure. + public var isSwiftRuntimeFailure: Bool { get } + + /// Provide a textual version of the backtrace. + public var description: String { get } +} +``` + +Example usage: + +```swift +import Runtime + +var backtrace = Backtrace.capture() + +print(backtrace) + +var symbolicated = backtrace.symbolicated() + +print(symbolicated) +``` + +## Source compatibility + +This proposal is entirely additive. There are no source compatibility +concerns. + +## Effect on ABI stability + +The addition of this API will not be ABI-breaking, although as with any +new additions to the standard library it will constrain future versions +of Swift to some extent. + +## Effect on API resilience + +Once added, some changes to this API will be ABI and source-breaking +changes. Changes to the new structs/classes will be restricted as +described in the [library evolution +document](https://github.com/apple/swift/blob/master/docs/LibraryEvolution.rst) +in the Swift repository. + +## Alternatives considered + +This could have been addressed by creating a separate Swift package, +or by updating the existing [swift-server/swift-backtrace +package](https://github.com/swift-server/swift-backtrace). + +The latter focuses explicitly on Linux and Windows, and has +significant limitations, in addition to which we would like for this +functionality to be built in to Swift---just as it is built into +competing languages. This is why we felt it should be built into the +Swift runtime itself. + +The `Address` type could have been a fixed width integer, but that +loses some flexibility, both in terms of backtrace storage, and in our +ability to cope with backtraces from a platform other than the host. +It could also have been a protocol, but that then necessitates the use +of existentials; or it could have been a generic parameter, but doing +that makes it difficult to cope with a backtrace unless you already +know what kind of addresses it contains at compile time. + +The `frames` member variables could have been arrays, but implementing +them instead as a sequence means that we have the flexibility to use +a different backing store where doing so makes sense. An example where +we might want that is where we're capturing very large numbers of +backtraces, in which case doing some kind of delta compression on the +frame addresses might enable us to save significant amounts of memory. + +Some desirable features are intentionally left out of this proposal; +the intent is that while some of these may even be implemented, they +will remain SPI and may be promoted to API at a later date. Examples +include the ability to construct a `Backtrace` from an array of +addresses that have been gathered through some other mechanism; +provision for advanced formatting of backtraces; and features to +allow backtraces to be captured from another thread or process. + +## Acknowledgments + +Thanks to Jonathan Grynspan and Mike Ash for their helpful comments +on this proposal. diff --git a/proposals/0420-inheritance-of-actor-isolation.md b/proposals/0420-inheritance-of-actor-isolation.md new file mode 100644 index 0000000000..14f83d2603 --- /dev/null +++ b/proposals/0420-inheritance-of-actor-isolation.md @@ -0,0 +1,580 @@ +# Inheritance of actor isolation + +* Proposal: [SE-0420](0420-inheritance-of-actor-isolation.md) +* Authors: [John McCall](https://github.com/rjmccall), [Holly Borla](https://github.com/hborla), [Doug Gregor](https://github.com/douggregor) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-inheriting-the-callers-actor-isolation/68391)) ([review](https://forums.swift.org/t/se-0420-inheritance-of-actor-isolation/69638)) ([acceptance](https://forums.swift.org/t/accepted-se-0420-inheritance-of-actor-isolation/69913)) + +[SE-0302]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md +[SE-0304-propagation]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0304-structured-concurrency.md#actor-context-propagation +[SE-0306]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.md +[SE-0313]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0313-actor-isolation-control.md +[SE-0316]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0316-global-actors.md +[SE-0336]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0336-distributed-actor-isolation.md +[SE-0338]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0338-clarify-execution-non-actor-async.md +[SE-0392]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md + +## Introduction + +Under Swift's [actors design][SE-0306], every function in Swift has +an actor isolation: it is either isolated to some specific actor or +non-isolated. It is sometimes useful to be able to give a function +the same actor isolation as its caller, either to give it access to +actor-isolated data or just to avoid unnecessary suspensions. This +proposal allows `async` functions to opt in to this behavior. + +## Motivation + +The actor isolation of a function controls whether and how the +function can access actor-isolated data. An isolated function +can synchronously access the isolated storage of its actor, such +as the isolated properties of an [`actor` declaration][SE-0306] +or a global variable annotated with a [global actor attribute][SE-0316]. +When called from another function with the same isolation, it can +also pass and return non-[`Sendable`][SE-0302] values that are +isolated to the actor. A non-isolated function cannot do these +things, so making sure that functions share the same formal actor +isolation is sometimes important in order to safely express certain +patterns. + +Actor isolation also affects how the function is executed. Calls +and returns between functions with different actor isolations +may require the task to be suspended and then enqueued on a +different executor.[^1] Even when this is not required, there is +typically some overhead associated with the switch. Programmers +trying to optimize `async` code often find that avoiding these +overheads is important. Avoiding extra suspensions from +actor-isolated code can also be semantically important because +code from other tasks can interleave on the actor during suspensions, +potentially changing the values stored in isolated storage; +this is guaranteed not to happen at the moments of call and return +between functions with the same isolation. + +[^1]: This always happens when one of the functions is isolated +to an actor with a [custom actor executor][SE-0392], such as the +main actor (which uses a custom executor to ensure that execution +always happens on the main thread). For other actors, it typically +only happens when the actor is contended. + +Non-isolated synchronous functions dynamically inherit the isolation +of their caller. For example, an `actor` method can call a non-isolated +synchronous function, and the function will behave dynamically as if it +is isolated to the actor. While the function cannot directly access +actor-isolated storage --- it would need to be statically isolated to +the actor to do that --- it can be passed and return non-`Sendable` +values that are isolated to the actor. Among other things, this means +that you can call a function like `map` on an actor-isolated `Array` +of non-`Sendable` values; you can even pass it an actor-isolated +function, and everything will run synchronously and without suspension. + +However, there is currently no way to get this same effect from an +asynchronous function. [SE-0338][] clarified that non-isolated +`async` functions do not inherit isolation in this same way; instead, +they reset isolation.[^2] This means that these functions cannot +get passed and return non-`Sendable` data when called from an isolated +context, which can be a serious expressivity restriction, especially +for higher-order functions. It may also cause unwanted suspensions. + +[^2]: Prior to SE-0338, non-isolated asynchronous functions still +didn't properly inherit their caller's isolation: they just didn't +actively switch away. As a result, they ran with whatever isolation +they happened to the called or resumed with. That is not good enough +to allow them to safely be passed actor-isolated data or to make +strong guarantees of a lack of suspensions. This is now an ABI +constraint: even if we wanted to change the language to make these +functions inherit their caller's isolation by default, they aren't +passed that information reliably and have no way to implement those +semantics. + +For example, consider the following code that calls `next` on an instance +of `AsyncStream.Iterator` from the `@MainActor`: + +```swift +@MainActor func iterate(over stream: AsyncStream) async { + var iterator = stream.makeAsyncIterator() + while let element = await iterator.next() { + // do something with 'element' + } +} +``` + +The above code produces a warning: + +``` +warning: passing argument of non-sendable type 'inout AsyncStream.Iterator' outside of main actor-isolated context may introduce data races + while let element = await iterator.next() { + ^ +``` + +This happens because `AsyncIteratorProtocol.next()` is a non-isolated +asynchronous function, and most concrete `AsyncIteratorProtocol` types +including `AsyncStream.Iterator` are not `Sendable`. If `next()` is called +from another non-isolated asynchronous function, everything's okay: +it can be passed an arbitrary function and work with arbitrary types. +But if it's called from an *isolated* asynchronous function, Swift will +treat the call as crossing an isolation barrier and enforce three restrictions: + +- First, the result of the call must be `Sendable`. This restriction prevents + `next()` from being used from an actor to produce non-`Sendable` element + values. + +- Second, the `self` argument to the call (the iterator) must + be `Sendable`. This restriction prevents `next()` from being + used from an actor for concrete async iterator types that are not + `Sendable`. + +- Finally, any other arguments to the function must be `Sendable`. + This particular example doesn't have other function arguments, but + this restriction prevents non-isolated `async` functions from using + any other data that's isolated to the actor in the general case. + +In summary, these restrictions unnecessarily limit the capability of +the API when used from an isolated context. Furthermore, even +if the API is usable (e.g. because all the types involved are +`Sendable`), it may be unexpectedly inefficient if, say, the +element-producing closure is actor isolated, because `next()` will +hop to the generic executor only to immediately hop to the isolation +domain of the closure. + +This proposal addresses this problem by giving programmers better +tools for formally inheriting isolation from their caller, allowing +non-`Sendable` data to be safely passed back and forth and +avoiding unnecessary suspensions. + +## Proposed solution + +This proposal makes two changes to the language: + +- First, [SE-0313][]'s `isolated` parameters can now have optional + type. This is required in order for them to express that the + function should be dynamically non-isolated. + +- Second, default argument expressions can now have the special form + `#isolation`, which will be filled in with the actor isolation of + the caller. If the default argument is for an `isolated` parameter, + this allows isolation to be implicitly passed down. + +## Detailed design + +### Design approach + +The basic design approach of this proposal is to first enable +polymorphism over actor isolation, so that a function can declare +itself to have an arbitrary dynamic isolation, then add features +to allow that to be implicitly propagated in calls to the function. +The isolation logic can then recognize calls that propagate the +caller's isolation in sufficiently obvious ways and know that the +callee will share the current context's isolation. + +A function can be non-isolated, isolated to a specific actor instance, +or isolated to a global actor type. Dynamically, however, global actor +isolation is really just isolation to the `shared` instance of the +global actor, so a function's isolation can actually be dynamically +represented as just an optional actor reference, with `nil` +representing non-isolation. + +Since isolation is unavoidably value-dependent (an actor method is +isolated to a *specific* actor reference, not just any actor of that +type), polymorphism over it can't be expressed with just generics. +The natural next choice is to just use a parameter of polymorphic +type, such as `(any Actor)?`. This matches [SE-0313][]'s `isolated` +parameter feature, except that `isolated` parameters are currently +required to be non-optional actor types: either a concrete `actor` +type or a protocol type which implies `Actor`. Generalizing this +is straightforward and gives us the ability to make functions +explicitly polymorphic over an arbitrary isolation. + +Allowing arbitrary isolation to implicitly propagate from caller to +callee is a little trickier. If isolation is specified as a parameter, +then the caller must implicitly provide an argument to it; the most +obvious way to do that is to create a new special form for default +arguments, like `#line`, which expands to an expression that +evaluates to the isolation of the caller. + +### Generalized `isolated` parameters + +The type of an `isolated` parameter must be an *isolation type*. +Currently, the only kind of isolation is a possibly-optional actor type, +which is to say, either `T` or `Optional`, where `T` either conforms +to `Actor` or is a protocol type that implies `Actor`. + +If a function's `isolated` parameter has an optional actor type, then +the dynamic isolation of the function depends on whether the argument +value is `nil`. If it is `nil`, then the function behaves dynamically +as it were non-isolated; for example, if the function is `async`, it +resets isolation on entry under [SE-0338][] just as a non-isolated +function would. Otherwise, the function behaves dynamically as it +were isolated to the unwrapped actor reference. + +According to [SE-0304][SE-0304-propagation], closures passed directly +to the `Task` initializer (i.e. `Task { /*here*/ }`) inherit the +statically-specified isolation of the current context if: + +- the current context is non-isolated, +- the current context is isolated to a global actor, or +- the current context has an `isolated` parameter (including the + implicit `self` of an actor method) and that parameter is strongly + captured by the closure. + +The third clause is modified by this proposal to say that isolation +is also inherited if a non-optional binding of an isolated parameter +is captured by the closure. A non-optional binding of an isolated +parameter is defined in the +[generalized isolation checking](#generalized-isolation-checking) section. + +### Isolated distributed actors + +There is currently no type or protocol that enables abstracting over both +actor and distributed actor isolation using isolated parameters. The +[Distributed actor isolation][SE-0336] proposal introduced the +`DistributedActor` protocol as a separate protocol from `Actor` because +distributed actors only behave like actors when they are known to be +local. An `isolated` distributed actor parameter is known to be local, so +it has the capabilities of an actor. The following local API on +`DistributedActor` is provided to return a local actor instance from a +distributed actor, enabling distributed actors to be used with isolated +parameters of type `isolated any Actor` and `isolated (any Actor)?`: + +```swift +@available(SwiftStdlib 5.7, *) +extension DistributedActor { + /// Produces an erased `any Actor` reference to this known to be local distributed actor. + /// + /// Since this method is not distributed, it can only be invoked when the underlying + /// distributed actor is known to be local, e.g. from a context that is isolated + /// to this actor. + /// + /// Such reference can be used to work with APIs accepting `isolated any Actor`, + /// as only a local distributed actor can be isolated on and may be automatically + /// erased to such `any Actor` when calling methods implicitly accepting the + /// caller's actor isolation, e.g. by using the `#isolation` macro. + @backDeployed(before: SwiftStdlib 6.0) + public var asLocalActor: any Actor { +} +``` + +### Generalized isolation checking + +When calling a function with an `isolated` parameter, the function +shares the same isolation as the current context if: + +- the current context is non-isolated, the parameter type is optional, + and the argument expression is `nil` or a reference to `Optional.none`; + +- the current context has an `isolated` parameter (including the + implicitly-`isolated` `self` parameter of an actor function) and + the argument expression is a reference to that parameter, a + non-optional derivation of it (see below), or a local actor derivation + from a distributed actor using `DistributedActor.asAnyActor`; or + +- the current context is isolated to a global actor type `T` and the + argument expression is `T.shared`, where `shared` is `GlobalActor`'s + protocol requirement or the concrete declaration which provides it + in `T`'s conformance to `GlobalActor`. + +An expression is a non-optional derivation of an isolated parameter +`param` if it is: +- `param?` (the optional-chaining operator); +- `param!` (the force-unwrapping operator); or +- a reference to a *non-optional binding* of `param`, i.e. a `let` + constant initialized by a successful pattern-match which removes + the optionality from `param`, such as `ref` in `if let ref = param`. + +When analyzing an argument expression in all cases above, certain +non-instrumental differences in expression syntax and behavior must +be ignored: +- parentheses; +- the effect-marking operators `try`, `try?`, `try!`, and `await`;[^5] +- the type coercion operator `as` (in the cases where it doesn't + perform a dynamic bridging conversion); and +- implicit type conversions such as promotion to `Optional` type. + +[^5]: The restrictions on the underlying expression should make it +pointless to use these operators, but they must be ignored anyway. + +Note that the special `#isolation` default argument form should +always be replaced by something matching the rule above, so calls +using this default argument for an isolated parameter will always be +to a context that shares isolation. + +For example: + +```swift +/// This class type is not Sendable. +class Counter { + var count = 0 +} + +extension Counter { + /// Since this is an async function, if it were just declared + /// non-isolated, calling it from an isolated context would be + /// forbidden because it requires sharing a non-Sendable value + /// between concurrency domains. Inheriting isolation makes it + /// okay. This is a contrived example chosen for its simplicity. + func incrementAndSleep(isolation: isolated (any Actor)?) async { + count += 1 + await Task.sleep(nanoseconds: 1_000_000) + } +} + +actor MyActor { + var counter = Counter() +} + +extension MyActor { + func testActor(other: MyActor) { + // allowed + await counter.incrementAndSleep(isolation: self) + + // not allowed + await counter.incrementAndSleep(isolation: other) + + // not allowed + await counter.incrementAndSleep(isolation: MainActor.shared) + + // not allowed + await counter.incrementAndSleep(isolation: nil) + } +} + +@MainActor func testMainActor(counter: Counter) { + // allowed + await counter.incrementAndSleep(isolation: MainActor.shared) + + // not allowed + await counter.incrementAndSleep(isolation: nil) +} + +func testNonIsolated(counter: Counter) { + // allowed + await counter.incrementAndSleep(isolation: nil) + + // not allowed + await counter.incrementAndSleep(isolation: MainActor.shared) +} +``` + +### `#isolation` default argument + +The special expression form `#isolation` can be used in arbitrary +expression position: + +```swift +extension AsyncIteratorProtocol { + func next(isolation: isolated (any Actor)? = #isolation) async -> Element { + ... + } +} +``` + +When a call uses `#isolation` as the argument to an isolated parameter, +it behaves as if the argument was an expression representing the static +actor isolation of the current context: + +- if the current context is statically non-isolated, the parameter + must have optional type, and the argument is `nil`; +- if the current context is isolated to a global actor `T`, the argument + is `T.shared`; +- if the current context has an `isolated` actor parameter (including the + implicit `self` parameter of an actor method), the argument is a + reference to that parameter; +- if the current context has an `isolated` distributed actor parameter + `d` (including the implicit `self` parameter of a distributed actor + method), the argument is `d.asAnyActor`; +- otherwise, the current context must be a closure which captures + an `isolated` parameter or a non-optional binding of it, and the + argument is a reference to that capture. + +The type of `#isolation` depends on the type annotation provided in the +context of the expression, similar to other builtin macros such as `#file` +and `#line`, with a default type of `(any Actor)?` if no contextual type +is provided. When type-checking considers a candidate function for a call +that would use `#isolation` as an argument for a parameter, +it assumes that the notional argument expression above can be coerced +to the parameter type. If the call is actually resolved to use that +candidate, the coercion must succeed or the call is ill-formed. +This rule is necessary in order to avoid the need to decide the isolation +of the calling context before resolving calls from it. + +The parameter does not have to be an `isolated` parameter. + +## Source compatibility + +This proposal is largely additive and should not affect the behavior +of existing code. + +The new rules for isolation checking permit more calls to be +recognized as sharing isolation. This should strictly allow +more code to be compiled; it cannot cause source-compatibility +regressions by allowing different overloads to be picked because +isolation checking is performed separately from type-checking. + +## ABI compatibility + +This proposal does not change how any existing code is compiled. + +## Implications for adoption + +This proposal does not add any new types and does not require new +runtime or library support. It can be implemented purely in the compiler. + +Adding `#isolation` as a default argument to an existing parameter is not +ABI-breaking, but this is probably an uncommon situation. Adding a new +parameter to an existing declaration is ABI-breaking, of course. + +Making a library function inherit isolation is effectively a promise that +it can work when called from any isolated context. While this might seem +superficially like a pretty strong guarantee, it's not very different +in practice from just making the library function non-isolated: in both +cases, the function does not have any isolation preconditions that it can +rely on. Library authors should not be reserved about adopting this +proposal on that account. + +A better reason to be cautious about adopting this feature is that it +can cause more work to be done while actor-isolated, potentially creating +significant "hangover" on the actor lock and a less effective use of +concurrency. It may be better for the whole system if functions that do +significant computational work, including doing a lot of object +allocation and initialization, stay non-isolated rather than +isolation-inheriting. On the other hand, `async` functions with "fast +paths" --- functions that usually return quickly and only occasionally +need to set up more expensive work --- may see real benefits from +extracting the fast path into a function that inherits isolation and +then leaving the slow path in a non-isolated function. + +## Future directions + +### Syntax sugar for inheriting actor isolation + +Isolated parameters have three downsides. + +The first downside is that the use pattern we expect to dominate --- +declaring a function to inherit its caller's isolation --- is pretty +cumbersome: + +```swift +func foo(isolation: isolated (any Actor)? = #isolation) +``` + +The second downside is we can only do this if we can add formal +parameters to a function. Unfortunately, there are several places +in the language where we really can't do that, most importantly +accessors for computed properties: + +```swift +var count: Int { + get { // How do we add an isolated parameter here? + ... + } +} +``` + +The third downside is minor in comparison, but this pattern naturally +turns into passing an actor reference, which isn't the most efficient +way of passing down actor isolation because it still requires dynamic +dispatch in order to extract the executor. It would be better for the +implementation if we could pass down the exact `UnownedSerialExecutor?` +value that's needed at runtime. While we do not currently want to +encourage programmers to work with values of this type directly because +of its tricky lifetime semantics, the compiler can manage it fairly +easily. + +All of these downsides could be addressed by adding an attribute +which causes an entity (including an accessor) to inherit its caller's +isolation. This would be equivalent to receiving an `isolated` parameter +with the same value as would be produced by `#isolated`, but it's easier +to write, can be used in a few places that can't add arbitrary parameters, +and may be more efficiently implementable. + +### Isolated function types + +This proposal is focused on propagating isolation information *into* +functions, but it's also interesting to look at propagating isolation +*out* of functions. Currently, the Swift type system only allows +function isolation to be expressed in limited ways: functions can be +declared as isolated to a global actor (e.g. `@MainActor () -> ()`), but +all other kinds of isolation must be "type-erased", leaving a value +whose type appears to be non-isolated. + +One way to solve this would be to introduce value-dependent isolated +function types. With such a feature, you could declare a value to have +type, say, `@isolated(myActor) () -> ()`, where `myActor` is a `let` +constant in the local scope. This kind of value dependence, however, +is a large step in complexity for a type system, and it's not a likely +path for Swift in the foreseeable future. + +A more promising approach would be to allow the isolation to be +statically erased but still make it dynamically recoverable by carrying +it along in the function value, essentially as an extra value of type +`(any Actor)?`. A function type that supports dynamically recovering +the isolation would look something like `@isolated () -> ()`, +and it could be used to e.g. dynamically propagate the isolation of +a function into something like the `Task` initializer so that the task +can immediately start on the right executor. This would compose well +with the features in this proposal because it would be natural to allow +such functions to be used as `isolated` parameters. This would be very +nice for functions like `sequentialMap` that should probably be isolated +not to their *caller* but to the *function they've been passed*: + +```swift +extension Collection { + func sequentialMap(transform: (Element) async -> R) async -> [R] { + var results: [R] = [] + for elt in self { + results.append(await transform(elt)) + } + return results + } +} +``` + +## Alternatives Considered + +### Allowing isolation to `SerialExecutor` types + +This proposal observes that it is more efficient to pass down an +`UnownedSerialExecutor` value instead of an actor reference. However, a +function cannot use this more efficient pattern because an `isolated` +parameter must be an actor type. This is an intentional decision. + +Philosophically, Swift programmers should be encouraged to think about +actors in terms of isolation rather than execution policy. There are many +ways for actors to provide isolation, many of which don't require taking +over execution; in fact, Swift's actors use one such approach by default. +Keeping the focus on actors rather than executors supports this. + +Putting that aside, there also just isn't a reasonable type that could +be used here: + +- `UnownedSerialExecutor` is an unsafe type that requires the compiler +to implicitly manage a dependency on the underlying actor or executor +reference in order to safely use. While this is not difficult for the +compiler, we do not want to encourage programmers to use this type +directly. If Swift introduces a safe replacement in the future, possibly +using future language support for value dependencies, we can consider +allowing that to be used as an `isolated` parameter type at that time. + +- A managed serial executor reference such as `any SerialExecutor` +would be a safe alternative, but it's a surprisingly complex one. +For one, normal isolated contexts would not to be able to implement +`#isolation` forwarding to such a parameter, because there's currently no +way to get a managed serial executor reference from an actor, only an +`UnownedSerialExecutor`. For another, actor types can (and often do) +also conform to `SerialExecutor`, but there's nothing in the language +requiring those actors to always use `self` as their executor. This +greatly complicates the logic for both establishing and forwarding +isolation; e.g. an isolated *actor* parameter must not be forwarded +directly as an isolated *executor* (as opposed to extracting the +correct executor reference) even if the actor's type would normally +implicitly convert. Furthermore, the decision logic for whether a call +crosses isolation would have to recognize expressions that extract serial +executors, as well as appropriately reasoning about actor/executor +differences. And finally, getting an `UnownedSerialExecutor` from an +`any SerialExecutor` still requires calling a protocol method, so it's +not really enabling any sort of optimization. + +## Acknowledgments + +I'd like to especially thank Konrad Malawski, and Doug Gregor +for their help in developing the ideas in this proposal. diff --git a/proposals/0421-generalize-async-sequence.md b/proposals/0421-generalize-async-sequence.md new file mode 100644 index 0000000000..c0c19c4550 --- /dev/null +++ b/proposals/0421-generalize-async-sequence.md @@ -0,0 +1,271 @@ +# Generalize effect polymorphism for `AsyncSequence` and `AsyncIteratorProtocol` + +* Proposal: [SE-0421](0421-generalize-async-sequence.md) +* Authors: [Doug Gregor](https://github.com/douggregor), [Holly Borla](https://github.com/hborla) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-generalize-asyncsequence-and-asynciteratorprotocol/69283))([review](https://forums.swift.org/t/se-0421-generalize-effect-polymorphism-for-asyncsequence-and-asynciteratorprotocol/69662)) ([acceptance](https://forums.swift.org/t/accepted-se-0421-generalize-effect-polymorphism-for-asyncsequence-and-asynciteratorprotocol/69973)) + +## Introduction + +This proposal generalizes `AsyncSequence` in two ways: +1. Proper `throws` polymorphism is accomplished with adoption of typed throws. +2. A new overload of the `next` requirement on `AsyncIteratorProtocol` includes an isolated parameter to abstract over actor isolation. + +## Table of Contents + +* [Introduction](#introduction) +* [Motivation](#motivation) +* [Proposed solution](#proposed-solution) +* [Detailed design](#detailed-design) + + [Adopting typed throws](#adopting-typed-throws) + - [Error type inference from `for try await` loops](#error-type-inference-from-for-try-await-loops) + + [Adopting primary associated types](#adopting-primary-associated-types) + + [Adopting isolated parameters](#adopting-isolated-parameters) + + [Default implementations of `next()` and `next(isolation:)`](#default-implementations-of-next-and-nextisolation) + + [Associated type inference for `AsyncIteratorProtocol` conformances](#associated-type-inference-for-asynciteratorprotocol-conformances) +* [Source compatibility](#source-compatibility) +* [ABI compatibility](#abi-compatibility) +* [Implications on adoption](#implications-on-adoption) +* [Future directions](#future-directions) + + [Add a default argument to `next(isolation:)`](#add-a-default-argument-to-nextisolation) +* [Alternatives considered](#alternatives-considered) + + [Avoiding an existential parameter in `next(isolation:)`](#avoiding-an-existential-parameter-in-nextisolation) +* [Acknowledgments](#acknowledgments) + +## Motivation + +`AsyncSequence` and `AsyncIteratorProtocol` were intended to be polymorphic over the `throws` effect and actor isolation. However, the current API design has serious limitations that impact expressivity in generic code, `Sendable` checking, and runtime performance. + +Some `AsyncSequence`s can throw during iteration, and others never throw. To enable callers to only require `try` when the given sequence can throw, `AsyncSequence` and `AsyncIteratorProtocol` used an experimental feature to try to capture the throwing behavior of a protocol. However, this approach was insufficiently general, which has also [prevented `AsyncSequence` from adopting primary associated types](https://forums.swift.org/t/se-0346-lightweight-same-type-requirements-for-primary-associated-types/55869/70). Primary associated types on `AsyncSequence` would enable hiding concrete implementation details behind constrained opaque or existential types, such as in transformation APIs on `AsyncSequence`: + +```swift +extension AsyncSequence { + // 'AsyncThrowingMapSequence' is an implementation detail hidden from callers. + public func map( + _ transform: @Sendable @escaping (Element) async throws -> Transformed + ) -> some AsyncSequence { ... } +} +``` + +Additionally, `AsyncSequence` types are designed to work with `Sendable` and non-`Sendable` element types, but it's currently impossible to use an `AsyncSequence` with non-`Sendable` elements in an actor-isolated context: + +```swift +class NotSendable { ... } + +@MainActor +func iterate(over stream: AsyncStream) { + for await element in stream { // warning: non-sendable type 'NotSendable?' returned by implicitly asynchronous call to nonisolated function cannot cross actor boundary + + } +} +``` + +Because `AsyncIteratorProtocol.next()` is `nonisolated async`, it always runs on the generic executor, so calling it from an actor-isolated context crosses an isolation boundary. If the result is non-`Sendable`, the call is invalid under strict concurrency checking. + +More fundamentally, calls to `AsyncIteratorProtocol.next()` from an actor-isolated context are nearly always invalid in practice today. Most concrete `AsyncIteratorProtocol` types are not `Sendable`; concurrent iteration using `AsyncIteratorProtocol` is a programmer error, and the iterator is intended to be used/mutated from the isolation domain that formed it. However, when an iterator is formed in an actor-isolated context and `next()` is called, the non-`Sendable` iterator is passed across isolation boundaries, resulting in a diagnostic under strict concurrency checking. + +Finally, `next()` always running on the generic executor is the source of unnecessary hops between an actor and the generic executor. + +## Proposed solution + +This proposal introduces a new associated type `Failure` to `AsyncSequence` and `AsyncIteratorProtocol`, adopts both `Element` and `Failure` as primary associated types, adds a new protocol requirement to `AsyncIteratorProtocol` that generalizes the existing `next()` requirement by throwing the `Failure` type, and adds an `isolated` parameter to the new requirement to abstract over actor isolation: + +```swift +@available(SwiftStdlib 5.1, *) +protocol AsyncIteratorProtocol { + associatedtype Element + + mutating func next() async throws -> Element? + + @available(SwiftStdlib 6.0, *) + associatedtype Failure: Error = any Error + + @available(SwiftStdlib 6.0, *) + mutating func next(isolation actor: isolated (any Actor)?) async throws(Failure) -> Element? +} + +@available(SwiftStdlib 5.1, *) +public protocol AsyncSequence { + associatedtype AsyncIterator: AsyncIteratorProtocol + associatedtype Element where AsyncIterator.Element == Element + + @available(SwiftStdlib 6.0, *) + associatedtype Failure = AsyncIterator.Failure where AsyncIterator.Failure == Failure + + func makeAsyncIterator() -> AsyncIterator +} +``` + +The new `next(isolation:)` has a default implementation so that conformances will continue to behave as they do today. Code generation for `for-in` loops will switch over to calling `next(isolation:)` instead of `next()` when the context has appropriate availability. + +## Detailed design + +### Adopting typed throws + +Concrete `AsyncSequence` and `AsyncIteratorProtocol` types determine whether calling `next()` can `throw`. This can be described in each protocol with a `Failure` associated type that is thrown by the `AsyncIteratorProtocol.next(isolation:)` requirement. Describing the thrown error with an associated type allows conformances to fulfill the requirement with a type parameter, which means that libraries do not need to expose separate throwing and non-throwing concrete types that otherwise have the same async iteration functionality. + +#### Error type inference from `for try await` loops + +The `Failure` associated type is only accessible at runtime in the Swift 6.0 standard library; code running against older standard library versions does not include the `Failure` requirement in the witness tables for `AsyncSequence` and `AsyncIteratorProtocol` conformances. This impacts error type inference from `for try await` loops. + +When the thrown error type of an `AsyncIteratorProtocol` is available, either through the associated type witness (because the context has appropriate availability) or because the iterator type is concrete, iteration over an async sequence throws its `Failure` type: + +```swift +struct MyAsyncIterator: AsyncIteratorProtocol { + typealias Failure = MyError + ... +} + +func iterate(over s: S) where S.AsyncIterator == MyAsyncIterator { + let closure = { + for try await element in s { + print(element) + } + } +} +``` + +In the above code, the type of `closure` is `() async throws(MyError) -> Void`. + +When the thrown error type of an `AsyncIteratorProtocol` is not available, iteration over an async sequence throws `any Error`: + +```swift +@available(SwiftStdlib 5.1, *) +func iterate(over s: some AsyncSequence) { + let closure = { + for try await element in s { + print(element) + } + } +} +``` + +In the above code, the type of `closure` is `() async throws(any Error) -> Void`. + +When the `Failure` type of the given async sequence is constrained to `Never`, `try` is not required in the `for-in` loop: + +```swift +struct MyAsyncIterator: AsyncIteratorProtocol { + typealias Failure = Never + ... +} + +func iterate(over s: S) where S.AsyncIterator == MyAsyncIterator { + let closure = { + for await element in s { + print(element) + } + } +} +``` + +In the above code, the type of `closure` is `() async -> Void`. + +### Adopting primary associated types + +The `Element` and `Failure` associated types are promoted to primary associated types. This enables using constrained existential and opaque `AsyncSequence` and `AsyncIteratorProtocol` types, e.g. `some AsyncSequence` or `any AsyncSequence`. + +### Adopting isolated parameters + +The `next(isolation:)` requirement abstracts over actor isolation using [isolated parameters](/proposals/0313-actor-isolation-control.md). For callers to `next(isolation:)` that pass an iterator value that cannot be transferred across isolation boundaries under [SE-0414: Region based isolation](/proposals/0414-region-based-isolation.md), the call is only valid if it does not cross an isolation boundary. Explicit callers can pass in a value of `#isolation` to use the isolation of the caller, or `nil` to evaluate `next(isolation:)` on the generic executor. + +Desugared async `for-in` loops will call `AsyncIteratorProtocol.next(isolation:)` instead of `next()` when the context has appropriate availability, and pass in an isolated argument value of `#isolation` of type `(any Actor)?`. The `#isolation` macro always expands to the isolation of the caller so that the call does not cross an isolation boundary. + +### Default implementations of `next()` and `next(isolation:)` + +Because existing `AsyncIteratorProtocol`-conforming types only implement `next()`, the standard library provides a default implementation of `next(isolation:)`: + +```swift +extension AsyncIteratorProtocol { + /// Default implementation of `next(isolation:)` in terms of `next()`, which is + /// required to maintain backward compatibility with existing async iterators. + @available(SwiftStdlib 6.0, *) + @available(*, deprecated, message: "Provide an implementation of 'next(isolation:)'") + public mutating func next(isolation actor: isolated (any Actor)?) async throws(Failure) -> Element? { + nonisolated(unsafe) var unsafeIterator = self + do { + let element = try await unsafeIterator.next() + self = unsafeIterator + return element + } catch { + throw error as! Failure + } + } +} +``` + +Note that the default implementation of `next(isolation:)` necessarily violates `Sendable` checking in order to pass `self` from a possibly-isolated context to a `nonisolated` one. Though this is generally unsafe, this is how calls to `next()` behave today, so existing conformances will maintain the behavior they already have. Implementing `next(isolation:)` directly will eliminate the unsafety. + +To enable conformances of `AsyncIteratorProtocol` to only implement `next(isolation:)`, a default implementation is also provided for `next()`: + +```swift +extension AsyncIteratorProtocol { + @available(SwiftStdlib 6.0, *) + public mutating func next() async throws -> Element? { + // Callers to `next()` will always run `next(isolation:)` on the generic executor. + try await next(isolation: nil) + } +} +``` + +Both function requirements of `AsyncIteratorProtocol` have default implementations that are written in terms of each other, meaning that it is a programmer error to implement neither of them. Types that are available prior to the Swift 6.0 standard library must provide an implementation of `next()`, because the default implementation is only available with the Swift 6.0 standard library. + +To avoid silently allowing conformances that implement neither requirement, and to facilitate the transition of conformances from `next()` to `next(isolation:)`, we add a new availability rule where the witness checker diagnoses a protocol conformance that uses an deprecated, obsoleted, or unavailable default witness implementation. Deprecated implementations will produce a warning, while obsoleted and unavailable implementations will produce an error. + +Because the default implementation of `next(isolation:)` is deprecated, conformances that do not provide a direct implementation will produce a warning. This is desirable because the default implementation of `next(isolation:)` violates `Sendable` checking, so while it's necessary for source compatibility, it's important to aggressively suggest that conforming types implement the new method. + +### Associated type inference for `AsyncIteratorProtocol` conformances + +When an `AsyncIteratorProtocol`-conforming type provides a `next(isolation:)` function, the `Failure` type is inferred based on whether (and what) `next(isolation:)` throws using the rules described in [SE-0413](/proposals/0413-typed-throws.md). + +If the `AsyncIteratorProtocol`-conforming type uses the default implementation of `next(isolation:)`, then the `Failure` associated type is inferred from the `next` function instead. Whatever type is thrown from the `next` function (including `Never` if it is non-throwing) is inferred as the `Failure` type. + +## Source compatibility + +The new requirements to `AsyncSequence` and `AsyncIteratorProtocol` are additive, with default implementations and `Failure` associated type inference heuristics that ensure that existing types that conform to these protocols will continue to work. + +The experimental "rethrowing conformances" feature used by `AsyncSequence` and `AsyncIteratorProtocol` presents some challenges for source compatibility. Namely, one can declare a `rethrows` function that considers conformance to these rethrowing protocols as sources of errors for rethrowing. For example, the following `rethrows` function is currently valid: + +```swift +extension AsyncSequence { + func contains(_ value: Element) rethrows -> Bool where Element: Hashable { ... } +} +``` + +With the removal of the experimental "rethrowing conformances" feature, this function becomes ill-formed because there is no closure argument that can throw. To preserve source compatibility for such functions, this proposal introduces a specific rule that allows requirements on `AsyncSequence` and `AsyncIteratorProtocol` to be involved in `rethrows` checking: a `rethrows` function is considered to be able to throw `T.Failure` for every `T: AsyncSequence` or `T: AsyncIteratorProtocol` conformance requirement. In the case of this `contains` operation, that means it can throw `Self.Failure`. The rule permitting the definition of these `rethrows` functions will only be permitted prior to Swift 6. + +## ABI compatibility + +This proposal is purely an extension of the ABI of the standard library and does not change any existing features. Note that the addition of a new `next(isolation:)` requirement, rather than modifying the existing `next()` requirement, is necessary to maintain ABI compatibility, because changing `next()` to abstract over actor isolation requires passing the actor as a parameter in order to hop back to that actor after any `async` calls in the implementation. The typed throws ABI is also different from the rethrows ABI, so the adoption of typed throws alone necessitates a new requirement. + +## Implications on adoption + +The associated `Failure` types of `AsyncSequence` and `AsyncIteratorProtocol` are only available at runtime with the Swift 6.0 standard library, because code that runs against prior standard library versions does not have a witness table entry for `Failure`. Code that needs to access the `Failure` type through the associated type, e.g. to dynamic cast to it or constrain it in a generic signature, must be availability constrained. For this reason, the default implementations of `next()` and `next(isolation:)` have the same availability as the Swift 6.0 standard library. + +This means that concrete `AsyncIteratorProtocol` conformances cannot switch over to implementing `next(isolation:)` only (without providing an implementation of `next()`) if they are available earlier than the Swift 6.0 standard library. + +Similarly, primary associated types of `AsyncSequence` and `AsyncIteratorProtocol` must be gated behind Swift 6.0 availability. + +Once the concrete `AsyncIteratorProtocol` types in the standard library, such as `Async{Throwing}Stream.Iterator`, implement `next(isolation:)` directly, code that iterates over those concrete `AsyncSequence` types in an actor-isolated context may exhibit fewer hops to the generic executor at runtime. + +## Future directions + +### Add a default argument to `next(isolation:)` + +Most calls to `next(isolation:)` will pass the isolation of the enclosing context. We could consider lifting the restriction that protocol requirements cannot have default arguments, and adding a default argument value of `#isolated` as described in the [pitch for actor isolation inheritance](https://forums.swift.org/t/pitch-inheriting-the-callers-actor-isolation/68391). + +## Alternatives considered + +### Avoiding an existential parameter in `next(isolation:)` + +The isolated parameter to `next(isolation:)` has existential type `(any Actor)?` because a `nil` value is used to represent `nonisolated`. There is no concrete `Actor` type that describes a `nonisolated` context, which necessitates using `(any Actor)?` instead of `some Actor` or `(some Actor)?`. Potential alternatives to this are: + +1. Represent `nonisolated` with some other value than `nil`, or a specific declaration in the standard library that has a concrete optional actor type to enable `(some Actor)?`. Any solution in this category requires the compiler to have special knowledge of the value that represents `nonisolated` for actor isolation checking of the call. +2. Introduce a separate entrypoint for `next(isolation:)` that is always `nonisolated`. This defeats the purpose of having a single implementation of `next(isolation:)` that abstracts over actor isolation. + +Note that the use of an existential type `(any Actor)?` means that [embedded Swift](/visions/embedded-swift.md) would need to support class existentials in order to use `next(isolation:)`. + +## Acknowledgments + +Thank you to Franz Busch and Konrad Malawski for starting the discussions about typed throws and primary associated type adoption for `AsyncSequence` and `AsyncIteratorProtocol` in the [Typed throws in the Concurrency module](https://forums.swift.org/t/pitch-typed-throws-in-the-concurrency-module/68210/1) pitch. Thank you to John McCall for specifying the rules for generalized isolated parameters in the [pitch for inheriting the caller's actor isolation](https://forums.swift.org/t/pitch-inheriting-the-callers-actor-isolation/68391). diff --git a/proposals/0422-caller-side-default-argument-macro-expression.md b/proposals/0422-caller-side-default-argument-macro-expression.md new file mode 100644 index 0000000000..92b9f6f836 --- /dev/null +++ b/proposals/0422-caller-side-default-argument-macro-expression.md @@ -0,0 +1,196 @@ +# Expression macro as caller-side default argument + +* Proposal: [SE-0422](0422-caller-side-default-argument-macro-expression.md) +* Authors: [Apollo Zhu](https://github.com/ApolloZhu) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-expression-macro-as-caller-side-default-argument/69019)), ([review](https://forums.swift.org/t/se-0422-expression-macro-as-caller-side-default-argument/69730)), ([acceptance](https://forums.swift.org/t/accepted-se-0422-expression-macro-as-caller-side-default-argument/70050)) + +## Introduction + +This proposal aims to lift the restriction afore set in [SE-0382 "Expression macros"](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md) to allow non-built-in expression macros as caller-side default argument expressions. + +## Motivation + +Built-in magic identifiers like [#line](https://developer.apple.com/documentation/swift/line()) and [#fileID](https://developer.apple.com/documentation/swift/fileID()) are documented as expression macros in the official documentation, but if Swift developers try to implement a similar macro themselves and use it as the default argument for some function, the code will not compile: + +```swift +public struct MakeLabeledPrinterMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) throws -> ExprSyntax { + return "{ value in print(\"\\(#fileID):\\(#line): \\(value)\") }" + } +} + +public macro LabeledPrinter() -> (T) -> Void += #externalMacro(module: ..., type: "MakeLabeledPrinterMacro") + +public func greet( + _ thing: T, + print: (T) -> Void = #LabeledPrinter +// error: ^ non-built-in macro cannot be used as default argument +) { + print("Hello, \(thing)") +} +``` + +This is because built-in expression macros/magic identifiers have a special behavior: when used as default arguments, instead of been expanded at where the expressions are written like all other macros, they are expanded by the caller using the source-location information of the call site: + +```swift +// in MyLibrary.swift +public func greet(_ thing: T, file: String = #fileID) { + print("\(fileID): Hello, \(thing)" +} + +// in main.swift +greet("World") +// prints "main.swift: Hello, World" instead of "MyLibrary.swift: ... +``` + +This a useful existing behavior that should be supported, but could be surprising as it differs from all other macro expansions, and might not be desired for all expression macros. + +## Proposed solution + +The proposal lifts the restriction and makes non-built-in expression macros behave consistently as built-in magic identifier expression macros: + +* if expression macros are used as default arguments, they’ll be expanded with caller side source location information and context; +* if they are used as sub-expressions of default arguments, they’ll be expanded at where they are written + +```swift +// in MyLibrary.swift ======= +@freestanding(expression) +macro MyFileID() -> T = ... + +public func callSiteFile(_ file: String = #MyFileID) { file } + +public func declarationSiteFile(_ file: String = (#MyFileID)) { file } + +public func alsoDeclarationSiteFile( + file: String = callSiteFile(#MyFileID) +) { file } + +// in main.swift ============ +print(callSiteFile()) // print main.swift, the current file +print(declarationSiteFile()) // always prints MyLibrary.swift +print(alsoDeclarationSiteFile()) // always prints MyLibrary.swift +``` + +Macro author can inquire the source location information using `context.location(of:)` just like before and implement `#fileID`, `#line`, and `#column` as shown below: + +```swift +public struct MyFileIDMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + context.location( + of: node, at: .afterLeadingTrivia, filePathMode: .fileID + )!.file + } +} + +public struct MyLineMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + context.location(of: node)!.line + } +} + +public struct MyColumnMacro: ExpressionMacro { + public static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + context.location(of: node)!.column + } +} +``` + +## Detailed design + +### Type-checking default argument macro expressions + +Since the macro expanded expression might reference declarations that are not available in the scope where the function is declared, macro expressions are not expanded at the primary function declaration. However, macro expression used as a default argument is type checked without expansion to make sure that + +1. it is at least as visible as the function using it, +2. its return type matches what that parameter expects, and +3. its arguments, if any, are literals without string interpolation. + +### Type-checking macro expanded expressions + +For each call to a function that has an expression macro default argument, the macro will be expanded with each call-site’s source location and type-checked in the corresponding caller-side context, as if the macro expression is written at where it is expanded: + +```swift +@freestanding(expression) +// expands to `foo + bar` +public macro VariableReferences() -> String = ... + +public func preferVariablesFromCallerSide( + param: String = #VariableReferences +) { + print(param) +} + +// in another file ========== +var foo = "hi " +var bar = "caller" +preferVariablesFromCallerSide() // prints: hi caller +// ^ same as #VariableReferences written here +``` + +## Source compatibility + +As non-built-in macro expressions aren’t allowed as default argument, this change is purely additive and has no impact on existing code. + +## ABI compatibility + +This feature does not affect the ABI. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. + +## Future directions + +### Allow arguments to default argument macro expressions to be arbitrary expressions + +If these arguments can be arbitrary expressions, type-checking the macro expression at function declaration will require any declarations referenced in these expressions to be also in scope: + +```swift +@freestanding(expression) +// expands to: "Hello " + string +public macro PrependHello(_ string: String) -> String = ... + +// this is needed so it can be referenced in the default argument +public var shadowedVariable: String = "World" + +public func preferVariablesFromCallerSide( + param: String = #PrependHello(shadowedVariable) +) { + print(param) +} +``` + +However, as the expanded expression is type-checked in the caller-side context, it’s rather unintuitive that one must add the public variable in the example above, yet it might not be what the macro expanded expressions use. For example, if there's a variable with the same name in scope on the caller side, that variable will be used, and the call to the function might fail to type-check: + +```swift +// in another file ========== +var shadowedVariable: Int = 42 +preferVariablesFromCallerSide() +// #PrependHello(shadowedVariable) expands to "Hello " + 42 +// error: binary operator '+' cannot be applied to operands of type 'String' and 'Int' +``` + +## Alternatives considered + +### Expand non-built-in expression macro default arguments at the primary declaration + +While this allows all macro expansions to be expanded at where they are written, it creates an inconsistency for expression macros where they behave differently depending on whether they are built-in or not. Therefore, this alternative won’t be a solution for addressing the surprising behavior of built-in expression macros as caller-side default arguments, while the proposed solution unifies, and clarifies how to make expression macro default arguments expand at caller-side vs. at function declaration. + +## Acknowledgments + +Thanks to Doug Gregor, Richard Wei, and Holly Borla for early feedback and suggestions on design and implementation. diff --git a/proposals/0423-dynamic-actor-isolation.md b/proposals/0423-dynamic-actor-isolation.md new file mode 100644 index 0000000000..fb7215418e --- /dev/null +++ b/proposals/0423-dynamic-actor-isolation.md @@ -0,0 +1,182 @@ +# Dynamic actor isolation enforcement from non-strict-concurrency contexts + +* Proposal: [SE-0423](0423-dynamic-actor-isolation.md) +* Authors: [Holly Borla](https://github.com/hborla), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.0)** +* Upcoming Feature Flag: `DynamicActorIsolation` +* Review: ([pitch](https://forums.swift.org/t/pitch-dynamic-actor-isolation-enforcement/68354)) ([first review](https://forums.swift.org/t/se-0423-dynamic-actor-isolation-enforcement-from-non-strict-concurrency-contexts/70155)) ([second review](https://forums.swift.org/t/se-0423-second-review-dynamic-actor-isolation-enforcement-from-non-strict-concurrency-contexts/71159)) ([acceptance](https://forums.swift.org/t/accepted-se-0423-dynamic-actor-isolation-enforcement-from-non-strict-concurrency-contexts/71540)) + +## Introduction + +Many Swift programs need to interoperate with frameworks written in C/C++/Objective-C whose implementations cannot participate in static data race safety. Similarly, many Swift programs have dependencies that have not yet adopted strict concurrency checking. A `@preconcurrency import` statement downgrades concurrency-related error messages that the programmer cannot resolve because the fundamental issue is in one of the dependencies. To strengthen Swift's data-race safety guarantees while working with preconcurrency dependencies, this proposals adds actor isolation checking at runtime for synchronous isolated functions. + +## Motivation + +The ecosystem of Swift libraries has a vast surface area of APIs that predate strict concurrency checking, relying on carefully calling APIs from the appropriate thread or dispatch queue to avoid data races. Migrating all of these libraries to strict concurrency checking will happen incrementally, motivating [SE-0337: Incremental migration to concurrency checking](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md) which introduced the `@preconcurrency import` statement to suppress concurrency warnings from APIs that programmers do not control. + +If an actor isolation violation exists in the implementation of a preconcurrency library, the bug is only surfaced to clients as hard-to-debug data races on isolated state. `@preconcurrency` also does not apply to protocol conformances; there is no way to suppress concurrency diagnostics when conforming to a protocol from a preconcurrency library. This is unfortunate, because it's common for protocols to have a dynamic invariant that all requirements are called on the main thread or a specific dispatch queue provided by the client. + +For example, consider the following protocol in a library called `NotMyLibrary`, which provides a guarantee that its requirements are always called from the main thread: + +```swift +public protocol ViewDelegateProtocol { + func respondToUIEvent() +} +``` + +and a client of `NotMyLibrary` that contains a conformance to `ViewDelegateProtocol`: + +```swift +import NotMyLibrary + +@MainActor +class MyViewController: ViewDelegateProtocol { + func respondToUIEvent() { // error: @MainActor function cannot satisfy a nonisolated requirement + // implementation... + } +} +``` + +The above code is invalid because `MyViewController.respondToUIEvent()` is `@MainActor`-isolated, but it satisfies a `nonisolated` protocol requirement that can be called from generic code off the main actor. If the library provides a dynamic guarantee that the requirement is always called on the main actor, a sensible workaround is to resort to dynamic actor isolation checking by marking the function as `nonisolated` and wrapping the implementation in `MainActor.assumeIsolated`: + +```swift +import NotMyLibrary + +@MainActor +class MyViewController: ViewDelegateProtocol { + nonisolated func respondToUIEvent() { + MainActor.assumeIsolated { + // implementation... + } + } +} +``` + +With this workaround, the programmer must annotate every witness with `nonisolated` and wrap the implementation in `MainActor.assumeIsolated`. More importantly, the programmer loses static data-race safety in their own code, because internal callers of `respondToUIEvent()` are free to invoke it from any isolation domain without compiler errors. + +## Proposed solution + +This proposal adds dynamic actor isolation checking to: + + - Witnesses of synchronous `nonisolated` protocol requirements when the witness is isolated and the protocol conformance is annotated as `@preconcurrency`. For example: + + If `respondToUIEvent` is a witness to a synchronous `nonisolated` protocol requirement, the protocol conformance error can be suppressed using a `@preconcurrency` annotation on the protocol to indicate that the protocol itself predates concurrency: + + ```swift + import NotMyLibrary + + @MainActor + class MyViewController: @preconcurrency ViewDelegateProtocol { + func respondToUIEvent() { + // implementation... + } + } + ``` + + The witness checker diagnostic will be suppressed, the actor isolation assertion will fail if `respondToUIEvent()` is called inside `NonMyLibrary` from off the main actor, and the compiler will continue to emit diagnostics inside the module when called from off the main actor. + + These dynamic checks apply to any situation where a synchronous `nonisolated` requirement is implemented by an isolated method, including synchronous actor methods. + + - `@objc` thunks of synchronous actor-isolated members of classes. + + Similarly to the previous case if a class or its individual synchronous members are actor-isolated and marked as either `@objc` or `@objcMembers`, the thunks, synthesized by the compiler to make them available from Objective-C, would have a new precondition check to make sure that use always happens on the right actor. + + - Synchronous actor-isolated function values passed to APIs that erase actor isolation and haven't yet adopted strict concurrency checking. + + When API comes from a module that doesn't have strict concurrency checking enabled it's possible that it could introduce actor isolation violations that would not be surfaced to a client. In such cases actor isolation erasure should be handled defensively by introducing a runtime check at each position for granular protection. + + ```swift + @MainActor + func updateUI(view: MyViewController) { + NotMyLibrary.track(view.renderToUIEvent) + } + ``` + + The use of `track` here would be considered unsafe if it accepts a synchronous nonisolated function type due to loss of `@MainActor` from `renderToUIEvent` and compiler would transform the call site into a function equivalent of: + + ```swift + @MainActor + func updateUI(view: MyViewController) { + NotMyLibrary.track({ + MainActor.assumeIsolated { + view.renderToUIEvent() + } + }) + } + ``` + + - Call-sites of synchronous actor-isolated functions imported from Swift 6 libraries. + + When importing a module that was compiled with the Swift 6 language mode into code that is not, it's possible to call actor-isolated functions from outside the actor using `@preconcurrency`. For example: + + ```swift + // ModuleA built with -swift-version 6 + @MainActor public func onMain() { ... } + + // ModuleB built with -swift-version 5 -strict-concurrency=minimal + import ModuleA + + @preconcurrency @MainActor func callOnMain() { + onMain() + } + + func notIsolated() { + callOnMain() + } + ``` + + In the above code, `onMain` from ModuleA can be called from outside the main actor via a call to `notIsolated()`. To close this safety hole, a dynamic check is inserted at the call-site of `onMain()` when ModuleB is recompiled against ModuleA after ModuleA has migrated to the Swift 6 language mode. + +These are the most common circumstances when losing actor isolation could be problematic and restricting runtime checking to them significantly limits negative performance impact of the new checks. The strategy of only emitting runtime checks when there’s potential for the function to be called from unchecked code is desirable, because it means the dynamic checks will be eliminated as more of the Swift ecosystem transitions to Swift 6. + + +## Detailed design + +### Runtime actor isolation checking + +For all of the situations described in the previous section the compiler will emit a runtime check to assert that the current executor matches the expected executor of the isolated actor. Calling an isolated synchronous function from outside the isolation domain will result in a runtime error that halts program execution. + +Runtime checking for actor isolation is not necessary for `async` functions, because switching to the callee's actor is always performed by the callee. `async` functions cannot be unsafely called from non-Swift code because they are not available directly in C/C++/Objective-C. + +### `@preconcurrency` conformances + +A `@preconcurrency` protocol conformance is scoped to the implementation of the protocol requirements in the conforming type. A `@preconcurrency` conformance can be written at the primary declaration or in an extension, and witness checker diagnostics about actor isolation will be suppressed. Like other `@preconcurrency` annotations, if no diagnotsics are suppressed, a warning will be emitted at the `@preconcurrency` annotation stating that the annotation has no effect and it should be removed. + +### Disabling dynamic actor isolation checking + +The dynamic actor isolation checks can be disabled using the flag `-disable-dynamic-actor-isolation`. Disabling dynamic actor isolation is discouraged, but it may be necessary if code that you don't control violates actor isolation in a way that causes the program to crash, such as by passing a non-`Sendable` function argument outside of a main actor context. `-disable-dynamic-actor-isolation` is similar to the `-enforce-exclusivity=unchecked` flag, which was a tool provided when staging in dynamic memory exclusivity enforcement under the Swift 5 language mode. + +## Source compatibility + +Dynamic actor isolation checking can introduce new runtime assertions for existing programs. Therefore, dynamic actor isolation is only performed for synchronous functions that are witnesses to an explicitly annotated `@preconcurrency` protocol conformance, or that are compiled under the Swift 6 language mode. + +## ABI compatibility + +This proposal has no impact on ABI compatibility of existing code. There are runtime implications for code that explicitly adopts this feature; see the following section. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. However, as noted in the Source compatibility section, adoption of this feature has runtime implications, because actor-isolated code called incorrectly from preconcurrency code will crash instead of race. + +## Alternatives considered + +### Always emit dynamic checks upon entry to synchronous isolated functions + +A previous iteration of this proposal specified that dynamic actor isolation checks are always emitted upon entry to a synchronous isolated function. This approach is foolproof; there's little possibility for missing a dynamic check for code that can be called from another module that does not have strict concurrency checking at compile time. However, the major downside of this approach is that code will be paying the price of runtime overhead for actor isolation checking even when actor isolation is fully enforced at compile time in Swift 6. + +The current approach in this proposal has a very desirable property of eliminated more runtime overhead as more of the Swift ecosystem transitions to Swift 6 at the cost of introducing the potential for missing dynamic checks where synchronous functions can be called from not-statically-checked code. We believe this is the right tradeoff for the long term arc of data race safety in Swift 6 and beyond, but it may require more special cases when we discover code patterns that are not covered by the specific set of rules in this proposal. + +### `@preconcurrency(unsafe)` to downgrade dynamic actor isolation violations to warnings + +If adoption of this feature exposes a bug in existing binaries because actor-isolated code is run outside the actor, a `@preconcurrency(unsafe)` annotation (or similar) could be provided to downgrade assertion failures to warnings. However, it's not clear whether allowing a known data race exhibited at runtime is the right approach to solving such a problem. + +## Revision history + +* Changes from the first review + * Insert dynamic checks at direct calls to synchronous actor-isolated functions imported from Swift 6 libraries. + * Add a flag to disable all dynamic actor isolation checking. + +## Acknowledgments + +Thank you to Doug Gregor for implementing the existing dynamic actor isolation checking gated behind `-enable-actor-data-race-checks`. diff --git a/proposals/0424-custom-isolation-checking-for-serialexecutor.md b/proposals/0424-custom-isolation-checking-for-serialexecutor.md new file mode 100644 index 0000000000..cc7a2d0d29 --- /dev/null +++ b/proposals/0424-custom-isolation-checking-for-serialexecutor.md @@ -0,0 +1,206 @@ +# Custom isolation checking for SerialExecutor + +* Proposal: [SE-0424](0424-custom-isolation-checking-for-serialexecutor.md) +* Author: [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-custom-isolation-checking-for-serialexecutor/69786)) ([review](https://forums.swift.org/t/se-0424-custom-isolation-checking-for-serialexecutor/70195)) ([acceptance](https://forums.swift.org/t/accepted-se-0424-custom-isolation-checking-for-serialexecutor/70480)) + +## Introduction + +[SE-0392 (Custom Actor Executors)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md) added support for custom actor executors, but its support is incomplete. Safety checks like [`Actor.assumeIsolated`](https://developer.apple.com/documentation/swift/actor/assumeisolated(_:file:line:)) work correctly when code is running on the actor through a task, but they don't work when code is scheduled to run on the actor's executor through some other mechanism. For example, if an actor uses a serial `DispatchQueue` as its executor, a function dispatched _directly_ to the queue with DispatchQueue.async cannot use `assumeIsolated` to assert that the actor is currently isolated. This proposal fixes this by allowing custom actor executors to provide their own logic for these safety checks. + +## Motivation + +The Swift concurrency runtime dynamically tracks the current executor of a running task in thread-local storage. To run code on behalf of a task, an executor must call into the runtime, and the runtime will set up the tracking appropriately. APIs like `assertIsolated` and `assumeIsolated` are built on top of that functionality and perform their checks by comparing the expected executor with the current executor tracked by the runtime. If the current thread is not running a task, the runtime treats it as if it were running a non-isolated function, and the comparison will fail. + +This logic is not sufficient to handle the situation in which code is running on an actor's serial executor, but the code is not associated with a task. Swift's default actor executors currently do not provide any way to enqueue work on them that is not associated with a task, so this situation does not apply to them. However, many custom executors do provide other APIs for enqueuing work, such as the `async` method on `DispatchSerialQueue`. These APIs are not required to inform the Swift concurrency runtime before running the code. As a result, the runtime will be unaware that the current thread is associated with an actor's executor, and checks like `assumeIsolated` will fail. This is undesirable because, as long as the executor still acts like a serial executor for any non-task code it runs this way, the code will still be effectively actor-isolated: no code that accesses the actor's isolated state can run concurrently with it. + +The following example demonstrates such a situation: + +```swift +import Dispatch + +actor Caplin { + let queue: DispatchSerialQueue = .init(label: "CoolQueue") + + var num: Int // actor isolated state + + // use the queue as this actor's `SerialExecutor` + nonisolated var unownedExecutor: UnownedSerialExecutor { + queue.asUnownedSerialExecutor() + } + + nonisolated func connect() { + queue.async { + // guaranteed to execute on `queue` + // which is the same as self's serial executor + self.queue.assertIsolated() // CRASH: Incorrect actor executor assumption + self.assumeIsolated { caplin in // CRASH: Incorrect actor executor assumption + caplin.num += 1 + } + } + } +} +``` + +Even though the code is executing on the correct Dispatch**Serial**Queue, the assertions trigger and we're left unable to access the actor's state, even though isolation-wise it would be safe and correct to do so. + +Being able to assert isolation for non-task code this way is important enough that the Swift runtime actually already has a special case for it: even if the current thread is not running a task, isolation checking will succeed if the target actor is the `MainActor` and the current thread is the *main thread*. This problem is more general than the main actor, however; it exists for all kinds of threads which may be used as actor executors. The most important example of this is `DispatchSerialQueue`, especially because it is so commonly used in pre-concurrency code bases to provide actor-like isolation. Allowing types like `DispatchSerialQueue` to hook into isolation checking makes it much easier to gradually migrate code to actors: if an actor uses a queue as its executor, existing code that uses the queue don't have to be completely rewritten in order to access the actor's state. + +One way to think of this proposal is that gives all `SerialExecutor`s the power to provide a "fallback" check like this, rather than keeping it special-cased to `MainActor`. + +## Proposed solution + +We propose to add a new last-resort mechanism to executor comparison, which will be used by all the isolation-checking APIs in the concurrency library. + +This will be done by providing a new `checkIsolated()` protocol requirement on `SerialExecutor`: + +```swift +protocol SerialExecutor: Executor { + // ... + + /// Invoked as last resort when the Swift concurrency runtime is performing an isolation + /// assertion and could not confirm that the current execution context belongs to the + /// expected executor. + /// + /// This function MUST crash the program with a fatal error if it is unable + /// to prove that this thread can currently be safely treated as isolated + /// to this ``SerialExecutor``. That is, if a synchronous function calls + /// this method, and the method does not crash with a fatal error, + /// then the execution of the entire function must be well-ordered + /// with any other job enqueued on this executor, as if it were part of + /// a job itself. + /// + /// A default implementation is provided that unconditionally causes a fatal error. + func checkIsolated() +} + +extension SerialExecutor { + public func checkIsolated() { + fatalError("Incorrect actor executor assumption, expected: \(self)") + } +} +``` + +## Detailed design + +This proposal adds another customization point to the Swift concurrency runtime that hooks into isolation context comparison mechanisms used by `assertIsolated`, `preconditionIsolated`, and `assumeIsolated`, as well as any implicitly injected assertions used in `@preconcurrency` code. + +### Extended executor comparison mechanism + +With this proposal, the logic for checking if the current executor is the same as an expected executor changes, and can be expressed using the following pseudo-code: + +```swift +// !!!! PSEUDO-CODE !!!! Simplified for readability. + +let current = Task.current.executor + +guard let current else { + // no current executor, last effort check performed by the expected executor: + expected.checkIsolated() + + // e.g. MainActor: + // MainActorExecutor.checkIsolated() { + // guard Thread.isMain else { fatalError("Expected main thread!") + // return // ok! + // } +} + +if isSameSerialExecutor(current, expected) { + // comparison takes into account "complex equality" as introduced by 'SE-0392 + return // ok! +} else { + // executor comparisons failed... + + // give the expected executor a last chance to check isolation by itself: + expected.checkIsolated() + + // as the default implementation of checkIsolated is to unconditionally crash, + // this call usually will result in crashing -- as expected. +} + +return // ok, it seems the expected executor was able to prove isolation +``` + +This pseudo code snippet explains the flow of the executor comparisons. There are two situations in which the new `checkIsolated` method may be invoked: when there is no current executor present, or if all other comparisons have failed. +For more details on the executor comparison logic, you can refer to [SE-0392: Custom Actor Executors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md). + +Specific use-cases of this API include `DispatchSerialQueue`, which would be able to implement the requirement as follows: + +```swift +// Dispatch + +extension DispatchSerialQueue { + public func checkIsolated() { + dispatchPrecondition(condition: .onQueue(self)) // existing Dispatch API + } +} +``` + +An executor that wishes to take advantage of this proposal will need to have some mechanism to identity its active worker thread. If that's not possible or desired, the executor should leave the default implementation (that unconditionally crashes) in place. + +### Impact on async code and isolation assumptions + +The `assumeIsolated(_:file:line:)` APIs purposefully only accept a **synchronous** closure. This is correct, and it remains correct with these proposed additions. An isolation check on an executor ensures that any actor using the executor is synchronously isolated, and the closure provided to `assumeIsolated` will execute prior to any possible async suspension. This is what makes it safe to access actor-isolated state within the closure. + +This means that the following code snippet, while a bit unusual remains correct isolation-wise: + +```swift +actor Worker { + var number: Int + + nonisolated func canOnlyCallMeWhileIsolatedOnThisInstance() -> Int { + self.preconditionIsolated("This method must be called while isolated to \(self)") + + return self.assumeIsolated { // () throws -> Int + // suspensions are not allowed in this closure. + + self.number // we are guaranteed to be isolated on this actor; read is safe + } + } + +``` + +As such, there is no negative impact on the correctness of these APIs. + +Asynchronous functions should not use dynamic isolation checking. Isolation checking is useful in synchronous functions because they naturally inherit execution properties like their caller's isolation without disturbing it. A synchronous function may be formally non-isolated and yet actually run in an isolated context dynamically. This is not true for asynchronous functions, which switch to their formal isolation on entry without regard to their caller's isolation. If an asynchronous function is not formally isolated to an actor, its execution will never be dynamically in an isolated context, so there's no point in checking for it. + +## Future directions + +### Introduce `globalMainExecutor` global property and utilize `checkIsolated` on it + +This proposal also paves the way to clean up this hard-coded aspect of the runtime, and it would be possible to change these heurystics to instead invoke the `checkIsolated()` method on a "main actor executor" SerialExecutor reference if it were available. + +This proposal does not introduce a `globalMainActorExecutor`, however, similar how how [SE-0417: Task ExecutorPreference](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0417-task-executor-preference.md) introduced a: + +```swift +nonisolated(unsafe) +public var globalConcurrentExecutor: any TaskExecutor { get } +``` + +the same could be done to the MainActor's executor: + +```swift +nonisolated(unsafe) +public var globalMainExecutor: any SerialExecutor { get } +``` + +The custom heurystics that are today part of the Swift Concurrency runtime to detect the "main thread" and "main actor executor", could instead be delegated to this global property, and function correctly even if the MainActor's executor is NOT using the main thread (which can happen on some platforms): + +```swift +// concurrency runtime pseudo-code +if expectedExecutor.isMainActor() { + expectedExecutor.checkIsolated() +} +``` + +This would allow the isolation model to support different kinds of main executor and properly assert their isolation, using custom logic, rather than hardcoding the main thread assumptions into the Swift runtime. + +## Alternatives considered + +### Do not provide customization points, and just hardcode DispatchQueue handling + +Alternatively, we could hardcode detecting dispatch queues and triggering `dispatchPrecondition` from within the Swift runtime. + +This is not a good direction though, as our goal is to have the concurrency runtime be less attached to Dispatch and allow Swift to handle each and every execution environment equally well. As such, introducing necessary hooks as official and public API is the way to go here. diff --git a/proposals/0425-int128.md b/proposals/0425-int128.md new file mode 100644 index 0000000000..809464b3f8 --- /dev/null +++ b/proposals/0425-int128.md @@ -0,0 +1,193 @@ +# 128-bit Integer Types + +* Proposal: [SE-0425](0425-int128.md) +* Author: [Stephen Canon](https://github.com/stephentyrone) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.0)** +* Review: ([Pitch](https://forums.swift.org/t/pitch-128-bit-integer-types/70188)) ([Review](https://forums.swift.org/t/se-0425-128-bit-integer-types/70456)), ([Acceptance](https://forums.swift.org/t/accepted/71063)) + +## Motivation + +128b integers are the largest fixed-size type that is currently commonly +used in "general-purpose" code. They are much less common than 64b types, +but common enough that adding them to the standard library makes sense. +We use them internally in the standard library already (e.g. as an +implementation detail of Duration). + +## Proposed solution + +Introduce two new structs, `UInt128` and `Int128`, conforming to all of the +usual fixed-width integer protocols. + +## Detailed design + +The `[U]Int128` types are 16B aligned on 64b targets¹ and have the same +alignment as `[U]Int64` on 32b targets. They will match the endianness of +all other integer types. + +The clang importer will be updated to bridge `__uint128_t` to `UInt128` and +`__int128_t` to `Int128`. We will not bridge `_BitInt()` types until +the ABI problems with those types have been clearly resolved (see Alternatives +Considered for sordid history). + +The `[U]Int128` types conform to `AtomicRepresentable` on targets with +`_hasAtomicBitWidth(_128)` set (notably x86\_64, arm64, and arm64\_32). + +The actual API of the types is uninteresting; they are entirely constrained by +their protocol conformances. Notably, these types conform to the following +protocols, and hence to any protocol that they refine: + +- Hashable +- Equatable +- Comparable +- Codable +- Sendable +- LosslessStringConvertible +- ExpressibleByIntegerLiteral +- AdditiveArithmetic +- [Signed]Numeric +- BinaryInteger +- FixedWidthInteger +- [Unsigned|Signed]Integer + +------- +¹ For the purposes of this discussion, arm64\_32 and similar architectures +are "64b targets." + +### Codable details + +An earlier version of this proposal conformed `[U]Int128` to `Codable` +with a representation as a pair of `[U]Int64` values. During the first review +period several people made excellent points: + +- Making it possible for encoders to customize how they represent these types +is desirable. Some cannot represent all 64b values or might prefer to use a +string representation, others might prefer to treat 128b integers as native +values to encode. + +- If we make it customizable but provide a default behavior, some decoders +would have to support that default as well as their desired representation, +for compatibility with any encodings created between when we added support +and when they defined their preferred encoding. + +For this reason, the proposal has been updated to add new protocol requirements +for encoders and decoders to support `[U]Int128`, but with default +implementations that throw an EncodingError or DecodingError unconditionally, +allowing implementations to choose their preferred behavior when they add +support without worrying about compatibility with a defaulted implementation. + +Thus, the following requirements will be added: +```swift +protocol KeyedEncodingContainerProtocol { + mutating func encode(_ value: Int128, forKey key: Key) throws + mutating func encode(_ value: UInt128, forKey key: Key) throws + mutating func encodeIfPresent(_ value: Int128?, forKey key: Key) throws + mutating func encodeIfPresent(_ value: UInt128?, forKey key: Key) throws + // And matching changes to KeyedEncodingContainer +} + +protocol KeyedDecodingContainerProtocol { + func decode(_ type: Int128.Type, forKey key: Key) throws -> Int128 + func decode(_ type: UInt128.Type, forKey key: Key) throws -> UInt128 + func decodeIfPresent(_ type: Int128.Type, forKey key: Key) throws -> Int128? + func decodeIfPresent(_ type: UInt128.Type, forKey key: Key) throws -> UInt128? + // And matching changes to KeyedDecodingContainer +} + +protocol UnkeyedEncodingContainer { + mutating func encode(_ value: Int128) throws + mutating func encode(_ value: UInt128) throws + mutating func encode( + contentsOf sequence: T + ) throws where T.Element == Int128 + mutating func encode( + contentsOf sequence: T + ) throws where T.Element == UInt128 +} + +protocol UnkeyedDecodingContainer { + mutating func decode(_ type: Int128.Type) throws -> Int128 + mutating func decode(_ type: UInt128.Type) throws -> UInt128 + mutating func decodeIfPresent(_ type: Int128.Type) throws -> Int128? + mutating func decodeIfPresent(_ type: UInt128.Type) throws -> UInt128? +} + +protocol SingleValueEncodingContainer { + mutating func encode(_ value: Int128) throws + mutating func encode(_ value: UInt128) throws +} + +protocol SingleValueDecodingContainer { + func decode(_ type: Int128.Type) throws -> Int128 + func decode(_ type: UInt128.Type) throws -> UInt128 +} +``` +and given default implementations. The default encode implementations throw +`EncodingError.invalidValue`, and the default decode implementations throw +`DecodingError.typeMismatch`. + +## Source compatibility + +This proposal has no effect on source compatibility. + +## ABI compatibility + +This proposal has no effect on ABI compatibility. + +## Implications on adoption + +Adopting this feature will require a target with runtime support. + +## Future directions + +Implement clang importer support for `_BitInt(128)` on any platforms where +the finalized ABI is compatible with our layout. + +## Alternatives considered + +### Alignment and `_BitInt()` types +Clang and GCC have historically exposed the extension types `__uint128_t` and +`__int128_t` on 64b platforms only. These types basically behave like C +builtin integer types--their size and alignment are 16B. + +The C23 standard introduces `_BitInt(N)` as a means to spell arbitrary-width +integer types, but these still have some warts. In particular, `_BitInt(128)` +as implemented in clang has 8B alignment on x86\_64 and arm64. For arm64, +this is clearly a bug; the AAPCS specifies that it should have 16B alignment. +For x86\_64, the situation is less clear. The x86\_64 psABI document specifies +that it should have 8B alignment, but the authors of the proposal that added +the feature tell me that it _should_ be 16B aligned and that they are +attempting to change the psABI. + +We would like to be layout-compatible with `_BitInt(128)` on all platforms, +but given the currently-murky state of the layout of those types, it makes +the most sense to guarantee compatibility with the widely-used but non- +standard `__[u]int128_t` and find mechanisms to make `_BitInt(128)` work +once its ABI has been finalized on Swift's targeted platforms. + +### Generic-sized fixed width integers +Rather than adding `[U]Int128`, we could implement some form of generic- +sized fixed-width integer (like `\_BitInt()` in C). Given both the lack +of consensus around what integer generic parameters ought to look like in +Swift (or if they ought to exist at all), and the growing pains that +`\_BitInt()` is currently going through, such a design would be premature. +While other fixed-width integer types are interesting, 128 bits is a couple +orders of magnitude more useful than all the others for general-purpose +software at this point in time. + +### NSNumber bridging +`[U]Int128` will not bridge to `NSNumber`. In the future, Swift will need +a careful rethinking of how best to handle type-erased numbers, but we don't +want to pile on the debt by including ever more types in an existing system +that isn't supported on all platforms. In addition, the most common use for +such bridging, unpacking type-erased fields from encoded dictionaries, is +somewhat moot since most existing coders do not support 128b integers. We +will likely revisit this more holistically in the future. + +### Codable errors +It would be nice to introduce new `unsupportedType` error cases for the +default Codable conformances, but we cannot add new cases with associated +values and constrained availability to existing enums, which prevents +attaching context or a useful debug description. It's more useful for users +if we use existing error cases `invalidValue` and `typeMismatch` but put an +actionable message in that description field. diff --git a/proposals/0426-bitwise-copyable.md b/proposals/0426-bitwise-copyable.md new file mode 100644 index 0000000000..f057c4dd5e --- /dev/null +++ b/proposals/0426-bitwise-copyable.md @@ -0,0 +1,493 @@ +# BitwiseCopyable + +* Proposal: [SE-0426](0426-bitwise-copyable.md) +* Authors: [Kavon Farvardin](https://github.com/kavon), [Guillaume Lessard](https://github.com/glessard), [Nate Chandler](https://github.com/nate-chandler), [Tim Kientzle](https://github.com/tbkka) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Implementation: in main branch of compiler (https://github.com/apple/swift/pull/73235) +* Status: **Implemented (Swift 6.0)** +* Review: ([Pitch](https://forums.swift.org/t/pitch-bitwisecopyable-marker-protocol/69943)) ([First review](https://forums.swift.org/t/se-0426-bitwisecopyable/70479)) ([Returned for revision](https://forums.swift.org/t/returned-for-revision-se-0426-bitwisecopyable/70892)) ([Second review](https://forums.swift.org/t/se-0426-second-review-bitwisecopyable/71316)) ([Acceptance](https://forums.swift.org/t/accepted-se-0426-bitwisecopyable/71600)) + + + +## Introduction + +We propose a new, [limited](#limitations) protocol `BitwiseCopyable` that _can_ be conformed to by types that are "bitwise-copyable"[^1]--that is, that can be moved or copied with direct calls to `memcpy` and which require no special destroy operation. +When compiling generic code with such constraints, the compiler can emit these efficient operations directly, only requiring minimal overhead to look up the size of the value at runtime. +Alternatively, developers can use this constraint to selectively provide high-performance variations of specific operations, such as bulk copying of a container. + +[^1]: The term "trivial" is used in [SE-138](0138-unsaferawbufferpointer.md) and [SE-0370](0370-pointer-family-initialization-improvements.md) to refer to types with this property. The discussion below will explain why certain generic or exported types that are bitwise-copyable will not in fact be `BitwiseCopyable`. + +## Motivation + +Swift can compile generic code into an unspecialized form in which the compiled function receives a value and type information about that value. +Basic operations are implemented by the compiler as calls to a table of "value witness functions." + +This approach is flexible, but can represent significant overhead. +For example, using this approach to copy a buffer with a large number of `Int` values requires a function call for each value. + +Constraining the types in generic functions to `BitwiseCopyable` allows the compiler (and in some cases, the developer) to instead use highly efficient direct memory operations in such cases. + +The standard library already contains many examples of functions that could benefit from such a concept, and more are being proposed: + +The `UnsafeMutablePointer.initialize(to:count:)` function introduced in [SE-0370](0370-pointer-family-initialization-improvements.md) could use a bulk memory copy whenever it statically knew that its argument was `BitwiseCopyable`. + +The proposal for [`StorageView`](nnnn-safe-shared-contiguous-storage.md) includes the ability to copy items to or from potentially-unaligned storage, which requires that it be safe to use bulk memory operations: +```swift +public func loadUnaligned( + fromByteOffset: Int = 0, as: T.Type +) -> T + +public func loadUnaligned( + from index: Index, as: T.Type +) -> T +``` + +And this proposal includes the addition of three overloads of existing standard library functions. + +## Proposed solution + +We add a new protocol `BitwiseCopyable` to the standard library: +```swift +@_marker public protocol BitwiseCopyable {} +``` + +That a type conforms to the protocol [implies](#transient-and-permanent) that the type is bitwise-copyable; the reverse is _not_ true. + +Many basic types in the standard library will conformed to this protocol. + +Developer's own types may be conformed to the protocol, as well. +The compiler will check any such conformance and emit a diagnostic if the type contains elements that are not `BitwiseCopyable`. + +Furthermore, when building a module, the compiler will infer conformance to `BitwiseCopyable` for any non-exported struct or enum defined within the module whose stored members are all `BitwiseCopyable`, +except those for which conformance is explicitly [suppressed](#suppression). + +Developers cannot conform types defined in other modules to the protocol. + +## Detailed design + +Our design first conforms a number of core types to `BitwiseCopyable`, and then extends that to aggregate types. + +### Standard library changes + +Many types and a few key protocols are constrained to `BitwiseCopyable`. +A few highlights: + +* Integer types +* Floating point types +* SIMD types +* Pointer types +* `Unmanaged` +* `Optional` + +For an exhaustive list, see the [appendix](#all-stdlib-conformers). + +### Additional BitwiseCopyable types + +In addition to the standard library types marked above, the compiler will recognize several other types as `BitwiseCopyable`: + +* Tuples of `BitwiseCopyable` elements. + +* `unowned(unsafe)` references. + Such references can be copied without reference counting operations. + +* `@convention(c)` and `@convention(thin)` function types do not carry a reference-counted capture context, unlike other Swift function types, and are therefore `BitwiseCopyable`. + +### Explicit conformance to `BitwiseCopyable` + +Enum and struct types can be explicitly declared to conform to `BitwiseCopyable`. +When a type is declared to conform, the compiler will check that its elements are all `BitwiseCopyable` and emit an error otherwise. + +For example, the following struct can conform to `BitwiseCopayble` +```swift +public struct Coordinate : BitwiseCopyable { + var x: Int + var y: Int +} +``` +because `Int` is `BitwiseCopyable`. + +Similarly, the following enum can conform to `BitwiseCopyable` +```swift +public enum PositionUpdate : BitwiseCopyable { + case begin(Coordinate) + case move(x_change: Int, y_change: Int) + case end +} +``` +because both `Coordinate` and `(x_change: Int, y_change: Int)` are `BitwiseCopyable`. + +The same applies to generic types. For example, the following struct can conform to `BitwiseCopyable` +```swift +struct BittyBox : BitwiseCopyable { + var first: Value +} +``` +because its field `first` is a of type `Value` which is `BitwiseCopyable`. + +Generic types may be `BitwiseCopyable` only some of the time. +For example, +```swift +struct RegularBox { + var first: Value +} +``` +cannot conform unconditionally because `Value` needn't conform to `BitwiseCopyable`. +In this case, a conditional conformance may be written: + +```swift +extension Box : BitwiseCopyable where Value : BitwiseCopyable {} +``` + +### Automatic inference for aggregates + +As a convenience, unconditional conformances will be inferred for structs and enums[^2] much of the time. +When the module containing the type is built, if all of the type's fields are `BitwiseCopyable`, the compiler will generate a conformance for it to `BitwiseCopyable`. + +For generic types, a conformance will only be inferred if its fields unconditionally conform to `BitwiseCopyable`. +In the `RegularBox` example above, a conditional conformance will not be inferred. +If such a conformance is desired, the developer must explicitly write the conditional conformance. + +[^2]: This includes raw-value enums. While such enums do include a conformance to `RawRepresentable` where `RawValue` could be a non-conforming type (`String`), the instances of the enums themselves are `BitwiseCopyable`. + +### Inference for imported types + +The same inference will be done on imported C and C++ types. + +For an imported C or C++ enum, the compiler will always generate a conformance to to `BitwiseCopyable`. + +For an imported C struct, if all its fields are `BitwiseCopyable`, the compiler will generate a conformance to `BitwiseCopyable`. +The same is true for an imported C++ struct or class, unless the type is non-trivial[^3]. + +For an imported C or C++ struct, if any of its fields cannot be represented in Swift, the compiler will not generate a conformance. +This can be overridden, however, by annotating the type `__attribute__((__swift_attr__("BitwiseCopyable")))`. + +[^3]: A C++ type is considered non-trivial (for the purpose of calls, as defined by the Itanium ABI) if any of the following is non-default: its constructor; its copy-constructor; its destructor. + +### Inference for exported types + +This does not apply to exported (`public`, `package`, or `@usableFromInline`) types. +In the case of a library built with library evolution, while all the type's fields may be `BitwiseCopyable` at the moment, the compiler can't predict that they will always be. +If this is the developer's intent, they can explicitly conform the type. +To avoid having semantics that vary based on library evolution, the same applies to all exported (`public`, `package`, or `@usableFromInline`) types. + +For `@frozen` types, however, `BitwiseCopyable` conformance will be inferred. +That's allowed, even in the case of a library built with library evolution, because the compiler can see that the type's fields are all `BitwiseCopyable` and knows that they will remain that way. + +For example, the compiler will infer a conformance of the following struct +```swift +@frozen +public struct Coordinate3 { + public var x: Int + public var y: Int +} +``` +to `BitwiseCopyable`. + +### Suppressing inferred conformance + +To suppress the inference of `BitwiseCopyable`, `~BitwiseCopyable` can be added to the type's inheritance list. + +```swift +struct Coordinate4 : ~BitwiseCopyable {...} +``` + +Suppression must be declared on the type declaration itself, not on an extension. + +### Transient and permanent notions + +The Swift runtime already describes[^4] whether a type is bitwise-copyable. +It is surfaced, among other places, in the standard library function `_isPOD`[^5]. + +[^4]: The `IsNonPOD` value witness flag is set for every type that is _not_ bitwise-copyable. + +[^5]: "POD" here is an acronym for "plain old data" which is yet another name for the notion of bitwise-copyable or trivial. + +If a type conforms to `BitwiseCopyable`, then `_isPOD` must be true for the type. +The converse is not true, however. + +As a type evolves, it may [both gain _and_ lose bitwise-copyability](#fluctuating-bitwise-copyability). +A type may only _gain_ a conformance to `BitwiseCopyable`, however; +it cannot _lose_ its conformance without breaking source and ABI. + +The two notions are related, but distinct: +That a type `_isPOD` is a statement that the type is currently bitwise-copyable. +That a type conforms to `BitwiseCopyable` is a promise that the type is now and will remain bitwise-copyable as the library evolves. +In other words returning true from `_isPOD` is a transient property, and conformance to `BitwiseCopyable` is a permanent one. + +For this reason, conformance to `BitwiseCopyable` is not inherent. +Its declaration on a public type provides a guarantee that the compiler cannot infer. + +### Limitations of BitwiseCopyable + +Being declared with `@_marker`, `BitwiseCopyable` is a limited protocol. +Its limited nature allows the protocol's runtime behavior to be defined later, as needed. + +1. `BitwiseCopyable` cannot be extended. +This limitation is similar to that on `Sendable` and `Any`: +it prevents polluting the namespace of conforming types, especially types whose conformance is inferred. + +2. Because conformance to `BitwiseCopyable` is distinct from being bitwise-copyable, +the runtime cannot use the `IsNonPOD` bit as a proxy for conformance (although actual [conformance could be ignored](#casting-by-duck-typing)). +A separate mechanism would be necessary. +Until such a mechanism is added, `is`, `as?` and usage as a generic constraint to enable conditional conformance to another protocol is not possible. + +### Standard library API improvements + +The standard library includes a load method on both `UnsafeRawPointer` and `UnsafeMutableRawPointer` + +```swift +@inlinable +@_alwaysEmitIntoClient +public func loadUnaligned( + fromByteOffset offset: Int = 0, + as type: T.Type +) -> T +``` + +and a corresponding write method on `UnsafeMutableRawPointer` + +```swift +@inlinable +@_alwaysEmitIntoClient +public func storeBytes( + of value: T, toByteOffset offset: Int = 0, as type: T.Type +) +``` + +that must be called with a trivial `T`. + +We propose adding overloads of these methods to constrain the value to `BitwiseCopyable`: + +```swift +// on both UnsafeRawPointer and UnsafeMutableRawPointer +@inlinable +@_alwaysEmitIntoClient +public func loadUnaligned( + fromByteOffset offset: Int = 0, + as type: T.Type +) -> T + +// on UnsafeMutableRawPointer +@inlinable +@_alwaysEmitIntoClient +public func storeBytes( + of value: T, toByteOffset offset: Int = 0, as type: T.Type +) +``` + +This allows for optimal code generation because `memcpy` instead of value witnesses can be used. + +The existing methods that use a runtime assert instead of a type constraint will still be available (see [alternatives considered](#deprecation)). + +## Effect on ABI stability + +The addition of the `BitwiseCopyable` constraint to either a type or a protocol in a library will not cause an ABI break for users. + +## Source compatibility + +This addition of a new protocol will not impact existing source code that does not use it. + +Removing the `BitwiseCopyable` conformance from a type is source-breaking. +As a result, future versions of Swift may conform additional existing types to `BitwiseCopyable`, but will not remove it from any type already conforming to `BitwiseCopyable`. + +## Effect on API resilience + +Adding a `BitwiseCopyable` constraint on a generic type will not cause an ABI break. +As with any protocol, the additional constraint can cause a source break for users. + +## Future Directions + +### Automatic derivation of conditional conformances + +The wrapper type mentioned above +```swift +struct RegularBox { + var first: Value +} +``` +cannot conform to `BitwiseCopyable` unconditionally. +It can, however, so long as `Value` is `BitwiseCopyable`. + +With this proposal, such a conditional conformance can be added manually: + +```swift +extension Box : BitwiseCopyable where Value : BitwiseCopyable {} +``` + +In the future we may in some cases be able to derive it automatically. + +### Dynamic casting + +Being a [limited](#limitations) protocol, `BitwiseCopyable` does not currently have any runtime representation. +While a type's [transient](#transient-and-permanent) bitwise-copyability has a preexisting runtime representation, that is different from the type conforming to `BitwiseCopyable`. + +Being a low-level, performance-enabling feature, it is not clear that dynamic casting should be allowed at all. +If it were to be allowed at some point, a few different approaches can already be foreseen: + +#### Explicitly record a type's conformance + +The standard way to support dynamic casting would be to represent a type's conformance to the protocol and query the type at runtime. + +This approach has the virtue that dynamic casting behaves as usual. +A type could only be cast to `BitwiseCopyable` if it actually conformed to the protocol. +For example, casting a type which suppressed a conformance to `BitwiseCopyable` would fail. + +If this approach were taken, such casting could be back-deployed as far as the oldest OS in which this runtime representation was added. +Further back deployment would be possible by adding conformance records to back deployed binaries. + +#### Duck typing for BitwiseCopyable + +An alternative would be to dynamically treat any type that's bitwise-copyable as if it conformed to `BitwiseCopyable`. + +This is quite different from typical Swift casting behavior. +Rather than relying on a permanent characteristic of the type, it would rely on a [transient](#transient-and-permanent) one. +This would be visible to the programmer in several ways: +- different overloads would be selected for a value of concrete type from those selected for a value dynamically cast to `BitwiseCopyable` +- dynamic casts to `BitwiseCopyable` could fail, then succeed, then fail again in successive OS versions + +On the other hand, these behavioral differences may be desirable. + +Considering that this approach would just ignore the existence of conformances to `BitwiseCopyable`, +it would be reasonable to ignore the existence of a suppressed conformance as well. + +This approach also has the virtue of being completely back-deployable[^6]. +[^6]: All runtimes have had the `IsNonPOD` bit. + +### BitwiseMovable + +Most Swift types have the property that their representation can be relocated in memory with direct memory operations. +This could be represented with a `BitwiseMovable` protocol that would be handled similarly to `BitwiseCopyable`. + +### BitwiseCopyable as a composition + +Some discussion in the pitch thread discussed how `BitwiseCopyable` could be defined as the composition of several protocols. +For example, +```swift +typealias BitwiseCopyable = Bitwise & Copyable & DefaultDeinit +``` +Such a definition remains possible after this proposal. + +Because `BitwiseCopyable` is annotated `@_marker`, its ABI is rather limited. +Specifically, it only affects name mangling. +If, in a subsequent proposal, the protocol were redefined as a composition, symbols into which `BitwiseCopyable` was mangled could still be mangled in the same way, ensuring ABI compatibility. + +## Alternatives considered + +### Alternate Spellings + +**Trivial** is widely used within the compiler and Swift evolution discussions to refer to the property of bitwise copyability. `BitwiseCopyable`, on the other hand, is more self-documenting. + +### Deprecation of unconstrained functions dependent on `isPOD` + +The standard library has a few pre-existing functions that receive a generic bitwise-copyable value as a parameter. These functions work with types for which the `_isPOD()` function returns true, even though they do not have a `BitwiseCopyable` conformance. If we were to deprecate these unconstrained versions, we would add unresolvable warnings to some of the codebases that use them. For example, they might use types that could be conditionally `BitwiseCopyable`, but come from a module whose types have not been conformed to `BitwiseCopyable` by their author. Furthermore, as explained [above](#transient-and-permanent), it is not necessarily the case that a transiently bitwise-copyable type can be permanently annotated as `BitwiseCopyable`. + +At present, the unconstrained versions check that `_isPOD()` returns true in debug mode only. We may in the future consider changing them to check at all times, since in general their use in critical sections will have been updated to use the `BitwiseCopyable`-constrained overloads. + +## Acknowledgments + +This proposal has benefitted from discussions with John McCall, Joe Groff, Andrew Trick, Michael Gottesman, and Arnold Schwaigofer. + +## Appendix: Standard library conformers + +The following protocols in the standard library will gain the `BitwiseCopyable` constraint: + +- `_Pointer` +- `SIMDStorage`, `SIMDScalar`, `SIMD` + + +The following types in the standard library will gain the `BitwiseCopyable` constraint: + +- `Optional` when `T` is `BitwiseCopyable` +- The fixed-precision integer types: + - `Bool` + - `Int8`, `Int16`, `Int32`, `Int64`, `Int` + - `UInt8`, `UInt16`, `UInt32`, `UInt64`, `UInt` + - `StaticBigInt` + - `UInt8.Words`, `UInt16.Words`, `UInt32.Words`, `UInt64.Words`, `UInt.Words` + - `Int8.Words`, `Int16.Words`, `Int32.Words`, `Int64.Words`, `Int.Words` +- The fixed-precision floating-point types: + - `Float`, `Double`, `Float16`, `Float80` + - `FloatingPointSign`, `FloatingPointClassification` +- The family of `SIMDx` types +- The family of unmanaged pointer types: + - `OpaquePointer` + - `UnsafeRawPointer`, `UnsafeMutableRawPointer` + - `UnsafePointer`, `UnsafeMutablePointer`, `AutoreleasingUnsafeMutablePointer` + - `UnsafeBufferPointer`, `UnsafeMutableBufferPointer` + - `UnsafeRawBufferPointer`, `UnsafeMutableRawBufferPointer` + - `Unmanaged` + - `CVaListPointer` +- Some types related to collections + - `EmptyCollection` + - `UnsafeBufferPointer.Iterator`, `UnsafeRawBufferPointer.Iterator`, `EmptyCollection.Iterator` + - `String.Index`, `CollectionDifference.Index` +- Some types related to unicode + - `Unicode.ASCII`, `Unicode.UTF8`, `Unicode.UTF16`, `Unicode.UTF32`, `Unicode.Scalar` + - `Unicode.ASCII.Parser`, `Unicode.UTF8.ForwardParser`, `Unicode.UTF8.ReverseParser`, `Unicode.UTF16.ForwardParser`, `Unicode.UTF16.ReverseParser`, `Unicode.UTF32.Parser` + - `Unicode.Scalar.UTF8View`, `Unicode.Scalar.UTF16View` + - `UnicodeDecodingResult` +- Some fieldless types + - `Never`, `SystemRandomNumberGenerator` +- `StaticString` +- `Hasher` +- `ObjectIdentifier` +- `Duration` +- Atomic changes + - `AtomicRepresentable.AtomicRepresentation` + - `AtomicOptionalRepresentable.AtomicOptionalRepresentation` + +## Appendix: Fluctuating bitwise-copyability + +Let's say the following type is defined in a framework built with library evolution. + +```swift +public struct Dish {...} +``` + +In the first version of the framework, the type only contains bitwise-copyable fields: + +```swift +/// NoodleKit v1.0 + +public struct Dish { + public let substrate: Noodle + public let isTopped: Bool +} +``` + +So in version `1.0`, the type is bitwise-copyable. + +In the next version of the framework, to expose more information to its clients, the stored `Bool` is replaced with a stored `Array`: + +```swift +/// NoodleKit v1.1 + +public struct Dish { + public let substrate: Noodle + public let toppings: [Topping] + public var isTopped: Bool { toppings.count > 0 } +} +``` + +As a result, in version `1.1`, the type is _not_ bitwise-copyable. + +In a subsequent version, as an optimization, the stored `Array` is replaced with an `OptionSet` + +```swift +/// NoodleKit v2.0 + +public struct Dish { + public let substrate: Noodle + private let toppingOptions: Topping + public let toppings: [Topping] { ... } + public var isTopped: Bool { toppings.count > 0 } +} +``` + +In release `2.0` the type is once again bitwise-copyable. diff --git a/proposals/0427-noncopyable-generics.md b/proposals/0427-noncopyable-generics.md new file mode 100644 index 0000000000..321c9090e8 --- /dev/null +++ b/proposals/0427-noncopyable-generics.md @@ -0,0 +1,700 @@ +# Noncopyable Generics + +* Proposal: [SE-0427](0427-noncopyable-generics.md) +* Authors: [Kavon Farvardin](https://github.com/kavon), [Tim Kientzle](https://github.com/tbkka), [Slava Pestov](https://github.com/slavapestov) +* Review Manager: [Holly Borla](https://github.com/hborla), [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.0)** +* Implementation: On `main` gated behind `-enable-experimental-feature NoncopyableGenerics` +* Previous Proposal: [SE-0390: Noncopyable structs and enums](0390-noncopyable-structs-and-enums.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-noncopyable-generics/68180)) ([first review](https://forums.swift.org/t/se-0427-noncopyable-generics/70525)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0427-noncopyable-generics/72039)) ([second review](https://forums.swift.org/t/second-review-se-0427-noncopyable-generics/72881)) ([acceptance](https://forums.swift.org/t/accepted-se-0427-noncopyable-generics/73560)) + + +**Table of Contents** + +- [Noncopyable Generics](#noncopyable-generics) + - [Introduction](#introduction) + - [Motivation](#motivation) + - [Proposed Solution](#proposed-solution) + - [The `Copyable` protocol](#the-copyable-protocol) + - [Default conformance to `Copyable`](#default-conformance-to-copyable) + - [Suppression of `Copyable`](#suppression-of-copyable) + - [Detailed Design](#detailed-design) + - [The `Copyable` protocol](#the-copyable-protocol-1) + - [Default conformances and suppression](#default-conformances-and-suppression) + - [Struct, enum and class extensions](#struct-enum-and-class-extensions) + - [Protocol extensions](#protocol-extensions) + - [Protocol inheritance](#protocol-inheritance) + - [Conformance to `Copyable`](#conformance-to-copyable) + - [Classes](#classes) + - [Existential types](#existential-types) + - [Source Compatibility](#source-compatibility) + - [ABI Compatibility](#abi-compatibility) + - [Alternatives Considered](#alternatives-considered) + - [Alternative spellings](#alternative-spellings) + - [Associated types without defaulting behavior](#associated-types-without-defaulting-behavior) + - [Inferred conditional copyability](#inferred-conditional-copyability) + - [Extension defaults](#extension-defaults) + - [Recursive `Copyable`](#recursive-copyable) + - [`~Copyable` as logical negation](#copyable-as-logical-negation) + - [Future Directions](#future-directions) + - [Suppressed associated types](#suppressed-associated-types) + - [Standard library adoption](#standard-library-adoption) + - [Tuples and parameter packs](#tuples-and-parameter-packs) + - [`~Escapable`](#escapable) + - [Acknowledgments](#acknowledgments) + + + +## Introduction + +The noncopyable types introduced in +[SE-0390: Noncopyable structs and enums](0390-noncopyable-structs-and-enums.md) +cannot be used with generics, protocols, or existentials, +leaving an expressivity gap in the language. This proposal extends Swift's +type system to fill this gap. + +## Motivation + +Noncopyable structs and enums are intended to express value types for which +it is not meaningful to have multiple copies of the same value. + +Support for noncopyable generic types was omitted from SE-0390. For example, +`Optional` could not be instantiated with a noncopyable type, +which prevented declaration of a failable initializer: +```swift +struct FileDescriptor: ~Copyable { + init?(filename: String) { // error: cannot form a Optional + ... + } +} +``` + +Practical use of generics also requires conformance to protocols, however +noncopyable types could not conform to protocols. + +In order to broaden the utility of noncopyable types in the language, we need +a consistent and sound way to relax the fundamental assumption of copyability +that permeates Swift's generics system. + +## Proposed Solution + +We begin by recalling the restrictions from SE-0390: + +1. A noncopyable type could not appear in the generic argument of some other generic type. +2. A noncopyable type could not conform to protocols. +3. A noncopyable type could not be boxed as an existential. + +This proposal builds on the `~Copyable` notation introduced in SE-0390, and +introduces three fundamental concepts that together eliminate these +restrictions: + +1. A new `Copyable` protocol abstracts over types whose values can be copied. +2. Every struct, enum, class, generic parameter, protocol and associated type +now conforms to `Copyable` _by default_. +3. The `~Copyable` notation is used to _suppress_ this default conformance +requirement anywhere it would otherwise be inferred. + +**Note**: The adoption of noncopyable generics in the standard library will be +covered in a subsequent proposal. + +### The `Copyable` protocol + +The notion of copyability of a value is now expressed as a special kind of +protocol. The existing `~Copyable` notation is re-interpreted as _suppressing_ +a conformance to this protocol, as we detail below. This protocol has no +explicit requirements, and it has some special behaviors. For example, +metatypes and tuples cannot normally conform to other protocols, +but they do conform to `Copyable`. + +A key goal of the design is _progressive disclosure_. The idea of _default_ conformance to +`Copyable` means that a user never interacts with noncopyable generics unless +they choose to do so, using the `~Copyable` notation to _suppress_ +the default conformance. + +The meaning of existing code remains the same; all generic parameters and +protocols now require conformance to `Copyable`, but all existing concrete +types do in fact conform. + +### Default conformance to `Copyable` + +Every struct and enum now has a default conformance to `Copyable`, unless the +conformance is suppressed by writing `~Copyable` in the inheritance clause. In +this proposal, we will show these inferred requirements in comments. For example, +a definition of a copyable struct is understood as if the user wrote the +conformance to `Copyable`: +```swift +struct Polygon /* : Copyable */ {...} +``` + +Furthermore, generic parameters now conform to `Copyable` by +default, so the following generic function can only be called with `Copyable` types: +```swift +func identity(x: T) /* where T: Copyable */ { return x } +``` + +Finally, protocols also have a default conformance to `Copyable`, thus +only `Copyable` types can conform to `Shape` below: +```swift +protocol Shape /*: Copyable */ {} +``` + +### Suppression of `Copyable` + +So far, we haven't described anything new, just formalized existing behavior with +a protocol. Now, we allow writing `~Copyable` in some new positions. + +For example, to generalize our identity function to also allow noncopyable types, we +suppress the default `Copyable` conformance on `T` as follows: +```swift +func identity(x: consuming T) { return x } +``` +This function imposes _no_ requirements on the generic parameter `T`. All possible +types, both `Copyable` and noncopyable, can be substituted for `T`. +This is the reason why we refer to `~Copyable` as _suppressing_ the conformance +rather than _inverting_ or _negating_ it. + +As with a concrete noncopyable type, a noncopyable generic parameter type must +be prefixed with one of the ownership modifiers `borrowing`, +`consuming`, or `inout`, when it appears as the type of a function's parameter. +For details on these parameter ownership modifiers, +see [SE-377](0377-parameter-ownership-modifiers.md). + +A protocol can allow noncopyable conforming types by suppressing its inherited +conformance to `Copyable`: +```swift +protocol Resource: ~Copyable { + consuming func dispose() +} + +extension FileDescriptor: Resource {...} +``` +A `Copyable` type can still conform to a `~Copyable` protocol. + +What it means to write `~Copyable` in each position will be fully explained in +the **Detailed Design** section. + +## Detailed Design + +This proposal does not fundamentally change the abstract theory of Swift +generics, with its four fundamental kinds of requirements that can appear in a +`where` clause; namely conformance, superclass, `AnyObject`, and same-type +requirements. + +The proposed mechanism of default conformance to `Copyable`, and its suppression by +writing `~Copyable`, is essentially a new form of syntax sugar; the transformation +is purely syntactic and local. + +### The `Copyable` protocol + +While `Copyable` is a protocol in the current implementation, it is unlike a +protocol in some ways. In particular, protocol extensions of `Copyable` are not +allowed: +```swift +extension Copyable { // error + func f() {} +} +``` +Such a protocol extension would effectively add new members to _every_ +copyable type, which would complicate overload resolution and possibly lead to +user confusion. + +### Default conformances and suppression + +Default conformance to `Copyable` is inferred in each position below, +unless explicitly suppressed: + +1. A struct, enum or class declaration. +2. A generic parameter declaration. +3. A protocol declaration. +4. An associated type declaration; does not support suppression (see Future Directions). +5. The `Self` type of a protocol extension. +6. The generic parameters of a concrete extension. + +The `~Copyable` notation is also permitted to appear as the _member_ of +a protocol composition type. This ensures that the following three declarations +have the same meaning, as one might expect: +```swift +func f(_: T) {} +func f(_: T) where T: Resource & ~Copyable {} +func f(_: T) where T: Resource, T: ~Copyable {} +``` + +A conformance to `Copyable` cannot be suppressed if it must hold for +some _other_ reason. In the above declaration of `f()`, we can suppress +`Copyable` on `T` because `Resource` suppresses its own `Copyable` requirement +on `Self`: +```swift +protocol Resource: ~Copyable {...} +``` +Thus, nothing else forces `f()`'s generic parameter `T` to be `Copyable`. On the +other hand, let's look at a copyable protocol like `Shape` below: +```swift +protocol Shape /*: Copyable */ {...} +``` +If we try to suppress the `Copyable` conformance on a generic parameter that also +conforms to `Shape`, we get an error: +```swift +func f(_: T) {...} // error +``` +The reason being that the conformance `T: Copyable` is _implied_ by `T: Shape`, and +cannot be suppressed. + +Furthermore, a `Copyable` conformance can only be suppressed if the subject type +is a generic parameter declared in the innermost scope. That is, the following +is an error: +```swift +struct S { + func f(_: T, _: U) where T: ~Copyable // error! +} +``` +The rationale here is that since `S` must be instantiated with a copyable type, +it does not make sense for a method of `S` to operate on an `S` where `T` +might be noncopyable. For a similar reason the same rule applies to nested +generic types. + +### Struct, enum and class extensions + +We wish to allow existing types to adopt noncopyability without changing the +meaning of existing code. Thus, an extension of a concrete type must introduce +a default `T: Copyable` requirement on every generic parameter of the +extended type: +```swift +struct Pair: ~Copyable {...} + +extension Pair /* where T: Copyable */ {...} +``` +The conformance can be suppressed to get an unconstrained extension of `Pair`: +```swift +extension Pair where T: ~Copyable {...} +``` + +An extension presents a copyable view of the world by default, behaving as if +`Pair` were declared like so: +```swift +struct Pair /* : Copyable */ {...} +``` + +An extension of a nested type introduces default conformance requirements for +all outer generic parameters of the extended type, and each conformance +can be individually suppressed: +```swift +struct Outer { + struct Inner {} +} + +extension Outer.Inner /* where T: Copyable, U: Copyable */ {} +extension Outer.Inner where T: ~Copyable /* , U: Copyable */ {} +extension Outer.Inner where /* T: Copyable, */ U: ~Copyable {} +``` + +An extension of a type whose generic parameters must be copyable cannot +suppress conformances: +```swift +struct Horse {...} +extension Horse where Hay: ~Copyable {...} // error +``` + +### Protocol extensions + +Where possible, we wish to allow the user to change an existing protocol to +accommodate noncopyable conforming types, without changing the meaning of existing +code. + +For this reason, an extension of a `~Copyable` protocol also introduces a default +`Self: Copyable` requirement, because this is the behavior expected from +existing clients: +```swift +protocol EventLog: ~Copyable { + ... +} + +extension EventLog /* where Self: Copyable */ { + func duplicate() -> Self { + return copy self // OK + } +} +``` + +To write an unconstrained protocol extension, suppress the conformance on `Self`: +```swift +extension EventLog where Self: ~Copyable { + ... +} +``` +Associated types cannot have their `Copyable` requirement suppressed +(see Future Directions). + +### Protocol inheritance + +Another consequence that immediately follows from the rules as explained so far +is that protocol inheritance must re-state `~Copyable` if needed: +```swift +protocol Token: ~Copyable {} +protocol ArcadeToken: Token /* , Copyable */ {} +protocol CasinoToken: Token, ~Copyable {} +``` +Again, because `~Copyable` suppresses a default conformance instead of introducing +a new kind of requirement, it is not propagated through protocol inheritance. + +### Conformance to `Copyable` + +Structs and enums conform to `Copyable` unconditionally by default, but a +conditional conformance can also be defined. For example, take this +noncopyable generic type: +```swift +enum List: ~Copyable { + case empty + indirect case element(T, List) +} +``` +We would like `List` to be `Copyable` since `Int` is, while still being +able to use a noncopyable element type, like `List`. We do +this by declaring a _conditional conformance_: +```swift +extension List: Copyable where T: Copyable {} +``` +Note that the `where` clause needs to be written, because a conformance to +`Copyable` declared in an extension does _not_ automatically add any other +requirements, unlike other extensions. + +A conditional `Copyable` conformance is not permitted if the +struct or enum declares a `deinit`. Deterministic destruction requires the +type to be unconditionally noncopyable. + +A conformance to `Copyable` is checked by verifying that every stored property +(of a struct) or associated value (of an enum) itself conforms to `Copyable`. +For a conditional `Copyable` conformance, the conditional requirements must be +sufficient to ensure this is the case. For example, the following is rejected, +because the struct cannot unconditionally conform to `Copyable`, having a +stored property of the noncopyable type `T`: +```swift +struct Holder /* : Copyable */ { + var value: T // error +} +``` + +There are two situations when it is permissible for a copyable type to +have a noncopyable generic parameter. The first is when the generic parameter +is not stored inside the type itself: +```swift +struct Factory /* : Copyable */ { + let fn: () -> T // ok +} +``` +The above is permitted, because a _function_ of type `() -> T` is still copyable, +even if a _value_ of type `T` is not copyable. + +The second case is when the type is a class. The contents of a class is never +copied, so noncopyable types can appear in the stored properties of a class: +```swift +class Box { + let value: T // ok + + init(value: consuming T) { self.value = value } +} +``` + +For a conditional `Copyable` conformance, the conditional requirements must be +of the form `T: Copyable` where `T` is a generic parameter of the type. It is +not permitted to make `Copyable` conditional on any other kind of requirement: +```swift +extension Pair: Copyable where T == Array {} // error +``` + +Conditional `Copyable` conformance must be declared in the same source +file as the struct or enum itself. Unlike conformance to other protocols, +copyability is a deep, inherent property of the type itself. + +### Classes + +This proposal supports classes with noncopyable generic parameters, +but it does not permit classes to themselves be `~Copyable`. +Similarly, an `AnyObject` or superclass requirement cannot be combined with +`~Copyable`: +```swift +func f(_ t: T) where T: AnyObject, T: ~Copyable { ... } // error +``` + +### Existential types + +The type `Any` is no longer the supertype of all types in the type system's +implicit conversion rules. + +The constraint type of an existential type is now understood as being a +protocol composition, with a default `Copyable` _member_. So +the empty protocol composition type `Any` is really `any Copyable`, and the +supertype of all types is now `any ~Copyable`: + +``` + any ~Copyable + / \ + / \ + Any == any Copyable + | + +``` + +This default conformance is suppressed by writing `~Copyable` as a member of a +protocol composition: + +```swift +protocol Pizza: ~Copyable {} +struct UniquePizza: Pizza, ~Copyable {} + +let t: any Pizza /* & Copyable */ = UniquePizza() // error +let _: any Pizza & ~Copyable = UniquePizza() // ok +``` + +## Source Compatibility + +The default conformance to `Copyable` is inferred anywhere it is not explicitly +suppressed with `~Copyable`, so this proposal does not change the interpretation +of existing code. + +Similarly, the re-interpretation of the SE-0390 restrictions in terms of +conformance to `Copyable` preserves the meaning of existing code that makes use of +noncopyable structs and enums. + +## ABI Compatibility + +This proposal does not change the ABI of existing code. + +Adding `~Copyable` to +an existing generic parameter is generally an ABI-breaking change, even when +source-compatible. + +Targeted mechanisms are being developed to preserve ABI compatibility when +adopting `~Copyable` on previously-shipped generic code. This will enable adoption +of this feature by standard library types such as `Optional`. Such mechanisms will +require extreme care to use correctly. + +## Alternatives Considered + +### Alternative spellings + +The spelling of `~Copyable` generalizes the existing syntax introduced in +SE-0390, and changing it is out of scope for this proposal. + +### Associated types without defaulting behavior + +A simple design for suppressed associated types was considered, where the +default conformance in a protocol extension applies only to `Self`, and not +the associated types of `Self`. For example, we first declare a protocol with a +`~Copyable` associated type: +```swift +protocol Manager { + associatedtype Resource: ~Copyable +} +``` +Now, a protocol extension of `Manager` does _not_ carry an implicit +`Self.Resource: Copyable` requirement: +```swift +extension Manager { + func f(resource: Resource) { + // `resource' cannot be copied here! + } +} +``` +For this reason, while adding `~Copyable` to the inheritance clause of a protocol +is a source-compatible change, the same with an _associated type_ is not +source compatible. The designer of a new protocol must decide which associated +types are `~Copyable` up-front. + +Requirements on associated types can be written in the associated type's +inheritance clause, or in a `where` clause, or on the protocol itself. As +with ordinary requirements, all three of the following forms define the same +protocol: +```swift +protocol P { associatedtype A: ~Copyable } +protocol P { associatedtype A where A: ~Copyable } +protocol P where A: ~Copyable { associatedtype A } +``` + +If a base protocol declares an associated type with a suppressed conformance +to `Copyable`, and a derived protocol re-states the associated type, a +default conformance is introduced in the derived protocol, unless it is again +suppressed: +```swift +protocol Base { + associatedtype A: ~Copyable + func f() -> A +} + +protocol Derived: Base { + associatedtype A /* : Copyable */ + func g() -> A +} +``` + +Finally, conformance to `Copyable` cannot be conditional on the copyability of +an associated type: +```swift +struct ManagerManager: ~Copyable {} +extension ManagerManager: Copyable where T.Resource: Copyable {} // error +``` + +This design for associated types was initially implemented but ultimately +removed from this proposal, because of the source compatibility issues. A more +comprehensive design that allows for some way of preserving source compatibility +requires a separate proposal due to the open design issues. + +### Inferred conditional copyability + +A struct or enum can opt out of copyability with `~Copyable`, and then possibly +declare a conditional conformance. It would be possible to automatically infer +this conditional conformance. For example, in the below, +```swift +struct MaybeCopyable { + var t: T +} +``` +The only way this _could_ be valid is if we had inferred the conditional +conformance: +``` +extension MaybeCopyable: Copyable /* where T: Copyable */ {} +``` +Feedback from early attempts at implementing this form of inference suggested +it was more confusing than helpful, so it was removed. + +### Extension defaults + +One possible downside is that extensions of types with noncopyable generic +parameters must suppress the conformance on each generic parameter. + +It would be possible to allow library authors to explicitly control this +behavior, with a new syntax allowing the default `where` clause of an +extension to be written inside of a type declaration. For example, +```swift +public enum Either { + case a(T) + case b(U) + + // Hypothetical syntax: + default extension where T: Copyable, U: ~Copyable +} + +// `T` is copyable, but `U` is not, because of the defaults above: +extension Either /* where T: Copyable */ { ... } + +``` + +This becomes much more complex for protocols that impose conformance +requirements on their own associated types: +```swift +protocol P: ~Copyable { + associatedtype A: P, ~Copyable + + // Hypothetical syntax: + default extension where A: Copyable +} + +extension P { + // A is Copyable. What about A.A? A.A.A? ... +} +``` + +Besides the unclear semantics with associated types, it was also felt this +approach could lead to user confusion about the meaning of a particular +extension. As a result, we feel that explicitly suppressing `Copyable` on +every extension is the best approach. + +### Recursive `Copyable` + +The behavior of default `Copyable` conformance on associated types prevents +existing protocols from adopting `~Copyable` on their associated types in a +source compatible way. + +For example, suppose we attempt to change `IteratorProtocol` to accommodate +noncopyable element types: +```swift +protocol IteratorProtocol: ~Copyable { + associatedtype Element: ~Copyable + mutating func next() -> Element? +} +``` +An existing program might declare a generic function that assumes `T.Element` is +`Copyable`: +```swift +func f(iter: inout T) { + let value = iter.next()! + let copy = value // error +} +``` +Since `IteratorProtocol` suppresses its `Copyable` conformance, the generic +parameter `T` defaults to `Copyable`. However, `T.Element` is no longer +`Copyable`, thus the above code would not compile. + +One can imagine a design where instead of a single default conformance +requirement `T: Copyable` being introduced above, we also add a requirement +`T.Element: Copyable`. This would preserve source compatibility and our +function `f()` would continue to work as before. + +However, this approach introduces major complications, if we again consider +protocols that impose conformance requirements on their associated types. + +Consider this simple protocol and function that uses it: +```swift +protocol P: ~Copyable { + associatedtype A: P, ~Copyable +} + +func f(_: T) {} +``` +Our hypothetical design would actually introduce an infinite sequence of +requirements here unless suppressed: +```swift +func f(_: T) /* where T: Copyable, T.A: Copyable, T.A.A: Copyable, ... */ {} +``` +Of course, it seems natural to represent this infinite sequence of requirements +as a new kind of "recursive conformance" requirement instead. + +Swift generics are based on the mathematical theory of +_string rewriting_, and requirements and associated types define certain _rewrite +rules_ which operate on a set of terms. In this formalism, a +hypothetical "recursive conformance" requirement corresponds to a rewrite +rule that can match an infinite set of terms given by a _regular expression_. +We would then need to generalize the algorithms for deciding term equivalence to +handle regular expressions. While there has been research in this area, +the design for such a system is far beyond the scope of this proposal. + +### `~Copyable` as logical negation + +Instead of the syntactic desugaring presented in this proposal, one can attempt to +formalize `T: ~Copyable` as the _logical negation_ +of a conformance, extending the theory of Swift generics with a fifth requirement kind to +represent this negation. It is not apparent how this leads to a sound and +usable model and we have not explored this further. + +## Future Directions + +### Suppressed associated types + +Supporting the full generality of associated types with suppressed Copyable +requirements, while providing a mechanism to preserve source compatibility is +a highly desirable goal. At the same time, it is a large, open design problem. +A few ideas were considered (see Alternatives Considered) but it was ultimately +determined to be too complex to tackle in this proposal. + +### Standard library adoption + +The `Optional` and `UnsafePointer` family of types can support noncopyable types +in a straightforward way. In the future, we will also explore noncopyable +collections, and so on. All of this requires significant design work and is out +of scope for this proposal. + +### Tuples and parameter packs + +Noncopyable tuples and parameter packs are a straightforward generalization +which will be discussed in a separate proposal. + +### `~Escapable` + +The ability to "escape" the current context is another implicit capability +of all current Swift types. +Suppressing this requirement provides an alternative way to control object lifetimes. +A companion proposal will provide details. + +## Acknowledgments + +Thank you to Joe Groff and Ben Cohen for their feedback throughout the +development of this proposal. diff --git a/proposals/0428-resolve-distributed-actor-protocols.md b/proposals/0428-resolve-distributed-actor-protocols.md new file mode 100644 index 0000000000..e7b4128e3d --- /dev/null +++ b/proposals/0428-resolve-distributed-actor-protocols.md @@ -0,0 +1,472 @@ +# Resolve DistributedActor protocols + +* Proposal: [SE-0428](0428-resolve-distributed-actor-protocols.md) +* Author: [Konrad 'ktoso' Malawski](https://github.com/ktoso), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-resolve-distributedactor-protocols-for-server-client-apps/69933)) ([review](https://forums.swift.org/t/se-0428-resolve-distributedactor-protocols/70669)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0428-resolve-distributedactor-protocols/71366)) + +## Introduction + +Swift's distributed actors offer developers a flexible bring-your-own-runtime approach to building distributed systems using the actor paradigm. The initial design of the feature aimed for systems where all nodes of a distributed actor system (such as nodes in a [cluster](https://github.com/apple/swift-distributed-actors)) share the same binary, and therefore all have access to the concrete `distributed actor` declarations which may be resolved and made remote calls on. + +Although this works well for peer-to-peer systems, distributed actors are also useful for systems where a client/server split is necessary. Examples of such use-cases include isolating some failure prone logic into another process, or split between a client without the necessary libraries or knowledge how to implement an API and delegating this work to a backend service. + +## Motivation + +Distributed actors allow to abstract over the location (i.e. in process or not) of an actor -- often referred to as "location transparency". Currently, Swift's distributed actors have a practical limitation in how well this property can be applied to server/client split applications because the only way to obtain a *remote reference* on which a *remote call* can be made, is by invoking the resolve method on a concrete distributed actor type, like this: + +```swift +import Distributed +import DistributedCluster // github.com/apple/swift-distributed-actors + +protocol Greeter: DistributedActor { + distributed func greet(name: String) -> String +} + +distributed actor EnglishGreeter: Greeter { + typealias ActorSystem = ClusterSystem + + func greet(name: String) -> String { + "Hello, \(name)!" + } +} + +let system: ClusterSystem = ... + +let knownID: EnglishGreeter.ID = /* obtained ID using discovery mechanisms, see ClusterSystem */ +let remote: EnglishGreeter = try EnglishGreeter.resolve(id: knownID, using: system) + +// remote call (implicitly async and throwing, due to e.g. network errors) +let greeting = try await remote.greet(name: "Caplin") +assert(greeting == "Hello, Caplin!") +``` + +This is common and acceptable in a **peer-to-peer system**, where all nodes of a cluster share the same types -- or at least a "client side" of a connection can only discover types it knows about, e.g. during version upgrade rollouts. However, this pattern is problematic in a **client/server deployment**, where the two applications do not share the concrete implementation of the `Greeter` type. It is also worth calling out that typical inter-process communications (IPC) use-cases often fall into the category of a client/server setup, where e.g. a daemon process serves as a "server" and an application "client" calls into it. + +The goal of this proposal is to allow the following module approach to sharing distributed actor APIs: +- **API** module: allow sharing a `DistributedActor` constrained protocol, which only declares the API surface the server is going to expose +- **Server** module: which depends on API module, and implements the API description using a concrete distributed actor +- **Client** module: which depends on API module, and `$Greeter` type (synthesized by the `@Resolvable` macro) to resolve a remote actor reference it can then invoke distributed methods on + +```swift + ┌────────────────────────────────────────┐ + │ API Module │ + │========================================│ + │ @Resolvable │ + │ protocol Greeter: DistributedActor { │ + ┌───────┤ distributed func greet(name: String) ├───────┐ + │ │ } │ │ + │ └────────────────────────────────────────┘ │ + │ │ + ▼ ▼ +┌────────────────────────────────────────────────┐ ┌──────────────────────────────────────────────┐ +│ Client Module │ │ Server Module │ +│================================================│ │==============================================│ +│ let g = try $Greeter.resolve(...) /*new*/ │ │ distributed actor EnglishGreeter: Greeter { │ +│ try await greeter.hello(name: ...) │ │ distributed func greet(name: String) { │ +└────────────────────────────────────────────────┘ │ "Greeting in english, for \(name)!" │ +/* Client cannot know about EnglishGreeter type */ │ } │ + │ } │ + └──────────────────────────────────────────────┘ +``` + +In this scenario the client module has no knowledge of the concrete distributed actor type or implementation. + +In order to achieve this, this proposal improves upon _three_ aspects of distributed actors: +- introduce the `@Resolvable` macro that can be attached to distributed actor protocols, and enable the use of `resolve(id:using:)` with such types, +- allow distributed actors to be generic over their `ActorSystem` +- extend the distributed metadata section such that a distributed method which is witness to a distributed requirement, is also recorded using the protocol method's `RemoteCallTarget.identifier` and not only the concrete method's + +## Proposed solution + +### The `@Resolvable` macro + +At the core of this proposal is the `@Resolvable` macro. It is an attached declaration macro, which introduces a number of declarations which allow the protocol, or rather, a "stub" type for the protocol, to be used on a client without knowledge about the server implementation's concrete distributed actor type. + +The macro must be attached to the a `protocol` declaration that is a `DistributedActor` constrained protocol, like this: + +```swift +import Distributed + +@Resolvable +protocol Greeter: DistributedActor where ActorSystem: DistributedActorSystem { + distributed func greet(name: String) -> String +} +``` + +The protocol must specify a constraint on the `ActorSystem` which specifies the kind of `SerializationRequirement` it is able to work with. This serialization requirement must be a protocol, and existing distributed actor functionality already will be verifying this. + +Checking of distributed functions works as before, and the compiler will check that the `distributed` declarations all fulfill the `SerializationRequirement` constraint. E.g. in the example above, the parameter type `String` and return type `String` both conform to the `Codable` protocol, so this distributed protocol is well formed. + +It is possible to for a distributed actor protocol to contain non-distributed requirements. However in practice, it will be impossible to ever invoke such methods on a remote distributed actor reference. It is possible to call such methods if one were to obtain a local distributed actor reference implementing such protocol, and use the existing `whenLocal(operation:)` method on it. + +The `@Resolvable` macro generates a number of internal declarations necessary for the distributed machinery to work, however the only type users should care about is always a `$`-prefixed concrete distributed actor declaration, that is the "stub" type. This stub type can be used to resolve a remote actor using this protocol stub: + +E.g. if we knew a remote has a `Greeter` instance for a specific ID we have discovered using some external mechanism, this is how we'd resolve it: + +```swift +let clusterSystem: ClusterSystem // example system + +let greeter = try $Greeter.resolve(id: id, using: clusterSystem) +``` + +As the `ClusterSystem` is using `Codable` as it's serialization requirement, the resolve compiles and produces a valid reference. + +### Distributed actors generic over their `ActorSystem` + +The previous section made use of a distributed actor that abstracted over a generic `ActorSystem`. Today (in Swift 5.10) this is not possible, and would result in a compile time error. + +Previously, the compiler would require that the `ActorSystem` typealias refer to a specific distributed actor system type (not a protocol, a concrete nominal type). For example, the following actor can only be used with the `ClusterSystem`: + +```swift +import Distributed +import DistributedCluster // github.com/apple/swift-distributed-actors provides `ClusterSystem` + +distributed actor DistributedAsyncSequence where Element: Sendable & Codable { + typealias ActorSystem = ClusterSystem + + // not real implementation; simplified method to showcase introduced capabilities + distributed func gimmeNextElement() async throws -> Element? { ... } +} +``` + +And while such generally useful "distributed async sequence" actor can be written generically, to work with the vast majority of actor systems, today's language did not allow to write such generic actor, and the `ActorSystem` type always was forced to be a _concrete_ type. + +To support this new pattern, the `DistributedActorSystem` protocol gains a *primary associated type* for the `SerializationRequirement` associated type: + +```swift +// before: +// protocol DistributedActorSystem: Sendable { +// associatedtype SerializationRequirement where ... +// // ... +// } + +// now: +protocol DistributedActorSystem: Sendable { + /// The serialization requirement that will be applied to all distributed targets used with this system. + associatedtype SerializationRequirement + where SerializationRequirement == InvocationEncoder.SerializationRequirement, + SerializationRequirement == InvocationDecoder.SerializationRequirement, + SerializationRequirement == ResultHandler.SerializationRequirement + // ... +} + +// +``` + +The `SerializationRequirement` must be specified for all actors and protocols attempting to abstract over an actor system, because it is necessary to compile-time guarantee the correctness of values passed to such distributed actor methods. The compiler uses this associated type to verify all argument types and returned values are able to be serialized when performing remote calls, and will refuse to compile invocations would otherwise would have failed at runtime. + +Thanks to this new primary associated type, it is now possible to spell our `DistributedAsyncSequence` as a generic actor, implement it once, and re-use it across any compatible actor system implementation: + +```swift +distributed actor DistributedAsyncSequence + where Element: Sendable & Codable, + ActorSystem: DistributedActorSystem { + + // not real implementation; simplified method to showcase introduced capabilities + distributed func exampleNextElement() async throws -> Element? { ... } +} +``` + +Note that since the `ActorSystem` specifies a concrete `SerializationRequirement` the compiler is still able to check that all types invoked in a distributed function call conform to this protocol, i.e. we're guaranteed to be able to serialize `Element` because the ActorSystem provided must be able to handle this serialization mechanism. + +This also extends to distributed protocols, which are now able to abstract over an actor system, while specifying what serialization requirement they support: + +```swift +protocol DistributedAsyncSequence: DistributedActor + where ActorSystem: DistributedActorSystem { + associatedtype Element: Sendable & Codable + + distributed func exampleNextElement() async throws -> Element? { ... } +} +``` + +Failing to specify the serialization requirement is a compile time error: + +```swift +protocol DistributedAsyncSequence: DistributedActor + where ActorSystem: DistributedActorSystem { + // error: distributed actor protocol must specify `ActorSystem.SerializationRequirement`, + // you can provide it like this: DistributedActorSystem + associatedtype Element: Sendable & Codable + + distributed func exampleNextElement() async throws -> Element? { ... } +} +``` + +The serialization requirement must be a `protocol` type; This was previously enforced, and remains so after this proposal. The important part is that the macro is able to synthesize an stub implementation type, that the existing resolution mechanisms can be invoked on. + +## Detailed design + +The `@Resolvable` macro generates a concrete `distributed actor` declaration as well as an extension which implements the protocol's method requirements with "stub" implementations. + +> **NOTE:** The exact details of the macro synthesized code are not guaranteed to remain the same, and may change without notice. The existence of the $-prefixed generated type is guaranteed however, as it is the public API how developers resolve and obtain remote references, I.e. for a `protocol Greeter` annotated with the `@Resolvable` macro, developers may rely on the existence of the `distributed actor $Greeter` with the same access level as the protocol. + +This proposal also introduces an empty `DistributedActorStub` protocol: + +```swift +public protocol DistributedActorStub where Self: DistributedActor {} +``` + +The `@Resolvable` macro synthesizes a concrete distributed actor which accepts a generic `ActorSystem`. The generated actor declaration matches access level with the original declaration, and implements the protocol as well as the `DistributedActorStub` protocol: + +```swift +protocol Greeter: DistributedActor where ActorSystem: DistributedActorSystem { + distributed func greet(name: String) -> String +} + +// "stub" type +distributed actor $Greeter: Greeter, DistributedStubActor + where DistributedActorSystem { + private init() {} // cannot initialize, can only resolve(id:using:) +} + +extension Greeter where Self: DistributedActorStub { + // ... stub implementations for protocol requirements ... +} +``` + +Default implementations for all the protocol's requirements (including non-distributed requirements) are provided by extensions utilizing the `DistributedActorStub` protocol. + +It is possible for a protocol type to inherit other protocols, in that case the parent protocol must either have default implementations for all its requirements, or it must also apply the `@Resolvable` protocol which generates such default implementations. + +The default method "stub" implementations provided by the `@Resolvable` simply fatal error if they were to ever be invoked. In practice, invoking those methods is not possible, because resolving a stub will always return a remote reference, and therefore calls on these methods are redirected to `DistributedActorSystem`'s `remoteCall` rather than invoking the "local" methods. + +It is recommended to use `some Greeter` or `any Greeter` rather than `$Greeter` when passing around resolved instances of a distributed actor protocol. This way none of your code is tied to the fact of using a specific type of proxy, but rather, can accept any greeter, be it local or resolved through a proxy reference. This can come in handy when refactoring a codebase, and merging modules in such way where the greeter may actually be a local instance in some situations. + +### Interaction with the `DefaultDistributedActorSystem` + +Since the introduction of distributed actors, it is possible to declare a module-wide `DefaultDistributedActorSystem` type alias, like this: + +```swift +typealias DefaultDistributedActorSystem = ClusterSystem +``` + +This makes it easier to declare distributed actors as the `ActorSystem` type requirement is witnessed by an implicit type alias generated in every concrete distributed actor, like this: + +```swift +distributed actor Worker { + // synthesized: + // typealias ActorSystem = ClusterSystem // because 'DefaultDistributedActorSystem = ClusterSystem' + + distributed func work() +} +``` + +The newly introduced ability to abstract over the `ActorSystem` in concrete distributed actors _wins_ over the synthesized typealias, causing the typealias to not be emitted: + +```swift +distributed actor Worker where ActorSystem: DistributedActorSystem { + distributed func work() +} +``` + +The `ActorSystem` type requirement of the `DistributedActorProtocol` in this case is witnessed by the generic parameter, and not by the "default" fallback type. + +This is the right behavior because this generic type works with _any_ distributed actor system where the `SerializationRequirement` is Codable, and not only on the `ClusterSystem`. + +### Extend Distributed metadata for protocol method identifier lookups + +The way distributed method invocations work on the recipient node is that a message is parsed from some incoming transport, and a `RemoteCallTarget` is recovered. The remote call target in currently is a mangled encoding of the concrete distributed method the call was made for, like this: + +A shared module introducing the `Capybara` protocol: + +```swift +// Module "Shared" +@Resolvable +public protocol Capybara where ActorSystem: DistributedActorSystem { + distributed var name: String { get } + distributed func eat() +} +``` + +The distributed actor runtime stores static metadata about distributed methods, such that the `executeDistributedTarget(on:target:invocationDecoder:handler:)` method is able to turn the mangled `RemoteCallIdentifier` into a concrete method handle that it then invokes. This proposal introduces a way to mangle calls on distributed protocol requirements, in such a way that they refer to the `$`-prefixed name, and invocations on such accessor are performed on the recipient side using the concrete actor's witness tables. + +We can illustrate the new `remoteCall` flow like this: + +**Caller, i.e. client side:** + +```swift +// Module MyClientApp + +let discoveredCaplinID = ... +let capybara: some Capybara = try $Caybara.resolve(id: discoveredCaplinID, using: system) + +// make the remote call, without knowing the concrete +// capybara type that we'll invoke on the remote side +let name = try await capybara.name + +// invokes selected actor system's remoteCall: +// DistributedActorSystem.remoteCall( +// on: someCapybara, +// target: RemoteCallIdentifier("Shared.$Capybara.name"), <<< PROTOCOL REQUIREMENT MANGLING +// invocation: , +// throwing: Never.self, +// returnType: String.self) +// ------- + +assert("Hello, \(name)!" == + "Hello, Caplin!") +``` + +The caller just performed a normal remote call as usual, however the Distributed runtime offered a protocol based remote call identifier (`RemoteCallIdentifier("Shared.$Capybara.name")`) rather than one based on some underlying type that is hidden under the `any Capybara` existential. + +The client-side does not need to know or care about what concrete implementation type is used to implement this call on the recipient system, as it will only ever be performing such distributed protocol based calls. + +**Recipient, i.e. server side:** + +```swift +// Module MyActorSystem + +final class MySampleDistributedActorSystem: DistributedActorSystem, ... { + // ... + + func findById(_ id: ActorID) -> (any DistributedActor)? { ... } + + func receiveMessage() async throws { + let envelope: MyTransportEnvelope = try await readFromNetwork() + + guard let actor = findById(envelope.id) else { + throw TargetActorNotFound(envelope.id) + } + + // RemoteCallIdentifier("Shared.Capybara.name") <<< + let target: RemoteCallTarget = envelope.target + try await executeDistributedTarget( + on: actor, + target: target, // the protocol method identifier + invocationDecoder: invocation.makeDecoder(), + handler: resultHandler + } +) +``` + +This logic is exactly the same as any existing `DistributedActorSystem` implementation -- however, changes in the Distributed runtime will handle the protocol method invocation and be able to route it to the concrete resolved actor (returned by `findByID` in this snippet, e.g. the `Caplin` concrete type). + +## Source compatibility + +The changes proposed are purely additive. + +The introduced macros do not introduce any new capabilities to the `DistributedActorSystem` protocol itself, but rather introduce new source generation techniques. + +## ABI compatibility + +This proposal is purely ABI additive. + +We introduce new static, accessible at runtime, metadata necessary for the identification and location of distributed protocol methods. + +## Wire compatibility + +> Since distributed actors are used across processes, an additional kind of compatibility is necessary to discuss in proposals which may impact how messages are sent or methods identified and invoked. + +This proposal is additive and provides additional metadata such that "distributed protocol" methods may be invoked across process. Such calls were previously not supported. + +Remote calls are identified using the `RemoteCallTarget` struct, which contains an `identifier` of the target method. In today's distributed actors these identifiers are the mangled name of the target method. + +This proposal introduces a special way to mangle calls made on default implementations of distributed protocol requirements, in such a way that the target type identifier of the protocol (e.g. `Greeter`) is replaced with the stub type (e.g. `$Greeter`), and the server performs the invocation on a specific target actor using the concrete types witness and generic accessor thunk when such calls are made. + +## Future directions + +### Improve tools for non-breaking protocol evolution + +While this approach allows sharing protocols as source of "truth" for APIs vended by a server, their capability to evolve over time is limited. + +We find that the needs of distributed protocol evolution overlap in some parts with protocol evolution in binary stable libraries. While removing an API would always be breaking, it should be possible to automatically deprecate one method, and delegate to another by adding a new parameter and defaulting it in the deprecated version. + +This could be handled with declaration macros, introducing a peer method with the expected new API: + +```swift +@Resolvable(deprecatedName: "Greeter") +protocol DeprecatedGreeter: DistributedActor { + @Distributed.Deprecated(newVersion: greet(name:), defaults: [name: nil]) + distributed func greet() -> String + + // whoops, we forgot about a name parameter and need to add it... + // We can delegate from greet() to greet(name: nil) automatically though! +} +``` + +The deprecation macro could generate the necessary delegation code, like this: + +```swift +protocol Greeter: DistributedActor { + @Distributed.Deprecated(newVersion: greet(name:), defaults: [name: nil]) + distributed func greet() -> String + +/*** + distributed func greet(name: String?) -> String + ***/ +} + +/*** +extension Greeter { + /// Default implementation for deprecated ``greet()`` + /// Delegates to ``greet(name:)`` + distributed func greet() -> String { + self.greet(name: nil) + } +} + ***/ +``` + +This simplified snippet does not solve the problem about introducing the new protocol requirement in a binary compatible way, and we'd have to come up with some pattern for it -- however the general direction of allowing introducing new versions of APIs with easier deprecation of old ones is something we'd like to explore in the future. + +Support for renaming methods could also be provided, such that the legacy method can be called `__deprecated_greet()` for example, while maintaining the "legacy name" of "`greet()`". Overall, we believe that the protocol evolution story here is something we will have to flesh out in the near future, anf feel we have the tools to do so. + +### Consider customization points for distributed call target metadata assignment + +Currently the metadata used for remote call targets is based on Swift's mangling scheme. This is sub-optimal as it includes slightly "too much" information, such as the parameters being classes or structs, un-necessarily wire causing wire-incompatible changes when one could handle them more gracefully. + +Another downside of using the mangling for the keys of the distributed method accessor identifiers is that the names can be rather long as mangling is pretty verbose. It is possible to avoid sending then complete metadata by using compression schemes such that each identifier can only be sent at-most-once over the wire, and later on a numeric representation is used between the peers. Such scheme would need to be implemented dynamically at runtime and involves some tricky logic. It would be interesting to provide a hook in the actor system to allow for consistent remote call target identifier assignment, such that the identifiers could be both small, and predictable on both sides of a system. + +This would require introducing a dynamic lookup table in the runtime and it would need to interoperate with any given distributed actor system... It remains unclear if this is a net win, or un-necessary complexity since each actor system may handle this slightly differently... + +### Utilize distributed method metadata for auditing + +Given the information in distributed metadata, we could provide a command line application, or rather extend `swift-inspect` to be able to inspect a binary or running application for the distributed entry points to the application. + +This is useful as it allows auditors to quickly scan for all potential distributed entry points into an application, making auditing easier and more reliable than source scanning as it can be performed on the final artifact of a build. + +## Alternatives considered + +### Restrict distributed actors only to peer-to-peer systems + +Since this feature expands the usefulness of distributed actors to client/server settings, we should discuss wether or not this is a good idea to begin with. + +This topic has been one of heated discussions in various ecosystems every time distributed actor systems are compared to source-generation based RPC systems (such as gRPC, or OpenAPI, and others like SOAP and others in earlier days). + +Distributed actors in Swift have the unique position of being placed in a language that deeply embraces the actor model. There are valid reasons to NOT use distributed actors in some situations, and e.g. prefer exposing your APIs over OpenAPI or other source generation tools, especially if one wants to treat these as the "source of truth." + +At the same time though, Swift is used in many exciting domains where the use of OpenAPI, or gRPC would be deemed problematic. We are interested in supporting IPC and other low-level systems which are tightly integrated with eachother, and frequently even maintained by the same teams or organizations. Deeply integrating the language, with auditing capabilities and control over distributed process boundaries, without having to step out to secondary source generation and tools is a very valuable goal in these scenarios. + +### Handle stub synthesis in the compiler + +An earlier attempt at implementation of this feature attempted to handle synthesis in the compiler, and emit ad-hoc distributed actor declaration types as triggered by the _call site_ of `resolve(id:using:)` - this is problematic in being a very custom and special path in the compiler, complicating the language and giving distributed actors more "privileges" than normal code. + +The idea was as follows: + +```swift +protocol Greeter: DistributedActor { + distributed func greet(name: String) -> String +} + +let someSystem: some DistributedActorSystem = ... +let g: any Greeter = try .resolve(id: id, using: someSystem) +``` + +This would have to synthesize an ad-hoc created anonymous declaration for a `$Greeter` and at the site of the `resolve` type-check if the declaration can be used with the `someSystem`'s serialization requirement. We would have to check if the distributed greet method's parameters and return type conform to `Codable` etc, and all this would have to happen lazily -- triggered by the existence of a `.resolve` method combining a protocol with a specific actor system. + +The only possible spelling of such API would have been this: `let g: any Greeter = try .resolve(...)` as the concrete type that is used to implement this `any Greeter` is not user visible, and cannot be. This is a lot of complexity, to what amounts to just a simple stub type. + +We believe that the macro stub approach is a good balance between convenience and lack of "magic" compiler support, as for this specific piece of the design no deep integration in the type system is necessary. + + +## Revisions + +- 1.2 + - Change implementation to not need `#resolve` macro, but rely on generic distributed actors + - General cleanup + - Change implementation approach to macros, introduce `#resolve` macro +- 1.0 + - Initial revision diff --git a/proposals/0429-partial-consumption.md b/proposals/0429-partial-consumption.md new file mode 100644 index 0000000000..d760ce11b9 --- /dev/null +++ b/proposals/0429-partial-consumption.md @@ -0,0 +1,334 @@ +# Partial consumption of noncopyable values + +* Proposal: [SE-0429](0429-partial-consumption.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm), [Nate Chandler](https://github.com/nate-chandler) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch #1](https://forums.swift.org/t/request-for-feedback-partial-consumption-of-fields-of-noncopyable-types/65884)) ([pitch #2](https://forums.swift.org/t/pitch-piecewise-consumption-of-noncopyable-values/70045)) ([review](https://forums.swift.org/t/se-0429-partial-consumption-of-noncopyable-values/70675)) ([acceptance](https://forums.swift.org/t/accepted-se-0429-partial-consumption-of-noncopyable-values/70972)) + +## Introduction + +We propose allowing noncopyable fields in deinit-less aggregates to be consumed individually, +so long as they are defined in the current module or frozen. +Additionally, we propose allowing fields of such an aggregate with a deinit to be consumed individually _within that deinit_. +This permits common patterns to be used with many noncopyable values. + +## Motivation + +In Swift today, it can be challenging to manipulate noncopyable fields of an aggregate. + +For example, consider a `Pair` of noncopyable values: + +```swift +struct Unique : ~Copyable {...} +struct Pair : ~Copyable { + let first: Unique + let second: Unique +} +``` + +It is currently not straightforward to write a function that forms a new `Pair` with the values reversed. +For example, the following code is not currently allowed: + +```swift +extension Pair { + consuming func swap() -> Pair { + return Pair( + first: second, // error: cannot partially consume 'self' + second: first // error: cannot partially consume 'self' + ) + } +} +``` + +There are various workarounds for this, but they are not ideal. + +## Proposed solution + +We allow noncopyable aggregates without deinits to be consumed field-by-field, if they are defined in the current module or frozen. +That makes `swap` above legal as written. + +This initial proposal is deliberately minimal: +- We do not allow partial consumption of [noncopyable aggregates that have deinits](#future-direction-discard). +- We do not support [reinitializing](#future-direction-partial-reinitialization) fields after they are consumed. + +[Imported aggregates](#imported-aggregates) can never be partially consumed, unless they are frozen. + +## Detailed design + +We relax the requirement that a noncopyable aggregate be consumed at most once on each path. +Instead we require only that each of its noncopyable fields be consumed at most once on each path. +Imported aggregates (i.e. those defined in another module and marked either `public` or `package`), however, cannot be partially consumed unless they are marked `@frozen`. + +Extending the `Pair` example above, the following becomes legal: + +```swift +func takeUnique(_ elt: consuming Unique) {} +extension Pair { + consuming func passUniques(_ forward: Bool) { + if forward { + takeUnique(first) + takeUnique(second) + } else { + takeUnique(second) + takeUnique(first) + } + } +} +``` + +The struct `Pair` has two noncopyable fields, `first` and `second`. +And there are two paths through the function: the paths taken when `forward` is `true` and when it is `false`. +On both paths, `first` and `second` are both consumed exactly once. + +It's not necessary to consume every field on every path, however. +For example, the following is allowed as well: + +```swift +extension Pair { + consuming func passUnique(_ front: Bool) { + if front { + takeUnique(first) + } else { + takeUnique(second) + } + } +} +``` + +Here, only `first` is consumed on the path taken when `front` is `true` and only `second` on that taken when `front` is `false`. + +### Field lifetime extension + +When a field is _not_ consumed on some path, its destruction is deferred as long as possible. +Here, that looks like this: + +```swift +extension Pair { + consuming func passUnique(_ front: Bool) { + if front { + takeUnique(first) + // second is destroyed + } else { + takeUnique(second) + // first is destroyed + } + } +} +``` + +Neither `first` nor `second` can be destroyed _after_ the `if`/`else` blocks because that would require a copy. + +### Explicit field consumption + +Fields can also be consumed explicitly via the `consume` keyword. +This enables overriding the [extension of a field's lifetime](#lifetime-extension). + +Continuing the example, if it were necessary that `first` always be destroyed before `second`, the following could be written: + +```swift +extension Pair { + consuming func passUnique(_ front: Bool) { + if front { + takeUnique(first) + // second is destroyed + } else { + _ = consume first + takeUnique(second) + } + } +} +``` + +### Imported aggregates + +Partial consumption of a non-copyable type is always allowed when the type is defined in the module where it is consumed. +If the type is defined in another module, partial consumption is only permitted if the type is marked `@frozen`. + +The reason for this limitation is that as the module defining a type changes, +the type itself may change, adding or removing fields, changing fields to computed properties, and so on. +A partial consumption of the type's fields that makes sense as the type is defined by one version of the module +may not make sense as the type is defined in another version. +That consideration does not apply to frozen types, however, +because by marking them `@frozen`, the module's author promises not to change their layouts. + +These rules are unavoidable for libraries built with library evolution +and are applied universally to avoid having language rules differ based on the build mode. + +### Copyable fields + +It is currently legal to have multiple consuming uses of a copyable field of a noncopyable aggregate. +For example: + +```swift +func takeString(_ name: consuming String) {} +struct Named : ~Copyable { + let unique: Unique + let name: String + consuming func use() { + takeString(name) + takeString(name) + takeString(name) + takeString(name) + // unique is consumed + } +} +``` + +This remains true when a value is partially consumed: + +```swift +extension Named { + consuming func unpack() { + takeString(name) + takeString(name) + takeUnique(unique) + takeString(name) + takeString(name) + } +} +``` + +### Partial consumption within deinits + +There are two related reasons to limit partial consumption to fields of types without deinits: +First, the deinit of such types can't be run if it is partially consumed. +Second, no proposed mechanism to indicate that the deinit should not be run has been accepted. + +Neither applies when partially consuming a value within its own deinit. +We propose allowing a value to be partially consumed there. + +```swift +struct Pair2 : ~Copyable { + let first: Unique + let second: Unique + + deinit { + takeUnique(first) // partially consumes self + takeUnique(second) // partially consumes self + } +} +``` + +This enables noncopyable structs to dispose of any resources they own on destruction. + +## Source compatibility + +No effect. +The proposal makes more code legal. + +## ABI compatibility + +No effect. + +## Implications on adoption + +This proposal makes more code legal. +And the code it makes legal is code written in a style familiar to Swift developers used to working with copyable values. +It alleviates some pain points associated with writing noncopyable code, easing further adoption. + +## Future directions + +### Discard + +This document proposes limiting partial consumption to aggregates without deinit. +In the future, another proposal could lift that restriction. +The trouble with lifting it is that the deinit can no longer be run, which may be surprising. +That trouble could be mitigated by requiring the value be `discard`'d prior to partial consumption, +indicating that the deinit should not be run. + +```swift +struct Box : ~Copyable { + var unique: Unique + deinit {...} + + consuming func unpack() -> Unique { + discard self + return unique + } +} +``` + +### Partial reinitialization + +This document only proposes allowing the fields of an aggregate to be consumed individually. +It does not allow for those fields to be _reinitialized_ in order to return the aggregate to a legal state. +In the future, though, another proposal could lift that restriction. + +That would enable further code patterns--already legal with copyable values--to be written in noncopyable contexts +For example: + +```swift +struct Unique : ~Copyable {} +struct Pair : ~Copyable { + var first: Unique + var second: Unique +} + +extension Pair { + mutating func swap() { + let tmp = first + first = second + second = tmp + } +} +``` + +### Partial consumption of copyable fields + +This document only proposes allowing the noncopyable fields of a noncopyable aggregate to be consumed individually. +In the future, the ability to explicitly consume (via the `consume` keyword) the copyable fields of a copyable aggregate could be added. + +```swift +class C {} +func takeC(_ c: consuming C) +struct PairPlusC : ~Copyable { + let first: Unique + let second: Unique + let c: C +} + +func disaggregate(_ p: consuming PairPlusC) { + takeUnique(p.first) + takeC(consume p.c) // p.c's lifetime ends + takeUnique(p.second) +} +``` + +That would provide the ability to specify the point at which the lifetime of a copyable field should end. + +### Partial consumption of copyable aggregates + +This document only proposes allowing noncopyable aggregates to be partially consumed. +There is a natural extension of this to copyable aggregates: + +```swift +class C {} +struct CopyablePairOfCs { + let c1: C + let c2: C +} +func tearDownInOrder(_ p: consuming CopyablePairOfCs) { + takeC(consume p.c2) + takeC(consume p.c1) +} +``` + +## Alternatives considered + +### Explicit destructuring + +Instead of consuming the fields of a struct piecewise, an alternative would be to simultaneously bind every field to a variable: + +```swift +let (a, b) = destructure s +``` + +Something like this might be desirable eventually, but it would be best introduced as part of support for pattern matching for structs. +Even with such a feature, the behavior proposed here would remain desirable: +fields of a copyable aggregate can be consumed field-by-field, +so consuming fields of a noncopyable aggregate should be supported as much as possible too. + +## Acknowledgments + +Thanks to Andrew Trick for extensive design conversations and implementation review. diff --git a/proposals/0430-transferring-parameters-and-results.md b/proposals/0430-transferring-parameters-and-results.md new file mode 100644 index 0000000000..031685c0ee --- /dev/null +++ b/proposals/0430-transferring-parameters-and-results.md @@ -0,0 +1,539 @@ +# `sending` parameter and result values + +* Proposal: [SE-0430](0430-transferring-parameters-and-results.md) +* Authors: [Michael Gottesman](https://github.com/gottesmm), [Holly Borla](https://github.com/hborla), [John McCall](https://github.com/rjmccall) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 6.0)** +* Previous Proposal: [SE-0414: Region-based isolation](/proposals/0414-region-based-isolation.md) +* Previous Revisions: [1](https://github.com/apple/swift-evolution/blob/87943205551af43682ef50260816f3ff2ef9b7ea/proposals/0430-transferring-parameters-and-results.md) [2](https://github.com/apple/swift-evolution/blob/4dded8ed382b526a5a301c225a1d45018f8d556b/proposals/0430-transferring-parameters-and-results.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-transferring-isolation-regions-of-parameter-and-result-values/70240)) ([first review](https://forums.swift.org/t/se-0430-transferring-isolation-regions-of-parameter-and-result-values/70830)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0430-transferring-isolation-regions-of-parameter-and-result-values/71297)) ([second review](https://forums.swift.org/t/se-0430-second-review-sendable-parameter-and-result-values/71685)) ([acceptance with modifications](https://forums.swift.org/t/accepted-with-modifications-se-0430-second-review-sendable-parameter-and-result-values/71850)) ([amendment pitch](https://forums.swift.org/t/pitch-revise-se-0430-to-adopt-sending-on-unsafecontinuation/72289)) ([amendment review](https://forums.swift.org/t/amendment-se-0430-sending-parameter-and-result-values/72653)) + + +## Introduction + +This proposal extends region isolation to enable the application of an explicit +`sending` annotation to function parameters and results. A function parameter +or result that is annotated with `sending` is required to be disconnected at +the function boundary and thus possesses the capability of being safely sent +across an isolation domain or merged into an actor-isolated region in the +function's body or the function's caller respectively. + +## Motivation + +SE-0414 introduced region isolation to enable non-`Sendable` typed values to be +safely sent over isolation boundaries. In most cases, function argument and +result values are merged together into the same region for any given call. This +means that non-`Sendable` typed parameter values can never be sent: + +```swift +// Compiled with -swift-version 6 + +class NonSendable {} + +@MainActor func main(ns: NonSendable) {} + +func trySend(ns: NonSendable) async { + // error: sending 'ns' can result in data races. + // note: sending task-isolated 'ns' to main actor-isolated + // 'main' could cause races between main actor-isolated + // and task-isolated uses + await main(ns: ns) +} +``` + +Actor initializers have a special rule that requires their parameter values to be +sent into the actor instance's isolation region. Actor initializers are +`nonisolated`, so a call to an actor initializer does not cross an isolation +boundary, meaning the argument values would be usable in the caller after the +initializer returns under the standard region isolation rules. SE-0414 consider +actor initializer parameters as being sent into the actor's region to allow +initializing actor-isolated state with those values: + +```swift +class NonSendable {} + +actor MyActor { + let ns: NonSendable + init(ns: NonSendable) { + self.ns = ns + } +} + +func send() { + let ns = NonSendable() + let myActor = MyActor(ns: ns) // okay; 'ns' is sent into the 'myActor' region +} + +func invalidSend() { + let ns = NonSendable() + + // error: sending 'ns' may cause a data race + // note: sending 'ns' from nonisolated caller to actor-isolated + // 'init'. Later uses in caller could race with uses on the actor. + let myActor = MyActor(ns: ns) + + print(ns) // note: note: access here could race +} +``` + +In the above code, if the local variable `ns` in the function `send` was instead +a function parameter, it would be invalid to send `ns` into `myActor`'s region +because the caller of `send()` may use the argument value after `send()` +returns: + +```swift +func send(ns: NonSendable) { + // error: sending 'ns' may cause a data race + // note: task-isolated 'ns' to actor-isolated 'init' could cause races between + // actor-isolated and task-isolated uses. + let myActor = MyActor(ns: ns) +} + +func callSend() { + let ns = NonSendable() + send(ns: ns) + print(ns) +} +``` + +The "sending parameter" behavior of actor initializers is a generally +useful concept, but it is not possible to explicitly specify that functions +and methods can send away specific parameter values. Consider the following +code that uses `CheckedContinuation`: + +```swift +@MainActor var mainActorState: NonSendable? + +nonisolated func test() async { + let ns = await withCheckedContinuation { continuation in + Task { @MainActor in + let ns = NonSendable() + // Oh no! 'NonSendable' is passed from the main actor to a + // nonisolated context here! + continuation.resume(returning: ns) + + // Save 'ns' to main actor state for concurrent access later on + mainActorState = ns + } + } + + // 'ns' and 'mainActorState' are now the same non-Sendable value; + // concurrent access is possible! + ns.mutate() +} +``` + +In the above code, the closure argument to `withCheckedContinuation` crosses an +isolation boundary to get onto the main actor, creates a non-`Sendable` typed +value, then resumes the continuation with that non-`Sendable` typed value. The +non-`Sendable` typed value is then returned to the original `nonisolated` context, +thus crossing an isolation boundary. Because `resume(returning:)` does not +impose a `Sendable` requirement on its argument, this code does not produce any +data-race safety diagnostics, even under `-strict-concurrency=complete`. + +Requiring `Sendable` on the parameter type of `resume(returning:)` is a harsh +restriction, and it's safe to pass a non-`Sendable` typed value as long as the value +is in a disconnected region and all values in that disconnected region are not +used again after the call to `resume(returning:)`. + +## Proposed solution + +This proposal enables explicitly specifying parameter and result values as +possessing the capability of being sent over an isolation boundary by annotating +the value with a contextual `sending` keyword: + +```swift +public struct CheckedContinuation: Sendable { + public func resume(returning value: sending T) +} + +public func withCheckedContinuation( + function: String = #function, + _ body: (CheckedContinuation) -> Void +) async -> sending T +``` + +## Detailed design + +### Sendable Values and Sendable Types + +A type that conforms to the `Sendable` protocol is a thread-safe type: values of +that type can be shared with and used safely from multiple concurrent contexts +at once without causing data races. If a value does not conform to `Sendable`, +Swift must ensure that the value is never used concurrently. The value can still +be sent between concurrent contexts, but the send must be a complete transfer of +the value's entire region implying that all uses of the value (and anything +non-`Sendable` typed that can be reached from the value) must end in the source +concurrency context before any uses can begin in the destination concurrency +context. Swift achieves this property by requiring that the value is in a +disconnected region and we say that such a value is a `sending` value. + +Thus a newly-created value with no connections to existing regions is always a +`sending` value: + +```swift +func f() async { + // This is a `sending` value since we can transfer it safely... + let ns = NonSendable() + + // ... here by calling 'sendToMain'. + await sendToMain(ns) +} +``` + +Once defined, a `sending` value can be merged into other isolation +regions. Once merged, such regions, if not disconnected, will prevent the value +from being sent to another isolation domain implying that the value is no longer +a `sending` value: + +```swift +actor MyActor { + var myNS: NonSendable + + func g() async { + // 'ns' is initially a `sending` value since it is in a disconnected region... + let ns = NonSendable() + + // ... but once we assign 'ns' into 'myNS', 'ns' is no longer a sending + // value... + myNS = ns + + // ... causing calling 'sendToMain' to be an error. + await sendToMain(ns) + } +} +``` + +If a `sending` value's isolation region is merged into another disconnected +isolation region, then the value is still considered to be `sending` since two +disconnected regions when merged form a new disconnected region: + +```swift +func h() async { + // This is a `sending` value. + let ns = Nonsending() + + // This also a `sending` value. + let ns2 = NonSendable() + + // Since both ns and ns2 are disconnected, the region associated with + // tuple is also disconnected and thus 't' is a `sending` value... + let t = (ns, ns2) + + // ... that can be sent across a concurrency boundary safely. + await sendToMain(t) +} +``` + +### sending Parameters and Results + +A `sending` function parameter requires that the argument value be in a +disconnected region. At the point of the call, the disconnected region is no +longer in the caller's isolation domain, allowing the callee to send the +parameter value to a region that is opaque to the caller: + +```swift +@MainActor +func acceptSend(_: sending NonSendable) {} + +func sendToMain() async { + let ns = NonSendable() + + // error: sending 'ns' may cause a race + // note: 'ns' is passed as a 'sending' parameter to 'acceptSend'. Local uses could race with + // later uses in 'acceptSend'. + await acceptSend(ns) + + // note: access here could race + print(ns) +} +``` + +What the callee does with the argument value is opaque to the caller; the callee +may send the value away, or it may merge the value to the isolation region of +one of the other parameters. + +A `sending` result requires that the function implementation returns a value in +a disconnected region: + +```swift +@MainActor +struct S { + let ns: NonSendable + + func getNonSendableInvalid() -> sending NonSendable { + // error: sending 'self.ns' may cause a data race + // note: main actor-isolated 'self.ns' is returned as a 'sending' result. + // Caller uses could race against main actor-isolated uses. + return ns + } + + func getNonSendable() -> sending NonSendable { + return NonSendable() // okay + } +} +``` + +The caller of a function returning a `sending` result can assume the value is +in a disconnected region, enabling non-`Sendable` typed result values to cross +an actor isolation boundary: + +```swift +@MainActor func onMain(_: NonSendable) { ... } + +nonisolated func f(s: S) async { + let ns = s.getNonSendable() // okay; 'ns' is in a disconnected region + + await onMain(ns) // 'ns' can be sent away to the main actor +} +``` + +### Function subtyping + +For a given type `T`, `sending T` is a subtype of `T`. `sending` is +contravariant in parameter position; if a function type is expecting a regular +parameter of type `T`, it's perfectly valid to pass a `sending T` value +that is known to be in a disconnected region. If a function is expecting a +parameter of type `sending T`, it is not valid to pass a value that is not +in a disconnected region: + +```swift +func sendingParameterConversions( + f1: (sending NonSendable) -> Void, + f2: (NonSendable) -> Void +) { + let _: (sending NonSendable) -> Void = f1 // okay + let _: (sending NonSendable) -> Void = f2 // okay + let _: (NonSendable) -> Void = f1 // error +} +``` + +`sending` is covariant in result position. If a function returns a value +of type `sending T`, it's valid to instead treat the result as if it were +merged with the other parameters. If a function returns a regular value of type +`T`, it is not valid to assume the value is in a disconnected region: + +```swift +func sendingResultConversions( + f1: () -> sending NonSendable, + f2: () -> NonSendable +) { + let _: () -> sending NonSendable = f1 // okay + let _: () -> sending NonSendable = f2 // error + let _: () -> NonSendable = f1 // okay +} +``` + +### Protocol conformances + +A protocol requirement may include `sending` parameter or result annotations: + +```swift +protocol P1 { + func requirement(_: sending NonSendable) +} + +protocol P2 { + func requirement() -> sending NonSendable +} +``` + +Following the function subtyping rules in the previous section, a protocol +requirement with a `sending` parameter may be witnessed by a function with a +non-`sending` parameter: + +```swift +struct X1: P1 { + func requirement(_: sending NonSendable) {} +} + +struct X2: P1 { + func requirement(_: NonSendable) {} +} +``` + +A protocol requirement with a `sending` result must be witnessed by a function +with a `sending` result, and a requirement with a plain result of type `T` may +be witnessed by a function returning a `sending T`: + +```swift +struct Y1: P1 { + func requirement() -> sending NonSendable { + return NonSendable() + } +} + +struct Y2: P1 { + let ns: NonSendable + func requirement() -> NonSendable { // error + return ns + } +} +``` + +### `inout sending` parameters + +A `sending` parameter can also be marked as `inout`, meaning that the argument +value must be in a disconnected region when passed to the function, and the +parameter value must be in a disconnected region when the function +returns. Inside the function, the `inout sending` parameter can be merged with +actor-isolated callees or further sent as long as the parameter is +re-assigned a value in a disconnected region upon function exit. + +### Ownership convention for `sending` parameters + +When a call passes an argument to a `sending` parameter, the caller cannot +use the argument value again after the callee returns. By default `sending` +on a function parameter implies that the callee consumes the parameter. Like +`consuming` parameters, a `sending` parameter can be re-assigned inside +the callee. Unlike `consuming` parameters, `sending` parameters do not +have no-implicit-copying semantics. + +To opt into no-implicit-copying semantics or to change the default ownership +convention, `sending` may also be paired with an explicit `consuming` ownership modifier: + +```swift +func sendingConsuming(_ x: consuming sending T) { ... } +``` + +### Adoption in the Concurrency library + +There are several APIs in the concurrency library that send a parameter across +isolation boundaries and don't need the full guarantees of `Sendable`. These +APIs will instead adopt `sending` parameters: + +* `CheckedContinuation.resume(returning:)` +* `UnsafeContinuation.resume(returning:)` +* `Async{Throwing}Stream.Continuation.yield(_:)` +* `Async{Throwing}Stream.Continuation.yield(with:)` +* The `Task` creation APIs + +## Source compatibility + +In the Swift 5 language mode, `sending` diagnostics are suppressed under +minimal concurrency checking, and diagnosed as warnings under strict concurrency +checking. The diagnostics are errors in the Swift 6 language mode, as shown in +the code examples in this proposal. This diagnostic behavior based on language +mode allows `sending` to be adopted in existing Concurrency APIs including +`CheckedContinuation`. + +## ABI compatibility + +This proposal does not change how any existing code is compiled. + +## Implications on adoption + +Adding `sending` to a parameter is more restrictive at the caller, and +more expressive in the callee. Adding `sending` to a result type is more +restrictive in the callee, and more expressive in the caller. + +For libraries with library evolution, `sending` changes name mangling, so +any adoption must preserve the mangling using `@_silgen_name`. Adoping +`sending` must preserve the ownership convention of parameters; no +additional annotation is necessary if the parameter is already (implicitly or +explicitly) `consuming`. + +## Future directions + +### `Disconnected` types + +`sending` requires parameter and result values to be in a disconnected +region at the function boundary, but there is no way to preserve that a value +is in a disconnected region through stored properties, collections, function +calls, etc. To preserve that a value is in a disconnected region through the +type system, we could introduce a `Disconnected` type into the Concurrency +library. The `Disconnected` type would suppress copying via `~Copyable`, it +would conform to `Sendable`, constructing a `Disconnected` instance would +require the value it wraps to be in a disconnected region, and a value of type +`Disconnected` can never be merged into another isolation region. + +This would enable important patterns that take a `sending T` parameter, store +the value in a collection of `Disconnected`, and later remove values from the +collection and return them as `sending T` results. This would allow some +`AsyncSequence` types to return non-`Sendable` typed buffered elements as +`sending` without resorting to unsafe opt-outs in the implementation. + +## Alternatives considered + +### Use `transferring` or `sending` instead of `sendable` + +This proposal originally used the word `transferring` for `sendable`. The idea +was that this would superficially match parameter modifiers like `consuming` and +`borrowing`. But, this ignored that we are not actually `transferring` the +parameter into another isolation domain at the function boundary point. Instead, +we are requiring that the value at that point be in a disconnected region and +thus have the _capability_ to be sent to another isolation domain or merged into +actor isolated state. This is in contrast to `consuming` and `borrowing` which +actively affect the value at the function boundary point by consuming or +borrowing the value. Additionally, by using `transferring` would introduce a new +term of art into the language unnecessarily and contrasts with already +introduced terms like `@Sendable` and the `Sendable` protocol. + +It was also suggested that perhaps instead of renaming `transferring` to +`sendable`, it should have been renamed to `sending`. This was rejected by the +authors since it runs into the same problem as `transferring` namely that it is +suggesting that the value is actively being moved to another isolation domain, +when we are expressing a latent capability of the value. + +### Exclude `UnsafeContinuation` + +An earlier version of this proposal excluded +`UnsafeContinuation.resume(returning:)` from the list of standard library +APIs that adopt `sending`. This meant that `UnsafeContinuation` didn't +require either the return type to be `Sendable` or the return value to be +`sending`. Since `UnsafeContinuation` is unconditionally `Sendable`, this +effectively made it a major hole in sendability checking. + +This was an intentional choice. The reasoning was that `UnsafeContinuation` +was already an explicitly unsafe type, and so it's not illogical for it +to also work as an unsafe opt-out from sendability checks. There are some +uses of continuations that do need an opt-out like this. For example, it +is not uncommon for a continuation to be resumed from a context that's +isolated to the same actor as the context that called `withUnsafeContinuation`. +In this situation, the data flow through the continuation is essentially +internal to the actor. This means there's no need for any sendability +restrictions; both `sending` and `Sendable` would be over-conservative. + +However, the nature of the unsafety introduced by this exclusion is very +different from the normal unsafety of an `UnsafeContinuation`. +Continuations must be resumed exactly once; that's the condition that +`CheckedContinuation` checks. If a programmer can prove that +they will do that to their own satisfaction, they should be able to use +`UnsafeContinuation` instead of `CheckedContinuation` in full confidence. +Making `UnsafeContinuation` *also* a potential source of concurrency-safety +holes is likely to be surprising to programmers. + +Conversely, if a programmer needs to opt out of sendability checks but +is *not* confident about how many times their continuation will be +resumed --- for example, if it's resumed from an arbitrary callback --- +forcing them to adopt `UnsafeContinuation` in order to achieve their +goal is actively undesirable. + +Not requiring `sending` in `UnsafeContinuation` also makes the high-level +interfaces of `UnsafeContinuation` and `CheckedContinuation` inconsistent. +This means that programmers cannot always easily move from an unsafe to a +checked continuation. That is a common need, for example when fixing +a bug and trying to prove that the unsafe continuation is not implicated. + +Swift has generally resisted adding new dimensions of unsafety to unsafe +types this way. For example, `UnsafePointer` was originally specified as +unconditionally `Sendable` in [SE-0302][], but that conformance was +removed in [SE-0331][], and pointers are now unconditionally +non-`Sendable`. The logic in both of those proposals closely parallels +this one: at first, `UnsafePointer` was seen as an unsafe type that should +not be burdened with partial safety checks, and then the community +recognized that this was actually adding a new dimension of unsafety to +how the type interacted with concurrency. + +Finally, there is already a general unsafe opt-out from sendability +checking: `nonisolated(unsafe)`. It is better for Swift to encourage +consistent use of a single unsafe opt-out than to build *ad hoc* +opt-outs into many different APIs, because it is much easier to find, +recognize, and audit uses of the former. + +For these reasons, `UnsafeContinuation.resume(returning:)` now requires +its argument to be `sending`, and the result of `withUnsafeContinuation` +is correspondingly now marked as `sending`. + +[SE-0302]: https://github.com/apple/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md +[SE-0331]: https://github.com/apple/swift-evolution/blob/main/proposals/0331-remove-sendable-from-unsafepointer.md diff --git a/proposals/0431-isolated-any-functions.md b/proposals/0431-isolated-any-functions.md new file mode 100644 index 0000000000..1797df6616 --- /dev/null +++ b/proposals/0431-isolated-any-functions.md @@ -0,0 +1,863 @@ +# `@isolated(any)` Function Types + +* Proposal: [SE-0431](0431-isolated-any-functions.md) +* Authors: [John McCall](https://github.com/rjmccall) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.0)** +* Previous revision: [1](https://github.com/swiftlang/swift-evolution/blob/b35498bf6f198477be50809c0fec3944259e86d0/proposals/0431-isolated-any-functions.md) +* Review: ([pitch](https://forums.swift.org/t/isolated-any-function-types/70562))([review](https://forums.swift.org/t/se-0431-isolated-any-function-types/70939))([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0431-isolated-any-function-types/71611)) + +[SE-0316]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0316-global-actors.md +[SE-0392]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md +[isolated-captures]: https://forums.swift.org/t/closure-isolation-control/70378 +[generalized-isolation]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0420-inheritance-of-actor-isolation.md#generalized-isolation-checking +[regions]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0414-region-based-isolation.md +[region-transfers]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0430-transferring-parameters-and-results.md + +## Introduction + +The actor isolation of a function is an important part of how it's +used. Swift can reason precisely about the isolation of a specific +function *declaration*, but when functions are passed around as +*values*, Swift's function types are not expressive enough to keep up. +This proposal adds a new kind of function type that carries its function's +actor isolation dynamically. This solves a variety of expressivity +problems in the language. It also allows features such as the standard +library's task-creation APIs to be implemented more efficiently and +with stronger semantic guarantees. + +## Motivation + +The safety of Swift concurrency relies on understanding the isolation +requirements of functions. The caller of an isolated synchronous +function must run it in an appropriately-isolated context or else the +function will almost certainly introduce data races. + +Function declarations and closures in Swift support three different +forms of actor isolation: + +- They can be non-isolated. +- They can be isolated to a specific [global actor][SE-0316] type. +- They can be isolated to a specific parameter or captured value. + +A function's isolation can be specified or inferred in many ways. +Non-isolation is the default if no other rules apply, and it can also +be specified explicitly with the `nonisolated` modifier. Global actor +isolation can be expressed explicitly with a global actor attribute, +such as `@MainActor`, but it can also be inferred from context, such +as in the methods of main-actor-isolated types. A function can +explicitly declare one of its parameters as `isolated` to isolate +itself to the value of that parameter; this is also done implicitly to +the `self` parameter of an actor method if the method doesn't explicitly +use some other isolation. Closure expressions can be declared with a +global actor attribute, and there is a [proposal currently being +developed][isolated-captures] to also allow them to have an explicit +`isolated` capture or to be explicitly non-isolated. Additionally, +when you pass a closure expression directly to the `Task` initializer, +that closure is inferred to have the isolation of the enclosing context.[^1] +These rules are fairly complex, but at the end of the day, they all +boil down to this: every function is assigned one of the three kinds of +actor isolation above. + +[^1]: Currently, if the enclosing context is isolated to a value, the +closure is only isolated to it if it actually captures that value (by +using it somewhere in its body). This is often seen as confusing, and +the `isolated` captures proposal is considering lifting this restriction +by unconditionally capturing the value. + +When a function is called directly, Swift's isolation checker can +analyze its isolation precisely and compare that to the isolation of the +calling context. However, when a call expression calls an opaque value +of function type, Swift is limited by what can be expressed in the type +system: + +- A function type with no isolation specifiers, such as `() -> Int`, + represents a non-isolated function. + +- A function type with a global actor attribute, such as + `@MainActor () -> Int`, represents a function that's isolated to that + global actor. + +- A function type with an `isolated` parameter, such as + `(isolated MyActor) - > Int`, represents a function that's isolated to + that parameter. + +But there's a very important case that can't be expressed in the type +system like this: a closure can be isolated to one of its captures. In +the following example, the closure is isolated to its captured `self` +value: + +```swift +actor WorldModelObject { + var position: Point3D + + func linearMove(to finalPosition: Point3D, over time: Duration) { + let originalPosition = self.position + let motion = finalPosition - originalPosition + + gradually(over: time) { [isolated self] progressProportion in + self.position = originalPosition + progressProportion * motion + } + } + + func updateLater() { + Task { + // This closure doesn't have an explicit isolation + // specification, and it's being passed to the `Task` + // initializer, so it will be inferred to have the same + // isolation as its enclosing context. The enclosing + // context is isolated to its `self` parameter, which this + // closure captures, so this closure will also be isolated + // that value. + self.update() + } + } +} +``` + +This inexpressible case also arises with a partial application of an +actor method, such as `myActor.methodName`: the resulting function +value captures `myActor` and is isolated to it. For now, these are +the only two cases of isolated captures. However, the upcoming +[closure isolation control][isolated-captures] proposal is expected +to give this significantly greater prominence and importance. Under +that proposal, isolated captures will become a powerful general tool +for controlling the isolation of a specific piece of code. But there +will still not be a way to express the isolation of that closure in +the type system.[^2] + +[^2]: Expressing this exactly would require the use of value-dependent +types. Value dependence is an advanced type system feature that we +cannot easily add to Swift. This is discussed in greater depth in +the Future Directions section. + +This is a very unfortunate limitation, because it actually means that +there's no way for a function to accept a function argument with +arbitrary isolation without completely erasing that isolation. Swift +does allow functions with arbitrary isolation to be converted to a +non-isolated function type, but this comes with three severe drawbacks. +The first is that the resulting function type must be `async` so that +it can switch to the right isolation internally. The second is that, +because the function changes isolation internally, it is limited in its +ability to work with non-`Sendable` values because any argument or return +value must cross an isolation boundary. And the third is that the +isolation is completely dynamically erased: there is no way for the +recipient of the function value to recover what isolation the function +actually wants, which often puts the recipient in the position of doing +unnecessary work. + +Here's an example of that last problem. The `Task` initializer receives +an opaque value of type `() async throws -> ()`. Because it cannot +dynamically recover the isolation from this value, the initializer has +no choice but to start the task on the global concurrent executor. If +the function passed to the initializer is actually isolated to an actor, +it will immediately switch to that actor on entry. This requires +additional synchronization and may require re-suspending the task. +Perhaps more importantly, it means that the order in which tasks are +actually enqueued on the actor is not necessarily the same as the order +in which they were created. It would be much better --- both semantically +and for performance --- if the initializer could immediately enqueue the +task on the right executor to begin with. + +The straightforward solution to these problems is to add a type which +is capable of expressing a function with an arbitrary (but statically +unknown) isolation. That is what we propose to do. + +## Proposed solution + +This proposal adds a new attribute that can be placed on function types: + +```swift +func gradually(over: Duration, operation: @isolated(any) (Double) -> ()) +``` + +A function value with this type dynamically carries the isolation of +the function that was used to initialize it. + +When such a function is called from an arbitrary context, it must be +assumed to always cross an isolation boundary. This means, among other +things, that the call is effectively asynchronous and must be `await`ed. + +```swift +await operation(timePassed / overallDuration) +``` + +The isolation can be read using the special `isolation` property +of these types: + +```swift +func traverse(operation: @isolated(any) (Node) -> ()) { + let isolation = operation.isolation +} +``` + +The isolation checker knows that the value of this special property +matches the isolation of the function, so calls to the function from +contexts that are isolated to the `isolation` value do not cross +an isolation boundary. + +Finally, every task-creation API in the standard library will be updated +to take a `@isolated(any)` function value and synchronously enqueue the +new task on the appropriate executor. + +## Detailed design + +### Grammar and structural rules + +`@isolated(any)` is a new type attribute that can only be applied to +function types. It is an isolation specification, and it is an error +to combine it with other isolation specifications such as a global +actor attribute or an `isolated` parameter. + +`@isolated(any)` is not a *concrete* isolation specification and cannot +be directly applied to a declaration or a closure. That is, you cannot +declare a function *entity* as having `@isolated(any)` isolation, +because Swift needs to know what the actual isolation is, and +`@isolated(any)` does not provide a rule for that. + +### Conversions + +Let `F` and `G` be function types, and let `F'` and `G'` be the corresponding +function types with any isolation specifier removed (including but not +limited to `@isolated(any)`. If either `F` or `G` specifies +`@isolated(any)` then a value of type `F` can be converted to type `G` +if a value of type `F'` could be converted to type `G'` *and* the +following conditions apply: + +- If `F` and `G` both specify `@isolated(any)`, there are no further + conditions. The resulting function is dynamically isolated to the + same value as the original function. + +- If only `G` specifies `@isolated(any)`, then the behavior depends on the + specified isolation of `F`: + + - If `F` has an `isolated` parameter, the conversion is invalid. + - Otherwise, the conversion is valid, and the dynamic isolation of + the resulting function is determined as follows: + - If the converted value is the result of an expression that is a + closure expression (including an implicit autoclosure), a function + reference, or a partial application of a method reference, the + resulting function is dynamically isolated to the isolation of the + function or closure. This looks through syntax that has no impact + on the value produced by the expression, such as parentheses; the + list is the same as in [SE-0420][generalized-isolation]. + - Otherwise, if `F` is non-isolated, the resulting function is + dynamically non-isolated. + - Otherwise, `F` must be isolated to a global actor, and the resulting + function is dynamically isolated to that global actor. + +- If only `F` specifies `@isolated(any)`, then `G` must be an `async` function + type. `G` may have any isolation specifier, but it will be ignored and the + function will run with the isolation of the original value. The arguments + and result must be sendable across an isolation boundary. It is unspecified + whether the task will dynamically suspend when calling or returning from + the resulting value. + +### Effects of intermediate conversions + +In general, all of the isolation semantics and runtime behaviors laid out +here are affected by intermediate conversions to non-`@isolated(any)` +function types. For example, if you coerce a non-isolated function or +closure to the type `@MainActor () -> ()`, the resulting function will +thereafter be treated as a `MainActor`-isolated function; the fact that +it was originally a non-isolated function is both statically and +dynamically erased. + +### Runtime behavior + +Calling a `@isolated(any)` function value behaves the same way as a direct +call to a function with that isolation would: + +- If the function is `async`, it will run with its formal isolation. This + includes leaving isolated contexts if the function is dynamically + non-isolated, as specified by [SE-0338](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0338-clarify-execution-non-actor-async.md). + +- If the function is synchronous, it will run with its formal isolation + only if it is dynamically isolated. If it is dynamically non-isolated, + it will simply run synchronously in the current context, even if that + is isolated, just like an ordinary call to a non-isolated synchronous + function would. + +### `isolation` property + +Values of `@isolated(any)` function type have a special `isolation` +property. The property is read-only and has type `(any Actor)?`. The +value of the property is determined by the dynamic isolation of the +function value: + +- If the function is dynamically non-isolated, the value of `isolation` + is `nil`. +- If the function is dynamically isolated to a global actor type `G`, + the value of `isolation` is `G.shared`. +- If the function is dynamically isolated to a specific actor reference, + the value of `isolation` is that actor reference. + +### Distributed actors + +Function values cannot generally be isolated to a distributed actor +unless the actor is known to be local. When a distributed actor *is* +local, function values isolated to the actor can be converted to +`@isolated(any)` type as above. The `isolation` property presents +the distributed actor as an `(any Actor)?` using the same mechanism +as [`#isolation`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0420-inheritance-of-actor-isolation.md#isolated-distributed-actors). + +### Isolation checking + +Since the isolation of an `@isolated(any)` function value is +statically unknown, calls to it typically cross an isolation boundary. +This means that the call must be `await`ed even if the function is +synchronous, and the arguments and result must satisfy the usual +sendability restrictions for cross-isolation calls. The function +value itself must satisfy a slightly less restrictive rule: it must +be a sendable value only if it is `async` and the current +context is not statically known to be non-isolated.[^4] + +[^4]: The reasoning here is as follows. All actor-isolated functions +are inherently `Sendable` because they will only use their captures from +an isolated context.[^5] There is only a data-race risk for the +captures of a non-`Sendable` `@isolated(any)` function in the case +where the function is dynamically non-isolated. The sendability +restrictions therefore boil down to the same restrictions we would +impose on calling a non-isolated function. A call to a non-isolated +function never crosses an isolation boundary if the function is +synchronous or if the current context is non-isolated. + +[^5]: Sending an isolated function value may cause its captures to be +*destroyed* in a different context from the function's formal isolation. +Swift pervasively assumes this is okay: copies of non-`Sendable` values +must still be managed in a thread-safe manner. This is a significant +departure from Rust, where non-`Send` values cannot necessarily be safely +managed concurrently, and it means that `Sendable` is not sufficient +to enable optimizations like non-atomic reference counting. Swift +accepts this in exchange for being more permissive, as long as the code +avoids "user-visible" data races. Note that this assumption is not new +to this proposal. + +In order for a call to an `@isolated(any)` function to be treated as +not crossing an isolation boundary, the caller must be known to have +the same isolation as the function. Since the isolation of an +`@isolated(any)` parameter is necessarily an opaque value, this would +require the caller to be declared with value-specific isolation. It +is currently not possible for a local function or closure to be +isolated to a specific value that isn't already the isolation of the +current context.[^6] The following rules lay out how `@isolated(any)` +should interact with possible future language support for functions +that are explicitly isolated to a captured value. In order to +present these rules, this proposal uses the syntax currently proposed +by the [closure isolation control pitch][isolated-captures], where +putting `isolated` before a capture makes the closure isolated to +that value. This should not be construed as accepting the terms of +that pitch. Accepting this proposal will leave most of this +section "suspended" until a feature with a similar effect is added +to the language. + +[^6]: Technically, it is possible to achieve this effect in Swift +today in a way that Swift could conceivably look through: the caller +could be a closure with an `isolated` parameter, and that closure +could be called with an expression like `fn.isolation` as the argument. +Swift could analyze this to see that the parameter has the value of +`fn.isolation` and then understand the connection between the caller's +isolation and `fn`. This would be very cumbersome, though, and it +would have significant expressivity gaps vs. an isolated-captures +feature. + +If `f` is an immutable binding of `@isolated(any)` function type, +then a call to `f` does not cross an isolation boundary if the +current context is isolated to a *derivation* of the expression +`f.isolation`. + +In the isolated captures pitch, a closure can be isolated to a specific +value by using the `isolated` modifier on an entry in its capture list. +So this question would reduce to whether that capture was initialized +to a derivation of `f.isolation`. + +An expression is a derivation of some expression form `E` if: + +- it has the exact form required by `E`; +- it is a reference to a capture or immutable binding immediately + initialized with a derivation of `E`; +- it is the result of `?` (the optional-chaining operator) or `!` + (the optional-forcing operator) applied to a derivation of `E`; or +- it is a reference to a non-optional binding (an immutable binding + initialized by a successful pattern-match which removes optionality, + such as `x` in `if let x = E`) of a derivation of `E`. + +The term *immutable binding* in the rules above means a `let` constant +or immutable (non-`inout`) parameter that is neither `weak` nor +`unowned`. The analysis ignores syntax that has no effect on the +value of an expression, such as parentheses; the exact set of cases +are the same as described in [SE-0420][generalized-isolation]. + +For example: + +```swift +func delay(operation: @isolated(any) () -> ()) { + let isolation = operation.isolation + Task { [isolated isolation] in // <-- tentative syntax from the isolated captures pitch + print("waking") + operation() // <-- does not cross an isolation barrier and so is synchronous + print("finished") + } +} +``` + +In this example, the expression `operation()` calls `operation`, +which is an immutable binding (a parameter) of `@isolated(any)` +function type. The call therefore does not cross an isolation +boundary if the calling context is isolated to a derivation of +`operation.isolation`. The calling context is the closure passed +to `Task.init`, which has an explicit `isolated` capture named +`isolation` and so is isolated to that value of that capture. +The capture is initialized with the value of the enclosing +variable `isolation`, which is an immutable binding (a `let` +constant) initialized to `operation.isolation`. As such, the +calling context is isolated to a derivation of `operation.isolation`, +so the call does not cross an isolation boundary. + +The primary intent of the rules above is simply to extend the +generalized isolation checking rules laid out in +[SE-0420][generalized-isolation] to work with an underlying +expression like `fn.isolation`. However, the rules above go +beyond the SE-0420 rules in some ways, most importantly by looking +through local `let`s. Looking through such bindings was not especially +important for SE-0420, but it is important for this proposal. In +order to keep the rules consistent, the isolation checking rules from +SE-0420 will be "rebased" on top of the rules in this proposal, +as follows: + +- When calling a function with an `isolated` parameter `calleeParam`, + if the current context also has an `isolated` parameter or capture + `callerIsolation`, the function has the same isolation as the current + context if the argument expression corresponding to `calleeParam` is + a derivation of either: + + - a reference to `callerIsolation` or + - a call to `DistributedActor.asAnyActor` applied to a derivation of + `calleeIsolation`. + +As a result, the following code is now well-formed: + +```swift +func operate(actor1: isolated MyActor) { + let actor2 = actor1 + actor2.isolatedMethod() // Swift now knows that actor2 is isolated +} +``` + +There is no reason to write this code instead of just using `actor1`, +but it's good to have consistent rules. + +### Adoption in task-creation routines + +There are a large number of functions in the standard library that create +tasks: +- `Task.init` +- `Task.detached` +- `TaskGroup.addTask` +- `TaskGroup.addTaskUnlessCancelled` +- `ThrowingTaskGroup.addTask` +- `ThrowingTaskGroup.addTaskUnlessCancelled` +- `DiscardingTaskGroup.addTask` +- `DiscardingTaskGroup.addTaskUnlessCancelled` +- `ThrowingDiscardingTaskGroup.addTask` +- `ThrowingDiscardingTaskGroup.addUnlessCancelled` + +This proposal modifies all of these APIs so that the task function has +`@isolated(any)` function type. These APIs now all synchronously enqueue +the new task directly on the appropriate executor for the task function's +dynamic isolation. + +Swift reserves the right to optimize the execution of tasks to avoid +"unnecessary" isolation changes, such as when an isolated `async` function +starts by calling a function with different isolation.[^3] In general, this +includes optimizing where the task initially starts executing: + +```swift +@MainActor class MyViewController: UIViewController { + @IBAction func buttonTapped(_ sender : UIButton) { + Task { + // This closure is implicitly isolated to the main actor, but Swift + // is free to recognize that it doesn't actually need to start there. + let image = await downloadImage() + display.showImage(image) + } + } +} +``` + +[^3]: This optimization doesn't change the formal isolation of the functions +involved and so has no effect on the value of either `#isolation` or +`.isolation`. + +As an exception, in order to provide a primitive scheduling operation with +stronger guarantees, Swift will always start a task function on the +appropriate executor for its formal dynamic isolation unless: +- it is non-isolated or +- it comes from a closure expression that is only *implicitly* isolated + to an actor (that is, it has neither an explicit `isolated` capture + nor a global actor attribute). This can currently only happen with + `Task {}`. + +As a result, in the following code, these two tasks are guaranteed +to start executing on the main actor in the order in which they were +created, even if they immediately switch away from the main actor without +having done anything that requires isolation:[^4] + +```swift +func process() async { + Task { @MainActor in + ... + } + + // do some work + + Task { @MainActor in + ... + } +} +``` + + +[^4]: This sort of guarantee is important when working with a FIFO +"pipeline", which is a common pattern when working with explicit queues. +In a pipeline, code responds to an event by performing work on a series +of queues, like so: + + ```swift + func handleEvent(event: Event) {} + queue1.async { + let x = makeX(event) + queue2.async { + let y = makeY(event) + queue3.async { + handle(x, y) + } + } + } + } + ``` + + As long as execution always goes through the exact same sequence of FIFO + queues, each queue will execute its stage of the overall pipeline in + the same order as the events were originally received. This can be a + difficult property to maintain --- concurrency at any stage will destroy + it, as will skipping any stages of the pipeline --- but it's not uncommon + for systems to be architected around it. + +The exception here to allow more optimization for implicitly-isolated +closures is an effort to avoid turning `Task {}` into a surprising +performance bottleneck. Programmers often reach for `Task {}` just to +do something concurrently with the current context, such as downloading +a file from the internet and then storing it somewhere. However, if +`Task {}` is used from an isolated context (such as from a `@MainActor` +event handler), the closure passed to `Task` will implicitly formally +inherit that isolation. A strict interpretation of the scheduling +guarantee in this proposal would require the closure to run briefly +on the current actor before it could do anything else. That would mean +that the task could never begin the download immediately; it would have +to wait, not just for the current operation on the actor to finish, but +for the actor to finish processing everything else currently in its +queue. If this is needed, it is not unreasonable to ask programmers +to state it explicitly, just as they would have to from a non-isolated +context. + +## Source compatibility + +Most of this proposal is additive. The exception is the adoption +in the standard library, which changes the types of certain API +parameters. Calls to these APIs should continue to work, as any +function that could be passed to the current parameter should also +be convertible to an `@isolated(any)` type. The observed type of +the API will change, however, if anyone does an abstract reference +such as `Task.init`. Contravariant conversion should allow these +unapplied references to work in any concrete type context that +would accept the current function, but references in other contexts +can lead to source breaks (such as `var fn = Task.init`). This is +unlikely to be an issue in practice. More importantly, I believe +Swift has a general policy of declining to guarantee stable types +for unapplied function references in the standard library this way. +Doing so would prevent a wide variety of reasonable code evolution +for the library, such as generalizing the type of a parameter (as +this proposal does) or adding a new defaulted parameter. + +## ABI compatibility + +This feature does not change the ABI of any existing code. + +## Implications on adoption + +The basic functionality of `@isolated(any)` function types is +implemented directly in generated code and does not require runtime +support. + +Using a type as a generic argument generally requires runtime type +metadata support for the type. For `@isolated(any)` function types, +that metadata support requires a new Swift runtime. It will therefore +not possible to use a type such as `[@isolated(any) () -> ()]` when +back-deploying code on a platform with ABI stability. However, +wrapping the function in a `struct` with a single field will generally +work around this problem. (It also generally allows the function to +be stored more efficiently.) + +The task-creation APIs in the standard library have been implemented +in a way that allows their signatures to be changed without ABI +considerations. Direct enqueuing on the isolated actor does require +runtime support, but fortunately that support has present in the +concurrency runtime since the first release. Therefore, there should +not be any back-deployment problems supporting the proposed changes. + +Adopters of `@isolated(any)` function types will generally face the +same source-compatibility considerations as this proposal does with +the task-creation APIs: it requires generalizing some parameter types, +which generally should not cause incompatibilities with direct callers +but can introduce problems in the somewhat unlikely case that anyone +is using those function as values. + +### When to use `@isolated(any)` + +It is recommended that APIs which take functions that are likely to run +concurrently and don't have a predetermined isolation take those functions +as `@isolated(any)`. This allows the API to make more intelligent +scheduling decisions about the function. + +Examples that should usually use `@isolated(any)` include: +- functions that wrap the creation of a task +- algorithms that call a function multiple times in parallel, such as a + parallel `map` + +Examples that should usually not use `@isolated(any)` include: +- algorithms that preserve the current isolation, such as a non-parallel + `map`; these functions should usually take a non-`Sendable` function + instead +- APIs that intend to call the function with a specific isolation, such + as UI frameworks that expect their event handlers to be `@MainActor` + or actor functions that run an operation on the actor + +## Future directions + +### Interaction with `assumeIsolated` + +It would be convenient in some cases to be able to assert that the +current synchronous context is already isolated to the isolation of +an `@isolated(any)` function, allowing the function to be called without +crossing isolation. Similar functionality is provided by the +`assumeIsolated` function introduced by [SE-0392][SE-0392]. +Unfortunately, the current `assumeIsolated` function is inadequate +for this purpose for several reasons. + +The first problem is that `assumeIsolated` only works on a +non-optional actor reference. We could add a version of this API +which does work on optional actors, but it's not clear what it +should actually do if given a `nil` reference. A `nil` isolation +represents non-isolation, which of course does not actually isolate +anything. Should `assumeIsolated` check that the current context +has exactly the given isolation, or should it check that it is safe +to use something with the given isolation requirement from the current +context? The first rule is probably the one that most people would +assume when they first heard about the feature. However, it implies +that `assumeIsolated(nil)` should check that no actors are currently +isolated, and that is not something we can feasibly check in general: +Swift's concurrency runtime does track the current isolation of a task, +but outside of a task, arbitrary things can be isolated without Swift +knowing about them. It is also needlessly restrictive, because there +is nothing that is unsafe to do in an isolated context that would be +safe if done in a non-isolated context.[^7] The second rule is less +intuitive but more closely matches the safety properties that static +isolation checking tests for. It implies that `assumeIsolated(nil)` +should always succeed. This is notably good enough for `@isolated(any)`: +since `assumeIsolated` is a synchronous function, only synchronous +`@isolated(any)` functions can be called within it, and calling a +synchronous non-isolated function always runs immediately without +changing the current isolation. + +[^7]: As far as data-race safety goes, at least. A specific actor +could conceivably have important semantic restrictions against doing +certain operations in its isolated code. Of course, such an actor should +generally not be calling arbitrary functions that are handed to it. + +The second problem is that `assumeIsolated` does not currently establish +a link back to the original expression passed to it. Code such as +the following is invalid: + +```swift +myActor.assumeIsolated { + myActor.property += 1 // invalid: Swift doesn't know that myActor is isolated +} +``` + +The callback passed to `assumeIsolated` is isolated because it takes +an `isolated` parameter, and while this parameter is always bound to +the actor that `assumeIsolated` was called on, Swift's isolation checking +doesn't know that. As a result, it is necessary to use the parameter +instead of the original actor reference, which is a persistent annoyance +when using this API: + +```swift +myActor.assumeIsolated { myActor2 in + myActor2.property += 1 +} +``` + +For `@isolated(any)`, we would naturally want to write this: + +```swift +myFn.isolation.assumeIsolated { + myFn() +} +``` + +However, since Swift doesn't understand the connection between the +closure's `isolated` parameter and `myFn`, this call will not work, +and there is no way to make it work. + +One way to fix this would be to add some new way to assert that an +`@isolated(any)` function is currently isolated. This could even +destructure the function value, giving the program access it to as +a non-`@isolated(any)` function. But it seems like a better approach +to allow isolation checking to understand that the `isolated` parameter +and the `self` argument of `assumeIsolated` are the same value. +That would fix both the usability problem with actors and the +expressivity problem with `@isolated(any)`. Decomposition could +be done as a general rule that permits isolation to be removed from +a function value as long as that isolation matches the current +context and the resulting function is non-`Sendable`. + +This is all sufficiently complex that it seems best to leave it for +a future direction. However, it should be relatively approachable. + +### Statically-isolated function types + +`@isolated(any)` function types are effectively an "existential +erasure" of the isolation of the function, removing the type system's +static knowledge of the isolation while dynamically preserving it. +This is directly analogous to how `Any` erases the type of the value +you store into it: the type system no longer knows statically what +type is stored there, but it's still possible to recover it dynamically. +This analogy is why this proposal uses the keyword `any` in the +attribute name. + +Where there's an existential, there's also a generic. The generic +analogue to `@isolated(any)` would be a type that expressed that it +was isolated to a specific value, like so: + +```swift +func delay(on operationActor: A, + operation: @isolated(to: operationActor) () async -> ()) +``` + +This is a kind of value-dependent type. Value-dependent types add a +lot of complexity to a type system. Consider how the arguments interact +in the example above: both value and type information from the first +argument flows into the second. This is not something to do lightly, +and we think Swift is relatively unlikely to ever add such a feature +as `@isolated(to:)`. + +Fortunately, it is unlikely to be necessary. We believe that +`@isolated(any)` function types are superior from a usability perspective +for all the dominant patterns of higher-order APIs. The main thing that +`@isolated(to:)` can express in an API signature that `@isolated(any)` +cannot is multiple functions that share a common isolation. It is +quite uncommon for APIs to take multiple closely-related functions +this way, especially `@Sendable` functions where there's an expected +isolation change from the current context. When only a single function +is required in an API, `@isolated(any)` allows its isolation to bound +up with it in a single value, which is both more convenient and likely +to have a more performant representation. + +If Swift ever does explore in the direction of `@isolated(to:)`, +nothing in this proposal would interfere with it. In fact, the +features would support each other well. Erasing the isolation of +an `@isolated(to:)` function into an `@isolated(any)` type would +be straightforward, much like erasing an `Int` into an `Any`. +Similarly, an `@isolated(any)` function could be "opened" into a +pair of an `@isolated(to:)` function and its known isolation. +Since the common cases will still be more convenient to express +with `@isolated(any)`, the community is unlikely to regret having +added this proposal first. + +## Alternatives considered + +### Other spellings + +`isolated` and `nonisolated` are used as bare-word modifiers in several +places already in Swift: you can declare a parameter as `isolated`, and +you can declare methods and properties as `nonisolated`. Using `@isolated` +as a function type attribute therefore risks confusion about whether +`isolated` should be written with an `@` sign. + +One alternative would be to drop the `@` sign and spell these function +types as e.g. `isolated(any) () -> ()`. However, this comes with its own +problems. Modifiers typically affect a specific entity without changing +its type; for example, the `weak` modifier makes a variable or property +a weak reference, but the type of that reference is unchanged (although +it is required to be optional). This wouldn't be too confusing if +modifiers and types were written in fundamentally different places, but +it's expected that `@isolated(any)` will usually be used on parameter +functions, and parameter modifiers are written immediately adjacent to +the parameter type. As a result, removing the `@` would create this +unfortunate situation: + +```swift +// This means `foo` is isolated to the actor passed in as `actor`. +func foo(actor: isolated MyActor) {} + +// This means `operation` is a value of isolated(any) function type; +// it has no impact on the isolation of `bar`. +func bar(operation: isolated(any) () -> ()) +``` + +It is better to preserve the current rule that type modifiers are +written with an `@` sign. + +Another alternative would be to not spell the attribute `@isolated(any)`. +For example, it could be spelled `@anyIsolated` or `@dynamicallyIsolated`. +The spelling `@isolated(any)` was chosen because there's an expectation +that this will be one of a family of related isolation-specifying +attributes. For example, if Swift wanted to make it easier to inherit +actor isolation from one's caller, it could add an `@isolated(caller)` +attribute. Another example is the `@isolated(to:)` future direction +listed above. There's merit in having these attributes be closely +related in spelling. Using a common `Isolated` suffix could serve as +that connection, but in the author's opinion, `@isolated` is much +clearer. + +If programmers do end up confused about when to use `@` with `isolated`, +it should be relatively straightforward to provide a good compiler +experience that corrects misuses. + +### Implying `@Sendable` + +An earlier version of this proposal made `@isolated(any)` imply `@Sendable`. +The logic behind this implication was that `@isolated(any)` is only +really useful if the function is going to be passed to a different +concurrent context. If a function cannot be passed to a different +concurrent context, the reasoning goes, there's really no point in +it carrying its isolation dynamically, because it can only be used +if that isolation is compatible with the current context. There's +therefore no reason not to eliminate the redundant `@Sendable` attribute. + +However, this logic subtly misunderstands the meaning of `Sendable` +in a world with [region-based isolation][regions]. A type conforming +to `Sendable` means that its values are intrinsically thread-safe and +can be used from multiple concurrent contexts *concurrently*. +Values of non-`Sendable` type are still safe to use from different +concurrent contexts as long as those uses are well-ordered: if the +value is properly [transferred][region-transfers] between contexts, +everything is fine. Given that, it is sensible for a non-`Sendable` +function to be `@isolated(any)`: if the function can be transferred +to a different concurrent context, it's still useful for it to carry +its isolation dynamically. + +In particular, something like a task-creation function ought to declare +the initial task function as a non-`@Sendable` but still transferrable +`@isolated(any)` function. This permits closures passed in to capture +non-`Sendable` state as long as that state can be transferred into the +closure. (Ideally, the initial task function would then be able to +transfer that captured state out of the closure. However, this would +require the compiler to understand that the task function is only +called once.) + +## Acknowledgments + +I'd like to thank Holly Borla and Konrad Malawski for many long +conversations about the design and implementation of this feature. diff --git a/proposals/0432-noncopyable-switch.md b/proposals/0432-noncopyable-switch.md new file mode 100644 index 0000000000..5eb6171845 --- /dev/null +++ b/proposals/0432-noncopyable-switch.md @@ -0,0 +1,324 @@ +# Borrowing and consuming pattern matching for noncopyable types + +* Proposal: [SE-0432](0432-noncopyable-switch.md) +* Authors: [Joe Groff](https://github.com/jckarter) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.0)** +* Experimental Feature Flag: `BorrowingSwitch` +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/86cf6eadcdb35a09eb03330bf5d4f31f2599da02/proposals/ABCD-noncopyable-switch.md) +* Review: ([review](https://forums.swift.org/t/se-0432-borrowing-and-consuming-pattern-matching-for-noncopyable-types/71158)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0432-borrowing-and-consuming-pattern-matching-for-noncopyable-types/71656)) + +## Introduction + +Pattern matching over noncopyable types, particularly noncopyable enums, can +be generalized to allow for pattern matches that borrow their subject, in +addition to the existing support for consuming pattern matches. + +## Motivation + +[SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md) +introduced noncopyable types, allowing for programs to define +structs and enums whose values cannot be copied. However, it restricted +`switch` over noncopyable values to be a `consuming` operation, meaning that +nothing can be done with a value after it's been matched against. This +severely limits the expressivity of noncopyable enums in particular, +since switching over them is the only way to access their associated values. + +## Proposed solution + +We lift the restriction that noncopyable pattern matches must consume their +subject value, and formalize the ownership behavior of patterns during +matching and dispatch to case blocks. `switch` statements **infer their +ownership behavior** based on a combination of whether the subject expression +refers to storage or a temporary value, in addition to the necessary ownership +behavior of the patterns in the `switch`. We also introduce **borrowing +bindings** into patterns, as a way of explicitly declaring a binding as a +borrow that doesn't allow for implicit copies. + +## Detailed design + +### Determining the ownership behavior of a `switch` operation + +Whether a `switch` borrows or consumes its subject can be determined from +the subject expression and the patterns involved in the switch. Based on +the criteria below, a switch may be one of: + +- **copying**, meaning that the subject is semantically copied, and additional + copies of some or all of the subject value may be formed to execute the + pattern match. +- **borrowing**, meaning that the subject is borrowed for the duration of the + `switch` block. +- **consuming**, meaning that the subject is consumed by the `switch` block. + +These modes can be thought of as being increasing in strictness. The compiler +looks recursively through the patterns in the `switch` and increases the +strictness of the `switch` behavior when it sees a pattern requiring stricter +ownership behavior. For copyable subjects, *copying* is the baseline mode, +whereas for noncopyable subjects, the baseline mode depends on the subject +expression: + +- If the expression refers to a variable or stored property, and is not + explicitly consumed using the `consume` operator, then the baseline + mode is *borrowing*. (Properties and subscripts which use the experimental + `_read`, `_modify`, or `unsafeAddress` accessors also get a baseline mode + of borrowing.) +- Otherwise, the baseline mode is *consuming*. + +For example, given the following copyable definition: + +```swift +enum CopyableEnum { + case foo(Int) + case bar(Int, String) +} +``` + +then the following patterns have ownership behavior as indicated below: + +```swift +case let x: // copying +case .foo(let x): // copying +case .bar(let x, let y): // copying +``` + +And for a noncopyable enum definition: + +```swift +struct NC: ~Copyable {} + +enum NoncopyableEnum: ~Copyable { + case copyable(Int) + case noncopyable(NC) +} +``` + +then the following patterns have ownership behavior as indicated below: + +```swift +var foo: NoncopyableEnum // stored variable + +switch foo { +case let x: // borrowing + +case .copyable(let x): // borrowing (because `x: Int` is copyable) + +case .noncopyable(let x): // borrowing +} + +func bar() -> NoncopyableEnum {...} // function returning a temporary + +switch bar() { +case let x: // consuming +case .copyable(let x): // borrowing (because `x: Int` is copyable) +case .noncopyable(let x): // consuming +} +``` + +### Refining the ownership behavior of `switch` + +The order in which `switch` patterns are evaluated is unspecified in Swift, +aside from the property that when multiple patterns can match a value, +the earliest matching `case` condition takes priority. Therefore, it is +important that matching dispatch **cannot mutate or consume the subject** +until a final match has been chosen. For copyable values, this means that +pattern matching operations can't mutate the subject, but they can be copied +as necessary to keep an instance of the subject available throughout the +pattern match even if a match operation wants to consume an instance of +part of the value. + +Copying isn't an option for noncopyable types, so +**noncopyable types strictly cannot undergo `consuming` operations until +the pattern match is complete**. For many kinds of pattern matches, this +doesn't need to affect their expressivity, since checking whether a type +matches the pattern criteria can be done nondestructively separate from +consuming the value to form variable bindings. Matching enum cases and tuples +(when noncopyable tuples are supported) for instance is still possible +even if they contain consuming `let` or `var` bindings as subpatterns: + +```swift +extension Handle { + var isReady: Bool { ... } +} + +let x: MyNCEnum = ... +switch consume x { +// OK to have `let y` in multiple patterns because we can delay consuming +// `x` to form bindings until we establish a match +case .foo(let y) where y.isReady: + y.close() +case .foo(let y): + y.close() +} +``` + +However, when a pattern has a `where` clause, variable bindings cannot be +consumed in the `where` clause even if the binding is consumable in the `case` +body: + +```swift +extension Handle { + consuming func tryClose() -> Bool { ... } +} + +let x: MyNCEnum = ... +switch consume x { +// error: cannot consume `y` in a "where" clause +case .foo(let y) where y.tryClose(): + // OK to consume in the case body + y.close() +case .foo(let y): + y.close() +} +``` + +Similarly, an expression subpattern whose `~=` operator consumes the subject +cannot be used to test a noncopyable subpattern. + +```swift +extension Handle { + static func ~=(identifier: Int, handle: consuming Handle) -> Bool { ... } +} + +switch consume x { +// error: uses a `~=` operator that would consume the subject before +// a match is chosen +case .foo(42): + .... +case .foo(let y): + ... +} +``` + +Noncopyable types do not yet support dynamic casting, but it is worth +anticipating how `is` and `as` patterns will work given this restriction. +An `is T` pattern only needs to determine whether the value being matched can +be cast to `T` or not, which can generally be answered nondestructively. +However, in order to form the value of type `T`, many kinds of casting, +including casts that bridge or which wrap the value in an existential +container, need to consume or copy parts of the input value in order to form +the result. The cast can still be separated into a check whether the type +matches, using a borrowing access, followed by constructing the actual cast +result by consuming if necessary. To do this, the switch would have already +be a consuming switch. But also, for a consuming `as T` pattern to work, the +subpattern `p` of the `p as T` pattern would need to be irrefutable, and the +pattern could not have an associated `where` clause, since we would be unable +to back out of the pattern match once a consuming cast is performed. + +### `case` conditions in `if`, `while`, `for`, and `guard` + +Patterns can also appear in `if`, `while`, `for`, and `guard` forms as part +of `case` conditions, such as `if case = { }`. These behave +just like `switch`es with one `case` containing the pattern, corresponding +to a true condition result with bindings, and a `default` branch corresponding +to a false condition result. Therefore, the ownership behavior of the `case` +condition on the subject follows the behavior of that one pattern. + +## Source compatibility + +SE-0390 explicitly required that a `switch` over a noncopyable variable +use the `consume` operator. This will continue to work in most cases, forcing +the lifetime of the binding to end regardless of whether the `switch` actually +consumes it or not. In some cases, the formal lifetime of the value or parts +of it may end up different than the previous implementation, but because +enums cannot yet have `deinit`s, noncopyable tuples are not yet supported, +and structs with `deinit`s cannot be partially destructured and must be +consumed as a whole, it is unlikely that this will be noticeable in real +world code. + +Previously, it was theoretically legal for noncopyable `switch`es to use +consuming `~=` operators, or to consume pattern bindings in the `where` +clause of a pattern. This proposal now expressly forbids these formulations. +We believe it is impossible to exploit these capabilities in practice under the +old implementation, since doing so would leave the value partially or fully +consumed on the failure path where the `~=` match or `where` clause fails, +leading to either mysterious ownership error messages, compiler crashes, or +both. + +## ABI compatibility + +This proposal has no effect on ABI. + +## Future directions + +### `inout` pattern matches + +With this proposal, pattern matches are able to *borrow* and *consume* their +subjects, but they still aren't able to take exclusive `inout` access to a +value and bind parts of it for in-place mutation. This proposal lays the +groundwork for supporting this in the future; we could introduce `inout` +bindings in patterns, and introducing **mutating** switch behavior as a level +of ownership strictness between *borrowing* and *consuming*. + +### Automatic borrow deduction for `let` bindings, and explicitly `consuming` bindings + +When working with copyable types, although `let` and `var` bindings formally +bind independent copies of their values, in cases where it's semantically +equivalent, the compiler optimizes aways the copy and borrows the original +value in place, with the idea that developers do not need to think about +ownership if the compiler does an acceptable job of optimizing their code. +By similar means, we could say that `let` pattern bindings for noncopyable types +borrow rather than consume their binding automatically if the binding is +not used in a way that requires it to consume the binding. This would +give developers a "do what I mean" model for noncopyable types closer to the +convenience of copyable types. This should be a backward compatible change +since it would allow for strictly more code to compile than does currently +when `let` bindings are always consuming. + +Conversely, performance-minded developers would also like to have explicit +control over ownership behavior and copying, while working with either +copyable or noncopyable types. To that end, we could add explicitly `consuming` +bindings to patterns as well, which would not be implicitly copyable, and +which would force the switch behavior mode on the subject to become *consuming* +even if the subject is copyable. + +### enum `deinit` + +SE-0390 left `enum`s without the ability to have a `deinit`, based on the fact +that the initial implementation of noncopyable types only supported consuming +`switch`es. Noncopyable types with `deinit`s generally cannot be decomposed, +since doing so would bypass the `deinit` and potentially violate invariants +maintained by `init` and `deinit` on the type, so an `enum` with a `deinit` +would be completely unusable when the only primitive operation supported on it +is consuming `switch`. Now that this proposal allows for `borrowing` switches, +we could allow `enum`s to have `deinit`s, with the restriction that such +enums cannot be decomposed by a consuming `switch`. + +### Explicit `borrow` operator + +The [`borrow` operator](https://forums.swift.org/t/selective-control-of-implicit-copying-behavior-take-borrow-and-copy-operators-noimplicitcopy/60168) +could be used in the future to explicitly mark the subject of a switch as +being borrowed, even if it is normally copyable or would be a consumable +temporary, as in: + +```swift +let x: String? = "hello" + +switch borrow x { +case .some(let y): // ensure y is bound from a borrow of x, no copies + ... +} +``` + +### `borrowing` bindings in patterns + +In the future, we want to support `borrowing` and `inout` local bindings +in functions and potentially even as fields in nonescapable types. It might +also be useful to specify explicitly `borrowing` bindings within patterns. +Although the default behavior for a `let` binding within a noncopyable +borrowing `switch` pattern is to borrow the matched value, an explicitly +`borrowing` binding could be used to indicate that a copyable binding should +have its local implicit copyability suppressed, like a `borrowing` parameter +binding. + +## Alternatives considered + +### Determining pattern match ownership wholly from patterns + +The [first pitched revision](https://github.com/swiftlang/swift-evolution/blob/86cf6eadcdb35a09eb03330bf5d4f31f2599da02/proposals/ABCD-noncopyable-switch.md) +of this proposal kept `let` bindings in patterns as always being consuming +bindings, and required the use of `borrowing` bindings in every pattern in order +for a `switch` to act as a borrow. Early feedback using the feature found this +tedious; `borrowing` is more often a better default for accessing values +stored in variables and stored properties. This led us to the design now +proposed, where `let` behaves as a copying, consuming, or borrowing binding +based on the subject expression. diff --git a/proposals/0433-mutex.md b/proposals/0433-mutex.md new file mode 100644 index 0000000000..cdb7946574 --- /dev/null +++ b/proposals/0433-mutex.md @@ -0,0 +1,351 @@ +# Synchronous Mutual Exclusion Lock 🔒 + +* Proposal: [SE-0433](0433-mutex.md) +* Author: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Stephen Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-synchronous-mutual-exclusion-lock/69889)), ([review](https://forums.swift.org/t/se-0433-synchronous-mutual-exclusion-lock/71174)), ([acceptance](https://forums.swift.org/t/accepted-se-0433-synchronous-mutual-exclusion-lock/71463)) + +## Introduction + +This proposal introduces a mutual exclusion lock, or a mutex, to the standard library. `Mutex` will be a new synchronization primitive in the synchronization module. + +## Motivation + +In concurrent programs, protecting shared mutable state is one of the core fundamental problems to ensuring reading and writing data is done in an explainable fashion. Synchronizing access to shared mutable state is not a new problem in Swift. We've introduced many features to help protect mutable data. Actors are a good default go-to solution for protecting mutable state because it isolates the stored data in its own domain. At any given point in time, only one task will be executing "on" the actor, and have exclusive access to it. Multiple tasks cannot access state protected by the actor at the same time, although they may interleave execution at potential suspension points (indicated by `await`). In general, the actor approach also lends itself well to code organization, since the actor's state, and operations on this state are logically declared in the same place: inside the actor. + +Not all code may be able (or want) to adopt actors. Reasons for this can be very varied, for example code may have to execute synchronously without any potential for other tasks interleaving with it. Or the `async` effect introduced on methods may prevent legacy code which cannot use Swift Concurrency from interacting with the protected state. + +Whatever the reason may be, it may not be feasible to use an actor. In such cases, Swift currently is missing standard tools for developers to ensure proper synchronization in their concurrent data-structures. Many Swift programs opt to use ad-hoc implementations of a mutual exclusion lock, or a mutex. A mutex is a simple to use synchronization primitive to help protect shared mutable data by ensuring that a single execution context has exclusive access to the related data. The main issue is that there isn't a single standardized implementation for this synchronization primitive resulting in everyone needing to roll their own. + +## Proposed solution + +We propose a new type in the Standard Library Synchronization module: `Mutex`. This type will be a wrapper over a platform-specific mutex primitive, along with a user-defined mutable state to protect. Below is an example use of `Mutex` protecting some internal data in a class usable simultaneously by many threads: + +```swift +class FancyManagerOfSorts { + let cache = Mutex<[String: Resource]>([:]) + + func save(_ resource: Resource, as key: String) { + cache.withLock { + $0[key] = resource + } + } +} +``` + +Use cases for such a synchronized type are common. Another common need is a global cache, such as a dictionary: + +```swift +let globalCache = Mutex<[MyKey: MyValue]>([:]) +``` + +### Toolchains + +You can try out `Mutex` using one of the following toolchains: + +macOS: https://ci.swift.org/job/swift-PR-toolchain-macos/1207/artifact/branch-main/swift-PR-71383-1207-osx.tar.gz + +Linux (x86_64): https://download.swift.org/tmp/pull-request/71383/779/ubuntu2004/PR-ubuntu2004.tar.gz + +Windows: `https://ci-external.swift.org/job/swift-PR-build-toolchain-windows/1200/artifact/*zip*/archive.zip` + +Note that these toolchains don't currently have the `transferring inout` implemented, but the functions are marked `@Sendable` to at least enforce sendability. + +## Detailed design + +### Underlying System Mutex Implementation + +The `Mutex` type proposed is a wrapper around a platform's implementation. + +* macOS, iOS, watchOS, tvOS, visionOS: + * `os_unfair_lock` +* Linux: + * `futex` +* Windows: + * `SRWLOCK` + +These mutex implementations all have different capabilities and guarantee different levels of fairness. Our proposed `Mutex` type does not guarantee fairness, and therefore it's okay to have different behavior from platform to platform. We only guarantee that only one execution context at a time will have access to the critical section, via mutual exclusion. + +### API Design + +Below is the complete API design for the new `Mutex` type: + +```swift +/// A synchronization primitive that protects shared mutable state via +/// mutual exclusion. +/// +/// The `Mutex` type offers non-recursive exclusive access to the state +/// it is protecting by blocking threads attempting to acquire the lock. +/// At any one time, only one execution context at a time has access to +/// the value stored within the `Mutex` allowing for exclusive access. +/// +/// An example use of `Mutex` in a class used simultaneously by many +/// threads protecting a `Dictionary` value: +/// +/// class Manager { +/// let cache = Mutex<[Key: Resource]>([:]) +/// +/// func saveResource(_ resource: Resource, as key: Key) { +/// cache.withLock { +/// $0[key] = resource +/// } +/// } +/// } +/// +/// - Warning: Instances of this type are not recursive. Calls +/// to `withLock(_:)` (and related functions) within their +/// closure parameters will have platform-dependent behavior. +/// Some platforms may choose to panic the process, deadlock, +/// or leave this behavior unspecified. +/// +public struct Mutex: ~Copyable { + /// Initializes an instance of this mutex with the given initial state. + /// + /// - Parameter state: The initial state to give to the mutex. + public init(_ state: transferring consuming State) +} + +extension Mutex: Sendable where State: ~Copyable {} + +extension Mutex where State: ~Copyable { + /// Calls the given closure after acquiring the lock and then releases + /// ownership. + /// + /// This method is equivalent to the following sequence of code: + /// + /// mutex.lock() + /// defer { + /// mutex.unlock() + /// } + /// return try body(&value) + /// + /// - Warning: Recursive calls to `withLock` within the + /// closure parameter has behavior that is platform dependent. + /// Some platforms may choose to panic the process, deadlock, + /// or leave this behavior unspecified. This will never + /// reacquire the lock however. + /// + /// - Parameter body: A closure with a parameter of `State` + /// that has exclusive access to the value being stored within + /// this mutex. This closure is considered the critical section + /// as it will only be executed once the calling thread has + /// acquired the lock. + /// + /// - Returns: The return value, if any, of the `body` closure parameter. + public borrowing func withLock( + _ body: (transferring inout State) throws(E) -> transferring Result + ) throws(E) -> transferring Result + + /// Attempts to acquire the lock and then calls the given closure if + /// successful. + /// + /// If the calling thread was successful in acquiring the lock, the + /// closure will be executed and then immediately after it will + /// release ownership of the lock. If we were unable to acquire the + /// lock, this will return `nil`. + /// + /// This method is equivalent to the following sequence of code: + /// + /// guard mutex.tryLock() else { + /// return nil + /// } + /// defer { + /// mutex.unlock() + /// } + /// return try body(&value) + /// + /// - Warning: Recursive calls to `withLockIfAvailable` within the + /// closure parameter has behavior that is platform dependent. + /// Some platforms may choose to panic the process, deadlock, + /// or leave this behavior unspecified. This will never + /// reacquire the lock however. + /// + /// - Parameter body: A closure with a parameter of `State` + /// that has exclusive access to the value being stored within + /// this mutex. This closure is considered the critical section + /// as it will only be executed if the calling thread acquires + /// the lock. + /// + /// - Returns: The return value, if any, of the `body` closure parameter + /// or nil if the lock couldn't be acquired. + public borrowing func withLockIfAvailable( + _ body: (transferring inout State) throws(E) -> transferring Result? + ) throws(E) -> transferring Result? +} +``` + +## Interaction with Existing Language Features + +`Mutex` will be decorated with the `@_staticExclusiveOnly` attribute, meaning you will not be able to declare a variable of type `Mutex` as `var`. These are the same restrictions imposed on the recently accepted `Atomic` and `AtomicLazyReference` types. Please refer to the [Atomics proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0410-atomics.md) for a more in-depth discussion on what is allowed and not allowed. These restrictions are enabled for `Mutex` for all of the same reasons why it was restricted for `Atomic`. We do not want to introduce dynamic exclusivity checking when accessing a value of `Mutex` as a class stored property for instance. + +### Interactions with Swift Concurrency + +`Mutex` is unconditionally `Sendable` regardless of the value it's protecting. We can ensure the safetyness of this value due to the `transferring` marked parameters of both the initializer and the closure `inout` argument. (Please refer to [SE-0430 `transferring` isolation regions of parameter and result values](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0430-transferring-parameters-and-results.md)) This allows us to statically determine that the non-sendable value we're initializing the mutex with will have no other uses after initialization. Within the closure body, it ensures that if we tried to escape the protected non-sendable value, it would require us to replace the stored value for the notion of transferring out and then transferring back in. + +Consider the following example of a mutex to a non-sendable class. + +```swift +class NonSendableReference { + var prop: UnsafeMutablePointer +} + +// Some non-sendable class reference somewhere, perhaps a global. +let nonSendableRef = NonSendableReference(...) + +let lockedPointer = Mutex>(...) + +func something() { + lockedPointer.withLock { + // error: isolated parameter transferred out + // but hasn't had a value transferred back in. + nonSendableRef.prop = $0 + } +} +``` + +Had this closure not been marked `transferring inout` or perhaps `@Sendable`, then `Mutex` would not have protected this class references or any underlying memory referenced by pointers. Transferring inout allows us to plug this safety hole of shared mutable state by statically requiring that if we need to escape the closure or `Mutex` isolation domain as a whole, that we transfer in a new value in this domain (or it could be the same value perhaps). + +```swift +func something() { + lockedPointer.withLock { + // OK because we assign '$0' to a new value + nonSendableRef.prop = $0 + + // OK because we're transferring a new value in + $0 = ... + } +} +``` + +By marking the closure as such, we've effectively declared that the mutex is in itself its own isolation domain. We must not let non-sendable values it holds onto be unsafely sent across isolation domains to prevent these holes of shared mutable state. + +### Differences between mutexes and actors + +The mutex type we're proposing is a synchronous lock. This means when other participants want to acquire the lock to access the protected shared data, they will halt execution until they are able to do so. Threads that are waiting to acquire the lock will not be able to make forward progress until their request to acquire the lock has completed. This can lead to thread contention if the acquired thread's critical section is not able to be executed relatively quickly exhausting resources for the rest of the system to continue making forward progress. Synchronous locks are also prone to deadlocks (which Swift's actors cannot currently encounter due to their re-entrant nature) and live-locks which can leave a process in an unrecoverable state. These scenarios can occur when there is a complex hierarchy of different locks that manage to depend on the acquisition of each other. + +Actors work very differently. Typical use of an actor doesn't request access to underlying shared data, but rather instruct the actor to perform some operation or service that has exclusive access to that data. An execution context making this request may need to await on the return value of that operation, but with Swift's `async`/`await` model it can immediately start doing other work allowing it to make forward progress on other tasks. The actor executes requests in a serial fashion in the order they are made. This ensures that the shared mutable state is only accessed by the actor. Deadlocks are not possible with the actor model. Asynchronous code that is dependent on a specific operation and resource from an actor can be later resumed once the actor has serviced that request. While deadlocking is not possible, there are other problems actors have such as the actor reentrancy problem where the state of the actor has changed when the executing operation got resumed after a suspension point. + +Mutexes and actors are very different synchronization tools that help protect shared mutable state. While they can both achieve synchronization of that data access, they do so in varying ways that may be desirable for some and undesirable for others. The proposed `Mutex` is yet another primitive that Swift should expose to help those achieve concurrency safe programs in cases where actors aren't suitable. + +## Source compatibility + +Source compatibility is preserved with the proposed API design as it is all additive as well as being hidden behind an explicit `import Synchronization`. Users who have not already imported the Synchronization module will not see this type, so there's no possibility of potential name conflicts with existing `Mutex` named types for instance. Of course, the standard library already has the rule that any type names that collide will disfavor the standard library's variant in favor of the user's defined type anyway. + +## ABI compatibility + +The API proposed here is fully addative and does not change or alter any of the existing ABI. + +`Mutex` as proposed will be a new `@frozen` struct which means we cannot change its layout in the future on ABI stable platforms, namely the Darwin family. Because we cannot change the layout, we will most likely not be able to change to a hypothetical new and improved system mutex implementation on those platforms. If said new system mutex were to share the layout of the currently proposed underlying implementation, then we _may_ be able to migrate over to that implementation. Keep in mind that Linux and Windows are non-ABI stable platforms, so we can freely change the underlying implementation if the platform ever supports something better. + +## Future directions + +There are quite a few potential future directions this new type can take as well as new future similar types. + +### Mutex Guard API + +A token based approach for locking and unlocking may also be highly desirable for mutex API. This is similar to C++'s `std::lock_guard` or Rust's `MutexGuard`: + +```swift +extension Mutex { + public struct Guard: ~Copyable, ~Escapable { + // Hand waving some syntax to borrow Mutex, or perhaps + // we just store a pointer to it. + let mutex: borrow Mutex + + public var value: Value {...} + + deinit { + mutex.unlock() + } + } +} + +extension Mutex { + public borrowing func lock() -> borrow(self) Guard {...} + + public borrowing func tryLock() -> borrow(self) Guard? {...} +} + +func add(_ i: Int, to mutex: Mutex) { + // This acquires the lock by calling the platform's + // underlying 'lock()' primitive. + let mGuard = mutex.lock() + + mGuard.value += 1 + + // At the end of the scope, mGuard is immediately deinitialized + // and releases the mutex by calling the platform's + // 'unlock()' primitive. +} +``` + +The above example shows an API similar to Rust's `MutexGuard` which allows access to the protected state in the mutex. C++'s guard on the other hand just performs `lock()` and `unlock()` for the user (because `std::mutex` doesn't protect any state). Of course the immediate issue with this approach right now is that we don't have access to non-escapable types. When one were to call `lock()`, there's nothing preventing the user from taking the guard value and escaping it from the scope that the caller is in. Rust resolves this issue with lifetimes, but C++ doesn't solve this at all and just declares: + +> The behavior is undefined if m is destroyed before the `lock_guard` object is. + +Which is not something we want to introduce in Swift if it's something we can eventually prevent. If we had this feature today, the primitive `lock()`/`unlock()` operations would be better suited in the form of the guard API. I don't believe we'd have those methods if we had guards. + +### Reader-Writer Locks, Recursive Locks, etc. + +Another interesting future direction is the introduction of new kinds of locks to be added to the standard library, such as a reader-writer lock. One of the core issues with the proposal mutual exclusion lock is that anyone who takes the lock, either a reader and/or writer, must be the only person with exclusive access to the protected state. This is somewhat unfortunate for models where there are infinitely more readers than there will be writers to the state. A reader-writer lock resolves this issue by allowing multiple readers to take the lock and enforces that writers who need to mutate the state have exclusive access to the value. Another potential lock is a recursive lock who allows the lock to be acquired multiple times by the acquired thread. In the same vein, the acquired thread needs to be the one to release the lock and needs to release X amount of times equal to the number of times it acquired it. + +## Alternatives considered + +### Implement `lock()`, `unlock()`, and `tryLock()` + +Seemingly missing from the `Mutex` type are the primitive locking and unlocking functions. These functions are fraught with peril in both Swift's concurrency model and in its ownership model. + +In the face of `async`/`await`, these primitives are very dangerous. The example below highlights incorrect usage of these operations in an `async` function: + +```swift +func test() async { + mutex.lock() // Called on Thread A + await downloadImage(...) // <--- Potential suspension point + mutex.unlock() // May be called on Thread A, B, C, etc. +} +``` + +The potential suspension point may cause the proceeding code to be called on a different thread than the one that initiated the `await` call. We can make these primitives safe in asynchronous contexts though by disallowing their use altogether by marking them `@available(*, noasync)`. Calling `withLock` in an asynchronous function is \_okay\_ because the same thread that calls `lock()` will be the same one that calls `unlock()` because there will not be any suspension points between the calls. + +Another bigger issue is how these functions interact with the ownership model. + +```swift +// borrow access begins +mutex.lock() +// borrow access ends +... +// borrow access begins +mutex.unlock() +// borrow access ends +``` + +In the above, I've modeled where the borrowing accesses occur when calling these functions. This is an important distinction to make because unlike C++'s similar synchronization primitives, `Mutex` (and similarly `Atomic`) can be _moved_. These types guarantee a stable address "for the duration of a borrow access", but as you can see there's nothing guaranteeing that the unlock is occurring on the same address as the call to lock. As opposed to the closure based API (and hopefully in the future the guard based API): + +```swift +// borrow access begins +mutex.withLock { + ... +} +// borrow access ends + +do { + // borrow access begins + let locked = mutex.lock() + ... + + // borrow access ends when 'locked' gets deinitialized +} +``` + +In the first example with the closure, we syntactically define our borrow access with the closure because the entire closure will be executed during the borrow access of the mutex. In the second example, the guarded value we get back from a guard based `lock()` will extend the duration of the borrow access for as long as the `locked` binding is available. + +Providing these APIs on `Mutex` would be incredibly unsafe. We feel that the proposed `withLock` closure based API is much safer and sufficient for most use cases of a mutex. A guard based `lock()` API should cover most of the remaining use cases of needing these bare primitive operations in a much more safer fashion. + +### Rename to `Lock` or similar + +A very common name for this type in various codebases is simply `Lock`. This is a decent name because many people know immediately what the purpose of the type is, but the issue is that it doesn't describe _how_ it's implemented. I believe this is a very important aspect of not only this specific type, but of synchronization primitives in general. Understanding that this particular lock is implemented via mutual exclusion conveys to developers who have used something similar in other languages that they cannot have multiple readers and cannot call `lock()` again on the acquired thread for instance. Many languages similar to Swift have opted into a similar design, naming this type mutex instead of a vague lock. In C++ we have `std::mutex` and in Rust we have `Mutex`. + +### Include `Mutex` in the default Swift namespace (either in `Swift` or in `_Concurrency`) + +This is another intriguing idea because on one hand misusing this type is significantly harder than misusing something like `Atomic`. Generally speaking, we do want folks to reach for this when they just need a simple traditional lock. However, by including it in the default namespace we also unintentionally discouraging folks from reaching for the language features and APIs they we've already built like `async/await`, `actors`, and so much more in this space. Gating the presence of this type behind `import Synchronization` is also an important marker for anyone reading code that the file deals with managing their own synchronization through the use of synchronization primitives such as `Atomic` and `Mutex`. diff --git a/proposals/0434-global-actor-isolated-types-usability.md b/proposals/0434-global-actor-isolated-types-usability.md new file mode 100644 index 0000000000..da254db685 --- /dev/null +++ b/proposals/0434-global-actor-isolated-types-usability.md @@ -0,0 +1,257 @@ +# Usability of global-actor-isolated types + +* Proposal: [SE-0434](0434-global-actor-isolated-types-usability.md) +* Authors: [Sima Nerush](https://github.com/simanerush), [Matt Massicotte](https://github.com/mattmassicotte), [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.0)** +* Upcoming Feature Flag: `GlobalActorIsolatedTypesUsability` +* Review: ([pitch](https://forums.swift.org/t/pitch-usability-of-global-actor-isolated-types/70799)) ([review](https://forums.swift.org/t/se-0434-usability-of-global-actor-isolated-types/71187)) ([acceptance](https://forums.swift.org/t/accepted-se-0434-usability-of-global-actor-isolated-types/72743)) + +## Introduction + +This proposal encompasses a collection of changes to concurrency rules concerning global-actor-isolated types to improve their usability. + +## Motivation + +Currently, there exist limitations in the concurrency model around types that are isolated to global actors. + +First, let's consider the stored properties of `struct`s isolated to global actors. `let` properties of such types are implicitly treated as `nonisolated` within the current module if they have `Sendable` type, but `var` properties are not. This poses a number of problems, such as when implementing a protocol conformance. Currently, the only solution is to declare the property `nonisolated(unsafe)`: + +```swift +@MainActor struct S { + nonisolated(unsafe) var x: Int = 0 +} + +extension S: Equatable { + static nonisolated func ==(lhs: S, rhs: S) -> Bool { + return lhs.x == rhs.x + } +} +``` + +However, there is nothing unsafe about treating `x` as `nonisolated`. The general rule is that concurrency is safe as long as there aren't data races. The type of `x` conforms to `Sendable`, and using a value of `Sendable` type from multiple concurrent contexts shouldn't ever introduce a data race, so any data race involved with an access to `x` would have to be on memory in which `x` is stored. But `x` is part of a value type, which means any access to it is always also an access to the containing `S` value. As long as Swift is properly preventing data races on that larger access, it's always safe to access the `x` part of it. So, first off, there's no reason for Swift to require `(unsafe)` when marking `x` `nonisolated`. + +We can do better than that, though. It should be possible to treat a `var` stored property of a global-actor-isolated value type as *implicitly* `nonisolated` under the same conditions that a `let` property can be. A stored property from a different module can be changed to a computed property in the future, and those future computed accessors may need to be isolated to the global actor, so allowing access across module boundaries would not be okay for source or binary compatibility without an explicit `nonisolated` annotation. But within the module that defines the property, we know that hasn't happened, so it's fine to use a more relaxed rule. + +Next, under the current concurrency rules, it is possible for a function type to be both isolated to a global actor and yet not required to be `Sendable`: + +```swift +func test(globallyIsolated: @escaping @MainActor () -> Void) { + Task { + // error: capture of 'globallyIsolated' with non-sendable type '@MainActor () -> Void' in a `@Sendable` closure + await globallyIsolated() + } +} +``` + +This is not a useful combination: such a function can only be used if the current context is isolated to the global actor, and in that case the global actor annotation is unnecessary because *all* non-`Sendable` functions will run with global actor isolation. It would be better for a global actor attribute to always imply `@Sendable`. + +Because a globally-isolated closure cannot be called concurrently, it's safe for it to capture non-`Sendable` values even if it's implicitly `@Sendable`. Such values just need to be transferred to the global actor's region (if they aren't there already). The same logic also applies to closures that are isolated to a specific actor reference, although it isn't currently possible to write such a closure in a context that isn't isolated to that actor. + + +Finally, the current diagnostic for a global-actor-isolated subclass of a non-isolated superclass is too restrictive: + +```swift +class NotSendable {} + + +@MainActor +class Subclass: NotSendable {} // error: main actor-isolated class 'Subclass' has different actor isolation from nonisolated superclass 'NotSendable' +``` + +Because global actor isolation on a class implies a `Sendable` conformance, adding isolation to a subclass of a non-`Sendable` superclass can circumvent `Sendable` checking: + +```swift +func computeCount() async -> Int { ... } + +class NotSendable { + var mutableState = 0 + func mutate() async { + let count = await computeCount() + mutableState += count + } +} + +@MainActor +class Subclass: NotSendable {} + +func test() async { + let c = Subclass() + await withDiscardingTaskGroup { group in + group.addTask { + await c.mutate() + } + + group.addTask { @MainActor in + await c.mutate() + } + } +} +``` + +In the above code, an instance of `Subclass` can be passed across isolation boundaries because `@MainActor` implies that the type is `Sendable`. However, `Subclass` inherits non-isolated, mutable state from the superclass, so this `Sendable` conformance allows smuggling unprotected shared mutable state across isolation boundaries to create potential for concurrent access. For this reason, the warning about adding isolation to a subclass was added in Swift 5.10, but this restriction could be lifted by instead preventing the subclass from being `Sendable`. + +## Proposed solution + +We propose that: + +- Stored properties of `Sendable` type in a global-actor-isolated value type can be declared as `nonisolated` without using `(unsafe)`. +- Stored properties of `Sendable` type in a global-actor-isolated value type are treated as `nonisolated` when used within the module that defines the property. +- `@Sendable` is inferred for global-actor-isolated functions and closures. +- Global-actor-isolated closures are allowed to capture non-`Sendable` values despite being `@Sendable`. +- A global-actor-isolated subclass of a non-isolated, non-`Sendable` class is allowed, but it must be non-`Sendable`. + + +## Detailed design + + +### Inference of `nonisolated` for `var` properties of globally isolated value types + +Let's look at the first problem with usability of a `var` property of a main-actor-isolated struct: + +```swift +@MainActor +struct S { + var x: Int = 0 // okay ('nonisolated' is inferred within the module) +} + +extension S: Equatable { + static nonisolated func ==(lhs: S, rhs: S) -> Bool { + return lhs.x == rhs.x // okay + } +} +``` + +In the above code, `x` is implicitly `nonisolated` within the module. Under this proposal, `nonisolated` is inferred for in-module access to `Sendable` properties of a global-actor-isolated value type. A `var` with `Sendable` type within a value type can also have an explicit `nonisolated` modifier to allow synchronous access from outside the module. Once added, `nonisolated` cannot later be removed without potentially breaking clients. The programmer can still convert the property to a computed property, but it has to be a `nonisolated` computed property. + +Because `nonisolated` access only applies to stored properties, wrapped properties and `lazy`-initialized properties with `Sendable` type still must be isolated because they are computed properties: + +```swift +@propertyWrapper +struct MyWrapper { ... } + +@MainActor +struct S { + @MyWrapper var x: Int = 0 +} + +extension S: Equatable { + static nonisolated func ==(lhs: S, rhs: S) -> Bool { + return lhs.x == rhs.x // error + } +} +``` + +### `@Sendable` inference for global-actor-isolated functions and closures + +To improve usability of globally-isolated functions and closures, under this proposal `@Sendable` is inferred: + +```swift +func test(globallyIsolated: @escaping @MainActor () -> Void) { + Task { + await globallyIsolated() //okay + } +} +``` + +The `globallyIsolated` closure in the above code is global-actor isolated because it has the `@MainActor` attribute. Because it will always run isolated, it's fine for it to capture and use values that are isolated the same way. It's also safe to share it with other isolation domains because the captured values are never directly exposed to those isolation domains. This means that there's no reason not to always treat these functions as `@Sendable`. + +#### Non-`Sendable` captures in isolated closures + +Under this proposal, globally-isolated closures are allowed to capture non-`Sendable` values: + +```swift +class NonSendable {} + +func test() { + let ns = NonSendable() + + let closure = { @MainActor in + print(ns) + } + + Task { + await closure() // okay + } +} +``` + +The above code is data-race safe, since a globally-isolated closure will never operate on the same instance of `NonSendable` concurrently. + +Note that under region isolation in SE-0414, capturing a non-`Sendable` value in an actor-isolated closure will transfer the region into the actor, so it is impossible to have concurrent access on non-`Sendable` captures even if the isolated closure is formed outside the actor: + +```swift +class NonSendable {} + +func test(ns: NonSendable) async { + let closure = { @MainActor in + print(ns) // error: task-isolated value 'ns' can't become isolated to the main actor + } + + await closure() +} +``` + +### Global actor isolation and inheritance + +Subclasses may add global actor isolation when inheriting from a nonisolated, non-`Sendable` superclass. In this case, an implicit conformance to `Sendable` will not be added, and explicitly specifying a `Sendable` conformance is an error: + +```swift +class NonSendable { + func test() {} +} + +@MainActor +class IsolatedSubclass: NonSendable { + func trySendableCapture() { + Task.detached { + self.test() // error: Capture of 'self' with non-sendable type 'IsolatedSubclass' in a `@Sendable` closure + } + } +} +``` + +Inherited and overridden methods still must respect the isolation of the superclass method: + +```swift +class NonSendable { + func test() { ... } +} + +@MainActor +class IsolatedSubclass: NonSendable { + var mutable = 0 + override func test() { + super.test() + mutable += 0 // error: Main actor-isolated property 'mutable' can not be referenced from a non-isolated context + } +} +``` + +Matching the isolation of the superclass method is necessary because the superclass method implementation may internally rely on the static isolation, such as when hopping back to the isolation after any asynchronous calls, and because there are a variety of ways to call the subclass method that don't preserve its isolation, including: + +* Upcasting to the superclass type +* Erasing to an existential type based on conformances of the superclass type +* Passing the isolated subclass as a generic argument to a type parameter that requires a conformance implemented by the superclass + +## Source compatibility + +This proposal changes the interpretation of existing code that uses global-actor-isolated function types that are not already marked with `@Sendable`. This can cause minor changes in type inference and overload resolution. However, the proposal authors have not encountered any such issues in source compatibility testing, so this proposal does not gate the inference change behind an upcoming feature flag. + +An alternative choice would be to introduce an upcoming feature flag that's enabled by default in the Swift 6 language mode, but this flag could not be enabled by default under `-strict-concurrency=complete` without risk of changing behavior in existing projects that adopt complete concurrency checking. Gating the `@Sendable` inference change behind a separate upcoming feature flag may lead to more code churn than necessary when migrating to complete concurrency checking unless the programmer knows to enable the flags in a specific order. + +## ABI compatibility + +`@Sendable` is included in name mangling, so treating global-actor-isolated function types as implicitly `@Sendable` changes mangling. This change only impacts resilient libraries that use global-actor-isolated-but-not-`Sendable` function types in effectively-public APIs. However, as noted in this proposal, such a function type is not useful, and the proposal authors expect that any API that uses a global-actor-isolated function type either already has `@Sendable`, or should add `@Sendable`. Because the only ABI impact of `@Sendable` is mangling, `@_silgen_name` can be used to preserve ABI in cases where `@Sendable` should be added, and the API is not already `@preconcurrency` (in which case the mangling will strip both the global actor and `@Sendable`). + +## Implications on adoption + +The existing adoption implications of `@Sendable` and global actor isolation adoption apply when making use of the rules in this proposal. For example, `@Sendable` and `@MainActor` can be staged into existing APIs using `@preconcurrency`. See [SE-0337: Incremental migration to concurrency checking](/proposals/0337-support-incremental-migration-to-concurrency-checking.md) for more information. + +## Alternatives considered + +Instead of implicitly suppressing a `Sendable` conformance on isolated subclasses of non-`Sendable`, non-isolated superclasses, the compiler could instead require an explicit opt-out, such as `~Sendable` in the conformance clause. This would make it obvious that the subclass does not have a `Sendable` conformance. However, the programmer does not need to understand that the class does not conform to `Sendable` until they use the type in a way that requires `Sendable`, and the reason for the class not conforming to `Sendable` can be explained with notes attached to the diagnostic. It is also not always the case that global actor isolation implies `Sendable`. Notably, `@MainActor` on a protocol does not imply that the protocol refines `Sendable`, so requiring more boilerplate for programmers in the isolated subclass case does not leave the programmer with a simple rule to remember about when `@MainActor` implies a conformance to `Sendable`. + +## Acknowledgments + +Thank you to Frederick Kellison-Linn for surfacing the problem with global-actor-isolated function types, and to Kabir Oberai for exploring the implications more deeply. diff --git a/proposals/0435-swiftpm-per-target-swift-language-version-setting.md b/proposals/0435-swiftpm-per-target-swift-language-version-setting.md new file mode 100644 index 0000000000..c9c62e089a --- /dev/null +++ b/proposals/0435-swiftpm-per-target-swift-language-version-setting.md @@ -0,0 +1,54 @@ +# Swift Language Version Per Target + +* Proposal: [SE-0435](0435-swiftpm-per-target-swift-language-version-setting.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 6.0)** +* Review: ([pitch](https://forums.swift.org/t/pitch-swiftpm-swift-language-version-per-target/71067)) ([review](https://forums.swift.org/t/se-0435-swift-language-version-per-target/71546)) ([acceptance](https://forums.swift.org/t/accepted-se-0435-swift-language-version-per-target/71846)) + +## Introduction + +The current Swift Package Manager manifest API for specifying Swift language version(s) applies to an entire package which is limiting when adopting new language versions that have implications for source compatibility. + +Swift-evolution thread: [Pitch: [SwiftPM] Swift Language Version Per Target](https://forums.swift.org/t/pitch-swiftpm-swift-language-version-per-target/71067) + +## Motivation + +Adopting new language versions at the target granularity allows for gradual migration to prevent possible disruptions. Swift 6, for example, turns on strict concurrency by default, which can have major implications for the project in the form of new errors that were previously downgraded to warnings. SwiftPM should allow to specify a language version per target so that package authors can incrementally transition their project to the newer version. + +## Proposed solution + +Add a new Swift target setting API, similar to `enable{Upcoming, Experimental}Feature`, to specify a Swift language version that should be used to build the target, if such version is not specified, fallback to the current language version determination logic. + +## Detailed design + +Add a new `swiftLanguageVersion` API to `SwiftSetting` limited to manifests >= 6.0: + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.0) + public static func swiftLanguageVersion( + _ version: SwiftVersion, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting { + ... + } +} +``` + +## Security + +New language version setting has no implications on security, safety or privacy. + +## Impact on existing packages + +Since this is a new API, all existing packages will use the default behavior - version specified at the package level when set or determined based on the tools version. + +## Alternatives considered + +- Add a new setting for 'known-safe' flags, of which `-swift-version` could be the first. This seems less user-friendly and error prone than re-using `SwiftLanguageVersion`, which has known language versions as its cases (with a plaintext escape hatch when necessary). + +- Add new initializer parameter to `Target` API and all of the convenience static functions, this is less flexible because it would require a default value. + diff --git a/proposals/0436-objc-implementation.md b/proposals/0436-objc-implementation.md new file mode 100644 index 0000000000..517599b667 --- /dev/null +++ b/proposals/0436-objc-implementation.md @@ -0,0 +1,348 @@ +# Objective-C implementations in Swift + +* Proposal: [SE-0436](0436-objc-implementation.md) +* Authors: [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.1)** +* Implementation: [swiftlang/swift#73309](https://github.com/swiftlang/swift/pull/73309), [swiftlang/swift#74801](https://github.com/swiftlang/swift/pull/74801) +* Review: ([first pitch](https://forums.swift.org/t/pitch-objective-c-implementations-in-swift/61907)) ([second pitch](https://forums.swift.org/t/pitch-2-objective-c-implementations-in-swift/68090)) ([third pitch](https://forums.swift.org/t/pitch-3-objective-c-implementations-in-swift/71315)) ([review](https://forums.swift.org/t/se-0436-objective-c-implementations-in-swift/71712)) ([acceptance](https://forums.swift.org/t/accepted-se-0436-objective-c-implementations-in-swift/72053)) + +## Introduction + +We propose an alternative to `@objc` classes where Objective-C header `@interface` declarations are implemented by Swift `extension`s marked with `@objc @implementation`. The resulting classes will be implemented in Swift, but will be indistinguishable from Objective-C classes, fully supporting Objective-C subclassing and runtime trickery. + +Swift-evolution thread: [first pitch](https://forums.swift.org/t/pitch-objective-c-implementations-in-swift/61907), [second pitch](https://forums.swift.org/t/pitch-2-objective-c-implementations-in-swift/68090), [third pitch](https://forums.swift.org/t/pitch-3-objective-c-implementations-in-swift/71315) + +## Motivation + +Swift has always had a mechanism that allows Objective-C code to use Swift types: the `@objc` attribute. When a class is marked with `@objc` (or, more typically, inherits from an `@objc` or imported Objective-C class), Swift generates sufficient Objective-C metadata to allow it to be used through the Objective-C runtime, and prints a translated Objective-C declaration into a generated header file that can be imported into Objective-C code. The same goes for members of the class. + +This feature works really well for mixed-language apps and project-internal frameworks, but it's poorly suited to exposing private and especially public APIs to Objective-C. There are three key issues: + +1. To avoid circularity while building the Swift half of the module, the generated header cannot be included into other headers in the same module, which can make it difficult to use the Swift-implemented parts of the API in the Objective-C-implemented parts. Worse, some build systems install the headers for all modules and then build binaries for them out of order; generated headers can't really be used across modules in these systems. + +2. Objective-C programmers expect API headers to serve as a second source of documentation on the APIs, but generated headers are disorganized, unreadable messes because Swift cannot mechanically produce the formatting that a human engineer would add to a handwritten header. + +3. While `@objc` classes can be *used* from Objective-C, they are not truly Objective-C types. They still contain Swift vtables and other Swift-specific data that the Objective-C compiler and runtime don't fully understand. This limits their capabilities—for instance, Objective-C code cannot subclass an `@objc` class or reliably swizzle its methods. + +Together, these issues make it very hard for frameworks with a lot of Objective-C clients to implement their functionality in Swift. If they have classes that are meant to be subclassed, it's actually impossible to fully port them to Swift, because it would break existing Objective-C subclasses. And yet the trade-offs made by `@objc` are really good for the things it's designed for, like writing custom views and view controllers in Swift app targets. We don't want to radically change the existing `@objc` feature. + +Swift also quietly supports a hacky pseudo-feature that allows a different model for Objective-C interop: It will not diagnose a selector conflict if a Swift extension redeclares members already imported from Objective-C, so you can declare a method or property in a header and then implement it in a Swift extension. However, this feature has not really been designed to work properly; it doesn't check that your implementation's name and signature match the header, there's no protection against forgetting to implement a method, and you still need an `@implementation` for the class metadata itself. Nevertheless, a few projects use this and find it helpful because it avoids the issues with normal interop. Formalizing and improving this pattern seems like a promising direction for Objective-C interop. + +## Proposed solution + +We propose adding a new attribute, `@implementation`, which, when paired with an interop attribute like `@objc`, tells Swift that it is to implement a declaration it has imported from another language, rather than creating a new declaration and exporting it *to* that language. + +Specifically, in this proposal, `@objc @implementation` allows a Swift `extension` to replace an Objective-C `@implementation` block. You write headers as normal for an Objective-C class, but instead of writing an `@implementation` in an Objective-C file, you write an `@objc @implementation extension` in a Swift file. You can even port an existing class’s implementation to Swift one category at a time without breaking backwards compatibility. + +For instance, if you were adding a new class, you would start by writing a normal Objective-C header, as though you were planning to implement the class in an Objective-C .m file: + +```objc +#import + +NS_HEADER_AUDIT_BEGIN(nullability, sendability) + +@interface MYFlippableViewController : UIViewController + +@property (strong) UIViewController *frontViewController; +@property (strong) UIViewController *backViewController; +@property (assign,getter=isShowingFront) BOOL showingFront; + +- (instancetype)initWithFrontViewController:(UIViewController *)front backViewController:(UIVIewController *)back; + +@end + +@interface MYFlippableViewController (Animation) + +- (void)setShowingFront:(BOOL)isShowingFront animated:(BOOL)animated NS_SWIFT_NAME(setIsShowingFront(_:animated:)); +- (void)setFrontViewController:(UIViewController *)front animated:(BOOL)animated; +- (void)setBackViewController:(UIViewController *)back animated:(BOOL)animated; + +@end + +@interface MYFlippableViewController (Actions) + +- (IBAction)flip:(id)sender; + +@end + +NS_HEADER_AUDIT_END(nullability, sendability) +``` + +And you would arrange for Swift to import it through an umbrella or bridging header. You would then write an `extension` for each `@interface` you wish to implement in Swift. For example, you could implement the main `@interface` (plus any visible class extensions) in Swift by writing: + +```swift +@objc @implementation extension MYFlippableViewController { + ... +} +``` + +And the `Animation` category by writing: + +```swift +@objc(Animation) @implementation extension MYFlippableViewController { + ... +} +``` + +Note that there is nothing special in the header which indicates that a given `@interface` is implemented in Swift. The header can use all of the usual Swift annotations—like `NS_SWIFT_NAME`, `NS_NOESCAPE` etc.—but they simply affect how the member is imported. Swift does not even require you to implement every declared `@interface` in Swift, so you can implement some parts of a class in Objective-C and others in Swift. But if you choose to implement a particular `@interface` in Swift, each Objective-C member in that `@interface` must be matched by a Swift declaration in the extension that has the same Swift name; these special members are called "member implementations". + +Specifically, member implementations must be non-`final`, not be overrides, and have an `open`, `public`, `package`, or `internal` access level. Every member implementation must match a member declared in the Objective-C header, and every member declared in the Objective-C header must have a matching member implementation. This ensures that everything declared by the header is correctly implemented without any accidental misspellings or type signature mismatches. + +In addition to member implementations, an `@objc @implementation` extension can also contain three other kinds of members: + +1. **Fileprivate or private non-`final` members** are helper methods (think `@IBAction`s or callback selectors). They must *not* match a member from the imported headers, but they are accessible from Objective-C by performing a selector or declaring them in a place that is not visible to Swift. (Objective-C `@implementation` blocks often declare private members like this, so it's helpful to allow them in `@objc @implementation` extensions too.) + +2. **Members with an `override` modifier** override superclass members and function normally. (Again, Objective-C `@implementation` blocks often override superclass members without declaring the override in their headers, so it's helpful to allow `@objc @implementation` extensions to do the same.) + +3. **Members with a `final` modifier (or `@nonobjc` on an initializer)** are Swift-only and can use Swift-only types or features. These may be Swift-only implementation details (if `internal` or `private`) or Swift-only APIs (if `public` or `package`). (This feature is necessary to support stored properties containing Swift-only types because ordinary extensions cannot declare stored properties; in theory, we could require other `final` members to be declared in a separate ordinary extension, but we also allow them in an `@objc @implementation` extension as a convenience.) + +Within an `@objc @implementation` extension, `final` members are `@nonobjc` by default. + +As a special exception to the usual rule, a non-category `@objc @implementation extension` can declare stored properties and other members that are normally only allowed in the main `class` body. They can be (perhaps implicitly) `@objc` or they can also be `final`; in the latter case they are only accessible from Swift. Note that `@implementation` does not have an equivalent to Objective-C's implicit `@synthesize`—you must declare a `var` explicitly for each `@property` in the header that you want to be backed by a stored property. + +## Detailed design + +### Custom category names + +As an enabling step, we propose that using `@objc(CustomName)` on an `extension` should cause the Objective-C category created for that extension to be named `CustomName`, rather than using a category name generated by the compiler. This name should appear in both generated headers and Objective-C metadata. The compiler should enforce that there is only one extension per class/category-name combination. + +`@objc(CustomName) extension` also has the effect of `@objc extension`, namely, making members of the extension `@objc` by default. + +### `@implementation` + +The compiler will accept a new attribute, `@implementation`, which turns a declaration that would normally be exported to another language into a declaration that implements something imported from another language. This attribute takes no arguments; it should be used alongside another attribute specifying the foreign language, such as `@objc`. + +In general, `@implementation` will cause the affected declarations to be matched against imported declarations, and will emit errors if a good enough match cannot be found. It cannot necessarily be applied to all declarations that can be exported to a foreign language; for instance, `@objc @implementation` is only supported on whole extensions, not on individual members. + +### `@objc @implementation extension`s + +`@objc @implementation extension` causes an extension to be used as the implementation of an Objective-C `@interface` declaration for the class it extends. If the `@objc` attribute specifies a custom category name, it will implement a category with that name; otherwise it will implement the main `@interface` for the class (plus any class extensions). + +```swift +@objc @implementation extension SomeClass { + // Equivalent to `@implementation SomeClass`; + // implements everything in `@interface SomeClass` and + // all `@interface SomeClass ()` extensions. +} + +@objc(SomeCategory) @implementation extension SomeClass { + // Equivalent to `@implementation SomeClass (SomeCategory)`; + // implements everything in `@interface SomeClass (SomeCategory)`. +} +``` + +As in any `@objc extension`, all members are implicitly `@objc` by default, and all `@objc` members are implicitly `dynamic`; unlike other `@objc extension`s, adding the `final` keyword makes the declaration *not* `@objc` by default. + +As a special exception to the usual rule, an `@objc @implementation extension` which implements the main Objective-C interface of a class can declare stored properties, designated and required `init`s, and `deinit`s. + +#### Rules + +An `@objc @implementation extension` must: + +* Extend a non-root class imported from Objective-C which does not use lightweight generics. +* If a category name is present, match a category by that name for that class (if no category name is present, the extension matches the main interface). +* Provide a member implementation (see below) for each member of the `@interface` it implements. +* Not declare conformances. (Conformances should be declared in the header if they are for Objective-C protocols, or in an ordinary extension otherwise.) +* Contain only `@objc`, `override`, `final`, or (for initializers) `@nonobjc` members. (Note that member implementations are implicitly `@objc`, as mentioned below, so this effectively means that non-`override`, non-`final`, non-`@nonobjc` members *must* be member implementations.) +* `@nonobjc` initializers must be convenience initializers, not designated or required initializers. + +> **Note**: `@objc @implementation` cannot support Swift-only designated and required initializers because subclasses with additional stored properties must be able to override designated and required initializers, but `@implementation` only supports overriding of `@objc` members. The Swift-only metadata that would be used for dynamic dispatch in an ordinary `@objc` class is not present in an `@implementation` class. + +### Member implementations + +Any non-`override` open, public, package, or internal `@objc` member of an `@objc @implementation extension` is a “member implementation”; that is, it implements some imported Objective-C member of the class it is extending. Member implementations are special because much of the compiler completely ignores them: + +* Access control denies access to member implementations in most contexts. +* Module interfaces and generated interfaces do not include member implementations. +* Objective-C generated headers do not include member implementations. + +This means that calls in expressions will *always* ignore the member implementation and use the imported Objective-C member instead. In other words, even other Swift code in the same module will behave as though the member is implemented in Objective-C. + +Some members cannot be implemented in an `@objc @implementation` extension because `@objc` does not support some of the features they use; see "Future Directions" for more specific discussion of this. These members will have to be moved to a separate category and implemented in Objective-C. + +#### Rules + +A member implementation must: + +* Have the same Swift name as the member it implements. +* Have the same selector as the member it implements. +* Have the same foreign error convention and foreign async convention as the member it implements. +* Not have other traits, like an overload signature, `@nonobjc`/`final` attribute, `class` modifier, or mutability, which is incompatible with the member it implements. +* Not have `@_spi` attributes (they would be pointless since the visibility of the imported Objective-C attribute is what will make the member usable or not). + +Both the Swift name and the Objective-C selector of a member implementation must match the corresponding Objective-C declaration; Swift will diagnose an error if one matches but the other doesn't. This checking respects both the `@objc(custom:selector:)` in Swift implementations and the Swift name (`NS_SWIFT_NAME(custom(_:name:))`) attribute in Objective-C headers. + +Member implementations must have an overload signature that closely matches the Objective-C declaration’s. However, types that are non-optional in the Objective-C declaration may be implicitly unwrapped optionals in the member implementation if this is ABI-compatible; this is because Objective-C does not prevent clients from passing `nil` or implementations from returning `nil` when `nonnull` is used, and member implementations may need to implement backwards compatibility logic for this situation. + +### Objective-C metadata generation + +When Swift generates metadata for an `@objc @implementation extension`, it will generate metadata that matches what clang would have generated for a similar `@implementation`. That is: + +* `@objc` members will only have Objective-C metadata, not Swift metadata. (`final` members may require Swift metadata.) +* If the extension is for the main class interface, it will generate complete Objective-C class metadata with an ivar for each stored property, and without setting the Swift bit or using any features incompatible with clang subclasses or categories. + +## Source compatibility + +These changes are additive and don't affect existing code. Replacing an Objective-C `@implementation` declaration with a Swift `@objc @implementation extension` is invisible to the library's Objective-C and Swift clients, so it should not be source-breaking unless the implementations have observable behavior differences. + +Previous versions of Swift have accepted the `@objc(CustomName) extension` syntax and simply ignored the custom name, so merely adding a custom category name won't break source compatibility. + +## Effect on ABI stability + +All `@objc` members of an `@implementation extension`—member implementation or otherwise—have the ABI of an `@objc dynamic` member, so turning one into the other is not ABI-breaking. `@implementation extension` classes generate only Objective-C metadata, not Swift metadata, so existing Objective-C subclasses will continue to function as normal. + +Because `@implementation` attributes and member implementations are not printed into module interfaces, this proposal has no direct effect on Swift ABI stability. + +## Implications on adoption + +`@implementation` extensions that implement categories are back-deployable to Swift 5.0 runtimes and later, and many `@implementation` extensions that implement classes are too. However, if a class's ivar layout cannot be computed at compile time, that class will require new runtime support and will not be back-deployable to old platforms. + +Affected classes are ones whose stored properties contain a non-frozen enum or struct imported from another module that has library evolution enabled. (This property is transitive—if your stored properties contain a struct in your own module, but that struct has a stored property of an affected type, that also limits back deployment.) In practice, it is usually possible to work around this problem by boxing affected values in a class or existential, at the cost of some overhead. + +> **Note**: Some of the required runtime changes are in the Objective-C runtime, so even a development toolchain will not be sufficient to actually run modules with affected classes. However, you can test the diagnostics and code generation by compiling with the experimental feature flag `ObjCImplementationWithResilientStorage`; OS version 99.99 will be treated as high enough to have the necessary runtime support. + +## Future directions + +### Extending `@objc` capabilities to extend `@objc @implementation` capabilities + +`@objc @implementation` extensions cannot implement `@interface` members that cannot be created by `@objc`. The most notable limitations include: + +* Factory convenience initializers (those implemented as class methods, like `+[NSString stringWithCharacters:length:]`). +* `__attribute__((objc_direct))` methods and `@property (direct)` properties. +* Members with nonstandard memory management behavior, even if it is correctly annotated. +* Members which deviate from the Objective-C error convention in certain subtle ways, such as by having the `NSError**` parameter in the wrong place. + +Additionally, `@objc @implementation` cannot implement global declarations that cannot be created by `@objc`, such as: + +* Free functions, global variables, cases of `NS_TYPED_ENUM` typedefs, and other non-member Objective-C declarations. +* Global declarations imported as members of a type using `NS_SWIFT_NAME`'s import-as-member capabilities. + +`@objc @implementation` heavily piggybacks on `@objc`'s code emission, so in most cases, the best approach to expanding `@implementation`'s support would be to extend `@objc` to support the feature and then make sure `@objc @implementation` supports it too. + +### `@implementation` for plain C declarations + +Many of the capabilities mentioned as future directions for `@objc` would also be useful for plain-C clients, including those on non-Darwin platforms. Once again, the best approach here would probably be to stabilize and extend something like `@_cdecl` to support creating these with a generated header, and then make sure `@implementation` supports this attribute too. + +The compiler currently has experimental support for `@_cdecl @implementation` for global functions; it's behind a separate experimental feature flag because it's not part of this proposal. + +### `@implementation` for C++ declarations + +One could similarly imagine a C++ version of this feature: + +```cpp +// C++ header file + +class CppClass { +private: + int myStorage = 0; +public: + int someMethod() { ... } + int swiftMethod(); +}; +``` + +```swift +// Swift implementation file + +@_expose(Cxx) @implementation extension CppClass { + func swiftMethod() -> Int32 { return self.myStorage } +} +``` + +This would be tricker than Objective-C interop because for Objective-C interop, Swift already generates machine code thunks directly in the binary, whereas for C++ interop, Swift generates C++ source code thunks in a generated header. Swift could either compile this generated code internally, or it could emit it to a file and expect the build system to build and link it. + +We believe that there wouldn't be a problem with sharing the `@implementation` attribute with this feature because `@implementation` is always paired with a language-specific attribute. + +### Supporting lightweight generics + +Classes using Objective-C lightweight generics have type-erased generic parameters; this imposes a lot of tricky limitations on Swift extensions of these classes. Since very few classes use lightweight generics, we have chosen to ban the combination for now. If there turns out to be a lot of demand for implementing Objective-C generic classes in Swift, we can lift the ban after we've figured out how to make the combination more usable. + +### `@objc @implementation(unchecked)` + +One could imagine an option that disables `@objc @implementation`'s exhaustiveness checking so that Swift implementations can use dynamic mechanisms like `+instanceMethodForSelector:` to create methods at runtime. This change would be purely additive, so we can consider it if there's demand. + +### Implementation-only bridging header + +This feature would work extremely well with a feature that allowed Swift to import an implementation-only bridging header alongside the umbrella header when building a public framework. This would not only give the Swift module access to internal Objective-C declarations, but also allow it to implement those declarations. However, the two features are fully orthogonal, so I’ll leave that to a different proposal. + +### Improvements to private Objective-C modules + +This feature would also work very well with some improvements to private Objective-C modules: + +1. The Swift half of a mixed-source framework could implicitly import the private Clang module with `internal`; this would allow you to easily provide implementations for Objective-C-compatible SPI. +2. We could perhaps set up some kind of equivalence between `@_spi` and private Clang modules so that `final` Swift members could be made public. + +Again, that’s something we can flesh out over time. + +## Alternatives considered + +### A different attribute spelling + +We've chosen the proposed spelling—`@objc(CategoryName) @implementation`—because it makes `@implementation` orthogonal to the specific language being implemented. Everything Objective-C-specific about it is tied to the `@objc` attribute, and it's pretty clear how it would be expanded in the future to support other languages. Many alternatives—such as the original pitch's `@objcImplementation(CategoryName)` and suggestions like `@implementation(objc, category: CategoryName)`—do not have this property. + +We chose the word "implementation" rather than "extern" because the attribute marks an implementation of something that was declared elsewhere; in most languages, "extern" works in the opposite direction, marking a declaration of something that is implemented elsewhere. Also, the name suggests a relationship to the `@implementation` keyword in Objective-C, which is appropriate since an `@objc @implementation extension` is a one-to-one drop-in replacement. + +`@implementation` has a similar spelling to the compiler-internal `@_implements` attribute, but there has been little impetus to stabilize `@_implements` in the seven years since it was added, and if it ever *is* stabilized it wouldn't be unreasonable to make it a variant of `@implementation` (e.g. `@implementation(forRequirement: SomeProto.someMethod)`). + +### Allow/require `@objc @implementation` extensions to declare conformances + +Rather than banning `@objc @implementation` extensions from declaring conformances, we could require them to re-declare conformances listed in the header and/or allow them to add additional conformances. This would more closely align with our design decisions about members, where we allow private members and overrides because Objective-C `@implementation` allows them. + +Unfortunately, every design here has at least one wart or inconsistency. There are actually four different alternatives here; let's tackle them separately: + +1. **Re-declares header conformances** + **Cannot declare extra conformances**: The conformances here are pure boilerplate; they duplicate what's in the header without any opportunity to add more information. By contrast, we force the extension to re-declare members because the redeclarations *add more information* (like the body), or at least they have the opportunity to add more information. It seems wasteful to require the developer to keep two conformance lists exactly in sync. + +2. **Doesn't re-declare header conformances** + **Can declare extra conformances**: The conformances behave very differently from the members. With members, you are *required* to declare what's in the header but also *allowed* to add certain kinds of extra members. Allowing a conformance list but requiring it to *not* include what's in the header is unintuitive. + +3. **Re-declares header conformances** + **Can declare extra conformances**: Unlike with members—where the additional members are made textually obvious by a modifier like `private` or `final`—there would be no visual distinction between re-declared conformances and extra conformances. That's unfortunate because their visibility is very different. The extra conformances would only be visible to clients which have imported the .swiftinterface, .swiftmodule, or -Swift.h files, and clients often don't know exactly which files they are importing from a module. Having a single, combined list of things which have invisible distinctions between them would be confusing. + +4. **Doesn't re-declare header conformances** + **Cannot declare extra conformances** (the design we selected): Does not map 1:1 to the behavior of `@implementation` in Objective-C (which can declare additional conformances). + +Since all of these options have unappealing aspects, we have chosen the simplest one with the smallest potential for mistakes, which is banning all use of the conformance list. It is still possible to write an ordinary extension which adds conformances to an `@objc @implementation` class; these conformances will have the same visibility behavior as extra conformances would, but putting them in a separate, ordinary extension creates a visible distinction to make this obvious. + +### `@implementation class` + +We considered using a `class` declaration, not an `extension`, to implement the main body of a class: + +```swift +// Header as above + +#if canImport(UIKit) +import UIKit +#else +import SwiftUIKitClone // hypothetical pure-Swift UIKit for non-Darwin platforms +#endif + +#if OBJC +@objc @implementation +#endif +class MYFlippableViewController: UIViewController { + var frontViewController: UIViewController { + didSet { ... } + } + var backViewController: UIViewController { + didSet { ... } + } + var isShowingFront: Bool { + didSet { ... } + } + + init(frontViewController: UIViewController, backViewController: UIViewController) { + ... + } +} + +#if OBJC +@objc(Animation) @implementation +#endif +extension MYFlippableViewController { + ... +} +``` + +This might be able to reduce code duplication for adopters who want to write cross-platform classes that only use `@implementation` on platforms which support Objective-C, and it would also mean we could remove the stored-property exception. However, it is a significantly more complicated model—there is much more we'd need to hide and a lot more scope for the `class` to have important mismatches with the `@interface`. And the reduction to code duplication would be limited because pure-Swift extension methods are non-overridable, so all methods you wanted clients to be able to override would have to be listed in the `class`. This means that in practice, mechanically generating pure-Swift code from the `@implementation`s might be a better approach. + +## Acknowledgments + +Doug Gregor gave a *ton* of input into this design; Allan Shortlidge reviewed much of the code; and Mike Ash provided timely help with a tricky Objective-C metadata issue. Thanks to all of them for their contributions, and thanks to all of the engineers who have provided feedback and bug reports on this feature in its experimental state. diff --git a/proposals/0437-noncopyable-stdlib-primitives.md b/proposals/0437-noncopyable-stdlib-primitives.md new file mode 100644 index 0000000000..ddef3d2c7c --- /dev/null +++ b/proposals/0437-noncopyable-stdlib-primitives.md @@ -0,0 +1,1760 @@ +# Noncopyable Standard Library Primitives + +* Proposal: [SE-0437](0437-noncopyable-stdlib-primitives.md) +* Authors: [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.0)** +* Roadmap: [Improving Swift performance predictability: ARC improvements and ownership control][Roadmap] +* Review: ([pitch](https://forums.swift.org/t/pitch-noncopyable-standard-library-primitives/71566)) ([review](https://forums.swift.org/t/se-0437-generalizing-standard-library-primitives-for-non-copyable-types/72020)) ([acceptance](https://forums.swift.org/t/accepted-se-0437-generalizing-standard-library-primitives-for-non-copyable-types/72275)) + +[Roadmap]: https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206 + +Related proposals: + +- [SE-0377] `borrowing` and `consuming` parameter ownership modifiers +- [SE-0390] Noncopyable structs and enums +- [SE-0426] BitwiseCopyable +- [SE-0427] Noncopyable generics +- [SE-0429] Partial consumption of noncopyable values +- [SE-0432] Borrowing and consuming pattern matching for noncopyable types + +[SE-0377]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0377-parameter-ownership-modifiers.md +[SE-0390]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md +[SE-0426]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0426-bitwise-copyable.md +[SE-0427]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md +[SE-0429]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0429-partial-consumption.md +[SE-0432]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0432-noncopyable-switch.md + +### Table of Contents + + * [Motivation](#motivation) + * [Low\-level memory management](#low-level-memory-management) + * [Generalized optional types](#generalized-optional-types) + * [Proposed Solution](#proposed-solution) + * [Generalizing function members](#generalizing-function-members) + * [Generalizing higher\-order functions](#generalizing-higher-order-functions) + * [Generalizing caller\-provided return types](#generalizing-caller-provided-return-types) + * [(Lack of) protocol generalizations](#lack-of-protocol-generalizations) + * [Unblocking basic construction work](#unblocking-basic-construction-work) + * [Detailed Design](#detailed-design) + * [protocol ExpressibleByNilLiteral](#protocol-expressiblebynilliteral) + * [enum Optional](#enum-optional) + * [enum Result](#enum-result) + * [enum MemoryLayout](#enum-memorylayout) + * [Unsafe Pointer Types](#unsafe-pointer-types) + * [Member generalizations](#member-generalizations) + * [Related enhancements in other types](#related-enhancements-in-other-types) + * [Setting temporary pointers on arbitrary entities](#setting-temporary-pointers-on-arbitrary-entities) + * [Unsafe Buffer Pointers](#unsafe-buffer-pointers) + * [Member generalizations](#member-generalizations-1) + * [Protocol conformances](#protocol-conformances) + * [Extracting parts of a buffer pointer](#extracting-parts-of-a-buffer-pointer) + * [Exceptions](#exceptions) + * [Related enhancements in other types](#related-enhancements-in-other-types-1) + * [Temporary buffer pointers over arbitrary entities](#temporary-buffer-pointers-over-arbitrary-entities) + * [Temporary Allocation Facility](#temporary-allocation-facility) + * [Managed Buffers](#managed-buffers) + * [Lifetime Management](#lifetime-management) + * [Swapping and exchanging items](#swapping-and-exchanging-items) + * [Source compatibility](#source-compatibility) + * [ABI compatibility](#abi-compatibility) + * [Alternatives Considered](#alternatives-considered) + * [Omitting UnsafeBufferPointer](#omitting-unsafebufferpointer) + * [Alternatives to UnsafeBufferPointer\.extracting()](#alternatives-to-unsafebufferpointerextracting) + * [Future Work](#future-work) + * [Non\-escapable Optional and Result](#non-escapable-optional-and-result) + * [Generalizing higher\-order functions](#generalizing-higher-order-functions-1) + * [Generalizing Optional\.unsafelyUnwrapped](#generalizing-optionalunsafelyunwrapped) + * [Generalized managed buffer headers](#generalized-managed-buffer-headers) + * [Additional raw pointer operations](#additional-raw-pointer-operations) + * [Protocol generalizations](#protocol-generalizations) + * [Additional future work](#additional-future-work) + * [Appendix: struct Hypoarray](#appendix-struct-hypoarray) + +## Motivation + +[SE-0427] allowed noncopyable types to participate in Swift generics, and introduced the protocol `Copyable` to the Standard Library. However, it stopped short of adapting the Standard Library to support using such constructs. + +The expectation that everything is copyable has been a crucial simplifying assumption throughout all previous API design work in Swift. It allowed and encouraged us to define and use interfaces without having to think too deeply about who is responsible for owning the entities we pass between functions; it let us define convenient container types with implicit copy-on-write value semantics; it has been a constant, familiar, friendly companion of every Swift programmer for almost a decade. Fully rethinking the Standard Library to facilitate working with noncopyable types is not going to happen overnight: it is going to take a series of proposals. This document takes the first step by focusing on an initial set of core changes that will enable building simple generic abstractions using noncopyable types. + +To achieve this, we need to tweak some core parts of the Standard Library to start eliminating the assumption of copyability. The changes proposed here only affect a small subset of the Standard Library's API surface; much more work remains to be done. But these changes are intended to be enough to let us start using Swift's new ownership control features in earnest, so that we can use them to solve real problems, but also so that we can gain crucial experience that will inform subsequent Standard Library work. + +This proposal concentrates on two particular areas: low-level memory management and generalized optional types. We propose to modify some preexisting generic constructs in the Standard Library to eliminate the assumption of copyability. Such a retroactive generalization is unlikely to be the right approach for every construct (especially not for copy-on-write container types), but it is the appropriate choice for these particular abstractions. + +### Low-level memory management + +First, we need to extend the existing low-level unsafe pointer operations to allow managing memory that holds noncopyable entities. + +- We need to teach `MemoryLayout` how to provide basic information on the memory layout of noncopyable types. +- `UnsafePointer` and `UnsafeMutablePointer` need to support noncopyable pointees. The existing pointer operations need to support working with such instances. This includes heap allocations, pointer conversions and comparisons, operations that bind or rebind raw memory, that initialize/deinitialize memory, etc. +- Similarly, `UnsafeBufferPointer` and `UnsafeMutableBufferPointer` must learn to support noncopyable elements. +- We need the standard low-level memory management facilities to allow working with noncopyable types: + - Scoped pointer-based access to arbitrary entities (`func withUnsafePointer(to:)`) + - Unmanaged heap memory allocations (`UnsafeMutablePointer.allocate`, `UnsafeMutableBufferPointer.allocate`) + - Managed tail-allocated storage allocations (`class ManagedBuffer`, `struct ManagedBufferPointer`). + - Allocating a temporary buffer (`func withUnsafeTemporaryAllocation`) + +Generalizing these constructs for noncopyable types does not fundamentally change their nature -- an `UnsafePointer` to a noncopyable pointee is still a regular, copyable pointer, conforming to much the same protocols as before, and providing many of the same operations. For example, given this simple noncopyable type `Foo`: + +```swift +struct Foo: ~Copyable { + var value: Int + mutating func increment() { value += 1 } +} +``` + +We want to be able to dynamically allocate memory for instances of `Foo`, and use the familiar pointer operations we've already learned while working with copyable types: + +```swift +let p = UnsafeMutablePointer.allocate(capacity: 2) +let q = p + 1 +p.initialize(to: Foo(value: 42)) +q.initialize(to: Foo(value: 23)) +p.pointee.increment() +print(p.pointee.value) // Prints "43" +print(p[0].value, p[1].value) // Prints "43 23" +print(p < q) // Prints "true" +let foo = p.move() +q.deinitialize(count: 1) +p.deallocate() +``` + +Most of the core pointer operations were already (implicitly) defined in terms of ownership control, and so they readily translate into noncopyable use. + +Of course, not all operations can be generalized: for example, `p.initialize(repeating: Foo(7), count: 2)` cannot possibly work, as repeating an item inherently requires making copies of it. That's not a problem though: we can continue to have such operations require a copyable pointee. + + +### Generalized optional types + +The second area that requires immediate attention is the `Optional` enumeration and its close sibling, `Result`. `Optional` is particularly frequently used in the definition of programming interfaces: it is the standard way a Swift function can take or return a potentially absent value. It is also deeply integrated into the language itself: features such as optional chaining, failable initializers and `try?` statements all rely on it, and we need these features to work even in noncopyable contexts. + +It is therefore very much desirable for optionals to start supporting noncopyable payloads, so that Swift functions can continue to use these well-known types in their interface definitions. + +Instances of `Optional` and `Result` directly contain the items they wrap. Therefore, an optional wrapping a noncopyable type will necessarily need to become noncopyable itself. This is a far more radical change than generalizing a pointer type: it means these enumerations must turn into conditionally copyable types. + +```swift +enum Optional: ~Copyable { + case none + case some(Wrapped) +} + +extension Optional: Copyable where Wrapped: Copyable {} +``` + +This is no small matter -- every existing use of `Optional` implicitly assumes its copyability, including all its protocol conformances. We need to lift this assumption without breaking source- and (on some platforms) binary compatibility with existing code that relies on it. + +Furthermore, compatibility expectations also go in the reverse direction, as `Optional` has been an unavoidable part of Swift since its initial release. On ABI stable platforms, we therefore also expect that code freshly built with this newly generalized `Optional` type will continue to be able to run on older versions of the Swift Standard Library. At minimum, we expect that all code that uses copyable types would be directly back-deployable. + +Allowing noncopyable use will necessarily involve defining new operations to help dealing with problems that are specific to noncopyable types. However, when doing that, we need to balance the need to help developers who embrace ownership control with the desire to avoid confusing folks when they continue relying on copyability. (These aren't necessarily different groups of people -- we expect developers will often find it useful to generally stay with copyable abstractions, only reaching for ownership control in specific parts of their code.) Retrofitting noncopyable support on existing types risks muddling up their semantics, hurting our desire for progressive disclosure and potentially overwhelming newcomers. + +However, in the particular case of `Optional`, the benefits of making copyability conditional greatly outweigh these drawbacks. Indeed, we don't have much choice but to retrofit `Optional`: we need a common idiom for representing a potentially absent item, shared across all contexts throughout the language. + +For example, introducing a separate version of `Optional` that's dedicated to noncopyable use would not be workable. This would prevent generic functions that want to support noncopyable type arguments from taking or returning "classic" optionals, so generic code would quickly standardize on using the new type, while existing interfaces would be stuck with the original -- causing universal confusion. + +The case for `Result` is less pressing, as it isn't tied as deeply into the language as `Optional` is. However, `Result` serves a similar purpose as `Optional`: it "merely" expands the nil case to explain the absence with an error value, to implement manual error propagation. It therefore makes sense to propose `Result`'s generalization alongside `Optional`: it involves solving effectively the same problems, and applying the same solutions. + +## Proposed Solution + +In this proposal, we extend the following generic types in the Standard Library with support for noncopyable type arguments: + +- `enum Optional` +- `enum Result` +- `struct MemoryLayout` +- `struct UnsafePointer` +- `struct UnsafeMutablePointer` +- `struct UnsafeBufferPointer` +- `struct UnsafeMutableBufferPointer` +- `class ManagedBuffer` +- `struct ManagedBufferPointer` + +`Optional` and `Result` become conditionally copyable, inheriting their copyability from their type arguments. All other types above remain unconditionally copyable, independent of the copyability of their type argument. + +We also update a single standard protocol to allow noncopyable conforming types: + +- `protocol ExpressibleByNilLiteral: ~Copyable` + +Additionally, we generalize the following top-level function families: + +- `func swap(_:_:)` +- `func withExtendedLifetime(_:_:)` +- `func withUnsafeTemporaryAllocation(byteCount:alignment:_:)` +- `func withUnsafeTemporaryAllocation(of:capacity:_:)` +- `func withUnsafePointer(to:_:)` +- `func withUnsafeMutablePointer(to:_:)` +- `func withUnsafeBytes(of:_:)` +- `func withUnsafeMutableBytes(of:_:)` + +We also generalize some low-level generic functions elsewhere in the stdlib that take or return the types above -- such as pointer conversion or rebinding operations. (See below for details.) + +In several example snippets, we'll be using the following (nonexistent) type to illustrate the use of noncopyable types: + +```swift +struct File: ~Copyable, Sendable { + init(opening path: FilePath) throws {...} + mutating func readByte() throws -> UInt8 {...} + consuming func close() throws {...} + deinit {...} +} +``` + +This is for illustrative purposes only -- we're not proposing to add an I/O facility to the stdlib in this document, and we do not expect a hypothetical future I/O feature would actually use this exact API surface. + +The rest of this section presents the principles this proposal follows in generalizing the constructs above. For a detailed list of changes, please see the [Detailed Design](#detailed-design) below. + +### Generalizing function members + +In past Swift versions, the difference between an operation consuming and borrowing its input argument was merely a subtle implementation detail: it was relevant for some performance optimization work, but generally there was little reason to learn or care about it. With noncopyable inputs, the distinction between consuming vs borrowing an argument rises to upmost importance -- it is one of the first things we need to learn when we write code that needs to use noncopyable types, whether we want to design our own APIs or to understand and use APIs provided by others. + +Retrofitting existing generic APIs for noncopyable use involves determining what ownership semantics to apply on their potentially noncopyable input arguments (including the special `self` argument). If the result of a function is noncopyable, then that inherently means the function is passing ownership of its output to its caller. This means that the function cannot keep hold of that value. + +When we cannot assume copyability, we need to carefully distinguish between consuming and borrowing use: functions need to declare this choice up front, for every parameter that isn't guaranteed to be copyable. + +For example, take the existing pointer operation that initializes the addressed location: + +```swift +extension UnsafeMutablePointer { + func initialize(to value: Pointee) { ... } +} +``` + +Semantically, this used to take a _copy_ of its input value, as that's how copyable values get passed to functions. (In this case, the implementation of the function is exposed to clients---it is marked `@inlinable`---so the copying can often be optimized away when the call happens to be the last use of the original instance. However, this depends on optimization heuristics; there is no strict guarantee that it will always happen.) + +With a noncopyable pointee, this will no longer work! The calling convention no longer makes sense. + +To support potentially non-copyable `Pointee` types, this function must now explicitly specify ownership semantics for its input argument. In this case, the operation wants to _take ownership_ of its input, because it needs to move it into its new place at the addressed location in memory. Therefore the function must explicitly consume its input argument: + +```swift +extension UnsafeMutablePointer where Pointee: ~Copyable { + func initialize(to value: consuming Pointee) { ... } +} +``` + +This new function remains source-compatible with the classic definition, except now it can work with noncopyable types. If a copyable value is passed as a `consuming` argument, Swift can pass an implicit copy of it as needed, based on whether or not the caller will need to continue using the original. Importantly, the explicit `consuming` keyword now _guarantees_ that the code will avoid making unnecessary copies when possible, even if `Pointee` happens to be copyable: we no longer rely on optimizer heuristics to avoid unnecessary copying overhead. + +By distinguishing between consuming and borrowing use, we gain more precise control over how our code behaves. The downside, of course, is that we pay for this control by having to think about it -- not only while defining these operations, but also while using them. To call this function with a noncopyable entity, we need to provide it with an item we own, and we need to be willing to let the function consume it. For example, we cannot call it on an instance that we're only borrowing from someone else. + +Preserving source compatibility is great, but unfortunately entirely replacing the old entry point with this new definition would be an ABI breaking change: the new function follows a new calling convention and it is exposed under a different linkage name. Existing binaries will keep linking with the original entry point, and we need to ensure they continue working. To allow this, on ABI stable platforms we continue to expose the old definition as an obsoleted `@usableFromInline internal` function. To allow newly built code to run on previously shipped Standard Library versions, the replacement needs to be defined in a back-deployable manner, such as by using the [`@backDeploy` attribute][SE-0376]. + +[SE-0376]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0376-function-back-deployment.md + +```swift +// Non-normative illustration of an implementation technique. +extension UnsafeMutablePointer where Pointee: ~Copyable { + @backDeployed(before: ...) + public func initialize(to value: consuming Pointee) { ... } +} +extension UnsafeMutablePointer /*where Pointee: Copyable*/ { + @available(swift, obsoleted: 1) + @usableFromInline + internal func initialize(to value: Pointee) { ... } +} +``` + +This way, existing binaries can continue to link with the original entry point, while newly built code will smoothly transition to the new definition.) + +The new function needs to be marked back-deployable, as it is replacing the original copyable version, and as such it needs to have matching availability. The function's implementation is directly embedded into binaries, so this means that in this particular case, newly introduced support for noncopyable use is also expected to "magically" work on older releases. (This will not necessarily extend to all noncopyable generalizations, as not every operation can retroactively learn to deal with noncopyable entities. In particular, older runtimes aren't expected to understand how to perform dynamic operations on noncopyable types (such as looking up metatype instances, performing conformance checks, dealing with existentials, downcasting or reflection); any operation that requires such features is not expected to deploy back without limits.) + +Declaring a function's input parameters `consuming` or `borrowing` can generally be done without tweaking the implementation, as long as the function does not happen to implicitly copy such arguments. If the implementation does rely on implicit copying, then it needs to be corrected to avoid doing that. + +### Generalizing higher-order functions + +For most operations, introducing support for noncopyable use is simply a matter of deciding what ownership rules they should follow and then adding the corresponding annotations. Sometimes the choice between consuming/borrowing use isn't obvious though -- the function may make sense in both flavors. + +Such is often the case with operations that take function arguments. For, example, take `Optional.map`: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws(E) -> U + ) throws(E) -> U? +} +``` + +Is this function supposed to consume or borrow the optional? Looking through existing (copyable) use cases, the answer seems to be both! + +In many cases, `map` is used to _transform_ the wrapped value into some other type, logically consuming it in the process. In some others, `map` is used to _project_ the wrapped value into some other entity, for example by copying a component or some computed property of it. + +The existing `map` name cannot be used to name both flavors, as consuming/borrowing annotations are not involved in overload resolution, so trying to do that would make the `map` name ambiguous. Of course, we could use the above distinction between consuming transformations and borrowing projections to replace `map` with two functions named `transform` and `project`. However, this terminology would be way too subtle, and it would not apply to similar cases elsewhere, such as `map`'s close sibling, `Optional.flatMap`. + +To resolve the ambiguity, we'll probably need to introduce a naming convention, such as to use `consuming` and/or `borrowing` as naming prefixes (as in `consumingMap` or `borrowingFlatMap`), or to invent `consuming` or `borrowing` views and move these operations there (as in `consuming.map` or `borrowing.flatMap`). Some of these choices depend on language features (non-escapable types, stored borrows, read accessors, consuming getters) that do not exist yet. Accordingly, we defer introducing consuming/borrowing higher-order functions until we can gain enough practical experience to make an informed decision. + +Functions like `Optional.map` will therefore keep requiring copyability for now. However, we can and should immediately generalize such functions in a different direction. + +### Generalizing caller-provided return types + +Many of the higher-order functions in the Standard Library are designed to return whatever value is returned by their function arguments. This includes the `Optional.map` function we saw above: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws(E) -> U + ) throws(E) -> U? +} +``` + +The generic return type `U` is implicitly required to be copyable here, which prevents the transformation from returning a noncopyable type: + +```swift +let name: FilePath? = ... +let file: File? = try name.map { try File(opening: $0) } +// error: noncopyable type 'File' cannot be substituted for copyable generic parameter 'U' in 'map' +``` + +Code that uses noncopyable types would hit this limitation surprisingly frequently, and working around it requires annoying and error-prone acrobatics, such as using an inout capture of an optional: + +```swift +let name: FilePath? = ... +var file: File? = nil +try name.map { file = try File(opening: $0) } // OK +``` + +To avoid forcing developers to use such workarounds, we systematically generalize such closure-taking APIs to allow noncopyable result types: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws(E) -> U + ) throws(E) -> U? +} + +let name: FilePath? = ... +let file: File? = try name.map { try File(opening: $0) } // OK +``` + +This is particularly important for interfaces like `ManagedBuffer.withUnsafeMutablePointers`, which are commonly used to access some container type's backing storage. A typical example is inside the implementation of a removal operation where the closure wants to return the item it removed from the container. + +(This generalization of outputs can be safely shipped separate from the consuming/borrowing generalizations of the input side. Doing these generalizations in two separate phases is not expected to cause any issues: it does not prevent getting us to whatever final APIs we want, and it does not introduce any unique compatibility problems that wouldn't also occur if we did both generalizations at the same time.) + +### (Lack of) protocol generalizations + +It would be very much desirable to generalize some of the existing Standard Library protocols for noncopyable use. However, each protocol needs careful consideration that is best deferred to subsequent proposals; therefore, in this particular document, we limit ourselves to generalizing just a single protocol, `ExpressibleByNilLiteral`. + +Generalizing `ExpressibleByNilLiteral` allows our newly noncopyable `Optional` to unconditionally conform to it, so that we can continue to use the `nil` keyword to refer to an empty optional instance, even if it happens to be wrapping a noncopyable type. + +```swift +var document: File? = nil // OK +``` + +All other public protocols in the Standard Library continue to require their conforming types as well as all their associated types to be copyable for now. This includes (but isn't limited to) such basic protocols as `Equatable`, `CustomStringConvertible`, `ExpressibleByArrayLiteral`, `Codable`, and `Sequence`. Some of these are directly generalizable, but most will require considerable design work, which we defer to future proposals. + +Therefore, all other conformances on `Optional` and `Result` will remain conditional on `Wrapped`'s copyability until future proposals. For example, while we'll be able to use the `== nil` form to check if an optional wraps no entity, the full `==` function is leaning on `Equatable` and thus it will only work for copyable types for now: + +```swift +let file: File? = try File(opening: "noncopyable-stdlib-primitives.md") +print(c == nil) // OK, prints "false" +let d: File? = nil +print(c == d) // error: operator function '==' requires that 'File' conform to 'Equatable' +``` + +On the other hand, unsafe pointer types remain unconditionally copyable, so some of their conformances can continue to remain unconditional. `UnsafePointer` and `UnsafeMutablePointer` can remain `Equatable`, `Comparable`, `Strideable` etc. even if their pointee happens to be noncopyable. + +```swift +let p: UnsafePointer = ... +var q: UnsafePointer = ... +print(p == q) // OK +q += 1 // OK +``` + +Unfortunately, the same isn't true for `UnsafeBufferPointer`, whose conformances to the `Collection` protocol hierarchy do not translate at all -- all our current container protocols require a copyable `Element`. + +Unsafe buffer pointers gain much of their core functionality from the `Collection` protocol: for instance, even the idea of accessing an item by subscripting with an integer index comes from that protocol. However, all buffer pointers need a way identify positions in themselves, and so we must nevertheless generalize the `Index` type and its associated index navigation methods and the crucial indexing subscript operation. + +```swift +extension UnsafeBufferPointer where Element: ~Copyable { + typealias Index = Int + var startIndex: Int { get } + var endIndex: Int { get } + var isEmpty: Bool { get } + var count: Int { get } + func index(after i: Int) -> Int + func index(before i: Int) -> Int + ... + subscript(i: Int) -> Element // (unstable accessor not shown) +} +``` + +Note that the generalized indexing subscript cannot provide a regular getter, as that would work by returning a copy of the item -- so the Standard Library currently has to resort to an unstable/unsafe language feature to provide direct borrowing access. (This isn't new, as we previously relied on this scheme to optimize performance; but its use now becomes unavoidable. Defining a stable language feature to implement such accessors is expected to be a topic of a future proposal.) + +While we can generalize the basic container primitives, sadly we need to leave the actual sequence & collection conformances conditional on copyability. Some of the protocol requirements also need to stick with copyability: + +```swift +extension UnsafeBufferPointer: Sequence /* where Element: Copyable */ { + struct Iterator: IteratorProtocol {...} + func makeIterator() -> Iterator {...} +} + +extension UnsafeBufferPointer: RandomAccessCollection /* where Element: Copyable */ { + typealias Indices = Range + typealias SubSequence = Slice> + var indices: Indices { get } + subscript(bounds: Range) -> Slice +} +``` + +For-in loops currently require a `Sequence` conformance, which means it will not yet be possible to iterate over the contents of an unsafe buffer pointer of noncopyable elements using a direct for-in loop. For now, we will need to write manual loops such as this one: + +```swift +let buffer: UnsafeBufferPointer> = ... +for i in buffer.startIndex ..< buffer.endIndex { + buffer[i].add(1, ordering: .sequentiallyConsistent) +} +``` + +This will have to bide us over until we invent new protocols for noncopyable containers. (Of course, that is expected to be the subject of subsequent work; however, we'll first need to introduce nonescapable types and use them to build some fundamental Standard Library constructs.) + +Another item of particular note is the loss of slicing subscript for noncopyable buffer pointers. The original slicing subscript returns a `Slice`, which requires a `Base` that conforms to `Collection`. Therefore, we can only provide the slicing subscript if `Element` happens to be copyable. Slicing buffer pointers is a very common operation, quite crucial for their usability. To make up for this, we propose to add an operation that returns a new standalone buffer over the supplied range of elements: + +```swift +extension UnsafeBufferPointer where Element: ~Copyable { + func extracting(_ bounds: Range) -> Self + func extracting(_ bounds: some RangeExpression) -> Self +} +``` + +The returned buffer does not share indices with the original; its indices start at zero. + +For buffer pointers with noncopyable elements, this operation will be the only (easy) way to split a buffer into small parts: + +```swift +import Synchronization + +// A bank of atomic integers +let bank = UnsafeMutableBufferPointer>.allocate(capacity: 4) +for i in 0 ..< 4 { + bank.initializeElement(at: i, to: Atomic(i)) +} + +let part = bank.extracting(2 ..< 4) +print(part[0].load(ordering: .sequentiallyConsistent)) // Prints "2" +print(part[1].load(ordering: .sequentiallyConsistent)) // Prints "3" + +bank.deinitialize().deallocate() +``` + +For copyable elements, the `extracting` operation is not crucial, but it is still useful: it is effectively a shorthand for slicing the buffer and immediately passing the returned slice to the `UnsafeBufferPointer.init(rebasing:)` initializer. This too is a common idiom, so it makes sense to provide a universally available shorter spelling for it. + +### Unblocking basic construction work + +The changes proposed here are enough to start constructing noncopyable containers, such as this illustrative noncopyable array variant, built entirely around an unsafe buffer pointer: + +```swift +struct Hypoarray: ~Copyable { + private var _storage: UnsafeMutableBufferPointer + private var _count: Int + + init() { + _storage = .init(start: nil, count: 0) + _count = 0 + } + + init(_ element: consuming Element) { + _storage = .allocate(capacity: 1) + _storage.initializeElement(at: 0, to: element) + _count = 1 + } + + deinit { + _storage.extracting(0 ..< count).deinitialize() + _storage.deallocate() + } +} +``` + +See the appendix for a full(er) definition of this sample type, including some of the fundamental array operations. + +Note that this type is presented here only to illustrate the use of the newly enhanced Standard Library; we do not propose to add such a type to the library as part of this proposal. On the other hand, building up a suite of basic noncopyable data structure implementations is naturally expected to be the subject of subsequent future work. + +## Detailed Design + +### `protocol ExpressibleByNilLiteral` + +In this proposal, we limit ourselves to generalizing just one standard protocol, `ExpressibleByNilLiteral`. We lift the requirement that conforming types must be copyable: + +```swift +protocol ExpressibleByNilLiteral: ~Copyable { + init(nilLiteral: ()) +} +``` + +This lets us continue to support the use of `nil` with noncopyable `Optional` types. + +(We do need to eventually generalize additional protocols, of course; as we mentioned above, such work is deferred to future proposals.) + +### `enum Optional` + +The `Optional` enum needs to be generalized to allow wrapping non-copyable types. This requires `Optional` to itself become conditionally copyable. + +```swift +@frozen +enum Optional: ~Copyable { + case none + case some(Wrapped) +} + +extension Optional: Copyable /* where Wrapped: Copyable */ {} +extension Optional: Sendable where Wrapped: ~Copyable & Sendable { } + +extension Optional: ExpressibleByNilLiteral where Wrapped: ~Copyable { + init(nilLiteral: ()) +} + +extension Optional where Wrapped: ~Copyable { + init(_ some: consuming Wrapped) +} +``` + +`Optional`'s `map` and `flatMap` members can be generalized to relax the copyability requirement on their return type: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws(E) -> U + ) throws(E) -> U? + + func flatMap( + _ transform: (Wrapped) throws(E) -> U? + ) throws(E) -> U? +} +``` + +However, these members cannot work on noncopyable optionals, as we have to distinguish between consuming and borrowing variants in that context. Choosing which of these two variants to generalize the existing names, and precisely what notation to use for the remaining variant is deferred to a future proposal. + +The current `unsafelyUnwrapped` property cannot currently be generalized for noncopyable types, so it is also kept restricted to the copyable case. + +[As foreshadowed in SE-0390](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md#noncopyable-optional), it is commonly useful to use noncopyable optionals to allow partial consumption of stored properties in [contexts where that isn't normally allowed][SE-0429]. To support such use, we introduce a brand new mutating member `Optional.take()` that resets `self` to nil, returning its original value: + +```swift +extension Optional where Wrapped: ~Copyable { + mutating func take() -> Self { + let result = consume self + self = nil + return result + } +} +``` + +Having a named operation for this common need establishes it as a universal idiom. + +The standard nil-coalescing `??` operator is updated to explicitly consume its first operand: + +``` +func ?? ( + optional: consuming T?, + defaultValue: @autoclosure () throws -> T +) rethrows -> T + +func ?? ( + optional: consuming T?, + defaultValue: @autoclosure () throws -> T? +) rethrows -> T? +``` + +This matches the behavior of the second argument, where ownership of the default value is passed to the `??` implementation. + +In this initial phase, `Equatable` continues to require conforming types to be copyable, so optionals containing noncopyable types cannot yet be compared for equality. However, `Optional` also provides special support for `== nil` and `!= nil` comparisons whether or not its wrapped type is Equatable. We do generalize this support to allow noncopyable wrapped types: + +```swift +extension Optional where Wrapped: ~Copyable { + static func ==( + lhs: borrowing Wrapped?, + rhs: _OptionalNilComparisonType + ) -> Bool + static func !=( + lhs: borrowing Wrapped?, + rhs: _OptionalNilComparisonType + ) -> Bool + static func ==( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool + static func !=( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool +} +``` + +We are also generalizing the standard `~=` operators to support pattern matching `nil` on noncopyable Optionals. + +```swift +extension Optional where Wrapped: ~Copyable { + static func ~=( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool +} +``` + +(The implementations above currently rely on an unstable `_OptionalNilComparisonType` type to represent a type-agnostic `nil` value. This type and this particular way of implementing `== nil` is an internal implementation detail of the Standard Library that remains subject to change in future versions. These signatures are listed to illustrate the changes we're making; they aren't intended to stabilize this particular implementation.) + +### `enum Result` + +The standard `Result` type similarly needs to be generalized to allow noncopyable `Success` values, itself becoming conditionally noncopyable. + +```swift +@frozen +enum Result: ~Copyable { + case success(Success) + case failure(Failure) +} + +extension Result: Copyable where Success: Copyable {} +extension Result: Sendable where Success: Sendable & ~Copyable {} +``` + +Like with `Optional`, some of `Result`'s existing members can be directly generalized not to require a copyable success type. + +```swift +extension Result where Success: ~Copyable { + init(catching body: () throws(Failure) -> Success) + + consuming func get() throws(Failure) -> Success + + consuming func mapError( + _ transform: (Failure) -> NewFailure + ) -> Result( + _ transform: (Failure) -> Result + ) -> Result +} +``` + +The `mapError` members need to potentially return the `Success` value they originally stored in `self`, so they need to become consuming functions -- we cannot provide any borrowing variants. + +Like we saw with `Optional`, unfortunately this does not apply to members that transform the success value. We can still generalize the type of the result, but not the input: + +```swift +extension Result { + func map( + _ transform: (Success) -> NewSuccess + ) -> Result + + func flatMap( + _ transform: (Success) -> Result + ) -> Result +} +``` + +We defer generalizing the "input side" into borrowing/consuming map/flatMap variants until a future proposal; until then, `map` and `flatMap` continue to require a copyable `Success`. + +### `enum MemoryLayout` + +We extend `MemoryLayout` to allow querying the layout properties of noncopyable types: + +```swift +enum MemoryLayout: Copyable {} + +extension MemoryLayout where T: ~Copyable { + static var size: Int { get } + static var stride: Int { get } + static var alignment: Int { get } + + static func size(ofValue value: borrowing T) -> Int + static func stride(ofValue value: borrowing T) -> Int + static func alignment(ofValue value: borrowing T) -> Int +} +``` + +Note that the current `offset(of:)` member continues to require `T` to be copyable, as key paths do not (currently) support noncopyable targets. + +### Unsafe Pointer Types + +We have two typed unsafe pointer types, `UnsafePointer` and `UnsafeMutablePointer`. To allow building noncopyable constructs, these types need to start supporting noncopyable pointee types. + +```swift +struct UnsafePointer: Copyable +struct UnsafeMutablePointer: Copyable +``` + +Pointers to noncopyable types still need to work like pointers -- in particular, the pointers themselves must always remain copyable. Unlike with `Optional` and `Result`, pointers can therefore continue to unconditionally conform to the `Equatable`, `Hashable`, `Comparable`, `Strideable` and `CVarArg` protocols, as well as the new `AtomicRepresentable`, `AtomicOptionalRepresentable` protocols, regardless of the copyability of their pointee type. + +```swift +extension Unsafe[Mutable]Pointer: Equatable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: Hashable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: Comparable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: Strideable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: CustomDebugStringConvertible where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: CustomReflectable where Pointee: ~Copyable {...} + +// module Synchronization: +extension Unsafe[Mutable]Pointer: AtomicRepresentable where Pointee: ~Copyable {...} +extension Unsafe[Mutable]Pointer: AtomicOptionalRepresentable where Pointee: ~Copyable {...} +``` + +#### Member generalizations + +Most existing members of unsafe pointers adapt directly into the noncopyable world, with some notable exceptions that inherently require copyability: + +- Some operations rely on duplicating or copying pointee values: + - `func initialize(repeating: Pointee, count: Int)` + - `func update(from source: UnsafePointer, count: Int)` + - `func initialize(from source: UnsafePointer, count: Int)` +- Others depend on key paths that have not been generalized for noncopyable types yet: + - `func pointer(to: KeyPath) -> Unsafe[Mutable]Pointer?` + - `func pointer(to: WritableKeyPath) -> UnsafeMutablePointer?` + +These members will continue to require that `Pointee` be copyable. + +All other standard pointer operations lift the copyability requirement: + +- In Swift 5.x, the `pointee` property and the standard offsetting subscript have already been defined with special accessors that provide in-place borrowing or mutating access to instances addressed by the pointer. These translate directly for noncopyable use. + + ```swift + extension Unsafe[Mutable]Pointer where Pointee: ~Copyable { + var pointee: Pointee // (unstable accessors not shown) + subscript(i: Int) -> Pointee // (unstable accessors not shown) + } + ``` + +- Of special note is the `withMemoryRebound` function, which needs to generalized not just for noncopyable pointees, but also for potentially noncopyable target and result types: + + ```swift + extension Unsafe[Mutable]Pointer where Pointee: ~Copyable { + func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: Unsafe[Mutable]Pointer) throws(E) -> Result + ) throws(E) -> Result + } + ``` + +- In previous Swift releases, the existing `UnsafeMutablePointer.initialize(to:)` member used to be defined to (effectively) borrow, rather than consume, its argument. This used to be a minor performance wrinkle, but with noncopyable pointees, it has now become a correctness problem. Therefore, in its newly generalized form, `initialize(to:)` now consumes its argument: + + ```swift + extension UnsafeMutablePointer where Pointee: ~Copyable { + func initialize(to value: consuming Pointee) + } + ``` + + This change does not affect source compatibility with existing copyable call sites, and its ABI impact is mitigated by continuing to expose the original borrowing entry point as an obsolete `@usableFromInline` function. + +- All other pointer members generalize in a straightforward way: + + ```swift + extension Unsafe[Mutable]Pointer where Pointee: ~Copyable { + init(_ other: Self) + init?(_ other: Self?) + init(_ from: OpaquePointer) + init?(_ from: OpaquePointer?) + init?(bitPattern: Int) + init?(bitPattern: UInt) + + func deallocate() + } + + extension UnsafeMutablePointer where Pointee: ~Copyable { + init(mutating other: UnsafePointer) + init?(mutating other: UnsafePointer?) + init(_ other: UnsafeMutablePointer) + init?(_ other: UnsafeMutablePointer?) + + static func allocate(capacity count: Int) -> UnsafeMutablePointer + + func move() -> Pointee + + func moveInitialize(from source: UnsafeMutablePointer, count: Int) + func moveUpdate(from source: UnsafeMutablePointer, count: Int) + + func deinitialize(count: Int) -> UnsafeMutableRawPointer + } + ``` + +#### Related enhancements in other types + +To keep the Standard Library's family of pointer types coherent, we also need to ensure that pointers to noncopyable types continue to interact well with other pointer types in the language, including `UnsafeRawPointer` and `OpaquePointer`: + +- The Swift Standard Library provides heterogeneous pointer comparison operators (`==`, `!=`, `<`, `<=`, `>`, `>=`) that allow comparing any two pointer values, no matter their type. We generalize these to extend their support to comparing pointers with noncopyable pointees. + +- Similarly, the `init(bitPattern:)` initializers on `Int` and `UInt` can work with any pointer type. These initializers must now also extend support to the newly generalized pointer types. + + (Note: We do not list interface updates for the last two enhancements, as they are currently implemented by generalizing the source-unstable `_Pointer` protocol, an implementation detail of the Standard Library.) + +- We need to generalize all generic conversion operations on raw and opaque pointers: + + ```swift + extension OpaquePointer { + init(_ from: Unsafe[Mutable]Pointer) + init?(_ from: Unsafe[Mutable]Pointer?) + } + + extension Unsafe[Mutable]RawPointer { + init(_ other: Unsafe[Mutable]Pointer) + init?(_ other: Unsafe[Mutable]Pointer?) + } + ``` + +- Operations that bind and initialize raw memory to arbitrary types also need to relax their copyability requirements: + + ```swift + extension Unsafe[Mutable]RawPointer { + func bindMemory( + to type: T.Type, capacity count: Int + ) -> Unsafe[Mutable]Pointer + + func withMemoryRebound( + to type: T.Type, + capacity count: Int, + _ body: (_ pointer: Unsafe[Mutable]Pointer) throws(E) -> Result + ) throws(E) -> Result + + func assumingMemoryBound( + to: T.Type + ) -> Unsafe[Mutable]Pointer + + func moveInitializeMemory( + as type: T.Type, from source: UnsafeMutablePointer, count: Int + ) -> UnsafeMutablePointer + } + ``` + +- As well as raw pointer operations that deal with a generic type's memory layout: + + ```swift + extension Unsafe[Mutable]RawPointer { + func alignedUp(for type: T.Type) -> Self + func alignedDown(for type: T.Type) -> Self + } + ``` + +#### Setting temporary pointers on arbitrary entities + +The standard `withUnsafe[Mutable]Pointer` top-level functions allow temporary pointer access to any inout value. These now need to be extended to support noncopyable types: + +```swift +func withUnsafeMutablePointer( + to value: inout T, + _ body: (UnsafeMutablePointer) throws(E) -> Result +) throws(E) -> Result + +func withUnsafePointer( + to value: inout T, + _ body: (UnsafePointer) throws(E) -> Result +) throws(E) -> Result +``` + +Beware that the pointer argument to `body` continues to be valid only during the execution of the function, even if `T` happens to be noncopyable. There is also no guarantee that the address will remain unchanged across repeated calls to `withUnsafe[Mutable]Pointer`. + +This also emphatically applies to the third `withUnsafePointer` variant that provides a temporary pointer to a borrowed instance. This one also gets generalized: + +```swift +func withUnsafePointer( + to value: borrowing T, + _ body: (UnsafePointer) throws(E) -> Result +) throws(E) -> Result +``` + +Borrows aren't exclusive, so it is possible to reentrantly call this function multiple times on the same noncopyable instance. When we do so, it may sometimes appear that the same (ostensibly noncopyable) entity is concurrently occupying multiple different locations in memory: + +```swift +struct Ghost: ~Copyable { + var value: Int +} + +let ghost = Ghost(value: 42) +withUnsafePointer(to: ghost) { p1 in + withUnsafePointer(to: ghost) { p2 in + print(p1 == p2) // Can print false! + } +} +``` + +Do not adjust your set -- this curiosity is inherent in the call-by-value calling convention that Swift normally uses for passing borrowed instances. (Semantically, there is still only a single extant copy, although it can sometimes be smeared over multiple locations.) + +### Unsafe Buffer Pointers + +Like pointers, typed buffer pointers need to start supporting noncopyable elements, without themselves becoming noncopyable. + +```swift +struct UnsafeBufferPointer: Copyable {} +struct UnsafeMutableBufferPointer: Copyable {} +``` + +#### Member generalizations + +Most existing buffer pointer operations directly translate to the noncopyable world: + +- Initializers adapt with no changes: + + ```swift + extension UnsafeBufferPointer where Element: ~Copyable { + init(start: UnsafePointer?, count: Int) + init(_ other: UnsafeMutableBufferPointer) + } + extension UnsafeMutableBufferPointer where Element: ~Copyable { + init(start: UnsafeMutablePointer?, count: Int) + init(mutating other: UnsafeBufferPointer) + } + ``` + +- So do the properties for accessing the components of a buffer pointer: + + ```swift + extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + var baseAddress: Unsafe[Mutable]Pointer? { get } + var count: Int { get } + } + ``` + +- As well as mutable/immutable deallocation: + + ```swift + extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + func deallocate() + } + ``` + +- And most mutating operations: + + ```swift + extension UnsafeMutableBufferPointer where Element: ~Copyable { + static func allocate(capacity count: Int) -> UnsafeMutableBufferPointer + + func moveInitialize(fromContentsOf source: Self) -> Index + + func moveUpdate(fromContentsOf source: Self) -> Index + + func deinitialize() -> UnsafeMutableRawBufferPointer + func deinitializeElement(at index: Index) + func moveElement(from index: Index) -> Element + } + ``` + +Like we saw with `UnsafeMutablePointer`, some operations need to be adjusted: + +- In Swift 5.10, `initializeElement(at:to:)` has an issue where it borrows, rather than consumes, its argument. We need to replace it with a source compatible variant that resolves this: + + ```swift + extension UnsafeMutableBufferPointer where Element: ~Copyable { + func initializeElement(at index: Index, to value: consuming Element) + } + ``` + + To ensure compatibility with current binaries, we also keep providing the old function as an obsolete entry point, like we did for `UnsafeMutablePointer.initialize(to:)`. + +- Memory rebinding operations again need to be generalized along multiple axes: + + ```swift + extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + public func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: Unsafe[Mutable]BufferPointer) throws(E) -> Result + ) throws(E) -> Result + } + ``` + +#### Protocol conformances + +The buffer pointer types also conform to the `Sequence`, `Collection`, `BidirectionalCollection`, `RandomAccessCollection` and `MutableCollection` protocols. We aren't generalizing these protocols in this proposal -- they continue to require copyable `Element` types. Therefore, buffer pointer conformances to these protocols must remain restricted to the pre-existing copyable cases. + +This also affects some related typealiases and nested types: the `UnsafeBufferPointer.Iterator` type and its `SubSequence` and `Indices` typealiases will only exist when `Element` is copyable. A buffer pointer of noncopyable elements is not a sequence, so as of this proposal it cannot be iterated over by a for-in loop. It also doesn't get any of the standard Sequence/Collection algorithms. (We expect to reintroduce these features in the future.) + +However, we do propose to generalize most of the core collection operations, even without carrying the actual conformance: + +```swift +extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + typealias Index = Int + var isEmpty: Bool { get } + var startIndex: Int { get } + var endIndex: Int { get } + func index(after i: Int) -> Int + func formIndex(after i: inout Int) + func index(before i: Int) -> Int + func formIndex(before i: inout Int) + func index(_ i: Int, offsetBy n: Int) -> Int + func index(_ i: Int, offsetBy n: Int, limitedBy limit: Int) -> Int? + func distance(from start: Int, to end: Int) -> Int +} + +extension UnsafeMutableBufferPointer where Element: ~Copyable { + func swapAt(_ i: Int, _ j: Int) +} +``` + +In Swift 5.x, the indexing subscript was already defined with special accessors that support in-place mutating access. To support in-place borrowing access, we can adapt the unstable/unsafe accessors from the unsafe pointers types, to define a subscript with direct support for use with noncopyable elements: + +```swift +extension Unsafe[Mutable]BufferPointer where Element: ~Copyable { + subscript(i: Int) -> Element // (special accessors not shown) +} +``` + +#### Extracting parts of a buffer pointer + +Unfortunately, the slicing subscript cannot be generalized, as its `Slice` return type requires a base container that conforms to `Collection`. + +We therefore propose to add the following new member methods for extracting a standalone buffer that covers a range of indices: + +```swift +extension UnsafeBufferPointer where Element: ~Copyable { + func extracting(_ bounds: Range) -> Self + func extracting(_ bounds: some RangeExpression) -> Self + func extracting(_ bounds: UnboundedRange) -> Self +} + +extension UnsafeMutableBufferPointer where Element: ~Copyable { + func extracting(_ bounds: Range) -> Self + func extracting(_ bounds: some RangeExpression) -> Self + func extracting(_ bounds: UnboundedRange) -> Self +} +``` + +Unlike with slicing, the returned buffer does not share indices with the original -- the result is a regular buffer that has its own 0-based indices. This operation is effectively equivalent to slicing the buffer and then immediately rebasing the slice into a standalone buffer pointer: `buffer.extracting(i ..< j)` produces the same result as the expression `UnsafeBufferPointer(rebasing: buffer[i ..< j])` did in Swift 5.x. + + +#### Exceptions + +There are also some buffer pointer operations that inherently cannot be generalized for noncopyable cases. These include: + +- Operations that require copying elements: + - `func initialize(repeating: Element)` + - `func update(repeating: Element)` +- Operations that operate on sequences or collections of items: + - `func initialize>(from: S) -> (unwritten: S.Iterator, index: Index)` + - `func initialize(fromContentsOf source: some Collection)` + - `func update>(from: S) -> (unwritten: S.Iterator, index: Index)` + - `func update(fromContentsOf: some Collection) -> Index` +- Members that involve buffer pointer slices: + - `init(rebasing slice: Slice>)` + - `func moveInitialize(fromContentsOf source: Slice) -> Index` + - `func moveUpdate(fromContentsOf source: Slice) -> Index` + +These operations are not in any way deprecated; they just continue requiring `Element` to be copyable. (We do expect to introduce noncopyable alternatives for the sequence/collection operations in a subsequent proposal.) + +#### Related enhancements in other types + +To keep the Standard Library's family of pointer types coherent, we also need to generalize some conversion/rebinding/initialization operations on raw buffer pointers: + +```swift +extension Unsafe[Mutable]RawBufferPointer { + init(_ buffer: UnsafeMutableBufferPointer) + init(_ buffer: UnsafeBufferPointer) + + func bindMemory( + to type: T.Type + ) -> Unsafe[Mutable]BufferPointer + + func withMemoryRebound( + to type: T.Type, + _ body: (_ buffer: Unsafe[Mutable]BufferPointer) throws(E) -> Result + ) throws(E) -> Result + + func assumingMemoryBound( + to: T.Type + ) -> Unsafe[Mutable]BufferPointer +} + +extension UnsafeMutableRawBufferPointer { + func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer +} +``` + + +#### Temporary buffer pointers over arbitrary entities + +We also need to similarly generalize the top-level `withUnsafe[Mutable]Pointer` functions that provide temporary buffer pointers over arbitrary values: + +```swift +func withUnsafeMutableBytes( + of value: inout T, + _ body: (UnsafeMutableRawBufferPointer) throws(E) -> Result +) throws(E) -> Result + +func withUnsafeBytes( + of value: inout T, + _ body: (UnsafeRawBufferPointer) throws(E) -> Result +) throws(E) -> Result + +func withUnsafeBytes( + of value: borrowing T, + _ body: (UnsafeRawBufferPointer) throws(E) -> Result +) throws(E) -> Result +``` + +All of these (and especially the borrowing variant) is subject to the same limitations as the original copyable variants: the pointers exposed are only valid for the duration of the function invocation, and multiple executions on the same instance may provide different locations for the same entity. + + +### Temporary Allocation Facility + +The [Standard Library's facility for allocating temporary uninitialozed buffers][SE-0322] needs to be generalized to support allocating storage for noncopyable types, as well as returning a potentially noncopyable type: + +[SE-0322]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0322-temporary-buffers.md + +```swift +func withUnsafeTemporaryAllocation( + byteCount: Int, + alignment: Int, + _ body: (UnsafeMutableRawBufferPointer) throws(E) -> R +) throws(E) -> R + +func withUnsafeTemporaryAllocation( + of type: T.Type, + capacity: Int, + _ body: (UnsafeMutableBufferPointer) throws(E) -> R +) throws(E) -> R + +``` + +### Managed Buffers + +Managed buffers provide a way for Swift container types to dynamically allocate storage for their contents in the form of a managed class reference. + +We generalize managed buffer types to support noncopyable element types, including all of their existing member operations. + +```swift +open class ManagedBuffer { + final var header: Header + init(_doNotCallMe: ()) +} + +@available(*, unavailable) +extension ManagedBuffer: Sendable where Element: ~Copyable {} + +extension ManagedBuffer where Element: ~Copyable { + // All existing members +} + +struct ManagedBufferPointer {...} + +extension ManagedBufferPointer where Element: ~Copyable { + // All existing members +} +``` + +The core `withUnsafeMutablePointer` interfaces are further generalized to allow noncopyable return types: + +```swift +extension ManagedBuffer where Element: ~Copyable { + final func withUnsafeMutablePointerToHeader( + _ body: (UnsafeMutablePointer
) throws(E) -> R + ) throws(E) -> R + + final func withUnsafeMutablePointerToElements( + _ body: (UnsafeMutablePointer) throws(E) -> R + ) throws(E) -> R + + final func withUnsafeMutablePointers( + _ body: ( + UnsafeMutablePointer
, UnsafeMutablePointer + ) throws(E) -> R + ) throws(E) -> R +} +``` + +```swift +extension ManagedBufferPointer where Element: ~Copyable { + func withUnsafeMutablePointerToHeader( + _ body: (UnsafeMutablePointer
) throws(E) -> R + ) throws(E) -> R + + func withUnsafeMutablePointerToElements( + _ body: (UnsafeMutablePointer) throws(E) -> R + ) throws(E) -> R + + func withUnsafeMutablePointers( + _ body: ( + UnsafeMutablePointer
, UnsafeMutablePointer + ) throws(E) -> R + ) throws(E) -> R +} +``` + +Notably, we preserve the requirement that the `Header` type must be copyable for now. It would be desirable to allow noncopyable `Header` types, but preserving compatibility with the stored property `ManagedBuffer.header` requires further work, so it is deferred. (We do not believe this to be a significant obstacle in practice.) + +### Lifetime Management + +The Standard Library offers the `withExtendedLifetime` family of functions to explicitly extend the lifetime of an entity to cover the entire duration of a closure. To support ownership control, we lift the copyability requirement on both the item whose lifetime is being extended, and for the return type of the function argument: + +```swift +func withExtendedLifetime( + _ x: borrowing T, + _ body: () throws(E) -> Result +) throws(E) -> Result +``` + +There exists a second variant of `withExtendedLifetime` whose function argument is passed the entity whose lifetime is being extended. This variant is less frequently used, but it still makes sense to generalize this to pass a borrowed instance: + +```swift +func withExtendedLifetime( + _ x: borrowing T, + _ body: (borrowing T) throws(E) -> Result +) throws(E) -> Result +``` + +### Swapping and exchanging items + +We have a standalone `swap` function that swaps the values of two `inout` values. We propose to generalize this operation to lift its copyability requirement. This is a good opportunity to make use of the new ownership control features to greatly simplify its implementation: + +```swift +func swap(_ a: inout T, _ b: inout T) { + let tmp = consume a + a = consume b + b = consume tmp +} +``` + +We also propose to add a new variant of this same operation that takes a single `inout` value, setting it to a given value and returning the original: + +```swift +public func exchange( + _ value: inout T, + with newValue: consuming T +) -> T { + var oldValue = consume value + value = consume newValue + return oldValue +} +``` + +This is a nonatomic analogue of the `exchange` operation on `struct Atomic`. This is a commonly invoked idiom, and having a standard operation for it will reduce the need to reinvent it from scratch with each use. (Thereby eliminating a potential source of errors, and improving readability.) By using `exchange`, we can avoid the need to manually introduce a second `inout` binding just to be able to invoke `swap`. + +## Source compatibility + +This proposal is heavily built on the assumption that removing the assumption of copyability on these constructs will not break existing code that relies on it. This is largely the case, although there are subtle cases where these generalizations break code that relies on shadowing standard declarations. + +For instance, code that used to substitute their own definition of `Optional.map` (or any other newly generalized function) in place of the stdlib's official definition may find that their declaration is no longer considered to shadow the original: + +```swift +extension Optional { + func map( + _ transform: (Wrapped) throws -> U + ) rethrows -> U? { + print("Hello from map!") + switch self { + case .some(let y): + return .some(try transform(y)) + case .none: + return .none + } + } +} + +let foo: Int? = 42 +foo.map { $0 + 1 } // error: ambiguous use of 'map' +``` + +The new `map` uses typed throws and it allows noncopyable return types, rendering it different enough to make this substitution no longer shadow the original. This makes such generalizations technically source breaking; however the breakage is similar in nature and severity as a source break that can arise from new API additions that happen to clash with preexisting extensions of similar names defined outside the Standard Library. If such issues prove harmful in practice, we can subsequently amend Swift's shadowing rules to ignore differences in throwing and noncopyability. + +## ABI compatibility + +We limited the changes proposed so that we allow maintaining full backward compatibility with existing binaries. + +Adding support for noncopyable type parameters generally changes linker-level mangled symbol names in emitted code, which would break ABI -- we avoid this either by continuing to ship the original function definitions as obsoleted `@usableFromInline internal` functions, or by overriding mangling to ignore `~Copyable` (using an unstable `@_preInverseGenerics` attribute). + +We also provide a measure of forward compatibility -- newly built code that calls newly generalized functions will continue to remain compatible with previously shipped versions of the Standard Library. This naturally must apply to the preexisting copyable cases, but it also extends to noncopyable use: the newly generalized generic operations are generally expected to work on older Swift runtime environments. Of course, older runtimes do not understand noncopyable generics (or even noncopyable types in general), so features that rely on runtime dynamism will come with a stricter deployment limit. (The feature set we propose in this document is not expected to hit this.) + +The `Optional` and `Result` types that shipped in previous versions of the Standard Library were naturally built with the assumption of copyability, but they tended to avoid making unnecessary copies, which means they are mostly expected to be also "magically" compatible with noncopyable use. (It is okay to break an assumption that was never actually relied on.) The places where we preserved mangling are the places where we think this applies -- we expect newly built code that invokes the old implementations will still run fine. (If we missed a case where an earlier implementation did rely on copying or runtime dynamism, we can correct it at any point by switching to the `@backDeplpoyed`/`@usableFromInline` implementation pattern.) + +## Alternatives Considered + +The primary alternative is to delay this work until it becomes possible to express more of the functionality that is deferred by this proposal. However, this would leave noncopyable types in a limbo state, where the language ships with rich functionality to support them, but the core Standard Library continues to treat them as second class entities. + +The inability to apply unsafe pointer APIs to noncopyable types would be a particularly severe obstacle to practical adoption -- it is tricky to fully embrace ownership control if we have no way to dynamically allocate storage for noncopyable entities. + +Avoiding the use of `Optional` is a similarly severe API design issue, with no elegant solutions. Forcing adopters of ownership control to define custom `Optional` types has proved impractical beyond simple throwaway prototypes; it's better to have a standard solution. + +We do not consider the generalization of `Result` to be anywhere near as important as `Optional`, although it does provide a standard way to implement manual error propagation. However, as it is a close relative to `Optional`, it seems undesirable to defer its generalization. + +### Omitting `UnsafeBufferPointer` + +`UnsafeBufferPointer` conforms to `Collection`, and it relies on the standard `Slice` type for its `SubSequence` concept. Neither `Collection` nor `Slice` can be directly generalized for noncopyable elements, and so these conformances need to continue require copyable elements. + +Given that buffer pointers are essentially useless without an idea of an index (which comes from `Collection`), we considered omitting them from this proposal, deferring their generalization until we have protocols for noncopyable container types. + +However, in practice, this would not be acceptable: the buffer pointer is Swift's native way to represent a region of direct memory, and we urgently need to enable dealing with memory regions that contain noncopyable instances. Leaving buffer pointers ungeneralized would strongly encourage Swift code to start passing around base pointers and counts as distinct items, which would be a significant step backwards -- we must avoid training Swift developers to do that. (We'd also lose the ability to generalize the `withUnsafeTemporaryAllocation` function, which is built on top of buffer pointers.) + +Therefore, this proposal generalizes buffer pointers, including the parts of `Collection` that we strongly believe will directly translate to noncopyable containers (the basic concept of an index, the index navigation members and the indexing subscript). + +### Alternatives to `UnsafeBufferPointer.extracting()` + +A different concern arises with buffer pointer slices. Regrettably, it seems we have to give up on the `buffer[i.., in buffer: UnsafeBufferPointer) +} + +// Usage: +UnsafeMutableBufferPointer(rebasing: i ..< j, in: buffer) +``` + +This easily fits into Swift API design conventions, but it doesn't feel like a good enough solution in practice. Specifically, it suffers from two distinct (but related) problems: + +1. It remains just as verbose, inconvenient and non-intuitive as the original rebasing initializer; and we have considered that a significant problem even in the copyable case. + + + + (Indeed, a large part of [SE-0370][SE-370-Slice] was dedicated to reducing the need to directly invoke this initializer, by cleverly extending the `Slice` type with direct methods that [hide the `init(rebasing:)` call](https://github.com/apple/swift/blob/swift-5.10-RELEASE/stdlib/public/core/UnsafeBufferPointerSlice.swift#L699-L702 +). This is very helpful, but in exchange for simplifying use sites, we've made it more difficult to define custom operations: each operation has to be defined on both the buffer pointer and the slice type, and the latter requires advanced generics trickery. Of course, none of this work helps the noncopyable case, as `Slice` does not translate there -- so we get back to where we started.) + +[SE-370-Slice]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0370-pointer-family-initialization-improvements.md#slices-of-bufferpointer + + + +2. The new initializer would also apply to the copyable case, but it would serve no discernible purpose in that context, other than to increase confusion. + +The solution we propose is to make the new operation a regular member function. This solves the first problem: `buffer.extracting(i..: ~Copyable, ~Escapable { + case none + case some(Wrapped) +} + +extension Optional: Copyable where Wrapped: ~Escapable {} +extension Optional: Escapable where Wrapped: ~Copyable {} +extension Optional: Sendable where Wrapped: ~Copyable & ~Escapable & Sendable {} + +extension Optional where Wrapped: ~Copyable & ~Escapable { + public init(_ some: consuming Wrapped) dependsOn(some) -> Self { + self = .some(some) + } +} +``` + +It is likely that we will want to generalize `MemoryLayout` as well. Allowing unsafe pointers to address non-escapable types is not nearly as straightforward, but it's possible we'll need to tackle that, too. + +### Generalizing higher-order functions + +This proposal does not allow `map` or `flatMap` to be called on noncopyable `Optional` or `Result` types yet, to avoid prematurely establishing a pattern before it becomes possible to express better solutions. + +As detailed in the [Proposed Solution]((#generalizing-higher-order-functions)) section, this is mostly a naming/presentation problem: we need distinct notations for the `map` that consumes `self` vs. the variant that merely borrows it. + +One straightforward idea is to simply use `consuming` and `borrowing` as naming prefixes: + +```swift +extension Optional where Wrapped: ~Copyable { + consuming func consumingMap( + _ transform: (consuming Wrapped) throws(E) -> U + ) throws(E) -> U? + + borrowing func borrowingMap( + _ transform: (borrowing Wrapped) throws(E) -> U + ) throws(E) -> U? +} +``` + +This is a somewhat verbose choice, but it makes the choice eminently clear at point of use: + +```swift +struct Wrapper: ~Copyable { + var value: T +} + +let v: Wrapper? +print(v.borrowingMap { $0.value }) + +let w: Wrapper? +let file = try v.consumingMap { try File(opening: $0) } +``` + +The primary drawback of this simple solution is that developers working with classic (i.e. copyable) `Optional` values would now be faced with three separate APIs for what is (from their viewpoint) the same operation. Making a distinction between guaranteed-consuming and guaranteed-borrowing transformations is not entirely pointless even in the copyable case, but it is mostly a nitpicky performance detail that wouldn't otherwise merit any new API additions. However, the distinction is crucial for noncopyable use, and that may excuse the new variations even if they mean additional noise for the classic copyable cases. + +A similar idea is to introduce `consuming` and `borrowing` views, and to move the ownership-aware operations into them, leaving us with the notations `v.borrowing.map { $0.value }` or `v.consuming.map { try File(opening: $0) }`. These are also eminently readable, and they would also be a good spiritual fit with the `.lazy` sequence view we already have. They also help with the noise issue, as the nitpicky variants with explicit ownership annotations would all get hidden away in views dedicated to ownership control. + +The idea of a "consuming view" is a bit of a stretch, as it doesn't seem particularly useful outside of this context; but a "borrowing view" certainly would have merit on its own -- it would be a type that consists of a "borrow" of an instance of some other type, which would be an independently useful construct. (E.g., it would allow us to generalize `Slice` into a `BorrowingSlice` while keeping it generic over the base container.) + +Therefore, the best choice may be to introduce the idea of a `borrowing` view (returning a standard `Borrow` (or `Ref`) type), but to avoid introducing a `consuming` view, preferring to instead generalize the existing `map`/`flatMap`/`filter`/`reduce` etc functions in the consuming sense. So `v.map { ... }` would be (implicitly) consuming, while `v.borrowing.map { ... }` would be explicitly borrowing. + +It isn't currently possible to implement generic borrowing views, as structs can only contain owned instances of another type, not borrowed ones. Therefore, we need to delay work on consuming/borrowing higher-order functions until it becomes possible to express such a thing. (We could implement the `consumingMap` and `borrowingMap` naming convention right now, but it seems likely that we'd regret that when it becomes possible to express the borrowing view concept.) + +### Generalizing `Optional.unsafelyUnwrapped` + +The `unsafelyUnwrapped` property of `Optional` implements an unsafe variant of the safe force-unwrap operation that is built into Swift (denoted `!`). (This property is _unsafe_ because it does not guarantee to check if the optional is empty before attempting to extract its wrapped value. Trying to access a value that isn't there is undefined behavior.) + +This proposal keeps this property in its original form, so it will be only available if `Wrapped` is copyable. + +Ideally, `unsafelyUnwrapped` would be generalized to follow the same adaptive behavior as the force-unwrap form, allowing both consuming and borrowing use. + +To achieve this, Swift would need to implement the following three enhancements: + +1. Provide a way to define a (coroutine based) borrowing accessor on a computed property +2. Provide a way to define an accessor on a computed property that consumes `self` (i.e., a consuming getter). +3. Allow these two accessors to coexist within the same property, with the language inferring which one to use based on usage context. + +Generalizing `unsafelyUnwrapped` needs to be deferred either until these become possible or until we decide not to do them. + +```swift +// Illustration; this is not real Swift +extension Optional where Wrapped: ~Copyable { + var unsafelyUnwrapped: Wrapped { + consuming get { ... } + read { ... } + modify { ... } // Let's throw this in the mix as well + } +} +``` + +In the meantime, we considered adding a separate `unsafeUnwrap()` member to provide a separate solution for point 2 above: + +```swift +extension Optional where Wrapped: ~Copyable { + consuming func unsafeUnwrap() -> Wrapped +} +``` + +However, if we do end up getting these enhancements, then this new function would become an unnecessary addition. As this is a rather obscure/niche operation, it doesn't seem worth this trouble. + +### Generalized managed buffer headers + +This proposal lifts the copyability requirement on `ManagedBuffer`'s `Element` type, but it continues to require `Header` to be copyable. + +Of course, it would be desirable to lift this requirement, too. Unfortunately, `ManagedBuffer` exposes the public (stored) property `header`, and lifting the copyability requirement would break this property's (implicit) ABI for low-level access. Until we find a way to mitigate this problem, we cannot generalize stored properties to remove the assumption of copyability; therefore, we need to postpone generalizing `Header`. + +Requiring a copyable `Header` does not appear to be a significant hurdle in most use cases, so it seems preferable to leave time to design a proper solution rather than attempting to ship a quick stopgap fix that may prove to be incomplete. + +### Additional raw pointer operations + +`Unsafe[Mutable]RawPointer` includes the `load(fromByteOffset:as:)` operation that directly returns a copy an instance of an arbitrary type at the indicated location. We kept this restricted to copyable types, and we refrained from providing noncopyable equivalents, such as the closure-based member below: + +```swift +extension Unsafe[Mutable]RawPointer { + func withValue( + atByteOffset offset: Int = 0, + as type: T.Type, + _ body: (borrowing T) throws(E) -> Result + ) throws(E) -> Result +} +``` + +We also do not provide a mutating operation that consumes an instance at a particular offset: + +```swift +extension UnsafeMutableRawPointer { + func move( + fromByteOffset offset: Int = 0, + as type: T.Type + ) -> T +} +``` + +We omitted these, as it is unclear if these would be the best ways to express these. For now, we instead recommend explicitly binding memory and using `Unsafe[Mutable]Pointer` operations. + +### Protocol generalizations + +[As noted above](#lack-of-protocol-generalizations), this proposal leaves most standard protocols as is, deferring their generalizations to subsequent future work. The single protocol we do generalize is `ExpressibleByNilLiteral` -- the `nil` syntax is so closely associated with the `Optional` type that it would not have been reasonable to omit it. + +This of course is not tenable; we expect that many (or even most) of our standard protocols will need to eventually get generalized for noncopyable use. + +For some protocols, this work is relatively straightforward. For example, we expect that generalizing `Equatable`, `Hashable` and `Comparable` would not be much of a technical challenge -- however, it will involve overhauling/refining `Equatable`'s semantic requirements, which I do not expect to be an easy process. (`Equatable` currently requires that "equality implies substitutability"; if the two equal instances happen to be noncopyable, such unqualified, absolute statements no longer seem tenable.) The `RawRepresentable` protocol is also in this category. + +In other cases, the generalization fundamentally requires additional language enhancements. For example, we may want to consider allowing noncopyable `Error` types -- but that implies that we'll also want to throw and catch noncopyable errors, and that will require a bit more work than adding a `~Copyable` clause on the protocol. It makes sense to defer generalizing the protocol until we decide to do this; if/when we do, the generalizations of `Result` can and should be part of the associated discussion and proposal. Another example is `ExpressibleByArrayLiteral`, which is currently built around an initializer with a variadic parameter -- to generalize it, we need to either figure out how to generalize those, or we need to design some alternative interface. + +In a third category of cases, the existing protocols make heavy use of copyability to (implicitly) unify concerns that need stay distinct when we introduce ownership control. Retroactively untangling these concerns is going to be difficult at best -- and sometimes it may in fact prove impractical. For instance, the current `Sequence` protocol is shaped like a consuming construct: `makeIterator` semantically consumes the sequence, and `Iterator.next()` passes ownership of the elements to its caller. However, the documentation of `Sequence` explicitly allows conforming types to implement multipass/nondestructive behavior, and it in fact it _requires_ `Collection` types to do precisely that. By definition, a consuming sequence cannot be multipass; such sequences are borrowing by nature. To support noncopyable elements, we'll need to introduce distinct abstractions for borrowing and consuming sequences. Generalizing the existing `Sequence` in either of these directions seems fraught with peril. + +Each of these protocol generalizations will require effort that's _at least_ comparable in complexity to this proposal; so it makes sense to consider them separately, in a series of future proposals. + +### Additional future work + +Fully supporting ownership control and noncopyable types will require overhauling much of the existing Standard Library. + +This includes generalizing dynamic runtime operations -- a huge area that includes facilities such as isa checks, downcasts, existentials, reflection, key paths, etc. (For instance, updating `print()` to fully support printing noncopyable types is likely to require many of these dynamic features to work.) + +On the way to generalizing the Standard Library's current sequence and collection abstractions, we'll also need to implement a variety of alternatives to the existing copy-on-write collection types, `Array`, `Set`, `Dictionary`, `String`, etc, providing clients direct control over (runtime and memory) performance: consider a fixed-capacity array type, or a stack-allocated dictionary construct. + +Many of these depend on future language enhancements, and as such they will be developed alongside those. + +## Appendix: `struct Hypoarray` + +Hypoarray is a simple noncopyable generic struct that is a very thin, safe wrapper around a piece of directly allocated memory. It is presented here as an illustration of the pointer improvements introduced in this document. + +This section is not normative: we are not proposing to add a `Hypoarray` type to the Standard Library. However, it illustrates the use of the proposed Standard Library extensions, and it does serve as a first prototype for a potential future addition. + +This type operates on a lower level of abstraction than the standard `Array` type. "Hypo" is greek for "under", so "hypoarray" is an apt name for such a construct. (In fact, if we started anew, the existing `Array` would potentially be built on top of such a construct.) + +A hypoarray is like an `Array` without the implicit copy-on-write machinery: it is still dynamically allocated, and it can still implicitly resize itself as needed, but it replaces copy-on-write behavior with strict ownership control. Its storage is always uniquely held, so every mutation can be done in place, resulting in more predictable performance. (Although implicit reallocations can still result in unexpected spikes of latency! To get rid of those, we'd need to introduce an even lower-level array variant that has a fixed capacity. We'll leave that as an exercise to the reader for now.) + +A hypoarray consists of a dynamically allocated storage buffer (of variable capacity) and an integer count that specifies how many initialized elements it contains. The elements of the array are all compacted at the beginning of storage, with any remaining slots serving as free capacity for future additions. + +```swift +struct Hypoarray: ~Copyable { + private var _storage: UnsafeMutableBufferPointer + private var _count: Int +``` + +The buffer's count is the current capacity of the hypoarray. We'll need to keep referring to it elsewhere, so it makes sense to introduce a name for it early on: + +```swift + var capacity: Int { _storage.count } +``` + +Initializing an empty array can be done by simply setting up an empty buffer, and setting the count to zero. + +```swift + init() { + _storage = .init(start: nil, count: 0) + _count = 0 + } +``` + +That wasn't very interesting, so to spruce things up, we can also provide a single-element initializer that needs to actually allocate and initialize some memory: + +```swift + init(_ element: consuming Element) { + _storage = .allocate(capacity: 1) + _storage.initializeElement(at: 0, to: element) + _count = 1 + } +``` + +This nontrivial initializer takes ownership of the element it is given, so naturally it has to be declared to consume its argument. + +(Of course, we will eventually also want to have an initializer that can take any sequence of elements; however, this needs the idea a sequence type that produces consumable items, and we do not yet have a protocol that could express that. The `Sequence` we currently have requires its `Element` to be copyable, and it inherently combines borrowing and consuming iteration into a single, convenient abstraction. Sadly it does not directly translate to noncopyable use.) + +When the array is destroyed, we need to properly deinitialize its elements and deallocate its storage. To do this, we need to define a deinitializer: + +```swift + deinit { + _storage.extracting(0 ..< count).deinitialize() + _storage.deallocate() + } +} +``` + +Note the use of the new `extracting` operation to get a buffer pointer that consists of just the slots that have been populated. We cannot call `_storage.deinitialize()` as it isn't necessarily fully initialized; and we also cannot use the classic slicing operation `_storage[.. Int { i + 1 } + func index(before i: Int) -> Int { i - 1 } + func distance(from start: Int, to end: Int) -> Int { end - start } + // etc. +} +``` + +The most fundamental `Collection` operation is probably its indexing subscript for accessing a particular element. Obviously, we need hypoarray to provide this functionality, too. + +Unfortunately, subscripts (and computed properties) cannot currently return noncopyable results without transferring ownership of the result to the caller. + +```swift +// Illustration: an array of atomic integers +import Synchronization +let array = Hypoarray(Atomic(42)) +print(array[0]) // This cannot work! +``` + +The subscript getter would need to move the item out of the array to give ownership to the caller, which we do not want. Getter accessors will need to be generalized into a coroutine-based read accessor that supports in-place borrowing access. (And setters need to be generalized to allow in-place mutating access.) [Introducing such accessors is still in progress][_modify], so for now, the best we can do is to provide closure-based access methods: + +[_modify]: https://forums.swift.org/t/modify-accessors/31872 + +```swift +extension Hypoarray where Element: ~Copyable { + func borrowElement ( + at index: Int, + by body: (borrowing Element) throws(E) -> R + ) throws(E) -> R { + precondition(index >= 0 && index < _count) + return try body(_storage[index]) + } + + mutating func updateElement ( + at index: Int, + by body: (inout Element) throws(E) -> R + ) throws(E) -> R { + precondition(index >= 0 && index < _count) + return try body(&_storage[index]) + } +} +``` + +These are quite clumsy, but they do work safely, and they provide in-place borrowing and mutating access to any element in a hypoarray, without having to change its ownership. + +```swift +// Example usage: +var array = Hypoarray(42) +array.updateElement(at: 0) { $0 += 1 } +array.borrowElement(at: 0) { print($0) } // Prints "43" +``` + +[[Aside: A future language extension will hopefully allow us to replace these with the subscript we actually want to write, along the lines of this hypothetical example: + +```swift +// This isn't real Swift yet: +extension Hypoarray where Element: ~Copyable { + subscript(position: Int) -> Element { + read { + precondition(position >= 0 && position < _count) + try yield _storage[position] + } + modify { + precondition(position >= 0 && position < _count) + try yield &_storage[position] + } + } +} +``` + +```swift +// Example usage: +var array = Hypoarray(42) +array[0] += 1 +print(array[0]) // Prints "43" +``` + +Note that the proposed `UnsafeMutableBufferPointer` changes already include a subscript that allows in-place borrowing and mutating use. However, the solution used there is tied to low-level unsafe pointer semantics that would not directly translate to a higher-level type like `Hypoarray`.]] + +It would be desirable to allow iteration over `Hypoarray` instances. Unfortunately, Swift's `for in` construct currently relies on `protocol Sequence`, and that protocol doesn't support noncopyable use. (Not only does it require copyable conforming types and copyable `Element`s, but its iterator is also defined to give ownership of returned elements to the caller; that is to say, it is shaped like a _consuming_ construct, not a _borrowing_ one.) Introducing a mechanism for borrowing iteration, and retooling `for in` loops to allow such use is future work. While that work is in progress, we can of course still manually iterate over the contents of a hypoarray by using its indices: + +```swift +// Example usage: +var array: Hypoarray = ... +for i in array.startIndex ..< array.endIndex { // a.k.a. 0 ..< array.count + array.borrowElement(at: i) { print($0) } +} +``` + +Not having noncopyable container protocols also means that `Hypoarray` cannot conform to any, so subsequently it will not get any of the standard generic container algorithms for free: there is no `firstIndex(of:)`, there is no `map`, no `filter`, no slicing, no `sort`, no `reverse`. Indeed, many of these standard algorithms expect to work on `Equatable` or `Comparable` items, and those protocols are also yet to be generalized. + +Okay, so all we have is `borrowElement` and `updateElement`, for borrowing and mutating access. What about consuming access, though? + +Consuming an item of an array at a particular index would require either removing the item from the array, or destroying and discarding the rest of the array. Neither of these looks desirable as a primitive operation for accessing an element. However, we do expect arrays to provide a named operation for removing items, `remove(at:)`. This operation is easily implementable on `Hypoarray`: + +```swift +extension Hypoarray where Element: ~Copyable { + @discardableResult + mutating func remove(at index: Int) -> Element { + precondition(index >= 0 && index < count) + let old = _storage.moveElement(from: index) + let source = _storage.extracting(index + 1 ..< count) + let target = _storage.extracting(index ..< count - 1) + let i = target.moveInitialize(fromContentsOf: source) + assert(i == target.endIndex) + _count -= 1 + return old + } +} +``` + +Note how this moves the removed element out of the array, so it can legitimately give ownership of it to the caller. Following our preexisting convention, the result of `remove(at:)` is marked discardable; if the caller decides to discard it, then the removed item immediately gets destroyed, as expected. + +(Implementing the classic `removeSubrange(_: some RangeExpression)` operation is left as an exercise for the reader.) + +Okay, so now we know how to create simple single-element hypoarrays, how to access their contents, and we are even able to remove elements from them. How do we add new elements, though? + +Hypoarray is supposed to implement a dynamically resizing array, so insertions generally need to be able to expand storage. Let's tackle this sub-problem first, by implementing `reserveCapacity`: + +```swift +extension Hypoarray where Element: ~Copyable { + mutating func reserveCapacity(_ n: Int) { + guard capacity < n else { return } + let newStorage: UnsafeMutableBufferPointer = .allocate(capacity: n) + let source = _storage.extracting(0 ..< count) + let i = newStorage.moveInitialize(fromContentsOf: source) + assert(i == count) + _storage.deallocate() + _storage = newStorage + } +} +``` + +Note again the use of `extracting` to operate on parts of a buffer -- in this case, we use it to move initialized items between the two allocations. + +We want insertions to have amortized O(1) complexity, so they need to be careful about the rate at which they grow the array's storage. In this simple illustration, we'll use a geometric growth factor of 2, so that each reallocation will at least double the capacity of the array: + +```swift +extension Hypoarray where Element: ~Copyable { + mutating func _ensureFreeCapacity(_ minimumCapacity: Int) { + guard capacity < _count + minimumCapacity else { return } + reserveCapacity(max(_count + minimumCapacity, 2 * capacity)) + } +} +``` + +With that done, we can finally implement insertions, starting with the `append` operation. Its implementation is fairly straightforward: + +```swift +extension Hypoarray where Element: ~Copyable { + mutating func append(_ item: consuming Element) { + _ensureFreeCapacity(1) + _storage.initializeElement(at: _count, to: item) + _count += 1 + } +} +``` + +Inserting at a particular index is complicated by the need to make room for the new item, but it's not that tricky, either: + +```swift +extension Hypoarray where Element: ~Copyable { + mutating func insert(_ item: consuming Element, at index: Int) { + precondition(index >= 0 && index <= count) + _ensureFreeCapacity(1) + if index < count { + let source = _storage.extracting(index ..< count) + let target = _storage.extracting(index + 1 ..< count + 1) + target.moveInitialize(fromContentsOf: source) + } + _storage.initializeElement(at: index, to: item) + _count += 1 + } +} +``` + +```swift +// Example usage: +var array = Hypoarray() +for i in 0 ..< 10 { + array.insert(i, at: 0) +} +// array now consists of 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 +``` + +Without noncopyable container protocols, we cannot yet implement `append(contentsOf:)`, `insert(contentsOf:)`, `replaceSubrange` operations. But we can still provide classic `Sequence`/`Collection`-based operations in cases where `Element` happens to copyable: + +```swift +extension Hypoarray { + mutating func append(contentsOf items: some Sequence) { + for item in items { + append(item) + } + } +} +``` + +Note how this extension omits the suppression of element copyability -- it does not have a `where Element: ~Copyable` clause. This means that the extension only applies if `Element` is copyable. + +These operations give us all primitive operations we expect an array type to provide. Of course, the `Hypoarray` we have now created is just the very first draft of a future dynamically sized noncopyable array type. There is plenty of work left: we need to add more operations; we need to implement noncopyable variants of more data structures; we need to define the general shape of a noncopyable container; we need to populate that shape with a family of standard generic algorithms. Implicit resizing is not always appropriate in memory-starved or low-latency applications, so for those use cases we also need to design data structure variants that work within some fixed storage capacity (or even a fixed count). We may want the backing store to be allocated dynamically, like we've seen, or we may want it to become part of the construct's representation ("inline storage"); perhaps we want to allocate storage on the stack, or statically reserve space for a global variable at compile time. We expect future work will tackle all these tasks, and plenty more. diff --git a/proposals/0438-metatype-keypath.md b/proposals/0438-metatype-keypath.md new file mode 100644 index 0000000000..e12bd4afb6 --- /dev/null +++ b/proposals/0438-metatype-keypath.md @@ -0,0 +1,136 @@ +# Metatype Keypaths + +* Proposal: [SE-0438](0438-metatype-keypath.md) +* Authors: [Amritpan Kaur](https://github.com/amritpan), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift#73242](https://github.com/apple/swift/pull/73242) +* Review: ([pitch](https://forums.swift.org/t/pitch-metatype-keypaths/70767)) ([review](https://forums.swift.org/t/se-0438-metatype-keypaths/72172)) ([acceptance](https://forums.swift.org/t/accepted-se-0438-metatype-keypaths/72878)) + +## Introduction + +Key path expressions access properties dynamically. They are declared with a concrete root type and one or more key path components that define a path to a resulting value via the type’s properties, subscripts, optional-chaining expressions, forced unwrapped expressions, or self. This proposal expands key path expression access to include static properties of a type, i.e., metatype keypaths. + +## Motivation + +Metatype keypaths were briefly explored in the pitch for [SE-0254](https://forums.swift.org/t/pitch-static-and-class-subscripts/21850) and the [proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0254-static-subscripts.md#metatype-key-paths) later recommended them as a future direction. Allowing key path expressions to directly refer to static properties has also been discussed on the Swift Forums for database lookups when used [in conjunction with @dynamicMemberLookup](https://forums.swift.org/t/dynamic-key-path-member-lookup-cannot-refer-to-static-member/30212) and as a way to avoid verbose hacks like [referring to a static property through another computed property](https://forums.swift.org/t/key-path-cannot-refer-to-static-member/28055). Supporting metatype keypaths in the Swift language will address these challenges and improve language semantics. + +## Proposed solution + +We propose to allow keypath expressions to define a reference to static properties. The following usage, which currently generates a compiler error, will be allowed as valid Swift code. + +```swift +struct Bee { + static let name = "honeybee" +} + +let kp = \Bee.Type.name +``` + +## Detailed design + +### Metatype syntax + +Keypath expressions where the first component refers to a static property will include `.Type` on their root types stated in the key path contextual type or in the key path literal. For example: + +```swift +struct Bee { + static let name = "honeybee" +} + +let kpWithContextualType: KeyPath = \.name // key path contextual root type of Bee.Type +let kpWithLiteral = \Bee.Type.name // key path literal \Bee.Type +``` + +Attempting to write the above metatype keypath without including `.Type will trigger an error diagnostic: + +```swift +let kpWithLiteral = \Bee.name // error: static member 'name' cannot be used on instance of type 'Bee' +``` + +Keypath expressions where the component referencing a static property is not the first component do not require `.Type`: +```swift +struct Species { + static let isNative = true +} + +struct Wasp { + var species: Species.Type {Species.self} +} + +let kpSecondComponentIsStatic = \Wasp.species.isNative +``` +### Access semantics + +Immutable static properties will form the read-only keypaths just like immutable instance properties. +```swift +struct Tip { + static let isIncluded = True + let isVoluntary = False +} + +let kpStaticImmutable: KeyPath = \.isIncluded +let kpInstanceImmutable: KeyPath = \.isVoluntary +``` +However, unlike instance members, keypaths to mutable static properties will always conform to `ReferenceWritableKeyPath` because metatypes are reference types. +```swift +struct Tip { + static var total = 0 + var flatRate = 20 +} + +let kpStaticMutable: ReferenceWriteableKeyPath = \.total +let kpInstanceMutable: WriteableKeyPath = \.flatRate +``` +## Effect on source compatibility + +This feature breaks source compatibility for key path expressions that reference static properties after subscript overloads. For example, the compiler cannot differentiate between subscript keypath components by return type in the following: + +```swift +struct S { + static var count: Int { 42 } +} + +struct Test { + subscript(x: Int) -> String { "" } + subscript(y: Int) -> S.Type { S.self } +} + +let kpViaSubscript = \Test.[42] // fails to typecheck +``` + +This keypath does not specify a contextual type, without which the key path value type is unknown. To form a keypath to the metatype subscript and return an `Int`, we can specify a contextual type with a value type of `S.Type` and chain the metatype keypath: + +```swift +let kpViaSubscript: KeyPath = \Test.[42] +let kpAppended = kpViaSubscript.appending(path: \.count) +``` + +## ABI compatibility + +This feature does not affect ABI compatibility. + +## Implications on adoption + +This feature is back-deployable but it requires emission of new (property descriptors) symbols for static properties. + +The type-checker wouldn't allow to form key paths to static properties of types that come from modules that are built by an older compiler that don't support the feature because dynamic or static library produced for such module won't have all of the required symbols. + +Attempting to form a key path to a static property of a type from a module compiled with a compiler that doesn't yet support the feature will result in the following error with a note to help the developers: + +```swift +error: cannot form a keypath to a static property of type +note: rebuild to enable the feature +``` + +## Future directions + +### Key Paths to Enum cases + +Adding language support for read-only key paths to enum cases has been widely discussed on the [Swift Forums](https://forums.swift.org/t/enum-case-key-paths-an-update/68436) but has been left out of this proposal as this merits a separate discussion around [syntax design and implementation concerns](https://forums.swift.org/t/enum-case-keypaths/60899/32). + +Since references to enum cases must be metatypes, extending keypath expressions to include references to metatypes will hopefully bring the Swift language closer to adopting keypaths to enum cases in a future pitch. + +## Acknowledgments + +Thank you to Joe Groff for providing pivotal feedback on this pitch and its possible implementation and to Becca Royal-Gordon for an insightful discussion around the anticipated hurdles in implementing this feature. diff --git a/proposals/0439-trailing-comma-lists.md b/proposals/0439-trailing-comma-lists.md new file mode 100644 index 0000000000..1de9044bbb --- /dev/null +++ b/proposals/0439-trailing-comma-lists.md @@ -0,0 +1,304 @@ +# Allow trailing comma in comma-separated lists + +* Proposal: [SE-0439](0439-trailing-comma-lists.md) +* Author: [Mateus Rodrigues](https://github.com/mateusrodriguesxyz) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.1)** +* Implementation: [swiftlang/swift#74522](https://github.com/swiftlang/swift/pull/74522) +* Previous Proposal: [SE-0084](0084-trailing-commas.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-allow-trailing-comma-in-tuples-arguments-and-if-guard-while-conditions/70170)), ([review](https://forums.swift.org/t/se-0439-allow-trailing-comma-in-comma-separated-lists/72876)), ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0439-allow-trailing-comma-in-comma-separated-lists/73216)) +* Previous Revision: ([1](https://github.com/swiftlang/swift-evolution/blob/7864fa20cfb3a43aa6874feedb5aedb8be02da2c/proposals/0439-trailing-comma-lists.md)) + +## Introduction + +This proposal aims to allow the use of trailing commas, currently restricted to array and dictionary literals, in symmetrically delimited comma-separated lists. + +## Motivation + +### Development Quality of Life Improvement + +A trailing comma is an optional comma after the last item in a list of elements: + +```swift +let rank = [ + "Player 1", + "Player 3", + "Player 2", +] +``` + +Swift's support for trailing commas in array and dictionary literals makes it as easy to append, remove, reorder, or comment out the last element as any other element. + +Other comma-separated lists in the language could also benefit from the flexibility enabled by trailing commas. Consider the function [`split(separator:maxSplits:omittingEmptySubsequences:)`](https://swiftpackageindex.com/apple/swift-algorithms/1.2.0/documentation/algorithms/swift/lazysequenceprotocol/split(separator:maxsplits:omittingemptysubsequences:)-4q4x8) from the [Algorithms](https://github.com/apple/swift-algorithms) package, which has a few parameters with default values. + + +```swift +let numbers = [1, 2, 0, 3, 4, 0, 0, 5] + +let subsequences = numbers.split( + separator: 0, +// maxSplits: 1 +) ❌ Unexpected ',' separator +``` + +### The Language Evolved + +Back in 2016, a similar [proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0084-trailing-commas.md) with a narrower scope was reviewed and rejected for Swift 3. Since that time, the language has evolved substantially that challenges the basis for rejection. The code style that "puts the terminating right parenthesis on a line following the arguments to that call" has been widely adopted by community, Swift standard library codebase, swift-format, DocC documentation and Xcode. Therefore, not encouraging or endorsing this code style doesn't hold true anymore. + +The language has also seen the introduction of [parameter packs](https://github.com/apple/swift-evolution/blob/main/proposals/0393-parameter-packs.md), which enables APIs that are generic over variable numbers of type parameters, and code generation tools like plugins and macros that, with trailing comma support, wouldn't have to worry about a special condition for the last element when generating comma-separated lists. + +## Proposed solution + +This proposal adds support for trailing commas in symmetrically delimited comma-separated lists, which are the following: + +- Tuples and tuple patterns. + + ```swift + let velocity = ( + 1.66007664274403694e-03, + 7.69901118419740425e-03, + 6.90460016972063023e-05, + ) + + let ( + velocityX, + velocityY, + velocityZ, + ) = velocity + ``` + +- Parameter and argument lists of initializers, functions, enum associated values, expression macros and attributes. + + ```swift + func foo( + input1: Int = 0, + input2: Int = 0, + ) { } + + foo( + input1: 1, + input2: 1, + ) + + struct S { + init( + input1: Int = 0, + input2: Int = 0, + ) { } + } + + enum E { + case foo( + input1: Int = 0, + input2: Int = 0, + ) + } + + @Foo( + "input 1", + "input 2", + "input 3", + ) + struct S { } + + #foo( + "input 1", + "input 2", + "input 3", + ) + + struct S { + #foo( + "input 1", + "input 2", + "input 3", + ) + } + + ``` + +- Subscripts, including key path subscripts. + + ```swift + let value = m[ + x, + y, + ] + + let keyPath = \Foo.bar[ + x, + y, + ] + + f(\.[ + x, + y, + ]) + ``` + +- Closure capture lists. + + ```swift + { [ + capturedValue1, + capturedValue2, + ] in + } + ``` + +- Generic parameters. + + ```swift + struct S< + T1, + T2, + T3, + > { } + ``` + +- String interpolation. + + ```swift + let s = "\(1, 2,)" + ``` + +## Detailed Design + +Trailing commas will be supported in comma-separated lists when symmetric delimiters (including `(...)`, `[...]`, and `<...>`) enable unambiguous parsing. + +Note that the requirement for symmetric delimiters means that the following cases will not support trailing comma: + +- `if`, `guard` and `while` condition lists. + + ```swift + if + condition1, + condition2, ❌ + { } + + while + condition1, + condition2, ❌ + { } + + guard + condition1, + condition2, ❌ + else { } + ``` + +- Enum case label lists. + + ```swift + enum E { + case + a, + b, + c, ❌ + } + ``` + +- `switch` case labels. + + ```swift + switch number { + case + 1, + 2, ❌ + : + ... + default: + .. + } + ``` + +- Inheritance clauses. + + ```swift + struct S: + P1, + P2, + P3, ❌ + { } + ``` + +- Generic `where` clauses. + + ```swift + struct S< + T1, + T2, + T3, + > where + T1: P1, + T2: P2, ❌ + { } + ``` + +Trailing commas will be allowed in single-element lists but not in zero-element lists, since the trailing comma is actually attached to the last element. +Supporting a zero-element list would require supporting _leading_ commas, which isn't what this proposal is about. + +```swift +(1,) // OK +(,) ❌ expected value in tuple +``` + + +## Source compatibility + +This is a purely additive change with no source compatibility impact. + +## Alternatives considered + +### Allow trailing comma in all comma-separated lists + +Comma-separated lists that are not symmetrically delimited could also benefit from trailing comma support; for example, condition lists, in which reordering is fairly common. +However, these lists currently rely on the comma after the penultimate element to determine that what comes next is the last element, and some of them present challenges if relying on opening/closing delimiters instead. + +At first sight, `{` may seem a reasonable closing delimiter for `if` and `while` condition lists, but conditions can have a `{` themselves. + +```swift +if + condition1, + condition2, + { true }(), +{ } +``` + +This particular case can be handled but, given how complex conditions can be, it's hard to conclude that there's absolutely no corner case where ambiguity can arise in currently valid code. + +Inheritance lists and generic `where` clauses can appear in protocol definitions where there's no clear delimiter, making it harder to disambiguate where the list ends. + +```swift +protocol Foo { + associatedtype T: + P1, + P2, ❌ Expected type + ... +} +``` + +Although some comma-separated lists without symmetric delimiters may have a clear terminator in some cases, this proposal restricts trailing comma support to symmetrically delimited ones where it's clear that the presence of a trailing comma will not cause parsing ambiguity. + +### Eliding commas + +A different approach to address similar motivations is to allow the comma between two expressions to be elided when they are separated by a newline. + +```swift +print( + "red" + "green" + "blue" +) +``` +This was even [proposed](https://forums.swift.org/t/se-0257-eliding-commas-from-multiline-expression-lists/22889/188) and returned for revision back in 2019. + +The two approaches are not mutually exclusive. There remain unresolved questions about how the language can accommodate elided commas, and adopting this proposal does not prevent that approach from being considered in the future. + +## Revision History + +- Update to address acceptance decision of restricting trailing comma to lists with symmetric delimiters. + +## Acknowledgments + +Thanks to Alex Hoppen, Xiaodi Wu and others for their help on the proposal text and implementation. diff --git a/proposals/0440-debug-description-macro.md b/proposals/0440-debug-description-macro.md new file mode 100644 index 0000000000..e374da0f85 --- /dev/null +++ b/proposals/0440-debug-description-macro.md @@ -0,0 +1,242 @@ +# DebugDescription Macro + +* Proposal: [SE-0440](0440-debug-description-macro.md) +* Authors: [Dave Lee](https://github.com/kastiglione) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.0)** +* Implementation: Present in `main` under experimental feature `DebugDescriptionMacro` [apple/swift#69626](https://github.com/apple/swift/pull/69626) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/fda6746506368c8c6d2933ee6d71c87e6ed92f94/proposals/0440-debug-description-macro.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-debug-description-macro/67711)) ([review](https://forums.swift.org/t/se-0440-debugdescription-macro/72958)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0440-debugdescription-macro/73270)) ([second review](https://forums.swift.org/t/second-review-se-0440-debugdescription-macro/73325))([acceptance](https://forums.swift.org/t/accepted-se-0440-debugdescription-macro/73741)) + +## Introduction + +This proposal introduces `@DebugDescription`, a new debugging macro to the standard library, which lets data types specify a custom summary to be presented by the debugger. This macro brings improvements to the debugging experience, and simplifies the maintenance and delivery of debugger type summaries. It can be used in place of `CustomDebugStringConvertible` conformance, or in addition to, for custom use cases. + +## Motivation + +Displaying data is a fundamental part of software development. Both the standard library and the debugger offer multiple ways of printing values - Swift's print and dump, and LLDB's `p` and `po` commands. These all share the ability to render an arbitrary value into human readable text. Out of the box, both the standard library and the debugger present data as a nested tree of property-value pairs. The similarities run deep, for example the standard library and the debugger provide control over how much of the tree is shown. This functionality requires no action from the developer. + +The utility of displaying a complete value depends on the size and complexity of the data type(s), or depends on the context the data is being presented. Displaying the entirety of a small/shallow structure is sufficient, but some data types reach sizes/complexities where the complete tree of data is too large to be useful. + +For types that are too large or complex, the standard library and debugger again both provide tools giving us control over how our data is displayed. In the standard library, Swift has the `CustomDebugStringConvertible` protocol, which allows types to represented not as the aforementioned property tree, but as an arbitrary string. Relatedly, Swift has `CustomReflectable`, which lets developers control the contents and structure of the rendered property tree. For brevity and convention, from this point on this document will refer to the `CustomDebugStringConvertible` and `CustomReflectable` protocols via their single properties: `debugDescription` and `customMirror` respectively. + +LLDB has analogous features, which are called Type Summaries (\~`debugDescription`) and Synthetic Children (\\~`customMirror`) respectively. However, Swift and the debugger don't share or interoperate these definitions. Implementing these customizing protocols provides limited benefit inside the debugger. Likewise, defining Type Summaries or Synthetic Children in LLDB will have no benefit to Swift. + +While LLDB’s `po` command provides a convenient way to evaluate a `debugDescription` property defined in Swift, there are downsides to expression evaluation: Running arbitrary code can have side effects, be unstable to the application, and is slower. Expression evaluation happens by JIT compiling code, pushing it to the device the application is running on, and executing it in the context of the application, which involves a lot of work. As such, LLDB only does expression evaluation when explicitly requested by a user, most commonly with the `po` command in the console. Debugger UIs (IDEs) often provide a variable view which is populated using LLDB’s variable inspection which does not perform expression evaluation and is built on top of reflection. In some cases, such as when viewing crashlogs, or working with a core file, expression evaluation is not even possible. For these reasons, rendering values is ideally done without expression evaluation. + +This proposal introduces the ability to share a `debugDescription` definition between Swift and the debugger. This has benefits for developers, and for the debugger. + +LLDB Type Summaries can be defined using LLDB’s own (non Turing-complete) string interpolation syntax, called [Summary Strings](https://lldb.llvm.org/use/variable.html#summary-strings). While similar to Swift string interpolation, LLDB Summary Strings have restrictions that Swift string interpolation does not have. The primary restriction is that it allows data/property access, but not computation. LLDB Summary Strings cannot evaluate function calls, which includes computed properties. For the purpose of definition sharing, LLDB Type Summaries can be viewed as a lower common denominator of the two. As a result, definition sharing can be achieved only when a `debugDescription` definition meets the criteria imposed by LLDB Summary Strings. The criteria is not overly limiting, LLDB Summary Strings have been in for some time. + +Swift macros provide a convenient means to implement automatic translation of compatible `debugDescription` definitions into LLDB Summary Strings. A macro provides benefits that LLDB Summary Strings do not currently offer, including the ability to do compile time static validation to produce typo-free LLDB Summary Strings. The previously mentioned criteria that `debugDescription` must meet in order to be converted to an LLDB Summary String will loosen over time. This will be achieved first through the macro implementation becoming more sophisticated, and second as LLDB’s Summary Strings gain advancements. + +## Proposed solution + +Consider this simple example data type: + +```swift +struct Organization: CustomDebugStringConvertible { + var id: String + var name: String + var manager: Person + var members: [Person] + // ... and more + + var debugDescription: String { + "#\(id) \(name) (\(manager.name))" + } +} +``` + +To see the results of `debugDescription` in the debugger, the user has to run `po team` in the console. + +``` +(lldb) po team +"#15 Shipping (Francis Carlson) +``` + +Running the `p` command, or viewing the value in the Debugger UI (IDE), will show the value’s property tree, which may have arbitrary size/nesting: + +``` +(lldb) p team +(Organization) { + id = "..." + name = "Shipping" + manager = { + name = "Francis Carlson" + ... + } + members = { + [0] = ... + } + ... +} +``` + +However, by introducing the `@DebugDescription` macro, we can teach the debugger how to generate a summary without expression evaluation. + +```swift +@DebugDescription +struct Organization: CustomDebugStringConvertible { + var id: String + var name: String + var manager: Person + var members: [Person] + var officeAddress: [Address] + // ... and more + + var debugDescription: String { + "#\(id) \(name) (\(manager.name))" + } +} +``` + +The macro expands the body of `debugDescription` into the following LLDB Summary String: + +``` +#${var.id} ${var.name} (${var.manage.name}) +``` + +This summary string is emitted into the binary, where LLDB will load it automatically. Using this definition, LLDB can now present this description in contexts it previously could not, including the variable view and other parts of the debugger UI. + +A notable difference between the debugger console and debugger UI is that that UI displays one level at a time. When viewing an Array for example, its children are not expanded. To distinguish between elements of an Array (or any other collection), a user must expand each child. By employing `@DebugDescription`, LLDB will show a summary for each element of a collection, so that users may know – at a glance – exactly which element(s) to expand. + +## Detailed design + +```swift +/// Converts description definitions to a debugger Type Summary. +/// +/// This macro converts compatible description implementations written in Swift +/// to an LLDB format known as a Type Summary. A Type Summary is LLDB's +/// equivalent to debugDescription, with the distinction that it does not +/// execute code inside the debugged process. By avoiding code execution, +/// descriptions can be produced faster, without potential side effects, and +/// shown in situations where code execution is not performed, such as the +/// variable list of an IDE. +/// +/// Consider this an example. This Team struct has a debugDescription which +/// summarizes some key details, such as the team's name. The debugger only +/// computes this string on demand - typically via the po command. By applying +/// the DebugDescription macro, a matching Type Summary is constructed. This +/// allows the user to show a string like "Rams [11-2]", without executing +/// debugDescription. This improves the usability, performance, and +/// reliability of the debugging experience. +/// +/// @DebugDescription +/// struct Team: CustomDebugStringConvertible { +/// var name: String +/// var wins, losses: Int +/// +/// var debugDescription: String { +/// "\(name) [\(wins)-\(losses)]" +/// } +/// } +/// +/// The DebugDescription macro supports both debugDescription, description, +/// as well as a third option: a property named lldbDescription. The first +/// two are implemented when conforming to the CustomDebugStringConvertible +/// and CustomStringConvertible protocols. The additional lldbDescription +/// property is useful when both debugDescription and description are +/// implemented, but don't meet the requirements of the DebugDescription +/// macro. If lldbDescription is implemented, DebugDescription choose it +/// over debugDescription and description. Likewise, debugDescription is +/// preferred over description. +/// +/// ### Description Requirements +/// +/// The description implementation has the following requirements: +/// +/// * The body of the description implementation must a single string +/// expression. String concatenation is not supported, use string interpolation +/// instead. +/// * String interpolation can reference stored properties only, functions calls +/// and other arbitrary computation are not supported. Of note, conditional +/// logic and computed properties are not supported. +/// * Overloaded string interpolation cannot be used. +@attached(member) +@attached(memberAttribute) +public macro DebugDescription() = + #externalMacro(module: "SwiftMacros", type: "DebugDescriptionMacro") + +/// Internal-only macro. See @DebugDescription. +@attached(peer, names: named(_lldb_summary)) +public macro _DebugDescriptionProperty( debugIdentifier: String, _ computedProperties: [String]) = + #externalMacro(module: "SwiftMacros", type: "_DebugDescriptionPropertyMacro") +``` + +Of note, the work is split between two macros `@DebugDescription` and `@_DebugDescriptionProperty`. By design, `@DebugDescription` is attached to the type, where it gathers type-level information, including gather a list of stored properties. This macro also determines which description property to attach @_DebugDescriptionProperty to. + +`@_DebugDescriptionProperty` is not intended for direct use by users. This macro is scoped to the inspect a single description property, not the entire type. This approach of splitting the work allows the compiler to avoid unnecessary work. + +The additional supported property, `lldbDescription`, is to support two different use cases. + +First, in some cases existing `debugDescription`/`description` cannot be changed, because doing so would be a breaking change to either `String(reflecting:)` or `String(describing:)`. In these circumstances, developers can define `lldbDescription` instead. + +Second, in some cases developer may want to use LLDB Summary String syntax directly. Since `lldbDescription` is not coupled to API, developers are free to include Summary String elements in their `lldbDescription`. This is expected to be uncommon, but lets developers have more control over Summary Strings. Since LLDB syntax has no meaning inside `debugDescription`/`description`, the macro performs escaping when translating those definitions. + +Using both `debugDescription` and `lldbDescription` is an intended use case. The design of this macro allows developers to have both an LLDB compatible `lldbDescription`, and a more complex `debugDescription`. This allows the debugger to show a simple summary, while providing enabling a more detailed or dynamic `debugDescription`. + +## Source compatibility + +This proposal adds a new macro to the standard library. There are no source compatibility concerns. + +## ABI compatibility + +The macro implementation emits metadata for the debugger, and does not affect ABI. + +## Implications on adoption + +The macro can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. + +## Future directions + +### Support for Generics + +The `@DebugDescription` macro cannot currently be attached to generic type definitions. This is because the macro's implementation emits a static property into the type, which is a use case not currently supported by the Swift language. To support generic types, Swift will need a separate proposal to enable code such as the following: + +```swift +struct Generic { + static let _lldb_summary = (23 as UInt8, 30 as UInt8, ...) +} +``` + +The goal is to allow static properties in generic types, specifically when the property's type is concrete and independent of the generic signature. In the above code, the property's type is a tuple of `UInt8`, which does not depend on `T`. + +### Beyond Summary Strings + +Future directions include generating Python instead of LLDB Summary Strings. This has the benefit of having fewer restrictions on the `debugDescription` definition. It has the downside of needing security scrutiny not required by LLDB Summary Strings. + +A similar future direction is to support sharing Swift `customMirror` definitions into LLDB Synthetic Children definitions. Unlike LLDB Type Summaries, LLDB has no "DSL" to expression LLDB Synthetic Children, currently the main option is Python. Given that there are two uses solved by generating Python, it's an approach worth considering in the future. While `customMirror` implementations are less common in Swift than their `debugDescription` counterpart, in LLDB, Synthetic Children are as important, or even more important than Summary Strings. The reason is that Synthetic Children allow data types to express their data "interface" rather than their implementation. Consider types like Array and Dictionary, which often have implementation complexity that provides optimal performance, not for data simplicity. + +## Alternatives considered + +### Explicit LLDB Summary Strings + +The simplest macro implementation is one that performs no Swift-to-LLDB translation and directly accepts an LLDB Summary String. This approach requires users to know LLDB Summary String syntax, which while not complex, still presents a hindrance to adoption. Such a macro would could create redundancy: `debugDescription` and the separate LLDB Summary String. These would need to be manually kept in sync. + +### Independent Property (No `debugDescription` Reuse) + +Instead of leveraging existing `debugDescription`/`description` implementations, the `@DebugDescription` macro could use a completely separate property. + +Reusing existing `debugDescription` implementations makes a tradeoff that may not be obvious to developers. The benefit is a single definition, and getting more out of a well known idiom. The risk comes from the requirements imposed by `@DebugDescription`. The requirements could lead to developers changing their existing implementation. Any changes to `debugDescription` will impact `String(reflecting:)`, and similarly changes to `description` will impact `String(describing:)`. + +The risk involving String conversion would be avoided by having the macro use an independent property. The macro would not support `debugDescription`/`description`. In this scenario, developers would be required to implement `lldbDescription`, even if the implementation is identical to the existing `debugDescription`. + +Our expectation is that most code, particularly application code, will not depend on String conversion (especially `String(reflecting:)`). For code that does depend on String conversion, it should have testing in place to catch breaking changes. Inside of an application, authors of code which has behavior that depends on String conversion initializers should already be aware of the consequences of changing `debugDescription`/`description`. Frameworks are a more challenging situation, where its authors are not always aware of if/how its clients depend on String conversion. + +The belief is that the benefits of reusing `debugDescription` will outweigh the downsides. Framework authors can make it a policy of their own to not reuse `debugDescription`, if they believe that presents a risk to clients of their framework. + +### Contextual Diagnostics + +To help address the potential risk around reuse of `debugDescription`, the macro could emit diagnostics that vary by the property being used. Specifically, if the developer implements `lldbDescription`, they will get the full diagnostics available, indicating how to fix its implementation. Conversely, when `debugDescription` is being reused, the diagnostics will not contain details of which requirements were not met, instead the diagnostics would tell the user that `debugDescription` is not compatible, and to define `lldbDescription` instead. This should make it less likely that the macro leads to changes affecting String conversion. + +## Revision history + +* Changes from the first review + * Document support for generic types as a future direction + * Rename `_debugDescription` to `lldbDescription` + * Direct use of LLDB Summary String syntax is supported only by `lldbDescription` + +## Acknowledgments + +Thank you to Doug Gregor and Alex Hoppen for their generous guidance, and PR reviews. Adrian Prantl, for the many productive discussions and implementation ideas. To Kuba Mracek, for implementing linkage macros which support this work. Thank you to Tony Parker and Steve Canon for their adoption feedback. To Holly Borla, for timely technical and process support. diff --git a/proposals/0441-formalize-language-mode-terminology.md b/proposals/0441-formalize-language-mode-terminology.md new file mode 100644 index 0000000000..d34b6eeacf --- /dev/null +++ b/proposals/0441-formalize-language-mode-terminology.md @@ -0,0 +1,279 @@ +# Formalize ‘language mode’ terminology + +* Proposal: [SE-0441](0441-formalize-language-mode-terminology.md) +* Author: [James Dempsey](https://github.com/dempseyatgithub) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.1)** +* Implementation: [swiftlang/swift-package-manager#7620](https://github.com/swiftlang/swift-package-manager/pull/7620), [swiftlang/swift#75564](https://github.com/swiftlang/swift/pull/75564) +* Review: ([first pitch](https://forums.swift.org/t/pitch-formalize-swift-language-mode-naming-in-tools-and-api/71733)) ([second pitch](https://forums.swift.org/t/pitch-2-formalize-language-mode-naming-in-tools-and-api/72136)) ([review](https://forums.swift.org/t/se-0441-formalize-language-mode-terminology/73182)) ([acceptance](https://forums.swift.org/t/accepted-se-0441-formalize-language-mode-terminology/73716)) + +## Introduction +The term "Swift version” can refer to either the toolchain/compiler version or the language mode. This ambiguity is a consistent source of confusion. This proposal formalizes the term _language mode_ in tool options and APIs. + +## Proposed Solution +The proposed solution is to use the term _language mode_ for the appropriate Swift compiler option and Swift Package Manager APIs. Use of "Swift version" to refer to language mode will be deprecated. + +### Terminology +The term _language mode_ has been consistently used to describe this compiler feature since it was introduced with Swift 4.0 and is an established term of art in the Swift community. + +The **Alternatives Considered** section contains a more detailed discussion of the term's history and usage. + +### Swift compiler option +Introduce a `-language-mode` option that has the same behavior as the existing `-swift-version` option, while de-emphasizing the `-swift-version` option in help and documentation. + +#### Naming note +The proposed compiler option uses the term 'language mode' instead of 'Swift language mode' because the context strongly implies a Swift language mode. The intent is that the `languageMode()` compiler condition described in **Future directions** would also use that naming convention for the same reason. + +### Swift Package Manager +Introduce four Swift Package Manager API changes limited to manifests \>= 6.0: + +#### 1. A new Package init method that uses the language mode terminology + +```swift +@available(_PackageDescription, introduced: 6) +Package( + name: String, + defaultLocalization: [LanguageTag]? = nil. + platforms: [SupportedPlatform]? = nil, + products: [Product] = [], + dependencies: [Package.Dependency] = [], + targets: [Target] = [], + swiftLanguageModes: [SwiftLanguageMode]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil +) +``` + +Add a new `init` method to `Package` with the following changes from the current `init` method: + +- The parameter `swiftLanguageVersions` is renamed to `swiftLanguageModes` +- The parameter type is now an optional array of `SwiftLanguageMode` values instead of `SwiftVersion` values + +The existing init method will be marked as `deprecated` and `renamed` allowing the compiler to provide a fix-it. + +#### 2. Rename `swiftLanguageVersions` property to `swiftLanguageModes` +Rename the public `Package` property `swiftLanguageVersions` to `swiftLanguageModes`. Add a `swiftLanguageVersions` computed property that accesses `swiftLanguageModes` for backwards compatibility. + + +#### 3. Rename `SwiftVersion` enum to `SwiftLanguageMode` +Rename the `SwiftVersion` enum to `SwiftLanguageMode`. Add `SwiftVersion` as a type alias for backwards compatibility. + + +#### 4. Add `swiftLanguageMode()` to `SwiftSetting` + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.0) + public static func swiftLanguageMode( + _ mode: SwiftLanguageMode, + _ condition: BuildSettingCondition? = nil + ) +``` + +The changes in [SE-0435: Swift Language Version Per Target](https://github.com/apple/swift-evolution/blob/main/proposals/0435-swiftpm-per-target-swift-language-version-setting.md) have been implemented and released in pre-release versions of Swift 6.0. This proposal will add `swiftLanguageMode()` as a `SwiftSetting` and deprecate the `swiftLanguageVersion()` setting added by SE-0435 with a `renamed` annotation. + +#### Naming note + +In Swift PM manifests, multiple languages are supported. For clarity, there is existing precedent for parameter and enum type names to have a language name prefix. + +For example the Package `init` method currently includes: + +```swift + ... + swiftLanguageVersions: [SwiftVersion]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil + ... +``` + +For clarity and to follow the existing precedent, the proposed Swift PM APIs will be appropriately capitalized versions of "Swift language mode". + +## Detailed design + +### New swift compiler option +A new `-language-mode` option will be added with the same behavior as the existing `-swift-version` option. + +The `-swift-version` option will continue to work as it currently does, preserving backwards compatibility. + +The `-language-mode` option will be presented in the compiler help. + +The `-swift-version` option will be suppressed from the top-level help of the compiler. + +### Swift Package Manager +Proposed Swift Package Manager API changes are limited to manifests \>= 6.0: + +### New Package init method and deprecated init method +A new `init` method will be added to `Package` that renames the `swiftLanguageVersions` parameter to `swiftLanguageModes` with the type of the parameter being an optional array of `SwiftLanguageMode` values instead of `SwiftVersion` values: + +```swift +@available(_PackageDescription, introduced: 6) +Package( + name: String, + defaultLocalization: [LanguageTag]? = nil. + platforms: [SupportedPlatform]? = nil, + products: [Product] = [], + dependencies: [Package.Dependency] = [], + targets: [Target] = [], + swiftLanguageModes: [SwiftLanguageMode]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil +) +``` + + +The existing init method will be marked as `deprecated` and `renamed`, allowing the compiler to provide a fix-it: + +``` +@_disfavoredOverload +@available(_PackageDescription, introduced: 5.3, deprecated: 6, renamed: +"init(name:defaultLocalization:platforms:pkgConfig:providers:products: +dependencies:targets:swiftLanguageModes:cLanguageStandard: +cxxLanguageStandard:)") + public init( + name: String, + ... + swiftLanguageVersions: [SwiftVersion]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil + ) { +``` + +See the **Source compatibility** section for more details about this change. + +### Rename `swiftLanguageVersions` property to `swiftLanguageModes` +Rename the `Package` public property `swiftLanguageVersions` to `swiftLanguageModes`. Introduce a computed property named `swiftLanguageModes` that accesses the renamed stored property for backwards compatibility. + +The computed property will be annotated as `deprecated` in Swift 6 and `renamed` to `swiftLanguageModes`. + +For packages with swift tools version less than 6.0, accessing the `swiftLanguageModes` property will continue to work. + +For 6.0 and later, that access will be a warning with a fix-it to use the new property name. + +```swift + @available(_PackageDescription, deprecated: 6.0, renamed: "swiftLanguageModes") + public var swiftLanguageVersions: [SwiftVersion]? { + get { swiftLanguageModes } + set { swiftLanguageModes = newValue } + } +``` + +See the **Source compatibility** section for more details about this change. + +### Rename `SwiftVersion` enum to `SwiftLanguageMode` +Rename the existing `SwiftVersion` enum to `SwiftLanguageMode` with `SwiftVersion` added back as a type alias for backwards compatibility. The type alias will be deprecated in 6.0 with a `renamed` annotation to `SwiftLanguageMode`. + +This change will not affect serialization of PackageDescription types. Serialization is handled by converting PackageDescription types into separate, corresponding Codable types. The existing serialization types will remain as-is. + + +### Add `swiftLanguageMode()` to `SwiftSetting` +Add a new `swiftLanguageMode()` static function to `SwiftSetting` with the same behavior of `swiftLanguageBehavior()` added by [SE-0435](https://github.com/apple/swift-evolution/blob/main/proposals/0435-swiftpm-per-target-swift-language-version-setting.md): + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.0) + public static func swiftLanguageMode( + _ mode: SwiftLanguageMode, + _ condition: BuildSettingCondition? = nil + ) +``` + +The name of the function is `swiftLanguageMode()` instead of `languageMode()` to keep naming consistent with the `swiftLanguageModes` parameter of the Package init method. The parameter label `mode` is used to follow the precedent set by the existing `interoperabilityMode()` method in `SwiftSetting`. + +Deprecate the `swiftLanguageVersion()` setting added by SE-0435 with a `renamed` annotation to provide a fix-it for developers who have adopted this API in pre-release versions of Swift 6.0.: + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.0, deprecated: 6.0, renamed: "swiftLanguageMode(_:_:)") + public static func swiftLanguageVersion( + _ version: SwiftVersion, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting { + ... + } +} +``` + +## Source compatibility +The new Package `init` method and deprecating the existing `init` method will cause a deprecation warning for package manifests that specify the existing `swiftLanguageVersions` parameter when updating to swift tools version 6.0 + +A search of manifest files in public repositories suggests that about 10% of manifest files will encounter this breakage. + +Because the deprecated `init` method is annotated as `renamed` the compiler will automatically provide a fix-it to update to the new `init` method. + +Renaming the public `swiftLanguageVersions` property of `Package` preserves backwards compatibility by introducing a computed property with that name. The computed property will be marked as `deprecated` in 6.0 and annotated as `renamed` to provide a fix-it. + +Searching manifest files in public repositories suggests that accessing the `swiftLanguageVersions` property directly is not common. + +The `SwiftVersion` type alias will be deprecated in favor of the `SwiftLanguageMode` enum. This also will provide a fix-it. + +Finally the `swiftLanguageVersion()` method in `SwiftSetting` added as part of SE-0435 will be deprecated in favor of the `swiftLanguageMode()` method with a fix-it. + +## ABI compatibility +This proposal has no effect on ABI stability. + +## Future directions +This proposal originally included the proposed addition of a `languageMode()` compilation condition to further standardize on the terminology and allow the compiler to check for valid language mode values. + +That functionality has been removed from this proposal with the intent to pitch it separately. Doing so keeps this proposal focused on the tools, including the source breaking API changes. The future direction is purely additive and would focus on the language change. + +## Alternatives considered + +### Alternate terminology + +In the pitch phase, a number of terms were suggested as alternatives for _language mode_. Some concerns were also expressed that the term _language mode_ may be too broad and cause future ambiguity. + +The intent of this proposal is to formalize established terminology in tool options and APIs. + +The term _language mode_ is a long-established term of art in the Swift community to describe this functionality in the compiler. + +This includes the [blog post](https://www.swift.org/blog/swift-4.0-released/) announcing the functionality as part of the release of Swift 4 in 2017 (emphasis added): + +> With Swift 4, you may not need to modify your code to use the new version of the compiler. The compiler supports two _language modes_… + +> The _language mode_ is specified to the compiler by the -swift-version flag, which is automatically handled by the Swift Package Manager and Xcode. +> +> One advantage of these _language modes_ is that you can start using the new Swift 4 compiler and migrate fully to Swift 4 at your own pace, taking advantage of new Swift 4 features, one module at a time. + +Usage also includes posts in the last year from LSG members about Swift 6 language mode: + +- [Design Priorities for the Swift 6 Language Mode](https://forums.swift.org/t/design-priorities-for-the-swift-6-language-mode/62408/27) +- [Progress toward the Swift 6 language mode](https://forums.swift.org/t/progress-toward-the-swift-6-language-mode/68315) + +Finally, searching for "language modes" and "language mode" in the Swift forums found that at least 90% of the posts use the term in this way. Many of the remaining posts use the term in the context of Clang. + +#### Alternatives mentioned + +Alternate terms raised as possibilities were: + - _Edition_: a term used by Rust for a similar concept + - _Standard_: similar to C or C++ standards + Language standards tend to be associated with a written specification, which Swift does not currently have. + Using the term _standard_ would preclude using the term in the future to describe a formal standard. + + +#### Potential overload of _language mode_ +Some reviewers raised concern that Embedded Swift could be considered a language mode and lead to future ambiguity. + +On consideration, this concern is mitigated in two ways: + +1. As noted above, the term _language mode_ is a well-established term of art in the Swift community. + +2. The term _Embedded Swift_ already provides an unambiguous, concise name that can be discussed without requiring a reference to modes. + + This is demonstrated by the following hypothetical FAQ based on the Embedded Swift vision document: +> _What is Embedded Swift?_ +> Embedded Swift is a subset of Swift suitable for restricted environments such as embedded and low-level environments. +> +> _How do you enable Embedded Swift?_ +> Pass the `-embedded` compiler flag to compile Embedded Swift. + +Considering these alternatives, it seems likely that introducing a new term to replace the long-established term _language mode_ and potentially giving the existing term a new meaning would lead to more ambiguity than keeping and formalizing the existing meaning of _language mode_. + +## Acknowledgments + +Thank you to Pavel Yaskevich for providing guidance and assistance in the implementation of this proposal. diff --git a/proposals/0442-allow-taskgroup-childtaskresult-type-to-be-inferred.md b/proposals/0442-allow-taskgroup-childtaskresult-type-to-be-inferred.md new file mode 100644 index 0000000000..8d64c12134 --- /dev/null +++ b/proposals/0442-allow-taskgroup-childtaskresult-type-to-be-inferred.md @@ -0,0 +1,191 @@ +# Allow TaskGroup's ChildTaskResult Type To Be Inferred + +* Proposal: [SE-0442](0442-allow-taskgroup-childtaskresult-type-to-be-inferred.md) +* Author: [Richard L Zarth III](https://github.com/rlziii) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift#74517](https://github.com/apple/swift/pull/74517) +* Review: ([pitch](https://forums.swift.org/t/allow-taskgroups-childtaskresult-type-to-be-inferred/72175))([review](https://forums.swift.org/t/se-0442-allow-taskgroups-childtaskresult-type-to-be-inferred/73397))([acceptance](https://forums.swift.org/t/accepted-se-0422-allow-taskgroups-childtaskresult-type-to-be-inferred/73747)) + +## Introduction + +`TaskGroup` and `ThrowingTaskGroup` currently require that one of their two generics (`ChildTaskResult`) always be specified upon creation. Due to improvements in closure parameter/result type inference introduced by [SE-0326](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0326-extending-multi-statement-closure-inference.md) this can be simplified by allowing the compiler to infer both of the generics in most cases. + +## Motivation + +Currently to create a new task group, there are two generics involved: `ChildTaskResult` and `GroupResult`. The latter can often be inferred in many cases, but the former must always be supplied as part of either the `withTaskGroup(of:returning:body:)` or `withThrowingTaskGroup(of:returning:body:)` function. For example: + +```swift +let messages = await withTaskGroup(of: Message.self) { group in + for id in ids { + group.addTask { await downloadMessage(for: id) } + } + + var messages: [Message] = [] + for await message in group { + messages.append(message) + } + return messages +} +``` + +The type of `messages` (which is the `GroupResult` type) is correctly inferred as `[Message]`. However, the return value of the `addTask(...)` closures is not inferred and currently must be supplied to the `of:` parameter of the `withTaskGroup(of:returning:body:)` function (e.g. `Message`). The correct value of the generic can be non-intuitive for new users to the task group APIs. + +Note that `withDiscardingTaskGroup(returning:body:)` and `withThrowingDiscardingTaskGroup(returning:body:)` do not have `ChildTaskResult` generics since their child tasks must always be of type `Void`. + +## Proposed solution + +Adding a default `ChildTaskResult.self` argument for `of childTaskResultType: ChildTaskResult.Type` will allow `withTaskGroup(of:returning:body:)` to infer the type of `ChildTaskResult` in most cases. The currently signature of `withTaskGroup(of:returning:body:)` looks like: + +```swift +public func withTaskGroup( + of childTaskResultType: ChildTaskResult.Type, + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout TaskGroup) async -> GroupResult +) async -> GroupResult where ChildTaskResult : Sendable +``` + +The function signature of `withThrowingTaskGroup(of:returning:body:)` is nearly identical, so only `withTaskGroup(of:returning:body:)` will be used as an example throughout this proposal. + +Note that the `GroupResult` generic is inferable via the `= GroupResult.self` default argument. This can also be applied to `ChildTaskResult` as of [SE-0326](0326-extending-multi-statement-closure-inference.md). As in: + +```swift +public func withTaskGroup( + of childTaskResultType: ChildTaskResult.Type = ChildTaskResult.self, // <- Updated. + returning returnType: GroupResult.Type = GroupResult.self, + body: (inout TaskGroup) async -> GroupResult +) async -> GroupResult where ChildTaskResult : Sendable +``` + +This allows the original example above to be simplified: + +```swift +// No need for `(of: Message.self)` like before. +let messages = await withTaskGroup { group in + for id in ids { + group.addTask { await downloadMessage(for: id) } + } + + var messages: [Message] = [] + for await message in group { + messages.append(message) + } + return messages +} +``` + +In the above snippet, `ChildTaskResult` is inferred as `Message` and `GroupResult` is inferred as `[Message]`. Not needing to specify the generics explicitly will simplify the API design for these functions and make it easier for new users of these APIs, as it can currently be confusing to understand the differences between `ChildTaskResult` and `GroupResult`. This can be especially true when one or both of those is `Void`. For example: + +```swift +let logCount = await withTaskGroup(of: Void.self) { group in + for id in ids { + group.addTask { await logMessageReceived(for: id) } + } + + return ids.count +} +``` + +In the above example, it can be confusing (and not intuitive) to know that `Void.self` is needed for `ChildTaskResult` and the compiler does not currently give great hints for what that type should be or steering the user into fixing the generic argument if it is mismatched (for example, if the user swaps `Int.self` for `Void.self` in the above example). With the proposed solution, the above can become the following example with type inference used for both generic arguments: + +```swift +let logCount = await withTaskGroup { group in + for id in ids { + group.addTask { await logMessageReceived(for: id) } + } + + return ids.count +} +``` + +## Detailed design + +Because type inference is top-down, it relies on the first statement that uses `group` to infer the generic arguments for `ChildTaskResult`. Therefore, it is possible to get a compiler error by creating a task group where the first use of `group` does not use `addTask(...)`, like so: + +```swift +// Expect `ChildTaskResult` to be `Void`... +await withTaskGroup { group in // Generic parameter 'ChildTaskResult' could not be inferred + // Since `addTask(...)` wasn't the first statement, this fails to compile. + group.cancelAll() + + for id in ids { + group.addTask { await logMessageReceived(for: id) } + } +} +``` + +This can be fixed by going back to specifying the generic like before: + +```swift +// Expect `ChildTaskResult` to be `Void`... +await withTaskGroup(of: Void.self) { group in + group.cancelAll() + + for id in ids { + group.addTask { await logMessageReceived(for: id) } + } +} +``` + +However, this is a rare case in general since `addTask(...)` is generally the first `TaskGroup`/`ThrowingTaskGroup` statement in a task group body. + +It is also possible to create a compiler error by returning two different values from an `addTask(...)` closure: + +```swift +await withTaskGroup { group in + group.addTask { await downloadMessage(for: id) } + group.addTask { await logMessageReceived(for: id) } // Cannot convert value of type 'Void' to closure result type 'Message' +} +``` + +The compiler will already give a good error message here, since the first `addTask(...)` statement is what determined (in this case) that the `ChildTaskResult` generic was set to `Message`. If this needs to be made more clear (instead of being inferred), the user can always specify the generic directly as before: + +```swift +await withTaskGroup(of: Void.self) { group in + // Now the error has moved here since the generic was specified up front... + group.addTask { await downloadMessage(for: id) } // Cannot convert value of type 'Message' to closure result type 'Void' + group.addTask { await logMessageReceived(for: id) } +} +``` + +## Source compatibility + +Omitting the `of childTaskResultType: ChildTaskResult.Type` parameter for both `withTaskGroup(of:returning:body:)` and `withThrowingTaskGroup(of:returning:body:)` is new, and therefore the inference of `ChildTaskResult` is opt-in and does not break source compatibility. + +## ABI compatibility + +No ABI impact since adding a default argument value is binary compatible change. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source +code with no deployment constraints and without affecting source or ABI +compatibility. + +## Future directions + +### TaskGroup APIs Without the "with..." Closures + +While not possible without more compiler features to enforce the safety of a task group not escaping a context, and having to await all of its results at the end of a "scope..." it is an interesting future direction to explore a `TaskGroup` API that does not need to resort to "with..." methods, like this: + +```swift +// Potential long-term direction that might be possible: +func test() async { + let group = TaskGroup() + group.addTask { /* ... */ } + + // Going out of scope would have to imply `group.waitForAll()`... +} +``` + +If we were to explore such API, the type inference rules would be somewhat different, and a `TaskGroup` would likely be initialized more similarly to a collection: `TaskGroup`. + +This proposal has no impact on this future direction, and can be accepted as is, without precluding future developments in API ergonomics like this. + +## Alternatives considered + +The main alternative is to do nothing; as in, leave the `withTaskGroup(of:returning:body:)` and `withThrowingTaskGroup(of:returning:body:)` APIs like they are and require the `ChildTaskResult` generic to always be specified. + +## Acknowledgments + +Thank you to both Konrad Malawski ([@ktoso](https://github.com/ktoso)) and Pavel Yaskevich ([@xedin](https://github.com/xedin)) for confirming the viability of this proposal/idea, and for Konrad Malawski ([@ktoso](https://github.com/ktoso)) for helping to review the proposal and implementation. diff --git a/proposals/0443-warning-control-flags.md b/proposals/0443-warning-control-flags.md new file mode 100644 index 0000000000..eb24169971 --- /dev/null +++ b/proposals/0443-warning-control-flags.md @@ -0,0 +1,242 @@ + +# Precise Control Flags over Compiler Warnings + +* Proposal: [SE-0443](0443-warning-control-flags.md) +* Authors: [Doug Gregor](https://github.com/douggregor), [Dmitrii Galimzianov](https://github.com/DmT021) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift#74466](https://github.com/swiftlang/swift/pull/74466) +* Review: ([pitch](https://forums.swift.org/t/warnings-as-errors-exceptions/72925)) ([review](https://forums.swift.org/t/se-0443-precise-control-flags-over-compiler-warnings/74116)) ([acceptance](https://forums.swift.org/t/accepted-se-0443-precise-control-flags-over-compiler-warnings/74377)) +* Previous revisions: [1](https://github.com/swiftlang/swift-evolution/blob/57fe29d5d55edb85b14c153b7f4cbead6b6539eb/proposals/0443-warning-control-flags.md), [2](https://github.com/swiftlang/swift-evolution/blob/7b12899ad0d96002c793d33ef8109ec47c5d256f/proposals/0443-warning-control-flags.md) + +## Introduction + +This proposal introduces new compiler options that allow fine-grained control over how the compiler emits certain warnings: as warnings or as errors. + +## Motivation + +The current compiler options for controlling how warnings are emitted are very inflexible. Currently, the following options exist: +- `-warnings-as-errors` - upgrades all warnings to errors +- `-no-warnings-as-errors` - cancels the upgrade of warnings to errors +- `-suppress-warnings` - disables the emission of all warnings + +This lack of flexibility leads to situations where users who want to use `-warnings-as-errors` find themselves unable to do so, or unable to upgrade to a new version of the compiler or SDK until all newly diagnosed warnings are resolved. The most striking example of this is deprecation warnings for certain APIs, though they are not limited to them. + +## Proposed solution + +This proposal suggests adding new options that will allow the behavior of warnings to be controlled based on their diagnostic group. +- `-Werror ` - upgrades warnings in the specified group to errors +- `-Wwarning ` - indicates that warnings in the specified group should remain warnings, even if they were previously suppressed or upgraded to errors + +The `` parameter is a string identifier of the diagnostic group. + +A diagnostic group is a stable identifier for an error or warning. It is an abstraction layer over the diagnostic identifiers used within the compiler. This is necessary because diagnostics within the compiler may change, but we need to provide stable user-facing identifiers for them. + +A diagnostic group may include errors, warnings, or other diagnostic groups. For example, the `DeprecatedDeclaration` diagnostic group includes warnings related to the use of an API marked with the `@available(..., deprecated: ...)` attribute. The `Deprecated` diagnostic group includes the `DeprecatedDeclaration` group and other groups related to deprecation. + +Diagnostic groups may expand over time, but they can never become narrower. When a new diagnostic is added to the compiler, it is either included in an existing group or a new group is created for it, which in turn can also be included in one of the broader groups, if appropriate. + +The order in which these flags are specified when invoking the compiler is important. If two or more options change the behavior of the same warning, we follow the rule "the last one wins." + +We also retain the existing compiler options but modify their handling algorithm so that they are considered in the general list with the new options and follow the "last one wins" rule as well. + +Thus, for example, you can use the combination `-warnings-as-errors -Wwarning Deprecated`, which will upgrade all warnings to errors except for those in the `Deprecated` group. However, if these flags are specified in the reverse order(`-Wwarning Deprecated -warnings-as-errors`) it will be interpreted as upgrading all warnings to errors, as the `-warnings-as-errors` flag is the last one. + +We are also introducing a new compiler flag, `-print-diagnostic-groups`, to display the names of diagnostic groups along with the textual representation of the warnings. When used, the warning message will be followed by the name of the narrowest group that includes that warning, enclosed in square brackets. For example: +``` +main.swift:33:1: warning: 'f()' is deprecated [#DeprecatedDeclaration] +``` + +## Detailed design + +### Diagnostic groups + +Diagnostic groups form an acyclic graph with the following properties: + +- A warning or error can only be included in one diagnostic group. This artificial restriction is introduced to solve two main problems: + - When using the `-print-diagnostic-groups` flag, it would be inconvenient if a warning corresponded to multiple groups. + - Documentation lookup will also be easier for the user if a diagnostic has only one identifier. + +- A diagnostic group may include any number of other diagnostic groups. This will allow organizing groups into sets with similar meanings but different specific diagnostics. For example, the warnings `DeprecatedDeclaration` and `UnsafeGlobalActorDeprecated` are part of the supergroup `Deprecated`. + +- A diagnostic group can be included in any number of diagnostic groups. This allows expressing the membership of a group in multiple supergroups, where appropriate. For example, the group `UnsafeGlobalActorDeprecated` is part of both the `Deprecated` and `Concurrency` groups. + +The internal structure of the graph may change to some extent. However, the set of diagnostics included in a diagnostic group (directly or transitively) should not shrink. There are two typical situations where the graph structure may change: +- When adding a new diagnostic to the compiler, consider creating a new group corresponding to that diagnostic. If the new group is created it can also be included in one or more existing groups if it belongs to them. For example, it is expected that the `Deprecated` group will continuously include new subgroups. +- If an existing diagnostic is split into more specific versions, and we want to allow users to use the more specific version in compiler options, a separate group is created for it, which **must** be included in the group of the original diagnostic. + + For example, suppose we split the `DeprecatedDeclaration` warning into a general version and a specialized version `DeprecatedDeclarationSameModule`, which the compiler emits if the deprecated symbol is declared in the same module. In this case, the `DeprecatedDeclarationSameModule` group must be added to the `DeprecatedDeclaration` group to ensure that the overall composition of the `DeprecatedDeclaration` group does not change. The final structure should look like this: + ``` + DeprecatedDeclaration (group) + ├─ DeprecatedDeclaration (internal diag id) + └─ DeprecatedDeclarationSameModule (group) + └─ DeprecatedDeclarationSameModule (internal diag id) + ``` + Thus, invoking the compiler with the `-Werror DeprecatedDeclaration` parameter will cover both versions of the warning, and the behavior will remain unchanged. At the same time, the user can control the behavior of the narrower `DeprecatedDeclarationSameModule` group if they want to. + +### Compiler options evaluation + +Each warning in the compiler is assigned one of three behaviors: `warning`, `error`, or `suppressed`. +Compiler options for controlling the behavior of groups are now processed as a single list. These options include: +``` +-Werror +-Wwarning +-warnings-as-errors +-no-warnings-as-errors +``` +When these options are passed to the compiler, we sequentially apply the specified behavior to all warnings within the specified group from left to right. For `-warnings-as-errors` and `-no-warnings-as-errors`, we apply the behavior to all warnings. + +Examples of option combinations: +- `-warnings-as-errors -Wwarning Deprecated` + + Warnings from the `Deprecated` group will be kept as warnings, but all the rest will be upgraded to errors. + +- `-Werror Deprecated -Wwarning DeprecatedDeclaration` + + Warnings from the `DeprecatedDeclaration` group will remain as warnings. Other warnings from the `Deprecated` group will be upgraded to errors. All others will be kept as warnings. + +It’s crucial to understand that the order in which these flags are applied can significantly affect the behavior of diagnostics. The rule is "the last one wins", meaning that if multiple flags apply to the same diagnostic group, the last one specified on the command line will determine the final behavior. + +It is also important to note that the order matters even if the specified groups are not explicitly related but have a common subgroup. +For example, as mentioned above, the `UnsafeGlobalActorDeprecated` group is part of both the `Deprecated` and `Concurrency` groups. So the order in which options for the `Deprecated` and `Concurrency` groups are applied will change the final behavior of the `UnsafeGlobalActorDeprecated` group. Specifically: + +- `-Wwarning Deprecated -Werror Concurrency` will make it an error, +- `-Werror Concurrency -Wwarning Deprecated` will keep it as a warning. + +#### Interaction with `-suppress-warnings` + +This proposal deliberately excludes `-suppress-warnings` and its group-based counterpart from the new unified model. We retain the behavior of the existing `-suppress-warnings` flag but forbid its usage with the new options. The following rules will be applied: + +- It is forbidden to combine `-suppress-warnings` with `-Wwarning` or `-Werror`. The compiler will produce an error if these options are present in the command line together. +- It is allowed to be combined with `-no-warnings-as-errors`. The current compiler behavior permits the usage of `-no-warnings-as-errors` or `-warnings-as-errors -no-warnings-as-errors` with `-suppress-warnings`. We will maintain this behavior. +- It remains position-independent. Whenever `-no-warnings-as-errors` and `-suppress-warnings` are combined, `-suppress-warnings` will always take precedence over `-no-warnings-as-errors`, regardless of the order in which they are specified. + +### Usage of `-print-diagnostic-groups` and `-debug-diagnostic-names` + +As mentioned earlier, we are adding support for the `-print-diagnostic-groups` compiler option, which outputs the group name in square brackets. + +A similar behavior already exists in the compiler and is enabled by the `-debug-diagnostic-names` option, but it prints the internal diagnostic identifiers used in the compiler. For example: +```swift +@available(iOS, deprecated: 10.0, renamed: "newFunction") +func oldFunction() { ... } + +oldFunction() +``` +When compiled with the `-debug-diagnostic-names` option, the following message will be displayed: +``` +'oldFunction()' is deprecated: renamed to 'newFunction' [#RenamedDeprecatedDeclaration] +``` +The string `RenamedDeprecatedDeclaration` is the internal identifier of this warning, not the group. Accordingly, it is not supported by the new compiler options. + +When compiling the same code with the `-print-diagnostic-groups` option, the following message will be displayed: +``` +'oldFunction()' is deprecated: renamed to 'newFunction' [#DeprecatedDeclaration] +``` +Here, the string `DeprecatedDeclaration` is the diagnostic group. + +Often, group names and internal diagnostic identifiers coincide, but this is not always the case. + +We retain support for `-debug-diagnostic-names` in its current form. However, to avoid confusion between diagnostic IDs and diagnostic groups, we prohibit the simultaneous use of these two options. + +## Source compatibility + +This proposal has no effect on source compatibility. + +## ABI compatibility + +This proposal has no effect on ABI compatibility. + +## Implications on adoption + +The adoption of diagnostic groups and the new compiler options will provide a foundation for flexible and precise control over warning behavior. However, to make this useful to end-users, significant work will be needed to mark existing diagnostics in diagnostic groups. It will also be necessary to develop a process for maintaining the relevance of diagnostic groups when new diagnostics are introduced in the compiler. + +## Future directions + +### Support in the language + +While diagnostic groups are introduced to support the compiler options, it may be possible in the future to standardize the structure of the group graph itself. This could open up the possibility of using these same identifiers in the language, implementing something analogous to `#pragma diagnostic` or `[[attribute]]` in C++. It could also address suppressing warnings entirely, which isn't covered by this proposal. However, such standardization and the design of new language constructs go far beyond the scope of this proposal, and we need to gain more experience with diagnostic groups before proceeding with this. + +### Support in SwiftPM + +If this proposal is accepted, it would make sense to support these parameters in SwiftPM as well, allowing the behavior of warnings to be conveniently specified in SwiftSetting. + +## Alternatives considered + +### Alternatives to diagnostic groups +#### Status quo +The lack of control over the behavior of specific diagnostics forces users to abandon the `-warnings-as-errors` compiler option and create ad-hoc compiler wrappers that filter its output. + +#### Using existing diagnostic identifiers +Warnings and errors in Swift can change as the compiler evolves. +For example, one error might be renamed or split into two that are applied in different situations to improve the clarity of the text message depending on the context. Such a change would result in a new ID for the new error variant. + +The example of `DeprecatedDeclarationSameModule` illustrates this well. If we used the warning ID, the behavior of the compiler with the `-Wwarning DeprecatedDeclaration` option would change when a new version of the warning is introduced, as this warning would no longer be triggered for the specific case of the same module. + +Therefore, we need a solution that allows us to modify errors and warnings within the compiler while providing a reliable mechanism for identifying diagnostics that can be used by the user. + +#### Flat list instead of a graph + +To solve this problem, we could use an additional alias-ID for diagnostics that does not change when the main identifier changes. + +Suppose we split the `DeprecatedDeclaration` diagnostic into a generic variant and `DeprecatedDeclarationSameModule`. To retain the existing name for the new variant, we could describe these two groups as +``` +DeprecatedDeclaration (alias: DeprecatedDeclaration) +DeprecatedDeclarationSameModule (alias: DeprecatedDeclaration) +``` +However, this solution would not allow specifying the narrower `DeprecatedDeclarationSameModule` or the broader group `Deprecated`. + +#### Using multiple alias IDs for diagnostics +To express a diagnostic's membership in multiple groups, we could allow multiple alias-IDs to be listed. +``` +DeprecatedDeclaration aliases: + DeprecatedDeclaration + Deprecated +DeprecatedDeclarationSameModule aliases: + DeprecatedDeclarationSameModule + DeprecatedDeclaration + Deprecated +``` +However, such a declaration lacks structure and makes it difficult to understand which alias-ID is the most specific. + +### Alternative names for the compiler options + +During the design process, other names for the compiler options were considered, which were formed as the singular form of the existing ones: +| Plural | Singular | +|--------------------------|--------------------------------| +| `-warnings-as-errors` | `-warning-as-error ` | +| `-no-warnings-as-errors` | `-no-warning-as-error ` | + +In Clang, diagnostic behavior is controlled through `-W...` options, but the format suffers from inconsistency. We adopt the `-W` prefix while making the format consistent. +| Clang | Swift | +|-------------------|----------------------| +| `-W` | `-Wwarning ` | +| `-Wno-` | | +| `-Werror=` | `-Werror ` | + +The option name `-Wwarning` is much better suited when it comes to enabling suppressed-by-default warnings. Today we have several of them behind dedicated flags like `-driver-warn-unused-options` and `-warn-concurrency`. It might be worth having a common infrastructure for warnings that are suppressed by default. + +### Alternative format for `-print-diagnostic-groups` + +Theoretically, we could allow the simultaneous use of `-debug-diagnostic-names` and `-print-diagnostic-groups`, but this would require choosing a different format for printing diagnostic groups. + +Since `-debug-diagnostic-names` has been available in the compiler for a long time, we proceed from the fact that there are people who rely on this option and its format with square brackets. + +To avoid overlap, we would need to use a different format, for example: +``` +'foo()' is deprecated [#DeprecatedDeclaration] [group:#DeprecatedDeclaration] +``` + +However, even this does not eliminate the possibility of breaking code that parses the compiler's output. + +Moreover, `-print-diagnostic-groups` provides a formalized version of the same functionality using identifiers suitable for user use. And thus it should supersede the usages of `-debug-diagnostic-names`. Therefore, we believe the best solution would be to use the same format for `-print-diagnostic-groups` and prohibit the simultaneous use of these two options. + +## Revision History + +- Revisions based on review feedback: + - `-Wsuppress` was excluded from the proposal. + - `-suppress-warnings` was excluded from the unified model and addressed separately by forbidding its usage with the new flags. + - The guideline in the "Diagnostic Groups" subsection for adding a new diagnostic has been softened to a consideration. + +## Acknowledgments + +Thank you to [Frederick Kellison-Linn](https://forums.swift.org/u/Jumhyn) for the idea of addressing the `-suppress-warnings` behavior without incorporating it into the new model. diff --git a/proposals/0444-member-import-visibility.md b/proposals/0444-member-import-visibility.md new file mode 100644 index 0000000000..7904ebc4d5 --- /dev/null +++ b/proposals/0444-member-import-visibility.md @@ -0,0 +1,157 @@ +# Member import visibility + +* Proposal: [SE-0444](0444-member-import-visibility.md) +* Authors: [Allan Shortlidge](https://github.com/tshortli) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Implemented (Swift 6.1)** +* Bug: [apple/swift#46493](https://github.com/apple/swift/issues/46493) +* Implementation: [apple/swift#72974](https://github.com/apple/swift/pull/72974), [apple/swift#73063](https://github.com/apple/swift/pull/73063) +* Upcoming Feature Flag: `MemberImportVisibility` +* Review: ([pitch](https://forums.swift.org/t/pitch-fixing-member-import-visibility/71432)) ([review](https://forums.swift.org/t/se-0444-member-import-visibility/74519)) ([acceptance](https://forums.swift.org/t/accepted-se-0444-member-import-visibility/74966)) + +## Introduction + +In Swift, there are rules dictating whether the name of a declaration in another module is considered in scope. For example, if you have a program that uses the `swift-algorithms` package and you want to use the global function [chain()](https://github.com/apple/swift-algorithms/blob/33abb694280321a84aa7dc9806de284afb8ca226/Sources/Algorithms/Chain.swift#L287) then you must write `import Algorithms` in the file that references that function or the compiler will consider it out of scope: + +``` swift +// Missing 'import Algorithms' +let chained = chain([1], [2]) // error: Cannot find 'chain' in scope +``` + +The visibility rules for a member declaration, such as a method declared inside of a struct, are different though. When resolving a name to a member declaration, the member is in scope even if the module introducing the member is only *transitively* imported. A transitively imported module could be imported directly in another source file, or it could be a dependency of some direct dependency of your program. This inconsistency may be best understood as a subtle bug rather than an intentional design decision, and in a lot of Swift code it goes unnoticed. However, the import rules for members become more surprising when you consider the members of extensions, since an extension and its nominal type can be declared in different modules. + +This proposal unifies the behavior of name lookup by changing the rules to bring both top-level declarations and members into scope using the same criteria. + +## Motivation + +Suppose you have an app that depends on an external library named `RecipeKit`. To start, your app contains a single source file `main.swift` that imports `RecipeKit`: + +```swift +// main.swift +import RecipeKit + +let recipe = "2 slices of bread, 1.5 tbs peanut butter".parse() +``` + +The interface of `RecipeKit` looks like this: + +```swift +// RecipeKit interface + +public struct Recipe { /*...*/ } + +extension String { + /// Returns the recipe represented by this string. + public func parse() -> Recipe? +} +``` + +Later, you decide to integrate with a new library named `GroceryKit`. You add a second file to your app that imports `GroceryKit`: + +```swift +// Groceries.swift +import GroceryKit + +var groceries = GroceryList() +// ... +``` + +Surprisingly, after adding the second file there's now a compilation error in `main.swift`: + +```swift +// main.swift +import RecipeKit + +let recipe = "2 slices of bread, 1.5 tbs peanut butter".parse() +// error: Ambiguous use of 'parse()' +``` + +The call to `parse()` is now ambiguous because `GroceryKit` happens to also declares its own `parse()` method in an extension on `String`: + +```swift +// GroceryKit interface +public struct GroceryList { /*...*/ } + +extension String { + /// Returns the grocery list represented by this string. + public func parse() -> GroceryList? +} + +``` + +Even though `GroceryKit` was not imported in `main.swift`, its `parse()` method is now a candidate in that file. To resolve the ambiguity, you can add a type annotation to the declaration of the variable `recipe` to give the compiler the additional context it needs to disambiguate the call: + +```swift +let recipe: Recipe? = "2 slices of bread, 1.5 tbs peanut butter".parse() // OK +``` + +This example demonstrates why Swift's existing "leaky" member visibility is undesirable. Although the fix for the new error is relatively simple in this code, providing disambiguation context to the compiler is not always so straightforward. Additionally, the fact that some declarations from `GroceryKit` are now visible in `main.swift` contradicts developer expectations, since visibility rules for top level declarations do not behave this way. This idiosyncrasy in Swift's import visibility rules harms local reasoning and results in confusing errors. + +## Proposed solution + +In a future language version, or whenever the `MemberImportVisibility` feature is enabled, both member declarations and top level declarations should be resolved from the same set of visible modules in a given source file. + +## Detailed design + +A reference to a member in a source file will only be accepted if that member is declared in a module that is contained in the set of visible modules for that source file. According to the existing rules for top level name lookup, a module is visible if any of the following statements are true: + +- The module is directly imported. In other words, some import statement in the source file names the module explicitly. +- The module is directly imported from the bridging header. +- The module is in the set of modules that is re-exported by any module that is either directly imported in the file or directly imported in the bridging header. + +A re-exported module is one that another module makes visible to any client that imports it. Clang modules list the modules that they re-export in their modulemap files, and it is common for a Clang module to re-export every module it imports using `export *`. There is no officially supported way to re-export a module from a Swift module, but the compiler-internal `@_exported` attribute is sometimes used for this purpose. Re-exports are also transitive, so if module `A` re-exports module `B`, and module `B` re-exports module `C`, then declarations from `A`, `B`, and `C` are all in scope in a file that only imports `A` directly. + +Note that there are some imports that are added to every source file implicitly by the compiler for normal programs. The implicitly imported modules include the standard library and the module being compiled. As a subtle consequence implicitly importing the current module, any module that the current module re-exports in any of its source files will be considered visible in every other source file. + +## Source compatibility + +The proposed change in behavior is source breaking because it adds stricter requirements to name lookup. There is much existing Swift code that will need to be updated to adhere to these new requirements, either by introducing additional import statements in some source files or by reorganizing code among files. This change in behavior therefore must be opt-in, which is why it should be limited to a future language mode with an upcoming feature identifier that allows opt-in with previous language modes. + +## ABI compatibility + +This change does not affect ABI. + +## Implications on adoption + +To make it easier to migrate to the new language mode, the compiler can attempt to identify whether a member reference would resolve to a member declared in a transitively imported module and emit a fix-it to suggest adding a direct import to resolve the errors caused by the stricter look up rules: + +```swift +// In this example, RecipeKit is imported in another file + +// note: add import of module 'RecipeKit' + +let recipe = "1 scoop ice cream, 1 tbs chocolate syrup".parse() +// error: instance method 'parse()' is inaccessible due to missing import of defining module 'RecipeKit' +``` + +With these fix-its, the burden of updating source code to be compatible with the new language mode should be significantly reduced. + +This feature will have some impact on source compatibility with older compilers and previous language modes. Adding new direct imports of modules that were previously only transitively imported is a backward compatible change syntactically. However, it's possible that source code could depend on the new language mode in order to make the code unambiguous. In these cases additional measures, like adding explicit type annotations, might be required to maintain backward compatibility with previous language modes. + +## Future directions + +#### Add module qualification syntax for extension members + +This proposal seeks to give developers explicit control over which members are visible in a source file because this control can be used to prevent and resolve ambiguities that arise when different modules declare conflicting members in extensions. With this proposal implemented, if an extension member ambiguity still arises in a source file then the developer has the option of curating the imports in that file to resolve the ambiguity. This may work in some situations, but in others it may be awkward to refactor code in order to avoid importing a module that introduces a conflict. For these cases it would be useful to have a syntax that unambiguously identifies the desired extension member at the use site. For example, here's a hypothetical syntax for explicitly calling the `parse()` method declared in the module `RecipeKit`: + +```swift +let recipe = "...".RecipeKit::parse() +``` + +#### Implement rules for retroactive conformance visibility + +A typical protocol conformance in Swift is declared in either the module that declares the protocol or the module of the conforming type and this allows the compiler to enforce that conformances are globally unique. However, Swift also allows conformances to be "retroactive", which means that the conformance is declared in some third module and cannot be guaranteed to be unique. Retroactive conformances make it possible for multiple declarations of the same protocol conformance from different dependency modules to be simultaneously visible to the compiler. Currently, there is no defined way to influence which conformance declaration gets chosen when such an ambiguity arises, but a rule similar to the one in this proposal could be implemented for conformance lookup as well. For instance, a direct import of a module could be required in order to bring the conformances from the module into scope in a source file. + +#### Improve lookup for operators and precedence groups + +Swift has bespoke rules for looking up operators and operator precedence groups that differ in important ways from the rules for both top-level names and members. Documenting the existing rules and reforming them to bring more consistency to name lookup in Swift would be worthy of a future proposal. + +## Alternatives considered + +#### Introduce module qualification syntax for extension members instead + +One alternative approach to the problem would be to rely exclusively on a new syntax for disambiguation of extension members (as discussed in Future directions). The limitation of that approach is that alone it only empowers the developer to solve conflicts *reactively*. On the other hand, the solution provided by this proposal is preventative because it stops unnecessary conflicts from arising in the first place. In the fullness of time, it would be best for both solutions to be available simultaneously. + +## Acknowledgments + +I would like to thank Doug Gregor for providing a proof-of-concept implementation of this pitch. diff --git a/proposals/0445-string-index-printing.md b/proposals/0445-string-index-printing.md new file mode 100644 index 0000000000..f1c2beaa89 --- /dev/null +++ b/proposals/0445-string-index-printing.md @@ -0,0 +1,180 @@ +# Improving `String.Index`'s printed descriptions + +* Proposal: [SE-0445](0445-string-index-printing.md) +* Authors: [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.1)** +* Implementation: [apple/swift#75433](https://github.com/swiftlang/swift/pull/75433) +* Review: ([pitch](https://forums.swift.org/t/improving-string-index-s-printed-descriptions/57027)) ([review](https://forums.swift.org/t/se-0445-improving-string-indexs-printed-descriptions/74643)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0445-improving-string-index-s-printed-descriptions/75108)) +* Previous Revision: [v1](https://github.com/swiftlang/swift-evolution/blob/682f7c293a3a05bff3e619c3b479bfb68541fb6e/proposals/0445-string-index-printing.md) + +## Introduction + +This proposal conforms `String.Index` to `CustomDebugStringConvertible`. + +## Motivation + +String indices represent offsets from the start of the string's underlying storage representation, referencing a particular UTF-8 or UTF-16 code unit, depending on the string's encoding. (Most Swift strings are UTF-8 encoded, but strings bridged over from Objective-C may remain in their original UTF-16 encoded form.) + +If you ever tried printing a string index, you probably noticed that the output is gobbledygook: + +```swift +let string = "👋🏼 Helló" + +print(string.startIndex) // ⟹ Index(_rawBits: 15) +print(string.endIndex) // ⟹ Index(_rawBits: 983047) +print(string.utf16.index(after: string.startIndex)) // ⟹ Index(_rawBits: 16388) +print(string.firstRange(of: "ell")!) // ⟹ Index(_rawBits: 655623).. + + + + + + +## Detailed design + +``` +@available(SwiftStdlib 6.1, *) +extension String.Index: CustomDebugStringConvertible {} + +extension String.Index { + @backDeployed(before: SwiftStdlib 6.1) + public var debugDescription: String {...} +} +``` + +## Source compatibility + +The new conformance changes the result of converting a `String.Index` value to a string. This changes observable behavior: code that attempts to parse the result of `String(describing:)` or `String(reflecting:)` can be mislead by the change of format. + +However, the documentation of these interfaces explicitly state that when the input type conforms to none of the standard string conversion protocols, then the result of these operations is unspecified. + +Changing the value of an unspecified result is not considered to be a source incompatible change. + +## ABI compatibility + +The proposal retroactively conforms a previously existing standard type to a previously existing standard protocol. This is technically an ABI breaking change -- on ABI stable platforms, we may have preexisting Swift binaries that assume that `String.Index is CustomDebugStringConvertible` returns `false`, or ones that are implementing this conformance on their own. + +We do not expect this to be an issue in practice. + +## Implications on adoption + +The `String.Index.debugDescription` property is defined to be backdeployable, but the conformance itself is not. (It cannot be.) + +Code that runs on ABI stable platforms will not get the nicer displays when running on earlier versions of the Swift Standard Library. + +```swift +let str = "🐕 Doggo" +print(str.firstRange(of: "Dog")!) +// older stdlib: Index(_rawBits: 327943).. NotEscapable { + let ne = NotEscapable() + borrowingFunc(ne) // OK to pass to borrowing function + let another = ne // OK to make local copies + globalVar = ne // 🛑 Cannot assign ~Escapable type to a global var + return ne // 🛑 Cannot return ~Escapable type +} +``` + +**Note**: +The section ["Returned nonescapable values require lifetime dependency"](#Returns) explains the implications for how you must write initializers. + +Without `~Escapable`, the default for any type is to be escapable. Since `~Escapable` suppresses a capability, you cannot declare it with an extension. + +```swift +// Example: Escapable by default +struct Ordinary { } +extension Ordinary: ~Escapable // 🛑 Extensions cannot remove a capability +``` + +Classes cannot be declared `~Escapable`. + +#### In generic contexts, `~Escapable` suppresses the default Escapable requirement + +When used in a generic context, `~Escapable` allows you to define functions or types that can work with values that might or might not be escapable. +That is, `~Escapable` indicates the default escapable requirement has been suppressed. +Since the values might not be escapable, the compiler must conservatively prevent the values from escaping: + +```swift +func f(_ value: MaybeEscapable) { + // `value` might or might not be Escapable + globalVar = value // 🛑 Cannot assign possibly-nonescapable type to a global var +} +f(NotEscapable()) // Ok to call with nonescapable argument +f(7) // Ok to call with escapable argument +``` + +[SE-0427 Noncopyable Generics](https://github.com/apple/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md) provides more detail on +how suppressible protocols such as `Escapable` are handled in the generic type system. + +**Note:** There is no relationship between `Copyable` and `Escapable`. +Copyable or noncopyable types can be escapable or nonescapable. + +#### Constraints on nonescapable local variables + +A nonescapable value can be freely copied and passed into other functions, including async and throwing functions, as long as the usage can guarantee that the value does not persist beyond the current scope: + +```swift +// Example: Local variable with nonescapable type +func borrowingFunc(_: borrowing NotEscapable) { ... } +func consumingFunc(_: consuming NotEscapable) { ... } +func inoutFunc(_: inout NotEscapable) { ... } +func asyncBorrowingFunc(_: borrowing NotEscapable) async -> ResultType { ... } + +func f() { + var value: NotEscapable + let copy = value // OK to copy as long as copy does not escape + globalVar = value // 🛑 May not assign to global + SomeType.staticVar = value // 🛑 May not assign to static var + async let r = asyncBorrowingFunc(value) // OK to pass borrowing + borrowingFunc(value) // OK to pass borrowing + inoutFunc(&value) // OK to pass inout + consumingFunc(value) // OK to pass consuming + // `value` was consumed above, but NotEscapable + // is Copyable, so the compiler can insert + // a copy to satisfy the following usage: + borrowingFunc(value) // OK +} +``` + +#### Constraints on nonescapable parameters + +A value of nonescapable type received as a parameter is subject to the same constraints as any other local variable. +In particular, a nonescapable `consuming` parameter (and all direct copies thereof) must actually be destroyed during the execution of the function. +This is in contrast to an _escapable_ `consuming` parameter which can be disposed of by being returned or stored to an instance property or global variable. + +#### Types that contain nonescapable values must be nonescapable + +Stored struct properties and enum payloads can have nonescapable types if the surrounding type is itself nonescapable. +Equivalently, an escapable struct or enum can only contain escapable values. +Nonescapable values cannot be stored as class properties, since classes are always inherently escaping. + +```swift +// Example +struct EscapableStruct { + // 🛑 Escapable struct cannot have nonescapable stored property + var nonesc: Nonescapable +} + +enum EscapableEnum { + // 🛑 Escapable enum cannot have a nonescapable payload + case nonesc(Nonescapable) +} + +struct NonescapableStruct: ~Escapable { + var nonesc: Nonescapable // OK +} + +enum NonescapableEnum: ~Escapable { + case nonesc(Nonescapable) // OK +} +``` + +#### Returned nonescapable values require lifetime dependency + +As mentioned earlier, a simple return of a nonescapable value is not permitted: +```swift +func f() -> NotEscapable { // 🛑 Cannot return a nonescapable type + var value: NotEscapable + return value // 🛑 Cannot return a nonescapable type +} +``` + +A future proposal will describe “lifetime dependency annotations” that can relax this requirement by tying the lifetime of the returned value to the lifetime of another binding. +In particular, struct and enum initializers (which build a new value and return it to the caller) cannot be written without some such mechanism. + +#### Globals and static variables cannot be nonescapable + +Nonescapable values must be constrained to some specific local execution context. +This implies that they cannot be stored in global or static variables. + +#### Closures and nonescapable values + +Escaping closures cannot capture nonescapable values. +Nonescaping closures can capture nonescapable values subject to the usual exclusivity restrictions. + +Returning a nonescapable value from a closure will only be possible with explicit lifetime dependency annotations, to be covered in a future proposal. + +#### Nonescapable values and concurrency + +All of the requirements on use of nonescapable values as function parameters and return values also apply to async functions, including those invoked via `async let`. + +The closures used in `Task.init`, `Task.detached`, or `TaskGroup.addTask` are escaping closures and therefore cannot capture nonescapable values. + +#### Conditionally `Escapable` types + +You can define types whose escapability varies depending on their generic arguments. +As with other conditional behaviors, this is expressed by using an extension to conditionally add a new capability to the type: + +```swift +// Example: Conditionally Escapable generic type +// By default, Box is itself nonescapable +struct Box: ~Escapable { + var t: T +} + +// Box gains the ability to escape whenever its +// generic argument is Escapable +extension Box: Escapable where T: Escapable { } +``` + +This can be used in conjunction with other suppressible protocols. +For example, many general library container types will need to be copyable and/or escapable depending on their contents. +Here's a compact way to declare such a type: +```swift +struct Wrapper: ~Copyable, ~Escapable { ... } +extension Wrapper: Copyable where T: Copyable, T: ~Escapable {} +extension Wrapper: Escapable where T: Escapable, T: ~Copyable {} +``` + +## Source compatibility + +The compiler will treat any type without explicit `~Escapable` as escapable. +This matches the current behavior of the language. + +Only when new types are marked as `~Escapable` does this have any impact. + +Adding `~Escapable` to an existing concrete type is generally source-breaking because existing source code may rely on being able to escape values of this type. +Removing `~Escapable` from an existing concrete type is not generally source-breaking since it effectively adds a new capability, similar to adding a new protocol conformance. + +## ABI compatibility + +As above, existing code is unaffected by this change. +Adding or removing a `~Escapable` constraint on an existing type is an ABI-breaking change. + +## Implications on adoption + +Manglings and interface files will only record the lack of escapability. +This means that existing interfaces consumed by a newer compiler will treat all types as escapable. +Similarly, an old compiler reading a new interface will have no problems as long as the new interface does not contain any `~Escapable` types. + +These same considerations ensure that escapable types can be shared between previously-compiled code and newly-compiled code. + +Retrofitting existing generic types so they can support both escapable and nonescapable type arguments is possible with care. + +## Future directions + +#### `Span` family of types + +This proposal is being driven in large part by the needs of the `Span` types that have been discussed elsewhere. +Briefly, this type would provide an efficient universal “view” of array-like data stored in contiguous memory. +Since values of this type do not own any data but only refer to data stored elsewhere, their lifetime must be limited to not exceed that of the owning storage. +We expect to publish a sample implementation and proposal for that type very soon. + +#### Initializers and Lifetime Dependencies + +Nonescapable function parameters may not outlive the function scope. +Consequently, nonescapable values can never be returned from a function. +Nonescapable values come into existence within the body of the initializer. +Naturally, the initializer must return its value, and this creates an exception to the rule. +The parameters to the initializer typically indicate a lifetime that the nonescapable value cannot outlive. +An initializer may, for example, create a nonescapable value that depends on a container variable that is bound to an object with its own lifetime: +```swift +struct Iterator: ~Escapable { + init(container: borrowing Container) { ... } +} + +let container = ... +let iterator = Iterator(container) +consume container // `container` lifetime ends here +use(iterator) // 🛑 'iterator' outlives `container` +``` + +Specifying a dependency from a function parameter to its nonescapable result currently requires an experimental lifetime dependency feature. +With lifetime dependencies, initialization of nonescapable types is safe: misuses similar to the one shown above are compile-time errors. +Adopting new syntax for lifetime dependencies merits a separate, focussed review. +Until then, initialization of nonescapable values remains experimental. + +#### Expanding standard library types + +We expect that many standard library types will need to be updated to support possibly-nonescapable types, including `Optional`, `Array`, `Set`, `Dictionary`, and the `Unsafe*Pointer` family of types. + +Some of these types will require first exploring whether it is possible for the `Collection`, `Iterator`, `Sequence`, and related protocols to adopt these concepts directly or whether we will need to introduce new protocols to complement the existing ones. + +The more basic protocols such as `Equatable`, `Comparable`, and `Hashable` should be easier to update. + +#### Refining `with*` closure-taking APIs + +The `~Escapable` types can be used to refine common `with*` closure-taking APIs by ensuring that the closures cannot save or hold their arguments beyond their own lifetime. +For example, this can greatly improve the safety of locking APIs that expect to unlock resources upon completion of the closure. + +#### Nonescapable classes + +We’ve explicitly excluded class types from being nonescapable. +In the future, we could allow class types to be declared nonescapable as a way to avoid most reference-counting operations on class objects. + +#### Concurrency + +Structured concurrency implies lifetime constraints similar to those outlined in this proposal. +It may be appropriate to incorporate `~Escapable` into the structured concurrency primitives. + +For example, the current `TaskGroup` type is supposed to never be escaped from the local context; +making it `~Escapable` would prevent this type of abuse and possibly enable other optimizations. + +#### Global nonescapable types with immortal lifetimes + +This proposal currently prohibits putting values with nonescapable types into global or static variables. +We expect to eventually allow this by explicitly annotating a “static” or “immortal” lifetime. + +## Alternatives considered + +#### Require `Escapable` to indicate escapable types without using `~Escapable` + +We could avoid using `~Escapable` to mark types that lack the `Escapable` property by requiring `Escapable` on all escapable types. +However, it is infeasible to require updating all existing types in all existing Swift code with a new explicit capability. + +Apart from that, we expect almost all types to continue to be escapable in the future, so the negative marker reduces the overall burden. +It is also consistent with progressive disclosure: +Most new Swift programmers should not need to know details of how escapable types work, since that is the common behavior of most data types in most programming languages. +When developers use existing nonescapable types, specific compiler error messages should guide them to correct usage without needing to have a detailed understanding of the underlying concepts. +With our current proposal, the only developers who will need detailed understanding of these concepts are library authors who want to publish nonescapable types. + +#### `Nonescapable` as a marker protocol + +We considered introducing `Nonescapable` as a marker protocol indicating that the values of this type required additional compiler checks. +With that approach, you would define a conditionally-escapable type such as `Box` above in this fashion: + +```swift +// Box does not normally require additional escapability checks +struct Box { + var t: T +} + +// But if T requires additional checks, so does Box +extension Box: Nonescapable where T: Nonescapable { } +``` + +However, this would imply that any `Nonescapable` type was a +subtype of `Any` and could therefore be placed within an `Any` existential box. +An `Any` existential box is both `Copyable` and `Escapable`, +so it cannot be allowed to contain a nonescapable value. + +#### Rely on `~Copyable` + +As part of the `Span` design, we considered whether it would suffice to use `~Copyable` instead of introducing a new type concept. +Andrew Trick's analysis in [Language Support for Bufferview](https://forums.swift.org/t/roadmap-language-support-for-bufferview/66211) concluded that making `Span` be non-copyable would not suffice to provide the full semantics we want for that type. +Further, introducing `Span` as `~Copyable` would actually preclude us from later expanding it to be `~Escapable`. + +The iterator example in the beginning of this document provides another motivation: +Iterators are routinely copied in order to record a particular point in a collection. +Thus we concluded that non-copyable was not the correct lifetime restriction for types of this sort, and it was worthwhile to introduce a new lifetime concept to the language. + +## Acknowledgements + +Many people discussed this proposal and gave important feedback, including: Kavon Farvardin, Meghana Gupta, John McCall, Slava Pestov, Joe Groff, Guillaume Lessard, and Franz Busch. diff --git a/proposals/0447-span-access-shared-contiguous-storage.md b/proposals/0447-span-access-shared-contiguous-storage.md new file mode 100644 index 0000000000..71b3ff8aa4 --- /dev/null +++ b/proposals/0447-span-access-shared-contiguous-storage.md @@ -0,0 +1,799 @@ +# Span: Safe Access to Contiguous Storage + +* Proposal: [SE-0447](0447-span-access-shared-contiguous-storage.md) +* Authors: [Guillaume Lessard](https://github.com/glessard), [Michael Ilseman](https://github.com/milseman), [Andrew Trick](https://github.com/atrick) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [BufferView Roadmap](https://forums.swift.org/t/66211) +* Bug: rdar://48132971, rdar://96837923 +* Implementation: [apple/swift#76406](https://github.com/swiftlang/swift/pull/76406) +* Review: ([Pitch 1](https://forums.swift.org/t/69888))([Pitch 2](https://forums.swift.org/t/72745))([Review](https://forums.swift.org/t/se-0447-span-safe-access-to-contiguous-storage/74676))([Acceptance](https://forums.swift.org/t/accepted-se-0447-span-safe-access-to-contiguous-storage/75508)) + +## Introduction + +We introduce `Span`, an abstraction for container-agnostic access to contiguous memory. It will expand the expressivity of performant Swift code without compromising on the memory safety properties we rely on: temporal safety, spatial safety, definite initialization and type safety. + +In the C family of programming languages, memory can be shared with any function by using a pointer and (ideally) a length. This allows contiguous memory to be shared with a function that doesn't know the layout of a container being used by the caller. A heap-allocated array, contiguously-stored named fields or even a single stack-allocated instance can all be accessed through a C pointer. We aim to enable a similar idiom in Swift, without compromising Swift's memory safety. + +This proposal builds on [Nonescapable types][PR-2304] (`~Escapable`,) and is a precursor to [Compile-time Lifetime Dependency Annotations][PR-2305], which will be proposed in the following weeks. The [BufferView roadmap](https://forums.swift.org/t/66211) forum thread was an antecedent to this proposal. This proposal also depends on the following proposals: + +- [SE-0426] BitwiseCopyable +- [SE-0427] Noncopyable generics +- [SE-0437] Non-copyable Standard Library Primitives +- [SE-0377] `borrowing` and `consuming` parameter ownership modifiers + +[SE-0426]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0426-bitwise-copyable.md +[SE-0427]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md +[SE-0437]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0437-noncopyable-stdlib-primitives.md +[SE-0377]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0377-parameter-ownership-modifiers.md +[PR-2304]: https://github.com/swiftlang/swift-evolution/pull/2304 +[PR-2305]: https://github.com/swiftlang/swift-evolution/pull/2305 +[PR-2305-pitch]: https://forums.swift.org/t/69865 + +## Motivation + +Swift needs safe and performant types for local processing over values in contiguous memory. Consider for example a program using multiple libraries, including one for [base64](https://datatracker.ietf.org/doc/html/rfc4648) decoding. The program would obtain encoded data from one or more of its dependencies, which could supply the data in the form of `[UInt8]`, `Foundation.Data` or even `String`, among others. None of these types is necessarily more correct than another, but the base64 decoding library must pick an input format. It could declare its input parameter type to be `some Sequence`, but such a generic function can significantly limit performance. This may force the library author to either declare its entry point as inlinable, or to implement an internal fast path using `withContiguousStorageIfAvailable()`, forcing them to use an unsafe type. The ideal interface would have a combination of the properties of both `some Sequence` and `UnsafeBufferPointer`. + +The `UnsafeBufferPointer` passed to a `withUnsafeXXX` closure-style API, while performant, is unsafe in multiple ways: + +1. The pointer itself is unsafe and unmanaged +2. `subscript` is only bounds-checked in debug builds of client code +3. It might escape the duration of the closure + +Even if the body of the `withUnsafeXXX` call does not escape the pointer, other functions called within the closure have to be written in terms of unsafe pointers. This requires programmer vigilance across a project and potentially spreads the use of unsafe types, even if the helper functions could have been written in terms of safe constructs. + +## Proposed solution + +#### `Span` + +`Span` will allow sharing the contiguous internal representation of a type, by providing access to a borrowed view of an interval of contiguous memory. `Span` does not copy the underlying data: it instead relies on a guarantee that the original container cannot be modified or destroyed while the `Span` exists. In the prototype that accompanies this first proposal, `Span`s will be constrained to closures from which they structurally cannot escape. Later, we will introduce a lifetime dependency between a `Span` and the binding of the type vending it, preventing its escape from the scope where it is valid for use. Both of these approaches guarantee temporal safety. `Span` also performs bounds-checking on every access to preserve spatial safety. Additionally `Span` always represents initialized memory, preserving the definite initialization guarantee. + +`Span` is intended as the currency type for local processing over values in contiguous memory. It is a replacement for many API currently using `Array`, `UnsafeBufferPointer`, `Foundation.Data`, etc., that do not need to escape the owning container. + +A `Span` provided by a container represents a borrow of that container. `Span` can therefore provide simultaneous access to a non-copyable container. It can also help avoid unwanted copies of copyable containers. Note that `Span` is not a replacement for a copyable container with owned storage; see [future directions](#Directions) for more details ([Resizable, contiguously-stored, untyped collection in the standard library](#Bytes).) + +In this initial proposal, no initializers are proposed for `Span`. Initializers for non-escapable types such as `Span` require a concept of lifetime dependency, which does not exist at this time. The lifetime dependency annotation will indicate to the compiler how a newly-created `Span` can be used safely. See also ["Initializers"](#Initializers) in [future directions](#Directions). + +#### `RawSpan` + +`RawSpan` allows sharing contiguous memory representing values which may be heterogeneously-typed, such as memory intended for parsing. It makes the same safety guarantees as `Span`. Since it is a fully concrete type, it can achieve great performance in debug builds of client code as well as straightforward performance in library code. + +A `RawSpan` can be obtained from containers of `BitwiseCopyable` elements, as well as be initialized directly from an instance of `Span`. + +## Detailed design + +`Span` is a simple representation of a region of initialized memory. + +```swift +@frozen +public struct Span: Copyable, ~Escapable { + internal var _start: UnsafeRawPointer? + internal var _count: Int +} + +extension Span: Sendable where Element: Sendable & ~Copyable {} +``` + +We store a `UnsafeRawPointer` value internally in order to explicitly support reinterpreted views of memory as containing different types of `BitwiseCopyable` elements. Note that the the optionality of the pointer does not affect usage of `Span`, since accesses are bounds-checked and the pointer is only dereferenced when the `Span` isn't empty, and the pointer cannot be `nil`. + +It provides a buffer-like interface to the elements stored in that span of memory: + +```swift +extension Span where Element: ~Copyable { + public var count: Int { get } + public var isEmpty: Bool { get } + + public typealias Index = Int + public var indices: Range { get } + + public subscript(_ index: Index) -> Element { _read } +} +``` + +Note that `Span` does _not_ conform to `Collection`. This is because `Collection`, as originally conceived and enshrined in existing source code, assumes pervasive copyability and escapability of the `Collection` itself as well as of element type. In particular a subsequence of a `Collection` is semantically a separate value from the instance it was derived from. In the case of `Span`, a sub-span representing a subrange of its elements _must_ have the same lifetime as the `Span` from which it originates. Another proposal will consider collection-like protocols to accommodate different combinations of `~Copyable` and `~Escapable` for the collection and its elements. + +Like `UnsafeBufferPointer`, `Span` uses a simple offset-based indexing. The first element of a given span is always at position zero, and its last element is always at position `count-1`. + +As a side-effect of not conforming to `Collection` or `Sequence`, `Span` is not directly supported by `for` loops at this time. It is, however, easy to use in a `for` loop via indexing: + +```swift +for i in mySpan.indices { + calculation(mySpan[i]) +} +``` + +### `Span` API: + +Initializers, required for library adoption, will be proposed alongside [lifetime annotations][PR-2305]; for details, see "[Initializers](#Initializers)" in the [future directions](#Directions) section. + +##### Basic API: + +The following properties, functions and subscripts have direct counterparts in the `Collection` protocol hierarchy. Their semantics shall be as described where they counterpart is declared (in `Collection` or `RandomAccessCollection`). + +```swift +extension Span where Element: ~Copyable { + /// The number of initialized elements in the span. + public var count: Int { get } + + /// A Boolean value indicating whether the span is empty. + public var isEmpty: Bool { get } + + /// The type that represents a position in `Span`. + public typealias Index = Int + + /// The range of indices valid for this `Span` + public var indices: Range { get } + + /// Accesses the element at the specified position. + public subscript(_ position: Index) -> Element { _read } +} +``` + +Note that we use a `_read` accessor for the subscript, a requirement in order to `yield` a borrowed non-copyable `Element` (see ["Coroutines"](#Coroutines).) This yields an element whose lifetime is scoped around this particular access, as opposed to matching the lifetime dependency of the `Span` itself. This is a language limitation we expect to resolve with a followup proposal introducing a new accessor model. The subscript will then be updated to use the new accessor semantics. We expect the updated accessor to be source-compatible, as it will provide a borrowed element with a wider lifetime than a `_read` accessor can provide. + +##### Unchecked access to elements: + +The `subscript` mentioned above has always-on bounds checking of its parameter, in order to prevent out-of-bounds accesses. We also want to provide unchecked variants as an alternative for cases where bounds-checking is proving costly, such as in tight loops: + +```swift +extension Span where Element: ~Copyable { + + /// Accesses the element at the specified `position`. + /// + /// This subscript does not validate `position`; this is an unsafe operation. + /// + /// - Parameter position: The offset of the element to access. `position` + /// must be greater or equal to zero, and less than `count`. + public subscript(unchecked position: Index) -> Element { _read } +} +``` + +When using the unchecked subscript, the index must be known to be valid. While we are not proposing explicit index validation API on `Span` itself, its `indices` property can be use to validate a single index, in the form of the function `Range.contains(_: Int) -> Bool`. We expect that `Range` will also add efficient containment checking of a subrange's endpoints, which should be generally useful for index range validation in this and other contexts. + +##### Identifying whether a `Span` is a subrange of another: + +When working with multiple `Span` instances, it is often desirable to know whether one is identical to or a subrange of another. We include functions to determine whether this is the case, as well as a function to obtain the valid offsets of the subrange within the larger span: + +```swift +extension Span where Element: ~Copyable { + /// Returns true if the other span represents exactly the same memory + public func isIdentical(to span: borrowing Self) -> Bool + + /// Returns the indices within `self` where the memory represented by `span` + /// is located, or `nil` if `span` is not located within `self`. + /// + /// Parameters: + /// - span: a span that may be a subrange of `self` + /// Returns: A range of offsets within `self`, or `nil` + public func indices(of span: borrowing Self) -> Range? +} +``` + +##### Interoperability with unsafe code: + +We provide two functions for interoperability with C or other legacy pointer-taking functions. + +```swift +extension Span where Element: ~Copyable { + /// Calls a closure with a pointer to the viewed contiguous storage. + /// + /// The buffer pointer passed as an argument to `body` is valid only + /// during the execution of `withUnsafeBufferPointer(_:)`. + /// Do not store or return the pointer for later use. + /// + /// - Parameter body: A closure with an `UnsafeBufferPointer` parameter + /// that points to the viewed contiguous storage. If `body` has + /// a return value, that value is also used as the return value + /// for the `withUnsafeBufferPointer(_:)` method. The closure's + /// parameter is valid only for the duration of its execution. + /// - Returns: The return value of the `body` closure parameter. + func withUnsafeBufferPointer( + _ body: (_ buffer: UnsafeBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} + +extension Span where Element: BitwiseCopyable { + /// Calls the given closure with a pointer to the underlying bytes of + /// the viewed contiguous storage. + /// + /// The buffer pointer passed as an argument to `body` is valid only + /// during the execution of `withUnsafeBytes(_:)`. + /// Do not store or return the pointer for later use. + /// + /// - Parameter body: A closure with an `UnsafeRawBufferPointer` + /// parameter that points to the viewed contiguous storage. + /// If `body` has a return value, that value is also + /// used as the return value for the `withUnsafeBytes(_:)` method. + /// The closure's parameter is valid only for the duration of + /// its execution. + /// - Returns: The return value of the `body` closure parameter. + func withUnsafeBytes( + _ body: (_ buffer: UnsafeRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} +``` + +These functions use a closure to define the scope of validity of `buffer`, ensuring that the underlying `Span` and the binding it depends on both remain valid through the end of the closure. They have the same shape as the equivalents on `Array` because they fulfill the same function, namely to keep the underlying binding alive. + +### RawSpan + +In addition to `Span`, we propose the addition of `RawSpan`, to represent heterogeneously-typed values in contiguous memory. `RawSpan` is similar to `Span`, but represents _untyped_ initialized bytes. `RawSpan` is a specialized type that is intended to support parsing and decoding applications, as well as applications where heavily-used code paths require concrete types as much as possible. Its API supports the data loading operations `unsafeLoad(as:)` and `unsafeLoadUnaligned(as:)`. + +#### `RawSpan` API: + +```swift +@frozen +public struct RawSpan: Copyable, ~Escapable { + internal var _start: UnsafeRawPointer + internal var _count: Int +} + +extension RawSpan: Sendable {} +``` + +Initializers, required for library adoption, will be proposed alongside [lifetime annotations][PR-2305]; for details, see "[Initializers](#Initializers)" in the [future directions](#Directions) section. + +##### Accessing the memory of a `RawSpan`: + +`RawSpan` has basic operations to access the contents of its memory: `unsafeLoad(as:)` and `unsafeLoadUnaligned(as:)`: + +```swift +extension RawSpan { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// The memory at this pointer plus `offset` must be properly aligned for + /// accessing `T` and initialized to `T` or another type that is layout + /// compatible with `T`. + /// + /// This is an unsafe operation. Failure to meet the preconditions + /// above may produce an invalid value of `T`. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance is memory-managed and unassociated + /// with the value in the memory referenced by this pointer. + public func unsafeLoad( + fromByteOffset offset: Int = 0, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// The memory at this pointer plus `offset` must be initialized to `T` + /// or another type that is layout compatible with `T`. + /// + /// This is an unsafe operation. Failure to meet the preconditions + /// above may produce an invalid value of `T`. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance isn't associated + /// with the value in the range of memory referenced by this pointer. + public func unsafeLoadUnaligned( + fromByteOffset offset: Int = 0, as: T.Type + ) -> T +``` + +These operations are not type-safe, in that the loaded value returned by the operation can be invalid, and violate type invariants. Some types have a property that makes the `unsafeLoad(as:)` function safe, but we don't have a way to [formally identify](#SurjectiveBitPattern) such types at this time. + +The `unsafeLoad` functions have counterparts which omit bounds-checking for cases where redundant checks affect performance: + +```swift + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// The memory at this pointer plus `offset` must be properly aligned for + /// accessing `T` and initialized to `T` or another type that is layout + /// compatible with `T`. + /// + /// This is an unsafe operation. This function does not validate the bounds + /// of the memory access, and failure to meet the preconditions + /// above may produce an invalid value of `T`. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance is memory-managed and unassociated + /// with the value in the memory referenced by this pointer. + public func unsafeLoad( + fromUncheckedByteOffset offset: Int, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + /// + /// The memory at this pointer plus `offset` must be initialized to `T` + /// or another type that is layout compatible with `T`. + /// + /// This is an unsafe operation. This function does not validate the bounds + /// of the memory access, and failure to meet the preconditions + /// above may produce an invalid value of `T`. + /// + /// - Parameters: + /// - offset: The offset from this pointer, in bytes. `offset` must be + /// nonnegative. The default is zero. + /// - type: The type of the instance to create. + /// - Returns: A new instance of type `T`, read from the raw bytes at + /// `offset`. The returned instance isn't associated + /// with the value in the range of memory referenced by this pointer. + public func unsafeLoadUnaligned( + fromUncheckedByteOffset offset: Int, as: T.Type + ) -> T +} +``` + +`RawSpan` provides `withUnsafeBytes` for interoperability with C or other legacy pointer-taking functions: + +```swift +extension RawSpan { + /// Calls the given closure with a pointer to the underlying bytes of + /// the viewed contiguous storage. + /// + /// The buffer pointer passed as an argument to `body` is valid only + /// during the execution of `withUnsafeBytes(_:)`. + /// Do not store or return the pointer for later use. + /// + /// - Parameter body: A closure with an `UnsafeRawBufferPointer` + /// parameter that points to the viewed contiguous storage. + /// If `body` has a return value, that value is also + /// used as the return value for the `withUnsafeBytes(_:)` method. + /// The closure's parameter is valid only for the duration of + /// its execution. + /// - Returns: The return value of the `body` closure parameter. + func withUnsafeBytes( + _ body: (_ buffer: UnsafeRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} +``` + +##### Examining `RawSpan` bounds: + +```swift +extension RawSpan { + /// The number of bytes in the span. + public var byteCount: Int { get } + + /// A Boolean value indicating whether the span is empty. + public var isEmpty: Bool { get } + + /// The range of valid byte offsets into this `RawSpan` + public var byteOffsets: Range { get } +} +``` + +##### Identifying whether a `RawSpan` is a subrange of another: + +When working with multiple `RawSpan` instances, it is often desirable to know whether one is identical to or a subrange of another. We include a function to determine whether this is the case, as well as a function to obtain the valid offsets of the subrange within the larger span. The documentation is omitted here, as it is substantially the same as for the equivalent functions on `Span`: + +```swift +extension RawSpan { + public func isIdentical(to span: borrowing Self) -> Bool + + public func byteOffsets(of span: borrowing Self) -> Range? +} +``` + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a new version of the standard library and runtime. + +## Alternatives considered + +##### Make `Span` a noncopyable type +Making `Span` non-copyable was in the early vision of this type. However, we found that would make `Span` a poor match to model borrowing semantics. This realization led to the initial design for non-escapable declarations. + +##### Use a non-escapable index type +A non-escapable index type implies that any indexing operation would borrow its `Span`. This would prevent using such an index for a mutation, since a mutation requires an _exclusive_ borrow. Noting that the usage pattern we desire for `Span` must also apply to `MutableSpan`(described [below](#MutableSpan),) a non-escapable index would make it impossible to also implement a mutating subscript, unless any mutating operation consumes the index. This seems untenable. + +##### Naming + +The ideas in this proposal previously used the name `BufferView`. While the use of the word "buffer" would be consistent with the `UnsafeBufferPointer` type, it is nevertheless not a great name, since "buffer" is commonly used in reference to transient storage. Another previous pitch used the term `StorageView` in reference to the `withContiguousStorageIfAvailable()` standard library function. We also considered the name `StorageSpan`, but that did not add much beyond the shorter name `Span`. `Span` clearly identifies itself as a relative of C++'s `std::span`. + +The OpenTelemetry project and its related libraries use the word "span" for a concept of a timespan. The domains of use between that and direct memory access are very distinct, and we believe that the confusability between the use cases should be low. We also note that standard library type names can always be shadowed by type names from packages, mitigating the risk of source breaks. + +##### Sendability of `RawSpan` + +This proposal makes `RawSpan` a `Sendable` type. We believe this is the right decision. The sendability of `RawSpan` could be used to unsafely transfer a pointer value across an isolation boundary, despite the non-sendability of pointers. For example, suppose a `RawSpan` were obtained from an existing `Array` variable. We could send the `RawSpan` across the isolation boundary, and there extract the pointer using `rawSpan.unsafeLoad(as: UnsafeRawPointer.self)`. While this is an unsafe outcome, a similar operation can be done encoding a pointer as an `Int`, and then using `UnsafeRawPointer(bitPattern: mySentInt)` on the other side of the isolation boundary. + +##### A more sophisticated approach to indexing + +This is discussed more fully in the [indexing appendix](#Indexing) below. + +## Future directions + +#### Initializing and returning `Span` instances + +A `Span` represents a region of memory and, as such, must be initialized using an unsafe pointer. This is an unsafe operation which will typically be performed internally to a container's implementation. In order to bridge to safe code, these initializers require new annotations that indicate to the compiler how the newly-created `Span` can be used safely. + +These annotations have been [pitched][PR-2305-pitch] and are expected to be formally [proposed][PR-2305] soon. `Span` initializers using lifetime annotations will be proposed alongside the annotations themselves. + +#### Obtaining variant `Span`s and `RawSpan`s from `Span` and `RawSpan` + +`Span`s representing subsets of consecutive elements could be extracted out of a larger `Span` with an API similar to the `extracting()` functions recently added to `UnsafeBufferPointer` in support of non-copyable elements: + +```swift +extension Span where Element: ~Copyable { + public func extracting(_ bounds: Range) -> Self +} +``` + +Each variant of such a function needs to return a `Span`, which requires a lifetime dependency. + +Similarly, a `RawSpan` should be initializable from a `Span`, and `RawSpan` should provide a function to unsafely view its content as a typed `Span`: + +```swift +extension RawSpan { + public init(_ span: Span) + + public func unsafeView(as type: T.Type) -> Span +} +``` + +We are subsetting these functions of `Span` and `RawSpan` until the lifetime annotations are proposed. + +#### Coroutine or Projection Accessors + +This proposal includes some `_read` accessors, the coroutine version of the `get` accessor. `_read` accessors are not an official part of the Swift language, but are necessary for some types to be able to provide borrowing access to their internal storage, in particular storage containing non-copyable elements. The correct solution may involve a projection of a different type than is provided by a coroutine. When correct, stable replacement for `_read` accessors is proposed and accepted, the implementation of `Span` and `RawSpan` will be adapted to the new syntax. + +#### Extensions to Standard Library and Foundation types + +The standard library and Foundation has a number of types that can in principle provide access to their internal storage as a `Span`. We could provide `withSpan()` and `withBytes()` closure-taking functions as safe replacements for the existing `withUnsafeBufferPointer()` and `withUnsafeBytes()` functions. We could also provide lifetime-dependent `span` or `bytes` properties. For example, `Array` could be extended as follows: + +```swift +extension Array { + public func withSpan( + _ body: (_ elements: Span) throws(E) -> Result + ) throws(E) -> Result + + public var span: Span { borrowing get } +} + +extension Array where Element: BitwiseCopyable { + public func withBytes( + _ body: (_ bytes: RawSpan) throws(E) -> Result + ) throws(E) -> Result where Element: BitwiseCopyable + + public var bytes: RawSpan { borrowing get } +} +``` + +Of these, the closure-taking functions can be implemented now, but it is unclear whether they are desirable. The lifetime-dependent computed properties require lifetime annotations, as initializers do. We are deferring proposing these extensions until the lifetime annotations are proposed. + +#### A `ContiguousStorage` protocol + +An earlier version of this proposal proposed a `ContiguousStorage` protocol by which a type could indicate that it can provide a `Span`. `ContiguousStorage` would form a bridge between generically-typed interfaces and a performant concrete implementation. It would supersede the rejected [SE-0256](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0256-contiguous-collection.md). + +For example, for the hypothetical base64 decoding library mentioned in the [motivation](#Motivation) section, a possible API could be: + +```swift +extension HypotheticalBase64Decoder { + public func decode(bytes: some ContiguousStorage) -> [UInt8] +} +``` + +`ContiguousStorage` would have the following definition: + +```swift +public protocol ContiguousStorage: ~Copyable, ~Escapable { + associatedtype Element: ~Copyable & ~Escapable + var storage: Span { _read } +} +``` + +Two issues prevent us from proposing it at this time: (a) the ability to suppress requirements on `associatedtype` declarations was deferred during the review of [SE-0427], and (b) we cannot declare a `_read` accessor as a protocol requirement. + +Many of the standard library collections could conform to `ContiguousStorage`. + +#### Index Validation Utilities + +This proposal originally included index validation utilities for `Span`. such as `boundsContain(_: Index) -> Bool` and `boundsContain(_: Range) -> Bool`. After review feedback, we believe that the utilities proposed would also be useful for index validation on `UnsafeBufferPointer`, `Array`, and other similar `RandomAccessCollection` types. `Range` already a single-element `contains(_: Bound) -> Bool` function which can be made even more efficient. We should add an additional function that identifies whether a `Range` contains the _endpoints_ of another `Range`. Note that this is not the same as the existing `contains(_: some Collection) -> Bool`, which is about the _elements_ of the collection. This semantic difference can lead to different results when examining empty `Range` instances. + +#### Support for `Span` in `for` loops + +This proposal does not define an `IteratorProtocol` conformance, since an iterator for `Span` would need to be non-escapable. This is not compatible with `IteratorProtocol`. As such, `Span` is not directly usable in `for` loops as currently defined. A `BorrowingIterator` protocol for non-escapable and non-copyable containers must be defined, providing a `for` loop syntax where the element is borrowed through each iteration. Ultimately we should arrive at a way to iterate through borrowed elements from a borrowed view: + +```swift +func doSomething(_ e: borrowing Element) { ... } +let span: Span = ... +for borrowing element in span { + doSomething(element) +} +``` + +In the meantime, it is possible to loop through a `Span`'s elements by direct indexing: + +```swift +let span: Span = ... +// either: +var i = 0 +while i < span.count { + doSomething(span[i]) + i += 1 +} + +// ...or: +for i in 0..Layout constraint for safe loading of bit patterns + +`RawSpan` has unsafe functions that interpret the raw bit patterns it contains as values of arbitrary `BitwiseCopyable` types. In order to have safe alternatives to these, we could add a layout constraint refining `BitwiseCopyable`, specifically for types whose mapping from bit pattern to values is a [surjective function](https://en.wikipedia.org/wiki/Surjective_function) (e.g. `SurjectiveBitPattern`). Such types would be safe to [load](#Load) from `RawSpan` instances. 1-byte examples are `Int8` (any of 256 values are valid) and `Bool` (256 bit patterns map to `true` or `false` because only one bit is considered.) + +An alternative to a layout constraint is to add a type validation step to ensure that if a given bit pattern were to be interpreted as a value of type `T`, then all the invariants of type `T` would be respected. This alternative would be more flexible, but may have a higher runtime cost. + +#### Byte parsing helpers + +We could add some API to `RawSpan` to make it better suited for binary parsers and decoders. + +```swift +extension RawSpan { + public struct Cursor: Copyable, ~Escapable { + public let base: RawSpan + + /// The current parsing position + public var position: Int + + /// Parse an instance of `T` and advance. + /// Returns `nil` if there are not enough bytes remaining for an instance of `T`. + public mutating func parse( + _ t: T.Type = T.self + ) -> T? + + /// Parse `numBytes`and advance. + /// Returns `nil` if there are fewer than `numBytes` remaining. + public mutating func parse( + numBytes: some FixedWidthInteger + ) -> RawSpan? + + /// The bytes that we've parsed so far + public var parsedBytes: RawSpan { get } + } +} +``` + +`Cursor` stores and manages a parsing subrange, which alleviates the developer from managing one layer of slicing. + +Alternatively, if some future `RawSpan.Iterator` were 3 words in size (start, current position, and end) instead of 2 (current pointer and end), making it a "resettable", it could host this API instead of introducing a new `Cursor` type or concept. + +##### Example: Parsing PNG + +The code snippet below parses a [PNG Chunk](https://www.w3.org/TR/png-3/#4Concepts.FormatChunks), using the byte parsing helpers defined above: + +```swift +// Parse a PNG chunk +let length = try cursor.parse(UInt32.self).bigEndian +let type = try cursor.parse(UInt32.self).bigEndian +let data = try cursor.parse(numBytes: length) +let crc = try cursor.parse(UInt32.self).bigEndian +``` + +#### Safe mutations of memory with `MutableSpan` + +Some data structures can delegate mutations of their owned memory. In the standard library the function `withMutableBufferPointer()` provides this functionality in an unsafe manner. + +The `UnsafeMutableBufferPointer` passed to a `withUnsafeMutableXXX` closure-style API is unsafe in multiple ways: + +1. The pointer itself is unsafe and unmanaged +2. `subscript` is only bounds-checked in debug builds of client code +3. It might escape the duration of the closure +4. Exclusivity of writes is not enforced +5. Initialization of any particular memory address is not ensured + +in other words, it is unsafe in all the same ways as `UnsafeBufferPointer`-passing closure APIs, in addition to enforcing neither exclusivity nor initialization state. + +Loading an uninitialized non-`BitwiseCopyable` value leads to undefined behavior. Loading an uninitialized `BitwiseCopyable` value does not immediately lead to undefined behavior, but it produces a garbage value which may lead to misbehavior of the program. + +A `MutableSpan` should provide a better, safer alternative to mutable memory in the same way that `Span` provides a better, safer read-only type. `MutableSpan` would apply to initialized memory and would enforce exclusivity of writes, thereby preserving the initialization state of its memory between mutations. + +#### Delegating initialization of memory with `OutputSpan` + +Some data structures can delegate initialization of their initial memory representation, and in some cases the initialization of additional memory. For example, the standard library features the initializer`Array.init(unsafeUninitializedCapacity:initializingWith:)`, which depends on `UnsafeMutableBufferPointer` and is known to be error-prone. A safer abstraction for initialization would make such initializers less dangerous, and would allow for a greater variety of them. + +We can define an `OutputSpan` type, which could support appending to the initialized portion of a data structure's underlying storage. `OutputSpan` allows for uninitialized memory beyond the last position appended. Such an `OutputSpan` would also be a useful abstraction to pass user-allocated storage to low-level API such as networking calls or file I/O. + +#### Resizable, contiguously-stored, untyped collection in the standard library + +The example in the [motivation](#Motivation) section mentions the `Foundation.Data` type. There has been some discussion of either replacing `Data` or moving it to the standard library. This document proposes neither of those. A major issue is that in the "traditional" form of `Foundation.Data`, namely `NSData` from Objective-C, it was easier to control accidental copies because the semantics of the language did not lead to implicit copying. + +Even if `Span` were to replace all uses of a constant `Data` in API, something like `Data` would still be needed, for the same reason as `Array` is needed: such a type allows for resizing mutations (e.g. `RangeReplaceableCollection` conformance.) We may want to add an untyped-element equivalent of `Array` to the standard library at a later time. + +#### Syntactic Sugar for Automatic Conversions + +Even with a `ContiguousStorage` protocol, a generic entry point in terms of `some ContiguousStorage` may add unwanted overhead to resilient libraries. As detailed above, an entry point in an evolution-enabled library requires an inlinable generic public entry point which forwards to a publicly-accessible function defined in terms of `Span`. If `Span` does become a widely-used type to interface between libraries, we could simplify these conversions with a bit of compiler help. + +We could provide an automatic way to use a `ContiguousStorage`-conforming type with a function that takes a `Span` of the appropriate element type: + +```swift +func myStrnlen(_ b: Span) -> Int { + guard let i = b.firstIndex(of: 0) else { return b.count } + return b.distance(from: b.startIndex, to: e) +} +let data = Data((0..<9).reversed()) // Data conforms to ContiguousStorage +let array = Array(data) // Array also conforms to ContiguousStorage +myStrnlen(data) // 8 +myStrnlen(array) // 8 +``` + +This would probably consist of a new type of custom conversion in the language. A type author would provide a way to convert from their type to an owned `Span`, and the compiler would insert that conversion where needed. This would enhance readability and reduce boilerplate. + +#### Interoperability with C++'s `std::span` and with llvm's `-fbounds-safety` + +The [`std::span`](https://en.cppreference.com/w/cpp/container/span) class template from the C++ standard library is a similar representation of a contiguous range of memory. LLVM may soon have a [bounds-checking mode](https://discourse.llvm.org/t/70854) for C. These are opportunities for better, safer interoperation with Swift, via a type such as `Span`. + +## Acknowledgments + +Joe Groff, John McCall, Tim Kientzle, Steve Canon and Karoy Lorentey contributed to this proposal with their clarifying questions and discussions. + +### Appendix: Index and slicing design considerations + +Early prototypes of this proposal defined an `Index` type, `Iterator` types, etc. We are proposing `Int`-based API and are deferring defining `Index` and `Iterator` until more of the non-escapable collection story is sorted out. The below is some of our research into different potential designs of an `Index` type. + +There are 3 potentially-desirable features of `Span`'s `Index` design: + +1. `Span` is its own slice type +2. Indices from a slice can be used on the base collection +3. Additional reuse-after-free checking + +Each of these introduces practical tradeoffs in the design. + +#### `Span` is its own slice type + +Collections which own their storage have the convention of separate slice types, such as `Array` and `String`. This has the advantage of clearly delineating storage ownership in the programming model and the disadvantage of introducing a second type through which to interact. + +When types do not own their storage, separate slice types can be [cumbersome](https://github.com/swiftlang/swift/blob/swift-5.10.1-RELEASE/stdlib/public/core/StringComparison.swift#L175). The reason `UnsafeBufferPointer` has a separate slice type is because it wants to allow indices to be reused across slices and its `Index` is a relative offset from the start (`Int`) rather than an absolute position (such as a pointer). + +`Span` does not own its storage and there is no concern about leaking larger allocations. It would benefit from being its own slice type. + +#### Indices from a slice can be used on the base collection + +There is very strong stdlib precedent that indices from the base collection can be used in a slice and vice-versa. + +```swift +let myCollection = [0,1,2,3,4,5,6] +let idx = myCollection.index(myCollection.startIndex, offsetBy: 4) +myCollection[idx] // 4 +let slice = myCollection[idx...] // [4, 5, 6] +slice[idx] // 4 +myCollection[slice.indices] // [4, 5, 6] +``` + +Code can be written to take advantage of this fact. For example, a simplistic parser can be written as mutating methods on a slice. The slice's indices can be saved for reference into the original collection or another slice. + +```swift +extension Slice where Base == UnsafeRawBufferPointer { + mutating func parse(numBytes: Int) -> Self { + let end = index(startIndex, offsetBy: numBytes) + defer { self = self[end...] } + return self[.. Int { + parse(numBytes: MemoryLayout.stride).loadUnaligned(as: Int.self) + } + + mutating func parseHeader() -> Self { + // Comments show what happens when ran with `myCollection` + + let copy = self + parseInt() // 0 + parseInt() // 1 + parse(numBytes: 8) // [2, 0, 0, 0, 0, 0, 0, 0] + parseInt() // 3 + parse(numBytes: 7) // [4, 0, 0, 0, 0, 0, 0] + + // self: [0, 5, 0, 0, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0] + parseInt() // 1280 (0x00_00_05_00 little endian) + // self: [0, 6, 0, 0, 0, 0, 0, 0, 0] + + return copy[..( + _ c: C +) -> C.Element where C.Index == Int { + c[0] +} + +getFirst(myCollection) // 0 +getFirst(slice) // Fatal error: Index out of bounds +``` + +#### Additional reuse-after-free checking + +`Span` bounds-checks its indices, which is important for safety. If the index is based around a pointer (instead of an offset), then bounds checks will also ensure that indices are not used with the wrong span in most situations. However, it is possible for a memory address to be reused after being freed and using a stale index into this reused memory may introduce safety problems. + +```swift +var idx: Span.Index + +let array1: Array = ... +let span1 = array1.span +idx = span1.startIndex.advanced(by: ...) +... +// array1 is freed + +let array2: Array = ... +let span2 = array2.span +// array2 happens to be allocated within the same memory of array1 +// but with a different base address whose offset is not an even +// multiple of `MemoryLayout.stride`. + +span2[idx] // misaligned load, what happens? +``` + +If `T` is `BitwiseCopyable`, then the misaligned load is not undefined behavior, but the value that is loaded is garbage. Whether the program is well-behaved going forwards depends on whether it is resilient to getting garbage values. + +If `T` is not `BitwiseCopyable`, then the misaligned load may introduce undefined behavior. No matter how well-written the rest of the program is, it has a critical safety and security flaw. + +When the reused allocation happens to be stride-aligned, there is no undefined behavior from undefined loads, nor are there "garbage" values in the strictest sense, but it is still reflective of a programming bug. The program may be interacting with an unexpected value. + +Bounds checks protect against critical programmer errors. It would be nice, pending engineering tradeoffs, to also protect against some reuse after free errors and invalid index reuse, especially those that may lead to undefined behavior. + +Future improvements to microarchitecture may make reuse after free checks cheaper, however we need something for the foreseeable future. Any validation we can do reduces the need to switch to other mitigation strategies or make other tradeoffs. + +#### Design approaches for indices + +##### Index is an offset (`Int` or a wrapper around `Int`) + +When `Index` is an offset, there is no undefined behavior from misaligned loads because the `Span`'s base address is advanced by `MemoryLayout.stride * offset`. + +However, there is no protection against invalidly using an index derived from a different span, provided the offset is in-bounds. + +Since `Span` is 2 words (base address and count), indices cannot be interchanged between slices and the base span. In order to do so, `Span` would need to additionally store a base offset, bringing it up to 3 words in size. + +##### Index is a pointer (wrapper around `UnsafeRawPointer`) + +When Index holds a pointer, `Span` only needs to be 2 words in size, as valid index interchange across slices falls out naturally. Additionally, invalid reuse of an index across spans will typically be caught during bounds checking. + +However, in a reuse-after-free situation, misaligned loads (i.e. undefined behavior) are possible. If stride is not a multiple of 2, then alignment checking can be expensive. Alternatively, we could choose not to detect these bugs. + +##### Index is a fat pointer (pointer and allocation ID) + +We can create a per-allocation ID (e.g. a cryptographic `UInt64`) for both `Span` and `Span.Index` to store. This would make `Span` 3 words in size and `Span.Index` 2 words in size. This provides the most protection possible against all forms of invalid index use, including reuse-after-free. However, making `Span` be 3 words and `Span.Index` 2 words for this feature is unfortunate. + +We could instead go with 2 word `Span` and 2 word `Span.Index` by storing the span's `baseAddress` in the `Index`'s second word. This will detect invalid reuse of indices across spans in addition to misaligned reuse-after-free errors. However, indices could not be interchanged without a way for the slice type to know the original span's base address (e.g. through a separate slice type or making `Span` 3 words in size). + +In either approach, making `Span.Index` be 2 words in size is unfortunate. `Range` is now 4 words in size, storing the allocation ID twice. Anything built on top of `Span` that wishes to store multiple indices is either bloated or must hand-extract the pointers and hand-manage the allocation ID. diff --git a/proposals/0448-regex-lookbehind-assertions.md b/proposals/0448-regex-lookbehind-assertions.md new file mode 100644 index 0000000000..744bdbd85c --- /dev/null +++ b/proposals/0448-regex-lookbehind-assertions.md @@ -0,0 +1,135 @@ +# Regex lookbehind assertions + +* Proposal: [SE-0448](0448-regex-lookbehind-assertions.md) +* Authors: [Jacob Hearst](https://github.com/JacobHearst) [Michael Ilseman](https://github.com/milseman) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Accepted** +* Implementation: https://github.com/swiftlang/swift-experimental-string-processing/pull/760 +* Review: + ([pitch](https://forums.swift.org/t/pitch-regex-reverse-matching/73482)) + ([review](https://forums.swift.org/t/se-0448-regex-lookbehind-assertions/74672)) + ([acceptance](https://forums.swift.org/t/accepted-se-0448-regex-lookbehind-assertions/75111)) + + +## Introduction + +Regex supports lookahead assertions, but does not currently support lookbehind assertions. We propose adding these. + +## Motivation + +Modern regular expression engines support lookbehind assertions, whether fixed length (Perl, PCRE2, Python, Java) or arbitrary length (.NET, Javascript). + +## Proposed solution + +We propose supporting arbitrary-length lookbehind regexes which can be achieved by performing matching in reverse. + +Like lookahead assertions, lookbehind assertions are _zero-width_, meaning they do not affect the current match position. + +Examples: + + +```swift +"abc".firstMatch(of: /a(?<=a)bc/) // matches "abc" +"abc".firstMatch(of: /a(?<=b)c/) // no match +"abc".firstMatch(of: /a(?<=.)./) // matches "ab" +"abc".firstMatch(of: /ab(?<=a)c/) // no match +"abc".firstMatch(of: /ab(?<=.a)c/) // no match +"abc".firstMatch(of: /ab(?<=a.)c/) // matches "abc" +``` + +Lookbehind assertions run in reverse, i.e. right-to-left, meaning that right-most eager quantifications have the opportunity to consume more of the input than left-most. This does not affect whether an input matches, but could affect the value of captures inside of a lookbehind assertion: + +```swift +"abcdefg".wholeMatch(of: /(.+)(.+)/) +// Produces ("abcdefg", "abcdef", "g") + +"abcdefg".wholeMatch(of: /.*(?<=(.+)(.+)/)) +// Produces ("abcdefg", "a", "bcdefg") +``` + +## Detailed design + + +### Syntax + +Lookbehind assertion syntax is already supported in the existing [Regex syntax](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0355-regex-syntax-run-time-construction.md#lookahead-and-lookbehind). + +The engine is currently incapable of running them, so a compilation error is thrown: + +```swift +let regex = /(?<=a)b/ +// error: Cannot parse regular expression: lookbehind is not currently supported +``` + +With this proposal, this restriction is lifted and the following syntactic forms will be accepted: + +```swift +// Positive lookbehind +/a(?<=b)c/ +/a(*plb:b)c/ +/a(*positive_lookbehind:b)c/ + +// Negative lookbehind +/a(?) -> Self + + /// The trait's canonical name. + /// + /// This is used when enabling the trait or when referring to it from other modifiers in the manifest. + /// + /// The following rules are enforced on trait names: + /// - The first character must be a [Unicode XID start character](https://unicode.org/reports/tr31/#Figure_Code_Point_Categories_for_Identifier_Parsing) + /// (most letters), a digit, or `_`. + /// - Subsequent characters must be a [Unicode XID continue character](https://unicode.org/reports/tr31/#Figure_Code_Point_Categories_for_Identifier_Parsing) + /// (a digit, `_`, or most letters), `-`, or `+`. + /// - `default` and `defaults` (in any letter casing combination) are not allowed as trait names to avoid confusion with default traits. + public var name: String + + /// The trait's description. + /// + /// Use this to explain what functionality this trait enables. + public var description: String? + + /// A set of other traits of this package that this trait enables. + public var enabledTraits: Set + + /// Initializes a new trait. + /// + /// - Parameters: + /// - name: The trait's canonical name. + /// - description: The trait's description. + /// - enabledTraits: A set of other traits of this package that this trait enables. + public init( + name: String, + description: String? = nil, + enabledTraits: Set = [] + ) + + /// Initializes a new trait. + /// + /// This trait is disabled by default and enables no other trait of this package. + public init(stringLiteral value: StringLiteralType) + + /// Initializes a new trait. + /// + /// - Parameters: + /// - name: The trait's canonical name. + /// - description: The trait's description. + /// - enabledTraits: A set of other traits of this package that this trait enables. + public static func trait( + name: String, + description: String? = nil, + enabledTraits: Set = [] + ) -> Trait +} +``` + +The `Package` class is extended to define a set of traits: + +```swift +public final class Package { + // ... + + /// The set of traits of this package. + public var traits: Set + + /// Initializes a Swift package with configuration options you provide. + /// + /// - Parameters: + /// - name: The name of the Swift package, or `nil` to use the package's Git URL to deduce the name. + /// - defaultLocalization: The default localization for resources. + /// - platforms: The list of supported platforms with a custom deployment target. + /// - pkgConfig: The name to use for C modules. If present, Swift Package Manager searches for a + /// `.pc` file to get the additional flags required for a system target. + /// - providers: The package providers for a system target. + /// - products: The list of products that this package makes available for clients to use. + /// - traits: The set of traits of this package. + /// - dependencies: The list of package dependencies. + /// - targets: The list of targets that are part of this package. + /// - swiftLanguageVersions: The list of Swift versions with which this package is compatible. + /// - cLanguageStandard: The C language standard to use for all C targets in this package. + /// - cxxLanguageStandard: The C++ language standard to use for all C++ targets in this package. + public init( + name: String, + defaultLocalization: LanguageTag? = nil, + platforms: [SupportedPlatform]? = nil, + pkgConfig: String? = nil, + providers: [SystemPackageProvider]? = nil, + products: [Product] = [], + traits: Set = [], + dependencies: [Dependency] = [], + targets: [Target] = [], + swiftLanguageVersions: [SwiftVersion]? = nil, + cLanguageStandard: CLanguageStandard? = nil, + cxxLanguageStandard: CXXLanguageStandard? = nil + ) +} +``` + +Furthermore, a new `Package.Dependency.Trait` type is introduced that can be used +to configure the traits of a dependency. + +```swift +extension Package.Dependency { + /// A struct representing an enabled trait of a dependency. + public struct Trait: Hashable, Sendable, ExpressibleByStringLiteral { + /// Enables the default traits of a package. + public static let default: Self + + /// A condition that limits the application of a dependencies trait. + public struct Condition: Hashable, Sendable { + /// Creates a package dependency trait condition. + /// + /// - Parameter traits: The set of traits that enable the dependencies trait. If any of the traits are enabled on this package + /// the dependencies trait will be enabled. + public static func when( + traits: Set + ) -> Self? + } + + /// The name of the enabled trait. + public var name: String + + /// The condition under which the trait is enabled. + public var condition: Condition? + + /// Initializes a new enabled trait. + /// + /// - Parameters: + /// - name: The name of the enabled trait. + /// - condition: The condition under which the trait is enabled. + public init( + name: String, + condition: Condition? = nil + ) + + public init(stringLiteral value: StringLiteralType) + + /// Initializes a new enabled trait. + /// + /// - Parameters: + /// - name: The name of the enabled trait. + /// - condition: The condition under which the trait is enabled. + public static func trait( + name: String, + condition: Condition? = nil + ) -> Trait + } +} +``` + +The dependency APIs are then extended with new variants that take a `Trait` parameter: + +```swift +extension Package.Dependency { + // MARK: Path + + public static func package( + path: String, + traits: Set + ) -> Package.Dependency + + public static func package( + name: String, + path: String, + traits: Set + ) -> Package.Dependency + + // MARK: Source repository + + public static func package( + url: String, + from version: Version, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + branch: String, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + revision: String, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + _ range: Range, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + _ range: ClosedRange, + traits: Set + ) -> Package.Dependency + + public static func package( + url: String, + exact version: Version, + traits: Set + ) -> Package.Dependency + + // MARK: Registry + + public static func package( + id: String, + from version: Version, + traits: Set + ) -> Package.Dependency + + public static func package( + id: String, + exact version: Version, + traits: Set + ) -> Package.Dependency + + public static func package( + id: String, + _ range: Range, + traits: Set + ) -> Package.Dependency + + public static func package( + id: String, + _ range: ClosedRange, + traits: Set + ) -> Package.Dependency +} +``` + +Lastly, traits can also be used to conditionalize `SwiftSettings`, `CSettings`, +`CXXSettings` and `LinkerSettings`. For this the `BuildSettingCondition` is extended. + +```swift +/// Creates a build setting condition. +/// +/// - Parameters: +/// - platforms: The applicable platforms for this build setting condition. +/// - configuration: The applicable build configuration for this build setting condition. +/// - traits: The applicable traits for this build setting condition. +public static func when( + platforms: [Platform]? = nil, + configuration: BuildConfiguration? = nil, + traits: Set? = nil +) -> BuildSettingCondition { + precondition(!(platforms == nil && configuration == nil)) + return BuildSettingCondition(platforms: platforms, config: configuration, traits: nil) +} +``` + +### Trait unification + +At this point, it is important to talk about the trait unification across the +entire dependency graph. After dependency resolution the union of enabled traits +per package is calculated. This is then used to determine both the enabled +optional dependencies and the enabled traits for the compile time checks. Since +the enabled traits of a dependency are specified on a per package level and not +from the root of the tree, any combination of enabled traits must be supported. +A consequence of this is that all traits **should** be _additive_. Enabling a trait +**should not** disable functionality i.e. remove API or lead to any other +**SemVer-incompatible** change. + +#### Mutally exclusive traits + +Some rare use-cases may want mutally exclusive traits which are incompatible to +be enabled at the same time. This should be avoided if possible because it +requires the whole dependency graph to coordinate on what trait to enable. In +the rare case where mutually exclusive traits are used consider adding a +compiler error to detect this during build time. + +```swift +#if Trait1 && Trait2 +#error("Trait1 and Trait2 are mutually exclusive") +#endif +``` + +A few options to avoid mutually exclusive traits: +- Separate the code into multiple packages +- Choose one trait over the other when possible +- Use platform checks `#if os(Windows)` when possible + +### Default traits + +Default traits allow package authors to define a set of traits that they think +cater to the majority use-cases of the package. When choosing the initial +default traits or adding a new default trait it is important to consider that +removing a default trait is a **SemVer-incompatible** change since it can potentially +remove APIs. + +### Trait specific command line options for `swift test/build/run` + +When executing one of `swift test/build/run` options can be passed to control which +traits for the root package are enabled: + +- `--traits` _TRAITS_: Enables the passed traits of the package. Multiple traits + can be specified by providing a comma separated list e.g. `--traits + Trait1,Trait2`. +- `--enable-all-traits`: Enables all traits of the package. +- `--disable-default-traits`: Disables all default traits of the package. + +### Trait namespaces + +Trait names are namespaced per package; hence, multiple packages can define the +same trait names. Moreover, it is an expected scenario that multiple packages +define the same trait name and conditionally enable the equivalent named trait +in their dependencies. + +### Trait limitations + +To prevent abuse, limit the complexity and make sure it integrates with the +compiler a few limitations are imposed. + +#### Number of traits + +[Other +ecosystems](https://blog.rust-lang.org/2023/10/26/broken-badges-and-23k-keywords.html) +have shown that a large number of traits can have significant impact on +registries and dependency managers. To avoid such a scenario an initial maximum +number of 300 defined traits per package is imposed. This can be revisited later +once traits have been used in the ecosystem extensively. + +### Allowed characters for trait names + +Since traits can show up both in the `Package.swift` and in source code when +checking if a trait is enabled, the allowed characters for a trait name are +restricted to [legal Swift +identifier](https://docs.swift.org/swift-book/documentation/the-swift-programming-language/summaryofthegrammar/). +Additional, the following rules are enforced on trait names: + +- `default` and `defaults` (in any letter casing combination) are not allowed as + trait names to avoid confusion with default traits. + +### swift package dump-package + +The `swift package dump-package` command will include information of the trait configuration for the +package and the dependencies it uses in the JSON output. + +## Impact on existing packages + +There is no impact on existing packages. Any package can start adopting package +traits but in doing so **must not** move existing API behind new traits. Even if +the trait is a enabled by default any consumer might have already disabled all +default traits; hence, moving API behind a new default trait could potentially +break them. + +### Impact on other build systems + +The initial impact on other build systems should be minimal. Exisiting packages +must not move exisiting APIs behind a trait. For new APIs that are guarded by a +trait a build system must pass the correct `SWIFT_ACTIVE_COMPILATION_CONDITIONS` +when building the modules of the package. Other build systems might want to +consider to expose a way to model traits in their target description. + +### Impact on documentation + +Traits are used to conditionally compile code. When building documentation the +symbol graph extracter will only see the code that is actually compiled. Systems +that produce documentation for packages should default to building with all +traits enabled so that all API documentation is visible. + +## Future directions + +### Consider traits during dependency resolution + +The implementation to this proposal only considers traits **after** the +dependency resolution when constructing the module graph. This is inline with +how platform specific dependencies are currently handled. In the future, both +platform specific dependencies and traits can be taken into consideration during +dependency resolution to avoid fetching an optional dependency that is not +enabled by a trait. Changing this **doesn't** require a Swift evolution proposal +since it is just an implementation detail of how dependency resolution currently +works. + +### Integrated compiler trait checking + +The current proposal passes enabled traits via custom defines to the compiler +and code can check it using regular define checks (`#if DEFINE`). In the future, +we can extend the compiler to make it aware of package traits to allows syntax +like `#if trait(FOO)` or implement an extensible configuration macro similar to +Rust's `cfg` macro. + +### Enabled trait compile time checking + +Since trait unification is done for every package in the graph during build time +the information which module enabled which trait of its dependencies is lost. +As a consequence it might be that a package accidentally uses an API from a +dependency which is guarded by a trait that another package in the graph has +enabled. Since the traits that any one package in the graph enables on its +dependencies are not considered part of the semantic version, it can happen that +disabling a trait could result in breaking a build. In the future, we could +integrate trait checking further into the compiler where it understands if an +API is only available if a certain trait is set. + +> Cargo currently [treats this +similar](https://users.rust-lang.org/t/is-disabling-features-of-a-dependency-considered-a-breaking-change/94302/2) +and doesn't consider disabling a cargo feature a breaking change. + +### Different default traits depending on platform + +A future evolution could allow to mark traits as default depending on the +platform that the package is build on. This would allow packages such as the +`swift-openapi-generator` to default the used transport depending on the +platform which makes it even easier to offer users the best out of box +experience. This is left as a future evolution since it intersects interestingly +with the future direction "Consider traits during dependency resolution". If +default traits depend on the target build platform then this must be an input to +the dependency resolution. + +### Globally configured traits + +One use-case where mutually exclusive traits are used is to configure the +behaviour of a single package globally. This means that only the final +executable is deciding what trait to enable. In a future proposal, we could +introduce a new setting for marking a trait as globally configured and check +that only executable targets are enabling such a trait. + +### Tooling to test different trait combinations + +Since a single package should support building with any combination of traits, +it would be helpful to offer package authors tooling to build and test all +combinations. A new option `--all-trait-combinations` could be added to +`swift test/build/run` make testing all combinations easy as possible. + +### Surface traits in documentation + +If the compiler gains knowledge about package traits in the future, we could +extract information if a public API is guarded by a trait and surface this in +the documentation. + +## Alternatives considered + +### Different naming + +During the implementation and writing of the proposal different names for +_package traits_ have been considered such as: +- Package features +- Package optional features +- Package options +- Package parameters +- Package flags +- Package configuration + +A lot of the other considered names have other meanings in the language already. +For example `feature` is already used in expressing compiler feature via +`enable[Upcoming|Experimental]Feature` and the `hasFeature` check. + +_Package traits_ are also consistent with [the "traits" concept in the +`swift-testing` +library](https://github.com/apple/swift-testing/blob/25d0eed9b339de36365ff16deb9a3d9c64322f1c/Sources/Testing/Traits/Trait.swift#L22). + +### Using SPI instead + +During the investigation how to solve the optional dependency problem `@_spi` +was considered; however, the problem with `@_spi` is that the code is still +compiled and present in the final binary. Optional dependencies can't work like +this since the symbols potentially aren't present during compile time. + +### Enum based traits + +Traits could be expressed via an enum in the package description which would +make sure that they are statically typed. This proposal decided to use `String` +based trait names instead to align with the other definitions inside the package +description such as `targets` or `products`. + +## Prior art + +Other dependency managers have similar features to control optional dependencies +and conditional compilation. + +- [Cargo](https://doc.rust-lang.org/cargo/) has [optional features](https://doc.rust-lang.org/cargo/reference/features.html) that allow conditional compilation and optional dependencies. +- [Maven](https://maven.apache.org/) has [optional dependencies](https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html). +- [Gradle](https://gradle.org/) has [feature variants](https://docs.gradle.org/current/userguide/feature_variants.html) that allow conditional compilation and optional dependencies. +- [Go](https://golang.org/) has [build constraints](https://golang.org/pkg/go/build/#hdr-Build_Constraints) which can conditionally include a file. +- [pip](https://pypi.org/project/pip/) dependencies can have [optional dependencies and extras](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#optional-dependencies). +- [Hatch](https://hatch.pypa.io/latest/) offers [optional dependencies](https://hatch.pypa.io/latest/config/metadata/#optional) and [features](https://hatch.pypa.io/latest/config/dependency/#features). diff --git a/proposals/0451-escaped-identifiers.md b/proposals/0451-escaped-identifiers.md new file mode 100644 index 0000000000..3105c97626 --- /dev/null +++ b/proposals/0451-escaped-identifiers.md @@ -0,0 +1,464 @@ +# Raw identifiers + +* Proposal: [SE-0451](0451-escaped-identifiers.md) +* Author: [Tony Allevato](https://github.com/allevato) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#76636](https://github.com/swiftlang/swift/pull/76636), [swiftlang/swift-syntax#2857](https://github.com/swiftlang/swift-syntax/pull/2857) +* Previous Proposal: [SE-0275](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-revisiting-backtick-delimited-identifiers-that-allow-more-non-identifier-characters/74432), [review](https://forums.swift.org/t/se-0451-raw-identifiers/75602), [acceptance](https://forums.swift.org/t/accepted-with-revision-se-0451-raw-identifiers/76387)) + + +## Introduction + +This proposal adds _raw identifiers_ to the Swift grammar, which are backtick-delimited identifiers that can contain characters other than the current set of identifier-allowed characters in the language. + +## Motivation + +When naming things in Swift, we use identifiers that are limited to a specific set of characters defined by the [Swift grammar](https://docs.swift.org/swift-book/documentation/the-swift-programming-language/summaryofthegrammar/). For the vast majority of identifiers, this set is perfectly reasonable: names of APIs should be no more complex than they need to be, and many natural names do fit within these requirements. However, there do exist some use cases that would be improved by more flexibility in naming: + +* Declarations whose names serve a purely descriptive purpose, like tests. +* Externally-defined entities that already have natural names that do not fit within Swift's rules for identifiers. + +The unifying principle behind these motivating cases is that information is lost or complexity is unnecessarily increased when a developer must forego a more natural name for one that meets Swift's requirements. Indeed, Swift already admits this to a degree by defining the backtick syntax to allow reserved keywords to be used as identifiers. + +### Descriptive test naming + +When this feature was originally proposed as [SE-0275](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md), one of the reasons for rejection was that ["[t]he core team feels that this use case could be better handled by test library design that allowed strings to be associated with test cases, rather than try to encode that information entirely in symbol names."](https://forums.swift.org/t/se-0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers/32538/46) + +We now have a new testing library in the Swift project called [swift-testing](https://github.com/apple/swift-testing) that implements the feature described above: + +```swift +@Test("square returns x * x") +func squareIsXTimesX() { + #expect(square(4) == 4 * 4) +} +``` + +Unfortunately, if the user wants to provide a descriptive name for the test, they are now forced to name it **twice**: once in the `@Test` macro and then again in the function itself. As the number of tests increases (see swift-testing's own tests, for [example](https://github.com/swiftlang/swift-testing/blob/main/Tests/TestingTests/ClockTests.swift)), so, too, does the burden on the developer. + +This is not merely redundant; it also creates an inconsistency in how the tests are referenced by different tools. Test result reports that are generated by the framework will use the display name, as may user interfaces that display the test hierarchy. However, lower-level tooling such as the compiler itself, the linker, the debugger, index data used for code navigation, and backtraces, will only show the Swift language symbol. + +Re-designing the testing framework to eliminate the function name would also not be the correct solution. Consider this hypothetical alternative using a trailing closure that would remove one of the redundant names: + +```swift +// Not being proposed +@Test("square returns x * x") { + #expect(square(4) == 4 * 4) +} +``` + +This would be unsatisfactory for a few reasons: + +* The current `@Test func` syntax used by swift-testing is a strength because it provides progressive disclosure rather than introducing a new bespoke DSL for tests. +* There are subtle differences between how the compiler processes closures and regular function declarations that could result in awkward behavior for test authors. +* The testing framework must still contrive a symbol name for the test by either mangling the user-provided description into something that Swift can support or creating a completely unique name. The user now has no control over that name and no easy way to discover it. That name would still appear in the debugger, index data, and backtraces, so the inconsistency described above still remains. + +### Naturally non-alphabetic identifiers + +There are some situations where the best name for a particular symbol is numeric or at least starts with a number. As an example, consider a hypothetical color design system that lets users choose a base hue and then a variant/shade of that hue from a fixed set. These design systems often give the color variants numeric names that indicate their intensity—100, 200, and so forth. A naïve API might represent the variant as a numeric value: + +```swift +struct Color { + init(hue: Hue, variant: Int) +} +``` + +But this API can only enforce correctness at runtime. A developer should naturally reach for an `enum` type for a fixed set of variants like this, but they must transform the name or add unnecessary ceremony to make it fit into Swift's naming rules: + +```swift +struct Color { + init(hue: Hue, variant: ColorVariant) +} + +// Not ideal; leading underscore usually means "don't use this" +enum ColorVariant { + case _50 + case _100 + case _200 + // ... +} +let color = Color(hue: .red, variant: ._100) + +// Repetitive +enum ColorVariant { + case variant50 + case variant100 + case variant200 + // ... +} +let color = Color(hue: .red, variant: .variant100) + +// "v" is for... version? No, that's not right. +enum ColorVariant { + case v50 + // ... +} +``` + +### Code generators and FFI + +Generating code from an external source like a data exchange format specification or transpiling an API from another language is common in programming. When generating such code, names in the original data source may not be usable as Swift identifiers. + +Another example is generating type-safe accessors for resources bundled in a library on the filesystem or in an asset catalog. The names of such resources may not necessarily obey Swift conventions—an example is Apple's own _SF Symbols_, which has images with names like `1.circle`. + +In these situations, the generator needs to implement a fixed set of transformation rules to convert names that aren't valid identifiers into names that are. One problem with this approach is that such conversions have a cascading effect: the result of converting a Swift-incompatible name into a Swift-compatible one might produce a name that could also be a plausible name in the original source. Therefore, code generators must complicate their logic to avoid such collisions so that an unexpected input doesn't produce invalid code and block the user. This collision-breaking logic often results in arbitrary and difficult-to-understand rules that the user must nevertheless internalize in order to know what to write in the destination language when faced with one of these names in the origin language. + +### Module naming at scale + +Swift modules (and Clang modules with which Swift interoperates) occupy a single, flat[^1] namespace. The prevailing code organization pattern among many Swift projects is to treat a module as a semantic unit and to give them short, simple names. This approach has already been proven to cause difficulties in real-world builds, as evidenced by the need for the [module aliasing](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0339-module-aliasing-for-disambiguation.md) feature added in Swift 5.7 to reduce the problems that arise when modules with conflicting names occur in the same dependency graph. + +[^1]: Clang supports submodules, but they are still _contained_ by their parent modules and importing a parent module also imports all of its (implicit) submodules. This is distinct from having independent compilation units that share a hierarchical namespace. Java packages are an example of the latter. + +Experience has shown that this model does not scale well to massively large Swift codebases that use different organizational and build models; for example, companies that use monorepos and build with distributed build systems like Bazel, which we will elaborate on below. + +First, many of the projects now using Swift historically used Objective-C or are using common components that are written in Objective-C. That Objective-C code was not written with modules in mind and uses header-file-based inclusions. In order to interoperate with Swift, all of that code would need to be grouped into modules. This would be a significant undertaking to do manually. + +Additionally, requiring human developers to choose their own module names has high cognitive overhead and easily leads to conflicts. A component in one part of the codebase can add a new dependency that breaks a different project elsewhere simply by having a conflicting module name with something else the project is already using. Module aliasing has originally designed and implemented does not immediately provide a satisfying remedy here, because it still requires that the project _consuming_ the modules opt into aliasing to resolve the conflict, and it forces them to choose a _second, contrived_ name for the module. + +To work around these issues, the approach Bazel has adopted is to automatically derive a module name for a build target based on the unique identifier—a path to its location in the repository—that Bazel already uses to refer to it. This removes the burden from the developer to choose a unique name and reduces the chance of collisions. However, this process still projects a name from a hierarchical namespace onto a flat one, so the build target's natural identifier must be contorted to fit into the identifier-safe naming required by a Swift module. For example, a module defined at the path `myapp/extensions/widget/common/utils` would be given the name `myapp_extensions_widget_common_utils`. + +While this works, it is not ideal. Users often know where the code that they want to import lives before they write the line of code that imports it. In order to write the `import` declaration, they must mentally transform the path to the code into the module name, or infrastructure developers must provide tooling to do so. And like the code generation case discussed above, the transformation is not usually reversible, which makes certain kinds of tooling difficult to write and does not the possibility of collisions. For example, a module named `myapp_extensions_widget_common_utils` could live at `myapp/extensions/widget/common/utils` or at `myapp/extensions/widget/common_utils`. Designing a reversible transform would require making the encoding less friendly to humans who need to read and write those names in their code. + +## Proposed Solution + +We propose extending the backtick-based syntax used to escape keywords as identifiers to support identifiers that can contain characters other than the standard identifier-safe characters allowed by Swift. + +We will distinguish these two uses of backticks with the following terminology: + +* An **escaped identifier** is a sequence of characters surrounded by backticks that starts with `identifier-head` and is followed by zero or more `identifier-character`, as defined in the Swift [grammar](https://docs.swift.org/swift-book/documentation/the-swift-programming-language/summaryofthegrammar/). This is what Swift supports today to treat keywords as identifiers. +* A **raw identifier** is a sequence of characters surrounded by backticks that contains characters other than those allowed in an escaped identifier. The exact contents are described in [Permitted Characters](#permitted-characters) below. + +In both cases, the backticks are **not** considered part of the identifier; they only delimit the identifier from surrounding tokens. + +Raw identifiers would provide more flexibility in naming for use cases like the ones discussed earlier in the document as we explain below. + +### Describing tests + +Test functions could use raw identifiers to describe their purpose clearly without redundancy: + +```swift +@Test func `square returns x * x`() { + #expect(square(4) == 4 * 4) +} +``` + +Here, the user need only provide a **single** description of the test function. That one name will be used anywhere that the test is referenced: test result reports, the debugger, index data, crash logs, and so forth. + +A key observation here is that when using swift-testing, or another framework like XCTest, the name of the test function serves **only to describe the test.** It is _not_ real API, and its only caller is the test framework itself that discovers the function through some dynamic or generative mechanism. Since the name will only be written once at the declaration site, it makes sense to allow for more flexible and verbose naming for the sake of expressibility and simplicity. + +Using raw identifiers for test naming fits very well with Swift's philosophy of progressive disclosure. Rather than using a bespoke API for descriptive naming, the author's journey follows a path through the following stages: + +* You learn how to write a regular Swift function. +* You learn how to make it a unit test by prepending `@Test` to it. +* You learn that raw identifiers exist and how to apply them to the names of your tests, and this knowledge applies for any identifiers in the language rather than just a specific framework. + +### Other kinds of identifiers + +Raw identifiers would provide a clean solution to the cases where the most natural name for an identifier is numeric. Considering the color design system again, we could write: + +```swift +enum ColorVariant { + case `50` + case `100` + case `200` +} +let color = Color(hue: .red, variant: .`100`) +``` + +One could claim that the backticks are just as much ceremony as using a different prefix, but we believe that raw identifiers are a better choice because they would be using **a common notation** that applies to **all** such symbols. This _reduces_ complexity across all Swift codebases, rather than requiring each developer/API to come up with their own conventions as they do today, which increases complexity for readers. + +Raw identifiers could likewise be used for other kinds of foreign identifiers introduced by FFI or code generation. For example, a tool that generates strongly-typed accessors for resources could use raw identifiers to produce a more direct mapping from the resource name to the name in code, reducing complexity by making collisions less likely. + +```swift +extension UIImage { + static var `10.circle`: UIImage { ... } +} +``` + +Raw identifiers also alleviate the naming problems for modules in very large codebases. Instead of deriving a name using a non-reversible transformation, the build system could simply name the module with the unique identifier that it already associates with that module: + +```swift +import `myapp/extensions/widget/common/utils` + +// if explicit disambiguation is needed: +`myapp/extensions/widget/common/utils`.SomeClass +``` + +Doing so greatly improves the experience of developers working in large codebases who can more easily map imports to where the code resides and vice versa, and it also trivializes writing automated tooling that manages build dependencies by scanning imports written in code and updating the build system's definition of the targets. + +### Support in other languages + +Several other modern programming languages acknowledge the occasional but important need that exists for identifiers that do not meet their standard requirements and support raw identifiers like the ones being proposed here. + +In [F#](https://fsharp.org/specs/language-spec/4.1/FSharpSpec-4.1-latest.pdf), identifiers that are surrounded by double backticks and can contain any characters excluding newlines, tabs, and double backticks. + +In [Groovy](https://groovy-lang.org/syntax.html#_quoted_identifiers), identifiers after the dot in a dot-expression can be written as quoted string literals containing characters other than those allowed in standard identifiers. + +In [Kotlin](https://kotlinlang.org/docs/reference/grammar.html#Identifier), identifiers can be surrounded by single backticks. The characters that are allowed inside backticks may differ based on the target backend (for example, the JVM places additional restrictions). The closest comparison to Swift would be Kotlin/Native, which permits any character other than carriage return, newline, or backtick. + +In [Scala](https://www.scala-lang.org/files/archive/spec/2.11/01-lexical-syntax.html), an identifier can contain an arbitrary string when surrounded by single backticks, but host systems may impose some restrictions on which strings are legal for identifiers. + +In [Zig](https://ziglang.org/documentation/master/#Identifiers), an identifier that does not meet the standard requirements may be expressed by using an `@` symbol followed by a string literal, such as `@"with a space"`. + +## Detailed Design + +### Permitted characters + +A raw identifier may contain any valid Unicode characters except for the following: + +* The backtick (`` ` ``) itself, which terminates the identifier. +* The backslash (`\`), which is reserved for potential future escape sequences. +* Carriage return (`U+000D`) or newline (`U+000A`); identifiers must be written on a single line. +* The NUL character (`U+0000`), which already emits a warning if present in Swift source but would be disallowed completely in a raw identifier. +* All other non-printable ASCII code units that are also forbidden in single-line Swift string literals (`U+0001...U+001F`, `U+007F`). + +In addition to these rules, some specific combinations that require special handling are discussed below. + +#### Whitespace + +A raw identifier may have leading, trailing, or internal whitespace; however, it may not consist of *only* whitespace. "Whitespace" is defined here to mean characters satisfying the Unicode `White_Space` property, exposed in Swift by `Unicode.Scalar.Properties.isWhitespace`. + +#### Operator characters + +A raw identifier may start with, contain, or end with operator characters, but it may not contain **only** operator characters. To avoid confusion, a raw identifier containing only operator characters is treated as a parsing error: it is neither a valid identifier nor an operator: + +```swift +func + (lhs: Int, rhs: Int) -> Int // ok +func `+` (lhs: Int, rhs: Int) -> Int // error + +let x = 1 + 2 // ok +let x = 1 `+` 2 // error +``` + +This leaves the door open for a future use of that syntax, should it be desired. There is more discussion of this in [Alternatives Considered](#alternatives-considered). + +### Symbol generation and mangling + +Backticks are required to delimit a raw identifier written in source code, but they are **not part** of the identifier as it relates to symbols generated by the compiler. For example, ``func `with a space`()`` defines a function whose name is `with a space` without backticks. This is the same as escaped identifiers today: ``func `for`()`` defines a function named `for`, not `` `for` ``. + +Fortunately, Swift's symbol mangler already handles non-ASCII-identifier code units in symbol names today by converting them using Punycode. The current algorithm already works for raw identifiers, with one change needed: it does not recognize that an identifier starting with a digit needs to be encoded. Consider the following example: + +```swift +// Module "test" +public struct FontWeight { + public func `100`() { ... } +} +``` + +Without Punycoding an identifier starting with a digit, the current implementation would mangle the function above as `$s4test10FontWeightV3X00yyF`, which round-trips incorrectly as `test.FontWeight.X00() -> ()`. This proposal would implement the necessary change so that it produces the correct mangling `$s4test10FontWeightV004_100_yyF`. + +### Property wrappers + +When a property wrapper wraps a variable named with a raw identifier, backticks must also be used to refer to its backing storage or its projected value. The prefix sigil (`_` or `$`, respectively) is part of those identifiers, so it is placed _inside_ the backticks. + +```swift +@propertyWrapper +struct Wrapper { + var wrappedValue: T + var projectedValue: Wrapper +} + +struct UsesWrapper { + @Wrapper var `with a space`: Int +} + +let x = UsesWrapper() +print(x.`_with a space`) // correct +doSomethingWith(x.`$with a space`) // correct + +print(x._`with a space`) // error +doSomethingWith(x.$`with a space`) // error +``` + +The dollar sign (`$`) has two special meanings in Swift when used at the beginning of an identifier: + +* When followed by an integer, it references an unnamed closure argument at that index (e.g., `$0`). +* When followed by an identifier, it references the projected value of a property wrapper (e.g., `$someBinding`). + +We must then decide what an identifier like `` `$0` `` means. The only correct choice is to treat `` `$0` `` as a regular identifier, not a closure argument. This is necessary to allow property wrapper projections to be accessed on properties whose names are numeric raw identifiers: + +```swift +@propertyWrapper +struct Wrapper { + var wrappedValue: T + var projectedValue: Wrapper +} + +let `$1` = "hello" // error: cannot declare entity named '$1'; the '$' prefix + // is reserved for implicitly-synthesized declarations + +struct UsesWrapper { + @Wrapper var `0`: Int + + func f() { + let closure: (Int) -> Int { + doSomethingWith(`$0`) // ok, refers to projected value of `0` + return $0 // ok, refers to unnamed closure argument + } + } +} +``` + +### Member access expressions + +When escaping a keyword as an identifer, Swift allows the backticks to be omitted in certain contexts when the compiler knows that it cannot be a keyword. Most commonly, this occurs with member access expressions: + +```swift +enum Access { + case `public` // must be escaped here + case `private` +} + +_ = Access.`public` // ok, but not necessary +_ = Access.public // also ok +``` + +Raw identifiers, on the other hand, **must be escaped by backticks in all contexts** since they can contain characters that could be treated as parsing delimiters: + +```swift +struct S { + var `with a space` = 0 +} + +S().with a space // error, the parser can't know where the member + // name is supposed to end +S().`with a space` // correct +``` + +This rule also guarantees an important invariant for tuples: raw identifiers are never confusable with tuple element indices. If a tuple member is accessed with a numeric raw identifier (i.e., `` tuple.`0` ``), that will only interpreted as a tuple element _label_ and never a tuple element _index_. Similarly, if a tuple member is accessed with an unescaped integer (i.e., `tuple.0`), the behavior has not changed: it is only interpreted as an _index_ and never as a _label_, even if there is a label with the same name. For example, + +```swift +let x = (0, 1) +_ = x.`0` // error + +let a = (5, `0`: 10) +let b = y.0 // z <- 5 +let c = y.`0` // z <- 10 +``` + +### Objective-C Compatibility + +A Swift declaration named with a raw identifier must be given an explicit Objective-C name to export it to Objective-C. The compiler will emit an error if a declaration is given an explicit name with the `@objc(...)` attribute and that name is not a valid Objective-C identifier, or if a declaration is otherwise inferred to be `@objc` and the Swift name of that declaration is not a valid Objective-C identifier. + +```swift +@objc class `Class with a Space` { // error + @objc(someFunction) func `some function`() {} // ok + @objc(`not valid`) func myFunction() {} // error +} + +@objc @objcMembers class AnotherClass { + var `some property`: Int // error +} +``` + +The Objective-C runtime can dynamically support class names and selectors that contain characters that are not expressible in source code, but we do not consider that to be something should be supported. Therefore, such names are forbidden even for symbols that would only be exposed to the runtime but not written to the generated header file. + +### Module names + +The Clang module map parser already supports modules with non-identifier characters in their names by writing their names in double quotes: + +``` +module "some/module/name" { + header "..." +} +``` + +Such a module would already import cleanly into Swift simply by allowing the module name in an `import` statement to be a raw identifier: + +```swift +import `some/module/name` +``` + +Swift modules pose a different challenge. The compiler's serialization and import search path logic assumes that compiled module artifacts have filenames that match the name of the module; for example, `SomeModule` would be named `SomeModule.swiftmodule`. Using raw identifiers as module names would limit us to only those characters supported in filenames, and these character sets differ between platforms. + +There is a feature in the compiler called an "explicit Swift module map" that uses a JSON file to explicitly list all dependencies without using search paths. This can be used to give a module a different name than its filename. However, it does not seem appropriate to restrict the use of raw identifier module names only to users of this feature; a solution should also include users of standard module resolution. + +This proposal suggests an alternative: Continue to require that the `-module-name` passed during compilation be a filesystem-compatible name, as is the case today, and then allow aliases set by `-module-alias` to be raw identifiers. This effectively separates the "physical" name of the module (its filesystem name and its ABI name) from what users write in source code. Build systems using this feature would generate the physical names behind the scenes for users and provide the aliases to the compiler, completely transparent to the user. This approach elegantly builds on top of existing features in the compiler and avoids adding more complexity to serialization. + +Finally, users who want the ABI name (the name that appears in symbol manglings) to match exactly what users are typing in source code—for example, to ensure consistency between the module names in source and what appears in the debugger and crash logs—could go one step further and pass the raw identifier using the `-module-abi-name` flag. + +### Impacts on tooling + +During the review of SE-0275, concerns were raised about whether identifiers containing whitespace (or other traditional delimiter characters) would make language tooling harder to write or use. While we cannot address all possible editors here, we believe that modern LSP-driven editors can handle raw identifiers gracefully. One specific concern raised was that double-clicking a word in a raw identifier would not be able to select the whole identifier. Since then, the Language Server Protocol has defined a `textDocument/selectionRange` request that can be used to determine an "interesting" selection range for a text position in a document. If this request is made for the text position that is inside a raw identifier, Swift's language server could return the range of that identifier in the response, allowing the host editor to select it in its entirety. + +Likewise, syntax highlighting for raw identifiers is straightforward; they are no more complicated than a single-line string literal. + +### Impacts on future reflective APIs + +During the review of SE-0275, the point was raised that allowing a name to contain common type delimiters could create ambiguity for future APIs that look up symbols via runtime reflection: + +```swift +struct X { + struct Y {} // 1 +} +struct `X.Y` {} // 2 + +// Does this return 1 or 2? +_ = hypotheticalTypeByName("X.Y") +``` + +While these APIs do not yet exist, we feel that they could be implemented unambiguously. Without designing such an API (it is certainly out of scope for this proposal), we can identify some basic principles to address that concern. + +First, we can imagine that such an API would—at least at a lower level—allow users to drill down through the module context and its descendant contexts explicitly; for example, something in the style of `myModule.type("X", genericArgs: [swiftStdlibModule.type("Int")]).type("Y")` would be clearly distinct from `myModule.type("X.Y")`. + +Then, if a simplified API like `hypotheticalTypeByName` is desired, we can observe that the type name is being passed in a form such that the library would need the capability to parse a subset of the language's type grammar in order to understand which parts are generic arguments, which are member references, and so forth. Therefore, it stands to reason that the argument to `hypotheticalTypeByName` should be **written as it would be in source**, including any raw identifier delimiters. In doing so, each case is easily distinguished: + +```swift +_ = hypotheticalTypeByName("X.Y") // returns 1 above +_ = hypotheticalTypeByName("`X.Y`") // returns 2 above +``` + +## Alternatives Considered + +[SE-0275](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0275-allow-more-characters-like-whitespaces-and-punctuations-for-escaped-identifiers.md) originally proposed using backticks to support qualified references to operators, such as the following: + +```swift +let add: (Int, Int) -> Int = Int.+ // not allowed today +let add: (Int, Int) -> Int = Int.`+` // would have been allowed by SE-0275 +``` + +As mentioned above, this proposal does not allow this; it is an error to write a raw identifier containing **only** identifier characters. We agree with the Core Team's original feedback that since backticks remove the "magic" around a special character sequence, there is potential for confusion about whether writing `` `+` `` refers to the operator or to a different regular identifier named `+`. If we intended the former, there are also open questions about how one would distinguish prefix, postfix, infix operators with such a syntax. This feature, if desired, demands its own in-depth design and separate proposal. + +## Future Directions + +There are natural extensions of the raw identifier syntax that could be used to support multi-line identifiers or identifiers containing backticks by adopting a similar syntax to that of raw string literals: + +~~~swift +let x = #`contains`some`backticks`# +let y = ##`contains`#single`#pound`#delimiters`## + +func ``` + This is a function that multiplies two numbers. + It's named this way to make sure you REALLY know + what you're getting into when you multiply two + numbers. + ```(_ x: Int, _ y: Int) { x * y } + +let fifteen = ``` + This is a function that multiplies two numbers. + It's named this way to make sure you REALLY know + what you're getting into when you multiply two + numbers. + ```(3, 5) +~~~ + +At this time, however, we do not believe there are any compelling use cases for such identifiers. + +### Escape sequences inside raw identifiers + +Raw identifiers follow similar parsing rules as string literals with respect to unprintable characters, which raises the question of how to handle backslashes. The use cases served by many backslash escapes—such as writing unprintable characters—are not desirable for identifiers, so we could choose to treat backslashes as regular literal characters. For example, `` `hello\now` `` would mean the identifier `hello\now`. This could be confusing for users though, who might expect the `\n` to be interpreted the same way that it would be in a string literal. Treating backslashes as literal characters today would also close the door on a viable method of escaping characters inside raw identifiers if we decide that it is needed later. For these reasons, we currently forbid backslashes and leave their purpose to be defined in the future. + +## Source compatibility + +This proposal is purely additive; it does not affect compatibility with existing source code. + +## ABI compatibility + +This proposal has no effect on existing ABI. It only makes new valid Swift symbol manglings for symbols that were previously invalid. + +## Implications on adoption + +For codebases built entirely from source using the same version of the Swift compiler would see no negative impact from using this feature. + +For example, if a resilient library used raw identifiers in any declarations that are serialized into the textual interface (e.g., `public` or `@usableFromInline`), that interface would not be consumable by versions of the compiler prior to when this feature was introduced because the older compilers would not be able to parse the identifiers. This, however, is the case for any feature that introduces a new syntax into the language. diff --git a/proposals/0452-integer-generic-parameters.md b/proposals/0452-integer-generic-parameters.md new file mode 100644 index 0000000000..b2ca7b2083 --- /dev/null +++ b/proposals/0452-integer-generic-parameters.md @@ -0,0 +1,625 @@ +# Integer Generic Parameters + +* Proposal: [SE-0452](0452-integer-generic-parameters.md) +* Authors: [Alejandro Alonso](https://github.com/Azoy), [Joe Groff](https://github.com/jckarter) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#75518](https://github.com/swiftlang/swift/pull/75518), [swiftlang/swift#78248](https://github.com/swiftlang/swift/pull/78248) +* Review: ([pitch](https://forums.swift.org/t/integer-generic-parameters/74181)) ([first review](https://forums.swift.org/t/se-0452-integer-generic-parameters/75844)) ([second review](https://forums.swift.org/t/second-review-se-0452-integer-generic-parameters/77043)) ([acceptance](https://forums.swift.org/t/accepted-se-0452-integer-generic-parameters/77507)) + +## Introduction + +In this proposal, we introduce the ability to parameterize generic types +on literal integer parameters. + +## Motivation + +Swift does not currently support fixed-size or fixed-capacity collections +with inline storage. (Or at least, it doesn't do so *well*, not without +forming a struct with some specific number of elements and doing horrible +things with `withUnsafePointer` to handle indexing.) Most of the implementation +of something like a fixed-size array, or a fixed-capacity growable array with +a maximum size, or a hash table with a fixed number of buckets, is agnostic +to any specific size or capacity, so that implementation +would ideally be generic over size so that a library implementation can +be reused for any given size. + +Beyond inline storage sizes, there are other use cases for carrying integers +in type information, such as to represent an operation with a particular +input or output size. Carrying this information in types can allow for APIs +with stronger static guarantees that chains of operations match in the +number of elements they consume or produce. + +## Proposed solution + +Generic types can now be parameterized by integer parameters, declared using +the syntax `let : Int` inside of the generic parameter angle brackets: + +```swift +struct Vector { + /*implementation TBD*/ +} +``` + +A generic type with integer parameters can be instantiated using literal +integer arguments: + +```swift +struct Matrix4x4 { + var matrix: Vector<4, Vector<4, Double>> +} +``` + +Or it can be instantiated using integer generic parameters from the surrounding +generic environment: + +```swift +struct Matrix { + var matrix: Vector> +} +``` + +Integer generic parameters become static constant members of the type, with +the same visibility as the type itself: + +```swift +public struct Matrix { + // implicitly has these members: + // public static var columns: Int { get } + // public static var rows: Int { get } +} + +// From another module: + +import struct Matrices.Matrix + +print(Matrix<4, 3>.columns) // prints 4 +print(Matrix<4, 3>.rows) // prints 3 +``` + +Generic functions and methods can also be parameterized by integer generic +parameters. As with other generic parameters, the values of the generic +arguments for a call are inferred from the types of the argument values +provided to the call: + +```swift +func matmul( + _ l: Matrix, + _ r: Matrix +) -> Matrix { ... } + +let m1 = Matrix<4, 2>(...) +let m2 = Matrix<2, 5>(...) + +let m3 = matmul(m1, m2) // a = 4, b = 2, c = 5, result type is Matrix<4, 5> +``` + +Within an expression, a reference to an integer generic parameter evaluates +the parameter as a value of type `Int`: + +```swift +extension Vector { + subscript(i: Int) -> Element { + get { + if i < 0 || i >= count { + fatalError("index \(i) out of bounds [0, \(count))") + } + return element(i) + } + } +} +``` + +## Detailed design + +The grammar for generic parameter lists expands to include value generic +parameters: + +```swift +generic-parameter --> 'let' type-name ':' type +``` + +Correspondingly, signed integer literals can now appear as elements in +generic argument lists and as operands of generic requirements: + +```swift +generic-argument --> '-'? integer-literal +same-type-requirement --> type-identifier '==' '-'? integer-literal +``` + +Although they can appear as elements in generic parameter lists, integer +literals are still not allowed to appear as types in and of themselves, and +cannot be used as bindings for type generic parameters. + +```swift +let x: 2 // error, 2 is not a type +let y: Array<2> // error, Array's Element is a type generic parameter +``` + +Likewise, integer generic parameters cannot be used as standalone types in their +generic context. + +```swift +struct Foo { + let y: x // Error, x is not a type + let metax: x.Type // Error, x has no member `.Type` +} +``` + +The type referenced by a value generic parameter declaration must resolve to +the `Swift.Int` standard library type. (Allowing other types of value generic +parameter is a future direction.) + +```swift +struct Foo { } // OK (assuming no shadowing `Int` declaration) +struct Foo2 { } // also OK + +struct BadFoo { } // Error, generic parameters of type Float not supported + +typealias MyInt = Swift.Int +struct Bar { } // OK + +struct Baz: P { + typealias A = Int +} + +struct Zim { } // OK + +func contrived() { + struct Int { } + + struct BadFoo { } // Error, local Int not supported + + struct Foo { } // OK +} +``` + +Integer generic parameters of types become static members of that type, +with the same visibility as the type itself. It is an error to try to +declare another static property with the same name as an integer generic +parameter within the type declaration, just as it would if the property +were independently declared: + +```swift +struct Vec { + static let count: Int // error: Vec already has a static property `count` +} +``` + +In a type reference, an integer generic argument can be provided as either +a literal integer, or as a reference to an integer generic parameter from +the enclosing generic context. References to type generic parameters, +type generic parameter packs, or declarations other than integer generic +parameters is an error. (Allowing references to constants of integer type, +or more elaborate constant expressions, as generic parameters is a future +direction.) + +```swift +struct IntParam { } + +let a: IntParam<2> // OK +let b: IntParam<-2> // OK + +struct AlsoIntParam { + let c: IntParam // OK + + static let someIntegerConstant = 42 + let d: IntParam // Error, not an Int generic parameter + + let e: IntParam // Error, is a type generic parameter + let f: IntParam // Error, is a pack generic parameter +} +``` + +Conversely, using an integer generic parameter as an argument for a type +generic parameter is also an error. + +```swift +struct IntAndTypeParam { + let xs: Array // Error, x is an integer type parameter +} +``` + +An integer generic parameter can be constrained to be equal to a specific +literal value using a same-value constraint, spelled with `==` as for a +same-type constraint. Two integer generic parameters can also be constrained +to be equal to each other. + +```swift +struct TwoIntParams {} + +extension TwoIntParams where n == 2 { + func foo() { ... } +} + +extension TwoIntParams where n == m { + func bar() { ... } +} + +let x: TwoIntParams +x.foo() // OK +x.bar() // Error, doesn't match constraint + +let y: TwoIntParams +y.foo() // Error, doesn't match constraint +y.bar() // OK +``` + +Integer generic parameters cannot be constrained to be equal to type generic +parameters, concrete types, or to declarations other than generic parameters. +Integer generic parameters also cannot be constrained to conform to protocols. + +```swift +extension TwoIntParams where n == T {} // error +extension TwoIntParams where T == n {} // error +extension TwoIntParams where n == Int {} // error + +let globalConstant = 42 +extension TwoIntParams where n == globalConstant {} // error + +extension TwoIntParams where n: Collection // error +``` + +(In the same way overload resolution already works in Swift, extensions or +functions with generic constraints on integer parameters will only be chosen +for call sites at which those constraints always hold; we won't "dispatch" +based on the value of an argument from a less-constrained call site.) + +```swift +struct Foo { + func foo() { print("foo #1") } + + func bar() { + // Always prints "foo #1" + self.foo() + } +} + +extension Foo where n == 2 { + func foo() { print("foo #2") } +} + +Foo<2>().bar() // prints "foo #1" +Foo<2>().foo() // prints "foo #2" +``` + +## Source compatibility + +This proposal is a strict extension of the existing language. The `let n: Type` +syntax should ensure source compatibility if we expand the feature to allow +value generic parameters of other types in the future. + +## ABI compatibility + +This proposal does not affect the ABI of existing code. Handling integer +generic parameters in full generality requires new functionality in the +Swift runtime to be able to encode and interpret them as part of type +metadata. + +As with generic parameters in general, adding or removing +integer generic parameters, replacing value parameters of a function with +integer generic parameters, reordering an integer generic parameter relative to +other generic parameters (whether value or type), and adding or removing +same-value constraints are all ABI-breaking changes. + +## Implications on adoption + +### Back-deployment limitations + +On platforms where the vendor ships the Swift runtime with the operating +system, there may be limitations on using integer generic parameters in +programs that want to target earlier versions of those platforms that don't +have the necessary runtime support. + +### Naming conventions + +Integer generic parameters are a new kind of declaration in Swift, and +conventions need to be established as to how they should be named. This +proposal recommends that integer generic parameters follow the convention +of other value bindings and be named using `lowerCamelCase` identifiers. + +## Future directions + +This proposal aims to establish the core functionality of integer generic +parameters. There are many possible improvements that could be built upon +this base: + +### Fixed-size and fixed-capacity collection types + +This proposal provides a foundational mechanism for fixed-size array and +fixed-capacity collection types, but does not itself introduce any +new standard library types or mechanisms for defining those types. We leave +it to future proposals to explore the design of those types. + +### Use of constant bindings as generic parameters + +It would be very useful to be able to use constant bindings as generic +parameter bindings, in addition to literals and existing generic parameter +bindings: + +```swift +static let bufferSize + = MemoryLayout.size * 64 + MemoryLayout.size * 8 + +var buffer = Vector(...) +``` + +This should be possible as long as the bindings referenced are known to be +constant (like `let` bindings are). + +A likely-fundamental limitation to this feature, as well as related +constant evaluation features explored below, is that the type checker will +likely be unable to reason about the value of these bindings. +Type checking influences overload resolution and the overall meaning of +expressions, so cannot rely on the evaluation of those expressions without +creating circularities. We may be able understand that two terms spelled +exactly the same way are equivalent, but we wouldn't recognize two different +expressions with the same result are the same type statically: + +``` +let fourShorts = 4 * MemoryLayout.size +let eightBytes = 8 * MemoryLayout.size + +var v1: Vector = [...] +var v2: Vector = [...] +v1 = v2 // Error, different types +``` + +This is similar to how opaque result types for different declarations are +type-checked as if they are potentially different types even when their +underlying types dynamically resolve to the same type. + +### Arithmetic in generic parameters + +There are many operations that would benefit from being able to express basic +arithmetic relationships among values. For instance, the concatenation of two +fixed-sized arrays would give an array whose length is the sum of the input +lengths: + +```swift +func concat( + _ a: Vector, _ b: Vector +) -> Vector +``` + +Due to the bidirectional nature of Swift's type-checking, there would be +limits to the sorts of relations we would be able to express this way. + +### Relating integer generic parameters and variadic pack shapes + +The "shape" of a parameter pack ultimately compiles down to its length. +Variadic packs don't currently have a way to directly reference or constrain +their shape or length, and integer generic parameters might be one way of doing +so. Among other things, this might allow for a variadic API to express that it +takes as many arguments as one of its integer generic parameters indicates: + +```swift +struct Vector { + // the initializer for a Vector takes one argument + // for every element + init(_ values: repeat each n * T) +} +``` + +### Non-integer value generic parameters + +We may want to eventually allow generic declarations to have value parameters +of type other than `Int`. The proposal's `let Parameter: Type` declaration +syntax maintains space for this: + +```swift +struct MatrixShape { var rows: Int, columns: Int } + +struct Matrix { + var elements: Vector> +} +``` + +Although the syntactic extension is straightforward, there are a lot of +questions to answer about how type equality is determined when values of +arbitrary type are involved, and what sorts of construction and destructuring +operations can be supported at type level. There is some precedent in +other languages to look at here, particularly C++'s non-type template +parameters or Rust's similar const generics feature. However, in relation to +those other languages, Swift puts a bit stronger emphasis on being able to +abstract the layout of types, but the type-level equality of parameters would +be heavily dependent on their types' layout and how initialization and property +access works. + +### Integer parameter packs + +There are use cases for variadic packs of integer generic parameters. +For instance, it might be a way of representing arbitrary multidimensional +matrices of values: + +```swift +struct MDMatrix { ... } + +let mat2d: MDMatrix<4, 4> = ... +let mat4d: MDMatrix<120, 24, 6, 2> = ... +``` + +## Alternatives considered + +### Variable-sized types instead of integer generic parameters + +One of the primary motivators for integer generic parameters is to represent +fixed-size and fixed-capacity collections. One of the reasons this is necessary +is because every value of a Swift type has to have a uniform size; since +a four-element array has a different size from a five-element array, that +implies that they have to be different types `Vector<4, T>` and `Vector<5, T>`. + +However, one could argue that the fundamental type of such a container +doesn't really change with its size; in most cases, a function that can +accept an array of some size can just as well accept an array of any size. +Forcing a type distinction between different-sized arrays forces the majority +of APIs that want to work with arrays to either be generic over their size, +be generic over some more abstract protocol like `Collection` that all +sized arrays conform to (along with unsized `Array` and non-array collections), +or work with the arrays indirectly through some handle type like +`UnsafeBufferPointer` or `Span`. + +So it's interesting to consider an alternative design where we instead +remove the "all values of a type have the same size" constraint. One could +say that the owner of a `Vector` value has to give it some size, but then +a `borrowing` or `inout Vector` can reference a `Vector` of any size, since +the reference representation would carry that size information from the +owner. There are however a lot of open questions following this design +path—if you want to have a two-dimensional `Vector` of `Vector`s, how do you +track the size information of both levels of nesting? There also *are* +functions that want to require taking two input arrays of the same size, +or promise to return an array as the same size as an argument. These +relationships are straightforward to express through the generics system, +and if sizes aren't propagated through types but some other means, it seems +likely we would need a parallel mechanism for reasoning about sizes generically. +Variable-sized types are an interesting idea to explore, but it isn't clear +that they lead to an overall simpler language design. + +### Declaring value parameters without `let` + +One could argue that, since `Int` clearly isn't a protocol constraint, that +it should be sufficient to declare integer generic parameters with the +syntax `` without an introducer like `let`. There are at least +a couple of reasons we choose to adopt the `let` introducer: + +- It makes it clear to the reader (and the compiler) what parameters are + value parameters without needing to do name resolution first. This may not + be a huge deal for `Int`, but if we expand the feature to allow other + types of value generic parameters, then it may not be obvious in an + unfamiliar codebase whether `T: Foo` refers to a protocol constraint + `Foo` or a concrete type `Foo`. +- If we do generalize value generic parameters to allow other types in the + future, it's not entirely out of the question that that could include + existential types, which would make `T: P` potentially ambiguous as to + whether it declares a type parameter constrained to `T` or a value + parameter of type `any P`. (There are perhaps other ways of dealing with that + ambiguity, such as requiring the value parameter form to be written + explicitly with `t: any P`.) + +### Arbitrary-precision integer generic parameters + +Instead of treating integer generic parameters as values of `Int` or any +finite type, another possible design would be to treat type-level integers +as independent of any concrete type, leaving them as ideal arbitrary-precision +integers. This would have some semantic advantages if we want to allow for +type-level arithmetic relationships, since these operations could be defined +in their ideal form without having to deal with overflow and other limitations +of concrete Swift types. In such a design, a reference to an integer generic +parameter in a value expression could be treated as polymorphic, in a similar +way to how integer literals can be used with any type that's +`ExpressibleByIntegerLiteral`. + +Although this model has some appeal, it also has some practical issues. If +type-level integers are arbitrary precision, but value-level integer types are +still finite, then there is the chance for overflow any time a type-level +integer is reified to a finite integer type. This model also would not extend +very naturally to non-integer value parameters if we introduce those in the +future. + +### Generic parameters of integer types other than `Int` + +We discuss generalizing value generic parameters to types other than `Int` +as a future direction above, but a narrower expansion might be to allow all +of Swift's primitive integer types, including all of the sized and +signed/unsigned variants, as types for generic value parameters. One could +argue that `UInt` is particular is desirable to use as the type for fixed-size +and fixed-capacity collections, of which instances can never actually be +constructed for negative sizes. + +However, we would like to continue to promote the use of `Int` as the common +currency type for integers, as we have already established for the standard +library. Introducing mixed integer types as type-level generic parameters +would inevitably lead to the need to be able to perform type conversions +at type level, and the associated need to deal with overflow during these +type-level conversions. + +The established API for the `Collection` protocol, `Array`, and the other +standard library types already use `Int` for `count` and array subscripting +operations, so establishing `Int` as the type for type-level size parameters +avoids the need for type conversions when mixing type- and value-level index +and size values. Types that use integer parameters for sizing can still +refuse to initialize values of types with negative parameters, so that a +type like `Vector<-1, Int>` is uninhabited. Given the restrictions in this +initial proposal, without type-level arithmetic, it is unlikely that +developers would intentionally form such a type with a negative size explicitly. + +### Alternative naming conventions + +We propose recommending a `lowerCamelCase` naming convention for integer +generic parameters, following the recommended convention for other value +declarations such as property and function declarations. This allows for +integer generic parameters to appear consistent with other value references +in expressions. If we add the ability to instantiate generics using value +constants or expressions as integer generic arguments in the future, this will +also maintain consistency between existing generic parameters and other value +declarations used as arguments: + +``` +struct Foo { + static let nSquared = n * n + + // These both consistently use lowerCamelCase + var vector: Vector + var matrix: Vector +} +``` + +Alternatively, we could recommend that integer generic parameters use +`UpperCamelCase`, following the convention of type generic parameters. We +believe that the distinction between values and types is better to prioritize +than the distinction between type-level and value-level parameters. + +### Syntactic separation of value and type parameters + +The angle brackets that enclose generic arguments have to be syntactically +disambiguated from the `<` and `>` operators. This disambiguation relies on +parsing ahead to determine whether the source code following a `<` is parsable +as a list of types followed by a closing `>` and a member access or initializer +call. This lookahead rule has worked well up to this point, but it could impose +constraints on our ability to allow for expressions to be used as generic +arguments, since allowing more expression productions to appear in generic +argument lists will lead to more potentially ambiguous parsing situations. + +It may be worth considering a design that separates value generic parameters +by putting them in a different set of brackets separate from the type +generic parameters, like `Vector[count]`. This would avoid the need +for disambiguation if an arbitrary expression can be used as a `count` in +the future. + +### Behavior of static properties corresponding to generic parameters + +In response to the first round of review, we added static properties +corresponding to the integer generic parameters of types, based on feedback +that, in many if not most cases, the values of the generic parameters would end +up being redeclared as static parameters, and the existence and names of the +generic parameters are already essentially part of its public API, since +usage of the type must be able to provide arguments to the parameters, and +extensions can refer to the parameters by their names. + +One good argument against doing this is that we don't already do anything +analogous for type generic parameters, such as presenting them as a typealias +member of the type. We think that it would be beneficial to do so, however, +and in discussion with the LSG, there were attempts to make this happen in +the past, but they ran into source compatibility issues. If there is a chance +to take a source break that enables this functionality for type generic +parameters, the LSG thinks it is worth considering. With integer generic +parameters, we have the opportunity to do the right thing out of the gate, +and we would have similar source-breaking considerations to do it later, so +we believe it is better to do the right thing up front. + +## Acknowledgments + +We would like to thank the following people for prototyping and design +contributions that helped shape this proposal: + +- Holly Borla +- Ben Cohen +- Erik Eckstein +- Doug Gregor +- Tim Kientzle +- Karoy Lorentey +- John McCall +- Kuba Mracek +- Slava Pestov +- Andrew Trick +- Pavel Yaskevich diff --git a/proposals/0453-vector.md b/proposals/0453-vector.md new file mode 100644 index 0000000000..d755bf66f1 --- /dev/null +++ b/proposals/0453-vector.md @@ -0,0 +1,774 @@ +# InlineArray, a fixed-size array + +* Proposal: [SE-0453](0453-vector.md) +* Authors: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [Approaches for fixed-size arrays](https://forums.swift.org/t/approaches-for-fixed-size-arrays/58894) +* Implementation: [swiftlang/swift#76438](https://github.com/swiftlang/swift/pull/76438) +* Review: ([pitch](https://forums.swift.org/t/vector-a-fixed-size-array/75264)) ([first review](https://forums.swift.org/t/se-0453-vector-a-fixed-size-array/76004)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0453-vector-a-fixed-size-array/76411)) ([second review](https://forums.swift.org/t/second-review-se-0453-vector-a-fixed-size-array/76412)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0453-inlinearray-formerly-vector-a-fixed-size-array/77678)) + +## Introduction + +This proposal introduces a new type to the standard library, `InlineArray`, which is +a fixed-size array. This is analogous to the +[classical C arrays `T[N]`](https://en.cppreference.com/w/c/language/array), +[C++'s `std::array`](https://en.cppreference.com/w/cpp/container/array), +and [Rust's arrays `[T; N]`](https://doc.rust-lang.org/std/primitive.array.html). + +## Motivation + +Arrays in Swift have served as the go to choice when needing to put items in an +ordered list. They are a great data structure ranging from a variety of +different use cases from teaching new developers all the way up to sophisticated +implementation details of something like a cache. + +However, using `Array` all the time doesn't really make sense in some scenarios. +It's important to understand that `Array` is a heap allocated growable data +structure which can be expensive and unnecessary in some situations. The next +best thing is to force a known quantity of elements onto the stack, probably by +using tuples. + +```swift +func complexAlgorithm() { + let elements = (first, second, third, fourth) +} +``` + +Unfortunately, using tuples in this way is very limited. They don't allow for +dynamic indexing or iteration: + +```swift +func complexAlgorithm() { + let elements = (first, second, third, fourth) + + // Have to manually know the tuple has N elements... + for i in 0 ..< 4 { + // error: cannot access element using subscript for tuple type + // '(Int, Int, Int, Int)'; use '.' notation instead + compute(elements[i]) + } +} +``` + +It wasn't until [SE-0322 Temporary uninitialized buffers](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0322-temporary-buffers.md), which proposed the `withUnsafeTemporaryAllocation` +facilities, that made this situation a little easier to work with by giving us a +direct `UnsafeMutableBufferPointer` pointing either somewhere on the stack or to +a heap allocation. This API allows us to get the indexing and iteration we want, +but it drops down to an unsafe layer which is unfortunate because there should +be much safer ways to achieve the same while not exposing unsafety to +developers. + +While we aren't getting rid of `Array` anytime soon, more and more folks are +looking towards Swift to build safer and performant code and having `Array` be +our only solution to an ordered list of things is less than ideal. `Array` is a +very general purpose array collection that can suit almost any need, but it is +always heap allocated, automatically resizable, and introduces retain/release +traffic. These implicit allocations are becoming more and more of a bottleneck, +especially in embedded domains where there might not be a lot of memory for many +allocations or even heap allocations at all. Swift should be able to provide +developers a safe API to have an ordered list of homogeneous items on the stack, +allowing for things like indexing, iteration, and many other collection +utilities. + +## Proposed solution + +We introduce a new top level type, `InlineArray`, to the standard library which is a +fixed-size contiguously inline allocated array. We're defining "inline" as using +the most natural allocation pattern depending on the context of where this is +used. It will be stack allocated most of the time, but as a class property +member it will be inline allocated on the heap with the rest of the properties. +`InlineArray` will never introduce an implicit heap allocation just for its storage +alone. + +```swift +func complexAlgorithm() { + // This is a stack allocation, no 'malloc's or reference counting here! + let elements: InlineArray<4, Int> = [1, 2, 3, 4] + + for i in elements.indices { + compute(elements[i]) // OK + } +} +``` + +InlineArrays of noncopyable values will be possible by using any of the closure based +taking initializers or the literal initializer: + +```swift +// [Atomic(0), Atomic(1), Atomic(2), Atomic(3)] +let incrementingAtomics = InlineArray<4, Atomic> { i in + Atomic(i) +} + +// [Sprite(), Sprite(), Sprite(), Sprite()] +// Where the 2nd, 3rd, and 4th elements are all copies of their previous +// element. +let copiedSprites = InlineArray<4, _>(first: Sprite()) { $0.copy() } + +// Inferred to be InlineArray<3, Mutex> +let literalMutexes: InlineArray = [Mutex(0), Mutex(1), Mutex(2)] +``` + +These closure based initializers are not limited to noncopyable values however! + +## Detailed design + +`InlineArray` will be a simple noncopyable struct capable of storing other potentially +noncopyable elements. It will be conditionally copyable only when its elements +are. + +```swift +public struct InlineArray: ~Copyable {} + +extension InlineArray: Copyable where Element: Copyable {} +extension InlineArray: BitwiseCopyable where Element: BitwiseCopyable {} +extension InlineArray: Sendable where Element: Sendable {} +``` + +### MemoryLayout + +The memory layout of a `InlineArray` is defined by taking its `Element`'s stride and +multiplying that by its `count` for its size and stride. Its alignment is equal +to that of its `Element`: + +```swift +MemoryLayout.stride == 1 +MemoryLayout.alignment == 1 + +MemoryLayout>.size == 4 +MemoryLayout>.stride == 4 +MemoryLayout>.alignment == 1 + +struct Uneven { + let x: UInt32 + let y: Bool +} + +MemoryLayout.stride == 8 +MemoryLayout.alignment == 4 + +MemoryLayout>.size == 32 +MemoryLayout>.stride == 32 +MemoryLayout>.alignment == 4 + +struct ACoupleOfUInt8s { + let x: InlineArray<2, UInt8> +} + +MemoryLayout.stride == 2 +MemoryLayout.alignment == 1 + +MemoryLayout>.size == 4 +MemoryLayout>.stride == 4 +MemoryLayout>.alignment == 1 +``` + +### Literal Initialization + +Before discussing any of the API, we need to discuss how the array literal +syntax will be used to initialize a value of `InlineArray`. While naively we could +conform to `ExpressibleByArrayLiteral`, the shape of the initializer always +takes an actual `Array` value. This could be optimized away in the simple cases, +but fundamentally it doesn't make sense to have to do an array allocation to +initialize a stack allocated `InlineArray`. Therefore, the array literal +initialization for `InlineArray` will be a special case, at least to start out with. +A stack allocated InlineArray using a InlineArray literal will do in place initialization +of each element at its stack slot. The two below are roughly equivalent: + +```swift +let numbers: InlineArray<3, Int> = [1, 2, 3] + +// Roughly gets compiled as: + +// This is not a real 'InlineArray' initializer! +let numbers: InlineArray<3, Int> = InlineArray() +numbers[0] = 1 +numbers[1] = 2 +numbers[2] = 3 +``` + +There shouldn't be any intermediary values being copied or moved into the InlineArray. + +Note that the array literal syntax will only create a `InlineArray` value when the +compiler knows concretely that it is a `InlineArray` value. We don't want to break +source whatsoever, so whatever current rules the compiler has will still be +intact. Consider the following uses of the array literal syntax and where each +call site creates either a `Swift.Array` or a `Swift.InlineArray`. + +```swift +let a = [1, 2, 3] // Swift.Array +let b: InlineArray<3, Int> = [1, 2, 3] // Swift.InlineArray + +func generic(_: T) {} + +generic([1, 2, 3]) // passes a Swift.Array +generic([1, 2, 3] as InlineArray<3, Int>) // passes a Swift.InlineArray + +func test(_: T) {} + +test([1, 2, 3]) // passes a Swift.Array +test([1, 2, 3] as InlineArray<3, Int>) // error: 'InlineArray<3, Int>' does not conform to 'ExpressibleByArrayLiteral' + +func array(_: [T]) {} + +array([1, 2, 3]) // passes a Swift.Array +array([1, 2, 3] as InlineArray<3, Int>) // error: 'InlineArray<3, Int>' is not convertible to 'Array' + +func inlineArray(_: InlineArray<3, T>) {} + +inlineArray([1, 2, 3]) // passes a Swift.InlineArray +inlineArray([1, 2, 3] as [Int]) // error: 'Array' is not convertible to 'InlineArray<3, Int>' +``` + +I discuss later about a hypothetical `ExpressibleByInlineArrayLiteral` and the design +challenges there in [Future Directions](#expressiblebyInlineArrayliteral). + +The literal initialization allows for more type inference just like the current +literal syntax does by inferring not only the element type, but also the count +as well: + +```swift +let a: InlineArray<_, Int> = [1, 2, 3] // InlineArray<3, Int> +let b: InlineArray<3, _> = [1, 2, 3] // InlineArray<3, Int> +let c: InlineArray<_, _> = [1, 2, 3] // InlineArray<3, Int> +let d: InlineArray = [1, 2, 3] // InlineArray<3, Int> + +func takesGenericInlineArray(_: InlineArray) {} + +takesGenericInlineArray([1, 2, 3]) // Ok, N is inferred to be '3'. +``` + +A compiler diagnostic will occur if the number of elements within the literal +do not match the desired count (as well as element with the usual diagnostic): + +```swift +// error: expected '2' elements in InlineArray literal, but got '3' +let x: InlineArray<2, Int> = [1, 2, 3] + +func takesInlineArray(_: InlineArray<2, Int>) {} + +// error: expected '2' elements in InlineArray literal, but got '3' +takesInlineArray([1, 2, 3]) +``` + +### Initialization + +In addition to literal initialization, `InlineArray` offers a few others forms of +initialization: + +```swift +extension InlineArray where Element: ~Copyable { + /// Initializes every element in this InlineArray running the given closure value + /// that returns the element to emplace at the given index. + /// + /// This will call the closure `count` times, where `count` is the static + /// count of the InlineArray, to initialize every element by passing the closure + /// the index of the current element being initialized. The closure is allowed + /// to throw an error at any point during initialization at which point the + /// InlineArray will stop initialization, deinitialize every currently initialized + /// element, and throw the given error back out to the caller. + /// + /// - Parameter next: A closure that returns an owned `Element` to emplace at + /// the passed in index. + public init(_ next: (Int) throws(E) -> Element) throws(E) + + /// Initializes every element in this InlineArray by running the closure with the + /// previously initialized element. + /// + /// This will call the closure `count - 1` times, where `count` is the static + /// count of the InlineArray, to initialize every element by passing the closure + /// an immutable borrow reference to the previously initialized element. The + /// closure is allowed to throw an error at any point during initialization at + /// which point the InlineArray will stop initialization, deinitialize every + /// currently initialized element, and throw the given error back out to the + /// caller. + /// + /// - Parameter first: The first value to insert into the InlineArray which will be + /// passed to the closure as a borrow. + /// - Parameter next: A closure that passes in an immutable borrow reference + /// of the previously initialized element of the InlineArray + /// which returns an owned `Element` instance to insert into + /// the InlineArray. + public init( + first: consuming Element, + next: (borrowing Element) throws(E) -> Element + ) throws(E) +} + +extension InlineArray where Element: Copyable { + /// Initializes every element in this InlineArray to a copy of the given value. + /// + /// - Parameter value: The instance to initialize this InlineArray with. + public init(repeating: Element) +} +``` + +### Deinitialization and consumption + +Once a InlineArray is no longer used, the compiler will implicitly destroy its value. +This means that it will do an element by element deinitialization, releasing any +class references or calling any `deinit`s on noncopyable elements. + +### Generalized `Sequence` and `Collection` APIs + +While we aren't conforming `InlineArray` to `Collection` (more information in future +directions), we do want to generalize a lot of APIs that will make this a usable +collection type. + +```swift +extension InlineArray where Element: ~Copyable { + public typealias Element = Element + public typealias Index = Int + + /// Provides the count of the collection statically without an instance. + public static var count: Int { count } + + public var count: Int { count } + public var indices: Range { 0 ..< count } + public var isEmpty: Bool { count == 0 } + public var startIndex: Int { 0 } + public var endIndex: Int { count } + + public borrowing func index(after i: Int) -> Int + public borrowing func index(before i: Int) -> Int + + public mutating func swapAt( + _ i: Int, + _ j: Int + ) + + public subscript(_ index: Int) -> Element + public subscript(unchecked index: Int) -> Element +} +``` + +## Source compatibility + +`InlineArray` is a brand new type in the standard library, so source should still be +compatible. + +Given the name of this type however, we foresee this clashing with existing user +defined types named `InlineArray`. This isn't a particular issue though because the +standard library has special shadowing rules which prefer user defined types by +default. Which means in user code with a custom `InlineArray` type, that type will +always be preferred over the standard library's `Swift.InlineArray`. By always I +truly mean _always_. + +Given the following two scenarios: + +```swift +// MyLib +public struct InlineArray { + +} + +print(InlineArray.self) + +// error: generic type 'InlineArray' specialized with too many type parameters +// (got 2, but expected 1) +print(InlineArray<3, Int>.self) +``` + +Here, we're exercising the fact that this `MyLib.InlineArray` has a different generic +signature than `Swift.InlineArray`, but regardless of that we will prefer `MyLib`'s +version even if we supply more generic arguments than it supports. + +```swift +// MyLib +public struct InlineArray { + +} + +// MyExecutable main.swift +import MyLib + +print(InlineArray.self) // OK + +// error: generic type 'InlineArray' specialized with too many type parameters +// (got 2, but expected 1) +print(InlineArray<3, Int>.self) + +// MyExecutable test.swift + +// error: generic type 'InlineArray' specialized with too few type parameters +// (got 1, but expected 2) +print(InlineArray.self) +``` + +And here, we exercise that a module with its own `InlineArray`, like `MyLib`, will +always prefer its own definition within the module, but even for dependents +who import `MyLib` it will prefer `MyLib.InlineArray`. For files that don't +explicitly `MyLib`, it will prefer `Swift.InlineArray`. + +## ABI compatibility + +`InlineArray` is a brand new type in the standard library, so ABI should still be +compatible. + +## Implications on adoption + +This is a brand new type which means there will be deployment version +requirement to be able to use this type, especially considering it is using new +runtime features from integer generics. + +## Future directions + +### `Equatable`, `Hashable`, `CustomStringConvertible`, and other protocols. + +There are a wide class of protocols that this type has the ability to conform to, +but the issue is that it can only conform to them when the element conforms to +them (this is untrue for `CustomStringConvertible`, but it still requires +copyability). We could introduce these conformances but have them be conditional +right now and generalize it later when we generalize these protocols, but if we +were to ship say Swift X.Y with: + +```swift +@available(SwiftStdlib X.Y) +extension InlineArray: Equatable where Element: Equatable // & Element: Copyable +``` + +and later down the road in Swift X.(Y + 1): + +```swift +@available(SwiftStdlib X.Y) +extension InlineArray: Equatable where Element: ~Copyable & Equatable +``` + +Suddenly, this availability isn't quite right because the conformance that +shipped in Swift X.Y doesn't support noncopyable elements. To prevent the +headache of this and any potential new availability feature, we're holding off on +these conformances until they are fully generalized. + +### `Sequence` and `Collection` + +Similarly, we aren't conforming to `Sequence` or `Collection` either. +While we could conform to these protocols when the element is copyable, `InlineArray` +is unlike `Array` in that there are no copy-on-write semantics; it is eagerly +copied. Conforming to these protocols would potentially open doors to lots of +implicit copies of the underlying InlineArray instance which could be problematic +given the prevalence of generic collection algorithms and slicing behavior. To +avoid this potential performance pitfall, we're explicitly not opting into +conforming this type to `Sequence` or `Collection`. + +We do plan to propose new protocols that look like `Sequence` and `Collection` +that avoid implicit copying making them suitable for types like `InlineArray` and +containers of noncopyable elements. +[SE-0437 Noncopyable Standard Library Primitives](0437-noncopyable-stdlib-primitives.md) +goes into more depth about this rationale and mentions that creating new +protocols to support noncopyable containers with potentially noncopyable +elements are all marked as future work. + +Much of the `Collection` API that we are generalizing here for this type are all +API we feel confident will be included in any future container protocol. Even if +we find that to not be the case, they are still useful API outside of generic +collection contexts in their own right. + +Remember, one can still iterate a `InlineArray` instance with the usual `indices` +property (which is what noncopyable InlineArray instances would have had to deal with +regardless until new container protocols have been proposed): + +```swift +let atomicInts: InlineArray<3, Atomic> = [Atomic(1), Atomic(2), Atomic(3)] + +for i in atomicInts.indices { + print(atomicInts[i].load(ordering: .relaxed)) +} +``` + +### `Span` APIs + +With the recent proposal +[SE-0447 Span: Safe Access to Contiguous Storage](0447-span-access-shared-contiguous-storage.md) +who defines a safe abstraction over viewing contiguous storage, it would make +sense to define API on `InlineArray` to be able to get one of these `Span`s. However, +the proposal states that: + +> We could provide `withSpan()` and `withBytes()` closure-taking functions as +> safe replacements for the existing `withUnsafeBufferPointer()` and +> `withUnsafeBytes()` functions. We could also also provide lifetime-dependent +> `span` or `bytes` properties. +> ... +> Of these, the closure-taking functions can be implemented now, but it is +> unclear whether they are desirable. The lifetime-dependent computed properties +> require lifetime annotations, as initializers do. We are deferring proposing +> these extensions until the lifetime annotations are proposed. + +All of which is exactly true for the current `InlineArray` type. We could propose a +`withSpan` style API now, but it's unclear if that's what we truly want vs. a +computed property that returns the span which requires lifetime annotation +features. For now, we're deferring such API until a lifetime proposal is +proposed and accepted. + +### `ExpressibleByInlineArrayLiteral` + +While the proposal does propose a literal initialization for `InlineArray` that +doesn't use `ExpressibleByArrayLiteral`, we are intentionally not exposing some +`ExpressibleByInlineArrayLiteral` or similar. It's unclear what this protocol would +look like because each design has a different semantic guarantee: + +```swift +public protocol ExpressibleByInlineArrayLiteral: ~Copyable { + associatedtype Element: ~Copyable + + init(InlineArrayLiteral: consuming InlineArray) +} +``` + +This naive approach would satisfy a lot of types like `Array`, `Set`, +some hypothetical future noncopyable array, etc. These types actually want a +generic count and can allocate just enough space to hold all of those elements. + +However, this shape doesn't quite work for `InlineArray` itself because initializing +a `InlineArray<4, Int>` should require that the literal has exactly 4 elements. Note +that we wouldn't be able to impose a new constraint just for the conformer, so +`InlineArray` couldn't require that `N == count` and still have this witness the +requirement. Similarly, a `Pair` type could be InlineArray initialized, but only if +the InlineArray has exactly 2 elements. If we had the ability to define +`associatedvalue`, then this makes the conformance pretty trivial for both of +these types: + +```swift +public protocol ExpressibleByInlineArrayLiteral: ~Copyable { + associatedtype Element: ~Copyable + associatedvalue count: Int + + init(InlineArrayLiteral: consuming InlineArray) +} + +extension InlineArray: ExpressibleByInlineArrayLiteral { + init(InlineArrayLiteral: consuming InlineArray) { ... } +} + +extension Pair: ExpressibleByInlineArrayLiteral { + init(InlineArrayLiteral: consuming InlineArray<2, Element>) { ... } +} +``` + +But even with this design it's unsuitable for `Array` itself because it doesn't +want a static count for the literal, it still wants it to be generic. + +It would be nice to define something like this either on top of `InlineArray`, +parameter packs, or something else that would let us define statically the +number of elements we need for literal initialization or be dynamic if we opt to. + +### `FixedCapacityArray` and `SmallArray` + +In the same vein as this type, it may make sense to introduce some `FixedCapacityArray` +type which would support appending and removing elements given a fixed-capacity. + +```swift +var numbers: FixedCapacityArray<4, Int> = [1, 2] +print(numbers.capacity) // 4 +print(numbers.count) // 2 +numbers.append(3) +print(numbers.count) // 3 +numbers.append(4) +print(numbers.count) // 4 +numbers.append(5) // error: not enough space +``` + +This type is significantly different than the type we're proposing because +`InlineArray` defines a fixed-size meaning you cannot append or remove from it, but +it also requires that every single element is initialized. There must never be +an uninitialized element within a `InlineArray`, however for `FixedCapacityArray` this is +not true. It would act as a regular array with an initialized prefix and an +uninitialized suffix, it would be inline allocated (stack allocated for locals, +heap allocated if it's a class member, etc.), and it would not be growable. + +The difficulty in proposing such a type right now is that we have no way of +informing the compiler what parts of `FixedCapacityArray` are initialized and what +parts are not. This is critical for copy operations, move operations, and +destroy operations. Assuming that an uninitialized element is initialized and +attempting to perform any of these operations on it may lead to runtime crashes +which is definitely undesirable. + +Once we have `FixedCapacityArray` and some hypothetical noncopyable heap allocated +array type (which [SE-0437 Noncopyable Standard Library Primitives](0437-noncopyable-stdlib-primitives.md) +dons as `HypoArray` as a placeholder), it should be very trivial to define a +`SmallArray` type similar to the one found in LLVM APIs `llvm::SmallVector`. + +```swift +public enum SmallArray: ~Copyable { + case small(FixedCapacityArray) + case large(HypoArray) +} +``` + +which would act as an inline allocated array until one out grew the inline +capacity and would fall back to a dynamic heap allocation. + +### Syntax sugar + +We feel that this type will become as fundamental as `Array` and `Dictionary` +both of which have syntactic sugar for declaring a type of them, `[T]` for +`Array` and `[K: V]` for `Dictionary`. It may make sense to define something +similar for `InlineArray`, however we leave that as a future direction as the +spelling for such syntax is not critical to landing this type. + +It should be fairly trivial to propose such a syntax in the future either via a +new proposal, or as an amendment to this one. Such a change should only require +a newer compiler that supports the syntax and nothing more. + +Some syntax suggestions: + +* `[N x T]` or `[T x N]` +* `[N * T]` or `[T * N]` +* `T[N]` (from C) +* `[T; N]` (from Rust) + +Note that it may make more sense to have the length appear before the type. I +discuss this more in depth in [Reorder the generic arguments](#reorder-the-generic-arguments-InlineArrayt-n-instead-of-InlineArrayn-t). + +### C Interop changes + +With the introduction of `InlineArray`, we have a unique opportunity to fix another +pain point within the language with regards to C interop. Currently, the Swift +compiler imports a C array of type `T[24]` as a tuple of `T` with 24 elements. +Previously, this was really the only representation that the compiler could pick +to allow interfacing with C arrays. It was a real challenge working with these +fields from C in Swift. Consider the following C struct: + +```c +struct section_64 { + char sectname[16]; + char segname[16]; + uint64_t addr; + uint64_t size; + uint32_t offset; + uint32_t align; + ... +}; +``` + +Today, this gets imported as the following Swift struct: + +```swift +struct section_64 { + let sectname: (CChar, CChar, CChar, CChar, CChar, CChar, ... 10 more times) + let segname: (CChar, CChar, CChar, CChar, CChar, CChar, ... 10 more times) + let addr: UInt64 + let size: UInt64 + let offset: UInt32 + let align: UInt32 + ... +} +``` + +Using an instance of `section_64` in Swift for the most part is really easy. +Accessing things like `addr` or `size` are simple and easy to use, but using the +`sectname` property introduces a level of complexity that isn't so fun to use. + +```swift +func getSectionName(_ section: section_64) -> String { + withUnsafePointer(to: section.sectname) { + // This is unsafe! 'sectname' isn't guaranteed to have a null byte + // indicating the end of the C string! + String(cString: $0) + } +} + +func iterateSectionNameBytes(_ section: section_64) { + withUnsafeBytes(to: section.sectname) { + for byte in $0 { + ... + } + } +} +``` + +Having to resort to using very unsafe API to do anything useful with imported C +arrays is not something a memory safe language like Swift should be in the +business of. + +Ideally we could migrate the importer from using tuples to this new `InlineArray` +type, however that would be massively source breaking. A previous revision of +this proposal proposed an _upcoming_ feature flag that modules can opt into, +but this poses issues with the current importer implementation with regards to +inlinable code. + +Another idea was to import struct fields with C array types twice, one with the +existing name with a tuple type (as to not break source) and another with some +`InlineArray` suffix in the name with the `InlineArray` type. This works pretty well for +struct fields and globals, but it leaves fields and functions who have pointers +to C arrays in question as well (spelt `char (*x)[4]`). Do we import such +functions twice using a similar method of giving it a different name? Such a +solution would also incur a longer deprecation period to eventually having just +`InlineArray` be imported and no more tuples. + +We're holding off on any C interop changes here as there are still lots of open +questions as to what the best path forward is. + +## Alternatives considered + +### Reorder the generic arguments (`InlineArray` instead of `InlineArray`) + +If we directly followed existing APIs from C++, then obviously the length should +follow the element type. However we realized that when reading this type aloud, +it's "a InlineArray of 3 integers" for example instead of "a InlineArray of integers of +size 3". It gets more interesting the more dimensions you add. +Consider an MxN matrix. In C, you'd write this as `T[N][M]` but index it as +`[m][n]`. We don't want to introduce that sort of confusion (which is a good +argument against `T[N]` as a potential syntax sugar for this type), so the +length being before the underlying element makes the most sense at least for any +potential sugared form. `[M * [N * T]]` would be indexed directly as it is spelt +out in the sugared form, `[m][n]`. In light of that, we wouldn't want the sugar +form to have a different ordering than the generic type itself leading us to +believe that the length must be before the element type. + +## Revisions + +Previously, this type was named `Vector`, but has since been renamed to `InlineArray`. + +### A name other than `Vector` + +For obvious reasons, we cannot name this type `Swift.Array` to match the +"term of art" that other languages like C, C++, and Rust are using for this +exact type. However, while this name is the de facto for other languages, it +actually mischaracterizes the properties and behaviors of this type considering +existing terminology in mathematics. A. Stepanov mentions in his book, "From +Mathematics to Generic Programming", that using the name `std::vector` for their +dynamically allocated growable array type was perhaps a mistake for this same +reason: + +> If we are coming up with a name for something, or overloading an existing name, +> we should follow these three guidelines: +> +> 1. If there is an established term, use it. +> 2. Do not use an established term inconsistently with its accepted meaning. In +> particular, overload an operator or function name only when you will be +> preserving its existing semantics. +> 3. If there are conflicting usages, the much more established one wins. +> +> The name _vector_ in STL was taken from the earlier programming languages +> Scheme and Common Lisp. Unfortunately, this was inconsistent with the much +> older meaning of the term in mathematics and violates Rule 3; this data structure +> should have been called _array_. Sadly, if you make a mistake and violate these +> principles, the result might stay around for a long time. +> +> \- Stepanov, A. A., Rose, D. E. (2014). _From Mathematics to Generic Programming_. United Kingdom: Addison-Wesley. + +Indeed, the `std::vector` type goes against the definition of vector by being a +growable container, having a non-fixed magnitude. + +We fully acknowledge that the Swift types, `Swift.Array` and `Swift.Vector`, are +complete opposites of the C++ ones, `std::vector` and `std::array`. While it may +be confusing at first, ultimately we feel that our names are more in line with +the mathematical term of art. + +If there was any type we could add to the standard library whose name could be +`Vector`, it must be this one. + +## Acknowledgments + +I would like the thank the following people for helping in the design process +of this type: + +* Karoy Lorentey +* Guillaume Lessard +* Joe Groff +* Kuba Mracek +* Andrew Trick +* Erik Eckstein +* Philippe Hausler +* Tim Kientzle diff --git a/proposals/0454-memory-allocator.md b/proposals/0454-memory-allocator.md new file mode 100644 index 0000000000..c7c6f0d755 --- /dev/null +++ b/proposals/0454-memory-allocator.md @@ -0,0 +1,68 @@ +# Custom Allocator for Toolchain + +* Proposal: [SE-0454](0454-memory-allocator.md) +* Authors: [Saleem Abdulrasool](https://github.com/compnerd) +* Review Manager: [Alastair Houghton](https://github.com/al45tair) +* Status: **Accepted** +* Implementation: [swiftlang/swift#76563](https://github.com/swiftlang/swift/pull/76563) +* Review: ([review](https://forums.swift.org/t/se-454-adopt-mimalloc-for-windows-toolchain/77096)) + ([acceptance](https://forums.swift.org/t/accepted-se-0454-custom-allocator-for-toolchain-adopt-mimalloc-for-windows-toolchain/77413)) + +## Introduction + +The tools in the Swift toolchain require allocating data structures for +compiling the code. Different memory allocators have differing performance +characteristics. Changing the default memory allocator away from the default +(system) allocator can yield benefits if the allocator is better tuned to the +allocation patterns of the compiler. + +## Motivation + +A more effecient memory allocator would improve the performance of the compiler +on Windows. This allows better developer productivity by reducing compile time. + +## Proposed solution + +We propose to adopt mimalloc as the memory allocator for the Swift toolchain on +Windows. + +## Detailed design + +Building a test codebase yielded a 4% build time decrease when the toolchain was +built with mimalloc. + +## Source compatibility + +This proposal does not affect source compatibility. + +## ABI compatibility + +This proposal does not affect ABI of code. + +## Implications on adoption + +Additional files will need to be built, packaged, and shipped as part of the +toolchain. The mimalloc build is relatively light and the overall build time +impact is minimal. + +This change has no implications for the runtime, only the toolchain is changed. + +## Future directions + +None at this time. + +## Alternatives considered + +Alternative memory allocators were considered, including +[tcmalloc](https://github.com/google/tcmalloc) and +[tbb](https://github.com/intel/tbb). mimalloc is well supported, developed by +Microsoft, and has better characteristics comparatively. + +Leaving the allocator on the default system allocator leaves the compiler +without the performance improvements of an alternative allocator. + +## Acknowledgements + +Special thanks to @hjyamauchi for performing the work to integrate the mimalloc +build into the Windows build and collecting the performance numbers that showed +the improvement. diff --git a/proposals/0455-swiftpm-testable-build-setting.md b/proposals/0455-swiftpm-testable-build-setting.md new file mode 100644 index 0000000000..186eb31892 --- /dev/null +++ b/proposals/0455-swiftpm-testable-build-setting.md @@ -0,0 +1,64 @@ +# SwiftPM @testable build setting + +* Proposal: [SE-0455](0455-swiftpm-testable-build-setting.md) +* Authors: [Jake Petroules](https://github.com/jakepetroules) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Accepted** +* Implementation: [swiftlang/swift-package-manager#8004](https://github.com/swiftlang/swift-package-manager/pull/8004) +* Review: ([pitch](https://forums.swift.org/t/pitch-swiftpm-testable-build-setting/75084)) ([review](https://forums.swift.org/t/se-0455-swiftpm-testable-build-setting/77100)) ([acceptance](https://forums.swift.org/t/accepted-se-0455-swiftpm-testable-build-setting/77510)) + +## Introduction + +The current Swift Package Manager build system is currently hardcoded to pass the `-enable-testing` flag to the Swift compiler to enable `@testable import` when building in debug mode. + +Swift-evolution thread: [Pitch: [SwiftPM] @testable build setting](https://forums.swift.org/t/pitch-swiftpm-testable-build-setting/75084) + +## Motivation + +Not all targets in a given package make use of the `@testable import` feature (or wish to use it at all), but all targets are presently forced to build their code with this support enabled regardless of whether it's needed. + +Developers should be able to disable `@testable import` when it's not needed or desired, just as they're able to do so in Xcode's build system. + +On Windows in particular, where a shared library is limited to 65k exported symbols, disabling `@testable import` provides developers an option to significantly reduce the exported symbol count of a library by hiding all of the unnecessary internal APIs. It can also improve debug build performance as fewer symbols exported from a binary can result in faster linking. + +## Proposed solution + +Add a new Swift target setting API to specify whether testing should be enabled for the specified target, falling back to the current behavior by default. + +## Detailed design + +Add a new `enableTestableImport` API to `SwiftSetting` limited to manifests >= 6.1: + +```swift +public struct SwiftSetting { + // ... other settings + + @available(_PackageDescription, introduced: 6.1) + public static func enableTestableImport( + _ enable: Bool, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting { + ... + } +} +``` + +The existing `--enable-testable-imports` / `--disable-testable-imports` command line flag to `swift-test` currently defaults to `--enable-testable-imports`. It will be changed to default to "unspecified" (respecting any target settings), and explicitly passing `--enable-testable-imports` or `--disable-testable-imports` will force all targets to enable or disable testing, respectively. + +Attempting to enable `@testable import` in release builds will result in a build warning. + +## Security + +New language version setting has no implications on security, safety or privacy. + +## Impact on existing packages + +Since this is a new API, all existing packages will use the default behavior - `@testable import` will be enabled when building for the debug configuration, and disabled when building for the release configuration. + +## Future directions + +In a future manifest version, we may want to change `@testable import` to be disabled by default when building for the debug configuration. + +## Alternatives considered + +None. diff --git a/proposals/0456-stdlib-span-properties.md b/proposals/0456-stdlib-span-properties.md new file mode 100644 index 0000000000..bffb608e00 --- /dev/null +++ b/proposals/0456-stdlib-span-properties.md @@ -0,0 +1,293 @@ +# Add `Span`-providing Properties to Standard Library Types + +* Proposal: [SE-0456](0456-stdlib-span-properties.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [BufferView Roadmap](https://forums.swift.org/t/66211) +* Implementation: [swift PR #78561](https://github.com/swiftlang/swift/pull/78561), [swift PR #80116](https://github.com/swiftlang/swift/pull/80116), [swift-foundation PR#1276](https://github.com/swiftlang/swift-foundation/pull/1276) +* Review: ([pitch](https://forums.swift.org/t/76138)) ([review](https://forums.swift.org/t/se-0456-add-span-providing-properties-to-standard-library-types/77233)) ([acceptance](https://forums.swift.org/t/77684)) + +[SE-0446]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md +[SE-0447]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md +[PR-2305]: https://github.com/swiftlang/swift-evolution/pull/2305 +[SE-0453]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md + +## Introduction + +We recently [introduced][SE-0447] the `Span` and `RawSpan` types, but did not provide ways to obtain instances of either from existing types. This proposal adds properties that vend a lifetime-dependent `Span` from a variety of standard library types, as well as vend a lifetime-dependent `RawSpan` when the underlying element type supports it. + +## Motivation + +Many standard library container types can provide direct access to their internal representation. Up to now, it has only been possible to do so in an unsafe way. The standard library provides this unsafe functionality with closure-taking functions such as `withUnsafeBufferPointer()`, `withContiguousStorageIfAvailable()` and `withUnsafeBytes()`. These functions have a few different drawbacks, most prominently their reliance on unsafe types, which makes them unpalatable in security-conscious environments. Closure-taking API can also be difficult to compose with new features and with one another. These issues are addressed head-on with non-escapable types in general, and `Span` in particular. With this proposal, compatible standard library types will provide access to their internal representation via computed properties of type `Span` and `RawSpan`. + +## Proposed solution + +Computed properties returning [non-escapable][SE-0446] copyable values represent a particular case of lifetime relationships between two bindings. While initializing a non-escapable value in general requires [lifetime annotations][PR-2305] in order to correctly describe the lifetime relationship, the specific case of computed properties returning non-escapable copyable values can only represent one type of relationship between the parent binding and the non-escapable instance it provides: a borrowing relationship. + +For example, in the example below we have an instance of type `A`, with a well-defined lifetime because it is non-copyable. An instance of `A` can provide access to a type `B` which borrows the instance `A`: + +```swift +struct A: ~Copyable, Escapable {} +struct B: ~Escapable, Copyable { + init(_ a: borrowing A) {} +} +extension A { + var b: B { B(self) } +} + +func function() { + var a = A() + var b = a.b // access to `a` begins here + read(b) + // `b` has ended here, ending access to `a` + modify(&a) // `modify()` can have exclusive access to `a` +} +``` +If we were to attempt using `b` again after the call to `modify(&a)`, the compiler would report an overlapping access error, due to attempting to mutate `a` (with `modify(&a)`) while it is already being accessed through `b`'s borrow. Note that the copyability of `B` means that it cannot represent a mutation of `A`; it therefore represents a non-exclusive borrowing relationship. + +Given this, we propose to enable the definition of a borrowing relationship via a computed property. With this feature we then propose to add `span` computed properties to standard library types that can share access to their internal typed memory. When a `span` has `BitwiseCopyable` elements, it will have a `bytes` computed property to share a view of the memory it represents as untyped memory. + +One of the purposes of `Span` is to provide a safer alternative to `UnsafeBufferPointer`. This proposal builds on it and allows us to rewrite code reliant on `withUnsafeBufferPointer()` to use `span` properties instead. Eventually, code that requires access to contiguous memory can be rewritten to use `Span`, gaining better composability in the process. For example: + +```swift +let result = try myArray.withUnsafeBufferPointer { buffer in + let indices = findElements(buffer) + var myResult = MyResult() + for i in indices { + try myResult.modify(buffer[i]) + } +} +``` + +This closure-based call is difficult to evolve, such as making `result` have a non-copyable type, adding a concurrent task, or adding typed throws. An alternative based on a vended `Span` property would look like this: + +```swift +let span = myArray.span +let indices = findElements(span) +var myResult = MyResult() +for i in indices { + try myResult.modify(span[i]) +} +``` + +In this version, code evolution is not constrained by a closure. Incorrect escapes of `span` will be diagnosed by the compiler, and the `modify()` function can be updated with typed throws, concurrency or other features as necessary. + +## Detailed Design + +Computed property getters returning non-escapable and copyable types (`~Escapable & Copyable`) become possible, requiring no additional annotations. The lifetime of their returned value depends on the type vending them. A `~Escapable & Copyable` value borrows another binding. In terms of the law of exclusivity, a borrow is a read-only access. Multiple borrows are allowed to overlap, but cannot overlap with any mutation. + +A computed property getter defined on an `Escapable` type and returning a `~Escapable & Copyable` value establishes a borrowing lifetime relationship of the returned value on the callee's binding. As long as the returned value exists (including local copies,) then the callee's binding remains borrowed. + +A computed property getter defined on a non-escapable and copyable (`~Escapable & Copyable`) type and returning a `~Escapable & Copyable` value copies the lifetime dependency of the callee. The returned value becomes an additional borrow of the callee's dependency, but is otherwise independent from the callee. + +A computed property getter defined on a non-escapable and non-copyable (`~Escapable & ~Copyable`) type returning a `~Escapable & Copyable` value establishes a borrowing lifetime relationship of the returned value on the callee's binding. As long as the returned value exists (including local copies,) then the callee's binding remains borrowed. + +By allowing the language to define lifetime dependencies in these limited ways, we can add `Span`-providing properties to standard library types. + +#### Extensions to Standard Library types + +The standard library and Foundation will provide `span` computed properties, returning lifetime-dependent `Span` instances. These computed properties are the safe and composable replacements for the existing `withUnsafeBufferPointer` closure-taking functions. + +```swift +extension Array { + /// Share this `Array`'s elements as a `Span` + var span: Span { get } +} + +extension ArraySlice { + /// Share this `Array`'s elements as a `Span` + var span: Span { get } +} + +extension ContiguousArray { + /// Share this `Array`'s elements as a `Span` + var span: Span { get } +} + +extension String.UTF8View { + /// Share this `UTF8View`'s code units as a `Span` + var span: Span { get } +} + +extension Substring.UTF8View { + /// Share this `UTF8View`'s code units as a `Span` + var span: Span { get } +} + +extension CollectionOfOne { + /// Share this `Collection`'s element as a `Span` + var span: Span { get } +} + +extension KeyValuePairs { + /// Share this `Collection`'s elements as a `Span` + var span: Span<(Key, Value)> { get } +} +``` + +Following the acceptance of [`InlineArray`][SE-0453], we will also add the following: + +```swift +extension InlineArray where Element: ~Copyable { + /// Share this `InlineArray`'s elements as a `Span` + var span: Span { get } +} +``` + +#### Accessing the raw bytes of a `Span` + +When a `Span`'s element is `BitwiseCopyable`, we allow viewing the underlying memory as raw bytes with `RawSpan`: + +```swift +extension Span where Element: BitwiseCopyable { + /// Share the raw bytes of this `Span`'s elements + var bytes: RawSpan { get } +} +``` + +The returned `RawSpan` instance will borrow the same binding as is borrowed by the `Span`. + +#### Extensions to unsafe buffer types + +We hope that `Span` and `RawSpan` will become the standard ways to access shared contiguous memory in Swift, but current API provide `UnsafeBufferPointer` and `UnsafeRawBufferPointer` instances to do this. We will provide ways to unsafely obtain `Span` and `RawSpan` instances from them, in order to bridge `UnsafeBufferPointer` to contexts that use `Span`, or `UnsafeRawBufferPointer` to contexts that use `RawSpan`. + +```swift +extension UnsafeBufferPointer { + /// Unsafely view this buffer as a `Span` + var span: Span { get } +} + +extension UnsafeMutableBufferPointer { + /// Unsafely view this buffer as a `Span` + var span: Span { get } +} + +extension UnsafeRawBufferPointer { + /// Unsafely view this raw buffer as a `RawSpan` + var bytes: RawSpan { get } +} + +extension UnsafeMutableRawBufferPointer { + /// Unsafely view this raw buffer as a `RawSpan` + var bytes: RawSpan { get } +} +``` + +All of these unsafe conversions return a value whose lifetime is dependent on the _binding_ of the UnsafeBufferPointer. This dependency does not keep the underlying memory alive. As is usual where the `UnsafePointer` family of types is involved, the programmer must ensure the memory remains allocated while it is in use. Additionally, the following invariants must remain true for as long as the `Span` or `RawSpan` value exists: + + - The underlying memory remains initialized. + - The underlying memory is not mutated. + +Failure to maintain these invariants results in undefined behaviour. + +#### Extensions to `Foundation.Data` + +While the `swift-foundation` package and the `Foundation` framework are not governed by the Swift evolution process, `Data` is similar in use to standard library types, and the project acknowledges that it is desirable for it to have similar API when appropriate. Accordingly, we would add the following properties to `Foundation.Data`: + +```swift +extension Foundation.Data { + // Share this `Data`'s bytes as a `Span` + var span: Span { get } + + // Share this `Data`'s bytes as a `RawSpan` + var bytes: RawSpan { get } +} +``` + +Unlike with the standard library types, we plan to have a `bytes` property on `Foundation.Data` directly. This type conceptually consists of untyped bytes, and `bytes` is likely to be the primary way to directly access its memory. As `Data`'s API presents its storage as a collection of `UInt8` elements, we provide both `bytes` and `span`. Types similar to `Data` may choose to provide both typed and untyped `Span` properties. + +#### Performance + +The `span` and `bytes` properties should be performant and return their `Span` or `RawSpan` with very little work, in O(1) time. This is the case for all native standard library types. There is a performance wrinkle for bridged `Array` and `String` instances on Darwin-based platforms, where they can be bridged to Objective-C types that may not be represented in contiguous memory. In such cases the implementation will eagerly copy the underlying data to the native Swift form, and return a `Span` or `RawSpan` pointing to that copy. + +This eager copy behaviour will be specific to the `span` and `bytes` properties, and therefore the memory usage behaviour of existing unchanged code will remain the same. New code that adopts the `span` and `bytes` properties will occasionally have higher memory usage due to the eager copies, but we believe this performance compromise is the right approach for the standard library. The alternative is to compromise the design for all platforms supported by Swift, and we consider that a non-starter. + +As a result of the eager copy behaviour for bridged `String.UTF8View` and `Array` instances, the `span` property for these types will have a documented performance characteristic of "amortized constant time performance." + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a version of the Swift standard library which include the `Span` and `RawSpan` types. + +## Alternatives considered + +#### Adding `withSpan()` and `withBytes()` closure-taking functions + +The `span` and `bytes` properties aim to be safe replacements for the `withUnsafeBufferPointer()` and `withUnsafeBytes()` closure-taking functions. We could consider `withSpan()` and `withBytes()` closure-taking functions that would provide an quicker migration away from the older unsafe functions. We do not believe the closure-taking functions are desirable in the long run. In the short run, there may be a desire to clearly mark the scope where a `Span` instance is used. The default method would be to explicitly consume a `Span` instance: +```swift +var a = ContiguousArray(0..<8) +var span = a.span +read(span) +_ = consume span +a.append(8) +``` + +In order to visually distinguish this lifetime, we could simply use a `do` block: +```swift +var a = ContiguousArray(0..<8) +do { + let span = a.span + read(span) +} +a.append(8) +``` + +A more targeted solution may be a consuming function that takes a non-escaping closure: +```swift +var a = ContiguousArray(0..<8) +var span = a.span +consuming(span) { span in + read(span) +} +a.append(8) +``` + +During the evolution of Swift, we have learned that closure-based API are difficult to compose, especially with one another. They can also require alterations to support new language features. For example, the generalization of closure-taking API for non-copyable values as well as typed throws is ongoing; adding more closure-taking API may make future feature evolution more labor-intensive. By instead relying on returned values, whether from computed properties or functions, we build for greater composability. Use cases where this approach falls short should be reported as enhancement requests or bugs. + +#### Different naming for the properties + +We originally proposed the name `storage` for the `span` properties introduced here. That name seems to imply that the returned `Span` is the storage itself, rather than a view of the storage. That would be misleading for types that own their storage, especially those that delegate their storage to another type, such as a `ContiguousArray`. In such cases, it would make sense to have a `storage` property whose type is the type that implements the storage. + +#### Disallowing the definition of non-escapable properties of non-escapable types + +The particular case of the lifetime dependence created by a property of a copyable non-escapable type is not as simple as when the parent type is escapable. There are two possible ways to define the lifetime of the new instance: it can either depend on the lifetime of the original instance, or it can acquire the lifetime of the original instance and be otherwise independent. We believe that both these cases can be useful, but that in the majority of cases the desired behaviour will be to have an independent return value, where the newly returned value borrows the same binding as the callee. Therefore we believe that is reasonable to reserve the unannotated spelling for this more common case. + +The original version of this pitch disallowed this. As a consequence, the `bytes` property had to be added on each individual type, rather than having `bytes` as a conditional property of `Span`. + +#### Omitting extensions to `UnsafeBufferPointer` and related types + +We could omit the extensions to `UnsafeBufferPointer` and related types, and rely instead of future `Span` and `RawSpan` initializers. The initializers can have the advantage of being able to communicate semantics (somewhat) through their parameter labels. However, they also have a very different shape than the `span` computed properties we are proposing. We believe that the adding the same API on both safe and unsafe types is advantageous, even if the preconditions for the properties cannot be statically enforced. + +## Future directions + +Note: The future directions stated in [SE-0447](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md#Directions) apply here as well. + +#### Safe mutations with `MutableSpan` + +Some data structures can delegate mutations of their owned memory. In the standard library the function `withMutableBufferPointer()` provides this functionality in an unsafe manner. We expect to add a `MutableSpan` type to support delegating mutations of initialized memory. Standard library types will then add a way to vend `MutableSpan` instances. This could be with a closure-taking `withMutableSpan()` function, or a new property, such as `var mutableStorage`. Note that a computed property providing mutable access needs to have a different name than the `span` properties proposed here, because we cannot overload the return type of computed properties based on whether mutation is desired. + +#### A `ContiguousStorage` protocol + +An early version of the `Span` proposal ( [SE-0447][SE-0447] ) proposed a `ContiguousStorage` protocol by which a type could indicate that it can provide a `Span`. `ContiguousStorage` would form a bridge between generically-typed interfaces and a performant concrete implementation. It would supersede the rejected [SE-0256](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0256-contiguous-collection.md), and many of the standard library collections could conform to `ContiguousStorage`. + +The properties added by this proposal are largely the concrete implementations of `ContiguousStorage`. As such, it seems like an obvious enhancement to this proposal. + +Unfortunately, a major issue prevents us from proposing it at this time: the ability to suppress requirements on `associatedtype` declarations was deferred during the review of [SE-0427](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md). Once this restriction is lifted, then we could propose a `ContiguousStorage` protocol. + +The other limitation stated in [SE-0447][SE-0447]'s section about `ContiguousStorage` is "the inability to declare a `_read` acessor as a protocol requirement." This proposal's addition to enable defining a borrowing relationship via a computed property is a solution to that, as long as we don't need to use a coroutine accessor to produce a `Span`. While allowing the return of `Span`s through coroutine accessors may be undesirable, whether it is undesirable is unclear until coroutine accessors are formalized in the language. + +`span` properties on standard library SIMD types + +This proposal as reviewed included `span` properties for the standard library `SIMD` types. We are deferring this feature at the moment, since it is difficult to define these succinctly. The primary issue is that the `SIMD`-related protocols do not explicitly require contiguous memory; assuming that they are represented in contiguous memory fails with theoretically-possible examples. We could define the `span` property systematically for each concrete SIMD type in the standard library, but that would be very repetitive (and expensive from the point of view of code size.) We could also fix the SIMD protocols to require contiguous memory, enabling a succinct definition of their `span` property. Finally, we could also rely on converting `SIMD` types to `InlineArray`, and use the `span` property defined on `InlineArray`. + +## Acknowledgements + +Thanks to Ben Rimmington for suggesting that the `bytes` property should be on `Span` rather than on every type. diff --git a/proposals/0457-duration-attosecond-represenation.md b/proposals/0457-duration-attosecond-represenation.md new file mode 100644 index 0000000000..b88984362b --- /dev/null +++ b/proposals/0457-duration-attosecond-represenation.md @@ -0,0 +1,110 @@ +# Expose attosecond representation of `Duration` + +* Proposal: [SE-0457](0457-duration-attosecond-represenation.md) +* Authors: [Philipp Gabriel](https://github.com/ph1ps) +* Review Manager: [Stephen Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#78202](https://github.com/swiftlang/swift/pull/78202) +* Review: ([pitch](https://forums.swift.org/t/pitch-adding-int128-support-to-duration))([review](https://forums.swift.org/t/se-0457-expose-attosecond-representation-of-duration/77249)) + +## Introduction +This proposal introduces public APIs to enable seamless integration of `Int128` into the `Duration` type. Specifically, it provides support for directly accessing a `Duration`'s attosecond representation via the newly available `Int128` type and simplifies the creation of `Duration` values from attoseconds. + +## Motivation +The `Duration` type currently offers two ways to construct and decompose itself: + +**Low and high bits** +```swift +public struct Duration: Sendable { + public var _low: UInt64 + public var _high: Int64 + public init(_high: Int64, low: UInt64) { ... } +} +``` +**Components** +```swift +extension Duration { + public var components: (seconds: Int64, attoseconds: Int64) { ... } + public init(secondsComponent: Int64, attosecondsComponent: Int64) { ... } +} +``` +However, both approaches have limitations when it comes to exposing `Duration`'s total attosecond representation: +- The `_low` and `_high` properties are underscored, indicating that their direct use is discouraged. +- The `components` property decomposes the value into seconds and attoseconds, requiring additional arithmetic operations for many use cases. + +This gap becomes particularly evident in scenarios like generating a random `Duration`, which currently requires verbose and potentially inefficient code: +```swift +func randomDuration(upTo maxDuration: Duration) -> Duration { + let attosecondsPerSecond: Int128 = 1_000_000_000_000_000_000 + let upperRange = Int128(maxDuration.components.seconds) * attosecondsPerSecond + Int128(maxDuration.components.attoseconds) + let (seconds, attoseconds) = Int128.random(in: 0.. Duration { + return Duration(attoseconds: Int128.random(in: 0.. Duration { ... } +} +``` + +However, this approach would introduce asymmetry to other factory methods which support both `Double` and `BinaryInteger` overloads: +```swift +extension Duration { + public static func microseconds(_ microseconds: T) -> Duration { ... } + public static func microseconds(_ microseconds: Double) -> Duration { ... } +} +``` +For attoseconds, adding these overloads would lead to practical issues: + +1. A `Double` overload is nonsensical because sub-attoseconds are not supported, meaning the method cannot represent fractional attoseconds. +2. A `BinaryInteger` overload introduces additional complexity. Since it would need to support types other than `Int128`, arithmetic operations would be necessary to ensure correct scaling and truncation, negating the simplicity and precision that the `Int128`-specific initializer aims to provide. + +Ultimately, the static func `attoseconds(_:)` would likely end up as a one-off method with only an `Int128` overload. This inconsistency diminishes the appeal of the factory method approach. The proposed `init(attoseconds:)` initializer avoids these issues, offering a direct and clear way to work with attoseconds, while remaining symmetrical with the existing `Duration` API structure. diff --git a/proposals/0458-strict-memory-safety.md b/proposals/0458-strict-memory-safety.md new file mode 100644 index 0000000000..dce70053c2 --- /dev/null +++ b/proposals/0458-strict-memory-safety.md @@ -0,0 +1,841 @@ +# Opt-in Strict Memory Safety Checking + +* Proposal: [SE-0458](0458-strict-memory-safety.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.2)** +* Feature name: `StrictMemorySafety` +* Vision: [Optional Strict Memory Safety](https://github.com/swiftlang/swift-evolution/blob/main/visions/memory-safety.md) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/f2cab4ddc3381d1dc7a970e813ed29e27b5ae43f/proposals/0458-strict-memory-safety.md) [2](https://github.com/swiftlang/swift-evolution/blob/9d180aea291c6b430bcc816ce12ef0174ec0237b/proposals/0458-strict-memory-safety.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-opt-in-strict-memory-safety-checking/76689)) ([review](https://forums.swift.org/t/se-0458-opt-in-strict-memory-safety-checking/77274)) ([first revision](https://forums.swift.org/t/se-0458-opt-in-strict-memory-safety-checking/77274/33)) ([second revision](https://forums.swift.org/t/se-0458-opt-in-strict-memory-safety-checking/77274/51)) ([acceptance](https://forums.swift.org/t/accepted-se-0458-opt-in-strict-memory-safety-checking/78116)) + +## Introduction + +[Memory safety](https://en.wikipedia.org/wiki/Memory_safety) is a property of programming languages and their implementations that prevents programmer errors from manifesting as [undefined behavior](https://en.wikipedia.org/wiki/Undefined_behavior) at runtime. Undefined behavior effectively breaks the semantic model of a language, with unpredictable results including crashes, data corruption, and otherwise-impossible program states. Such behavior can lead to hard-to-reproduce bugs as well as introduce security vulnerabilities. + +Swift provides memory safety with a combination of language affordances and runtime checking. However, Swift also deliberately includes some unsafe constructs, such as the `Unsafe` pointer types in the standard library, language features like `nonisolated(unsafe)`, and interoperability with unsafe languages like C. For most Swift developers, this is a pragmatic solution that provides an appropriate level of memory safety while not getting in the way. + +However, some projects want to require stronger memory-safety guarantees than Swift provides by default. These projects want to pay closer attention to uses of unsafe constructs in their code, and discourage casual use of unsafe constructs when a safe alternative exists. This proposal introduces opt-in strict memory safety checking to identify those places in Swift code that make use of unsafe language constructs and APIs. Any code written within this strictly-safe subset also works as “normal” Swift and can interoperate with existing Swift code. + +## Motivation + +Much of the recent focus on memory safety is motivated by security, because memory safety issues offer a fairly direct way to compromise a program: in fact, the lack of memory safety in C and C++ has been found to be the root cause for ~70% of reported security issues in various analyses [[1](https://msrc.microsoft.com/blog/2019/07/a-proactive-approach-to-more-secure-code/)][[2](https://www.chromium.org/Home/chromium-security/memory-safety/)]. + +### Dimensions of memory safety + +While there are a number of potential definitions for memory safety, the one provided by [this blog post](https://security.apple.com/blog/towards-the-next-generation-of-xnu-memory-safety/) breaks it down into five dimensions of safety: + +* **Lifetime safety** : all accesses to a value are guaranteed to occur during its lifetime. Violations of this property, such as accessing a value after its lifetime has ended, are often called use-after-free errors. +* **Bounds safety**: all accesses to memory are within the intended bounds of the memory allocation, such as accessing elements in an array. Violations of this property are called out-of-bounds accesses. +* **Type safety** : all accesses to a value use the type to which it was initialized, or a type that is compatible with that type. For example, one cannot access a `String` value as if it were an `Array`. Violations of this property are called type confusions. +* **Initialization safety** : all values are initialized properly prior to being used, so they cannot contain unexpected data. Violations of this property often lead to information disclosures (where data that should be invisible becomes available) or even other memory-safety issues like use-after-frees or type confusions. +* **Thread safety:** all values are accessed concurrently in a manner that is synchronized sufficiently to maintain their invariants. Violations of this property are typically called data races, and can lead to any of the other memory safety problems. + +### Memory safety in Swift + +Since its inception, Swift has provided memory safety for the first four dimensions. Lifetime safety is provided for reference types by automatic reference counting and for value types via [memory exclusivity](https://www.swift.org/blog/swift-5-exclusivity/); bounds safety is provided by bounds-checking on `Array` and other collections; type safety is provided by safe features for casting (`as?` , `is` ) and `enum` s; and initialization safety is provided by “definite initialization”, which doesn’t allow a variable to be accessed until it has been defined. Swift 6’s strict concurrency checking extends Swift’s memory safety guarantees to the last dimension. + +Swift achieves safety with a mixture of static and dynamic checks. Static checks are better when possible, because they are surfaced at compile time and carry no runtime cost. Dynamic checks are sometimes necessary and are still acceptable, so long as the failure can't escalate into a memory safety problem. Swift offers unsafe features to allow problems to be solved when neither static nor dynamic checks are sufficient. These unsafe features can still be used without compromising memory safety, but doing so requires more care because they have requirements that Swift can't automatically check. + +For example, Swift solves null references with optional types. Statically, Swift prevents you from using an optional reference without checking it first. If you're sure it's non-null, you can use the `!` operator, which is safe because Swift will dynamically check for `nil`. If you really can't afford that dynamic check, you can use [`unsafelyUnwrapped`](https://developer.apple.com/documentation/swift/optional/unsafelyunwrapped). This can still be correct if you can prove that the reference is definitely non-null for some reason that Swift doesn't know. But it is an unsafe feature because it admits violations if you're wrong. + +## Proposed solution + +This proposal introduces an opt-in strict memory safety checking mode that identifies all uses of unsafe behavior within the given module. There are several parts to this change: + +* A compiler flag `-strict-memory-safety` that enables warnings for all uses of unsafe constructs within a given module. All warnings will be in the diagnostic group `StrictMemorySafety`, enabling precise control over memory-safety-related warnings per [SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md). When strict memory safety is enabled, the `StrictMemorySafety` feature will be set: `#if hasFeature(StrictMemorySafety)` can be used to detect when Swift code is being compiled in this mode. +* An attribute `@unsafe` that indicates that a declaration is unsafe to use. Such declarations may use unsafe constructs within their signatures. +* A corresponding attribute `@safe` that indicates that a declaration whose signature contains unsafe constructs is actually safe to use. For example, the `withUnsafeBufferPointer` method on `Array` has an unsafe type in its signature (`self`), but is actually safe to use because it handles safety for the unsafe buffer pointer it vends to its closure argument. The closure itself will need to handle the unsafety when using that unsafe buffer pointer. +* An `unsafe` expression that marks any use of unsafe constructs in an expression, much like `try` and `await`. +* Standard library annotations to identify unsafe declarations. + +### Example of `unsafe` usage + +The `UnsafeBufferPointer` type will be marked with `@unsafe` in the Standard library, as will the other unsafe types (e.g., `UnsafePointer`, `UnsafeRawPointer`): + +```swift +@unsafe +public struct UnsafeBufferPointer { ... } +``` + +This indicates that use of this type is not memory-safe. Any declaration that has `UnsafeBufferPointer` as part of its type is implicitly `@unsafe`. + +```swift +// note: implicitly @unsafe due to the use of the unsafe type UnsafePointer +func sumIntBuffer(_ address: UnsafePointer?, _ count: Int) -> Int { ... } +``` + +Users of this function that enable strict safety checking will see warnings when using it. For example: + +```swift +extension Array { + func sum() -> Int { + withUnsafeBufferPointer { buffer in + // warning: use of unsafe function 'sumIntBuffer' and unsafe property 'baseAddress' + sumIntBuffer(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +Both the call to `sumIntBuffer` and access to the property `UnsafeBufferPointer.baseAddress` involve unsafe code, and therefore will produce a warning. Because `UnsafeBufferPointer` and `UnsafePointer` are `@unsafe` types, this code will get a warning regardless of whether the declarations were marked `@unsafe`, because having unsafe types in the signature of a declaration implies that they are `@unsafe`. This helps us identify more unsafe code even when the libraries we depend on haven't enabled strict safety checking themselves. + +To suppress these warnings, the expressions involving unsafe code must be marked with `unsafe` in the same manner as one would mark a throwing expression with `try` or an asynchronous expression with `async`. The warning-free version of this code is: + +```swift +extension Array { + func sum() -> Int { + withUnsafeBufferPointer { buffer in + unsafe sumIntBuffer(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +The `unsafe` keyword here indicates the presence of unsafe code within that expression. As with `try` and `await`, it can cover multiple sources of unsafety within that expression: the call to `sumIntBuffer` is unsafe, as is the use of `buffer` and `buffer.baseAddress`, yet they are all covered by one `unsafe`. It is up to the authors of an unsafe API to document the conditions under which it is safe to use that API, and the user of that API to ensure that those conditions are met within the `unsafe` expression. + +Unlike `try`, `unsafe` doesn't propagate outward: we do *not* require that the `sum` function be marked `@unsafe` just because it has unsafe code in it. Similarly, the call to `withUnsafeBufferPointer` doesn't have to be marked as `unsafe` just because it has a closure that is unsafe. The programmer may choose to indicate that `sum` is unsafe, but the assumption is that unsafe behavior is properly encapsulated when using `unsafe` if the signature doesn't contain any unsafe types. + +The function `Array.withUnsafeBufferPointer` has an unsafe type in its signature, because it passes an unsafe buffer pointer to its closure parameter. However, this function itself is addressing all of the memory-safety issues with providing such a pointer, and it's up to the closure itself to ensure that it is memory safe. Therefore, we mark `withUnsafeBufferPointer` with the `@safe` attribute to indicate that it iself is not introducing memory-safety issues: + +```swift +extension Array { + @safe func withUnsafeBufferPointer( + _ body: (UnsafeBufferPointer) throws(E) -> R + ) throws(E) -> R +} +``` + +The new attributes `@safe` and `@unsafe`, as well as the `unsafe` expression, are all available in Swift regardless of whether strict safety checking is enabled, and all code using these features retains the same semantics. Strict safety checking will *only* produce diagnostics. + +### A larger example: `swapAt` on unsafe pointers + +The operation `UnsafeMutableBufferPointer.swapAt` swaps the values at the given two indices in the buffer. Under the proposed strict safety mode, it would look like this: + +```swift +extension UnsafeMutableBufferPointer { + /*implicitly @unsafe*/ + public func swapAt(_ i: Index, _ j: Index) { + guard i != j else { return } + precondition(i >= 0 && j >= 0) + precondition(i < endIndex && j < endIndex) + let pi = unsafe (baseAddress! + i) + let pj = unsafe (baseAddress! + j) + let tmp = unsafe pi.move() + unsafe pi.moveInitialize(from: pj, count: 1) + unsafe pj.initialize(to: tmp) + } +} +``` + +The `swapAt` implementation uses a mix of safe and unsafe code. The code marked with `unsafe` identifies operations that Swift cannot verify memory safety for: + +* Performing pointer arithmetic on `baseAddress`: Swift cannot reason about the lifetime of that underlying pointer, nor whether the resulting pointer is still within the bounds of the allocation. +* Moving and initializing the actual elements. The elements need to already be initialized. + +The code itself has preconditions to ensure that the provided indices aren't out of bounds before performing the pointer arithmetic. However, there are other safety properties that cannot be checked with preconditions: that the memory associated with the pointer has been properly initialized, has a lifetime that spans the whole call, and is not being used simultaneously by any other part of the code. These safety properties are something that must be established by the *caller* of `swapAt`. Therefore, `swapAt` is considered `@unsafe` because callers of it need to reason about these properties. Because it's part of an unsafe type, it is implicitly `@unsafe`, although the author could choose to mark it as `@unsafe` explicitly. + +### Incremental adoption + +The strict memory safety checking proposed here enforces a subset of Swift. Code written within this subset must also be valid Swift code, and must interoperate with Swift code that does not use this strict checking. Compared to other efforts in Swift that introduce stricter checking or a subset, this mode is smaller and more constrained, providing better interoperability and a more gradual adoption curve: + +* Strict concurrency checking, the focus of the Swift 6 language mode, required major changes to the type system, including the propagation of `Sendable` and the understanding of what code must be run on the main actor. These are global properties that don't permit local reasoning, or even local fixes, making the interoperability problem particularly hard. In contrast, strict safety checking has little or no effect on the type system, and unsafety can be encapsulated with `unsafe` expressions or ignored by a module that doesn't enable the checking. +* [Embedded Swift](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md) is a subset of Swift that works without a runtime. Like the proposed strictly-safe subset, code written in Embedded Swift will also work as regular Swift. Embedded Swift and the strict safety checking proposed here are orthogonal and can be composed to (for example) ensure that firmware written in Swift has no runtime and provides the best memory-safety guarantees. + +A Swift module that adopts strict safety checking can address all of the resulting diagnostics by applying the `@unsafe` attribute and `unsafe` expression in the appropriate places, without changing any other code. This application of attributes can be automated through Fix-Its, making it possible to enable the mode and silence all diagnostics automatically. It would then be left to the programmer to audit those places that have used `unsafe` to encapsulate unsafe behavior, to ensure that they are indeed safe. Note that the strict safety checking does not by itself make the code more memory-safe: rather, it identifies those constructs that aren't safe, encouraging the use of safe alternatives and making it easier to audit for unsafe behavior. + +The introduction of the `@unsafe` attribute on a declaration has no effect on clients compiled without strict safety enabled. For clients that have enabled strict safety, they will start diagnosing uses of the newly-`@unsafe` API. However, these diagnostics are warnings with their own diagnostic group, so a client can ensure that they do not prevent the client from building. Therefore, modules can adopt strict safety checking at their own pace (or not) and clients of those modules are never "stuck" having to make major changes in response. + +## Detailed design + +This section describes how the primary proposed constructs, the `@unsafe` attribute, `@safe` attribute, and `unsafe` expression, interact with the strict memory safety mode, and enumerates the places in the language, standard library, and compiler that introduce non-memory-safe code. + +### Sources of unsafety + +There are a number of places in the language where one can introduce memory unsafety. This section enumerates the ways in which the language, library, and user code can introduce memory unsafety. + +#### Unsafe language constructs + +The following language constructs are always considered to be unsafe: + +* `unowned(unsafe)`: Used to store a reference without maintaining its reference count. The safe counterpart, `unowned`, uses dynamic checking to ensure that the reference isn't accessed after the corresponding object has been released. The `unsafe` variant disables that dynamic checking. Uses of `unowned(unsafe)` entities are not memory-safe. +* `unsafeAddressor`, `unsafeMutableAddressor`: These accessors vend an unsafe pointer, and are therefore unsafe to declare. Other accessors (e.g., `get` and `set`) can provide safe alternatives. The accessors are considered to be part of the signature of the property or subscript they're associated with, making the property implicitly `@unsafe` unless explicitly marked `@safe`. +* `@exclusivity(unchecked)`: Used to remove dynamic exclusivity checks from a particular variable, which can mean that dynamic exclusivity violations go undetected at run time, causing a memory safety violation. Uses of `@exclusivity(unchecked)` entities are not memory-safe. + +The following language constructs are considered to be unsafe when strict concurrency checking is enabled (i.e., in the Swift 6 language mode): + +* `nonisolated(unsafe)`: Allows a property to be accessed from concurrent code without ensuring that such accesses are done so safely. Uses of `nonisolated(unsafe)` entities are not memory-safe. +* `@preconcurrency` imports: Suppresses diagnostics related to data race safety when they relate to specific imported modules, which can introduce thread safety issues. The `@preconcurrency` import will need to be annotated with `@unsafe` in the strict safety mode. + +#### `@unsafe` attribute + +The `@unsafe` attribute can be applied to any declaration to indicate that use of that declaration can undermine memory safety. Here are some examples: + +```swift +@unsafe +public struct DataWrapper { + var buffer: UnsafeBufferPointer +} + +@unsafe +func writeToRegisterAt(integerAddress: Int, value: Int32) { ... } +``` + +Uses of `DataWrapper` and `writeToRegisterAt` within executable code will be considered to be memory-unsafe. + +When a declaration uses unsafe types within its signature, it is implicitly considered to be `@unsafe`. The signature of a declaration is the interface that the declaration presents to clients, including the parameter and result types of functions, the type of properties, and any generic parameters and requirements. For example, a method on `DataWrapper` defined as follows: + +```swift +extension DataWrapper { + public func checksum() -> Int32 { + crc32(0, buffer.baseAddress, buffer.count) + } +} +``` + +will be implicitly `@unsafe` because the type of the implicit `self` parameter is `@unsafe`. + +Generally speaking, a declaration's signature is everything that isn't within the "body" of the definition that's enclosed in braces or following the `=` of a property. One notable exception to this is default arguments, because default arguments of functions are part of the implementation of a function, not its signature. For example, the following function does not have any unsafe types in its signature, even though the default argument for `value` involves unsafe code. That unsafe code is effectively part of the body of the function, so it follows the rules for `unsafe` expressions. + +```swift +func hasDefault(value: Int = unsafe getIntegerUnsafely()) { ... } +``` + +#### Unsafe conformances + +A given type might implement a protocol in a manner that introduces unsafety, for example because the operations needed to satisfy protocol requirements cannot ensure that all uses through the protocol can maintain memory safety. For example, the `UnsafeBufferPointer` type conforms to the `Collection` protocol, but it cannot do so in a safe way because `UnsafeBufferPointer` does not provide the lifetime, bounds, or initialization safety that clients of the `Collection` protocol expect. Such conformances should be explicitly marked `@unsafe`, e.g., + +```swift +extension UnsafeBufferPointer: @unsafe Collection { ... } +``` + +Unsafe conformances are similar to unsafe types in that their presence in a declaration's signature will make that declaration implicitly `@unsafe`. For example, the use of a collection algorithm such as `firstIndex(where:)` with an `UnsafeBufferPointer` will be considered unsafe because the conformance above is `@unsafe`. + +#### Unsafe standard library APIs + +Much of the identification of unsafe Swift code that becomes available with the strict memory safety mode is due to the identification of unsafe declarations within the standard library itself, and their propagation to other types that use them. In the standard library, the following functions and types would be marked `@unsafe` : + +* `Unsafe(Mutable)(Raw)(Buffer)Pointer`, `OpaquePointer`, `CVaListPointer`: These types provide neither lifetime nor bounds safety. Over time, Swift code is likely to move toward their safe replacements, such as `(Raw)Span`. +* `(Closed)Range.init(uncheckedBounds:)`: This operation makes it possible to create a range that doesn't satisfy invariants on which other bounds safety checking (e.g., in `Array.subscript`) relies. +* `Span.subscript(unchecked:)` : An unchecked subscript whose use can introduce bounds safety problems. +* `Unmanaged`: Wrapper over reference-counted types that explicitly disables reference counting, potentially introducing lifetime safety issues. +* `unsafeBitCast`: Allows type casts that are not known to be safe, which can introduce type safety problems. +* `unsafeDowncast`: An unchecked form of an `as!` cast that can introduce type safety problems. +* `Optional.unsafelyUnwrapped`: An unchecked form of the postfix `!` operation on optionals that can introduce various type, initialization, or lifetime safety problems when `nil` is interpreted as a typed value. +* `UnsafeContinuation`, `withUnsafe(Throwing)Continuation`: An unsafe form of `withChecked(Throwing)Continuation` that does not verify that the continuation is called exactly once, which can cause various safety problems. +* `withUnsafeCurrentTask` and `UnsafeCurrentTask`: The `UnsafeCurrentTask` type does not provide lifetime safety, and must only be used within the closure passed to `withUnsafeCurrentTask`. +* `UnownedSerialExecutor`: This type is intentionally not lifetime safe. It's primary use is the `unownedExecutor` property of the `Actor` protocol, which documents the lifetime assumptions of the `UnownedSerialExecutor` instance it produces. + +All of these APIs will be marked `@unsafe`. For standard library APIs that involve unsafe types, those that are safe to use will be marked `@safe` while those that require the user to maintain some aspect of safety will be marked `@unsafe`. Unless mentioned above, standard library APIs that do not have an unsafe type in their signature, but use unsafe constructs in their implementation, will be considered to be safe. + +There are also a number of unsafe conformances in the standard library: + +* `Unsafe(Mutable)(Raw)BufferPointer`: The conformances of these types to `Sequence` and the `Collection` protocol hierarchy are all `@unsafe`. +* `Unsafe(Mutable)(Raw)Pointer`: The conformances of these types to `Strideable` are `@unsafe`. + +#### Unsafe compiler flags + +There are a number of compiler flags that intentionally disable some safety-related checking. For each of these flags, the compiler will produce a diagnostic if they are used with strict memory safety: + +* `-Ounchecked`, which disables some checking in the standard library, including (for example) bounds checking on array accesses. +* `-enforce-exclusivity=unchecked` and `-enforce-exclusivity=none`, which disables exclusivity checking that is needed for memory safety. +* `-strict-concurrency=` for anything other than "complete", because the memory safety model requires strict concurrency to eliminate thread safety issues. +* `-disable-access-control`, which allows one to break invariants of a type that can lead to memory-safety issues, such as breaking the invariant of `Range` that the lower bound not exceed the upper bound. + +#### C(++) interoperability + +The C family of languages does not provide an equivalent to the strict safety mode described in this proposal, and unlike Swift, the defaults tend to be unsafe along all of the dimensions of memory safety. C(++) libriaries used within Swift can, therefore, introduce memory safety issues into the Swift code. + +The primary issue with memory safety in C(++) concerns the presence of pointers. C(++) pointers will generally be imported into Swift as an `Unsafe*Pointer` type of some form. For C functions (and C++ member functions), that means that a potentially unsafe API such as + +```swift +char *strstr(const char * haystack, const char *needle); +``` + +will be treated as implicitly `@unsafe` in Swift because its signature contains unsafe types: + +```swift +func strstr( + _ haystack: UnsafePointer?, + _ needle: UnsafePointer? +) -> UnsafeMutablePointer? +``` + +A C function that doesn't use pointer types, on the other hand, will implicitly be considered to be safe, because there are no unsafe types in its Swift signature. For example, the following would be considered safe: + +```swift +// int getchar(void); +func getchar() -> CInt +``` + +C and C++ also have user-defined types in the form of `struct`s, `enum`s, `union`s, and (in C++) `class`es. For such types, this proposal infers them to be `@unsafe` when their non-static data contains any C pointers or C types that are explicitly marked as unsafe. For example, a `Point` struct could be considered safe: + +```cpp +struct Point { + double x, y; +}; +``` + +but a `struct` with a pointer or C++ reference in it would be implicitly `@unsafe` in Swift: + +```swift +struct ListNode { + void *element; + struct ListNode *next; +}; +``` + +Note that C `enum`s will never be inferred to be `@unsafe` because they don't carry any values other than their underlying integral type, which is always a safe type. + +### Acknowledging unsafety + +All of the features described above are available in Swift regardless of whether strict memory safety checking is enabled. When strict memory safety checking is enabled, each use of an unsafe construct of any form must be ackwnowledged in the source code with one of the forms below, which provides an in-source auditable indication of where memory unsafety issues can arise. The following section describes each of the features for acknowledging memory unsafety. + +#### `unsafe` expression + +Any time there is executable code that makes use of unsafe constructs, the compiler will produce a diagnostic that indicates the use of those unsafe constructs unless it is within an `unsafe` expression. For example, consider the `DataWrapper` example from an earlier section: + +```swift +public struct DataWrapper { + var buffer: UnsafeBufferPointer +} + +extension DataWrapper { + public func checksum() -> Int32 { + crc32(0, buffer.baseAddress, buffer.count) + } +} +``` + +The property `buffer` uses an unsafe type, `UnsafeBufferPointer`. When using that property in the implementation of `checksum`, the Swift compiler will produce a warning when strict memory safety checking is enabled: + +```swift +warning: expression uses unsafe constructs but is not marked with 'unsafe' +``` + +This warning can be suppressed using the `unsafe` expression, as follows: + +```swift +extension DataWrapper { + public func checksum() -> Int32 { + unsafe crc32(0, buffer.baseAddress, buffer.count) + } +} +``` + +The `unsafe` expression is much like `try` and `await`, in that it acknowledges that unsafe constructs (`buffer`) are used within the subexpression but otherwise does not change the type. Unlike `try` and `await`, which require the enclosing context to handle throwing or be asynchronous, respectively, the `unsafe` expression does not imply any requirements about the enclosing block: it is purely a marker to indicate the presence of unsafe code, silencing a diagnostic. + +#### `@safe` attribute + +The `@safe` attribute is used on declarations whose signatures involve unsafe types but are, nonetheless, safe to use. For example, marking `UnsafeBufferPointer` as `@unsafe` means that all operations involving an unsafe buffer pointer are implicitly considered `@unsafe`. The `@safe` attribute can be used to say that those certain operations are actually safe. For example, any operation involving buffer indices or count are safe, because they don't touch the memory itself. This can be indicated by marking these APIs `@safe`: + +```swift +extension UnsafeBufferPointer { + @safe public let count: Int + @safe public var startIndex: Int { 0 } + @safe public var endIndex: Int { count } +} +``` + +For an array, the `withUnsafeBufferPointer` operation itself also involves the unsafe type that it passes along to the closure. The array itself takes responsibility for the memory safety of the unsafe buffer pointer it vends, ensuring that the elements have been initialized (which is always the case for array elements), that the bounds are correct, and that nobody else has access to the buffer when it is provided. From that perspective, `withUnsafeBufferPointer` itself can be marked `@safe`, and any unsafety will be in the closure's use of the `UnsafeBufferPointer`. + +```swift +extension Array { + @safe func withUnsafeBufferPointer( + _ body: (UnsafeBufferPointer) throws(E) -> R + ) throws(E) -> R +} +``` + +A use of this API with the `c_library_sum_function` would look like this: + +```swift +extension Array { + func sum() -> Int { + withUnsafeBufferPointer { buffer in + unsafe c_library_sum_function(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +The `@safe` annotation on a declaration takes responsibility for any variables of unsafe type that are used as its direct arguments (including the `self`). If such a variable is used to access a `@safe` property or subscript, or in a function call to a `@safe` function, it will not be diagnosed as unsafe: + +```swift +extension Array { + func sum() -> Int { + withUnsafeBufferPointer { buffer in + let count = buffer.count // count is `@safe`, no diagnostic even though 'buffer' has unsafe type + let address = buffer.baseAddress // warning: 'buffer' and 'baseAddress' are both unsafe + c_library_sum_function(address, count, 0) // warning: 'c_library_sum_function' and 'address' are both unsafe + } + } +} +``` + +#### Types with unsafe storage + +Types that wrap unsafe types will often encapsulate the unsafe behavior to provide safe interfaces. However, this requires deliberate design and implementation, potentially involving adding specific preconditions. When strict safety checking is enabled, a type whose storage includes any unsafe types or conformances will be diagnosed as involving unsafe code. For example, the `DataWrapper` struct from the prior section + +```swift +public struct DataWrapper { + var buffer: UnsafeBufferPointer +} +``` + +contains storage of an unsafe type (`UnsafeBufferPointer`), so the Swift compiler will produce a warning when strict memory safety checking is enabled: + +```swift +warning: type `DataWrapper` that includes unsafe storage must be explicitly marked `@unsafe` or `@safe` +``` + +As the warning implies, this diagnostic can be suppressed by marking the type as `@safe` or `@unsafe`. The `DataWrapper` type doesn't appear to provide safety over its storage, so it should likely be marked `@unsafe`. In contrast, one can wrap unsafe types to provide safe types. For example: + +```swift +// @safe is required to suppress a diagnostic about the 'buffer' property's use +// of an unsafe type. +@safe +struct ImmortalBufferWrapper : Collection { + let buffer: UnsafeBufferPointer + + @unsafe init(_ withImmortalBuffer: UnsafeBufferPointer) { + self.buffer = unsafe buffer + } + + subscript(index: Index) -> Element { + precondition(index >= 0 && index < buffer.count) + return unsafe buffer[index] + } + + /* Also: Index, startIndex, endIndex, index(after:) */ +} +``` + +Much of the standard library is built up as safe abstractions over unsafe code, and it's expected that user code will do the same. + +A type has unsafe storage if: + +* Any stored instance property (for `actor`, `class`, and `struct` types) or associated value (for cases of `enum` types) have a type that involves an unsafe type or conformance. +* Any stored instance property uses one of the unsafe language features (such as `unowned(unsafe)`). + +#### Unsafe witnesses + +When a type conforms to a given protocol, it must satisfy all of the requirements of that protocol. Part of this process is determining which declaration (called the *witness*) satisfies a given protocol requirement. If a particular witness is unsafe but the corresponding requirement is safe, the compiler will produce a warning: + +```swift +protocol P { + func f() +} + +struct ConformsToP { } + +extension ConformsToP: P { + @unsafe func f() { } // warning: unsafe instance method 'f()' cannot satisfy safe requirement +} +``` + +This unsafety can be acknowledged by marking the conformance as `@unsafe`, e.g., + +```swift +extension ConformsToP: @unsafe P { + @unsafe func f() { } // okay, it's an unsafe conformance +} +``` + +#### Unsafe overrides + +Overriding a safe method within an `@unsafe` one could introduce unsafety, so it will produce a diagnostic in the strict safety mode: + +```swift +class Super { + func f() { } +} + +class Sub: Super { + @unsafe override func f() { ... } // warning: override of safe instance method with unsafe instance method +} +``` + +to suppress this warning, the `Sub` class itself can be marked as `@unsafe`, e.g., + +```swift +@unsafe +class Sub: Super { + override func f() { ... } // no more warning +} +``` + +The `@unsafe` annotation is at the class level because any use of the `Sub` type can now introduce unsafe behavior, and any indication of that unsafe behavior will be lost once that `Sub` is converted to a `Super` instance. + +#### `for..in` loops + +Swift's `for..in` loops are effectively implemented as syntactic sugar over the `Sequence` and `IteratorProtocol` protocols, where the `for..in` creates a new iterator (with `Sequence.makeIterator()`) and then calls its `next()` operation for each loop iteration. If the conformances to `Sequence` or `IteratorProtocol` are `@unsafe`, the loop will introduce a warning: + +```swift +let someUnsafeBuffer: UnsafeBufferPointer = unsafe ... +for x in someBuffer { // warning: use of unsafe conformance of 'UnsafeBufferPointer' to 'Sequence' + // and someBuffer has unsafe type 'UnsafeBufferPointer' + // ... +} +``` + +Following the precedent set by [SE-0298](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0298-asyncsequence.md), which introduced effects in loops, the `unsafe` keyword that acknowledges unsafe behavior in the iteration will follow the `for`: + +```swift +let someUnsafeBuffer: UnsafeBufferPointer = unsafe ... +for unsafe x in someBuffer { // still warns that someBuffer has unsafe type 'UnsafeBufferPointer' + // ... +} +``` + +This may not be the only unsafe behavior in the `for..in` loop. For example, the expression that produces the sequence itself (via the reference to `someBuffer`) is also unsafe, so it needs to be acknowledged: + +```swift +let someUnsafeBuffer: UnsafeBufferPointer = unsafe ... +for unsafe x in unsafe someBuffer { + // ... +} +``` + +This repeated `unsafe` also occurs with the other effects: if an `async throws` function `getAsyncSequence()` produces an `AsyncSequence` whose iteration can throw, one will end up with two `try` and `await` keywords: + +```swift +for try await x in try await getAsyncSequence() { ... } +``` + +### Strict safety mode and escalatable warnings + +The strict memory safety mode can be enabled with the new compiler flag `-strict-memory-safety`. + +All of the memory-safety diagnostics produced by the strict memory safety mode will be warnings. These warnings be in the group `StrictMemorySafety` (possibly organized into subgroups) so that one can choose to escalate them to errors or keep them as warnings using the compiler flags introduced in [SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md). For example, one can choose to enable the mode and make memory-safety issues errors using: + +``` +swiftc -strict-memory-safety -Werror StrictMemorySafety +``` + +### SwiftPM integration + +Swift package manifests will need a way to enable strict memory safety mode on a per-module and per-package basis. This proposal extends the `SwiftSetting` type in the manifest with a new option to enable that checking: + +```swift +static func strictMemorySafety( + _ condition: BuildSettingCondition? = nil +) -> SwiftSetting +``` + +## Source compatibility + +The `unsafe` keyword in this proposal will be introduced as a contextual keyword following the precedent set by `await`'s' introduction in [SE-0296](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md) and `consume`'s introduction in [SE-0366](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0366-move-function.md). This allows `unsafe` to continue to be used as an identifier, albeit with a small potential to break existing source that uses `unsafe` as a function that is then called with a trailing closure, like this: + +```swift +func unsafe(_ body: () -> Void) { } + +unsafe { + // currently calls 'unsafe(_:)', but will become an unsafe expression +} +``` + +As with those proposals, the impact of this source break in expected to be small enough that it is acceptable. If not, the parsing of the `unsafe` expression can be limited to code that has enabled strict safety checking. + +Other than the source break above, the introduction of this strict safety checking mode has no impact on source compatibility for any module that does not enable it. When enabling strict safety checking, source compatibility impact is limited to the introduction of warnings that will not break source compatibility (and can be treated as warnings even under `-warnings-as-errors` mode using the aforementioned diagnostic flags). The interoperability story is covered in detail in prior sections. + +## ABI compatibility + +The attributes, `unsafe` expression, and strict memory-safety checking model proposed here have no impact on ABI. + +## Future Directions + +### The `SerialExecutor` and `Actor` protocols + +The `SerialExecutor` protocol provides a somewhat unique challenge for the strict memory safety mode. For one, it is impossible to implement this protocol with entirely safe code due to the presence of the `unownedExecutor` requirement: + +```swift +protocol SerialExecutor: Executor { + // ... + @unsafe var unownedExecutor: UnownedSerialExecutor { get } +} +``` + +To make it possible to safely implement `SerialExecutor`, the protocol will need to be extended with a safe form of `unownedExecutor`, which itself will likely require a [non-escapable](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md) form of `UnownedSerialExecutor` to provide lifetime safety without introducing any overhead. The `Actor` protocol has the same `unownedExecutor` requirement, so it will need the corresponding safe variant. The Swift implementation will need to start using this new requirement for scheduling work on actors to eliminate the implicit use of unsafe constructs. + +The `SerialExecutor` protocol has additional semantic constraints involving the serial execution of the jobs provided to the executor. While conformance to any protocol implies that the conforming type meets the documented semantic requirements, `SerialExecutor` is unique in that the data-race safety model (and therefore the memory safety model) depends on it correctly implementing these semantics: a conforming type that currently executes two jobs will create memory-safety violations. There are a few options for addressing this: + +* The `SerialExecutor` protocol itself could be marked `@unsafe`, meaning that any use of this protocol must account for unsafety. +* Some requirements of the `SerialExecutor` protocol (such as the replacement for `unownedExecutor`) could be marked `@unsafe`, so any use of this protocol's requirements must account for unsafety. +* Conformance to the `SerialExecutor` protocol could require some attestation (such as `@safe(unchecked)`) to make it clear from the source code that there is some unsafety encapsulated in the conformance. + +The first two options are the most straightforward, but the fact that actors have implicit uses of `SerialExecutor` means that it would effectively make every actor `@unsafe`. This pushes the responsibility for acknowledging the memory unsafety to clients of `SerialExecutor`, rather than at the conforming type where the responsibility for a correct implementation lies. The third option appears best, because it provides an auditable place to assert memory safety that corresponds with where extra care must be taken to avoid introducing a problem. + +It is unclear whether `SerialExecutor` is or will be the only protocol of this nature. If there are others, it could be worth providing a special form of the `@unsafe` attribute on the protocol itself, such as `@unsafe(conforms)`, that is only considered unsafe for conforming types. + +### Handling of `@unsafe` cases + +When an enum case is explicitly marked `@unsafe`, but involves no associated data that is unsafe, this proposal doesn't have a way to suppress safety diagnostics when pattern matching that case. For example: + +```swift +enum WeirdAddress { + @unsafe case rawOffsetIntoGlobalArray(Int) +} + +func example(_ address: WeirdAddress) { + if case .rawOffsetIntoGlobalArray(let offset) = weirdAddress { // reference to @unsafe case rawOffsetIntoGlobalArray that can't be suppressed + } +} + +``` + +We have several options here: + +* We could suppress the diagnostic for this use of an `@unsafe case`. One would still get diagnostics when constructing such a case. + +* We could reject `@unsafe` on case declarations that don't involve any unsafe types. + +* We could extend the pattern grammar with an `unsafe` pattern to suppress this diagnostic, e.g., + ```swift + if case unsafe .rawOffsetIntoGlobalArray(let offset) = weirdAddress { ... } + ``` + +### Handling unsafe code in macro expansions + +A macro can expand to any code. If the macro-expanded code contains uses of unsafe constructs not properly covered by `@safe`, `@unsafe`, or an `unsafe` expression within the macro, then strict safety checking will diagnose those safety issues within the macro expansion. In this case, the client of the macro does not have any way to suppress diagnostics within the macro expansion itself without modifying the implementation of the macro. + +There are a number of possible approaches that one could use for suppression. The `unsafe` expression could be made to apply to everything in the macro expansion, which would also require some spelling for attached attributes and other places where expressions aren't permitted. Alternatively, Swift could introduce a general syntax for suppressing a class of warnings within a block of code, and that could be used to surround the macro expansion. + +Note that both of these approaches trade away some of the benefits of the strict safety mode for the convenience of suppressing safety-related diagnostics. + +## Alternatives considered + +### Prohibiting unsafe conformances and overrides entirely + +This proposal introduces two places where polymorphism interacts with unsafety: protocol conformances and overrides. In both cases, a safe abstraction (e.g., a superclass or protocol) has a specific implementation that is unsafe, and there is a way to note the unsafety: + +* When overriding a safe declaration with an unsafe one, the overriding subclass must be marked `@unsafe`. +* When implementing a safe protocol requirement with an unsafe declaration, the corresponding conformance must be marked `@unsafe`. + +In both cases, the current proposal will consider uses of the type (in the overriding case) or conformance (for that case) as unsafe, respectively. However, that unsafety is not localized, because code that's generally safe can now cause safety problems when calling through polymorphic operations. For example, consider a function that operates on a general collection: + +```swift +func parse(_ input: some Collection) -> ParseResult +``` + +Calling this function with an unsafe buffer pointer will produce a diagnostic due to the use of the unsafe conformance of `UnsafeBufferPointer` to `Collection`: + +```swift +let result = parse(unsafeBufferPointer) // warning: use of unsafe conformance +``` + +Marking the call as `unsafe` will address the diagnostic. However, because `UnsafeBufferPointer` doesn't perform bounds checking, the `parse` function itself can introduce a memory safety problem if it subscripts into the collection with an invalid index. There isn't a way to communicate how the code that is `unsafe` is addressing memory safety issues within the context of the call. + +This proposal could prohibit use of unsafe conformances and overrides entirely, for example by making it impossible to suppress the diagnostics associated with their definition and use. This would require the `parse(unsafeBufferPointer)` call to be refactored to avoid the unsafe conformance, for example by introducing a wrapper type: + +```swift +@safe struct ImmortalBufferWrapper : Collection { + let buffer: UnsafeBufferPointer + + @unsafe init(_ withImmortalBuffer: UnsafeBufferPointer) { + self.buffer = unsafe buffer + } + + subscript(index: Index) -> Element { + precondition(index >= 0 && index < buffer.count) + return unsafe buffer[index] + } + + /* Also: Index, startIndex, endIndex, index(after:) */ +} +``` + +The call would then look like this: + +```swift +let wrapper = unsafe ImmortalBufferWrapper(withImmortalBuffer: buffer) +let result = parse(wrapper) +``` + +This approach is better than the prior one: it improves bounds safety by introducing bounds checking. It clearly documents the assumptions made around lifetime safety. It is both functionally safer (due to bounds checks) and makes it easier to reason that the `unsafe` is correctly used. It does require a lot more code, and the code itself requires careful reasoning about safety (e.g., the right preconditions for bounds checking; the right naming to capture the lifetime implications). + +Unsafe conformances and overrides remain part of this proposal because prohibiting them doesn't fundamentally change the safety model. Rather, it requires the introduction of more abstractions that could be safer--or could just be boilerplate. Swift has a number of constructs that are functionally similar to unsafe conformances, where safety checking can be disabled locally despite that having wide-ranging consequences: `@unchecked Sendable`, `nonisolated(unsafe)`, `unowned(unsafe)`, and `@preconcurrency` all fall into this category. + +### `@unsafe` implying `unsafe` throughout a function body + +A function marked `@unsafe` is unsafe to use, so any clients that have enabled strict safety checking will need to put uses of the function into an `unsafe` expression. The implementation of that function is likely to use unsafe code (possibly a lot of it), which could result in a large number of annotations: + +```swift +extension UnsafeMutableBufferPointer { + @unsafe public func swapAt(_ i: Index, _ j: Index) { + guard i != j else { return } + precondition(i >= 0 && j >= 0) + precondition(unsafe i < endIndex && j < endIndex) + let pi = unsafe (baseAddress! + i) + let pj = unsafe (baseAddress! + j) + let tmp = unsafe pi.move() + unsafe pi.moveInitialize(from: pj, count: 1) + unsafe pj.initialize(to: tmp) + } +} +``` + +We could choose to make `@unsafe` on a function acknowledge all uses of unsafe code within its definition. For example, this would mean that marking `swapAt` with `@unsafe` means that one need not have any `unsafe` expressions in its body: + +```swift +extension UnsafeMutableBufferPointer { + @unsafe public func swapAt(_ i: Index, _ j: Index) { + guard i != j else { return } + precondition(i >= 0 && j >= 0) + precondition(i < endIndex && j < endIndex) + let pi = (baseAddress! + i) + let pj = (baseAddress! + j) + let tmp = pi.move() + pi.moveInitialize(from: pj, count: 1) + pj.initialize(to: tmp) + } +} +``` + +This approach reduces the annotation burden in unsafe code, but makes it much harder to tell exactly what aspects of the implementation are unsafe. Indeed, even unsafe functions should still strive to minimize the use of unsafe constructs, and benefit from having the actual unsafe behavior marked in the source. It also conflates the notion of "exposes an unsafe interface" from "has an unsafe implementation". + +Rust's `unsafe` functions have this behavior, where an `unsafe fn` in Rust implies an `unsafe { ... }` block around the entire function body. [Rust RFC #2585](https://rust-lang.github.io/rfcs/2585-unsafe-block-in-unsafe-fn.html) argues for Rust to remove this behavior; the motivation there generally applies to Swift as well. + +### Making "encapsulation" of unsafe behavior explicit + +In the proposed design, a function with no unsafe types in its signature is considered safe unless the programmer explicitly marked it `@unsafe`. The implementation may contain any amount of unsafe code, so long as it is covered by an `unsafe` expression: + +```swift +extension Array { + // this function is considered safe + func sum() -> Int { + withUnsafeBufferPointer { buffer in + unsafe sumIntBuffer(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +This differs somewhat from the way in which throwing and asynchronous functions work. A function that has a `try` or `await` in the body needs to be `throws` or `async`, respectively. Essentially, the effect from the body has to also be reflected in the signature. With unsafe code, this could mean that having `unsafe` expressions in the function body requires you to either make the function `@unsafe` or use some other suppression mechanism to acknowledge that you are using unsafe constructs to provide a safe interface. + +There are several options for such a suppression mechanism. An attribute form, `@safe(unchecked)`, is described below as an alternative to the `unsafe` expression. Another approach would be to provide an `unsafe!` form the `unsafe` expression, which (like `try!`) acknowledges the effect but doesn't propagate that effect out to the function. For the `sum` function, it would be used as follows: + +```swift +extension Array { + // this function is considered safe + func sum() -> Int { + withUnsafeBufferPointer { buffer in + unsafe! sumIntBuffer(buffer.baseAddress, buffer.count, 0) + } + } +} +``` + +This proposal chooses not to go down this path, because having a function signature involving no unsafe types is already a strong indication that the function is providing a safe interface, and there is little to be gained from requiring additional ceremony (whether an attribute like `@safe(unchecked)` or the `unsafe!` form described above). + +### `@safe(unchecked)` attribute to allow unsafe code + +Early iterations of this proposal introduced a `@safe(unchecked)` attribute as an alternative to `unsafe` expressions. The `@safe(unchecked)` attribute would be placed on a function to suppress diagnostics about use of unsafe constructs within its definition. This has all of the same downsides as having `@unsafe` imply cover for all of the uses of unsafe code within the body of a function, albeit while providing a safe interface. + +### `unsafe` blocks + +The `unsafe` expression proposed here covers unsafe constructs within a single expression. For unsafe-heavy code, this can introduce a large number of `unsafe` keywords. There is an alternative formulation for acknowledging unsafe code that is used in some peer languages like C# and Rust: an `unsafe` block, which is a statement that suppresses diagnostics about uses of unsafe code within it. For example: + +```swift +extension UnsafeMutableBufferPointer { + @unsafe public func swapAt(_ i: Index, _ j: Index) { + guard i != j else { return } + precondition(i >= 0 && j >= 0) + precondition(i < endIndex && j < endIndex) + unsafe { + let pi = (baseAddress! + i) + let pj = (baseAddress! + j) + let tmp = pi.move() + pi.moveInitialize(from: pj, count: 1) + pj.initialize(to: tmp) + } + } +} +``` + +For Swift, an `unsafe` block would be a statement that can also be used as an expression when its body is an expression, much like `if` or `switch` expressions following [SE-0380](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0380-if-switch-expressions.md). + +`unsafe` blocks are more coarse-grained than the proposed `unsafe` expressions, which represents a trade-off: `unsafe` blocks will be less noisy for unsafe-heavy code, because one `unsafe { ... }` can cover a lot of code. On the other hand, doing so hides which code within the block is actually unsafe, making it harder to audit the unsafe parts. In languages that have `unsafe` blocks, it's considered best practice to make the `unsafe` blocks as narrow as possible. The proposed `unsafe` expressions enforce that best practice at the language level. + +### Strictly-safe-by-default + +This proposal introduced strict safety checking as an opt in mode and not an [*upcoming* language feature](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md) because there is no intent to make this feature the default behavior in a future language mode. There are several reasons why this checking should remain an opt-in feature for the foreseeable future: + +* The various `Unsafe` pointer types are the only way to work with contiguous memory in Swift today, and the safe replacements (e.g., `Span`) are new constructs that will take a long time to propagate through the ecosystem. Some APIs depending on these `Unsafe` pointer types cannot be replaced because it would break existing clients (either source, binary, or both). +* Interoperability with the C family of languages is an important feature for Swift. Most C(++) APIs are unlikely to ever adopt the safety-related attributes described above, which means that enabling strict safety checking by default would undermine the usability of C(++) interoperability. +* Swift's current (non-strict) memory safety by default is likely to be good enough for the vast majority of users of Swift, so the benefit of enabling stricter checking by default is unlikely to be worth the disruption it would cause. + +### Overloading to stage in safe APIs + +When adopting the strict memory safety mode, it's likely that a Swift module will want to replace existing APIs that traffic in unsafe types (such as `UnsafeMutablePointer`) with safer equivalents (such as `Span`). To retain compatibility for older clients, the existing APIs will need to be left in place. Unfortunately, this might mean that the best name for the API is already taken. For example, perhaps we have a data packet that exposes its bytes via a property: + +```swift +public class DataPacket { + @unsafe public let bytes: UnsafeRawBufferPointer +} +``` + +The `bytes` property is necessarily unsafe. Far better would be to produce a `RawSpan`, which we can easily do with another property: + +```swift +extension DataPacket { + public var byteSpan: RawSpan +} +``` + +Clients using the existing `bytes` will continue to work, and those that care about memory safety can choose to move to `byteSpan`. All of this works, but is somewhat annoying because the good name, `bytes`, has been taken for the API we no longer want to use. + +Swift does allow type-based overloading, including on the type of properties, so one could introduce an overloaded `bytes` property, like this: + +```swift +extension DataPacket { + public var bytes: RawSpan +} +``` + +This works for code that accesses `bytes` and then uses it in a context where type inference can figure out whether we need an `UnsafeRawBufferPointer` or a `RawSpan`, but fails if that context does not exist: + +```swift +let unsafeButGoodBytes: UnsafeRawBufferPointer = dataPacket.bytes // ok, uses @unsafe bytes +let goodBytes: RawSpan = dataPacket.bytes // ok, uses safe bytes +let badBytes = dataPacket.bytes // error: ambiguous! +``` + +We could consider extending Swift's overloading rules to make this kind of evolution possible. For example, one could introduce a pair of rules into the language: + +1. When strict memory safety checking is enabled, `@unsafe` declarations are dis-favored vs. safe ones, so the unsafe `bytes: UnsafeRawBufferPointer` would be a worse solution for type inference to pick than the safe alternative, `bytes: RawSpan`. + +2. Overloads that were introduced to replace unsafe declarations could be marked with a new attribute `@safe(unsafeDisfavored)` so that they would be disfavored only when building with strict memory safety checking disabled. + +Assuming these rules, and that the safe `bytes: RawSpan` had the `@safe(unsafeDisfavored)` attribute, the example uses of `DataPacket` would resolve as follows: + +* `unsafeButGoodBytes` would always be initialized with the unsafe `bytes`. If strict memory safety were enabled, this use would produce a warning. +* `goodBytes` would always be initialized with the safe `bytes`. +* `badBytes` would be initialized differently based on whether strict memory safety was enabled: + * If enabled, `badBytes` would choose the safe version of `bytes` to produce the safest code, because the unsafe one is disfavored (rule #1). + * If disabled, `badBytes` would choose the unsafe version of `bytes` to provide source compatibility with existing code, because the safe one is disfavored (rule #2). + +There are downsides to this approach. It partially undermines the source compatibility story for the strict safety mode, because type inference now behaves differently when the mode is enabled. That means, for example, there might be errors---not warnings---because some code like `badBytes` above would change behavior, causing additional failures. Changing the behavior of type inference is also risky in an of itself, because it is not always easy to reason about all of the effects of such a change. That said, the benefit of being able to move toward a more memory-safe future might be worth it. + +### Optional `message` for the `@unsafe` attribute + +We could introduce an optional `message` argument to the `@unsafe` attribute, which would allow programmers to indicate *why* the use of a particular declaration is unsafe and, more importantly, how to safely write code that uses it. However, this argument isn't strictly necessary: a comment could provide the same information, and there is established tooling to expose comments to programmers that wouldn't be present for this attribute's message, so we have omitted this feature. + +## Revision history + +* **Revision 3 (following second review extension)** + * Do not require declarations with unsafe types in their signature to be marked `@unsafe`; it is implied. They may be marked `@safe` to indicate that they are actually safe. + * Add `unsafe` for iteration via the `for..in` syntax. + * Add C(++) interoperability section that infers `@unsafe` for C types that involve pointers. + * Document the unsafe conformances of the `UnsafeBufferPointer` family of types to the `Collection` protocol hierarchy. + * Restructure detailed design to separate out "sources of unsafety" from "acknowledging unsafety". + +* **Revision 2 (following first review extension)** + * Specified that variables of unsafe type passed in to uses of `@safe` declarations (e.g., calls, property accesses) are not diagnosed as themselves being unsafe. This makes means that expressions like `unsafeBufferePointer.count` will be considered safe. + * Require types whose storage involves an unsafe type or conformance to be marked as `@safe` or `@unsafe`, much like other declarations that have unsafe types or conformances in their signature. + * Add an Alternatives Considered section on prohibiting unsafe conformances and overrides. + * Add a Future Directions section on handling unsafe code in macro expansions. + +## Acknowledgments + +This proposal has been greatly improved by the feedback from Félix Cloutier, Geoff Garen, Gábor Horváth, Frederick Kellison-Linn, Karl Wagner, and Xiaodi Wu. diff --git a/proposals/0459-enumerated-collection.md b/proposals/0459-enumerated-collection.md new file mode 100644 index 0000000000..14071906cd --- /dev/null +++ b/proposals/0459-enumerated-collection.md @@ -0,0 +1,199 @@ +# Add `Collection` conformances for `enumerated()` + +* Proposal: [SE-0459](0459-enumerated-collection.md) +* Author: [Alejandro Alonso](https://github.com/Azoy) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#78092](https://github.com/swiftlang/swift/pull/78092) +* Previous Proposal: [SE-0312](0312-indexed-and-enumerated-zip-collections.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-add-collection-conformance-for-enumeratedsequence/76680)) ([review](https://forums.swift.org/t/se-0459-add-collection-conformances-for-enumerated/77509)) ([acceptance](https://forums.swift.org/t/accepted-with-modification-se-0459-add-collection-conformances-for-enumerated/78082)) + +## Introduction + +This proposal aims to fix the lack of `Collection` conformance of the sequence returned by `enumerated()`, preventing it from being used in a context that requires a `Collection`. + +## Motivation + +Currently, `EnumeratedSequence` type conforms to `Sequence`, but not to any of the collection protocols. Adding these conformances was impossible before [SE-0234 Remove `Sequence.SubSequence`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0234-remove-sequence-subsequence.md), and would have been an ABI breaking change before the language allowed `@available` annotations on protocol conformances ([PR](https://github.com/apple/swift/pull/34651)). Now we can add them! + +Conformance to the collection protocols can be beneficial in a variety of ways, for example: +* `(1000..<2000).enumerated().dropFirst(500)` becomes a constant time operation. +* `"abc".enumerated().reversed()` will return a `ReversedCollection` rather than allocating a new array. +* SwiftUI’s `List` and `ForEach` views will be able to directly take an enumerated collection as their data. + +## Detailed design + +Conditionally conform `EnumeratedSequence` to `Collection`, `BidirectionalCollection`, `RandomAccessCollection`. + +```swift +@available(SwiftStdlib 6.1, *) +extension EnumeratedSequence: Collection where Base: Collection { + // ... +} + +@available(SwiftStdlib 6.1, *) +extension EnumeratedSequence: BidirectionalCollection + where Base: BidirectionalCollection +{ + // ... +} + +@available(SwiftStdlib 6.1, *) +extension EnumeratedSequence: RandomAccessCollection + where Base: RandomAccessCollection {} +``` + +## Source compatibility + +All protocol conformances of an existing type to an existing protocol are potentially source breaking because users could have added the exact same conformances themselves. However, given that `EnumeratedSequence` do not expose their underlying sequences, there is no reasonable way anyone could have conformed to `Collection` themselves. + +## Effect on ABI stability + +These conformances are additive to the ABI, but will affect runtime casting mechanisms like `is` and `as`. On ABI stable platforms, the result of these operations will depend on the OS version of said ABI stable platforms. Similarly, APIs like `underestimatedCount` may return a different result depending on if the OS has these conformances or not. + +## Alternatives considered + +#### Add `LazyCollectionProtocol` conformance for `EnumeratedSequence`. + +Adding `LazySequenceProtocol` conformance for `EnumeratedSequence` is a breaking change for code that relies on the `enumerated()` method currently not propagating `LazySequenceProtocol` conformance in a lazy chain: + +```swift +extension Sequence { + func everyOther_v1() -> [Element] { + let x = self.lazy + .enumerated() + .filter { $0.offset.isMultiple(of: 2) } + .map(\.element) + + // error: Cannot convert return expression of type 'LazyMapSequence<...>' to return type '[Self.Element]' + return x + } + + func everyOther_v2() -> [Element] { + // will keep working, the eager overload of `map` is picked + return self.lazy + .enumerated() + .filter { $0.offset.isMultiple(of: 2) } + .map(\.element) + } +} +``` + +We chose to keep this proposal very small to prevent any such potential headaches of source breaks. + +#### Keep `EnumeratedSequence` the way it is and add an `enumerated()` overload to `Collection` that returns a `Zip2Sequence, Self>`. + +This is tempting because `enumerated()` is little more than `zip(0..., self)`, but this would cause an unacceptable amount of source breakage due to the lack of `offset` and `element` tuple labels that `EnumeratedSequence` provides. + +#### Only conform `EnumeratedSequence` to `BidirectionalCollection` when the base collection conforms to `RandomAccessCollection` rather than `BidirectionalCollection`. + +Here’s what the `Collection` conformance could look like: + +```swift +extension EnumeratedSequence: Collection where Base: Collection { + struct Index { + let base: Base.Index + let offset: Int + } + var startIndex: Index { + Index(base: _base.startIndex, offset: 0) + } + var endIndex: Index { + Index(base: _base.endIndex, offset: 0) + } + func index(after index: Index) -> Index { + Index(base: _base.index(after: index.base), offset: index.offset + 1) + } + subscript(index: Index) -> (offset: Int, element: Base.Element) { + (index.offset, _base[index.base]) + } +} + +extension EnumeratedSequence.Index: Comparable { + static func == (lhs: Self, rhs: Self) -> Bool { + return lhs.base == rhs.base + } + static func < (lhs: Self, rhs: Self) -> Bool { + return lhs.base < rhs.base + } +} +``` + +Here’s what the `Bidirectional` conformance could look like. The question is: should `Base` be required to conform to `BidirectionalCollection` or `RandomAccessCollection`? + +```swift +extension EnumeratedSequence: BidirectionalCollection where Base: ??? { + func index(before index: Index) -> Index { + let currentOffset = index.base == _base.endIndex ? _base.count : index.offset + return Index(base: _base.index(before: index.base), offset: currentOffset - 1) + } +} +``` + +Notice that calling `index(before:)` with the end index requires computing the `count` of the base collection. This is an O(1) operation if the base collection is `RandomAccessCollection`, but O(n) if it's `BidirectionalCollection`. + +##### Option 1: `where Base: BidirectionalCollection` + +A direct consequence of `index(before:)` being O(n) when passed the end index is that some operations like `last` are also O(n): + +```swift +extension BidirectionalCollection { + var last: Element? { + isEmpty ? nil : self[index(before: endIndex)] + } +} + +// A bidirectional collection that is not random-access. +let evenNumbers = (0 ... 1_000_000).lazy.filter { $0.isMultiple(of: 2) } +let enumerated = evenNumbers.enumerated() + +// This is still O(1), ... +let endIndex = enumerated.endIndex + +// ...but this is O(n). +let lastElement = enumerated.last! +print(lastElement) // (offset: 500000, element: 1000000) +``` + +However, since this performance pitfall only applies to the end index, iterating over a reversed enumerated collection stays O(n): + +```swift +// A bidirectional collection that is not random-access. +let evenNumbers = (0 ... 1_000_000).lazy.filter { $0.isMultiple(of: 2) } + +// Reaching the last element is O(n), and reaching every other element is another combined O(n). +for (offset, element) in evenNumbers.enumerated().reversed() { + // ... +} +``` + +In other words, this could make some operations unexpectedly O(n), but it’s not likely to make operations unexpectedly O(n²). + +##### Option 2: `where Base: RandomAccessCollection` + +If `EnumeratedSequence`’s conditional conformance to `BidirectionalCollection` is restricted to when `Base: RandomAccessCollection`, then operations like `last` and `last(where:)` will only be available when they’re guaranteed to be O(1): + +```swift +// A bidirectional collection that is not random-access. +let str = "Hello" + +let lastElement = str.enumerated().last! // error: value of type 'EnumeratedSequence' has no member 'last' +``` + +That said, some algorithms that can benefit from bidirectionality such as `reversed()` and `suffix(_:)` are also available on regular collections, but with a less efficient implementation. That means that the code would still compile if the enumerated sequence is not bidirectional, it would just perform worse — the most general version of `reversed()` on `Sequence` allocates an array and adds every element to that array before reversing it: + +```swift +// A bidirectional collection that is not random-access. +let str = "Hello" + +// This no longer conforms to `BidirectionalCollection`. +let enumerated = str.enumerated() + +// As a result, this now returns a `[(offset: Int, element: Character)]` instead +// of a more efficient `ReversedCollection>`. +let reversedElements = enumerated.reversed() +``` + +The base collection needs to be traversed twice either way, but the defensive approach of giving the `BidirectionalCollection` conformance a stricter bound ultimately results in an extra allocation. + +Taking all of this into account, we've gone with option 1 for the sake of giving collections access to more algorithms and more efficient overloads of some algorithms. Conforming this collection to `BidirectionalCollection` when the base collection conforms to the same protocol is less surprising. We don’t think the possible performance pitfalls pose a large enough risk in practice to negate these benefits. diff --git a/proposals/0460-specialized.md b/proposals/0460-specialized.md new file mode 100644 index 0000000000..8c5ead4bd2 --- /dev/null +++ b/proposals/0460-specialized.md @@ -0,0 +1,263 @@ +# Explicit Specialization + +* Proposal: [SE-0460](0460-specialized.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Implemented (Swift Next)** +* Review: ([pitch](https://forums.swift.org/t/pitch-explicit-specialization/76967)) ([review](https://forums.swift.org/t/se-0460-explicit-specialization/77541)) ([acceptance](https://forums.swift.org/t/accepted-se-0460-explicit-specialization/78583)) + +## Introduction + +The Swift compiler has the ability to "specialize" a generic function at compile time. This specialization creates a custom implementation of the function, where the generic placeholders are substituted with specific types. This can unlock optimizations of that specialized function that can be dramatically faster than the unspecialized version in some circumstances. The compiler can generate this specialized version and call it when it can see at the call site with what concrete types a function is being called, and the body of the function to specialize. + +In some cases, though, this information is obscured from the compiler. This proposal introduces a new attribute, `@specialized`, which allows the author of a generic function to generate pre-specialized versions of that function for specific types. When the unspecialized version of the function is called with one of those types, the compiler will generate code that will re-dispatch to those prespecialized versions if available. + +## Motivation + +Consider the following generic function that sums an array of any binary integer type: + +```swift +extension Sequence where Element: BinaryInteger { + func sum() -> Double { + reduce(0) { $0 + Double($1) } + } +} +``` + +If you call this function directly on an array of a specific integer type (e.g. `Array`): + +``` +let arrayOfInt: [Int] = // ... +let result = arrayOfInt.sum() +``` + +in optimized builds, the Swift compiler will generate a specialized version of `sum` for that type. + +If you inspect the binary, you will see this specialized version under a symbol `_$sST3sumSz7ElementRpzrlEAASdyFSaySiG_Tg5`, which demangles to `generic specialization <[Swift.Int]> of (extension in example):Swift.Sequence< where A.Element: Swift.BinaryInteger>.sum() -> Swift.Double`, alongside the unspecialized version. + +This specialized version of `sum` will be optimized specifically for `Array`. It can move a pointer directly over the the array's buffer, loading the elements into registers directly from memory. On modern hardware, it can make use of dedicated instructions to convert the integers to floating point. It can do this because it can see the implementation of `sum`, and can see the exact types on which it is being called. What exact assembly instructions are generated will differ significantly between e.g. `Array.sum` and `Array.sum`. + +Here is that `Array`-specialized code when compiled to x86-64 with `swiftc -Osize`: + +```assembly +_$sST3sumSz7ElementRpzrlEAASdyFSaySiG_Tg5: + mov rax, qword ptr [rdi + 16] + xorpd xmm0, xmm0 + test rax, rax + je .LBB1_3 + xor ecx, ecx +.LBB1_2: + cvtsi2sd xmm1, qword ptr [rdi + 8*rcx + 32] + inc rcx + addsd xmm0, xmm1 + cmp rax, rcx + jne .LBB1_2 +.LBB1_3: + ret +``` + +Now consider some code that places an optimization barrier between the call site and the concrete type: + +```swift +protocol Summable: Sequence where Element: BinaryInteger { } +extension Array: Summable where Element: BinaryInteger { } + +var summable: any Summable + +// later, when summable has been populated with an array of some kind of integer +let result = summable.sum() +``` + +The compiler now has no way of knowing what type `Summable` is at the call site – neither the sequence type, nor the element type. So its only option is to execute the fully unspecialized code. Instead of advancing a pointer over a buffer and loading values directly from memory, it must iterate the sequence by first calling `makeIterator()` and then calling `next()`, unwrapping optional values until it reaches `nil`. And instead of using a single instruction to convert an `Int` to a `Double`, it must call the generic initializer on `Double` that uses methods on the `BinaryInteger` protocol to convert the value into floating point representation. Unsurprisingly, this code path will be about 2 orders of magnitude slower than the specialized version. This is true even if the actual type being summed ends up being `[Int]` at runtime. + +A similar situation would occur when `sum` is a generic function in a binary framework which has not provided an `@inlinable` implementation. While the Swift compiler might know the types at the call site, it has no ability to generate a specialized version because it cannot see the implementation in order to generate it. So it must call the unspecialized version of `sum`. + +The best way to avoid this situation is just to avoid type erasure in the first place. This is why concrete types and generics should be preferred whenever practical over existential types for performance-sensitive code. But sometimes type erasure, particularly for heterogenous storage, is necessary. + +Similarly, ABI-stable binary framework authors do not always want to expose their implementations to the caller, as this tends to generate a very large and complex ABI surface that cannot be easily changed later. It also prevents upgrades of binary implementations without recompiling the caller, allowing for e.g. an operating system to fix bugs or security vulnerabilities without an app needing to be recompiled. + +## Proposed solution + +A new attribute, `@specialized`, will allow the author of a function to cause the compiler to generate specializations of that function. In the body of the unspecialized version, the types are first checked to see if they are of one of the specialized types. If they are, the specialized version will be called. + +So in our example above: + +```swift +extension Sequence where Element: BinaryInteger { + @specialized(where Self == [Int]) + func sum() -> Double { + reduce(0) { $0 + Double($1) } + } +} +``` + +A specialized version of `[Int].sum` will be generated in the same way as if it had been specialized for a callsite. And inside the unspecialized generic code, the additional check-and-redispatch logic will be inserted at the start of the function. + +Doing this restores the performance of the specialized version of `sum` (less a check and branch of the calling type) even when using the existential `any Summable` type. + +## Detailed design + +The `@specialized` attribute can be placed on any generic function. It takes an argument with the same syntax as a `where` clause for a generic signature. + +Multiple specializations can be listed: + +```swift +extension Sequence where Element: BinaryInteger { + @specialized(where Self == [Int]) + @specialized(where Self == [UInt32]) + @specialized(where Self == [Int8]) + func sum() -> Double { + reduce(0) { $0 + Double($1) } + } +} +``` + +Within the unspecialized function, redispatch will be based on the _exact_ type. That is, in pseudocode: + +```swift +extension Sequence where Element: BinaryInteger { + @specialized(where Self == [Int]) + @specialized(where Self == [Int8]) + func sum() -> Double { + if Self.self == [Int].self { + + } else if Self.self == [Int8].self { + + } else { + reduce(0) { $0 + Double($1) } + } + } +} +``` + +(Note that this is just for illustrative purposes, the actual dispatch mechanism to the specific specialization may differ and change over time) + +No attempt to implicitly convert the type is made (so for example, a specialization for `Int?` would not be executed when called on `Int`), nor is a specialization for a superclass executed when called with the generic type of the subclass. + +These specializations are the same as the ones generated by the caller. They replace the dynamic dispatch of the unspecialized generic protocol with static dispatch to the methods on the concrete type, and this in turn unlocks other optimizations within the specialized function. Note that within these specializations, no different overload resolution takes place now that the concrete type is known. That is, the functions being called will be only those made available via the protocol witness table. This ensures that there is no _semantic_ effect from using `@specialized`, only a change in performance. + +As well as protocol extensions, it can also be used on extensions of generic types, on computed properties (note, it must be put on `get` explicitly, not on the shorthand where it is elided), and on free functions: + +```swift +extension Array where Element: BinaryInteger { + @specialized(where Element == Int) + func sum() -> Double { + reduce(0) { $0 + Double($1) } + } + + var product: Double { + @specialized(where Element == Int8) + get { reduce(1) { $0 * Double($1) } } + } +} + +@specialized(where T == Int) +func sum(_ numbers: T...) -> Double { + numbers.reduce(0) { $0 + Double($1) } +} +``` + +The `where` clause must fully specialize all the generic placeholders of the types in the function signature. In the case of a protocol extension, as seen above, that includes specifying `Self`. All placeholders must be fully specified even if they do not appear to be used in the function body: + +```swift +extension Dictionary where Value: BinaryInteger { + // error: Too few generic parameters are specified in 'specialize' attribute (got 1, but expected 2) + // note: Missing equality constraint for 'Key' in 'specialize' attribute + @specialized(where Value == Int) + func sum() -> Double { + values.reduce(0) { $0 + Double($1) } + } +} +``` + +Bear in mind that even when not explicitly used in the body, they may be used implicitly. For example, in calculating the location of stored properties. Depending on how `Dictionary` is laid out, it may be important to know the size of the `Key` even if key values aren't used. But even if there is absolutely no use of a generic type in the body being specialized, the type must be explicitly specified. This requirement could be loosened in future, see "Partial Specialization" in future directions. + +Where multiple placeholders need to be specified separately, they can be separated by commas, such as in the fix for the above example: + +```swift +extension Dictionary where Value: BinaryInteger { + @specialized(where Value == Int, Key == Int) + func sum() -> Double { + values.reduce(0) { $0 + Double($1) } + } +} +``` + +## Source compatibility + +The addition or removal of explicit specializations has no impact on the caller, and is opt-in. As such it has no source compatability implications. + +## ABI compatibility + +This proposal covers only _internal_ specializations, dispatched to from within the implementation of an unspecialized generic function. As such it has no impact on ABI. It can be applied to existing ABI-stable functions in libraries built for distribution, and can be removed later without ABI impact. Generic functions can continue to be inlinable to be specialized by the caller, and the code to dispatch to specialized versions will not appear in the inlinable code emitted into the swift interface file. + +A future direction where explicit specializations are exposed as ABI appears in Future Directions. + +## Implications on adoption + +Explicit specializations impose no new runtime requirements on either the caller or the called function, so use of them can be back-deployed to earlier Swift runtimes. + +## Future directions + +### Partial Specialization + +The need to fully specify every type when using `@specialized` is somewhat limiting. For example, in the `Dictionary` example above, no reference is made in the code to the `Key` type in summing the values. Having to explicitly state the concrete type for `Key` means you need separate specializations for `[String:Int]`, `[Int:Int]` and so on. + +Even in cases where dynamic dispatch through the protocol is still required for the unspecialized aspects (such as determining where to find the values in a dictionary's layout), this might not be in the hot part of the function and so partial specialization might be the better trade-off. + +Closely related to partial specialization is the ability to specialize once for all particular type layouts. In the `Dictionary.sum` example, it might only matter what the size of the key type is in order to efficiently iterate the values. + +Another example of this is `Array.append`. There is only one single implementation needed for appending an `AnyObject`-constrained to an array. The append operation needs to retain the object, but does not need any other properties. So two class instances could be appended using the same method irrespective of their actual type. Similar techniques could be used to provide shared specializations for `BitwiseCopyable` types, possibly with the addition of a supplied size argument to avoid having to use value witness calls. + +### Making Symbols for Specializations Publicly Available + +This proposal keeps entry points for explicit specializations internal only. Callers outside the module call the generic version of the function, which redispatches to the specialized version. For ABI-stable frameworks, a useful future direction would be to make these symbols publicly available and listed in the `.swiftinterface` file, for callers to link to directly. + +Doing this allows ABI-stable framework authors to expose specializations without exposing the full implementation details of a function as inlinable code. While adding a public specialized symbol to a framework is ABI, it is a much more limited ABI surface compared to providing an inlinable implementation, which requires any future changes to a type to consider the previous inlinable code's behavior to ensure it remains compatible forever in older binaries. A specialized entry point could be updated in a framework to fix a bug without recompiling the caller. Specializations in binary frameworks also have the benefit of avoiding duplication of code into the caller. + +### Requiring Specialization + +A related feature is the ability to _require_ specialization either at the call site, or on a protocol. Specialization does not always have to happen at the call site even when it could – it remains an optimization (albeit a very aggressively applied one currently). If the specializatino does not happen, it would be useful to force the compiler to override its heuristics, similar to forcing linlining of a long but critical function. + +There are also protocols that are only meant to be used in specialized form in optimized builds. Arguably `BinaryInteger` is one of them. It may be worth exploring in these cases an annotation to indicate this to the caller, either via a compile-time errror/warning, or a runtime error. + +### Marking types or extensions as `@specialized` + +It may desirable to mark a number of generic functions "en-mass" as specialized for a particular type, by annotating either the type or an extension grouping. + +This would mostly be just sugar for annotating each function individually. The exception could be annotation of classes or protocols where an entire specialized witness or vtable could then be passed around and used from other unspecialized functions. + +### Tooling Directions + +This proposal only outlines language syntax the developer can use to instruct the compiler to emit a specialized version of a function. Complimentary to this is development of tooling to identify profitable specializations to add, for example by profiling typical usage of an app. It should be noted that specialization is only required in highly performance sensitive code. In many cases, explicit specialization will have little impact on overall performance while increasing binary size. + +The current implementation requires the attribute to be attached directly to the function being specialized. Tooling to produce specializations would benefit from an additional syntax that could be added in a separate file, or even into a separately compiled binary. + +## Alternatives considered + +Many alternatives to this proposal – such as whole-program analysis to determine prespecializations automatically – are complimentary to this technique. Even in the presence of better optimizer heroics, there are still benefits to having explicit control over specialization. + +This proposal takes as a given Swift's current dispatch mechanisms. Some alternatives to this proposal tend to end up requiring fundamental changes to Swift's core generics implementation, and are therefore out of scope for this proposal. + +### Execution of custom functions based on type + +Sometimes, it is desirable to execute not a compiler-generated specialization of a function, but a very different implementation based on the type: + +``` +extension Sequence where Element: BinaryInteger { + func sum() -> Double { + if let arrayOfInt = self as? [Int] { + arrayOfInt.handVectorizedImplementation() + } else { + reduce(0) { $0 + Double($1) } + } + } +} +``` + +The specializations created by this proposal are entirely referentially transparent, using Swift's protocol dispatch semantics to ensure the only (reasonably )observable difference is how quickly the code runs. They are just optimizations where dynamic dispatch is replaced by static, code inlined, low-level optimizations applied to that code etc. + +This is distinct from "when it's this type run this code, when it's that type, run that code", where this code and that code might do very different things semantically. You might not mean for them to differ semantically, but they can. + +This is an interesting area to explore, but it's important to be clear that it's a very different feature (and hence not included in future directions). + diff --git a/proposals/0461-async-function-isolation.md b/proposals/0461-async-function-isolation.md new file mode 100644 index 0000000000..47633dda33 --- /dev/null +++ b/proposals/0461-async-function-isolation.md @@ -0,0 +1,1137 @@ +# Run nonisolated async functions on the caller's actor by default + +* Proposal: [SE-0461](0461-async-function-isolation.md) +* Authors: [Holly Borla](https://github.com/hborla), [John McCall](https://github.com/rjmccall) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.2)** +* Vision: [Improving the approachability of data-race safety](/visions/approachable-concurrency.md) +* Upcoming Feature Flag: `NonisolatedNonsendingByDefault` +* Previous Proposal: [SE-0338](0338-clarify-execution-non-actor-async.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-inherit-isolation-by-default-for-async-functions/74862)) ([first review](https://forums.swift.org/t/se-0461-run-nonisolated-async-functions-on-the-callers-actor-by-default/77987)) ([acceptance with focused re-review](https://forums.swift.org/t/accepted-with-modifications-and-focused-re-review-se-0461-run-nonisolated-async-functions-on-the-callers-actor-by-default/78920)) ([second review](https://forums.swift.org/t/focused-re-review-se-0461-run-nonisolated-async-functions-on-the-callers-actor-by-default/78921)) ([second acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0461-run-nonisolated-async-functions-on-the-caller-s-actor-by-default/79117)) + +## Introduction + +Swift's general philosophy is to prioritize safety and ease-of-use over +performance, while still providing tools to write more efficient code. The +current behavior of nonisolated async functions prioritizes main actor +responsiveness at the expense of usability. + +This proposal changes the behavior of nonisolated async functions to run on +the caller's actor by default, and introduces an explicit way to state that an +async function always switches off of an actor to run. + +## Table of Contents + +- [Motivation](#motivation) +- [Proposed solution](#proposed-solution) +- [Detailed design](#detailed-design) + - [`nonisolated(nonsending)` functions](#nonisolatednonsending-functions) + - [`@concurrent` functions](#concurrent-functions) + - [Task isolation inheritance](#task-isolation-inheritance) + - [`#isolation` macro expansion](#isolation-macro-expansion) + - [Isolation inference for closures](#isolation-inference-for-closures) + - [Function conversions](#function-conversions) + - [Non-`@Sendable` function conversions](#non-sendable-function-conversions) + - [Region isolation rules](#region-isolation-rules) + - [Executor switching](#executor-switching) + - [Dynamic actor isolation APIs in async contexts](#dynamic-actor-isolation-apis-in-async-contexts) + - [Import-as-async heuristic](#import-as-async-heuristic) +- [Source compatibility](#source-compatibility) +- [ABI compatibility](#abi-compatibility) +- [Implications on adoption](#implications-on-adoption) +- [Alternatives considered](#alternatives-considered) + - [Changing isolation inference behavior to implicitly capture isolated parameters](#changing-isolation-inference-behavior-to-implicitly-capture-isolated-parameters) + - [Use `nonisolated` instead of a separate `@concurrent` attribute](#use-nonisolated-instead-of-a-separate-concurrent-attribute) + - [Alternative syntax choices](#alternative-syntax-choices) + - [No explicit spelling for `nonisolated(nonsending)`](#no-explicit-spelling-for-nonisolatednonsending) + - [Justification for `@concurrent`](#justification-for-concurrent) + - [`@executor`](#executor) + - [`@isolated`](#isolated) + - [`nonisolated` argument spelling](#nonisolated-argument-spelling) + - [Deprecate `nonisolated`](#deprecate-nonisolated) + - [Don't introduce a type attribute for `@concurrent`](#dont-introduce-a-type-attribute-for-concurrent) +- [Revisions](#revisions) + +## Motivation + +[SE-0338: Clarify the Execution of Non-Actor-Isolated Async Functions][SE-0338] +specifies that nonisolated async functions never run on an actor's executor. +This design decision was made to prevent unnecessary serialization and +contention for the actor by switching off of the actor to run the nonisolated +async function, and any new tasks it creates that inherit isolation. The actor +is then free to make forward progress on other work. This behavior is +especially important for preventing unexpected overhang on the main actor. + +This decision has a number of unfortunate consequences. + +**`nonisolated` is difficult to understand.** There is a semantic difference +between the isolation behavior of nonisolated synchronous and asynchronous +functions; nonisolated synchronous functions always run on the caller's actor, +while nonisolated async functions always switch off of the caller's actor. This +means that sendable checking applies to arguments and results of nonisolated +async functions, but not nonisolated synchronous functions. + +For example: + +```swift +class NotSendable { + func performSync() { ... } + func performAsync() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + x.performSync() // okay + + await x.performAsync() // error + } +} +``` + +The call to `performAsync` from the actor results in a data-race safety error +because the call leaves the actor to run the function. This frees up the actor +to run other tasks, but those tasks can access the non-`Sendable` value `x` +concurrently with the call to `performAsync`, which risks a data race. + +It's confusing that the two calls to methods on `NotSendable` have different +isolation behavior, because both methods are `nonisolated`. + +**Async functions that run on the caller's actor are difficult to express.** +It's possible to write an async function that does not leave an actor to run +using isolated parameters and the `#isolation` macro as a default argument: + +```swift +class NotSendable { + func performAsync( + isolation: isolated (any Actor)? = #isolation + ) async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + await x.performAsync() // okay + } +} +``` + +This resolves the data-race safety error because `performAsync` now runs on the +actor. However, this isn't an obvious solution, it's onerous boilerplate to +write, and the default argument is lost if the method is used in a higher-order +manner. + +**It's easy to write invalid async APIs.** If the `performAsync` method were in +a library that the programmer doesn't own, it's not possible to workaround the +data-race safety error without using unsafe opt outs. It's common for library +authors to mistakenly vend an API like this, because the data-race safety error +only manifests when calling the API from an actor. + +The concurrency library itself has made this mistake, and many of the async +APIs in the concurrency library have since transitioned to inheriting the +isolation of the caller using isolated parameters; see [SE-0421][SE-0421] for +an example. + +**It's difficult to write higher-order async APIs.** Consider the following +async API which provides a `with`-style method for acquiring a resource and +performing a scoped operation: + +```swift +public struct Resource { + internal init() {} + internal mutating func close() async {} +} + +public func withResource( + isolation: isolated (any Actor)? = #isolation, + _ body: (inout Resource) async -> Return +) async -> Return { + var resource = Resource() + let result = await body(&resource) + await resource.close() + return result +} +``` + +Despite `withResource` explicitly running on the caller's actor by default, +there's no way to specify that the async `body` function value should also run +in the same context. The compiler treats the async function parameter as +switching off of the actor to run, so it requires sendable checking on the +arguments and results. This particular example happens to pass a value in a +disconnected region to `body`, but passing an argument in the actor's region +would be invalid. In most cases, the call doesn't cross an isolation boundary +at runtime, because the function type is not `@Sendable`, so calling the API +from an actor-isolated context and passing a trailing closure will treat the +closure as isolated to the same actor. This sendable checking is often a source +of false positives that make higher-order async APIs extremely difficult to +write. The checking can't just be eliminated, because it's valid to pass a +nonisolated async function that will switch off the actor to run, which would +lead to a data race if actor-isolated state is passed to the `body` parameter. + +Moreover, the above explanation of isolation rules for async closures is +extremely difficult to understand; the default isolation rules are too +complicated. + +## Proposed solution + +This proposal changes the execution semantics of nonisolated async functions +to always run on the caller's actor by default. This means that nonisolated +functions will have consistent execution semantics by default, regardless of +whether the function is synchronous or asynchronous. + +This change makes the following example from the motivation section +valid, because the call to `x.performAsync()` does not cross an isolation +boundary: + +```swift +class NotSendable { + func performSync() { ... } + func performAsync() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + x.performSync() // okay + + await x.performAsync() // okay + } +} +``` + +Changing the default execution semantics of async functions can change the +behavior of existing code, so the change is gated behind the +`NonisolatedNonsendingByDefault` upcoming feature flag. To help stage in the new +behavior, new syntax can be used to explicitly specify the +execution semantics of an async function in any language mode. + +A new `nonsending` argument can be written with `nonisolated` to indicate +that by default, the argument and result values are not sent over an +isolation boundary when the function is called: + +```swift +class NotSendable { + nonisolated(nonsending) + func performAsync() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + await x.performAsync() // okay + } +} +``` + +The `@concurrent` attribute is an explicit spelling for the behavior of +async functions in language modes <= Swift 6. `@concurrent` indicates +that calling the function always switches off of an actor to run, so +the function will run concurrently with other tasks on the caller's actor: + +```swift +class NotSendable { + @concurrent + func alwaysSwitch() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + await x.alwaysSwitch() // error + } +} +``` + +`@concurrent` is the current default for nonisolated async +functions. `nonisolated(nonsending)` will become the default for async functions +when the `NonisolatedNonsendingByDefault` upcoming feature is enabled. + +## Detailed design + +The sections below will explicitly use `@concurrent` and +`nonisolated(nonsending)` to demonstrate examples that will behave consistently +independent of upcoming features or language modes. However, note that the +end state under the `NonisolatedNonsendingByDefault` upcoming feature will mean +that `(nonsending)` is not necessary to explicitly write, and +`@concurrent` will likely be used sparingly because it has far +stricter data-race safety requirements. + +### `nonisolated(nonsending)` functions + +Async functions annotated with `nonisolated(nonsending)` will always run on the +caller's actor: + +```swift +class NotSendable { + func performSync() { ... } + + nonisolated(nonsending) + func performAsync() async { ... } +} + +actor MyActor { + let x: NotSendable + + func call() async { + x.performSync() // okay + + await x.performAsync() // okay + } +} +``` + +In the above code, the call to `x.performAsync()` continues running on the +`self` actor instance. The code does not produce a data-race safety error, +because the `NotSendable` instance `x` does not leave the actor. In other +words, the arguments are not sent across an isolation boundary when calling +`performAsync` by default. + +This behavior is accomplished by implicitly passing an optional actor parameter +to the async function. The function will run on this actor's executor. See the +[Executor switching](#executor-switching) section for more details on why the +actor parameter is necessary. + +The type of an `nonisolated(nonsending)` function declaration is an +`nonisolated(nonsending)` function type. For example: + +```swift +class NotSendable { ... } + +@MainActor let global: NotSendable = .init() + +nonisolated(nonsending) +func runOnActor(ns: NotSendable) async {} + +@MainActor +func callSendableClosure() async { + // the type of 'closure' is '@Sendable nonisolated(nonsending) (NotSendable) -> Void' + let closure = runOnActor(ns:) + + let ns = NotSendable() + await closure(ns) // okay + await closure(global) // okay +} + +callSendableClosure() +``` + +In the above code, the calls to `closure` from `callSendableClosure` run on the +main actor, because `closure` is `nonisolated(nonsending)` and `callSendableClosure` +is main actor isolated. + +### `@concurrent` functions + +Async functions can be declared to always switch off of an actor to run using +the `@concurrent` attribute: + +```swift +struct S: Sendable { + @concurrent + func alwaysSwitch() async { ... } +} +``` + +Only (implicitly or explicitly) `nonisolated` functions can be marked with +`@concurrent`; it is an error to use the these attributes with +an isolation other than `nonisolated`, including global actors, isolated +parameters, and `@isolated(any)`: + +```swift +actor MyActor { + var value = 0 + + // error: '@concurrent' can only be used with 'nonisolated' methods + @concurrent + func isolatedToSelf() async { + value += 1 + } + + @concurrent + nonisolated func canRunAnywhere() async { + // cannot access 'value' or other actor-isolated state + } +} +``` + +`@concurrent` can be used together with `@Sendable` or `sending`. + +`@concurrent` cannot be applied to synchronous +functions. This is an artificial limitation that could later be lifted if use +cases arise. + +The type of an `@concurrent` function declaration is an +`@concurrent` function type. Details on function conversions are +covered in a [later section](#function-conversions). + +When an `@concurrent` function is called from a context that can +run on an actor, including `nonisolated(nonsending)` functions or actor-isolated +functions, sendable checking is performed on the argument and result values. +Either the argument and result values must have a type that conforms to +`Sendable`, or the values must be in a disconnected region so they can be sent +outside of the actor: + +```swift +class NotSendable {} + +@concurrent +func alwaysSwitch(ns: NotSendable) async { ... } + +actor MyActor { + let ns: NotSendable = .init() + + func callConcurrent() async { + await alwaysSwitch(ns: ns) // error + + let disconnected = NotSendable() + await alwaysSwitch(ns: disconnected) // okay + } +} +``` + +### Task isolation inheritance + +Unstructured tasks created in nonisolated functions never run on an actor +unless explicitly specified. This behavior is consistent for all nonisolated +functions, including synchronous functions, `nonisolated(nonsending)` async +functions, and `@concurrent` async functions. + +For example: + +```swift +class NotSendable { + var value = 0 +} + +nonisolated(nonsending) +func createTask(ns: NotSendable) async { + Task { + // This task does not run on the same actor as `createTask` + + ns.value += 1 // error + } +} +``` + +Capturing `ns` in the unstructured task is an error, because the value can +be used concurrently between the caller of `createTask` and the newly +created task. + +This decision is deliberate to match the semantics of unstructured task +creation in nonisolated synchronous functions. Note that unstructured task +creation in methods with isolated parameters also do not inherit isolation +if the isolated parameter is not explicitly captured. + +### `#isolation` macro expansion + +Uses of the `#isolation` macro will expand to the implicit isolated parameter. +For example, the following program prints `Optional(Swift.MainActor)`: + +```swift +nonisolated func printIsolation() async { + let isolation = #isolation + print(isolation) +} + +@main +struct Program { + // implicitly isolated to @MainActor + static func main() async throws { + await printIsolation() + } +} +``` + +This behavior allows async function calls that use `#isolation` as a default +isolated argument to run on the same actor when called from an +`nonisolated(nonsending)` function. For example, the following code is valid because +the call to `explicitIsolationInheritance` does not cross an isolation +boundary: + +```swift +class NotSendable { ... } + +func explicitIsolationInheritance( + ns: NotSendable, + isolation: isolated (any Actor)? = #isolation +) async { ... } + +nonisolated(nonsending) +func printIsolation(ns: NotSendable) async { + await explicitIsolationInheritance(ns: ns) // okay +} +``` + +Note that this introduces a semantic difference compared to synchronous +nonisolated functions, where there is no implicit isolated parameter and +`#isolation` always expands to `nil`. For example, the following program prints +`nil`: + +```swift +func printIsolation() { + let isolation = #isolation + print(isolation) +} + +@main +struct Program { + // implicitly isolated to @MainActor + static func main() async throws { + printIsolation() + } +} +``` + +In an `@concurrent` function, the `#isolation` macro expands to +`nil`. + +### Isolation inference for closures + +Note that the rules in this section are not new with this proposal. However, +these rules have not been specified in any other proposal, and they are +necessary for understanding the execution semantics of async closures. + +The isolation of a closure can be explicitly specified with a type annotation +or in the closure signature. If no isolation is specified, the inferred +isolation for a closure depends on two factors: +1. The isolation of the context where the closure is formed. +2. Whether the contextual type of the closure is `@Sendable` or `sending`. + +If the contextual type of the closure is neither `@Sendable` nor `sending`, the +inferred isolation of the closure is the same as the enclosing context: + +```swift +class NotSendable { ... } + +@MainActor +func closureOnMain(ns: NotSendable) async { + let syncClosure: () -> Void = { + // inferred to be @MainActor-isolated + + // capturing main-actor state is okay + print(ns) + } + + // runs on the main actor + syncClosure() + + let asyncClosure: (NotSendable) async -> Void = { + // inferred to be @MainActor-isolated + + print($0) + } + + // runs on the main actor; + // passing main-actor state is okay + await asyncClosure(ns) +} +``` + +If the type of the closure is `@Sendable` or if the closure is passed to a +`sending` parameter, the closure is inferred to be `nonisolated`. + +The closure is also inferred to be `nonisolated` if the enclosing context +has an isolated parameter (including `self` in actor-isolated methods), and +the closure does not explicitly capture the isolated parameter. This is done to +avoid implicitly capturing values that are invisible to the programmer, because +this can lead to reference cycles. + +### Function conversions + +Function conversions can change isolation. You can think of this like a +closure with the new isolation that calls the original function, asynchronously +if necessary. For example, a function conversion from one global-actor-isolated +type to another can be conceptualized as an async closure that calls the +original function with `await`: + +```swift +@globalActor actor OtherActor { ... } + +func convert( + closure: @OtherActor () -> Void +) { + let mainActorFn: @MainActor () async -> Void = closure + + // The above conversion is the same as: + + let mainActorEquivalent: @MainActor () async -> Void = { + await closure() + } +} +``` + +A function conversion that crosses an isolation boundary must only +pass argument and result values that are `Sendable`; this is checked +at the point of the function conversion. For example, converting an +actor-isolated function type to a `nonisolated` function type requires +that the argument and result types conform to `Sendable`: + +```swift +class NotSendable {} +actor MyActor { + var ns = NotSendable() + + func getState() -> NotSendable { ns } +} + +func invalidResult(a: MyActor) async -> NotSendable { + let grabActorState: nonisolated(nonsending) () async -> NotSendable = a.getState // error + + return await grabActorState() +} +``` + +In the above code, the conversion from the actor-isolated method `getState` +to a `nonisolated(nonsending)` function is invalid, because the +result type does not conform to `Sendable` and the result value could be +actor-isolated state. The `nonisolated` function can be called from +anywhere, which would allow access to actor state from outside the actor. + +Not all function conversions cross an isolation boundary, and function +conversions that don't can safely pass non-`Sendable` arguments and results. +For example, a `nonisolated(nonsending)` function type can always be converted to an +actor-isolated function type, because the `nonisolated(nonsending)` function will +simply run on the actor: + +```swift +class NotSendable {} + +nonisolated(nonsending) +func performAsync(_ ns: NotSendable) async { ... } + +@MainActor +func convert(ns: NotSendable) async { + // Okay because 'performAsync' will run on the main actor + let runOnMain: @MainActor (NotSendable) async -> Void = performAsync + + await runOnMain(ns) +} +``` + +The following table enumerates each function conversion rule and specifies +which function conversions cross an isolation boundary. Function conversions +that cross an isolation boundary require `Sendable` argument and result types, +and the destination function type must be `async`. Note that the function +conversion rules for synchronous `nonisolated` functions and asynchronous +`nonisolated(nonsending)` functions are the same; they are both +represented under the "Nonisolated" category in the table: + +| Old isolation | New isolation | Crosses Boundary | +|----------------------|------------------------|------------------| +| Nonisolated | Actor isolated | No | +| Nonisolated | `@isolated(any)` | No | +| Nonisolated | `@concurrent` | Yes | +| Actor isolated | Actor isolated | Yes | +| Actor isolated | `@isolated(any)` | No | +| Actor isolated | Nonisolated | Yes | +| Actor isolated | `@concurrent` | Yes | +| `@isolated(any)` | Actor isolated | Yes | +| `@isolated(any)` | Nonisolated | Yes | +| `@isolated(any)` | `@concurrent` | Yes | +| `@concurrent` | Actor isolated | Yes | +| `@concurrent` | `@isolated(any)` | No | +| `@concurrent` | Nonisolated | Yes | + +#### Non-`@Sendable` function conversions + +If a function type is not `@Sendable`, only one isolation domain can +reference the function at a time, and calls to the function may never +happen concurrently. These rules for non-`Sendable` types are enforced +through region isolation. When a non-`@Sendable` function is converted +to an actor-isolated function, the function value itself is merged into the +actor's region, along with any non-`Sendable` function captures: + +```swift +class NotSendable { + var value = 0 +} + +nonisolated(nonsending) +func convert(closure: () -> Void) async { + let ns = NotSendable() + let disconnectedClosure = { + ns.value += 1 + } + let valid: @MainActor () -> Void = disconnectedClosure // okay + await valid() + + let invalid: @MainActor () -> Void = closure // error + await invalid() +} +``` + +The function conversion for the `invalid` variable is an error because the +non-`Sendable` captures of `closure` could be used concurrently from the caller +of `convert` and the main actor. + +Converting a non-`@Sendable` function type to an actor-isolated one is invalid +if the original function must leave the actor in order to be called: + +```swift +nonisolated(nonsending) +func convert( + fn1: @escaping @concurrent () async -> Void, +) async { + let fn2: @MainActor () async -> Void = fn1 // error + + await withDiscardingTaskGroup { group in + group.addTask { await fn2() } + group.addTask { await fn2() } + } +} +``` + +In general, a conversion from an actor-isolated function type to a +`nonisolated` function type crosses an isolation boundary, because the +`nonisolated` function type can be called from an arbitrary isolation domain. +However, if the conversion happens on the actor, and the new function type is +not `@Sendable`, then the function must only be called from the actor. In this +case, the function conversion is allowed, and the resulting function value +is merged into the actor's region: + +```swift +class NotSendable {} + +@MainActor class C { + var ns: NotSendable + + func getState() -> NotSendable { ns } +} + +func call(_ closure: () -> NotSendable) -> NotSendable { + return closure() +} + +@MainActor func onMain(c: C) { + // 'result' is in the main actor's region + let result = call(c.getState) +} +``` + +### Region isolation rules + +`nonisolated(nonsending)` functions have the same region isolation rules as +synchronous `nonisolated` functions. When calling an `nonisolated(nonsending)` +function, all non-`Sendable` parameter and result values are merged into +the same region, but they are only merged into the caller's actor region if +one of those non-`Sendable` values is already in the actor's region. + +For example: + +```swift +class NotSendable {} + +nonisolated(nonsending) +func identity(_ t: T) async -> T { + return t +} + +actor MyActor { + func isolatedToSelf() async -> sending NotSendable { + let ns = NotSendable() + return await identity(ns) + } +} +``` + +The above code is valid; the implementation of `identity` can't access the +actor's state unless isolated state is passed in via one of the parameters. +Note that this code would be invalid if `identity` accepted an isolated +parameter, because the non-`Sendable` parameters and results would always be +merged into the actor's region. + +This proposal allows you to access `#isolation` in the implementation of an +`nonisolated(nonsending)` function for the purpose of forwarding it along to a +method that accepts an `isolated (any Actor)?`. This is still safe, because +there's no way to access the actor's isolated state via the `Actor` protocol, +and dynamic casting to a concrete actor type will not result in a value that +the function is known to be isolated to. + +### Executor switching + +Async functions switch executors in the implementation when entering the +function, and after any calls to other async functions. Note that synchronous +functions do not have the ability to switch executors. If a call to a +synchronous function crosses an isolation boundary, the call must happen in an +async context and the executor switch happens at the caller. + +`@concurrent` async functions switch to the generic executor, and +all other async functions switch to the isolated actor's executor. + +```swift +@MainActor func runOnMainExecutor() async { + // switch to main actor executor + + await runOnGenericExecutor() + + // switch to main actor executor +} + +@concurrent func runOnGenericExecutor() async { + // switch to generic executor + + await Task { @MainActor in + // switch to main actor executor + + ... + }.value + + // switch to generic executor +} +``` + +`nonisolated(nonsending)` functions will switch to the executor of the implicit +actor parameter passed from the caller instead of switching to the generic +executor: + +```swift +@MainActor func runOnMainExecutor() async { + // switch to main actor executor + ... +} + +class NotSendable { + var value = 0 +} + +actor MyActor { + let ns: NotSendable = .init() + + func callNonisolatedFunction() async { + await inheritIsolation(ns) + } +} + +nonisolated func inheritIsolation(_ ns: NotSendable) async { + // switch to isolated parameter's executor + + await runOnMainExecutor() + + // switch to isolated parameter's executor + + ns.value += 1 +} +``` + +For most calls, the switch upon entering the function will have no effect, +because it's already running on the executor of the actor parameter. + +A task executor preference can still be used to configure where a nonisolated +async function runs. However, if the nonisolated async function was called from +an actor with a custom executor, the task executor preference will not apply. +Otherwise, the code will risk a data-race, because the task executor preference +does not apply to actor-isolated methods with custom executors, and the +nonisolated async method can be passed mutable state from the actor. + +### Dynamic actor isolation APIs in async contexts + +Because nonisolated async functions may now execute on a specific actor at +runtime, the APIs in the Concurrency library for enforcing actor isolation +assertions and preconditions are now useful in these contexts. As such, the +`noasync` attribute will be removed from `assertIsolated`, `assumeIsolated`, +and `preconditionIsolated` on `Actor` and `MainActor`. + +### Import-as-async heuristic + +Nonisolated functions imported from Objective-C that match the import-as-async +heuristic from [SE-0297: Concurrency Interoperability with Objective-C][SE-0297] +will implicitly be imported as `nonisolated(nonsending)`. Objective-C async +functions already have bespoke code generation that continues running on +the caller's actor to match the semantics of the original completion handler +function, so `nonisolated(nonsending)` already better matches the semantics of these +imported `async` functions. This change will eliminate many existing data-race +safety issues that happen when calling an async function on an Objective-C +class from the main actor. Because the only effect of this change is +eliminating concurrency diagnostics -- the runtime behavior of the code will +not change -- it will not be gated behind the upcoming feature. + +## Source compatibility + +This proposal changes the semantics of nonisolated async functions when the +upcoming feature flag is enabled. Without the upcoming feature flag, the default +for nonisolated async functions is `@concurrent`. When the upcoming +feature flag is enabled, the default for nonisolated async functions changes to +`nonisolated(nonsending)`. This applies to both function declarations and function +values that are nonisolated (either implicitly or explicitly). + +Changing the default execution semantics of nonisolated async functions has +minor source compatibility impact if the implementation calls an +`@concurrent` function and passes non-Sendable state in the actor's +region. In addition to the source compatibility impact, the change can also +regress performance of existing code if, for example, a specific async function +relied on running off of the main actor when called from the main actor to +maintain a responsive UI. + +To avoid breaking source compatibility or silently changing behavior of +existing code, this change must be gated behind an upcoming feature flag. +However, unlike most other changes gated behind upcoming feature flags, this +change allows writing code that is valid with and without the upcoming feature +flag, but means something different. Many programmers have internalized the +SE-0338 semantics, and making this change several years after SE-0338 was +accepted creates an unfortunate intermediate state where it's difficult to +understand the semantics of a nonisolated async function without understanding +the build settings of the module you're writing code in. + +To make it easy to discover what kind of async function you're working with, +SourceKit will surface the implicit `nonisolated(nonsending)` or `@concurrent` +attribute for IDE inspection features like Quick Help in Xcode and Hover in +VSCode. To ease the transition to the upcoming feature flag, [migration +tooling][adoption-tooling] will provide fix-its to preserve behavior by +annotating nonisolated async functions with `@concurrent`. + +## ABI compatibility + +Adopting the semantics to run on the caller's actor for an existing nonisolated +async function is an ABI change, because the caller's actor must be passed as +a parameter. However, a number of APIs in the concurrency library have staged +in similar changes using isolated parameters and `#isolation`, and it may be +possible to offer tools to do this transformation automatically for resilient +libraries that want to adopt this behavior. + +For example, if a nonisolated async function is ABI-public and is available +earlier than a version of the Swift runtime that includes this change, the +compiler could emit two separate entry points for the function: + +```swift +@_alwaysEmitIntoClient +public func myAsyncFunc() async { + // original implementation +} + +@concurrent +@_silgen_name(...) // to preserve the original symbol name +@usableFromInline +internal func abi_myAsyncFunc() async { + // existing compiled code will continue to always run calls to this function + // on the generic executor. + await myAsyncFunc() +} +``` + +This transformation only works if the original function implementation +can be made inlinable. + +## Implications on adoption + +`nonisolated(nonsending)` functions must accept an implicit actor parameter. This +means that adding `nonisolated(nonsending)` to a function that is actor-isolated, or +changing a function from `@concurrent` to `nonisolated(nonsending)`, is +not a resilient change. + +## Alternatives considered + +### Changing isolation inference behavior to implicitly capture isolated parameters + +The current isolation inference behavior in contexts with isolated parameters +is often surprising with respect to data-race safety. However, this proposal +does not suggest changing the rules, because implicitly capturing an isolated +parameter can lead to silently causing new memory leaks in existing code. One +potential compromise is to keep the current isolation inference behavior, and +offer fix-its to capture the actor if there are any data-race safety errors +from capturing state in the actor's region. + +### Use `nonisolated` instead of a separate `@concurrent` attribute + +It's tempting to not introduce a new attribute to control where an async +function executes, and instead control this behavior with an explicit +`nonisolated` annotation. However, this approach falls short for the following +reasons: + +1. It does not accomplish the goal of having consistent semantics for + `nonisolated` by default, regardless of whether it's applied to synchronous + or async functions. +2. This approach cuts off the future direction of allowing + `@concurrent` on synchronous functions. + +### Alternative syntax choices + +Several different options for the spelling of `nonisolated(nonsending)` +and `@concurrent` were explored. An earlier iteration of this proposal +used the same base attribute for both annotations. However, these two +annotations serve very different purposes. `@concurrent` is the long-term +right way to move functions and closures off of actors. +`nonisolated(nonsending)` is necessary for the transition to the new behavior, +but it's not a syntax that will stick around long term in Swift codebases; the +ideal end state is that this is expressed via the default behavior for +(explicitly or implicitly) `nonisolated` async functions. + +Note that it is well understood that there is no perfect syntax which will +explain the semantics without other context such as educational material or +documentation. This is true for all syntax design decisions. + +#### No explicit spelling for `nonisolated(nonsending)` + +It's reasonable to question whether `nonisolated(nonsending)` is necessary +at all given that its only purpose is transitioning to the new behavior +for async functions. An explicit spelling that has consistent behavior +independent of upcoming features and language modes is valuable when +undertaking a transition that changes the meaning of existing code. + +An explicit, transitory attribute is valuable because there will be a period of +time where it is not immediately clear from source what kind of async function +a programmer is working with. It's necessary to be able to discover that +information from source, such as by showing an inferred attribute explicitly +in SourceKit's cursor info request (surfaced by "Quick Help" in Xcode and +"Hover" in LSP / VSCode). An explicit spelling that has consistent behavior +independent of language mode is also valuable for code generation tools like +macros, so that they do not have to consider build settings to determine the +right code to generate, it's valuable for posting code snippets on the forums +during the transition period, etc. + +#### Justification for `@concurrent` + +This proposal was originally pitched using the `@concurrent` syntax, and many +reviewers surfaced objects about why `@concurrent` may be misleading, such as: + +* `@concurrent` is not the only source of concurrency; concurrency can arise from + many other things. +* The execution of an `@concurrent` function is not concurrent from the local + perspective of the current task. + +It's true that concurrency can only arise if there are multiple "impetuses" +(such as tasks or event sources) in the program that are running with different +isolation. But for the most part, we can assume that there are multiple +impetuses; and while those impetuses might otherwise share isolation, +`@concurrent` is the only isolation specification under this proposal that +guarantees that they do not and therefore forces concurrency. Indeed, we expect +that programmers will be reaching for `@concurrent` exactly for that reason: +they want the current function to run concurrently with whatever else might +happen in the process. So, this proposal uses `@concurrent` because out of the +other alternatives we explored, it best reflects the programmer's intent for +using the attribute. + +#### `@executor` + +A previous iteration of this proposal used the syntax `@execution(concurrent)` +instead of `@concurrent`. The review thread explored several variations of +this syntax, including `@executor(concurrent)` and `@executor(global)`. + +However, `@execution` or `@executor` encourages +thinking about async function semantics in terms of the lower level model of +executors and threads, and we should be encouraging programmers to think about +these semantics at the higher abstraction level of actor isolation and tasks. +Trying to understand the semantics in proposal in terms of executors can also +be misleading, both because isolation does not always map naively to executor +requests and because executors are used for other things than isolation. +For example, an `@executor(global)` function could end up running on some +executor other than the global executor via task executor preferences. + +#### `@isolated` + +An alternative to `nonisolated(nonsending)` is to use the "isolated" +terminology, such as `@isolated(caller)`. However, this approach has very +unsatisfying answers for how it interacts with `nonisolated`. There are +two options: + +1. `@isolated(caller)` must be written together with `nonisolated`, + + This approach leads to the verbose and oxymoronic spelling + `@isolated(caller) nonisolated`. Though there + exists a perfectly reasonable explanation about how `nonisolated` is the + static isolation while `@isolated(caller)` is the dynamic isolation, most + programmers do not have this deep of an understanding of actor isolation, + and they should not have to in order to make basic use of nonisolated async + functions. +2. `@isolated(caller)` implies `nonisolated` and can be written alone as an + alternative. + + This direction means that programmers would sometimes write + `nonisolated` and sometimes write `@isolated(caller)`, which is not a good + end state to be in because programmers have to learn a separate syntax for + `async` functions that accomplishes the same effect as a `nonisolated` + synchronous function. Or, if we view `@isolated(caller)` as only used for + the transition to the new behavior, then the assumption is that some day + people will remove `@isolated(caller)` if it is written in source. If + `@isolated(caller)` implies `nonisolated`, then the code could change + behavior if it's in a context where global or instance actor isolation would + otherwise be inferred. + +Going in the oppose direction, this proposal could effectively deprecate +`nonisolated` and allow you to use `@isolated(caller)` everywhere that +`nonisolated` is currently supported, including synchronous methods, stored +properties, type declarations, and extensions. This direction was not chosen +for the following reasons: + +1. This would lead to much more code churn than the current proposal. Part of + the goal of this proposal is to minimize the change to only what is absolutely + necessary to solve the major usability problem with async functions on + non-`Sendable` types, because it's painful both to transition code and to + re-learn parts of the model that have already been internalized. +2. `nonisolated` is nicer to write than `@isolated(caller)` + or any other alternative attribute + argument syntax. + +#### `nonisolated` argument spelling + +An argument to `nonisolated` is more compelling than a separate attribute +to specify that an async function runs on the caller's actor because it +defines away the problem of whether this annotation implies `nonisolated` when +written alone. + +A few different options for the argument to `nonisolated` were explored. + +**`nonisolated(nosend)`**. +`nonisolated(nosend)` effectively the same as `nonisolated(nonsending)` as +proposed, but it states that the call itself does not constitute a "send", +rather than stating that the call is not "sending" its argument and result +values over an isolation boundary. `nonisolated(nosend)` is shorter, but +`nonisolated(nonsending)` is more consistent with existing Swift naming +conventions. + +**`nonisolated(caller)`**. +`nonisolated(caller)` is meant to indicate that the function is statically +`nonisolated` and dynamically isolated to the caller. However, putting those +terms together into one `nonisolated(caller)` attribute is misleading, because +it appears the mean exactly the opposite of what it actually means; +`nonisolated(caller)` reads "not isolated to the caller". + +**`nonisolated(nonconcurrent)`**. +If `@concurrent` is applied to a function, then the function must run +concurrently with the caller's actor (assuming multiple isolated tasks +in the program). `nonconcurrent` conveys the inverse; if `nonconcurrent` is +applied to an async function, then the function must not run concurrently +with the caller's actor. However, this statement isn't quite true, because the +implementation of the function can perform work concurrently, though that work +cannot involve the non-`Sendable` parameter values. + +**`nonisolated(static)`**. +`nonisolated(static)` is meant to convey that a function is only `nonisolated` +statically, but it may be dynamically isolated to a specific actor at runtime. +However, we have not yet introduced "static" into the language surface to mean +"at compile time". `static` also has an existing, different meaning; +`nonisolated static func` would mean something quite different from +`nonisolated(static) func`, despite having extremely similar spelling. + +## Revisions + +The proposal was revised with the following changes after the first review: + +* Renamed `@execution(concurrent)` back to `@concurrent`. +* Renamed `@execution(caller)` to `nonisolated(nonsending)` +* Removed the unconditional warning about nonisolated async functions that + don't explicitly specify `nonisolated(nonsending)` or `@concurrent`. +* Removed `noasync` from the `assumeIsolated` API family. +* Specified the region isolation rules for `nonisolated(nonsending)` functions [as + discussed in the first review][region-isolation]. + +The proposal was revised with the following changes after the pitch discussion: + +* Gate the behavior change behind an `NonisolatedNonsendingByDefault` upcoming + feature flag. +* Change the spelling of `@concurrent` to `@execution(concurrent)`, and add an + `@execution(caller)` attribute to allow expressing the new behavior this + proposal introduces when the upcoming feature flag is not enabled. +* Apply `@execution(caller)` to nonisolated async function types by default to + make the execution semantics consistent between async function declarations + and values. +* Change the terminology in the proposal to not use the "inherits isolation" + phrase. + +[SE-0297]: /proposals/0297-concurrency-objc.md +[SE-0338]: /proposals/0338-clarify-execution-non-actor-async.md +[SE-0421]: /proposals/0421-generalize-async-sequence.md +[adoption-tooling]: https://forums.swift.org/t/pitch-adoption-tooling-for-upcoming-features/77936 +[region-isolation]: https://forums.swift.org/t/se-0461-run-nonisolated-async-functions-on-the-callers-actor-by-default/77987/36 diff --git a/proposals/0462-task-priority-escalation-apis.md b/proposals/0462-task-priority-escalation-apis.md new file mode 100644 index 0000000000..f12b6dd375 --- /dev/null +++ b/proposals/0462-task-priority-escalation-apis.md @@ -0,0 +1,237 @@ +# Task Priority Escalation APIs + +* Proposal: [SE-0462](0462-task-priority-escalation-apis.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [Freddy Kellison-Linn](https://github.com/jumhyn) +* Status: **Implemented (Swift 6.2)** +* Implementation: https://github.com/swiftlang/swift/pull/78625 +* Review: ([pitch](https://forums.swift.org/t/pitch-task-priority-escalation-apis/77702)) ([review](https://forums.swift.org/t/se-0462-task-priority-escalation-apis/77997))([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0462-task-priority-escalation-apis/78488)) + +## Introduction + +A large part of Swift Concurrency is its Structured Concurrency model, in which tasks automatically form parent-child relationships, and inherit certain traits from their parent task. For example, a task started from a medium priority task, also starts on the medium priority, and not only that – if the parent task gets awaited on from a higher priority task, the parent's as well as all of its child tasks' task priority will be escalated in order to avoid priority inversion problems. + +This feature is automatic and works transparently for any structured task hierarchy. This proposal will discuss exposing user-facing APIs which can be used to participate in task priority escalation. + +## Motivation + +Generally developers can and should rely on the automatic task priority escalation happening transparently–at least for as long as all tasks necessary to escalate are created using structured concurrency primitives (task groups and `async let`). However, sometimes it is not possible to entirely avoid creating an unstructured task. + +One such example is the async sequence [`merge`](https://github.com/apple/swift-async-algorithms/blob/4c3ea81f81f0a25d0470188459c6d4bf20cf2f97/Sources/AsyncAlgorithms/AsyncAlgorithms.docc/Guides/Merge.md) operation from the [swift-async-algorithms](https://github.com/apple/swift-async-algorithms/) project where the implementation is forced to create an unstructured task for iterating the upstream sequences, which must outlive downstream calls. These libraries would like to participate in task priority escalation to boost the priority of the upstream consuming task, however today they lack the API to do so. + +```swift +// SIMPLIFIED EXAMPLE CODE +// Complete source: https://github.com/apple/swift-async-algorithms/blob/main/Sources/AsyncAlgorithms/Merge/MergeStorage.swift + +struct AsyncMergeSequenceIterator: AsyncIterator { + struct State { + var task: Task? // unstructured upstream consumer task + var buffer: Deque + var upstreamContinuations: [UnsafeContinuation] + var downstreamContinuation: UnsafeContinuation? + } + + let state = Mutex(State()) + + func next() async throws { + self.state.withLock { state in + if state.task == nil { + state.task = Task { + // Consume from the base iterators + // ... + } + } + } + + if let element = self.state.withLock { $0.buffer.popFirst() } { + return element + } else { + // We are handling cancellation here and need to handle task escalation here as well + try await withTaskCancellationHandler { + // HERE: need to handle priority escalation and boost `state.task` + try await withCheckedContinuation { cont in + self.state.withLock { $0.consumerContinuation = cont } + } + } onCancel: { + // trigger cancellation of tasks and fail continuations + } + } + } +} +``` + +The above example showcases a common pattern: often a continuation is paired with a Task used to complete it. Around the suspension on the continuation, waiting for it to be resumed, developers often install a task cancellation handler in order to potentially break out of potentially unbounded waiting for a continuation to be resumed. Around the same suspension (marked with `HERE` in the snippet above), we might want to insert a task priority escalation handler in order to priority boost the task that is used to resume the continuation. This can be important for correctness and performance of such operations, so we should find a way to offer these libraries a mechanism to participate in task priority handling. + +Another example of libraries which may want to reach for manual task priority escalation APIs are libraries which facilitate communication across process boundaries, and would like to react to priority escalation and propagate it to a different process. Relying on the built-in priority escalation mechanisms won't work, because they are necessarily in-process, so libraries like this need to be able to participate and be notified when priority escalation happens, and also be able to efficiently cause the escalation inside the other process. + +## Proposed solution + +In order to address the above use-cases, we propose to add a pair of APIs: to react to priority escalation happening within a block of code, and an API to _cause_ a priority escalation without resorting to trickery by creating new tasks whose only purpose is to escalate the priority of some other task: + +```swift +enum State { + case initialized + case task(Task) + case priority(TaskPriority) +} +let m: Mutex = .init(.initialized) + +await withTaskPriorityEscalationHandler { + await withCheckedContinuation { cc in + let task = Task { cc.resume() } + + let newPriority: TaskPriority? = state.withLock { state -> TaskPriority? in + defer { state = .task(task) } + switch state { + case .initialized: + return nil + case .task: + preconditionFailure("unreachable") + case .priority(let priority): + return priority + } + } + // priority was escalated just before we stored the task in the mutex + if let newPriority { + Task.escalatePriority(of: task, to: newPriority) + } + } onPriorityEscalated: { newPriority in + state.withLock { state in + switch state { + case .initialized, .priority: + // priority was escalated just before we managed to store the task in the mutex + state = .priority(newPriority) + case .task(let task): + Task.escalatePriority(of: task, to: newPriority) + } + } + } +} +``` + +The above snippet handles edge various ordering situations, including the task escalation happening after +the time the handler is registered but _before_ we managed to create and store the task. + +In general, task escalation remains a slightly racy affair, we could always observe an escalation "too late" for it to matter, +and have any meaningful effect on the work's execution, however this API and associated patterns handle most situations which +we care about in practice. + +## Detailed design + +We propose the addition of a task priority escalation handler, similar to task cancellation handlers already present in the concurrency library: + +```swift +public func withTaskPriorityEscalationHandler( + operation: () async throws(E) -> T, + onPriorityEscalated handler: @Sendable (TaskPriority) -> Void, + isolation: isolated (any Actor)? = #isolation +) async throws(E) -> T +``` + +The shape of this API is similar to the `withTaskCancellationHandler` API present since initial Swift Concurrency release, however–unlike a cancellation handler–the `onPriorityEscalated` callback may be triggered multiple times. The `TaskPriority` passed to the handler is the "new priority" the surrounding task was escalated to. + +It is guaranteed that priority is ever only increasing, as Swift Concurrency does not allow for a task priority to ever be lowered after it has been escalated. If attempts are made to escalate the task priority from multiple other threads to the same priority, the handler will only trigger once. However if priority is escalated to a high and then even higher priority, the handler may be invoked twice. + +Task escalation handlers are inherently racy, and may sometimes miss an escalation, for example if it happened immediately before the handler was installed, like this: + +```swift +// priority: low +// priority: high! +await withTaskPriorityEscalationHandler { + await work() +} onPriorityEscalated: { newPriority in // may not be triggered if ->high escalation happened before handler was installed + // do something +} +``` + +This is inherent to the nature of priority escalation and even with this behavior, we believe handlers are a worthy addition. One could also check for the `Task.currentPriority` and match it against our expectations inside the `operation` wrapped by the `withTaskPriorityEscalationHandler` if that could be useful to then perform the operation at an already _immediately_ heightened priority. + +Escalation handlers work with any existing task kind (child, unstructured, unstructured detached), and trigger at every level of the hierarchy in an "outside in" order: + +```swift +let t = Task { + await withTaskPriorityEscalationHandler { + await withTaskGroup { group in + group.addTask { + await withTaskPriorityEscalationHandler { + try? await Task.sleep(for: .seconds(1)) + } onPriorityEscalated: { newPriority in print("inner: \(newPriority)") } + } + } + } onPriorityEscalated: { newPriority in print("outer: \(newPriority)") } +} + +// escalate t -> high +// "outer: high" +// "inner: high" +``` + +The API can also be freely composed with `withTaskCancellationHandler` or there may even be multiple task escalation handlers registered on the same task (but in different pieces of the code). + +### Manually propagating priority escalation + +While generally developers should not rely on manual task escalation handling, this API also does introduce a manual way to escalate a task's priority. Primarily this should be used in combination with a task escalation handler to _propagate_ an escalation to an _unstructured task_ which otherwise would miss reacting to the escalation. + +The `escalatePriority(of:to:)` API is offered as a static method on `Task` in order to slightly hide it away from using it accidentally by stumbling upon it if it were directly declared as a member method of a Task. + +```swift +extension Task { + public static func escalatePriority(of task: Task, to newPriority: TaskPriority) +} + +extension UnsafeCurrentTask { + public static func escalatePriority(of task: UnsafeCurrentTask, to newPriority: TaskPriority) +} +``` + +It is possible to escalate both a `Task` and `UnsafeCurrentTask`, however great care must be taken to not attempt to escalate an unsafe task handle if the task has already been destroyed. The `Task` accepting API is always safe. + +Currently it is not possible to escalate a specific child task (created by `async let` or a task group) because those do not return task handles. We are interested in exposing task handles to child tasks in the future, and this design could then be easily amended to gain API to support such child task handles as well. + +## Source compatibility + +This proposal is purely additive, and does not cause any source compatibility issues. + +## ABI compatibility + +This proposal is purely ABI additive. + +## Alternatives considered + +### New Continuation APIs + +We did consider if offering a new kind of continuation might be easier to work with for developers. One shape this might take is: + +```swift +struct State { + var cc = CheckedContinuation? + var task: Task? +} +let C: Mutex + +await withCheckedContinuation2 { cc in + // ... + C.withLock { $0.cc = cc } + + let t = Task { + C.withLock { + $0.cc?.resume() // maybe we'd need to add 'tryResume' + } + } + C.withLock { $0.task = t } +} onCancel: { cc in + // remember the cc can only be resumed once; we'd need to offer 'tryResume' + cc.resume(throwing: CancellationError()) +} onPriorityEscalated: { cc, newPriority in + print("new priority: \(newPriority)") + C.withLock { Task.escalatePriority(of: $0.task, to: newPriority) } +} +``` + +While at first this looks promising, we did not really remove much of the complexity -- careful locking is still necessary, and passing the continuation into the closures only makes it more error prone than not since it has become easier to accidentally multi-resume a continuation. This also does not compose well, and would only be offered around continuations, even if not all use-cases must necessarily suspend on a continuation to benefit from the priority escalation handling. + +Overall, this seems like a tightly knit API that changes current idioms of `with...Handler ` without really saving us from the inherent complexity of these handlers being invoked concurrently, and limiting the usefulness of those handlers to just "around a continuation" which may not always be the case. + +### Acknowledgements + +I'd like to thank John McCall, David Nadoba for their input on the APIs during early reviews. diff --git a/proposals/0463-sendable-completion-handlers.md b/proposals/0463-sendable-completion-handlers.md new file mode 100644 index 0000000000..85729b6ae5 --- /dev/null +++ b/proposals/0463-sendable-completion-handlers.md @@ -0,0 +1,82 @@ +# Import Objective-C completion handler parameters as `@Sendable` + +* Proposal: [SE-0463](0463-sendable-completion-handlers.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Implemented (Swift 6.2)** +* Vision: [Improving the approachability of data-race safety](https://github.com/swiftlang/swift-evolution/blob/main/visions/approachable-concurrency.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-import-objective-c-completion-handler-parameters-as-sendable/77904)) ([review](https://forums.swift.org/t/se-0463-import-objective-c-completion-handler-parameters-as-sendable/78169)) ([acceptance](https://forums.swift.org/t/accepted-se-0463-import-objective-c-completion-handler-parameters-as-sendable/78489)) + +## Introduction + +This proposal changes the Objective-C importing rules such that completion handler parameters are `@Sendable` by default. + +## Motivation + +Swift's data-race safety model requires function declarations to codify their concurrency invariants in the function signature with annotations. The `@Sendable` annotation indicates that closure parameters are passed over an isolation boundary before they're called. A missing `@Sendable` annotation in a library has negative effects on clients who call the function; the caller can unknowingly introduce data races, and [SE-0423: Dynamic actor isolation enforcement from non-strict-concurrency contexts][SE-0423] injects runtime assertions for non-`Sendable` closure parameters that are passed into libraries that don't have data-race safety checking. This means that a missing `@Sendable` annotation can lead to a runtime crash for any code that calls the API from an actor isolated context, which is extremely painful for projects that are migrating to the Swift 6 language mode. + +There's a large category of APIs with closure parameters that can be automatically identified as `@Sendable` functions, even if the annotation is missing: Objective-C methods with completion handler parameters. `@Sendable` is nearly always the right default for Objective-C completion handlers, and [programmers have already been searching for an automatic way for completion handlers to be `@Sendable` by default when auditing Clang headers](https://forums.swift.org/t/clang-sendability-audit-for-closures/75557). + +## Proposed solution + +I propose automatically importing completion handler parameters from Objective-C methods as `@Sendable` functions. + +## Detailed design + +If an imported method has an async variant (as described in [SE-0297: Concurrency Interoperability with Objective-C][SE-0297]) and the method is (implicitly or explicitly) `nonisolated`, the original method will be imported with a `@Sendable` annotation on its completion handler parameter. + +For example, given the following Objective-C method signature: + +```objc +- (void)performOperation:(NSString * _Nonnull)operation + completionHandler:(void (^ _Nullable)(NSString * _Nullable, NSError * _Nullable))completionHandler; +``` + +Swift will import the method with `@Sendable` on the `completionHandler` parameter: + +```swift +@preconcurrency +func perform( + operation: String, + completionHandler: @Sendable @escaping ((String?, Error?) -> Void)? +) +``` + +When calling the `perform` method from a Swift actor, the inference rules that allow non-`Sendable` closures to be isolated to the context they're formed in will no longer apply. The closure will be inferred as `nonisolated`, and warnings will be produced if any mutable state in the actor's region is accessed from the closure. Note that all APIs imported from C/C++/Objective-C are automatically `@preconcurrency`, so data-race safety violations are only ever warnings, even in the Swift 6 language mode. + +### Completion handlers of global actor isolated functions + +Functions that are isolated to a global actor will not have completion handlers imported as `@Sendable`. Main actor isolated functions with completion handlers that are always called on the main actor is a very common Objective-C pattern, and this carve out will eliminate false positive warnings in the cases where the main actor annotation is missing on the completion handler parameter. This carve out will not add any new dynamic assertions. + +### Opting out of `@Sendable` completion handlers + +If a completion handler does not cross an isolation boundary before it's called, the parameter can be annotated in the header with the `@nonSendable` attribute using `__attribute__((swift_attr(“@nonSendable”)))`. The `@nonSendable` attribute is only for Clang header annotations; it is not meant to be used from Swift code. + +## Source compatibility + +This change has no effect in language modes prior to Swift 6 when using minimal concurrency checking, and it only introduces warnings when using complete concurrency checking, even in the Swift 6 language mode. Declarations imported from C/C++/Objective-C are implicitly `@preconcurrency`, which makes all data-race safety violations warnings. + +## ABI compatibility + +This proposal has no impact on existing ABI. + +## Alternatives considered + +### Import completion handlers as `sending` instead of `@Sendable` + +The choice to import completion handlers as `@Sendable` instead of `sending` is pragmatic - the experimental `SendableCompletionHandlers` implementation has existed since 2021 and has been extensively tested for source compatibility. Similarly, `@Sendable` has been explicitly adopted in Objective-C frameworks for several years, and source compatibility issues resulting from corner cases in the compiler implementation that were intolerable to `@Sendable` mismatches have shaken out over time. `sending` is still a relatively new parameter attribute, it has not been adopted as extensively as `@Sendable`, and it does not support downgrading diagnostics in the Swift 6 language mode when combined with `@preconcurrency`. The pain caused by the dynamic actor isolation runtime assertions is enough that it's worth solving this problem now conservatively using `@Sendable`. + +Changing this proposal later to use `sending` instead will pose source compatibility issues, because it would become invalid to have a protocol requirement that is imported with a `sending` completion handler and implement the requirement with a `@Sendable` completion handler. The same source compatibility issue exists for overridden class methods. If programmers want to take advantage of region isolation, the recommended path is to modernize the code using `async`/`await`. + +## Acknowledgments + +Thank you to Becca Royal-Gordon for implementing the `SendableCompletionHandlers` experimental feature, and thank you to Pavel Yaskevich for consistently fixing compiler bugs where the implementation was intolerant to `@Sendable` mismatches. + +[SE-0297]: /proposals/0297-concurrency-objc.md +[SE-0423]: /proposals/0423-dynamic-actor-isolation.md + +## Revisions + +The proposal was revised with the following changes after the pitch discussion: + +* Add a carve out where global actor isolated functions are still imported with non-`Sendable` completion handlers. diff --git a/proposals/0464-utf8span-safe-utf8-processing.md b/proposals/0464-utf8span-safe-utf8-processing.md new file mode 100644 index 0000000000..1f366a6004 --- /dev/null +++ b/proposals/0464-utf8span-safe-utf8-processing.md @@ -0,0 +1,837 @@ +# UTF8Span: Safe UTF-8 Processing Over Contiguous Bytes + +* Proposal: [SE-0464](0464-utf8span-safe-utf8-processing.md) +* Authors: [Michael Ilseman](https://github.com/milseman), [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (6.2)** +* Bug: rdar://48132971, rdar://96837923 +* Implementation: [swiftlang/swift#78531](https://github.com/swiftlang/swift/pull/78531) +* Review: ([first pitch](https://forums.swift.org/t/pitch-utf-8-processing-over-unsafe-contiguous-bytes/69715)) ([second pitch](https://forums.swift.org/t/pitch-safe-utf-8-processing-over-contiguous-bytes/72742)) ([third pitch](https://forums.swift.org/t/pitch-utf8span-safe-utf-8-processing-over-contiguous-bytes/77483)) ([review](https://forums.swift.org/t/se-0464-utf8span-safe-utf-8-processing-over-contiguous-bytes/78307)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0464-safe-utf-8-processing-over-contiguous-bytes/79218)) + + +## Introduction + +We introduce `UTF8Span` for efficient and safe Unicode processing over contiguous storage. `UTF8Span` is a memory safe non-escapable type [similar to `Span`](0447-span-access-shared-contiguous-storage.md). + +Native `String`s are stored as validly-encoded UTF-8 bytes in an internal contiguous memory buffer. The standard library implements `String`'s API as internal methods which operate on top of this buffer, taking advantage of the validly-encoded invariant and specialized Unicode knowledge. We propose making this UTF-8 buffer and its methods public as API for more advanced libraries and developers. + +## Motivation + +Currently, if a developer wants to do `String`-like processing over UTF-8 bytes, they have to make an instance of `String`, which allocates a native storage class, copies all the bytes, and is reference counted. The developer would then need to operate within the new `String`'s views and map between `String.Index` and byte offsets in the original buffer. + +For example, if these bytes were part of a data structure, the developer would need to decide to either cache such a new `String` instance or recreate it on the fly. Caching more than doubles the size and adds caching complexity. Recreating it on the fly adds a linear time factor and class instance allocation/deallocation and potentially reference counting. + +Furthermore, `String` may not be available on tightly constrained platforms, such as those that cannot support allocations. Both `String` and `UTF8Span` have some API that require Unicode data tables and that might not be available on embedded (String via its conformance to `Comparable` and `Collection` depend on these data tables while `UTF8Span` has a couple of methods that will be unavailable). + +### UTF-8 validity and efficiency + +UTF-8 validation is a particularly common concern and the subject of a fair amount of [research](https://lemire.me/blog/2020/10/20/ridiculously-fast-unicode-utf-8-validation/). Once an input is known to be validly encoded UTF-8, subsequent operations such as decoding, grapheme breaking, comparison, etc., can be implemented much more efficiently under this assumption of validity. Swift's `String` type's native storage is guaranteed-valid-UTF8 for this reason. + +Failure to guarantee UTF-8 encoding validity creates security and safety concerns. With invalidly-encoded contents, memory safety would become more nuanced. An ill-formed leading byte can dictate a scalar length that is longer than the memory buffer. The buffer may have bounds associated with it, which differs from the bounds dictated by its contents. + +Additionally, a particular scalar value in valid UTF-8 has only one encoding, but invalid UTF-8 could have the same value encoded as an [overlong encoding](https://en.wikipedia.org/wiki/UTF-8#Overlong_encodings), which would compromise code that checks for the presence of a scalar value by looking at the encoded bytes (or that does a byte-wise comparison). + + +## Proposed solution + +We propose a non-escapable `UTF8Span` which exposes `String` functionality for validly-encoded UTF-8 code units in contiguous memory. We also propose rich API describing the kind and location of encoding errors. + +## Detailed design + +`UTF8Span` is a borrowed view into contiguous memory containing validly-encoded UTF-8 code units. + +```swift +public struct UTF8Span: Copyable, ~Escapable, BitwiseCopyable {} +``` + +`UTF8Span` is a trivial struct and is 2 words in size on 64-bit platforms. + +### UTF-8 validation + +We propose new API for identifying where and what kind of encoding errors are present in UTF-8 content. + +```swift +extension Unicode.UTF8 { + /** + + The kind and location of a UTF-8 encoding error. + + Valid UTF-8 is represented by this table: + + ╔════════════════════╦════════╦════════╦════════╦════════╗ + ║ Scalar value ║ Byte 0 ║ Byte 1 ║ Byte 2 ║ Byte 3 ║ + ╠════════════════════╬════════╬════════╬════════╬════════╣ + ║ U+0000..U+007F ║ 00..7F ║ ║ ║ ║ + ║ U+0080..U+07FF ║ C2..DF ║ 80..BF ║ ║ ║ + ║ U+0800..U+0FFF ║ E0 ║ A0..BF ║ 80..BF ║ ║ + ║ U+1000..U+CFFF ║ E1..EC ║ 80..BF ║ 80..BF ║ ║ + ║ U+D000..U+D7FF ║ ED ║ 80..9F ║ 80..BF ║ ║ + ║ U+E000..U+FFFF ║ EE..EF ║ 80..BF ║ 80..BF ║ ║ + ║ U+10000..U+3FFFF ║ F0 ║ 90..BF ║ 80..BF ║ 80..BF ║ + ║ U+40000..U+FFFFF ║ F1..F3 ║ 80..BF ║ 80..BF ║ 80..BF ║ + ║ U+100000..U+10FFFF ║ F4 ║ 80..8F ║ 80..BF ║ 80..BF ║ + ╚════════════════════╩════════╩════════╩════════╩════════╝ + + ### Classifying errors + + An *unexpected continuation* is when a continuation byte (`10xxxxxx`) occurs + in a position that should be the start of a new scalar value. Unexpected + continuations can often occur when the input contains arbitrary data + instead of textual content. An unexpected continuation at the start of + input might mean that the input was not correctly sliced along scalar + boundaries or that it does not contain UTF-8. + + A *truncated scalar* is a multi-byte sequence that is the start of a valid + multi-byte scalar but is cut off before ending correctly. A truncated + scalar at the end of the input might mean that only part of the entire + input was received. + + A *surrogate code point* (`U+D800..U+DFFF`) is invalid UTF-8. Surrogate + code points are used by UTF-16 to encode scalars in the supplementary + planes. Their presence may mean the input was encoded in a different 8-bit + encoding, such as CESU-8, WTF-8, or Java's Modified UTF-8. + + An *invalid non-surrogate code point* is any code point higher than + `U+10FFFF`. This can often occur when the input is arbitrary data instead + of textual content. + + An *overlong encoding* occurs when a scalar value that could have been + encoded using fewer bytes is encoded in a longer byte sequence. Overlong + encodings are invalid UTF-8 and can lead to security issues if not + correctly detected: + + - https://nvd.nist.gov/vuln/detail/CVE-2008-2938 + - https://nvd.nist.gov/vuln/detail/CVE-2000-0884 + + An overlong encoding of `NUL`, `0xC0 0x80`, is used in Java's Modified + UTF-8 but is invalid UTF-8. Overlong encoding errors often catch attempts + to bypass security measures. + + ### Reporting the range of the error + + The range of the error reported follows the *Maximal subpart of an + ill-formed subsequence* algorithm in which each error is either one byte + long or ends before the first byte that is disallowed. See "U+FFFD + Substitution of Maximal Subparts" in the Unicode Standard. Unicode started + recommending this algorithm in version 6 and is adopted by the W3C. + + The maximal subpart algorithm will produce a single multi-byte range for a + truncated scalar (a multi-byte sequence that is the start of a valid + multi-byte scalar but is cut off before ending correctly). For all other + errors (including overlong encodings, surrogates, and invalid code + points), it will produce an error per byte. + + Other commonly reported error ranges can be constructed from this result. + For example, PEP 383's error-per-byte can be constructed by mapping over + the reported range. Similarly, constructing a single error for the longest + invalid byte range can be constructed by joining adjacent error ranges. + + ╔═════════════════╦══════╦═════╦═════╦═════╦═════╦═════╦═════╦══════╗ + ║ ║ 61 ║ F1 ║ 80 ║ 80 ║ E1 ║ 80 ║ C2 ║ 62 ║ + ╠═════════════════╬══════╬═════╬═════╬═════╬═════╬═════╬═════╬══════╣ + ║ Longest range ║ U+61 ║ err ║ ║ ║ ║ ║ ║ U+62 ║ + ║ Maximal subpart ║ U+61 ║ err ║ ║ ║ err ║ ║ err ║ U+62 ║ + ║ Error per byte ║ U+61 ║ err ║ err ║ err ║ err ║ err ║ err ║ U+62 ║ + ╚═════════════════╩══════╩═════╩═════╩═════╩═════╩═════╩═════╩══════╝ + + */ + public struct EncodingError: Error, Sendable, Hashable, Codable { + /// The kind of encoding error + public var kind: Unicode.UTF8.EncodingError.Kind + + /// The range of offsets into our input containing the error + public var range: Range + + public init( + _ kind: Unicode.UTF8.EncodingError.Kind, + _ range: some RangeExpression + ) + + public init(_ kind: Unicode.UTF8.EncodingError.Kind, at: Int) + } +} + +extension UTF8.EncodingError { + /// The kind of encoding error encountered during validation + public struct Kind: Error, Sendable, Hashable, Codable, RawRepresentable { + public var rawValue: UInt8 + + public init(rawValue: UInt8) + + /// A continuation byte (`10xxxxxx`) outside of a multi-byte sequence + public static var unexpectedContinuationByte: Self + + /// A byte in a surrogate code point (`U+D800..U+DFFF`) sequence + public static var surrogateCodePointByte: Self + + /// A byte in an invalid, non-surrogate code point (`>U+10FFFF`) sequence + public static var invalidNonSurrogateCodePointByte: Self + + /// A byte in an overlong encoding sequence + public static var overlongEncodingByte: Self + + /// A multi-byte sequence that is the start of a valid multi-byte scalar + /// but is cut off before ending correctly + public static var truncatedScalar: Self + } +} + +extension UTF8.EncodingError.Kind: CustomStringConvertible { + public var description: String { get } +} + +extension UTF8.EncodingError: CustomStringConvertible { + public var description: String { get } +} +``` + +### Creation and validation + +`UTF8Span` is validated at initialization time and encoding errors are diagnosed and thrown. + + +```swift + +extension UTF8Span { + /// Creates a UTF8Span containing `codeUnits`. Validates that the input is + /// valid UTF-8, otherwise throws an error. + /// + /// The resulting UTF8Span has the same lifetime constraints as `codeUnits`. + public init(validating codeUnits: Span) throws(UTF8.EncodingError) + + /// Creates a UTF8Span unsafely containing `uncheckedBytes`, skipping validation. + /// + /// `uncheckedBytes` _must_ be valid UTF-8 or else undefined behavior may + /// emerge from any use of the resulting UTF8Span, including any use of a + /// `String` created by copying the resultant UTF8Span + @unsafe + public init(unsafeAssumingValidUTF8 uncheckedCodeUnits: Span) +} +``` + +Similarly, `String`s can be created from `UTF8Span`s without re-validating their contents. + +```swift +extension String { + /// Create's a String containing a copy of the UTF-8 content in `codeUnits`. + /// Skips + /// validation. + public init(copying codeUnits: UTF8Span) +} +``` + +### Scalar processing + +We propose a `UTF8Span.UnicodeScalarIterator` type that can do scalar processing forwards and backwards. Note that `UnicodeScalarIterator` itself is non-escapable, and thus cannot conform to `IteratorProtocol`, etc. + +```swift +extension UTF8Span { + /// Returns an iterator that will decode the code units into + /// `Unicode.Scalar`s. + /// + /// The resulting iterator has the same lifetime constraints as `self`. + public func makeUnicodeScalarIterator() -> UnicodeScalarIterator + + /// Iterate the `Unicode.Scalar`s contents of a `UTF8Span`. + public struct UnicodeScalarIterator: ~Escapable { + public let codeUnits: UTF8Span + + /// The byte offset of the start of the next scalar. This is + /// always scalar-aligned. + public var currentCodeUnitOffset: Int { get private(set) } + + public init(_ codeUnits: UTF8Span) + + /// Decode and return the scalar starting at `currentCodeUnitOffset`. + /// After the function returns, `currentCodeUnitOffset` holds the + /// position at the end of the returned scalar, which is also the start + /// of the next scalar. + /// + /// Returns `nil` if at the end of the `UTF8Span`. + public mutating func next() -> Unicode.Scalar? + + /// Decode and return the scalar ending at `currentCodeUnitOffset`. After + /// the function returns, `currentCodeUnitOffset` holds the position at + /// the start of the returned scalar, which is also the end of the + /// previous scalar. + /// + /// Returns `nil` if at the start of the `UTF8Span`. + public mutating func previous() -> Unicode.Scalar? + + /// Advance `codeUnitOffset` to the end of the current scalar, without + /// decoding it. + /// + /// Returns the number of `Unicode.Scalar`s skipped over, which can be 0 + /// if at the end of the UTF8Span. + public mutating func skipForward() -> Int + + /// Advance `codeUnitOffset` to the end of `n` scalars, without decoding + /// them. + /// + /// Returns the number of `Unicode.Scalar`s skipped over, which can be + /// fewer than `n` if at the end of the UTF8Span. + public mutating func skipForward(by n: Int) -> Int + + /// Move `codeUnitOffset` to the start of the previous scalar, without + /// decoding it. + /// + /// Returns the number of `Unicode.Scalar`s skipped over, which can be 0 + /// if at the start of the UTF8Span. + public mutating func skipBack() -> Int + + /// Move `codeUnitOffset` to the start of the previous `n` scalars, + /// without decoding them. + /// + /// Returns the number of `Unicode.Scalar`s skipped over, which can be + /// fewer than `n` if at the start of the UTF8Span. + public mutating func skipBack(by n: Int) -> Int + + /// Reset to the nearest scalar-aligned code unit offset `<= i`. + public mutating func reset(roundingBackwardsFrom i: Int) + + /// Reset to the nearest scalar-aligned code unit offset `>= i`. + public mutating func reset(roundingForwardsFrom i: Int) + + /// Reset this iterator to code unit offset `i`, skipping _all_ safety + /// checks (including bounds checks). + /// + /// Note: This is only for very specific, low-level use cases. If + /// `codeUnitOffset` is not properly scalar-aligned, this function can + /// result in undefined behavior when, e.g., `next()` is called. + /// + /// For example, this could be used by a regex engine to backtrack to a + /// known-valid previous position. + /// + public mutating func reset(uncheckedAssumingAlignedTo i: Int) + + /// Returns the UTF8Span containing all the content up to the iterator's + /// current position. + /// + /// The resultant `UTF8Span` has the same lifetime constraints as `self`. + public func prefix() -> UTF8Span + + /// Returns the UTF8Span containing all the content after the iterator's + /// current position. + /// + /// The resultant `UTF8Span` has the same lifetime constraints as `self`. + public func suffix() -> UTF8Span + } +} + +``` + +### Character processing + +We similarly propose a `UTF8Span.CharacterIterator` type that can do grapheme-breaking forwards and backwards. + +The `CharacterIterator` assumes that the start and end of the `UTF8Span` is the start and end of content. + +Any scalar-aligned position is a valid place to start or reset the grapheme-breaking algorithm to, though you could get different `Character` output if resetting to a position that isn't `Character`-aligned relative to the start of the `UTF8Span` (e.g. in the middle of a series of regional indicators). + +```swift + +extension UTF8Span { + /// Returns an iterator that will construct `Character`s from the underlying + /// UTF-8 content. + /// + /// The resulting iterator has the same lifetime constraints as `self`. + public func makeCharacterIterator() -> CharacterIterator + + /// Iterate the `Character` contents of a `UTF8Span`. + public struct CharacterIterator: ~Escapable { + public let codeUnits: UTF8Span + + /// The byte offset of the start of the next `Character`. This is always + /// scalar-aligned. It is always `Character`-aligned relative to the last + /// call to `reset` (or the start of the span if not called). + public var currentCodeUnitOffset: Int { get private(set) } + + public init(_ span: UTF8Span) + + /// Return the `Character` starting at `currentCodeUnitOffset`. After the + /// function returns, `currentCodeUnitOffset` holds the position at the + /// end of the `Character`, which is also the start of the next + /// `Character`. + /// + /// Returns `nil` if at the end of the `UTF8Span`. + public mutating func next() -> Character? + + /// Return the `Character` ending at `currentCodeUnitOffset`. After the + /// function returns, `currentCodeUnitOffset` holds the position at the + /// start of the returned `Character`, which is also the end of the + /// previous `Character`. + /// + /// Returns `nil` if at the start of the `UTF8Span`. + public mutating func previous() -> Character? + + /// Advance `codeUnitOffset` to the end of the current `Character`, + /// without constructing it. + /// + /// Returns the number of `Character`s skipped over, which can be 0 + /// if at the end of the UTF8Span. + public mutating func skipForward() -> Int + + /// Advance `codeUnitOffset` to the end of `n` `Characters`, without + /// constructing them. + /// + /// Returns the number of `Character`s skipped over, which can be + /// fewer than `n` if at the end of the UTF8Span. + public mutating func skipForward(by n: Int) -> Int + + /// Move `codeUnitOffset` to the start of the previous `Character`, + /// without constructing it. + /// + /// Returns the number of `Character`s skipped over, which can be 0 + /// if at the start of the UTF8Span. + public mutating func skipBack() -> Int + + /// Move `codeUnitOffset` to the start of the previous `n` `Character`s, + /// without constructing them. + /// + /// Returns the number of `Character`s skipped over, which can be + /// fewer than `n` if at the start of the UTF8Span. + public mutating func skipBack(by n: Int) -> Int + + /// Reset to the nearest character-aligned position `<= i`. + public mutating func reset(roundingBackwardsFrom i: Int) + + /// Reset to the nearest character-aligned position `>= i`. + public mutating func reset(roundingForwardsFrom i: Int) + + /// Reset this iterator to code unit offset `i`, skipping _all_ safety + /// checks (including bounds checks). + /// + /// Note: This is only for very specific, low-level use cases. If + /// `codeUnitOffset` is not properly scalar-aligned, this function can + /// result in undefined behavior when, e.g., `next()` is called. + /// + /// If `i` is scalar-aligned, but not `Character`-aligned, you may get + /// different results from running `Character` iteration. + /// + /// For example, this could be used by a regex engine to backtrack to a + /// known-valid previous position. + /// + public mutating func reset(uncheckedAssumingAlignedTo i: Int) + + /// Returns the UTF8Span containing all the content up to the iterator's + /// current position. + /// + /// The resultant `UTF8Span` has the same lifetime constraints as `self`. + public func prefix() -> UTF8Span + + /// Returns the UTF8Span containing all the content after the iterator's + /// current position. + /// + /// The resultant `UTF8Span` has the same lifetime constraints as `self`. + public func suffix() -> UTF8Span + } +} +``` + +### Comparisons + +The content of a `UTF8Span` can be compared in a number of ways, including literally (byte semantics) and Unicode canonical equivalence. + +```swift +extension UTF8Span { + /// Whether this span has the same bytes as `other`. + public func bytesEqual(to other: UTF8Span) -> Bool + + /// Whether this span has the same bytes as `other`. + public func bytesEqual(to other: some Sequence) -> Bool + + /// Whether this span has the same `Unicode.Scalar`s as `other`. + public func scalarsEqual( + to other: some Sequence + ) -> Bool + + /// Whether this span has the same `Character`s as `other`, using + /// `Character.==` (i.e. Unicode canonical equivalence). + public func charactersEqual( + to other: some Sequence + ) -> Bool +} +``` + + +#### Canonical equivalence and ordering + +`UTF8Span` can perform Unicode canonical equivalence checks (i.e. the semantics of `String.==` and `Character.==`). + +```swift +extension UTF8Span { + /// Whether `self` is equivalent to `other` under Unicode Canonical + /// Equivalence. + public func isCanonicallyEquivalent( + to other: UTF8Span + ) -> Bool + + /// Whether `self` orders less than `other` under Unicode Canonical + /// Equivalence using normalized code-unit order (in NFC). + public func canonicallyPrecedes( + _ other: UTF8Span + ) -> Bool +} +``` + +#### Extracting sub-spans + +Slicing a `UTF8Span` is nuanced and depends on the caller's desired use. They can only be sliced at scalar-aligned code unit offsets or else it will break the valid-UTF8 invariant. Furthermore, if the caller desires consistent grapheme breaking behavior without externally managing grapheme breaking state, they must be sliced along `Character` boundaries. For this reason, we have exposed slicing as `prefix` and `suffix` operations on `UTF8Span`'s iterators instead of `Span`'s' `extracting` methods. + +### Queries + +`UTF8Span` checks at construction time and remembers whether its contents are all ASCII. Additional checks can be requested and remembered. + +```swift +extension UTF8Span { + /// Returns whether contents are known to be all-ASCII. A return value of + /// `true` means that all code units are ASCII. A return value of `false` + /// means there _may_ be non-ASCII content. + /// + /// ASCII-ness is checked and remembered during UTF-8 validation, so this + /// is often equivalent to is-ASCII, but there are some situations where + /// we might return `false` even when the content happens to be all-ASCII. + /// + /// For example, a UTF-8 span generated from a `String` that at some point + /// contained non-ASCII content would report false for `isKnownASCII`, even + /// if that String had subsequent mutation operations that removed any + /// non-ASCII content. + public var isKnownASCII: Bool { get } + + /// Do a scan checking for whether the contents are all-ASCII. + /// + /// Updates the `isKnownASCII` bit if contents are all-ASCII. + public mutating func checkForASCII() -> Bool + + /// Returns whether the contents are known to be NFC. This is not + /// always checked at initialization time and is set by `checkForNFC`. + public var isKnownNFC: Bool { get } + + /// Do a scan checking for whether the contents are in Normal Form C. + /// When the contents are in NFC, canonical equivalence checks are much + /// faster. + /// + /// `quickCheck` will check for a subset of NFC contents using the + /// NFCQuickCheck algorithm, which is faster than the full normalization + /// algorithm. However, it cannot detect all NFC contents. + /// + /// Updates the `isKnownNFC` bit. + public mutating func checkForNFC( + quickCheck: Bool + ) -> Bool +} +``` + +### `UTF8Span` from `String` + +We propose adding `utf8Span` properties to `String` and `Substring`, in line with [SE-0456](0456-stdlib-span-properties.md): + +```swift +extension String { + public var utf8Span: UTF8Span { borrowing get } +} +extension Substring { + public var utf8Span: UTF8Span { borrowing get } +} +``` + + +### `Span`-like functionality + +A `UTF8Span` is similar to a `Span`, but with the valid-UTF8 invariant and additional information such as `isASCII`. We propose a way to get a `Span` from a `UTF8Span` as well as some methods directly on `UTF8Span`: + +``` +extension UTF8Span { + public var isEmpty: Bool { get } + + public var span: Span { get } +} +``` + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a new version of the standard library and runtime. + +## Future directions + +### Streaming grapheme breaking + +Grapheme-breaking, which identifies where the boundaries between `Character`s are, is more complex than scalar decoding. Grapheme breaking can be ran from any scalar-aligned position, either with a given state from having processed previous scalars, or with a "fresh" state (as though that position were the start of new content). + +While the code units in a `UTF8Span` are always scalar-aligned (in order to be validly encoded), whether a span is grapheme-cluster aligned depends on its intended use. For example, `AttributedString` stores its content using rope-like storage, in which the entire content is a sequence of spans were each individual span is scalar-aligned but not necessarily grapheme-cluster aligned. + +A potential approach to exposing this functionality is to make the stdlib's `GraphemeBreakingState` public and define API for finding grapheme-breaks. + +```swift +extension Unicode { + public struct GraphemeBreakingState: Sendable, Equatable { + public init() + } +} +``` + +One approach is to add API to the grapheme breaking state so that the state can find the next break (while updating itself). Another is to pass grapheme breaking state to an iterator on UTF8Span, like below: + +```swift +extension UTF8Span { + public struct GraphemeBreakIterator: ~Escapable { + public var codeUnits: UTF8Span + public var currentCodeUnitOffset: Int + public var state: Unicode.GraphemeBreakingState + + public init(_ span: UTF8Span) + + public init(_ span: UTF8Span, using state: Unicode.GraphemeBreakingState) + + public mutating func next() -> Bool + + public mutating func previous() -> Bool + + + public mutating func skipForward() + + public mutating func skipForward(by n: Int) + + public mutating func skipBack() + + public mutating func skipBack(by n: Int) + + public mutating func reset( + roundingBackwardsFrom i: Int, using: Unicode.GraphemeBreakingState + ) + + public mutating func reset( + roundingForwardsFrom i: Int, using: Unicode.GraphemeBreakingState + ) + + public mutating func reset( + uncheckedAssumingAlignedTo i: Int, using: Unicode.GraphemeBreakingState + ) + + public func prefix() -> UTF8Span + public func suffix() -> UTF8Span + } +``` + + +### More alignments and alignment queries + +Future API could include word iterators (either [simple](https://www.unicode.org/reports/tr18/#Simple_Word_Boundaries) or [default](https://www.unicode.org/reports/tr18/#Default_Word_Boundaries)), line iterators, etc. + +Similarly, we could add API directly to `UTF8Span` for testing whether a given code unit offset is suitably aligned (including scalar or grapheme-cluster alignment checks). + +### `~=` and other operators + +`UTF8Span` supports both binary equivalence and Unicode canonical equivalence. For example, a textual format parser using `UTF8Span` might operate in terms of binary equivalence for processing the textual format itself and then in terms of Unicode canonical equivalnce when interpreting the content of the fields. + +We are deferring making any decision on what a "default" comparison semantics should be as future work, which would include defining a `~=` operator (which would allow one to switch over a `UTF8Span` and match against literals). + +It may also be the case that it makes more sense for a library or application to define wrapper types around `UTF8Span` which can define `~=` with their preferred comparison semantics. + + +### Creating `String` copies + +We could add an initializer to `String` that makes an owned copy of a `UTF8Span`'s contents. Such an initializer can skip UTF-8 validation. + +Alternatively, we could defer adding anything until more of the `Container` protocol story is clear. + +### Normalization + +Future API could include checks for whether the content is in a particular normal form (not just NFC). + +### UnicodeScalarView and CharacterView + +Like `Span`, we are deferring adding any collection-like types to non-escapable `UTF8Span`. Future work could include adding view types that conform to a new `Container`-like protocol. + +See "Alternatives Considered" below for more rationale on not adding `Collection`-like API in this proposal. + +### More algorithms + +We propose equality checks (e.g. `scalarsEqual`), as those are incredibly common and useful operations. We have (tentatively) deferred other algorithms until non-escapable collections are figured out. + +However, we can add select high-value algorithms if motivated by the community. + +### More validation API + +Future API could include a way to find and classify UTF-8 encoding errors in arbitrary byte sequences, beyond just `Span`. + +We could propose something like: + +```swift +extension UTF8 { + public static func findFirstError( + _ s: some Sequence + ) -> UTF8.EncodingError? + + public static func findAllErrors( + _ s: some Sequence + ) -> some Sequence? +``` + +We are leaving this as future work. It also might be better formulated in line with a segemented-storage `Container`-like protocol instead of `some Sequence`. + +For now, developers can validate UTF-8 and diagnose the location and type of error using `UTF8Span`'s validating initializer, which takes a `Span`. This is similar to how developers do UTF-8 validation [in Rust](https://doc.rust-lang.org/std/str/fn.from_utf8.html). + +### Transcoded iterators, normalized iterators, case-folded iterators, etc + +We could provide lazily transcoded, normalized, case-folded, etc., iterators. If we do any of these for `UTF8Span`, we should consider adding equivalents views on `String`, `Substring`, etc. + +### Regex or regex-like support + +Future API additions would be to support `Regex`es on `UTF8Span`. We'd expose grapheme-level semantics, scalar-level semantics, and introduce byte-level semantics. + +Another future direction could be to add many routines corresponding to the underlying operations performed by the regex engine, which would be useful for parser-combinator libraries who wish to expose `String`'s model of Unicode by using the stdlib's accelerated implementation. + +### Track other bits + +Future work include tracking whether the contents are NULL-terminated (useful for C bridging), whether the contents contain any newlines or only a single newline at the end (useful for accelerating Regex `.`), etc. + +### Putting more API on String + +`String` would also benefit from the query API, such as `isKnownNFC` and corresponding scan methods. Because a string may be a lazily-bridged instance of `NSString`, we don't always have the bits available to query or set, but this may become viable pending future improvements in bridging. + +### Generalize printing and logging facilities + +Many printing and logging protocols and facilities operate in terms of `String`. They could be generalized to work in terms of UTF-8 bytes instead, which is important for embedded. + +## Alternatives considered + +### Problems arising from the unsafe init + +The combination of the unsafe init on `UTF8Span` and the copying init on `String` creates a new kind of easily-accesible backdoor to `String`'s security and safety, namely the invariant that it holds validly encoded UTF-8 when in native form. + +Currently, String is 100% safe outside of crazy custom subclass shenanigans (only on ObjC platforms) or arbitrarily scribbling over memory (which is true of all of Swift). Both are highly visible and require writing many lines of advanced-knowledge code. + +Without these two API, it is in theory possible to skip validation and produce a String instance of the [indirect contiguous UTF-8](https://forums.swift.org/t/piercing-the-string-veil/21700) flavor through a custom subclass of NSString. But, it is only available on Obj-C platforms and involves creating a custom subclass of `NSString`, having knowledge of lazy bridging internals (which can and sometimes do change from release to release of Swift), and writing very specialized code. The product would be an unsafe lazily bridged instance of `String`, which could more than offset any performance gains from the workaround itself. + +With these two API, you can get to UB via a: + +```swift +let codeUnits = unsafe UTF8Span(unsafeAssumingValidUTF8: bytes) +... +String(copying: codeUnits) +``` + +We are (very) weakly in favor of keeping the unsafe init, because there are many low-level situations in which the valid-UTF8 invariant is held by the system itself (such as a data structure using a custom allocator). + + + +### Invalid start / end of input UTF-8 encoding errors + +Earlier prototypes had `.invalidStartOfInput` and `.invalidEndOfInput` UTF8 validation errors to communicate that the input was perhaps incomplete or not slices along scalar boundaries. In this scenario, `.invalidStartOfInput` is equivalent to `.unexpectedContinuation` with the range's lower bound equal to 0 and `.invalidEndOfInput` is equivalent to `.truncatedScalar` with the range's upper bound equal to `count`. + +This was rejected so as to not have two ways to encode the same error. There is no loss of information and `.unexpectedContinuation`/`.truncatedScalar` with ranges are more semantically precise. + +### An unsafe UTF8 Buffer Pointer type + +An [earlier pitch](https://forums.swift.org/t/pitch-utf-8-processing-over-unsafe-contiguous-bytes/69715) proposed an unsafe version of `UTF8Span`. Now that we have `~Escapable`, a memory-safe `UTF8Span` is better. + +### Alternatives to Iterators + +#### Functions + +A previous version of this pitch had code unit offset taking API directly on UTF8Span instead of using iterators as proposed. This lead to a large number of unweildy API. For example, instead of: + +```swift +extension UTF8Span.UnicodeScalarIterator { + public mutating func next() -> Unicode.Scalar? { } +} +``` + +we had: + +```swift +extension UTF8Span { +/// Decode the `Unicode.Scalar` starting at `i`. Return it and the start of + /// the next scalar. + /// + /// `i` must be scalar-aligned. + public func decodeNextScalar( + _ i: Int + ) -> (Unicode.Scalar, nextScalarStart: Int) + + /// Decode the `Unicode.Scalar` starting at `i`. Return it and the start of + /// the next scalar. + /// + /// `i` must be scalar-aligned. + /// + /// This function does not validate that `i` is within the span's bounds; + /// this is an unsafe operation. + public func decodeNextScalar( + unchecked i: Int + ) -> (Unicode.Scalar, nextScalarStart: Int) + + /// Decode the `Unicode.Scalar` starting at `i`. Return it and the start of + /// the next scalar. + /// + /// `i` must be scalar-aligned. + /// + /// This function does not validate that `i` is within the span's bounds; + /// this is an unsafe operation. + /// + /// + /// This function does not validate that `i` is scalar-aligned; this is an + /// unsafe operation if `i` isn't. + public func decodeNextScalar( + uncheckedAssumingAligned i: Int + ) -> (Unicode.Scalar, nextScalarStart: Int) +} +``` + +Every operation had `unchecked:` and `uncheckedAssumingAligned:` variants, which were needed to implement higher-performance constructs such as iterators and string processing features (including the `Regex` engine). + +This API made the caller manage the scalar-alignment invariant, while the iterator-style API proposed maintains this invariant internally, allowing it to use the most efficient implementation. + +Scalar-alignment can still be checked and managed by the caller through the `reset` API, which safely round forwards or backwards as needed. And, for high performance use cases where the caller knows that a given position is appropriately aligned already (for example, revisiting a prior point in a string during `Regex` processing), there's the `reset(uncheckedAssumingAlignedTo:)` API available. + +#### View Collections + +Another forumulation of these operations could be to provide a collection-like API phrased in terms of indices. Because `Collection`s are `Escapable`, we cannot conform nested `View` types to `Collection` so these would not benefit from any `Collection`-generic code, algorithms, etc. + +A benefit of such `Collection`-like views is that it could help serve as adapter code for migration. Existing `Collection`-generic algorithms and methods could be converted to support `UTF8Span` via copy-paste-edit. That is, a developer could interact with `UTF8Span` ala: + +```swift +// view: UTF8Span.UnicodeScalarView +var curIdx = view.startIndex +while curIdx < view.endIndex { + let scalar = view[curIdx] + foo(scalar) + view.formIndex(after: &curIndex) +} +``` + +in addition to the iterator approach of: + +```swift +// iter: UTF8Span.UnicodeScalarIterator (or UTF8Span.UnicodeScalarView.Iterator) +while let scalar = iter.next() { + foo(scalar) +} +``` + +However, the iterator-based approach is the more efficient and direct way to work with a `UTF8Span`. Even if we had `Collection`-like API, we'd still implement a custom iterator type and advocate its use as the best way to interact with `UTF8Span`. The question is whether or not, for a given `FooIterator` we should additionally provide a `FooView`, `FooView.Index`, `FooView.SubSequence`, (possibly) `FooView.Slice`, etc. + +Idiomatic `Collection`-style interfaces support index interchange, even if "support" means reliably crashing after a dynamic check. Any idiomatic index-based interface would need to dynamically check for correct alignment in case the received index was derived from a different span. (There is a whole design space around smart indices and their tradeoffs, discussed in a [lengthy appendix](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md#appendix-index-and-slicing-design-considerations) in the Span proposal). + +This means that `UTF8Span.UnicodeScalarView.subscript` would have to check for scalar alignment of its given index, as it does not know whether it originally produced the passed index or not. Similarly, `index(after:)`, `index(before:)`, `index(_:offsetBy:)`, etc., would make these checks on every call. + +If we want to give the developer access to efficient formulations of index-style interfaces, we'd additionally propose `uncheckedAssumingAligned:` variants of nearly every method: `subscript(uncheckedAssumingAligned i:)`, `index(uncheckedAssumingAlignedAfter:)`, `index(uncheckedAssumingAlignedBefore:)`, `index(uncheckedAssumingAligned:offsetBy:)`, etc.. This also undermines the value of having an adapter to existing code patterns. + +If we do provide view adapter code, the API could look a little different in that `UnicodeScalarIterator` is called `UnicodeScalarView.Iterator`, `prefix/suffix` are slicing, and the `reset()` functionality is expressed by slicing the view before creating an iterator. However, this would also have the effect of scattering the efficient API use pattern across multiple types, intermingled with inefficient or ill-advised adaptor interfaces which have the more idiomatic names. + +Finally, in the future there will likely be some kind of `Container` protocol for types that can vend segments of contiguous storage. In our case, the segment type is `UTF8Span`, while the element is decoded from the underlying UTF-8. It's likely easier and more straightforward to retrofit or deprecate a single `UnicodeScalarIterator` type than a collection of types interrelated to each other. + +## Acknowledgments + +Karoy Lorentey, Karl, Geordie_J, and fclout, contributed to this proposal with their clarifying questions and discussions. + + + diff --git a/proposals/0465-nonescapable-stdlib-primitives.md b/proposals/0465-nonescapable-stdlib-primitives.md new file mode 100644 index 0000000000..798025fef9 --- /dev/null +++ b/proposals/0465-nonescapable-stdlib-primitives.md @@ -0,0 +1,886 @@ +# Standard Library Primitives for Nonescapable Types + +* Proposal: [SE-0465](0465-nonescapable-stdlib-primitives.md) +* Authors: [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [Doug Gregor](https://github.com/douggregor) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [Improving Swift performance predictability: ARC improvements and ownership control][Roadmap] +* Implementation: https://github.com/swiftlang/swift/pull/73258 +* Review: ([Acceptance](https://forums.swift.org/t/accepted-se-0465-standard-library-primitives-for-nonescapable-type/78637)) ([Review](https://forums.swift.org/t/se-0465-standard-library-primitives-for-nonescapable-types/78310)) ([Pitch](https://forums.swift.org/t/pitch-nonescapable-standard-library-primitives/77253)) + +[Roadmap]: https://forums.swift.org/t/a-roadmap-for-improving-swift-performance-predictability-arc-improvements-and-ownership-control/54206 +[Pitch]: https://forums.swift.org/t/pitch-nonescapable-standard-library-primitives/77253 + + + +Related proposals: + +- [SE-0377] `borrowing` and `consuming` parameter ownership modifiers +- [SE-0390] Noncopyable structs and enums +- [SE-0426] `BitwiseCopyable` +- [SE-0427] Noncopyable generics +- [SE-0429] Partial consumption of noncopyable values +- [SE-0432] Borrowing and consuming pattern matching for noncopyable types +- [SE-0437] Noncopyable Standard Library Primitives +- [SE-0446] Nonescapable Types +- [SE-0447] `Span`: Safe Access to Contiguous Storage +- [SE-0452] Integer Generic Parameters +- [SE-0453] `InlineArray`, a fixed-size array +- [SE-0456] Add `Span`-providing Properties to Standard Library Types + + + +[SE-0370]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0370-pointer-family-initialization-improvements.md +[SE-0377]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0377-parameter-ownership-modifiers.md +[SE-0390]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md +[SE-0426]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0426-bitwise-copyable.md +[SE-0427]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md +[SE-0429]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0429-partial-consumption.md +[SE-0432]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0432-noncopyable-switch.md +[SE-0437]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0437-noncopyable-stdlib-primitives.md +[SE-0446]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md +[SE-0447]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md +[SE-0452]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0452-integer-generic-parameters.md +[SE-0453]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md +[SE-0456]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0456-stdlib-span-properties.md + +### Table of Contents + + * [Introduction](#introduction) + * [Motivation](#motivation) + * [Proposed Solution](#proposed-solution) + * [Nonescapable optionals](#nonescapable-optionals) + * [Nonescapable Result](#nonescapable-result) + * [Retrieving the memory layout of nonescapable types](#retrieving-the-memory-layout-of-nonescapable-types) + * [Lifetime management](#lifetime-management) + * [Metatype comparisons](#metatype-comparisons) + * [Object identifiers](#object-identifiers) + * [Odds and ends](#odds-and-ends) + * [Detailed Design](#detailed-design) + * [Inferred lifetime behavior of nonescapable enum types](#inferred-lifetime-behavior-of-nonescapable-enum-types) + * [Inferred lifetime behavior of Optional's notational conveniences](#inferred-lifetime-behavior-of-optionals-notational-conveniences) + * [protocol ExpressibleByNilLiteral](#protocol-expressiblebynilliteral) + * [enum Optional](#enum-optional) + * [enum Result](#enum-result) + * [enum MemoryLayout](#enum-memorylayout) + * [Lifetime Management](#lifetime-management-1) + * [Metatype equality](#metatype-equality) + * [struct ObjectIdentifier](#struct-objectidentifier) + * [ManagedBufferPointer equatability](#managedbufferpointer-equatability) + * [Making indices universally available on unsafe buffer pointers](#making-indices-universally-available-on-unsafe-buffer-pointers) + * [Buffer pointer operations on Slice](#buffer-pointer-operations-on-slice) + * [Source compatibility](#source-compatibility) + * [ABI compatibility](#abi-compatibility) + * [Note on protocol generalizations](#note-on-protocol-generalizations) + * [Alternatives Considered](#alternatives-considered) + * [Future Work](#future-work) + * [Acknowledgements](#acknowledgements) + +## Introduction + +This document proposes to allow `Optional` and `Result` to hold instances of nonescapable types, and continues the work of adding support for noncopyable and nonescapable types throughout the Swift Standard Library. + + +## Motivation + +[SE-0437] started integrating noncopyable types into our Standard Library abstractions, by generalizing existing APIs and introducing new ones. In the time since that proposal, [SE-0446] has introduced nonescapable types to Swift, adding a new direction of generalization. + +This proposal continues the work of [SE-0437] by extending some basic constructs to support nonescapable types, where it is already possible to do so. For now, we are focusing on further generalizing a subset of the constructs covered by [SE-0437]: `MemoryLayout`, `Optional`, and `Result` types. Our immediate aim is to unblock the use of nonescapable types, especially in API surfaces. We also smooth out some minor inconsistencies that [SE-0437] has left unresolved. + +Like before, our aim is to implement these generalizations with as little disruption as possible. Existing code implicitly assumes copyability and escapability, and it needs to continue working as before. + +## Proposed Solution + +This proposal is focusing on achieving the following results: + +- Allow `Optional` to wrap nonescapable types, itself becoming conditionally escapable. +- Do the same for `Result`, allowing its success case to hold a nonescapable item. +- Generalize `MemoryLayout` to allow querying basic information on the memory layout of nonescapable types. +- Continue generalizing basic lifetime management functions; introduce a new `extendLifetime()` function that avoids a closure argument. +- Allow generating `ObjectIdentifier` instances for noncopyable and/or nonescapable metatypes. +- Allow comparing noncopyable and nonescapable metatypes for equality. + +We also propose to fix a handful of minor copyability-related omissions that have been discovered since [SE-0437] was accepted: + +- Allow `ManagedBufferPointer` instances to be equatable even when `Element` is noncopyable. +- Make the `Unsafe[Mutable]BufferPointer.indices` property universally available for all `Element` types. +- Generalize select operations on unsafe buffer pointer slices, to restore consistency with the same operations on the buffer pointers themselves. + +### Nonescapable optionals + +We want `Optional` to support wrapping all Swift types, whether or not they're copyable or escapable. This means that `Optional` needs to become conditionally escapable, depending on the escapability of its `Wrapped` type. + +`Optional` must itself become nonescapable when it is wrapping a nonescapable type, and such optional values need to be subject to precisely the same lifetime constraints as their wrapped item: we cannot allow the act of wrapping a nonescapable value in an optional to allow that value to escape its intended context. + +There are many ways to construct optional values in Swift: for example, we can explicitly invoke the factory for the `.some` case, we can rely on Swift's implicit optional promotion rules, or we can invoke the default initializer. We propose to generalize all of these basic/primitive mechanisms to support nonescapable use. For instance, given a non-optional `Span` value, this code exercises these three basic ways of constructing non-nil nonescapable optionals: + +```swift +func sample(_ span: Span) { + let a = Optional.some(span) // OK, explicit case factory + let b: Optional = span // OK, implicit optional promotion + let c = Optional(span) // OK, explicit initializer invocation +} +``` + +`a`, `b`, and `c` hold the same span instance, and their lifetimes are subject to the same constraints as the original span -- they can be used within the context of the `sample` function, but they cannot escape outside of it. (At least not without explicit lifetime dependency annotations, to be introduced in the future.) + +Of course, it also needs to be possible to make empty `Optional` values that do not hold anything. We have three basic ways to do that: we can explicitly invoke the factory for the `none` case, we can reach for the special `nil` literal, or (for `var`s) we can rely on implicit optional initialization. This proposal generalizes all three mechanisms to support noncopyable wrapped types: + +```swift +func sample(_ span: Span) { + var d: Span? = .none // OK, explicit factory invocation + var e: Span? = nil // OK, nil literal expression + var f: Span? // OK, implicit nil default value +} +``` + +Empty optionals of nonescapable types are still technically nonescapable, but they aren't inherently constrained to any particular context -- empty optionals are born with "immortal" (or "static") lifetimes, i.e., they have no lifetime dependencies, and so they are allowed to stay around for the entire execution of a Swift program. Nil optionals can be passed to any operation that takes a nonescapable optional, no matter what expectations it may dictate about its lifetime dependencies; they can also be returned from any function that returns a nonescapable optional. (Note though that Swift does not yet provide a stable way to define such functions.) + +Of course, we also expect to be able to reassign variables, rebinding them to a new value. Reassignments of local variables are allowed to arbitrarily change lifetime dependencies. There is no expectation that the lifetime dependencies of the new value have any particular relation to the old: local variable reassignments can freely "narrow" or "widen" dependencies, as they see fit. + +For instance, the code below initializes an optional variable to an immortal nil value; it then assigns it a new value that has definite lifetime constraints; and finally it turns it back to an immortal nil value: + +```swift +func sample(_ span: Span) { + var maybe: Span? = nil // immortal + maybe = span // definite lifetime + maybe = nil // immortal again +} +``` + +(Assigning `span` to `maybe` is not an escape, as the local variable will be destroyed before the function returns, even without the subsequent reassignment.) + +This flexibility will not necessarily apply to other kinds of variables, like stored properties in custom nonescapable structs, global variables, or computed properties -- I expect those variables to carry specific lifetime dependencies that cannot vary through reassignment. (For example, a global variable of a nonescapable type may be required to hold immortal values only.) However, for now, we're limiting our reasoning to local variables. + +Of course, an optional is of limited use unless we are able to decide whether it contains a value, and (if so) to unwrap it and look at its contents. We need to be able to operate on nonescapable optionals using the familiar basic mechanisms: + + - `switch` and `if case`/`guard case` statements that pattern match over them: + + ```swift + // Variant 1: Full pattern matching + func count(of maybeSpan: Span?) -> Int { + switch maybeSpan { + case .none: return 0 + case .some(let span): return span.count + } + } + + // Variant 2: Pattern matching with optional sugar + func count(of maybeSpan: Span?) -> Int { + switch maybeSpan { + case nil: return 0 + case let span?: return span.count + } + } + ``` + + - The force-unwrapping `!` special form, and its unsafe cousin, the Standard Library's `unsafelyUnwrapped` property. + + ```swift + func count(of maybeSpan: Span?) -> Int { + if case .none = maybeSpan { return 0 } + return maybeSpan!.count + } + ``` + +- The optional chaining special form `?`: + + ```swift + func count(of maybeSpan: Span?) -> Int { + guard let c = maybeSpan?.count else { return 0 } + return c + } + ``` + +- Optional bindings such as `if let` or `guard let` statements: + + ```swift + func count(of maybeSpan: Span?) -> Int { + guard let span = maybeSpan else { return 0 } + return span.count + } + ``` + +These variants all work as expected. To avoid escapability violations, unwrapping the nonescapable optional results in a value with precisely the same lifetime dependencies as the original optional value. This applies to all forms of unwrapping, including pattern matching forms that bind copies of associated values to new variables, like `let span` above -- the resulting `span` value always has the same lifetime as the optional it comes from. + +The standard `Optional` type has custom support for comparing optional instances against `nil` using the traditional `==` operator, whether or not the wrapped type conforms to `Equatable`. [SE-0437] generalized this mechanism for noncopyable wrapped types, and it is reasonable to extend this to also cover the nonescapable case: + +```swift +func count(of maybeSpan: Span?) -> Int { + if maybeSpan == nil { return 0 } // OK! + return maybeSpan!.count +} +``` + +This core set of functionality makes nonescapable optionals usable, but it does not yet enable the use of more advanced APIs. Eventually, we'd also like to use the standard `Optional.map` function (and similar higher-order functions) to operate on (or to return) nonescapable optional types, as in the example below: + +```swift +func sample(_ maybeArray: Array?) { + // Assuming `Array.storage` returns a nonescapable `Span`: + let maybeSpan = maybeArray.map { $0.storage } + ... +} +``` + +These operations require precise reasoning about lifetime dependencies though, so they have to wait until we have a stable way to express lifetime annotations on their definitions. We expect lifetime semantics to become an integral part of the signatures of functions dealing with nonescapable entities -- for the simplest cases they can often remain implicit, but for something like `map` above, we'll need to explicitly describe how the lifetime of the function's result relates to the lifetime of the result of the function argument. We need to defer this work until we have the means to express such annotations in the language. + +One related omission from the list of generalizations above is the standard nil-coalescing operator `??`. This is currently defined as follows (along with another variant that returns an `Optional`): + +```swift +func ?? ( + optional: consuming T?, + defaultValue: @autoclosure () throws -> T +) rethrows -> T +``` + +To generalize this to also allow nonescapable `T` types, we'll need to specify that the returned value's lifetime is tied to the _intersection_ of the lifetime of the left argument and the lifetime of the result of the right argument (a function). We aren't currently able to express that, so this generalization has to be deferred as well until the advent of such a language feature. + +### Nonescapable `Result` + +We generalize `Result` along the same lines as `Optional`, allowing its `success` case to wrap a nonescapable value. For now, we need to mostly rely on Swift's general enum facilities to operate on nonescapable `Result` values: switch statements, case factories, pattern matching, associated value bindings etc. + +Important convenience APIs such as `Result.init(catching:)` or `Result.map` will need to require escapability until we introduce a way to formally specify lifetime dependencies. This is unfortunate, but it still enables intrepid Swift developers to experiment with defining interfaces that take (or perhaps even return!) `Result` values. + +However, we are already able to generalize a couple of methods: `get` and the error-mapping utility `mapError`. + +```swift +func sample(_ res: Result, E>) -> Int { + guard let span = try? res.get() else { return 42 } + return 3 * span.count + 9 +} +``` + +Like unwrapping an `Optional`, calling `get()` on a nonescapable `Result` returns a value whose lifetime requirements exactly match that of the original `Result` instance -- the act of unwrapping a result cannot allow its content to escape its intended context. + +### Retrieving the memory layout of nonescapable types + +This proposal generalizes `enum MemoryLayout` to support retrieving information about the layout of nonescapable types: + +```swift +print(MemoryLayout>.size) // ⟹ 16 +print(MemoryLayout>.stride) // ⟹ 16 +print(MemoryLayout>.alignment) // ⟹ 8 +``` + +(Of course, the values returned will vary depending on the target architecture.) + +The information returned is going to be of somewhat limited use until we generalize unsafe pointer types to support nonescapable pointees, which this proposal does not include -- but there is no reason to delay this work until then. + +To usefully allow pointers to nonescapable types, we'll need to assign precise lifetime semantics to their `pointee` (and pointer dereferencing in general), and we'll most likely also need a way to allow developers to unsafely override the resulting default lifetime semantics. This requires explicit lifetime annotations, and as such, that work is postponed to a future proposal. + +### Lifetime management + +We once again generalize the `withExtendedLifetime` family of functions, this time to support calling them on nonescapable values. + +```swift +let span = someArray.storage +withExtendedLifetime(span) { span in + // `someArray` is being actively borrowed while this closure is running +} +// At this point, `someArray` may be ready to be mutated +``` + +We've now run proposals to generalize `withExtendedLifetime` for (1) typed throws, (2) noncopyable inputs and results, and (3) nonescapable inputs. It is getting unwieldy to keep having to tweak these APIs, especially since in actual practice, `withExtendedLifetime` is most often called with an empty closure, to serve as a sort of fence protecting against early destruction. The closure-based design of these interfaces are no longer fitting the real-life practices of Swift developers. These functions were originally designed to be used with a non-empty closure, like in the example below: + +```swift +withExtendedLifetime(obj) { + weak var ref = obj + foo(ref!) +} +``` + +In most cases, the formulation we actually recommend these days is to use a defer statement, with the function getting passed an empty closure: + +```swift +weak var ref = obj +defer { withExtendedLifetime(obj) {} } // Ugh 😖 +foo(ref!) +``` + +These functions clearly weren't designed to accommodate this widespread practice. To acknowledge and embrace this new style, we propose to introduce a new public Standard Library function that simply extends the lifetime of whatever variable it is given: + +```swift +func extendLifetime(_ x: borrowing T) +``` + +This allows `defer` incantations like the one above to be reformulated into a more readable form: + +```swift +// Slightly improved reality +weak var ref = obj +defer { extendLifetime(obj) } +foo(ref!) +``` + +To avoid disrupting working code, this proposal does not deprecate the existing closure-based functions in favor of the new `extendLifetime` operation. (Introducing the new function will still considerably reduce the need for future Swift releases to continue repeatedly generalizing the existing functions -- for example, to allow async use, or to allow nonescapable results.) + +### Metatype comparisons + +Swift's metatypes do not currently conform to `Equatable`, but the Standard Library still provides top-level `==` and `!=` operators that implement the expected equality relation. Previously, these operators only worked on metatypes of `Copyable` and `Escapable` types; we propose to relax this requirement. + +```swift +print(Atomic.self == Span.self) // ⟹ false +``` + +The classic operators support existential metatypes `Any.Type`; the new variants also accept generalized existentials: + +```swift +let t1: any (~Copyable & ~Escapable).Type = Atomic.self +let t2: any (~Copyable & ~Escapable).Type = Span.self +print(t1 != t2) // ⟹ true +print(t1 == t1) // ⟹ true +``` + +### Object identifiers + +The `ObjectIdentifier` construct is primarily used to generate a Comparable/Hashable value that identifies a class instance. However, it is also able to identify metatypes: + +```swift +let id1 = ObjectIdentifier(Int.self) +let id2 = ObjectIdentifier(String.self) +print(id1 == id2) // ⟹ false +``` + +[SE-0437] did not generalize this initializer; we can now allow it to work with both noncopyable and nonescapable types: + +```swift +import Synchronization + +let id3 = ObjectIdentifier(Atomic.self) // OK, noncopyable input type +let id4 = ObjectIdentifier(Span.self) // OK, nonescapable input type +print(id3 == id4) // ⟹ false +``` + +The object identifier of a noncopyable/nonescapable type is still a regular copyable and escapable identifier -- for instance, it can be compared against other ids and hashed. + +### Odds and ends + +[SE-0437] omitted generalizing the `Equatable` conformance of `ManagedBufferPointer`; this proposal allows comparing `ManagedBufferPointer` instances for equality even if their `Element` happens to be noncopyable. + +[SE-0437] kept the `indices` property of unsafe buffer pointer types limited to cases where `Element` is copyable. This proposal generalizes `indices` to be also available on buffer pointers of noncopyable elements. (In the time since the original proposal, [SE-0447] has introduced a `Span` type that ships with an unconditional `indices` property, and [SE-0453] followed suit by introducing `InlineArray` with the same property. It makes sense to also provide this interface on buffer pointers, for consistency.) `indices` is useful for iterating through these collection types, especially until we ship a new iteration model that supports noncopyable/nonescapable containers. + +Finally, [SE-0437] neglected to generalize any of the buffer pointer operations that [SE-0370] introduced on the standard `Slice` type. In this proposal, we correct this omission by generalizing the handful of operations that can support noncopyable result elements: `moveInitializeMemory(as:fromContentsOf:)`, `bindMemory(to:)`, `withMemoryRebound(to:_:)`, and `assumingMemoryBound(to:)`. `Slice` itself continues to require its `Element` to be copyable (at least for now), preventing the generalization of other operations. + +## Detailed Design + +Note that Swift provides no way to define the lifetime dependencies of a function's nonescapable result, nor to set lifetime constraints on input parameters. Until the language gains an official way to express such constraints, the Swift Standard Library will define the APIs generalized in this proposal with unstable syntax that isn't generally available. In this text, we'll be using an illustrative substitute -- the hypothetical `@_lifetime` attribute. We will loosely describe its meaning as we go. + +Note: The `@_lifetime` attribute is not real; it is merely a didactic placeholder. The eventual lifetime annotations proposal may or may not propose syntax along these lines. We expect the Standard Library to immediately switch to whatever syntax Swift eventually embraces, as soon as it becomes available. + +### Inferred lifetime behavior of nonescapable enum types + +[SE-0446] has introduced the concept of a nonescapable enum type to Swift. While that proposal did not explicitly spell this out, this inherently included a set of implicit lifetime rules for the principal language features that interact with enum types: enum construction using case factories and pattern matching. To generalize `Optional` and `Result`, we need to understand how these implicit inference rules work for enum types with a single nonescapable associated value. + +1. When constructing an enum case with a single nonescapable associated value, the resulting enum value is inferred to carry precisely the same lifetime dependencies as the origional input. +2. Pattern matching over such an enum case exposes the nonescapable associated value, inferring precisely the same lifetime dependencies for it as the original enum. + +```swift +enum Foo { + case a(T) + case b +} + +func test(_ array: Array) { + let span = array.span + let foo = Foo.a(span) // (1) + switch foo { + case .a(let span2): ... // (2) + case .b: ... + } +} +``` + +In statement (1), `foo` is defined to implicitly copy the lifetime dependencies of `span`; neither variable can escape the body of the `test` function. The let binding in the pattern match on `.a` in statement (2) creates `span2` with exactly the same lifetime dependencies as `foo`. + +(We do not describe the implicit semantics of enum cases with _multiple_ nonescapable associated values here, as they are relevant to neither `Optional` nor `Result`.) + +### Inferred lifetime behavior of `Optional`'s notational conveniences + +The `Optional` enum comes with a rich set of notational conveniences that are built directly into the language. This proposal extends these conveniences to work on nonescapable optionals; therefore it inherently needs to introduce new implicit lifetime inference rules, along the same lines as the two existing once we described above: + +1. The result of implicit optional promotion of a nonescapable value is a nonescapable optional carrying precisely the same lifetime dependencies as the original input. +2. The force-unwrapping special form `!` and the optional chaining special form `?` both implicitly infer the lifetime dependencies of the wrapped value (if any) by directly copying those of the optional. + +### `protocol ExpressibleByNilLiteral` + +In order to generalize `Optional`, we need the `ExpressibleByNilLiteral` protocol to support nonescapable conforming types. By definition, the `nil` form needs to behave like a regular, escapable value; accordingly, the required initializer needs to establish "immortal" or "static" lifetime semantics on the resulting instance. + +```swift +protocol ExpressibleByNilLiteral: ~Copyable, ~Escapable { + @_lifetime(immortal) // Illustrative syntax + init(nilLiteral: ()) +} +``` + +In this illustration, `@_lifetime(immortal)` specifies that the initializer places no constraints on the lifetime of its result. We expect a future proposal to define a stable syntax for expressing such lifetime dependency constraints. + +Preexisting types that conform to `ExpressibleByNilLiteral` are all escapable, and escapable values always have immortal lifetimes, by definition. Therefore, initializer implementations in existing conformances already satisfy this new refinement of the initializer requirement -- it only makes a difference in the newly introduced `~Escapable` case. + +### `enum Optional` + +We generalize `Optional` to allow nonescapable wrapped types in addition to noncopyable ones. + +```swift +enum Optional: ~Copyable, ~Escapable { + case none + case some(Wrapped) +} + +extension Optional: Copyable where Wrapped: Copyable & ~Escapable {} +extension Optional: Escapable where Wrapped: Escapable & ~Copyable {} +extension Optional: BitwiseCopyable where Wrapped: BitwiseCopyable & ~Escapable {} +extension Optional: Sendable where Wrapped: ~Copyable & ~Escapable & Sendable {} +``` + +To allow the use of the `nil` syntax with nonescapable optional types, we generalize `Optional`'s conformance to `ExpressibleByNilLiteral`: + +```swift +extension Optional: ExpressibleByNilLiteral +where Wrapped: ~Copyable & ~Escapable { + @_lifetime(immortal) // Illustrative syntax + init(nilLiteral: ()) +} +``` + +As discussed above, `nil` optionals have no lifetime dependencies, and they continue to work like escapable values. + +We need to generalize the existing unlabeled initializer to support the nonescapable case. When passed a nonescapable entity, the initializer creates an optional that has precisely the same lifetime dependencies as the original entity. Once again, Swift has not yet provided a stable way to express this dependency; so to define such an initializer, the Standard Library needs to use some unstable mechanism. We use the hypothetical `@_lifetime(copying some)` syntax to do this -- this placeholder notation is intended to reflect that the lifetime dependencies of the result are copied verbatim from the `some` argument. + +```swift +extension Optional where Wrapped: ~Copyable & ~Escapable { + @_lifetime(copying some) // Illustrative syntax + init(_ some: consuming Wrapped) +} +``` + +As we've seen, the language also has built-in mechanisms for constructing `Optional` values that avoid invoking this initializer: it implements implicit optional promotions and explicit case factories. When given values of nonescapable types, these methods also _implicitly_ result in the result's lifetime dependencies being copied directly from the original input. + +Swift offers many built-in ways for developers to unwrap optional values: we have force unwrapping, optional chaining, pattern matching, optional bindings, etc. Many of these rely on direct compiler support that is already able to properly handle lifetime matters; but the stdlib also includes its own forms of unwrapping, and these require some API changes. + +In this proposal, we generalize `take()` to work on nonescapable optionals. It resets `self` to nil and returns its original value with precisely the same lifetime dependency as we started with. The `nil` value it leaves behind is still constrained to the same lifetime -- we do not have a way for a mutating function to affect the lifetime dependencies of its `self` argument. + +```swift +extension Optional where Wrapped: ~Copyable & ~Escapable { + @_lifetime(copying self) // Illustrative syntax + mutating func take() -> Self +} +``` + +We are also ready to generalize the `unsafelyUnwrapped` property: + +```swift +extension Optional where Wrapped: ~Escapable { + @_lifetime(copying self) // Illustrative syntax + var unsafelyUnwrapped: Wrapped { get } +} +``` + +This property continues to require copyability for now, as supporting noncopyable wrapped types requires the invention of new accessors that hasn't happened yet. + +As noted above, we defer generalizing the nil-coalescing operator `??`. We expect to tackle it when it becomes possible to express the lifetime dependency of its result as an intersection of the lifetimes of its left argument and the _result_ of the right argument (an autoclosure). We also do not attempt to generalize similar higher-order API, like `Optional.map` or `.flatMap`. + +The Standard Library provides special support for comparing arbitrary optional values against `nil`. We generalize this mechanism to support nonescapable cases: + +```swift +extension Optional where Wrapped: ~Copyable & ~Escapable { + static func ~=( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool + + static func ==( + lhs: borrowing Wrapped?, + rhs: _OptionalNilComparisonType + ) -> Bool + + static func !=( + lhs: borrowing Wrapped?, + rhs: _OptionalNilComparisonType + ) -> Bool + + static func ==( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool + + static func !=( + lhs: _OptionalNilComparisonType, + rhs: borrowing Wrapped? + ) -> Bool +} +``` + +### `enum Result` + +For `Result`, this proposal concentrates on merely allowing the success case to contain a nonescapable value. + +```swift +enum Result { + case success(Success) + case failure(Failure) +} + +extension Result: Copyable where Success: Copyable & ~Escapable {} +extension Result: Escapable where Success: Escapable & ~Copyable {} +extension Result: Sendable where Success: Sendable & ~Copyable & ~Escapable {} +``` + +We postpone generalizing most of the higher-order functions that make `Result` convenient to use, as we currently lack the means to reason about lifetime dependencies for such functions. But we are already able to generalize the one function that does not have complicated lifetime semantics: `mapError`. + +```swift +extension Result where Success: ~Copyable & ~Escapable { + @_lifetime(copying self) // Illustrative syntax + consuming func mapError( + _ transform: (Failure) -> NewFailure + ) -> Result +} +``` + +The returned value has the same lifetime constraints as the original `Result` instance. + +We can also generalize the convenient `get()` function, which is roughly equivalent to optional unwrapping: + +```swift +extension Result where Success: ~Copyable & ~Escapable { + @_lifetime(copying self) // Illustrative syntax + consuming func get() throws(Failure) -> Success +} +``` + +In the non-escapable case, this function returns a value with a lifetime that precisely matches the original `Result`. + +### `enum MemoryLayout` + +Swift is not yet ready to introduce pointers to nonescapable values -- we currently lack the ability to assign proper lifetime semantics to the addressed items. + +However, a nonescapable type does still have a well-defined memory layout, and it makes sense to allow developers to query the size, stride, and alignment of such instances. This information is associated with the type itself, and it is independent of the lifetime constraints of its instances. Therefore, we can generalize the `MemoryLayout` enumeration to allow its subject to be a nonescapable type: + +```swift +enum MemoryLayout +: ~BitwiseCopyable, Copyable, Escapable {} + +extension MemoryLayout where T: ~Copyable & ~Escapable { + static var size: Int { get } + static var stride: Int { get } + static var alignment: Int { get } +} + +extension MemoryLayout where T: ~Copyable & ~Escapable { + static func size(ofValue value: borrowing T) -> Int + static func stride(ofValue value: borrowing T) -> Int + static func alignment(ofValue value: borrowing T) -> Int +} +``` + +### Lifetime Management + +[SE-0437] generalized the `withExtendedLifetime` family of functions to support extending the lifetime of noncopyable entities. This proposal further generalizes these to also allow operating on nonescapable entities: + +```swift +func withExtendedLifetime< + T: ~Copyable & ~Escapable, + E: Error, + Result: ~Copyable +>( + _ x: borrowing T, + _ body: () throws(E) -> Result +) throws(E) -> Result + +func withExtendedLifetime< + T: ~Copyable & ~Escapable, + E: Error, + Result: ~Copyable +>( + _ x: borrowing T, + _ body: (borrowing T) throws(E) -> Result +) throws(E) -> Result +``` + +Note that the `Result` is still required to be escapable. + +We also propose the addition of a new function variant that eliminates the closure argument, to better accommodate the current best practice of invoking these functions in `defer` blocks: + +```swift +func extendLifetime(_ x: borrowing T) +``` + +### Metatype equality + +Swift's metatypes do not conform to `Equatable`, but the Standard Library does implement the `==`/`!=` operators over them: + +```swift +func == (t0: Any.Type?, t1: Any.Type?) -> Bool { ... } +func != (t0: Any.Type?, t1: Any.Type?) -> Bool { ... } +``` + +Note how these are defined on optional metatype existentials, typically relying on implicit optional promotion. We propose to generalize these to support metatypes of noncopyable and/or nonescapable types: + +```swift +func == ( + t0: (any (~Copyable & ~Escapable).Type)?, + t1: (any (~Copyable & ~Escapable).Type)? +) -> Bool { ... } +func != ( + t0: (any (~Copyable & ~Escapable).Type)?, + t1: (any (~Copyable & ~Escapable).Type)? +) -> Bool { ... } +``` + +### `struct ObjectIdentifier` + +The `ObjectIdentifier` construct is primarily used to generate a `Comparable`/`Hashable` value that identifies a class instance. However, it is also able to generate hashable type identifiers: + +```swift +extension ObjectIdentifier { + init(_ x: Any.Type) +} +``` + +We propose to generalize this initializer to allow generating identifiers for noncopyable and nonescapable types as well, using generalized metatype existentials: + +```swift +extension ObjectIdentifier { + init(_ x: any (~Copyable & ~Escapable).Type) +} +``` + +### `ManagedBufferPointer` equatability + +The `ManagedBufferPointer` type conforms to `Equatable`; its `==` implementation works by comparing the identity of the class instances it is referencing. [SE-0437] has generalized the type to allow a noncopyable `Element` type, but it did not generalize this specific conformance. This proposal aims to correct this oversight: + +```swift +extension ManagedBufferPointer: Equatable where Element: ~Copyable { + static func ==( + lhs: ManagedBufferPointer, + rhs: ManagedBufferPointer + ) -> Bool +} +``` + +Managed buffer pointers are pointer types -- as such, they can be compared whether or not they are addressing a buffer of copyable items. + +(Note: conformance generalizations like this can cause compatibility issues when newly written code is deployed on older platforms that pre-date the generalization. We do not expect this to be an issue in this case, as the generalization is compatible with the implementations we previously shipped.) + +### Making `indices` universally available on unsafe buffer pointers + +[SE-0437] kept the `indices` property of unsafe buffer pointer types limited to cases where `Element` is copyable. In the time since that proposal, [SE-0447] has introduced a `Span` type that ships with an unconditional `indices` property, and [SE-0453] followed suit by introducing `InlineArray` with the same property. For consistency, it makes sense to also allow developers to unconditionally access `Unsafe[Mutable]BufferPointer.indices`, whether or not `Element` is copyable. + +```swift +extension UnsafeBufferPointer where Element: ~Copyable { + var indices: Range { get } +} + +extension UnsafeMutableBufferPointer where Element: ~Copyable { + var indices: Range { get } +} +``` + +This allows Swift programmers to iterate over the indices of a buffer pointer with simpler syntax, independent of what `Element` they are addressing: + +```swift +for i in buf.indices { + ... +} +``` + +We consider `indices` to be slightly more convenient than the equivalent expression `0 ..< buf.count`. + +(Of course, we are still planning to introduce direct support for for-in loops over noncopyable/nonescapable containers, which will provide a far more flexible solution. `indices` is merely a stopgap solution to bide us over until we are ready to propose that.) + +### Buffer pointer operations on `Slice` + +Finally, to address an inconsistency that was left unresolved by [SE-0437], we generalize a handful of buffer pointer operations that are defined on buffer slices. This consists of the following list, originally introduced in [SE-0370]: + +- Initializing a slice of a mutable raw buffer pointer by moving items out of a typed mutable buffer pointer: + + ```swift + extension Slice where Base == UnsafeMutableRawBufferPointer { + func moveInitializeMemory( + as type: T.Type, + fromContentsOf source: UnsafeMutableBufferPointer + ) -> UnsafeMutableBufferPointer + } + ``` + +- Binding memory of raw buffer pointer slices: + + ```swift + extension Slice where Base == UnsafeMutableRawBufferPointer { + func bindMemory( + to type: T.Type + ) -> UnsafeMutableBufferPointer + } + + extension Slice where Base == UnsafeRawBufferPointer { + func bindMemory( + to type: T.Type + ) -> UnsafeBufferPointer + } + ``` + +- Temporarily rebinding memory of a (typed or untyped, mutable or immutable) buffer pointer slice for the duration of a function call: + + ```swift + extension Slice where Base == UnsafeMutableRawBufferPointer { + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeMutableBufferPointer) throws(E) -> Result + ) throws(E) -> Result + } + + extension Slice where Base == UnsafeRawBufferPointer { + func withMemoryRebound( + to type: T.Type, + _ body: (UnsafeBufferPointer) throws(E) -> Result + ) throws(E) -> Result + } + + extension Slice { + func withMemoryRebound< + T: ~Copyable, E: Error, Result: ~Copyable, Element + >( + to type: T.Type, + _ body: (UnsafeBufferPointer) throws(E) -> Result + ) throws(E) -> Result + where Base == UnsafeBufferPointer + + public func withMemoryRebound< + T: ~Copyable, E: Error, Result: ~Copyable, Element + >( + to type: T.Type, + _ body: (UnsafeMutableBufferPointer) throws(E) -> Result + ) throws(E) -> Result + where Base == UnsafeMutableBufferPointer + } + ``` + +- Finally, converting a slice of a raw buffer pointer into a typed buffer pointer, assuming its memory is already bound to the correct type: + + ```swift + extension Slice where Base == UnsafeMutableRawBufferPointer { + func assumingMemoryBound( + to type: T.Type + ) -> UnsafeMutableBufferPointer + } + + extension Slice where Base == UnsafeRawBufferPointer { + func assumingMemoryBound( + to type: T.Type + ) -> UnsafeBufferPointer + } + ``` + +All of these forward to operations on the underlying base buffer pointer that have already been generalized in [SE-0437]. These changes are simply restoring feature parity between buffer pointer and their slices, where possible. (`Slice` still requires its `Element` to be copyable, which limits generalization of other buffer pointer APIs defined on it.) + +These generalizations are limited to copyability for now. We do expect that pointer types (including buffer pointers) will need to be generalized to allow non-escapable pointees; however, we have to postpone that work until we are able to precisely reason about lifetime requirements. + + + +## Source compatibility + +Like [SE-0437], this proposal also heavily relies on the assurance that removing the assumption of escapability on these constructs will not break existing code that used to rely on the original, escaping definitions. [SE-0437] has explored a few cases where this may not be the case; these can potentially affect code that relies on substituting standard library API with its own implementations. With the original ungeneralized definitions, such custom reimplementations could have shadowed the originals. However, this may no longer be the case with the generalizations included, and this can lead to ambiguous function invocations. + +This proposal mostly touches APIs that were already changed by [SE-0437], and that reduces the likelihood of it causing new issues. That said, it does generalize some previously unchanged interfaces that may provide new opportunities for such shadowing declarations to cause trouble. + +Like previously, we do have engineering options to mitigate such issues in case we do encounter them in practice: for example, we can choose to amend Swift's shadowing rules to ignore differences in throwing, noncopyability, and nonescapability, or we can manually patch affected definitions to make the expression checker consider them to be less specific than any custom overloads. + +## ABI compatibility + +The introduction of support for nonescapable types is (in general) a compile-time matter, with minimal (or even zero) runtime impact. This greatly simplifies the task of generalizing previously shipping types for use in nonescapable contexts. Another simplifying aspect is that while it can be relatively easy for classic Swift code to accidentally copy a value, it tends to be rare for functions to accidentally _escape_ their arguments -- previous versions of a function are less likely to accidentally violate nonescapability than noncopyability. + +The implementation of this proposal adopts the same approaches as [SE-0437] to ensure forward and backward compatibility of newly compiled (and existing) binaries, including the Standard Library itself. We expect that code that exercises the new features introduced in this proposal will be able to run on earlier versions of the Swift stdlib -- to the extent that noncopyable and/or nonescapable types are allowed to backdeploy. + +[SE-0437] has already arranged ABI compatibility symbols to get exported as needed to support ABI continuity. It has also already reimplemented most of the entry points that this proposal touches, in a way that forces them to get embedded in client binaries. This allows the changes in this proposal to get backdeployed without any additional friction. + +Like its precursor, this proposal does assume that the `~Copyable`/`~Escapable` generalization of the `ExpressibleByNilLiteral` protocol will not have an ABI impact on existing conformers of it. However, it goes a step further, by also adding a lifetime annotation on the protocol's initializer requirement; this requires that such annotations must not interfere with backward/forward binary compatibility, either. (E.g., it requires that such lifetime annotations do not get mangled into exported symbol names.) + +### Note on protocol generalizations + +Like [SE-0437], this proposal mostly avoids generalizing standard protocols, with the sole exception of `ExpressibleByNilLiteral`, which has now been generalized to allow both noncopyable and nonescapable conforming types. + +As a general rule, protocol generalizations like that may not be arbitrarily backdeployable -- it seems likely that we'll at least need to support limiting the availability of _conformance_ generalizations, if not generalizations of the protocol itself. In this proposal, we follow [SE-0437] in assuming that this potential issue will not apply to the specific case of `ExpressibleByNilLiteral`, because of its particularly narrow use case. Our experience with [SE-0437] is reinforcing this assumption, but it is still possible there is an ABI back-compatibility issue that we haven't uncovered yet. In the (unlikely, but possible) case we do discover such an issue, we may need to do extra work to patch protocol conformances in earlier stdlibs, or we may decide to limit the use of `nil` with noncopyable/nonescapable optionals to recent enough runtimes. + +To illustrate the potential problem, let's consider `Optional`'s conformance to `Equatable`: + +```swift +extension Optional: Equatable where Wrapped: Equatable { + public static func ==(lhs: Wrapped?, rhs: Wrapped?) -> Bool { + switch (lhs, rhs) { + case let (l?, r?): return l == r + case (nil, nil): return true + default: return false + } + } +} +``` + +This conformance is currently limited to copyable and escapable cases, and it is using the classic, copying form of the switch statement, with `case let (l?, r?)` semantically making full copies of the two wrapped values. We do intend to soon generalize the `Equatable` protocol to support noncopyable and/or nonescapable conforming types. When that becomes possible, `Optional` will want to immediately embrace this generalization, to allow comparing two noncopyable/nonescapable instances for equality: + +```swift +extension Optional: Equatable where Wrapped: Equatable & ~Copyable & ~Escapable { + public static func ==(lhs: borrowing Wrapped?, rhs: borrowing Wrapped?) -> Bool { + switch (lhs, rhs) { + case let (l?, r?): return l == r + case (nil, nil): return true + default: return false + } + } +} +``` + +On the surface, this seems like a straightforward change. Unfortunately, switching to `borrowing` arguments changes the semantics of the implementation, converting the original copying switch statement to the borrowing form introduced by [SE-0432]. This new variant avoids copying wrapped values to compare them, enabling the use of this function on noncopyable data. However, the old implementation of `==` did assume (and exercise!) copyability, so the `Equatable` conformance cannot be allowed to dispatch to `==` implementations that shipped in Standard Library releases that predate this generalization. + +To mitigate such problems, we'll either need to retroactively patch/substitute the generic implementations in previously shipping stdlibs, or we need to somehow limit availability of the generalized conformance, without affecting the original copyable/escapable one. + +This issue is more pressing for noncopyable cases, as preexisting implementations are far more likely to perform accidental copying than to accidentally escape their arguments. + +Our hypothesis is that `ExpressibleByNilLiteral` conformances are generally free of such issues. + +## Alternatives Considered + +Most of the changes proposed here follow directly from the introduction of nonescapable types. The API generalizations follow the patterns established by [SE-0437], and are largely mechanical in nature. For the most part, the decision points aren't about the precise form of any particular change, but more about what changes we are ready to propose _right now_. + +The single exception is the `extendLifetime` function, which is a brand new API; it comes from our experience using (and maintaining) the `withExtendedLifetime` function family. + +## Future Work + +For the most part, this proposal is concentrating on resolving the first item from [SE-0437]'s wish list (nonescapable `Optional` and `Result`), and it adds minor coherency improvements to the feature set we shipped there. + +Most other items listed as future work in that proposal continue to remain on our agenda. The advent of nonescapable types extends this list with additional items, including the following topics: + +1. We need to define stable syntax for expressing lifetime dependencies as explicit annotations, and we need to define what semantics we apply by default on functions that do not explicitly specify these. + +2. We will need an unsafe mechanism to override lifetime dependencies of nonescapable entities. We also expect to eventually need to allow unsafe bit casting to and from nonescapable types. + +3. We will need to allow pointer types to address nonescapable items: `UnsafePointer`, `UnsafeBufferPointer` type families, perhaps `ManagedBuffer`. The primary design task here is to decide what lifetime semantics we want to assign to pointer dereferencing operations, including mutations. + +4. Once we have pointers, we will also need to allow the construction of generic containers of nonescapable items, with some Sequence/Collection-like capabilities (iteration, indexing, generic algorithms, etc.). We expect the noncopyable/nonescapable container model to heavily rely on the `Span` type, which we intend to use as the basic unit of iteration, providing direct access to contiguous storage chunks. For containers of nonescapables in particular, this means we'll also need to generalize `Span` to allow it to capture nonescapable elements. + +5. We'll want to generalize most of the preexisting standard library protocols to allow nonescapable conforming types and (if possible) associated types. This is in addition to supporting noncopyability. This work will require adding carefully considered lifetime annotations on protocol requirements, while also carefully maintaining seamless forward/backward compatibility with the currently shipping protocol versions. This is expected to take several proposals; in some cases, it may include carefully reworking existing semantic requirements to better match noncopyable/nonescapable use cases. Some protocols may not be generalizable without breaking existing code; in those cases, we may need to resort to replacing or augmenting them with brand-new protocols. However, protocol generalizations for nonescapables are generally expected to be a smoother process than it is for noncopyables. + +## Acknowledgements + +Many people contributed to the discussions that led to this proposal. We'd like to especially thank the following individuals for their continued, patient and helpful input: + +- Alejandro Alonso +- Steve Canon +- Ben Cohen +- Kavon Farvardin +- Doug Gregor +- Joe Groff +- Megan Gupta +- Tim Kientzle +- Guillaume Lessard +- John McCall +- Tony Parker +- Ben Rimmington +- Andrew Trick +- Rauhul Varma diff --git a/proposals/0466-control-default-actor-isolation.md b/proposals/0466-control-default-actor-isolation.md new file mode 100644 index 0000000000..5a1e975d25 --- /dev/null +++ b/proposals/0466-control-default-actor-isolation.md @@ -0,0 +1,255 @@ +# Control default actor isolation inference + +* Proposal: [SE-0466](0466-control-default-actor-isolation.md) +* Authors: [Holly Borla](https://github.com/hborla), [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Active review (July 8...15, 2025)** +* Vision: [Improving the approachability of data-race safety](/visions/approachable-concurrency.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-control-default-actor-isolation-inference/77482))([review](https://forums.swift.org/t/se-0466-control-default-actor-isolation-inference/78321))([acceptance](https://forums.swift.org/t/accepted-se-0466-control-default-actor-isolation-inference/78926))([amendment pitch](https://forums.swift.org/t/pitch-amend-se-0466-se-0470-to-improve-isolation-inference/79854))([amendment review](https://forums.swift.org/t/amendment-se-0466-control-default-actor-isolation-inference/80994)) + +## Introduction + +This proposal introduces a new compiler setting for inferring `@MainActor` isolation by default within the module to mitigate false-positive data-race safety errors in sequential code. + +## Motivation + +> Note: This motivation section was adapted from the [vision for approachable data-race safety](https://github.com/hborla/swift-evolution/blob/approachable-concurrency-vision/visions/approachable-concurrency.md#mitigating-false-positive-data-race-safety-errors-in-sequential-code). Please see the vision document for extended motivation. + +A lot of code is effectively “single-threaded”. For example, most executables, such as apps, command-line tools, and scripts, start running on the main actor and stay there unless some part of the code does something concurrent (like creating a `Task`). If there isn’t any use of concurrency, the entire program will run sequentially, and there’s no risk of data races — every concurrency diagnostic is necessarily a false positive! It would be good to be able to take advantage of that in the language, both to avoid annoying programmers with unnecessary diagnostics and to reinforce progressive disclosure. Many people get into Swift by writing these kinds of programs, and if we can avoid needing to teach them about concurrency straight away, we’ll make the language much more approachable. + +The easiest and best way to model single-threaded code is with a global actor. Everything on a global actor runs sequentially, and code that isn’t isolated to that actor can’t access the data that is. All programs start running on the global actor `MainActor`, and if everything in the program is isolated to the main actor, there shouldn’t be any concurrency errors. + +Unfortunately, it’s not quite that simple right now. Writing a single-threaded program is surprisingly difficult under the Swift 6 language mode. This is because Swift 6 defaults to a presumption of concurrency: if a function or type is not annotated or inferred to be isolated, it is treated as non-isolated, meaning it can be used concurrently. This default often leads to conflicts with single-threaded code, producing false-positive diagnostics in cases such as: + +- global and static variables, +- conformances of main-actor-isolated types to non-isolated protocols, +- class deinitializers, +- overrides of non-isolated superclass methods in a main-actor-isolated subclass, and +- calls to main-actor-isolated functions from the platform SDK. + +## Proposed solution + +This proposal allows code to opt in to being “single-threaded” by default, on a module-by-module basis. A new `-default-isolation` compiler flag specifies the default isolation within the module, and a corresponding `SwiftSetting` method specifies the default isolation per target within a Swift package. + +This would change the default isolation rule for unannotated code in the module: rather than being non-isolated, and therefore having to deal with the presumption of concurrency, the code would instead be implicitly isolated to `@MainActor`. Code imported from other modules would be unaffected by the current module’s choice of default. When the programmer really wants concurrency, they can request it explicitly by marking a function or type as `nonisolated` (which can be used on any declaration as of [SE-0449](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0449-nonisolated-for-global-actor-cutoff.md)), or they can define it in a module that doesn’t default to main-actor isolation. + +## Detailed design + +### Specifying default isolation per module + +#### `-default-isolation` compiler flag + +The `-default-isolation` flag can be used to control the default actor isolation for all code in the module. The only valid arguments to `-default-isolation` are `MainActor` and `nonisolated`. It is an error to specify both `-default-isolation MainActor` and `-default-isolation nonisolated`. If no `-default-isolation` flag is specified, the default isolation for the module is `nonisolated`. + +#### `SwiftSetting.defaultIsolation` method + +The following method on `SwiftSetting` can be used to specify the default actor isolation per target in a Swift package manifest: + +```swift +extension SwiftSetting { + @available(_PackageDescription, introduced: 6.2) + public static func defaultIsolation( + _ globalActor: MainActor.Type?, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting +} +``` + +The only valid values for the `globalActor` argument are `MainActor.self` and `nil`. The `nil` argument corresponds to `nonisolated`; `.defaultIsolation(nil)` will default to `nonisolated` within the module. When no `.defaultIsolation` setting is specified, the default isolation within the module is `nonisolated`. + +### Default actor isolation inference + +When the default actor isolation is specified as `MainActor`, declarations are inferred to be `@MainActor`-isolated by default. Default isolation does not apply in the following cases: + +* Declarations with explicit actor isolation +* Declarations with inferred actor isolation from a superclass, overridden method, protocol conformance, or member propagation +* All declarations inside an `actor` type, including static variables, methods, initializers, and deinitializers +* Declarations that cannot have global actor isolation, including typealiases, import statements, enum cases, and individual accessors +* Declarations whose primary definition directly conforms to a protocol that inherits `SendableMetatype` +* Declarations that are types nested within a nonisolated type + +The following code example shows the inferred actor isolation in comments given the code is built with `-default-isolation MainActor`: + +```swift +// @MainActor +func f() {} + +// @MainActor +class C { + // @MainActor + init() { ... } + + // @MainActor + deinit { ... } + + // @MainActor + struct Nested { ... } + + // @MainActor + static var value = 10 +} + +@globalActor +actor MyActor { + // nonisolated + init() { ... } + + // nonisolated + deinit { ... } + + // nonisolated + static let shared = MyActor() +} + +@MyActor +protocol P {} + +// @MyActor +struct S: P { + // @MyActor + func f() { ... } +} + +nonisolated protocol Q: Sendable { } + +// nonisolated +struct S2: Q { + // nonisolated + struct Inner { } + + // @MyActor + struct IsolatedInner: P +} + +// @MainActor +struct S3 { } + +extension S3: Q { } +``` + +This proposal does not change the default isolation inference rules for closures. Non-Sendable closures and closures passed to `Task.init` already have the same isolation as the enclosing context by default. When specifying `MainActor` isolation by default in a module, non-`@Sendable` closures and `Task.init` closures will have inferred `@MainActor` isolation when the default `@MainActor` inference rules apply to the enclosing context: + +```swift +// Built with -default-isolation MainActor + +// @MainActor +func f() { + Task { // @MainActor in + ... + } + + Task.detached { // nonisolated in + ... + } +} + +nonisolated func g() { + Task { // nonisolated in + ... + } +} +``` + +## Source compatibility + +Changing the default actor isolation for a given module or source file is a source incompatible change. The default isolation will remain the same for existing projects unless they explicitly opt into `@MainActor` inference by default via `-default-isolation MainActor` or `defaultIsolation(MainActor.self)` in a package manifest. + +## ABI compatibility + +This proposal has no ABI impact on existing code. + +## Implications on adoption + +This proposal does not change the adoption implications of adding `@MainActor` to a declaration that was previously `nonisolated` and vice versa. The source and ABI compatibility implications of changing actor isolation are documented in the Swift migration guide's [Library Evolution](https://github.com/apple/swift-migration-guide/blob/29d6e889e3bd43c42fe38a5c3f612141c7cefdf7/Guide.docc/LibraryEvolution.md#main-actor-annotations) article. + +## Future directions + +### Specify build settings per file + +There are some build settings that are applicable on a per-file basis, including specifying default actor isolation and controlling diagnostic behavior. We could consider allowing settings in individual files which the setting should apply to by introducing a `#pragma`-like compiler directive. This idea has been [pitched separately](https://forums.swift.org/t/pitch-compilersettings-a-top-level-statement-for-enabling-compiler-flags-locally-in-a-specific-file/77994). + +## Alternatives considered + +### Allow defaulting isolation to a custom global actor + +The `-default-isolation` flag could allow a custom global actor as the argument, and the `SwiftSetting` API could be updated to accept a string that represents a custom global actor in the target. + +This proposal only supports `MainActor` because any other global actor does not help with progressive disclosure. It has the opposite effect - it forces asynchrony on any main-actor-isolated caller. However, there's nothing in this proposal that prohibits generalizing these settings to supporting arbitrary global actors in the future if a compelling use case arises. + +### Infer `MainActor` by default as an upcoming feature + +Instead of introducing a separate mode for configuring default actor isolation inference, the default isolation could be changed to be `MainActor` under an upcoming feature that is enabled by default in a future Swift language mode. The upcoming feature approach was not taken because `MainActor` isolation is the wrong default for many kinds of modules, including libraries that offer APIs that can be used from any isolation domain, and highly-concurrent server applications. + +Similarly, a future language mode could enable main actor isolation by default, and require an opt out for using `nonisolated` as the default actor isolation. However, as the Swift package ecosystem grows, it's more likely for `nonisolated` to be the more common default amongst projects. If we discover that not to be true in practice, nothing in this proposal prevents changing the default actor isolation in a future language mode. + +See the approachable data-race safety vision document for an [analysis on the risks of introducing a language dialect](https://github.com/hborla/swift-evolution/blob/approachable-concurrency-vision/visions/approachable-concurrency.md#risks-of-a-language-dialect) for default actor isolation. + +### Alternative to `SendableMetatype` for suppressing main-actor inference + +The protocols to which a type conforms can affect the isolation of the type. Conforming to a global-actor-isolated protocol can infer global-actor isolatation for the type. When the default actor isolation is `MainActor`, it is valuable for protocols to be able to push inference toward keeping conforming types `nonisolated`, for example because conforming types are meant to be usable from any isolation domain. + +In this proposal, inheritance from `SendableMetatype` (introduced in [SE-0470](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0470-isolated-conformances.md)) is used as an indication that types conforming to the protocol should be `nonisolated`. The `SendableMetatype` marker protocol indicates when a type (but not necessarily its instances) can cross isolation domains, which implies that the type generally needs to be usable from any isolation domain. Additionally, protocols that inherit from `SendableMetatype` can only be meaningfully be used with nonisolated conformances, as discussed in SE-0470. Experience using default main actor isolation uncovered a number of existing protocols that reinforce the notion of `SendableMetatype` inheritance is a reasonable heuristic to indicate that a conforming type should be nonisolated: the standard library's [`CodingKey`](https://developer.apple.com/documentation/swift/codingkey) protocol inherits `Sendable` (which in turn inherits `SendableMetatype`) so a typical conformance will fail to compile with default main actor isolation: + +```swift +struct S: Codable { + var a: Int + + // error if CodingKeys is inferred to `@MainActor`. The conformance cannot be main-actor-isolated, and + // the requirements of the (nonisolated) CodingKey cannot be satisfied by main-actor-isolated members of + // CodingKeys. + enum CodingKeys: CodingKey { + case a + } +} +``` + +Other places that have similar issues with default main actor isolation include the [`Transferable`](https://developer.apple.com/documentation/coretransferable/transferable) protocol and the uses of key paths in the [`@Model` macro](https://developer.apple.com/documentation/swiftdata/model()). + +Instead of using `SendableMetatype` inheritance, this proposal could introduce new syntax for a protocol to explicitly indicate + +```swift +@nonisolatedConformingTypes +public protocol CodingKey { + // ... +} +``` + +This would make the behavior pushing conforming types toward `nonisolated` opt-in. However, it means that existing protocols (such as the ones mentioned above) would all need to adopt this spelling before code using default main actor isolation will work well. Given the strong semantic link between `SendableMetatype` and `nonisolated` conformances and types, the proposed rule based on `SendableMetatype` inheritance is likely to make more code work well with default main actor isolation. An explicit opt-in attribute like the above could be added at a later time if needed. + +### Use an enum for the package manifest API + +An alternative to using a `MainActor` metatype for the Swift package manifest API is to use an enum, e.g. + +```swift +public enum DefaultActorIsolation { + case mainActor + case nonisolated +} + +extension SwiftSetting { + @available(_PackageDescription, introduced: 6.2) + public static func defaultIsolation( + _ isolation: DefaultActorIsolation, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting +} + +// in a package manifest + +swiftSettings: [ + .defaultIsolation(.mainActor) +] +``` + +The enum approach introduces a different way of writing main actor isolation that does not involve the `MainActor` global actor type. The proposed design matches exactly the values used for `#isolation`, i.e. `MainActor.self` for main actor isolation and `nil` for `nonisolated`, which programmers are already familiar with. + +The primary argument for using an enum is that it can be extended in the future to support custom global actor types. This proposal deliberately puts supporting custom global actors in the alternatives considered and not future directions, because defaulting a module to a different global actor does not help improve progressive disclosure for concurrency. + +## Revision history + +* Changes in amendment review: + * Disable `@MainActor` inference when type conforms to a `SendableMetatype` protocol + +## Acknowledgments + +Thank you to John McCall for providing much of the motivation for this pitch in the approachable data-race safety vision document, and to Michael Gottesman for helping with the implementation. diff --git a/proposals/0467-MutableSpan.md b/proposals/0467-MutableSpan.md new file mode 100644 index 0000000000..846697c393 --- /dev/null +++ b/proposals/0467-MutableSpan.md @@ -0,0 +1,700 @@ +# MutableSpan and MutableRawSpan: delegate mutations of contiguous memory + +* Proposal: [SE-0467](0467-MutableSpan.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Joe Groff](https://github.com/jckarter) +* Status: **Implemented (Swift 6.2)** +* Roadmap: [BufferView Roadmap](https://forums.swift.org/t/66211) +* Implementation: [PR #79650](https://github.com/swiftlang/swift/pull/79650), [PR #80517](https://github.com/swiftlang/swift/pull/80517) +* Review: ([Pitch](https://forums.swift.org/t/pitch-mutablespan/77790)) ([Review](https://forums.swift.org/t/se-0467-mutablespan/78454)) ([Acceptance](https://forums.swift.org/t/accepted-se-0467-mutablespan/78875)) + +[SE-0446]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md +[SE-0447]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md +[SE-0456]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0456-stdlib-span-properties.md +[PR-2305]: https://github.com/swiftlang/swift-evolution/pull/2305 +[SE-0437]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0437-noncopyable-stdlib-primitives.md +[SE-0453]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md +[SE-0223]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0223-array-uninitialized-initializer.md +[SE-0176]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0176-enforce-exclusive-access-to-memory.md + +## Introduction + +We recently [introduced][SE-0447] the `Span` and `RawSpan` types, providing shared read-only access to borrowed memory. This proposal adds helper types to delegate mutations of exclusively-borrowed memory: `MutableSpan` and `MutableRawSpan`. + +## Motivation + +Many standard library container types can provide direct access to modify their internal representation. Up to now, it has only been possible to do so in an unsafe way. The standard library provides this unsafe functionality with closure-taking functions such as `withUnsafeMutableBufferPointer()` and `withContiguousMutableStorageIfAvailable()`. + +These functions have a few different drawbacks, most prominently their reliance on unsafe types, which makes them unpalatable in security-conscious environments. We continue addressing these issues with `MutableSpan` and `MutableRawSpan`, new non-copyable and non-escapable types that manage respectively mutations of typed and untyped memory. + +In addition to the new types, we will propose adding new API some standard library types to take advantage of `MutableSpan` and `MutableRawSpan`. + +## Proposed solution + +We introduced `Span` to provide shared read-only access to containers. The natural next step is to provide a similar capability for mutable access. A library whose API provides access to its internal storage makes a decision regarding the type of access it provides; it may provide read-only access or provide the ability to mutate its storage. That decision is made by the API author. If mutations were enabled by simply binding a `Span` value to a mutable binding (`var` binding or `inout` parameter), that decision would rest with the user of the API instead of its author. This explains why mutations must be modeled by a type separate from `Span`. + +Mutability requires exclusive access, per Swift's [law of exclusivity][SE-0176]. `Span` is copyable, and must be copyable in order to properly model read access under the law of exclusivity: a value can be simultaneously accessed through multiple read-only accesses. Exclusive access cannot be modeled with a copyable type, since a copy would represent an additional access, in violation of the law of exclusivity. This explains why the type which models mutations must be non-copyable. + +#### MutableSpan + +`MutableSpan` allows delegating mutations of a type's contiguous internal representation, by providing access to an exclusively-borrowed view of a range of contiguous, initialized memory. `MutableSpan`'s memory safety relies on guarantees that: +- it has exclusive access to the range of memory it represents, providing data race safety and enforced by `~Copyable`. +- the memory it represents will remain valid for the duration of the access, providing lifetime safety and enforced by `~Escapable`. +- each access is guarded by bounds checking, providing bounds safety. + +A `MutableSpan` provided by a container represents a mutation of that container, as an extended mutation access. Mutations are implemented by mutating functions and subscripts, which let the compiler statically enforce exclusivity. + +#### MutableRawSpan + +`MutableRawSpan` allows delegating mutations to memory representing possibly heterogeneously-typed values, such as memory intended for encoding. It makes the same safety guarantees as `MutableSpan`. A `MutableRawSpan` can be obtained from a `MutableSpan` whose `Element` is `BitwiseCopyable`. + +#### Extensions to standard library types + +The standard library will provide `mutableSpan` computed properties. These return a new lifetime-dependent `MutableSpan` instance, and that `MutableSpan` represents a mutation of the instance that provided it. The `mutableSpan` computed properties are the safe and composable replacements for the existing `withUnsafeMutableBufferPointer` closure-taking functions. For example, + +```swift +func(_ array: inout Array) { + var ms = array.mutableSpan + modify(&ms) // call function that mutates a MutableSpan + // array.append(2) // attempt to modify `array` would be an error here + _ = consume ms // access to `array` via `ms` ends here + array.append(1) +} +``` + +The `mutableSpan` computed property represents a case of lifetime relationships not covered until now. The `mutableSpan` computed properties proposed here will represent mutations of their callee. This relationship will be illustrated with a hypothetical `@_lifetime` attribute, which ties the lifetime of a return value to an input parameter in a specific way. + +Note: The `@_lifetime` attribute is not real; it is a placeholder. The eventual lifetime annotations proposal may or may not propose syntax along these lines. We expect that, as soon as Swift adopts a syntax do describe lifetime dependencies, the Standard Library will be modified to adopt that new syntax. + +```swift +extension Array { + public var mutableSpan: MutableSpan { + @_lifetime(inout self) + mutating get { ... } + } +} +``` + +Here, the lifetime of the returned `MutableSpan` is tied to an `inout` access of `self` (the `Array`.) As long as the returned instance exists, the source `Array` is being mutated, and no other access to the `Array` can occur. + +This lifetime relationship will apply to all the safe `var mutableSpan: MutableSpan` and `var mutableBytes: MutableRawSpan` properties described in this proposal. + +#### Slicing `MutableSpan` or `MutableRawSpan` instances + +An important category of use cases for `MutableSpan` and `MutableRawSpan` consists of bulk copying operations. Often times, such bulk operations do not necessarily start at the beginning of the span, thus having a method to select a sub-span is necessary. This means producing an instance derived from the callee instance. We adopt the nomenclature already introduced in [SE-0437][SE-0437], with a family of `extracting()` methods. + +```swift +extension MutableSpan where Element: ~Copyable { + @_lifetime(inout self) + public mutating func extracting(_ range: Range) -> Self +} +``` + +This function returns an instance of `MutableSpan` that represents a mutation of the same memory as represented by the callee. The callee can therefore no longer be accessed (read or mutated) while the returned value exists: + +```swift +var array = [1, 2, 3, 4, 5] +var span1 = array.mutableSpan +var span2 = span1.extracting(3..<5) +// neither array nor span1 can be accessed here +span2.swapAt(0, 1) +_ = consume span2 // explicitly end scope for `span2` +span1.swapAt(0, 1) +_ = consume span1 // explicitly end scope for `span1` +print(array) // [2, 1, 3, 5, 4] +``` + +As established in [SE-0437][SE-0437], the instance returned by the `extracting()` function does not share indices with the function's callee. + +## Detailed Design + +#### MutableSpan + +`MutableSpan` is a simple representation of a region of initialized memory. It is non-copyable in order to enforce exclusive access for mutations of its memory, as required by the law of exclusivity: + +````swift +@frozen +public struct MutableSpan: ~Copyable, ~Escapable { + internal var _start: UnsafeMutableRawPointer? + internal var _count: Int +} + +extension MutableSpan: @unchecked Sendable where Element: Sendable & ~Copyable {} +```` + +We store a `UnsafeMutableRawPointer` value internally in order to explicitly support reinterpreted views of memory as containing different types of `BitwiseCopyable` elements. Note that the the optionality of the pointer does not affect usage of `MutableSpan`, since accesses are bounds-checked and the pointer is only dereferenced when the `MutableSpan` isn't empty, when the pointer cannot be `nil`. + +Initializers, required for library adoption, will be proposed alongside [lifetime annotations][PR-2305]; for details, see "[Initializers](#initializers)" in the [future directions](#Directions) section. + +```swift +extension MutableSpan where Element: ~Copyable { + /// The number of initialized elements in this `MutableSpan`. + var count: Int { get } + + /// A Boolean value indicating whether the span is empty. + var isEmpty: Bool { get } + + /// The type that represents a position in a `MutableSpan`. + typealias Index = Int + + /// The range of indices valid for this `MutableSpan`. + var indices: Range { get } + + /// Accesses the element at the specified position. + subscript(_ index: Index) -> Element { borrow; mutate } + // accessor syntax from accessors roadmap (https://forums.swift.org/t/76707) + + /// Exchange the elements at the two given offsets + mutating func swapAt(_ i: Index, _ j: Index) + + /// Borrow the underlying memory for read-only access + var span: Span { @_lifetime(borrow self) borrowing get } +} +``` + +Like `Span` before it, `MutableSpan` does not conform to `Collection` or `MutableCollection`. These two protocols assume their conformers and elements are copyable, and as such are not compatible with a non-copyable type such as `MutableSpan`. A later proposal will consider generalized containers. + +The subscript uses a borrowing accessor for read-only element access, and a mutate accessor for element mutation. The read-only borrow is a read access to the entire `MutableSpan` for the duration of the access to the element. The `mutate` accessor is an exclusive access to the entire `MutableSpan` for the duration of the mutation of the element. + +`MutableSpan` uses offset-based indexing. The first element of a given span is always at offset 0, and its last element is always at position `count-1`. + +As a side-effect of not conforming to `Collection` or `Sequence`, `MutableSpan` is not directly supported by `for` loops at this time. It is, however, easy to use in a `for` loop via indexing: + +```swift +for i in myMutableSpan.indices { + mutatingFunction(&myMutableSpan[i]) +} +``` + +##### Bulk updates of a `MutableSpan`'s elements: + +We include functions to perform bulk copies of elements into the memory represented by a `MutableSpan`. Updating a `MutableSpan` from known-sized sources (such as `Collection` or `Span`) copies every element of a source. It is an error to do so when there is the span is too short to contain every element from the source. Updating a `MutableSpan` from `Sequence` or `IteratorProtocol` instances will copy as many items as possible, either until the input is empty or until the operation has updated the item at the last index. The bulk operations return the index following the last element updated. + +```swift +extension MutableSpan where Element: Copyable { + /// Updates every element of this span to the given value. + mutating func update( + repeating repeatedValue: Element + ) + + /// Updates the span's elements with the elements from the source + mutating func update( + from source: S + ) -> (unwritten: S.Iterator, index: Index) where S.Element == Element + + /// Updates the span's elements with the elements from the source + mutating func update( + from source: inout some IteratorProtocol + ) -> Index + + /// Updates the span's elements with every element of the source. + mutating func update( + fromContentsOf source: some Collection + ) -> Index +} + +extension MutableSpan where Element: ~Copyable + /// Updates the span's elements with every element of the source. + mutating func update( + fromContentsOf source: Span + ) -> Index + + /// Updates the span's elements with every element of the source. + mutating func update( + fromContentsOf source: borrowing MutableSpan + ) -> Index + + /// Updates the span's elements with every element of the source, + /// leaving the source uninitialized. + mutating func moveUpdate( + fromContentsOf source: UnsafeMutableBufferPointer + ) -> Index +} + +extension MutableSpan where Element: Copyable { + /// Updates the span's elements with every element of the source, + /// leaving the source uninitialized. + mutating func moveUpdate( + fromContentsOf source: Slice> + ) -> Index +} +``` + +##### Extracting sub-spans +These functions extract sub-spans of the callee. The first two perform strict bounds-checking. The last four return prefixes or suffixes, where the number of elements in the returned sub-span is bounded by the number of elements in the parent `MutableSpan`. + +```swift +extension MutableSpan where Element: ~Copyable { + /// Returns a span over the items within the supplied range of + /// positions within this span. + @_lifetime(inout self) + mutating public func extracting(_ bounds: Range) -> Self + + /// Returns a span over the items within the supplied range of + /// positions within this span. + @_lifetime(inout self) + mutating public func extracting(_ bounds: some RangeExpression) -> Self + + /// Returns a span containing the initial elements of this span, + /// up to the specified maximum length. + @_lifetime(inout self) + mutating public func extracting(first maxLength: Int) -> Self + + /// Returns a span over all but the given number of trailing elements. + @_lifetime(inout self) + mutating public func extracting(droppingLast k: Int) -> Self + + /// Returns a span containing the final elements of the span, + /// up to the given maximum length. + @_lifetime(inout self) + mutating public func extracting(last maxLength: Int) -> Self + + /// Returns a span over all but the given number of initial elements. + @_lifetime(inout self) + mutating public func extracting(droppingFirst k: Int) -> Self +} +``` + +##### Unchecked access to elements or sub-spans: + +The `subscript` and index-taking functions mentioned above always check the bounds of the `MutableSpan` before allowing access to the memory, preventing out-of-bounds accesses. We also provide unchecked variants of the `subscript`, the `swapAt()` and `extracting()` functions as alternatives in situations where repeated bounds-checking is costly and has already been performed: + +```swift +extension MutableSpan where Element: ~Copyable { + /// Accesses the element at the specified `position`. + /// + /// This subscript does not validate `position`; this is an unsafe operation. + /// + /// - Parameter position: The offset of the element to access. `position` + /// must be greater or equal to zero, and less than `count`. + @unsafe + subscript(unchecked position: Index) -> Element { borrow; mutate } + + /// Exchange the elements at the two given offsets + /// + /// This function does not validate `i` or `j`; this is an unsafe operation. + @unsafe + mutating func swapAt(unchecked i: Index, unchecked j: Index) + + /// Constructs a new span over the items within the supplied range of + /// positions within this span. + /// + /// This function does not validate `bounds`; this is an unsafe operation. + @unsafe + @_lifetime(inout self) + mutating func extracting(unchecked bounds: Range) -> Self + + /// Constructs a new span over the items within the supplied range of + /// positions within this span. + /// + /// This function does not validate `bounds`; this is an unsafe operation. + @unsafe + @_lifetime(inout self) + mutating func extracting(unchecked bounds: ClosedRange) -> Self +} +``` + +##### Interoperability with unsafe code + +```swift +extension MutableSpan where Element: ~Copyable { + /// Calls a closure with a pointer to the viewed contiguous storage. + func withUnsafeBufferPointer( + _ body: (_ buffer: UnsafeBufferPointer) throws(E) -> Result + ) throws(E) -> Result + + /// Calls a closure with a pointer to the viewed mutable contiguous + /// storage. + mutating func withUnsafeMutableBufferPointer( + _ body: (_ buffer: UnsafeMutableBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} + +extension MutableSpan where Element: BitwiseCopyable { + /// Calls a closure with a pointer to the underlying bytes of + /// the viewed contiguous storage. + func withUnsafeBytes( + _ body: (_ buffer: UnsafeRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result + + /// Calls a closure with a pointer to the underlying bytes of + /// the viewed mutable contiguous storage. + /// + /// Note: mutating the bytes may result in the violation of + /// invariants in the internal representation of `Element` + @unsafe + mutating func withUnsafeMutableBytes( + _ body: (_ buffer: UnsafeMutableRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} +``` +These functions use a closure to define the scope of validity of `buffer`, ensuring that the underlying `MutableSpan` and the binding it depends on both remain valid through the end of the closure. They have the same shape as the equivalents on `Array` because they fulfill the same function, namely to keep the underlying binding alive. + +#### MutableRawSpan + +`MutableRawSpan` is similar to `MutableSpan`, but represents untyped initialized bytes. `MutableRawSpan` specifically supports encoding and decoding applications. Its API supports `unsafeLoad(as:)` and `storeBytes(of: as:)`, as well as a variety of bulk copying operations. + +##### `MutableRawSpan` API: + +```swift +@frozen +public struct MutableRawSpan: ~Copyable, ~Escapable { + internal var _start: UnsafeMutableRawPointer? + internal var _count: Int +} + +extension MutableRawSpan: @unchecked Sendable +``` + +Initializers, required for library adoption, will be proposed alongside [lifetime annotations][PR-2305]; for details, see "[Initializers](#initializers)" in the [future directions](#Directions) section. + +```swift +extension MutableRawSpan { + /// The number of bytes in the span. + var byteCount: Int { get } + + /// A Boolean value indicating whether the span is empty. + var isEmpty: Bool { get } + + /// The range of valid byte offsets into this `RawSpan` + var byteOffsets: Range { get } +} +``` + +##### Accessing and modifying the memory of a `MutableRawSpan`: + +`MutableRawSpan` supports storing the bytes of a `BitwiseCopyable` value to its underlying memory: + +```swift +extension MutableRawSpan { + /// Stores the given value's bytes into raw memory at the specified offset. + mutating func storeBytes( + of value: T, toByteOffset offset: Int = 0, as type: T.Type + ) + + /// Stores the given value's bytes into raw memory at the specified offset. + /// + /// This function does not validate `offset`; this is an unsafe operation. + @unsafe + mutating func storeBytes( + of value: T, toUncheckedByteOffset offset: Int, as type: T.Type + ) +} +``` + +Additionally, the basic loading operations available on `RawSpan` are available for `MutableRawSpan`. These operations are not type-safe, in that the loaded value returned by the operation can be invalid, and violate type invariants. Some types have a property that makes the `unsafeLoad(as:)` function safe, but we don't have a way to [formally identify](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md#SurjectiveBitPattern) such types at this time. + +```swift +extension MutableRawSpan { + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + @unsafe + func unsafeLoad( + fromByteOffset offset: Int = 0, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + @unsafe + func unsafeLoadUnaligned( + fromByteOffset offset: Int = 0, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + @unsafe + func unsafeLoad( + fromUncheckedByteOffset offset: Int, as: T.Type + ) -> T + + /// Returns a new instance of the given type, constructed from the raw memory + /// at the specified offset. + @unsafe + func unsafeLoadUnaligned( + fromUncheckedByteOffset offset: Int, as: T.Type + ) -> T +} +``` + +We include functions to perform bulk copies into the memory represented by a `MutableRawSpan`. Updating a `MutableRawSpan` from a `Collection` or a `Span` copies every element of a source. It is an error to do so when there is are not enough bytes in the span to contain every element from the source. Updating `MutableRawSpan` from `Sequence` or `IteratorProtocol` instance copies as many items as possible, either until the input is empty or until there are not enough bytes in the span to store another element. + +```swift +extension MutableRawSpan { + /// Updates the span's bytes with the bytes of the elements from the source + mutating func update( + from source: S + ) -> (unwritten: S.Iterator, byteOffset: Int) where S.Element: BitwiseCopyable + + /// Updates the span's bytes with the bytes of the elements from the source + mutating func update( + from source: inout some IteratorProtocol + ) -> Int + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: C + ) -> Int where C.Element: BitwiseCopyable + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: Span + ) -> Int + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: borrowing MutableSpan + ) -> Int + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: RawSpan + ) -> Int + + /// Updates the span's bytes with every byte of the source. + mutating func update( + fromContentsOf source: borrowing MutableRawSpan + ) -> Int +} +``` + +##### Extracting sub-spans + +These functions extract sub-spans of the callee. The first two perform strict bounds-checking. The last four return prefixes or suffixes, where the number of elements in the returned sub-span is bounded by the number of elements in the parent `MutableRawSpan`. + +```swift +extension MutableRawSpan { + /// Returns a span over the items within the supplied range of + /// positions within this span. + @_lifetime(inout self) + mutating public func extracting(_ byteOffsets: Range) -> Self + + /// Returns a span over the items within the supplied range of + /// positions within this span. + @_lifetime(inout self) + mutating public func extracting(_ byteOffsets: some RangeExpression) -> Self + + /// Returns a span containing the initial elements of this span, + /// up to the specified maximum length. + @_lifetime(inout self) + mutating public func extracting(first maxLength: Int) -> Self + + /// Returns a span over all but the given number of trailing elements. + @_lifetime(inout self) + mutating public func extracting(droppingLast k: Int) -> Self + + /// Returns a span containing the final elements of the span, + /// up to the given maximum length. + @_lifetime(inout self) + mutating public func extracting(last maxLegnth: Int) -> Self + + /// Returns a span over all but the given number of initial elements. + @_lifetime(inout self) + mutating public func extracting(droppingFirst k: Int) -> Self +} +``` + +We also provide unchecked variants of the `extracting()` functions as alternatives in situations where repeated bounds-checking is costly and has already been performed: + +```swift +extension MutableRawSpan { + /// Constructs a new span over the items within the supplied range of + /// positions within this span. + /// + /// This function does not validate `byteOffsets`; this is an unsafe operation. + @unsafe + @_lifetime(inout self) + mutating func extracting(unchecked byteOffsets: Range) -> Self + + /// Constructs a new span over the items within the supplied range of + /// positions within this span. + /// + /// This function does not validate `byteOffsets`; this is an unsafe operation. + @unsafe + @_lifetime(inout self) + mutating func extracting(unchecked byteOffsets: ClosedRange) -> Self +} +``` + +##### Interoperability with unsafe code: + +```swift +extension MutableRawSpan { + /// Calls a closure with a pointer to the underlying bytes of + /// the viewed contiguous storage. + func withUnsafeBytes( + _ body: (_ buffer: UnsafeRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result + + /// Calls a closure with a pointer to the underlying bytes of + /// the viewed mutable contiguous storage. + mutating func withUnsafeMutableBytes( + _ body: (_ buffer: UnsafeMutableRawBufferPointer) throws(E) -> Result + ) throws(E) -> Result +} +``` +These functions use a closure to define the scope of validity of `buffer`, ensuring that the underlying `MutableSpan` and the binding it depends on both remain valid through the end of the closure. They have the same shape as the equivalents on `Array` because they fulfill the same purpose, namely to keep the underlying binding alive. + +#### Properties providing `MutableSpan` or `MutableRawSpan` instances + +##### Accessing and mutating the raw bytes of a `MutableSpan` + +When a `MutableSpan`'s element is `BitwiseCopyable`, we allow mutations of the underlying storage as raw bytes, as a `MutableRawSpan`. + +```swift +extension MutableSpan where Element: BitwiseCopyable { + /// Access the underlying raw bytes of this `MutableSpan`'s elements + /// + /// Note: mutating the bytes may result in the violation of + /// invariants in the internal representation of `Element` + @unsafe + var mutableBytes: MutableRawSpan { @_lifetime(inout self) mutating get } +} +``` + +The standard library will provide `mutating` computed properties providing lifetime-dependent `MutableSpan` instances. These `mutableSpan` computed properties are intended as the safe and composable replacements for the existing `withUnsafeMutableBufferPointer` closure-taking functions. + +##### Extensions to Standard Library types + +```swift +extension Array { + /// Access this Array's elements as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} + +extension ContiguousArray { + /// Access this Array's elements as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} + +extension ArraySlice { + /// Access this Array's elements as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} + +extension InlineArray { + /// Access this Array's elements as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} + +extension CollectionOfOne { + /// Access this Collection's element as mutable contiguous storage. + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } +} +``` + +##### Extensions to unsafe buffer types + +We hope that `MutableSpan` and `MutableRawSpan` will become the standard ways to delegate mutations of shared contiguous memory in Swift. Many current API delegate mutations via closure-based functions that receive an `UnsafeMutableBufferPointer` parameter. We will provide ways to unsafely obtain `MutableSpan` instances from `UnsafeMutableBufferPointer` and `MutableRawSpan` instances from `UnsafeMutableRawBufferPointer`, in order to bridge these unsafe types to newer, safer contexts. + +```swift +extension UnsafeMutableBufferPointer { + /// Unsafely access this buffer as a MutableSpan + @unsafe + var mutableSpan: MutableSpan { @_lifetime(borrow self) get } +} + +extension UnsafeMutableRawBufferPointer { + /// Unsafely access this buffer as a MutableRawSpan + @unsafe + var mutableBytes: MutableRawSpan { @_lifetime(borrow self) get } +} +``` + +These unsafe conversions returns a value whose lifetime is dependent on the _binding_ of the `UnsafeMutable[Raw]BufferPointer`. This dependency does not keep the underlying memory alive. As is usual where the `UnsafePointer` family of types is involved, the programmer must ensure the memory remains allocated while it is in use. Additionally, the following invariants must remain true for as long as the `MutableSpan` or `MutableRawSpan` value exists: + + - The underlying memory remains initialized. + - The underlying memory is not accessed through another means. + +Failure to maintain these invariants results in undefined behaviour. + +##### Extensions to `Foundation.Data` + +While the `swift-foundation` package and the `Foundation` framework are not governed by the Swift evolution process, `Data` is similar in use to standard library types, and the project acknowledges that it is desirable for it to have similar API when appropriate. Accordingly, we plan to propose the following additions to `Foundation.Data`: + +```swift +extension Foundation.Data { + // Access this instance's bytes as mutable contiguous storage + var mutableSpan: MutableSpan { @_lifetime(inout self) mutating get } + + // Access this instance's bytes as mutable contiguous bytes + var mutableBytes: MutableRawSpan { @_lifetime(inout self) mutating get } +} +``` + +#### Performance + +The `mutableSpan` and `mutableBytes` properties should be performant and return their `MutableSpan` or `MutableRawSpan` with very little work, in O(1) time. In copy-on-write types, however, obtaining a `MutableSpan` is the start of the mutation. When the backing buffer is not uniquely referenced then a full copy must be made ahead of returning the `MutableSpan`. + +Note that `MutableSpan` incurs no special behaviour for bridged types, since mutable bindings always require a defensive copy of data bridged from Objective-C data structures. + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a new version of the Swift standard library. + +## Alternatives considered + +#### Adding `withMutableSpan()` closure-taking functions + +The `mutableSpan` and `mutableBytes` properties aim to be safe replacements for the `withUnsafeMutableBufferPointer()` and `withUnsafeMutableBytes()` closure-taking functions. We could consider `withMutableSpan()` and `withMutableBytes()` closure-taking functions that would provide a quicker migration away from the older unsafe functions. We do not believe the closure-taking functions are desirable in the long run. In the short run, there may be a desire to clearly mark the scope where a `MutableSpan` instance is used. The default method would be to explicitly consume a `MutableSpan` instance: + +```swift +var a = ContiguousArray(0..<8) +var span = a.mutableSpan +modify(&span) +_ = consume span +a.append(8) +``` + +During the evolution of Swift, we have learned that closure-based API are difficult to compose, especially with one another. They can also require alterations to support new language features. For example, the generalization of closure-taking API for non-copyable values as well as typed throws is ongoing; adding more closure-taking API may make future feature evolution more labor-intensive. By instead relying on returned values, whether from computed properties or functions, we build for **greater** composability. Use cases where this approach falls short should be reported as enhancement requests or bugs. + +#### Omitting extensions to `UnsafeBufferPointer` and related types + +We could omit the extensions to `UnsafeMutableBufferPointer` and related types, and rely instead of future `MutableSpan` and `MutableRawSpan` initializers. The initializers can have the advantage of being able to communicate semantics (somewhat) through their parameter labels. However, they also have a very different shape than the `storage` computed properties we are proposing for the safe types such as `Array`. We believe that the adding the same API on both safe and unsafe types is advantageous, even if the preconditions for the properties cannot be statically enforced. + +## Future directions + +Note: The future directions stated in [SE-0447](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md#Directions) apply here as well. + +#### Initializing and returning `MutableSpan` instances + +`MutableSpan` represents a region of memory and, as such, must be initialized using an unsafe pointer. This is an unsafe operation which will typically be performed internally to a container's implementation. In order to bridge to safe code, these initializers require new annotations that indicate to the compiler how the newly-created `Span` can be used safely. + +These annotations have been [pitched][PR-2305-pitch] and, after revision, are expected to be pitched again soon. `MutableSpan` initializers using lifetime annotations will be proposed alongside the annotations themselves. + +#### Splitting `MutableSpan` instances – `MutableSpan` in divide-and-conquer algorithms + +It is desirable to have a way to split a `MutableSpan` in multiple parts, for divide-and-conquer algorithms or other reasons: + +```swift +extension MutableSpan where Element: ~Copyable { + public mutating func split(at index: Index) -> (part1: Self, part2: Self) +} +``` + +Unfortunately, tuples do not support non-copyable or non-escapable values yet. We may be able to use `InlineArray` ([SE-0453][SE-0453]), or a bespoke type, but destructuring the non-copyable constituent part remains a challenge. Solving this issue for `Span` and `MutableSpan` is a top priority. + +#### Mutating algorithms + +Algorithms defined on `MutableCollection` such as `sort(by:)` and `partition(by:)` could be defined on `MutableSpan`. We believe we will be able to define these more generally once we have a generalized container protocol hierarchy. + +#### Exclusive Access + +The `mutating` functions in this proposal generally do not represent mutations of the binding itself, but of memory being referenced. `mutating` is necessary in order to model the necessary exclusive access to the memory. We could conceive of an access level between "shared" (`let`) and "exclusive" (`var`) that would model an exclusive access while allowing the pointer and count information to be stored in registers. + +#### Harmonizing `extracting()` functions across types + +The range of `extracting()` functions proposed here expands upon the range accepted in [SE-0437][SE-0437]. If the prefix and suffix variants are accepted, we should add them to `UnsafeBufferPointer` types as well. `Span` and `RawSpan` should also have `extracting()` functions with appropriate lifetime dependencies. + +#### Delegated initialization with `OutputSpan` + +Some data structures can delegate initialization of parts of their owned memory. The standard library added the `Array` initializer `init(unsafeUninitializedCapacity:initializingWith:)` in [SE-0223][SE-0223]. This initializer relies on `UnsafeMutableBufferPointer` and correct usage of initialization primitives. We should present a simpler and safer model of initialization by leveraging non-copyability and non-escapability. + +We expect to propose an `OutputSpan` type to represent partially-initialized memory, and to support to the initialization of memory by appending to the initialized portion of the underlying storage. diff --git a/proposals/0468-async-stream-continuation-hashable-conformance.md b/proposals/0468-async-stream-continuation-hashable-conformance.md new file mode 100644 index 0000000000..5412d1734e --- /dev/null +++ b/proposals/0468-async-stream-continuation-hashable-conformance.md @@ -0,0 +1,130 @@ +# `Hashable` conformance for `Async(Throwing)Stream.Continuation` + +* Proposal: [SE-0468](0468-async-stream-continuation-hashable-conformance.md) +* Authors: [Mykola Pokhylets](https://github.com/nickolas-pohilets) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#79457](https://github.com/swiftlang/swift/pull/79457) +* Review: ([pitch](https://forums.swift.org/t/pitch-add-hashable-conformance-to-asyncstream-continuation/77897)) ([review](https://forums.swift.org/t/se-0468-hashable-conformance-for-async-throwing-stream-continuation/78487)) ([acceptance](https://forums.swift.org/t/accepted-se-0468-hashable-conformance-for-async-throwing-stream-continuation/79116)) + +## Introduction + +This proposal adds a `Hashable` conformance to `Async(Throwing)Stream.Continuation` +to simplify working with multiple streams. + +## Motivation + +Use cases operating with multiple `AsyncStream`s may need to store multiple continuations. +When handling `onTermination` callback, client code needs to remove the relevant continuation. + +To identify the relevant continuation, client code needs to be able to compare continuations. + +It is possible to associate a lookup key with each continuation, but this is inefficient. +`AsyncStream.Continuation` already stores a reference to `AsyncStream._Storage`, +whose identity can be used to provide simple and efficient `Hashable` conformance. + +Consider this simple Observer pattern with an `AsyncSequence`-based API. +To avoid implementing `AsyncSequence` from scratch it uses `AsyncStream` as a building block. +To support multiple subscribers, a new stream is returned every time. + +```swift +@MainActor private class Sender { + var value: Int = 0 { + didSet { + for c in continuations { + c.yield(value) + } + } + } + + var values: some AsyncSequence { + AsyncStream(bufferingPolicy: .bufferingNewest(1)) { continuation in + continuation.yield(value) + self.continuations.insert(continuation) + continuation.onTermination = { _ in + DispatchQueue.main.async { + self.continuations.remove(continuation) + } + } + } + } + + private var continuations: Set.Continuation> = [] +} +``` + +Without a `Hashable` conformance, each continuation needs to be associated with an artificial identifier. +E.g. wrapping continuation in a class, identity of the wrapper object can be used: + +```swift +@MainActor private class Sender { + var value: Int = 0 { + didSet { + for c in continuations { + c.value.yield(value) + } + } + } + + var values: some AsyncSequence { + AsyncStream { (continuation: AsyncStream.Continuation) -> Void in + continuation.yield(value) + let box = ContinuationBox(value: continuation) + self.continuations.insert(box) + continuation.onTermination = { _ in + DispatchQueue.main.async { + self.continuations.remove(box) + } + } + } + } + + private var continuations: Set = [] + + private final class ContinuationBox: Hashable, Sendable { + let value: AsyncStream.Continuation + + init(value: AsyncStream.Continuation) { + self.value = value + } + + static func == (lhs: Sender.ContinuationBox, rhs: Sender.ContinuationBox) -> Bool { + lhs === rhs + } + + func hash(into hasher: inout Hasher) { + hasher.combine(ObjectIdentifier(self)) + } + } +} +``` + +Note that capturing `continuation` or `box` in `onTermination` is safe, because `onTermination` is dropped after being called +(and it is _always_ called, even if `AsyncStream` is discarded without being iterated). + +## Proposed solution + +Add a `Hashable` conformance to `Async(Throwing)Stream.Continuation`. + +## Detailed design + +Every time when the `build` closure of the `Async(Throwing)Stream.init()` is called, +it receives a continuation distinct from all other continuations. +All copies of the same continuation should compare equal. +Yielding values or errors, finishing the stream, or cancelling iteration should not affect equality. +Assigning `onTermination` closures should not affect equality. + +## Source compatibility + +This is an additive change. + +Retroactive conformances are unlikely to exist, because current public API of the `Async(Throwing)Stream.Continuation` +does not provide anything that could be reasonably used to implement `Hashable` or `Equatable` conformances. + +## ABI compatibility + +This is an additive change. + +## Implications on adoption + +Adopters will need a new version of the standard library. diff --git a/proposals/0469-task-names.md b/proposals/0469-task-names.md new file mode 100644 index 0000000000..0d1faf7de7 --- /dev/null +++ b/proposals/0469-task-names.md @@ -0,0 +1,272 @@ +# Task Naming + +* Proposal: [SE-0469](0469-task-names.md) +* Authors: [Konrad Malawski](https://github.com/ktoso), [Harjas Monga](https://github.com/Harjas12) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#79600](https://github.com/swiftlang/swift/pull/79600) +* Review: ([pitch](https://forums.swift.org/t/pitch-task-naming-api/76115)) ([review](https://forums.swift.org/t/se-0469-task-naming/78509)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0469-task-naming/79438)) + +## Introduction + +In this proposal, we introduce several new APIs to allow developers to name their Swift Tasks for the purposes of identifying tasks in a human-readable way. These names can then be used to identify tasks by printing their names, programatically inspecting the name property, or by tools which dump and inspect tasks–such as debuggers, swift-inspect or others. + +## Motivation + +In previous generations of concurrency technologies, developer tools, such as debuggers, have had access to some kind of label to help describe a process’s concurrent work. Ex: Pthread names or Grand Central Dispatch queue names. These names are very helpful to provide extra context to developers when using debugging and profiling tools. + +Currently, Swift Concurrency has no affordances to allow developers to label a Task, which can be troublesome for developers trying to identify "which task" is taking a long time to process or similar questions when observing the system externally. In order to ease the debugging and profiling of Swift concurrency code, developers should be able to annotate their Swift Tasks to describe an asynchronous workload. + +## Proposed solution + +In order to allow developers to provide helpful names for Swift Tasks, the Swift Task creation APIs should be modified to *optionally* allow developers to provide a name for that task. + +Consider the example: + +```swift +let getUsers = Task { + await users.get(accountID)) +} +``` + +In order to ease debugging, a developer could create this unstructured task by passing in a name instead: + +```swift +let getUsers = Task(name: "Get Users") { + await users.get(accountID) +} +``` + +Or, if a developer has a lot of similar tasks, they can provide more contextual information using string interpolation. + +```swift +let getUsers = Task("Get Users for \(accountID)") { + await users.get(accountID) +} +``` + +By introducing this API in Swift itself, rather than developers each inventing their own task-local with a name, runtime inspection tools and debuggers can become aware of task names and show you exactly which accountID was causing the crash or a profiling tool could tell you which accountID request was slow to load. + +## Detailed design + +Naming tasks is only allowed during their creation, and modifying names is not allowed. + +Names are arbitrary user-defined strings, which may be computed at runtime because they often contain identifying information such as the request ID or similar runtime information. + +The following APIs will be provided on `Task`: + +```swift +extension Task where Failure == /* both Never and Error cases */ { + init( + name: String?, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + priority: TaskPriority? = nil, + operation: sending @escaping @isolated(any) () async /*throws */-> Success) + + static func detached( + name: String?, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + priority: TaskPriority? = nil, + operation: sending @escaping @isolated(any) () async /*throws */ -> Success) +} +``` + +In addition to these APIs to name unstructured Tasks, the following API will be added to all kinds of task groups: + +```swift +mutating func addTask( + name: String?, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + priority: TaskPriority? = nil, + operation: sending @escaping @isolated(any) () async -> ChildTaskResult + ) + + mutating func addTaskUnlessCancelled( + name: String?, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + priority: TaskPriority? = nil, + operation: sending @escaping @isolated(any) () async -> ChildTaskResult + ) +``` + +These APIs would be added to all kinds of task groups, including throwing, discarding ones. With the signature being appropriately matching the existing addTask signatures of those groups. + +> Concurrently under review with this proposal is the `Task.startSynchronously` (working name, pending changes) proposal; +> If both this and the synchronous starting tasks proposals are accepted, these APIs would also gain the additional `name: String? = nil` parameter. + +In addition to that, it will be possible to read a name off a task, similar to how the current task's priority is possible to be read: + +```swift +extension Task { + static var name: String? { get } +} + +extension UnsafeCurrentTask { + var name: String? { get } +} +``` + +### `UnsafeCurrentTask` access from `UnownedJob` + +In order to have an `Executor` be able to inspect a task name, either to print "Now running [Task A]" or for other reasons, we propose to offer the access to an `UnsafeCurrentTask` representation of a `ExecutorJob` (or `UnownedJob`): + +```swift +extension ExecutorJob / UnownedJob { + public var unsafeCurrentTask: UnsafeCurrentTask? { ... } +} +``` + +This allows executors to inspect the task name if the `job` is a task, and has a name: + +```swift +public nonisolated func enqueue(_ job: consuming ExecutorJob) { + log.trace("Running task named: \(job?.unsafeCurrentTask?.name ?? "")") +} +``` + +We use the `UnsafeCurrentTask` type because it is possible to obtain it from an `UnownedTask` and therefore it is not safe to refer to it without knowladge about the job's lifetime. +One should not refer to the unsafe current task after invoking `runSynchronously` on the job, as the job may have completed and been destroyed; therefore the use of the existing `UnsafeCurrentTask` type here is quite appropriate. + +This also allows us to expose other information off a task, such as task local values in the future, if the `UnsafeCurrentTask` were to gain such APIs, without having to replicate "the same" accessors into yet another API that would be accessible directly from an `ExecutorJob`. + +## Source compatibility + +This proposal only contains additive changes to the API surface. + +Since Swift Tasks names will be optional, there will be no source compatibility issues. + +## ABI compatibility + +This proposal is ABI additive and does not change any existing ABI. + +## Implications on adoption + +Because runtime changes are required, these new APIs will only be available on newer OSes. + +## Future directions + +This proposal does not contain a method to name Swift Tasks created using the `async let` syntax. Unlike the other methods of creating Tasks, the `async let` syntax didn’t have an obvious way to allow a developer to provide a string. A suggestion of how we may provide automatic names to Tasks created via this method will be shown below in the [Alternatives Considered section](##Alternatives-considered). + +### Task names for "startSynchronously" + +If the ["start synchronously" tasks proposal](https://github.com/swiftlang/swift-evolution/pull/2698) would be accepted, the name parameter would also be included in those APIs. + +## Alternatives considered + +### Actor & DistributedActor Identity + +#### Actor Identity + +> Note: While not really an alternative, we would like to explain why this proposal does not propose to change anything about how actors are identified. + +This proposal focuses on task names, however, another important part of Swift Concurrency is actors, so in this section we’d like to discuss how there isn’t an actual need for new API to address *actor naming* because of how actors can already conform to protocols. + +An actor can conform e.g. to the `Identifiable` protocol. This works well with constant identifiers, as an actor can have a constant let property implement the `id` requirement from this protocol: + +```swift +actor Worker: Identifiable { + let id: String + + init(id: String) { + self.id = id + } +} +``` + +It is also likely that such identity is how a developer might want to look up and identify such actor in traces or logs, so making use of `Identifiable` seems like a good pattern to follow. + +It is also worth reminding that thread-safety of an actor is ensured even if the `id` were to be implemented using a computed property, because it will be forced to be `nonisolated` because of Swift’s conformance and actor isolation rules: + +```swift +actor Worker: Identifiable { + let workCategory: String = "fetching" // "building" etc... + let workID: Int + + nonisolated var id: String { + "\(workCategory)-\(workID)" + } +} +``` + +#### Distributed Actor Identity + +Distributed actors already implicitly conform to `Identifiable` protocol and have a very useful `id` representation that is always assigned by the actor system by which an actor is managed. + +This id is the natural human readable representation of such actor identity, and tools which want to print an “actor identity” should rely on this. In other words, this simply follows the same general pattern that makes sense for other objects and actors of using Identifiable when available to identify things. + +```swift +distributed actor Worker { // implicitly Identifiable + // nonisolated var id: Self.ActorSystem.ActorID { get } +} +``` + +### AsyncLet Task Naming + +While there is no clear way on how to name Swift Task using `async let`, the following were considered. + +#### Approach 1: + +Since we effectively want to express that “the task” is some specific task, we had considered introducing some special casing where if the right hand side of an async let we want to say at creation time that this task is something specific, thus we arrive at the following: + +```swift +async let example: String = Task(name: "get-example") { "example" } +``` + +In order to make this syntax work, we need to avoid double creating tasks. When the compiler sees the `async let` syntax and the `Task {}` initializer, it would need to not create a Task to immediately create another Task inside it, but instead use that Task initializer that we explicitly wrote. + +While, this approach could in theory allow us to name Tasks created using `async let`. It has at least one major issue: + +It can cause surprising behavior and it can be unclear that this would only work when the Task initializer is visible from the async let declaration... I.e. moving the initialization into a method like this: + +```swift +async let example: String = getTask() // error String != Task + +func getTask() -> Task { Task(name: "get-example") { "example" } } +``` + +This would not only break refactoring, as the types are not the same; but also execution semantics, as this refactoring has now caused the task to become an unstructured task “by accident”. Therefore this approach is not viable because it introduces too many easy to make mistakes. + +#### Approach 2: + +Instead of attempting to adding a naming API to the `async let` syntax, we could instead take a different where if developers really want to name a structured Task they can use a `TaskGroup` and the compiler would generate a good default name for the Tasks created using the `async let` syntax. Drawing inspiration from how closures and dispatch blocks are named, we count the declaration in the scope and use that to name it. For example: + +```swift +func getUserImages() async -> [Image] { + + async let profileImg = getProfilePicture() // <- Named "getUserImages.asyncLet-1" + async let headerImg = getHeaderPicture() // <- Named "getUserImages.asyncLet-2" + + . + . + . +} +``` + +These names at the very least give some indication of what the task was created to do, and the developer can opt to use the `TaskGroup` API if more control is desired. + +A slight alternative to this suggestion, is instead of using the name of the surrounding scope, use the name of the parent task instead. For example: + +```swift +Task(name: "get user images for \(userID)") { + async let profileImg = getProfilePicture() // <- Named "getUserImages.asyncLet-1" + async let headerImg = getHeaderPicture() // <- Named "getUserImages.asyncLet-2" + + . + . + . +} +``` + +This approach doesn’t allow developers full control over naming tasks, but it is in same spirit of allowing developer tools to provide more context for a task. + +## Structured Names + +There was some thought given to the idea of allowing developers to group similar tasks (in name only). Consider programs that create hundreds of tasks for network requests; by allowing grouping a runtime analysis tool could surface that in a textual or graphical UI. The API needed would be similar to the one proposed, but with an additional optional `category` argument for the Task initializer. For example: + +```swift +Task(category: "Networking", name: "download profile image for \(userID)) { ... } +``` + +Then a debugger than wanted to print all the tasks running when a break point is hit, it could group them by this optional “Networking” category. + +This is not in the actual proposal in order to keep the API simple and doesn’t add much additional value over a simple name. diff --git a/proposals/0470-isolated-conformances.md b/proposals/0470-isolated-conformances.md new file mode 100644 index 0000000000..879bd8d05e --- /dev/null +++ b/proposals/0470-isolated-conformances.md @@ -0,0 +1,694 @@ +# Global-actor isolated conformances + +* Proposal: [SE-0470](0470-isolated-conformances.md) +* Authors: [Doug Gregor](https://github.com/DougGregor) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.2)** +* Vision: [Improving the approachability of data-race safety](https://github.com/swiftlang/swift-evolution/blob/main/visions/approachable-concurrency.md) +* Implementation: On `main` with the experimental features `IsolatedConformances` and `StrictSendableMetatypes`. +* Upcoming Feature Flag: `InferIsolatedConformances` +* Review: ([pitch](https://forums.swift.org/t/pre-pitch-isolated-conformances/77726)) ([review](https://forums.swift.org/t/se-0470-global-actor-isolated-conformances/78704)) ([acceptance](https://forums.swift.org/t/accepted-se-0470-global-actor-isolated-conformances/79189)) ([amendment pitch](https://forums.swift.org/t/pitch-amend-se-0466-se-0470-to-improve-isolation-inference/79854)) ([amendment review](https://forums.swift.org/t/amendment-se-0470-global-actor-isolated-conformances/80999)) ([amendment acceptance](https://forums.swift.org/t/amendment-accepted-se-0470-global-actor-isolated-conformances/81144)) + +## Introduction + +Types isolated to a global actor (such as `@MainActor`) are useful for representing data that can only ever be used from a single concurrency context. They occur both in single-threaded programs where all code is expected to run on the main actor as well as larger applications where interaction with the UI occurs through the main actor. Unfortunately, such types are unable to conform to most protocols due to isolation mismatches: + +```swift +@MainActor +class MyModelType: Equatable { + var name: String + + init(name: String) { + self.name = name + } + + // error: main-actor-isolated static function '==' cannot satisfy non-isolated requirement 'Equatable.==' + static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { + lhs.name == rhs.name + } +} +``` + +This proposal introduces the notion of an *isolated conformance*, which is a conformance that can only be used within the isolation domain of the type. For the code above, the conformance to `Equatable` can be specified as being isolated to the main actor as follows: + +```swift +@MainActor +class MyModelType: @MainActor Equatable { + // unchanged from the above ... +} +``` + +This allows `MyModelType` to provide a conformance to `Equatable` that works like every other conformance, except that it can only be used from the main actor. + +## Motivation + +Types isolated to the global actor are common in single-threaded programs and UI applications, among others, but their inability to conform to protocols without workarounds means that they cannot integrate with any Swift code using generics, cutting them off from interacting with many libraries. The workarounds themselves can be onerous: each operation that is used to satisfy a protocol requirement must be marked as `nonisolated`, e.g., + +```swift + nonisolated static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { + lhs.name == rhs.name + } +``` + +However, this is incompatible with using types or data on the main actor, and results in an error: + +```swift + 3 | @MainActor + 4 | class MyModelType: Equatable { + 5 | var name: String + | `- note: property declared here + 6 | + 7 | init(name: String) { + : +10 | +11 | nonisolated static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { +12 | lhs.name == rhs.name + | `- error: main actor-isolated property 'name' can not be referenced from a nonisolated context +13 | } +14 | } +``` + +We can work around this issue by assuming that this function will only ever be called on the main actor using [`MainActor.assumeIsolated`](https://developer.apple.com/documentation/swift/mainactor/assumeisolated(_:file:line:)): + +```swift + nonisolated static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { + MainActor.assumeIsolated { + lhs.name == rhs.name + } + } +``` + +This is effectively saying that `MyModelType` will only ever be considered `Equatable` on the main actor. Violating this assumption will result in a run-time error detected when `==` is called from outside the main actor. There are two problems with this approach. First, it's dynamically enforcing data-race safety for something that seems like it should be statically verifiable (but can't easily be expressed). Second, this same `nonisolated`/`assumeIsolated` pattern has to be replicated for every function that satisfies a protocol requirement, creating a lot of boilerplate. + +## Proposed solution + +This proposal introduces the notion of an *isolated conformance*. Isolated conformances are conformances whose use is restricted to a particular global actor. This is the same effective restriction as the `nonisolated`/`assumeIsolated` pattern above, but enforced statically by the compiler and without any boilerplate. The following defines a main-actor-isolated conformance of `MyModelType` to `Equatable`: + +```swift +@MainActor +class MyModelType: @MainActor Equatable { + var name: String + + init(name: String) { + self.name = name + } + + static func ==(lhs: MyModelType, rhs: MyModelType) -> Bool { + lhs.name == rhs.name + } +} +``` + +Any attempt to use this conformance outside of the main actor will result in a compiler error: + +```swift +/*nonisolated*/ func hasMatching(_ value: MyModelType, in modelValues: [MyModelType]) -> Bool { + // error: cannot use main-actor-isolated conformance of 'MyModelType' to 'Equatable' in + // non-isolated function. + return modelValues.contains(value) +} +``` + +Additionally, we need to make sure that generic code cannot take the conformance and send it to another isolation domain. The [`Sequence.contains`](https://developer.apple.com/documentation/swift/sequence/contains(_:)) operation above clearly won't do that, but one could imagine a similar operation that uses concurrency to attempt the search in parallel: + +```swift +extension Sequence { + func parallelContains(_ element: Element) -> Bool where Element: Equatable & Sendable { + // ... + } +} +``` + +This `parallelContains` function can send values of type `Element` to another isolation domain, and from there call the `Equatable.==` function. If the conformance to `Equatable` is isolated, this would violate the data race safety guarantees. Therefore, this proposal specifies that an isolated conformance cannot be used in conjunction with a `Sendable` conformance: + +```swift +@MainActor +func parallelHasMatching(_ value: MyModelType, in modelValues: [MyModelType]) -> Bool { + // error: isolated conformance of 'MyModelType' to 'Equatable' cannot be used to + // satisfy conformance requirement for a `Sendable` type parameter 'Element'. + return modelValues.parallelContains(value) +} +``` + +The corresponding restriction needs to be in place within generic functions, ensuring that they don't leak (potentially) isolated conformances across isolation boundaries. For example, the following code could introduce a data race if the conformance of `T` to `GlobalLookup` were isolated: + +```swift +protocol GlobalLookup { + static func lookupByName(_ name: String) -> Self? +} + +func hasNamed(_: T.Type, name: String) async -> Bool { + return await Task.detached { + return T.lookupByName(name) != nil + }.value +} +``` + +Here, the type `T` itself is not `Sendable`, but because *all* metatypes are `Sendable` it is considered safe to use `T` from another isolation domain within the generic function. The use of `T`'s conformance to `GlobalLookup` within that other isolation domain introduces a data-race problem if the conformance were isolated. To prevent such problems in generic code, this proposal introduces a notion of *non-sendable metatypes*. Specifically, if a type parameter `T` does not conform to either `Sendable` or to a new protocol, `SendableMetatype`, then its metatype, `T.Type`, is not considered `Sendable` and cannot cross isolation boundaries. The above code, which is accepted in Swift 6 today, would be rejected by the proposed changes here with an error message like: + +```swift +error: cannot capture non-sendable type 'T.Type' in 'sending' closure +``` + +A function like `hasNamed` can indicate that its type parameter `T`'s requires non-isolated conformance by introducing a requirement `T: SendableMetatype`, e.g., + +```swift +func hasNamed(_: T.Type, name: String) async -> Bool { + return await Task.detached { + return T.lookupByName(name) != nil + }.value +} +``` + +As with `Sendable`, an isolated conformance cannot be combined with a `SendableMetatype` constraint: + +```swift +extension MyModelType: isolated GlobalLookup { + static func lookupByName(_ name: String) -> Self? { ... } +} + +// error: isolated conformance of 'MyModelType' to 'MyModelType' cannot be used to +// satisfy conformance requirement for a `SendableMetatype` type parameter 'T'. +if hasNamed(MyModelType.self, "root") { ... } +``` + +Note that `Sendable` inherits from `SendableMetatype`, so any type `T` with a `Sendable` requirement also implies a requirement `T: SendableMetatype`. + +Protocol conformances can also be discovered dynamically with the `as?` and `is` operators. For example, one could try to produce an `any Equatable` from a value of unknown type in any isolation domain: + +```swift +func tryEquatable(_ lhs: Any, rhs: Any) -> Bool { + if let eLHS = lhs as? any Equatable { + // use Equatable.== + } else { + return false + } +} +``` + +The `Any` value could contain `MyModelType`, in which case the conformance to `Equatable` will be isolated. In such cases, the `as?` operation will check whether the code is running on the executor associated with the conformance's isolation. If so, the cast can succeed; otherwise, the case will fail (and produce `nil`). + +## Detailed design + +The proposed solution describes the basic shape of isolated conformances and how they interact with the type system. This section goes into more detail on the data-race safety issues that arise from the introduction of isolated conformances into the language. Then it details three rules that, together, ensure freedom from data race safety issues in the presence of isolated conformances: + +1. An isolated conformance can only be used within its isolation domain. +2. When an isolated conformance is used to satisfy a generic constraint `T: P`, the generic signature must not include either of the following constraints: `T: Sendable` or `T: SendableMetatype`. +3. A value using a conformance isolated to a given global actor is within the same region as that global actor. + +### Data-race safety issues + +An isolated conformance must only be used within its actor's isolation domain. Here are a few examples that demonstrate the kinds of problems that need to be addressed by a design for isolated conformances to ensure that this property holds. + +First, using an isolated conformance outside of its isolation domain creates immediate problems. For example: + +```swift +protocol Q { + static func g() { } +} + +extension C: @MainActor Q { + @MainActor static func g() { } +} + +nonisolated func callQG() { + let qType: Q.Type = C.self + qType.g() // problem: called @MainActor function from nonisolated code +} +``` + +Here, a call to `C.g()` would have been rejected because it's calling a `@MainActor` function from non-isolated code and cannot `await`. However, if we're allowed to use the isolated conformance of `C: Q`, we would subvert the checking because `Q.g()` is non-isolated. + +We can address this specific issue by prohibiting the use of an isolated conformance from outside its isolation domain, i.e., the use of `C: Q` to convert `C.Type` to `Q.Type` in a non-`@MainActor` function would be an error. + +However, this is not sufficient to ensure that this conformance `C: P` will only be used from the main actor. Consider a function like this: + +```swift +@MainActor func badReturn(c: C) -> any Sendable & P { // okay so far + c // uses C: P from the main actor context (okay) + // uses C: Sendable (okay) +} + +@MainActor func useBadReturn(c: C) { + let anyP = badReturn(c: c) + Task.detached { + anyP.f() // PROBLEM: C.f is called from off the main actor + } +} +``` + +Here, the conformance `C: P` is used from within a `@MainActor` function, but a value that stores the conformance (in the `any Sendable & P`) is returned that no longer carries the isolation restriction. The caller is free to copy that value to another isolation domain, and will end up calling `@MainActor` code from outside the main actor. + +The issue is not limited to return values. For example, a generic parameter might escape to another isolation domain: + +```swift +@MainActor func sendMe(_ value: T) { + Task.detached { + value.f() + } +} + +extension C: @unchecked Sendable { } + +@MainActor func doSend(c: C) { + sendMe(c) // uses C: P from the main actor context + // uses C: Sendable +} +``` + +Here, `sendMe` ends up calling `C.f()` from outside the main actor. The combination of an isolated conformance and a `Sendable` requirement on the same type underlies this issue. To address the problem, we can prohibit the use of an isolated conformance if the corresponding type parameter (e.g, `T` in the example above) also has a `Sendable` requirement. + +However, that doesn't address all issues, because region isolation permits sending non-`Sendable` values: + +```swift +@MainActor func badSendingReturn() -> sending any P { // okay so far + C() // uses C: P from the main actor context (okay) + // returned value is in its own region +} + +@MainActor func useBadSendingReturn(c: C) { + let anyP = badSendingReturn() + Task.detached { + anyP.f() // PROBLEM: C.f is called from off the main actor + } +} +``` + +There are similar examples for `sending` parameters, but they're not conceptually different from the return case. This particular issue can be addressed by treating a value that depends on an isolated conformance as being within the region as the actor it's isolated to. So a newly-created value of type `C` is in its own region, but if it's type-erased to an `any P`, its region is merged with the region for the main actor. This would make the return expression in `badSendingReturn` ill-formed, because the returned value is not in its own region. + +Conformances can cross isolation boundaries even if no values cross the boundary: + +```swift +nonisolated func callQGElsewhere(_: T.Type) { + Task.detached { + T.g() + } +} + +@MainActor func isolationWithStatics() { + callQGElsewhere(C.self) +} +``` + +Here, the generic type `T` is used from another isolation domain inside `callQGElsewhere`. When the isolated conformance of `C: Q` is provided to this function, it opens up a data-race safety hole because `C.g()` ends up getting called through generic code. Addressing this problem either means ensuring that there are no operations on the metatype that go through a potentially-isolated protocol conformance or that the metatype is itself does not leave the isolation domain. + +One last issue concerns dynamic casting. Generic code can query a conformance at runtime with a dynamic cast like this: + +```swift +nonisolated func f(_ value: Any) { + if let p = value as? any P { + p.f() + } +} +``` + +If the provided `value` is an instance of `C` , and this code is invoked off the main actor, allowing it to enter the `if` branch would introduce a data race. Therefore, dynamic casting will have to determine when the conformance it depends on is isolated to an actor and check whether the code is running on the executor for that actor. + +Additionally, a dynamic cast that involves a `Sendable` or `SendableMetatype` constraint should not accept an isolated conformance even if the code is running on that global actor, e.g., + +```swift + if let p = value as? any Sendable & P { // never allows an isolated conformance to P + p.f() + } +``` + +### Rule 1: Isolated conformance can only be used within its isolation domain + +Rule (1) is straightforward: the conformance can only be used within a context that is also isolated to the same global actor. This applies to any use of a conformance anywhere in the language. For example: + +```swift +struct S: @MainActor P { } + +struct WrapsP: P { + var value: T + + init(_ value: T) { self.value = value } +} + +func badFunc() -> WrapsP { } // error: non-@MainActor-isolated function uses @MainActor-isolated conformance `S: P` + +func badFunc2() -> any P { + S() // error: non-@MainActor-isolated function uses @MainActor-isolated conformance `S: P` +} + +func acceptsP(_ value: T) { } + +func badFunc3() { + acceptsP(S()) // error: non-@MainActor-isolated function uses @MainActor-isolated conformance `S: P` +} + +protocol P2 { + associatedtype A: P +} + +struct S2: P2 { // error: conformance of S2: P2 depends on @MainActor-isolated conformance `S: P` + // note: fix by making conformance of S2: P2 also @MainActor-isolated + typealias A = S +} + +protocol HasName { + var name: String { get } +} + +@MainActor class Named: @MainActor HasName { + var name: String + // ... +} + +@MainActor +func useName() { + let named = Named() + Task.detached { + named[keyPath: \HasName.name] // error: uses main-actor isolated conformance Named: P + // outside of the main actor + } +} +``` + +Note that the types can have different isolation from their conformances. For example, an `actor` or non-isolated type can have a `@MainActor` conformance: + +```swift +actor MyActor: @MainActor P { + // okay, so long as the declarations satisfying the requirements to P are + // @MainActor or nonisolated +} + +/*nonisolated*/ struct MyStruct: @MainActor P { + // okay, so long as the declarations satisfying the requirements to P are + // @MainActor or nonisolated +} +``` + +### Rule 2: Isolated conformances can only be abstracted away for non-`SendableMetatype` types + +Rule (2) ensures that when information about an isolated conformance is abstracted away by the generics system, the conformance cannot leave its original isolation domain. This requires a way to determine when a given generic function is permitted to pass a conformance it receives across isolation domains. Consider the example above where a generic function uses one of its conformances in different isolation domain: + +```swift +protocol Q { + static func g() { } +} + +nonisolated func callQGElsewhere(_: T.Type) { + Task.detached { + T.g() // use of the conformance T: Q in a different isolation domain + } +} + +extension C: @MainActor Q { ... } + +@MainActor func isolationWithStatics() { + callQGElsewhere(C.self) // passing an isolated conformance +} +``` + +The above code must be rejected to prevent a data race. There are two options for diagnosing this data race: + +1. Reject the definition of `callQGElsewhere` because it is using the conformance from a different isolation domain. +2. Reject the call to `callQGElsewhere` because it does not support isolated conformances. + +This proposal takes option (1): we assume that generic code accepts isolated conformances unless it has indicated otherwise with a `SendableMetatype` constraint. Since most generic code doesn't deal with concurrency at all, it will be unaffected. And generic code that does make use of concurrency should already have `Sendable` constraints (which imply `SendableMetatype` constraints) that indicate that it will not work with isolated conformances. + +The specific requirement for option (1) is enforced both in the caller to a generic function and in the implementation of that function. The caller can use an isolated conformance to satisfy a conformance requirement `T: P` so long as the generic function does not also contain a requirement `T: SendableMetatype`. This prevents isolated conformances to be used in conjunction with types that can cross isolation domains, preventing the data race from being introduced at the call site. Here are some examples of this rule: + +```swift +func acceptsSendableMetatypeP(_ value: T) { } +func acceptsAny(_ value: T) { } +func acceptsSendableMetatype(_ value: T) { } + +@MainActor func passIsolated(s: S) { + acceptsP(s) // okay: the type parameter 'T' requires P but not SendableMetatype + acceptsSendableMetatypeP(s) // error: the type parameter 'T' requires SendableMetatype + acceptsAny(s) // okay: no isolated conformance + acceptsSendableMetatype(s) // okay: no isolated conformance +} +``` + +The same checking occurs when the type parameter is hidden, for example when dealing with `any` or `some` types: + +```swift +@MainActor func isolatedAnyGood(s: S) { + let a: any P = s // okay: the 'any P' cannot leave the isolation domain +} + +@MainActor func isolatedAnyBad(s: S) { + let a: any SendableMetatype & P = s // error: the (hidden) type parameter for the 'any' is SendableMetatype +} + +@MainActor func returnIsolatedSomeGood(s: S) -> some P { + return s // okay: the 'any P' cannot leave the isolation domain +} + +@MainActor func returnIsolatedSomeBad(s: S) -> some SendableMetatype & P { + return s // error: the (hidden) type parameter for the 'any' is Sendable +} +``` + +Within the implementation, we ensure that a conformance that could be isolated cannot cross an isolation boundary. This is done by making the a metatype `T.Type` `Sendable` only when there existing a constraint `T: SendableMetatype`. Therefore, the following program is ill-formed: + +```swift +protocol Q { + static func g() { } +} + +nonisolated func callQGElsewhere(_: T.Type) { + Task.detached { + T.g() // error: non-sendable metatype of `T` captured in 'sending' closure + } +} +``` + +To correct this function, add a constraint `T: SendableMetatype`, which allows the function to send the metatype (along with its conformances) across isolation domains. As described above, it also prevents the caller from providing an isolated conformance to satisfy the `T: Q` requirement, preventing the data race. + +`SendableMetatype` is a new marker protocol that captures the idea that values of the metatype of `T` (i.e., `T.Type`) will cross isolation domains and can take conformances with them. It is less restrictive than a `Sendable` requirement, which specifies that *values* of a type can be sent across isolation boundaries. All concrete types (structs, enums, classes, actors) conform to `SendableMetatype` implicitly, so fixing `callQGElsewhere` will not affect any non-generic code: + +```swift +nonisolated func callQGElsewhere(_: T.Type) { + Task.detached { + T.g() + } +} + +struct MyTypeThatConformsToQ: Q { ... } +callQGElsewhere(MyTypeThatConformsToQ()) // still works +``` + +The `Sendable` protocol inherits from the new `SendableMetatype` protocol: + +```swift +/*@marker*/ protocol SendableMetatype { } +/*@marker*/ protocol Sendable: SendableMetatype { } +``` + +This means that a requirement `T: Sendable` implies `T: SendableMetatype`, so a generic function that uses concurrency along with `Sendable` requirements, like this:: + +```swift +func doSomethingElsewhere(_ value: T) { + Task.detached { + value.f() // okay + } +} +``` + +will continue to work with the stricter model for generic functions in this proposal. + +The proposed change for generic functions does have an impact on source compatibility, where functions like `callQGElsewhere` will be rejected. However, the source break is limited to generic code that: + +1. Passes the metatype `T.Type` of a generic parameter `T` across isolation boundaries; +2. Does not have a corresponding constraint `T: Sendable` requirement; and +3. Is compiled with strict concurrency enabled (either as Swift 6 or with warnings). + +Experiments with the prototype implementation of this feature uncovered very little code that was affected by this change. The benefit to introducing this source break is that the vast majority of existing generic code will work unmodified with isolated conformances, or (if it's using concurrency) correctly reject the use of isolated conformances in their callers. + +### Rule 3: Isolated conformances are in their global actor's region + +With [region-based isolation](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0414-region-based-isolation.md), values of non-`Sendable` type can be transferred to another isolation domain when it can be proven that they are in their own "region" of code that separate from all other regions. Isolated conformances are considered to be within the region of their global actor, so any value formed that involves an isolated conformance will have its region merged with that of the isolated conformance. For example: + +```swift +@MainActor func acceptSending(_ value: sending any P) { } + +@MainActor func passSending() { + let c1 = C() // in its own region + let ap1: any P = c1 // merges region of c1 with region of the conformance of C: P (MainActor) + acceptSending(ap1) // error: argument to sending parameter is within the MainActor region + + let c2 = C() // in its own region + let wp2 = WrapsP(c2) // merges region of c2 with region of the conformance of C: P (MainActor) + acceptSending(c) // error: argument to sending parameter is within the MainActor region +} +``` + +### Inferring global actor isolation for global-actor-isolated types + +Types that are isolated to a global actor are very likely to want to have their conformances to be isolated to that global actor. This is especially true because the members of global-actor isolated types are implicitly isolated to that global actor, so obvious-looking code is rejected: + +```swift +@MainActor +class MyModelType: P { + func f() { } // error: implements P.f, is implicitly @MainActor + // but conformance to P is not isolated +} +``` + +With this proposal, the fix is to mark the conformance as `@MainActor`: + +```swift +@MainActor +class MyModelType: @MainActor P { + func f() { } // okay: implements P.f, is implicitly @MainActor +} +``` + +However, the inference rule feels uneven: why is the `@MainActor` in one place inferred but not in the other? + +In the future, we'd like to extend the global actor inference rule for global-actor isolated types to also infer global actor isolated on their conformances. This makes the obvious code above also correct: + +```swift +@MainActor +class MyModelType: /*inferred @MainActor*/ P { + func f() { } // implements P.f, is implicitly @MainActor +} +``` + +If this inference is not desired, one can use `nonisolated` on the conformances: + +```swift +@MainActor +class MyModelType: nonisolated Q { + nonisolated static func g() { } // implements Q.g, is non-isolated +} +``` + +There are two additional inference rules that imply `nonisolated` on a conformance of a global-actor-isolated type: + +* If the protocol inherits from `SendableMetatype` (including indirectly, e.g., from `Sendable`), then the isolated conformance could never be used, so it is inferred to be `nonisolated`. +* If all of the declarations used to satisfy protocol requirements are `nonisolated`, the conformance will be assumed to be `nonisolated`. The conformance of `MyModelType` to `Q` would be inferred to be `nonisolated` because the static method `g` used to satisfy `Q.g` is `nonisolated.` + +This proposed change is source-breaking in the cases where a conformance is currently `nonisolated`, the rules above would not infer `nonisolated`, and the conformance crosses isolation domains. There, conformance isolation inference is staged in via an upcoming feature (`InferIsolatedConformances`) that can be folded into a future language mode. Fortunately, it is mechanically migratable: existing code migrating to `InferIsolatedConformances` could introduce `nonisolated` for each conformance of a global-actor-isolated type. + +### Infer `@MainActor` conformances + +[SE-0466](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0466-control-default-actor-isolation.md) provides the ability to specify that a given module will infer `@MainActor` on any code that hasn't explicitly stated isolated (or non-isolation, via `nonisolated`). In a module that infers `@MainActor`, the upcoming feature `InferIsolatedConformances` (from the prior section) should also be enabled. This means that types will get main-actor isolation and also have their conformances main-actor isolated, extending the "mostly single-threaded" view of SE-0466 to interactions with generic code: + +```swift +/*implicit @MainActor*/ +class MyClass: /*implicit @MainActor*/P { ... } +``` + +## Source compatibility + +As discussed in the section on rule (2), this proposal introduces a source compatibility break for code that is using strict concurrency and passes uses conformances of non-`Sendable` type parameters in other isolation domains. The overall amount of such code is expected to be small, because it's likely to be rare that the conformances of generic types cross isolation boundaries but values of those types do not. + +Initial testing of an implementation of this proposal found very little code that relied on `Sendable` metatypes where the corresponding type was not also `Sendable`. Therefore, this proposal suggests to accept this as a source-breaking change with strict concurrency (as a warning in Swift 5, error in Swift 6) rather than staging the change through an upcoming feature or alternative language mode. + +## ABI compatibility + +Isolated conformances can be introduced into the Swift ABI without any breaking changes, by extending the existing runtime metadata for protocol conformances. All existing (non-isolated) protocol conformances can work with newer Swift runtimes, and isolated protocol conformances will be usable with older Swift runtimes as well. There is no technical requirement to restrict isolated conformances to newer Swift runtimes. + +However, there is one likely behavioral difference with isolated conformances between newer and older runtimes. In newer Swift runtimes, the functions that evaluate `as?` casts will check of an isolated conformance and validate that the code is running on the proper executor before the cast succeeds. Older Swift runtimes that don't know about isolated conformances will allow the cast to succeed even outside of the isolation domain of the conformance, which can lead to different behavior that potentially involves data races. It should be possible to provide (optional) warnings when running on newer Swift runtimes when a cast fails due to isolated conformances but would incorrectly succeed on older platforms. + +## Future Directions + +### Actor-instance isolated conformances + +Actor-instance isolated conformances are considerably more difficult than global-actor isolated conformances, because the conformance needs to be associated with a specific instance of that actor. Even enforcing rule (1) is nonobvious. As with `isolated` parameters, we could spell actor-instance isolation to a protocol `P` with `isolated P`. The semantics would need to be similar to what follows: + +```swift +actor A: isolated P { + func f() { } // implements P.f() +} + +func instanceActors(a1: isolated A, a2: A) { + let anyP1: any P = a1 // okay: uses isolated conformance 'A: P' only on a1, to which this function is isolated + let anyP2: any P = a2 // error: uses isolated conformance 'A: P' on a2, which is not the actor to which this function is isolated + + let a3 = a1 + let anyP3: any P = a3 // okay? requires dataflow analysis to determine that a3 and + // a1 are in the isolation domain of this function + + let wrappedA1: WrapsP // error? isolated conformance 'A: P' used without being + // anchored to the actor instance a1 + var wrappedA2: WrapsP = .init(a1) // okay? isolated conformance 'A: P' is used with a1 + wrappedA2.value = a3 // error: isolated conformance 'A: P' used in the type is + // in a different isolation domain than 'a1' +} +``` + +It's possible that these problems can be addressed by relying more heavily on region-based isolation akin to rule (3). This can be revisited in the future if the need justifies the additional complexity and we find a suitable implementation strategy. + +## Alternatives considered + +### "Non-Sendable" terminology instead of isolated conformances + +Isolated conformances are a lot like non-`Sendable` types, in that they can be freely used within the isolation domain in which they are created, but can't necessarily cross isolation domain boundaries. We could consider using "sendable" terminology instead of "isolation" terminology, e.g., all existing conformances are "Sendable" conformances (you can freely share them across isolation domain boundaries) and these new conformances are "non-Sendable" conformances. Trying to send such a conformance across an isolation domain boundary is, of course, an error. + +However, the "sendable" analogy breaks down or causes awkwardness in a few places: + +* Values of non-`Sendable` type can be sent across isolation domain boundaries due to [region-based isolation](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0414-region-based-isolation.md), but the same cannot be said of isolated conformances, so they are more non-Sendable than most non-Sendable things. + +* Global-actor-isolated types are usually `Sendable`, but their conformances would generally need to be non-`Sendable`. + +* Usually things are non-`Sendable` but have to be explicitly opt-in to being `Sendable`, whereas conformances would be the opposite. + +* Diagnostics for invalid conformance declarations that could be addressed with isolated conformances are necessarily described in terms of isolation, e.g., + ```` + error: main-actor isolated method 'f' cannot satisfy non-isolated requirement `f` of protocol P + ```` + + It wouldn't make sense to recast that diagnostic in terms of "sendable", and would also be odd for the fix to an isolation-related error message to be "add non-Sendable." + +* There is no established spelling for "not Sendable" that would work well on a conformance. + +### Isolated conformance requirements + +This proposal introduces the notion of isolated conformances, which can satisfy a conformance requirement only when the corresponding type isn't `Sendable`. There is no way for a generic function to express that some protocol requirements are intended to allow isolated conformances while others are not. That could be made explicit, for example by allowing requirements of the form `T: isolated P` (which would work with both isolated and non-isolated conformances) and `T: nonisolated P` (which only allows non-isolated conformances). One could combine these in a given generic signature: + +```swift +func mixedConformances(_ x: [T]) { + for item in x { + item.foo() // Can use requirements of P + print(x.id) // Can use requirements of Identifiable + } + + Task.detached { + for item in x { + item.foo() // error: cannot capture isolated conformance of 'T' to 'P' in a closure in a different isolation domain + print(x.id) // okay: conformance to Identifable is nonisolated + } + } +} +``` + +This is a generalization of the proposed rules that makes more explicit when conformances can cross isolation domains within generic code, as well as allowing mixing of isolated and non-isolated conformances as in the example. One can explain this proposal's rule involving `SendableMetatype` requirements and isolated conformances in terms of (non)-isolated requirements. For a given conformance requirement `T: P` : + +* If `T: SendableMetatype`, `T: P` is interpreted as `T: nonisolated P`. +* If not `T: SendableMetatype`, `T: P` is interepreted as `T: isolated P`. + +The main down side of this alternative is the additional complexity it introduces into generic requirements. It should be possible to introduce this approach later if it proves to be necessary, by treating it as a generalization of the existing rules in this proposal. + +### Require `nonisolated` rather than inferring it + +Under the upcoming feature `InferIsolatedConformances`, this proposal infers `nonisolated` for conformances when all of the declarations that satisfy requirements of a protocol are themselves `nonisolated`. For example: + +```swift +nonisolated protocol Q { + static func create() -> Self +} + +@MainActor struct MyType: /*infers nonisolated*/ Q { + nonisolated static func create() -> MyType { ... } +} +``` + +This inference is important for providing source compatibility with and without `InferIsolatedConformances`, and is especially useful useful when combined with default main-actor isolation ([SE-0466](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0466-control-default-actor-isolation.md)), where many more types will become main-actor isolated. Experience with using these features together also identified some macros (such as [`@Observable`](https://developer.apple.com/documentation/observation/observable())) that produced `nonisolated` members for a protocol conformances, but had not yet been updated to mark the conformance as `nonisolated`. Macro-generated code is much harder for users to update when a source-compatibility issue arises, which makes `nonisolated` conformance inference particularly important for source compatibility. + +However, this inference rule has downsides. It means one needs to examine a protocol and how a type conforms to that protocol to determine whether the conformance might be `nonisolated`, which can be a lot of work for the developer reading the code as well as the compiler. It can also change over time: for example, a default implementation of a protocol requirement will likely be `nonisolated`, but a user-written one within a main-actor-isolated type would be `@MainActor` and, therefore, make the conformance `@MainActor`. + +One alternative would be to introduce this inference rule for source compatibility, but treat it as a temporary measure to be disabled again in some future language mode. Introducing the inference rule in this proposal does not foreclose on that possibility: if we find that the `nonisolated` conformance inference rule here is harmful to readability, a separate proposal can deprecate it in a future language mode, providing a suitable migration timeframe. + +## Revision history + +* Changes in amendment review: + * If the protocol inherits from `SendableMetatype` (including indirectly, e.g., from `Sendable`), then the isolated conformance could never be used, so it is inferred to be `nonisolated`. + * If all of the declarations used to satisfy protocol requirements are `nonisolated`, the conformance will be assumed to be `nonisolated`. +* Changes in review: + * Within a generic function, use sendability of metatypes of generic parameters as the basis for checking, rather than treating specific conformances as potentially isolated. This model is easier to reason about and fits better with `SendableMetatype`, and was used in earlier drafts of this proposal. diff --git a/proposals/0471-SerialExecutor-isIsolated.md b/proposals/0471-SerialExecutor-isIsolated.md new file mode 100644 index 0000000000..b91e3693ad --- /dev/null +++ b/proposals/0471-SerialExecutor-isIsolated.md @@ -0,0 +1,226 @@ +# Improved Custom SerialExecutor isolation checking for Concurrency Runtime + +* Proposal: [SE-0471](0471-SerialExecutor-isIsolated.md) +* Author: [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.2)** +* Implementation: https://github.com/swiftlang/swift/pull/79788 & https://github.com/swiftlang/swift/pull/79946 +* Review: [Pitch](https://forums.swift.org/t/pitch-serialexecutor-improved-custom-serialexecutor-isolation-checking/78237/), [Review](https://forums.swift.org/t/se-0471-improved-custom-serialexecutor-isolation-checking-for-concurrency-runtime/78834), [Acceptance](https://forums.swift.org/t/accepted-se-0471-improved-custom-serialexecutor-isolation-checking-for-concurrency-runtime/79894) + +## Introduction + +In [SE-0424: Custom isolation checking for SerialExecutor](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0424-custom-isolation-checking-for-serialexecutor.md) we introduced a way for custom executors implementing the `SerialExecutor` protocol to assert and assume the static isolation if the dynamic check succeeded. This proposal extends these capabilities, allowing custom executors to not only "check and crash if assumption was wrong", but also check and act on the result of the check. + +## Motivation + +The previously ([SE-0424](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0424-custom-isolation-checking-for-serialexecutor.md)) introduced family of `Actor/assertIsolated()`, `Actor/preconditionIsolated()` and `Actor/assumeIsolated(operation:)` all rely on the `SerialExecutor/checkIsolated()` API which was introduced in the same proposal. + +These APIs all follow the "*pass or crash*" pattern. Where the crash is caused in order to prevent an incorrect assumption about isolation resulting in unsafe concurrent access to some isolated state. This is frequently used by methods which are "known to be called" on some specific actor to recover the dynamic (known at runtime) isolation information, into its static equivalent and therefore safely access some isolated state, like in this example: + +```swift +@MainActor +var counter: Int = num + +protocol P { + // Always called by the main actor, + // yet protocol author forgot to annotate the method or protocol using @MainActor + func actuallyWeKnowForSureThisIsCalledFromTheMainActor() +} + +struct Impl: P { + func actuallyWeKnowForSureThisIsCalledFromTheMainActor() { + MainActor.assumeIsolated { // we know this is safe here + counter += 1 + } + } +} + +@MainActor +func call(p: some P) { + p.actuallyWeKnowForSureThisIsCalledFromTheMainActor() +} +``` + +This works fine for many situations, however some libraries may need to be more careful when rolling out strict concurrency checks like these, and instead of crashing may want to choose to issue warnings before they adopt a mode that enforces correct use. + +Currently all APIs available are using the `checkIsolated()` API which must crash if called from a context not managed by the serial executor it is invoked on. This method is often implemented using `dispatchPrecondition()` when the serial executor is backed using `Dispatch` or custom `fatalError()` messages otherwise: + +```swift +final class ExampleExecutor: SerialExecutor { + func checkIsolated() { + dispatchPrecondition(condition: .onQueue(self.queue)) + } +} +``` + +This approach is better than not being able to participate in the checks at all (i.e. this API allows for advanced thread/queue sharing between actor and non-actor code), but it has two severe limitations: + +- the crash **messages** offered by these `checkIsolated()` crashes **are often sub-optimal** and confusing + - messages often don't include crucial information about which actor/executor the calling context was _actually_ executing on. Offering only "expected [...]" messages, leading to hard to debug crashes. +- it is **impossible** for the Swift runtime to offer **isolation violation warnings** + - because the Swift runtime _must_ call into a custom executor to verify its isolation, the "pass or crash" method will crash, rather than inform the runtime that a violation occurred and we should warn about it. + +Today, it is not possible for the Swift runtime to issue _warnings_ if something is detected to be on not the expected executor, but somehow we'd still like to continue without crashing the application. + +And for existing situations when `assumeIsolated()` is called and _should_ crash, by using this new API the Swift runtime will be able to provide _more informative_ messages, including all available context about the execution environment. + +## Proposed solution + +We propose to introduce a new `isIsolatingCurrentContext` protocol requirement to the `SerialExecutor` protocol: + +```swift +protocol SerialExecutor { + + /// May be called by the Swift runtime before `checkIsolated()` + /// in order to check for isolation violations at runtime, + /// and potentially issue isolation warnings. + /// + /// [...] + func isIsolatingCurrentContext() -> Bool + + // existing API, since SE-0424 + @available(SwiftStdlib 6.0, *) + func checkIsolated() // must crash if run on a context not managed by this serial executor + + // ... +} + +extension SerialExecutor { + /// Default implementation for backwards compatibility. + func isIsolatingCurrentContext() -> Bool? { nil } +} +``` + +The Swift runtime is free to call the `isIsolated` function whenever it wants to verify if the current context is appropriately isolated by some serial executor. + +In most cases implementing this new API is preferable to implementing `checkIsolated()`, as the Swift runtime is able to offer more detailed error messages when an isolation failure detected by a call to `isIsolatingCurrentContext()` is detected. + +## Detailed design + +The newly proposed `isIsolatingCurrentContext()` function participates in the previously established runtime isolation checking flow, and happens _before_ any calls to `checkIsolated()` are attempted. The following diagram explains the order of calls issued by the runtime to dynamically verify an isolation when e.g. `assumeIsolated()` is called: + +![diagram illustrating which method is called when](0471-is-isolated-flow.png) + +There are a lot of conditions here and availability of certain features also impacts this decision flow, so it is best to refer to the diagram for detailed analysis of every situation. However the most typical situation involves executing on a task, which has a potentially different executor than the `expected` one. In such situation the runtime will: + +- check for the existence of a "current" task, +- (fast path) if a task is present: + - compare the current task's serial executor it is isolated to (if any) with the expected serial executor, + +- :new: if **`isIsolatingCurrentContext`** **is available** on the `expected` executor: + - invoke `isIsolatingCurrentContext` + - ✅ if it returned true: pass the check. + - :x: if it returned false: fail the check. + - The runtime will **not** proceed to call `checkIsolated` after `isIsolated` is invoked! + +- if **`isIsolatingCurrentContext`** is **not available** on the expected executor, but **`checkIsolated`** **is available**: + - invoke `expected.checkIsolated` which will crash :x: or pass :white_check_mark: depending on its internal checking. +- if neither `checkIsolated` or `isIsolatingCurrentContext` are available, + - :x: crash with a best effort message. + + +This proposal specifically adds the "if `isIsolatingCurrentContext` is available" branch into the existing logic for confirming the isolation expectation. + +If `isIsolatingCurrentContext` is available, effectively it replaces `checkIsolated` because it does offer a sub-par error message experience and is not able to offer a warning if Swift would be asked to check the isolation but not crash upon discovering a violation. + +### The `isIsolatingCurrentContext` checking mode + +The `isIsolatingCurrentContext` method effectively replaces the `checkIsolated` method, because it can answer the same question if it is implemented. + +Some runtimes may not be able to implement a the returning `isIsolatingCurrentContext`, and they are not required to implement the new protocol requirement. + +The default implementation returns `nil` which is to be interpreted by the runtime as "unknown" or "unable to confirm the isolation", and the runtime may proceeed to call futher isolation checking APIs when this function returned `nil`. + +The general guidance about which method to implement is to implement `isIsolatingCurrentContext` whenever possible. This method can be used by the Swift runtime in "warning mode". When running a check in this mode, the `checkIsolated` method cannot and will not be used because it would cause an unexpected crash. An executor may still want to implement the `checkIsolated` function if it truly is unable to return a true/false response to the isolation question, but can only assert on an illegal state. The `checkIsolated` function will not be used when the runtime cannot tollerate the potential of crashing while performing an isolation check (e.g. isolated conformance checks, or when issuing warnings). + +The runtime will always invoke the `isIsolatingCurrentContext` before making attempts to call `checkIsolated`, and if the prior returns either `true` or `false`, the latter (`checkIsolated`) will not be invoked at all. + +### Checking if currently isolated to some `Actor` + +We also introduce a way to obtain `SerialExecutor` from an `Actor`, which was previously not possible. + +This API needs to be scoped because the lifetime of the serial executor must be tied to the Actor's lifetime: + +```swift +extension Actor { + /// Perform an operation with the actor's ``SerialExecutor``. + /// + /// This converts the actor's ``Actor/unownedExecutor`` to a ``SerialExecutor`` while + /// retaining the actor for the duration of the operation. This is to ensure the lifetime + /// of the executor while performing the operation. + @_alwaysEmitIntoClient + @available(SwiftStdlib 5.1, *) + public nonisolated func withSerialExecutor(_ operation: (any SerialExecutor) throws(E) -> T) throws(E) -> T + + /// Perform an operation with the actor's ``SerialExecutor``. + /// + /// This converts the actor's ``Actor/unownedExecutor`` to a ``SerialExecutor`` while + /// retaining the actor for the duration of the operation. This is to ensure the lifetime + /// of the executor while performing the operation. + @_alwaysEmitIntoClient + @available(SwiftStdlib 5.1, *) + public nonisolated func withSerialExecutor(_ operation: (any SerialExecutor) async throws(E) -> T) async throws(E) -> T + +} +``` + +This allows developers to write "warn if wrong isolation" code, before moving on to enable preconditions in a future release of a library. This gives library developers, and their adopters, time to adjust their code usage before enabling more strict validation mode in the future, for example like this: + +```swift +func something(operation: @escaping @isolated(any) () -> ()) { + operation.isolation.withSerialExecutor { se in + if !se.isIsolatingCurrentContext() { + warn("'something' must be called from the same isolation as the operation closure is isolated to!" + + "This will become a runtime crash in future releases of this library.") + } + } +} +``` + + + +This API will be backdeployed and will be available independently of runtime version of the concurrency runtime. + +### Compatibility strategy for custom SerialExecutor authors + +New executor implementations should prioritize implementing `isIsolatingCurrentContext` when available, using an appropriate `#if swift(>=...)` check to ensure compatibility. Otherwise, they should fall back to implementing the crashing version of this API: `checkIsolated()`. + +For authors of custom serial executors, adopting this feature is an incremental process and they can adopt it at their own pace, properly guarding the new feature with necessary availability guards. This feature requires a new version of the Swift concurrency runtime which is aware of this new mode and therefore able to call into the new checking function, therefore libraries should implement and adopt it, however it will only manifest itself when the code is used with a new enough concurrency runtime + +As a result, this change should cause little to no disruption to Swift concurrency users, while providing an improved error reporting experience if using executors which adopt this feature. + +## Source Compatibility + +This proposal is source compatible, a default implementation of the new protocol requirement is introduced along with it, allowing existing `SerialExecutors` to compile without changes. + +## Binary Compatibility + +This proposal is ABI additive, and does not cause any binary incompatibility risks. + +## Future directions + +### Expose `isIsolated` on (Distributed)Actor and MainActor? + +We are _not_ including new public API on Actor types because of concerns of this function being abused. + +If we determine that there are significant good use-cases for this method to be exposed, we might reconsider this position. + +## Alternatives considered + +### Somehow change `checkIsolated` to return a bool value + +This would be ideal, however also problematic since changing a protocol requirements signature would be ABI breaking. + +### Deprecate `checkIsolated`? + +In order to make adoption of this new mode less painful and not cause deprecation warnings to libraries which intend to support multiple versions of Swift, the `SerialExcecutor/checkIsolated` protocol requirement remains _not_ deprecated. It may eventually become deprecated in the future, but right now we have no plans of doing so. + +### Model the SerialExecutor lifetime dependency on Actor using `~Escapable` + +It is currently not possible to express this lifetime dependency using `~Escapable` types, because combining `any SerialExecutor` which is an `AnyObject` constrained type, cannot be combined with `~Escapable`. Perhaps in a future revision it would be possible to offer a non-escapable serial executor in order to model this using non-escapable types, rather than a `with`-style API. + +## Changelog + +- added way to obtain `SerialExecutor` from `Actor` in a safe, scoped, way. This enables using the `isIsolatingCurrentContext()` API when we have an `any Actor`, e.g. from an `@isolated(any)` closure. +- changed return value of `isIsolatingCurrentContext` from `Bool` to `Bool?`, where the `nil` is to be interpreted as "unknown", and the default implementation of `isIsolatingCurrentContext` now returns `nil`. +- removed the manual need to signal to the runtime that the specific executor supports the new checking mode. It is now detected by the compiler and runtime, checking for the presence of a non-default implementation of the protocol requirement. diff --git a/proposals/0471-is-isolated-flow.graffle b/proposals/0471-is-isolated-flow.graffle new file mode 100644 index 0000000000..05f3a5aa8d Binary files /dev/null and b/proposals/0471-is-isolated-flow.graffle differ diff --git a/proposals/0471-is-isolated-flow.png b/proposals/0471-is-isolated-flow.png new file mode 100644 index 0000000000..6a9e939edf Binary files /dev/null and b/proposals/0471-is-isolated-flow.png differ diff --git a/proposals/0472-task-start-synchronously-on-caller-context.md b/proposals/0472-task-start-synchronously-on-caller-context.md new file mode 100644 index 0000000000..1d266bfba4 --- /dev/null +++ b/proposals/0472-task-start-synchronously-on-caller-context.md @@ -0,0 +1,512 @@ +# Starting tasks synchronously from caller context + +* Proposal: [SE-0472](0472-task-start-synchronously-on-caller-context.md) +* Authors: [Konrad 'ktoso' Malawski](https://github.com/ktoso) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 6.2)** +* Implementation: + * https://github.com/swiftlang/swift/pull/79608 + * https://github.com/swiftlang/swift/pull/81428 + * https://github.com/swiftlang/swift/pull/81572 +* Review: ([pitch](https://forums.swift.org/t/pitch-concurrency-starting-tasks-synchronously-from-caller-context/77960/)) ([first review](https://forums.swift.org/t/se-0472-starting-tasks-synchronously-from-caller-context/78883)) ([returned for revision](https://forums.swift.org/t/returned-for-revision-se-0472-starting-tasks-synchronously-from-caller-context/79311)) ([second review](https://forums.swift.org/t/second-review-se-0472-starting-tasks-synchronously-from-caller-context/79683)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0472-starting-tasks-synchronously-from-caller-context/80037)) + +## Introduction + +Swift Concurrency's primary means of entering an asynchronous context is creating a Task (structured or unstructured), and from there onwards it is possible to call asynchronous functions, and execution of the current work may _suspend_. + +Entering the asynchronous context today incurs the creating and scheduling of a task to be executed at some later point in time. This initial delay may be wasteful for tasks which perform minimal or no (!) work at all. + +This initial delay may also be problematic for some situations where it is known that we are executing on the "right actor" however are *not* in an asynchronous function and therefore in order to call some different asynchronous function we must create a new task and introduce subtle timing differences as compared to just being able to call the target function–which may be isolated to the same actor we're calling from–immediately. + +## Motivation + +Today, the only way to enter an asynchronous execution context is to create a new task which then will be scheduled on the global concurrent executor or some specific actor the task is isolated to, and only once that task is scheduled execution of it may begin. + +This initial scheduling delay can be problematic in some situations where tight control over execution is required. While for most tasks the general semantics are a good choice–not risking overhang on the calling thread–we have found through experience that some UI or performance sensitive use-cases require a new kind of semantic: immediately starting a task on the calling context. After a suspension happens the task will resume on the the executor as implied by the task operation's isolation, as would be the case normally. + +This new behavior can especially beneficial for tasks, which *may run to completion very quickly and without ever suspending.* + +A typical situation where this new API may be beneficial often shows up with @MainActor code, such as: + +```swift +@MainActor var thingsHappened: Int = 0 + +@MainActor func asyncUpdateThingsHappenedCounter() async { + // for some reason this function MUST be async + thingsHappened += 1 +} + +func synchronousFunction() { + // we know this executes on the MainActor, and can assume so: + MainActor.assumeIsolated { + // The following would error: + // await asyncUpdateThingsHappenedCounter() + // because is it is an async call; cannot call from synchronous context + } + + // Using the newly proposed Immediate Task: + let task = Task.immediate { + // Now we CAN call the asynchronous function below: + await asyncUpdateThingsHappenedCounter() + } + + // cannot await on the `task` since still in synchronous context +} +``` + +The above example showcases a typical situation where this new API can be useful. While `assumeIsolated` gives us a specific isolation, but it would not allow us to call the async functions, as we are still in a synchronous context. + +The proposed `Task.immediate` API forms an async context on the calling thread/task/executor, and therefore allows us to call into async code, at the risk of overhanging on the calling executor. + +While this should be used sparingly, it allows entering an asynchronous context *synchronously*. + +## Proposed solution + +We propose the introduction of a new family of Task creation APIs collectively called "**immediate tasks**", which create a task and use the calling execution context to run the task's immediately, before yielding control back to the calling context upon encountering the first suspension point inside the immediate task. + +Upon first suspension inside the immediate task, the calling executor is freed up and able to continue executing other work, including the code surrounding the creation of the immediate task. This happens specifically when when a real suspension happens, and not for "potential suspension point" (which are marked using the `await` keyword). + +The canonical example for using this new API is using an *unstructured immediate task* like this: + +```swift +func synchronous() { // synchronous function + // executor / thread: "T1" + let task: Task = Task.immediate { + // executor / thread: "T1" + guard keepRunning() else { return } // synchronous call (1) + + // executor / thread: "T1" + await noSuspension() // potential suspension point #1 // (2) + + // executor / thread: "T1" + await suspend() // potential suspension point #2 // (3), suspend, (5) + // executor / thread: "other" + } + + // (4) continue execution + // executor / thread: "T1" +} +``` + +The task created by the `immediate` function begins running immediately _on the calling executor (and thread)_ without any scheduling delay. This new task behaves generally the same as any other unstructured task, it gets a copy of the outer context's task locals, and uses the surrounding context's base priority as its base priority as well. + +Since the task started running immediately, we're able to perform some calls immediately inside it, and potentially return early, without any additional scheduling delays. + +If a potential suspension point does not actually suspend, we still continue running on the calling context. For example, if potential suspension point `#1` did not suspend, we still continue running synchronously until we reach potential suspension point `#2` which for the sake of discussion let's say does suspend. At this point the calling thread continues executing the code that created the unstructured task. + +> You can refer to the `(N)` numbers in the above snippet to follow the execution order of this example execution. Specifically, once the execution reaches (3) the calling thread stops executing the unstructured task, and continues executing at (4). Eventually, when the unstructured task is resumed, it gets woken up at (5) and continues running on some other executor and/or thread. + +## Detailed design + +We propose the introduction of a family of APIs that allow for the creation of *immediate tasks*. + +The most frequent use of this API is likely going to be the unstructured task one. This is because we are able to enter an asynchronous context from a synchronous function using it: + +```swift +extension Task { + + @discardableResult + public static func immediate( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: consuming (any TaskExecutor)? = nil, + @_inheritActorContext(always) operation: sending @escaping () async throws(Failure) -> Success + ) -> Task + + @discardableResult + public static func immediateDetached( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: consuming (any TaskExecutor)? = nil, + @_inheritActorContext(always) operation: sending @escaping () async throws(Failure) -> Success + ) -> Task +} +``` + +We also introduce the same API for all kinds of task groups. These create child tasks, which participate in structured concurrency as one would expect of tasks created by task groups. + +```swift +extension (Throwing)TaskGroup { + // Similar semantics as the usual 'addTask'. + func addImmediateTask( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + operation: sending @escaping () async throws -> ChildTaskResult + ) + + // Similar semantics as the usual 'addTaskUnlessCancelled'. + func addImmediateTaskUnlessCancelled( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + operation: sending @escaping () async throws -> ChildTaskResult + ) +} + +extension (Throwing)DiscardingTaskGroup { + // Similar semantics as the usual 'addTask'. + func addImmediateTask( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + operation: sending @escaping () async throws -> ChildTaskResult + ) + + // Similar semantics as the usual 'addTaskUnlessCancelled'. + func addImmediateTaskUnlessCancelled( + name: String? = nil, // Introduced by SE-0469 + priority: TaskPriority? = nil, + executorPreference taskExecutor: (any TaskExecutor)? = nil, + operation: sending @escaping () async throws -> ChildTaskResult + ) +} +``` + +The `addImmediateTask` function mirrors the functionality of `addTask`, unconditionally adding the task to the task group, while the `addImmediateTaskUnlessCancelled` mirrors the `addTaskUnlessCancelled` which only adds the task to the group if the group (or task we're running in, and therefore the group as well) are not cancelled. + +### Isolation rules + +Due to the semantics of "starting on the caller context", the isolation rules of the closure passed to `Task.immediate` need to be carefully considered. + +The isolation rules for the `immediate` family of APIs need to account for this synchronous "first part" of the execution. We propose the following set of rules to make this API concurrency-safe: + +- The operation closure is `sending`. +- The operation closure may only specify an isolation (e.g. `{ @AnyGlobalActor in }`) + +Another significant way in which `Task.immediate` differs from the `Task.init` initializer is that the inheritance of the surrounding actor context is performed more eagerly. This is because immediate tasks always attempt to execute on the "current" executor, unlike `Task.init` which only execute on the "surrounding" actor context when the task's operation closure closes over an isolated parameter, or was formed in a global actor isolated context: + +```swift +@MainActor +func alreadyDefinitelyOnMainActor() { + Task { + // @MainActor isolated, enqueue + } + Task.immediate { + // @MainActor isolated, run immediately + } +} +``` + +```swift +actor Caplin { + var anything: Int = 0 + + func act() { + Task { + // nonisolated, enqueue on global concurrent executor + } + Task { + // self isolated, enqueue + self.anything // any capture of 'self' + } + + Task.immediate { // regardless of captures + // self isolated, run immediately + } + } +} + +func go(with caplin: isolated Caplin) async { + Task { + // nonisolated, enqueue on global concurrent executor + } + Task { + // 'caplin' isolated, enqueue + caplin.anything // any capture of 'caplin' + } + + Task.immediate { // regardless of captures + // 'caplin' isolated, run immediately + } + } +} + +func notSpecificallyIsolatedAnywhere() { + Task { + // nonisolated, enqueue on global concurrent executor + } + + Task.immediate { + // nonisolated. + // attempt to run on current executor, + // or enqueue to global as fallback + } + } +} +``` + +The `Task.immediateDetached` does not inherit isolation automatically, same as it's non-immediate `Task.detached` equivalent. + +Task group methods which create immediate child tasks do not inherit isolation automatically, although they are allowed to specify an isolation explicitly. This is the same as existing TaskGroup APIs (`addTask`). + +### Scheduling immediate tasks given matching current and requested isolation + +The Swift concurrency runtime maintains a notion of the "current executor" in order to be able to perform executor switching and isolation checking dynamically. This information is managed runtime, and is closely related to compile time isolation rules, but it is also maintained throughout nonisolated and synchronous functions. + +Immediate tasks make use of this executor tracking to determine on which executor we're asking the task to "immediately" execute. It is possible to start an immediate task in a synchronous context, and even require it to have some specific isolation. + +The following example invokes the synchronous `sayHello()` function from a `@MainActor` isolated function. The static information about this isolation is _lost_ by the synchronous function. And the compiler will assume, that the `sayHello()` function is not isolated to any specific context -- after all, the actual isolated context would depending on where we call it from, and we're not passing an `isolated` parameter to this synchronous function. + +By using an immediate task the runtime is able to notice that the requested, and current, executor are actually the same (`MainActor`) and therefore execute the task _immediately_ on the caller's executor _and_ with the expected `@MainActor` isolation, which is guaranteed to be correct: + +```swift +@MainActor var counterUsual = 0 +@MainActor var counterImmediate = 0 + +@MainActor +func sayHelloOnMain() { + sayHello() // call synchronous function from @MainActor +} + +// synchronous function +func sayHello() { + // We are "already on" the main actor + MainActor.assertIsolated() + + // Performs an enqueue and will execute task "later" + Task { @MainActor in + counterUsual += 1 + } + // At this point (in this specific example), `counterUsual` is still 0. + // We did not "give up" the main actor executor, so the new Task could not execute yet. + + // Execute the task immediately on the calling context (!) + Task.immediate { @MainActor in + counterImmediate += 1 + } + // At this point (in this specific cexample), + // `counterImmediate` is guaranteed to == 1! +} +``` + +The difference between the use of `Task.init` and `Task.immediate` is _only_ in the specific execution ordering semantics those two tasks exhibit. + +Because we are dynamically already on the expected executor, the immediate task will not need to enqueue and "run later" the new task, but instead will take over the calling executor, and run the task body immediately (up until the first suspension point). + +This can have importand implications about the observed order of effects and task execution, so it is important for developers to internalize this difference in scheduling semantics. + +If the same `sayHello()` function were to be invoked from some execution context _other than_ the main actor, both tasks–which specify the requested isolation to be `@MainActor`–will perform the usual enqueue and "run later": + +```swift +@MainActor var counterUsual = 0 +@MainActor var counterImmediate = 0 + +actor Caplin { + func sayHelloFromCaplin() { + sayHello() // call synchronous function from Caplin + } +} + +func sayHello() { + Task { @MainActor in // enqueue, "run later" + counterUsual += 1 + } + Task.immediate { @MainActor in // enqueue, "run later" + counterImmediate += 1 + } + + // at this point, no guarantees can be made ablue the values of the `counter` variables +} +``` + +This means that a `Task.immediate` can be used to opportunistically attempt to "run immediately, if the caller matches my required isolation, otherwise, enqueue and run the task later". Which is a semantic that many developers have requested in the past, most often in relation to the `MainActor`. + +The same technique of specifying a required target isolation may be used with the new TaskGroup APIs, such as `TaskGroup/addImmediateTask`. + +### Interaction with `Actor/assumeIsolated` + +In [SE-0392: Custom Actor Executors](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0392-custom-actor-executors.md) we introduced the ability to dynamically recover isolation information using the `Actor/assumeIsolated` API. It can be used to dynamically recover the runtime information about whether we are executing on some specific actor. + +The `assumeIsolated` shares some ideas with `Task/immediate` however it is distinctly different. For example, while both APIs can effectively be used to "notice we are running on the expected actor, and therefore perform some work on its context". However, `assumeIsolated` does _not_ create a new asynchronous context, while `Task.immediate` does: + +```swift +@MainActor +var state: Int = 0 + +@MainActor +func asyncMainActorMethod() async { } + +func synchronous() { + // assert that we are running "on" the MainActor, + // and therefore can access its isolated state: + MainActor.assumeIsolated { + num +=1 // ✅ ok + + await asyncMainActorMethod() // ❌ error: 'async' call in a function that does not support concurrency + } + +} +``` + +We can compose `assumeIsolated` with `Task.immediate` to both assert that the current execution context must the the expected actor, and form a new asynchronous task that will immediately start on that actor: + +```swift +func alwaysCalledFromMainActor() { // we know this because e.g. documentation, but the API wasn't annotated + MainActor.assumeIsolated { // @MainActor isolated + assert(num == 0) + Task.immediate { // @MainActor isolated + num +=1 // ✅ ok + assert(num == 1) // since we are guaranteed nothing else executed since the 'num == 0' assertion + + await asyncMainActorMethod() // ✅ ok + } + } +} +``` + +The immediately started task will not suspend and context switch until any of the called async methods does. For example, we are guaranteed that there will be no interleaved code execution between the `assert(num == 0)` in our example, and the `num += 1` inside the synchronously started task. + +After the suspension point though, there may have been other tasks executed on the main actor, and we should check the value of `num` again. + +### Immediate child tasks + +Immediate child tasks tasks can be created using the various `*TaskGroup/addImmediateTask*` methods behave similarily to their normal structured child task API counterparts (`*TaskGroup/addTask*`). + +Child tasks, including immediate child tasks, do not infer their isolation from the enclosing context, and by default are `nonisolated`. + +```swift +actor Worker { + func workIt(work: Work) async { + await withDiscardingTaskGroup { + group.addImmediateTask { // nonisolated + work.synchronousWork() + } + } + } +} +``` + +While the immediate task in the above example is indeed `nonisolated` and does not inherit the Worker's explicit isolation, it will start out immediately on the Worker's executor. Since this example features _no suspension points_ in the task group child tasks, this is effectively synchronously going to execute those child tasks on the caller (`self`). In other words, this is not performing any of its work in parallel. + +If we were to modify the work to have potential suspension points like so: + +```swift +actor Worker { + func workIt(work: Work) async { + await withDiscardingTaskGroup { + group.addImmediateTask { // nonisolated + // [1] starts on caller immediately + let partialResult = await work.work() // [2] actually suspends + // [3] resumes on global executor (or task executor, if there was one set) + work.moreWork(partialResult) + } + } + } +} +``` + +The actual suspension happening in the `work()` call, means that this task group actually would exhibit some amount of concurrent execution with the calling actor -- the remainder between `[2]` and `[3]` would execute on the global concurrent pool -- concurrently to the enclosing actor. + +Cancellation, task locals, priority escalation, and any other structured concurrency semantics remain the same for structured child tasks automatically for unstructured tasks created using the `Task/immediate[Detached]` APIs. + +The only difference in behavior is where these synchronously started tasks _begin_ their execution. + +## Source compatibility + +This proposal is purely additive, and does not cause any source compatibility issues. + +## ABI compatibility + +This proposal is purely ABI additive. + +## Alternatives considered + +### Dynamically asserting isolation correctness + +An important use case of this API is to support calling into an actor isolated context when in a synchronous function that is dynamically already running on that actor. This situation can occur both with instance actors and global actors, however the most commonly requested situation where this shows up is synchronous handler methods in existing frameworks, and which often may have had assumptions about the main thread, and did not yet annotate their API surface with @MainActor annotations. + +It would be possible to create a _dynamically asserting_ version of `Task.immediate`, which does handle the happy path where indeed we "know" where we're going to be called quite well, but gives a *false sense of security* as it may crash at runtime, in the same way the `Actor/preconditionIsolated()` or `Actor/assumeIsolated` APIs do. We believe we should not add more such dynamically crashing APIs, but rather lean into the existing APIs and allow them compose well with any new APIs that should aim to complement them. + +The dynamically asserting version would be something like this: + +```swift +// Some Legacy API: documented to be invoked on main thread but NOT @MainActor annotated and NOT 'async' +func onSomethingHappenedAlwaysOnMainThread(something: Something) { + // we "know" we are on the MainActor, however this is a legacy API that is not an 'async' method + // so we cannot call any other async APIs unless we create a new task. + Task.immediate { @MainActor in + await showThingy() + } +} + +func onSomethingHappenedSometimesOnMainThread(something: Something) { + // 💥 Must assert at runtime if not on main thread + Task.immediate { @MainActor in + await showThingy() + } +} + +func showThingy() async { ... } +``` + +This implementation approach yields safe looking code which unfortunately may have to assert at runtime, rather than further improve the compile time safety properties of Swift Concurrency. + +> See *Future Directions: Dynamically "run synchronously if in right context, otherwise enqueue as usual"* for a future direction that would allow implementing somewhat related APIs in a more elegant and correct way. + +### Banning from use in async contexts (@available(*, noasync)) + +During earlier experiments with such API it was considered if this API should be restricted to only non-async contexts, by marking it `@available(*, noasync)` however it quickly became clear that this API also has specific benefits which can be used to ensure certain ordering of operations, which may be useful regardless if done from an asynchronous or synchronous context. + +## Future Directions + +### Partial not-sending closure semantics + +The isolation rules laid out in this proposal are slightly more conservative than necessary. + +Technically one could make use of the information that the part of the closure up until the first potential suspension point is definitely running synchronously, and therefore even access state that would not be able to be accessed even under region isolation analysis rules. + +We believe that most common situations will be handled well enough by region analysis, and sending closures, however this is a future direction that could be explored if it becomes more apparent that implementing these more complex semantics would be very beneficial. + +For example, such analysis could enable the following: + +```swift +actor Caplin { + var num: Int = 0 + + func check() { + Task.immediateDetached { + num += 1 // could be ok; we know we're synchronously executing on caller + + try await Task.sleep(for: .seconds(1)) + + num += 1 // not ok anymore; we're not on the caller context anymore + } + + num += 1 // always ok + } +} +``` + +### Implementation detail: Expressing closure isolation tied to function parameter: `@isolated(to:)` + +The currently proposed API is working within the limitations of what is expressible in today's isolation model. It would be beneficial to be able to express the immediate API if we could spell something like "this closure must be isolated to the same actor as the calling function" which would allow for the following code: + +```swift +@MainActor +func test() { + Task.immediate { /* inferred to be @MainActor */ + num += 1 + } +} + +@MainActor var num = 0 +``` + +The way to spell this in an API could be something like this: + +```swift +public static func immediate( + ... + isolation: isolated (any Actor)? = #isolation, + operation: @escaping @isolated(to: isolation) sending async throws(Failure) -> Success, +) -> Task +``` + +The introduction of a hypothetical `@isolated(to:)` paired with an `isolated` `#isolation` defaulted actor parameter, would allow us to express "the *operation* closure statically inherits the exact same isolation as is passed to the isolation parameter of the `immediate` method". This naturally expresses the semantics that the `immediate` is offering, and would allow to _stay_ on that isolation context after resuming from the first suspension inside the operation closure. + +Implementing this feature is a large task, and while very desirable we are not ready yet to commit to implementing it as part of this proposal. If and when this feature would become available, we would adopt it in the `immediate` APIs. + +### Changelog + +- Moved the alternative considered of "attempt to run immediately, or otherwise just enqueue as usual" into the proposal proper diff --git a/proposals/0473-clock-epochs.md b/proposals/0473-clock-epochs.md new file mode 100644 index 0000000000..c674352d3c --- /dev/null +++ b/proposals/0473-clock-epochs.md @@ -0,0 +1,60 @@ +# Clock Epochs + +* Proposal: [SE-0473](0473-clock-epochs.md) +* Authors: [Philippe Hausler](https://github.com/phausler) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Accepted** +* Implementation: [PR #80409](https://github.com/swiftlang/swift/pull/80409) +* Review: ([pitch](https://forums.swift.org/t/pitch-suspendingclock-and-continuousclock-epochs/78017)) ([review](https://forums.swift.org/t/se-0473-clock-epochs/78923)) ([acceptance](https://forums.swift.org/t/accepted-se-0473-clock-epochs/79221)) + +## Introduction + +[SE-0329: Clock, Instant, and Duration](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0329-clock-instant-duration.md) introduced three concrete clock types: `SuspendingClock`, `ContinuousClock`, and `UTCClock`. While not all clocks have a meaningful concept of a reference or zero instant, `SuspendingClock` and `ContinuousClock` do, and having access to it can be useful. + +## Motivation + +The `Instant` type of a `Clock` represents a moment in time as measured by that clock. `Clock` intentionally imposes very few requirements on `Instant` because different kinds of clocks can have very different characteristics. Just because something does not belong on the generic `Clock` protocol, however, does not mean it shouldn't be exposed in the interface of a concrete clock type. + +Many clocks have a concept of a reference instant, also called an "epoch", that has special meaning for the clock. For example, the Unix `gettimeofday` function measures the nominal elapsed time since 00:00 UTC on January 1st, 1970, an instant often called the "Unix epoch". Swift's `SuspendingClock` and `ContinuousClock` are defined using system facilities that similarly measure time relative to an epoch, and while the exact definition of the epoch is system-specific, it is at least consistent for any given system. This means that durations since the epoch can be meaningfully compared within the system, even across multiple processes or with code written in other languages (as long as they use the same system facilities). + +## Proposed solution + +Two new properties will be added, one to `SuspendingClock` and another to `ContinuousClock`. These properties define the system epoch that all `Instant` types for the clock are derived from; practically speaking, this is the "zero" point for these clocks. Since the values may be relative to the particular system they are being used on, their names reflect that they are a system-specific definition and should not be expected to be consistent (or meaningfully serializable) across systems. + +## Detailed design + +```swift +extension ContinuousClock { + public var systemEpoch: Instant { get } +} + +extension SuspendingClock { + public var systemEpoch: Instant { get } +} +``` + +On most platforms, including Apple platforms, Linux, and Windows, the system epoch of these clocks is set at boot time, and so measurements relative to the epoch can used to gather information such as the uptime or active time of a system: + +```swift +let clock = ContinousClock() +let uptime = clock.now - clock.systemEpoch +``` + +Likewise: + +```swift +let clock = SuspendingClock() +let activeTime = clock.now - clock.systemEpoch +``` + +However, this cannot be guaranteed for all possible platforms. A platform may choose to use a different instant for its system epoch, perhaps because the concept of uptime doesn't apply cleanly on the platform or because it is intentionally not exposed to the programming environment for privacy reasons. + +## ABI compatibility + +This is a purely additive change and provides no direct impact to existing ABI. It only carries the ABI impact of new properties being added to an existing type. + +## Alternatives considered + +We considered adding a constructor or static member to `SuspendingClock.Instant` and `ContinousClock.Instant` instead of on the clock. However, placing it on the clock itself provides a more discoverable and nameable location. + +As proposed, `systemEpoch` is an informal protocol that works across multiple clock implementations. We consider formalizing it as a new protocol, but ultimately we decided not to because no generic function made much sense that would not be better served with generic specialization or explicit clock parameter types. diff --git a/proposals/0474-yielding-accessors.md b/proposals/0474-yielding-accessors.md new file mode 100644 index 0000000000..a109cee548 --- /dev/null +++ b/proposals/0474-yielding-accessors.md @@ -0,0 +1,668 @@ +# Yielding accessors + +* Proposal: [SE-0474](0474-yielding-accessors.md) +* Authors: [Ben Cohen](https://github.com/airspeedswift), [Nate Chandler](https://github.com/nate-chandler), [Joe Groff](https://github.com/jckarter/) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Accepted** +* Vision: [A Prospective Vision for Accessors in Swift](https://github.com/rjmccall/swift-evolution/blob/accessors-vision/visions/accessors.md) +* Implementation: Partially available on main behind the frontend flag `-enable-experimental-feature CoroutineAccessors` +* Review: ([pitch 1](https://forums.swift.org/t/modify-accessors/31872)), ([pitch 2](https://forums.swift.org/t/pitch-modify-and-read-accessors/75627)), ([pitch 3](https://forums.swift.org/t/pitch-3-yielding-coroutine-accessors/77956)), ([review](https://forums.swift.org/t/se-0474-yielding-accessors/79170)), [Acceptance](https://forums.swift.org/t/accepted-se-0474-yielding-accessors/80273) + +## Introduction + +We propose the introduction of two new accessors, `yielding mutate` and `yielding borrow`, for implementing computed properties and subscripts alongside the current `get` and `set`. + +By contrast with `get` and `set`, whose bodies behave like traditional methods, the body of a `yielding` accessor will be a coroutine, using a new contextual keyword `yield` to pause the coroutine and lend access of a value to the caller. +When the caller ends its access to the lent value, the coroutine's execution will continue after `yield`. + +These `yielding` accessors enable values to be accessed and modified without requiring a copy. +This is essential for noncopyable types and often desirable for performance even with copyable types. + +This feature has been available (but not supported) since Swift 5.0 via the `_modify` and `_read` keywords. +Additionally, the feature is available via `read` and `modify` on recent builds from the compiler's `main` branch using the flag `-enable-experimental-feature CoroutineAccessors`. + +## Motivation + +### `yielding mutate` + +Swift's `get`/`set` syntax allows users to expose computed properties and subscripts that behave as l-values. +This powerful feature allows for the creation of succinct idiomatic APIs, such as this use of `Dictionary`'s defaulting subscript: + +```swift +var wordFrequencies: [String:Int] = [:] +wordFrequencies["swift", default: 0] += 1 +// wordFrequencies == ["swift":1] +``` + +While this provides the illusion of "in-place" mutation, it actually involves three separate operations: +1. a `get` of a copy of the value +2. the mutation, performed on that returned value +3. finally, a `set` replacing the original value with the mutated temporary value. + +This can be seen by performing side effects inside of the getter and setter; for example: + +```swift +struct GetSet { + var x: String = "👋🏽 Hello" + + var property: String { + get { print("Getting",x); return x } + set { print("Setting",newValue); x = newValue } + } +} + +var getSet = GetSet() +getSet.property.append(", 🌍!") +// prints: +// Getting 👋🏽 Hello +// Setting 👋🏽 Hello, 🌍! +``` + +#### Performance + +When the property or subscript is of copyable type, this simulation of in-place mutation works well for user ergonomics but has a major performance shortcoming, which can be seen even in our simple `GetSet` type above. +Strings in Swift aren't bitwise-copyable types; once they grow beyond a small fixed size, they allocate a reference-counted buffer to hold their contents. +Mutation is handled via the copy-on-write technique: +When you make a copy of a string, only the reference to the buffer is copied, not the buffer itself. +Then, when either copy of the string is mutated, the mutation operation checks if the buffer is uniquely referenced. +If it isn't (because the string has been copied), it first duplicates the buffer before mutating it, preserving the value semantics of `String` while avoiding unnecessary eager copies. + +Given this, we can see a performance problem when appending to `GetSet.property` in our example above: + +- `GetSet.property { get }` is called and returns a copy of `x`. +- Because a copy is returned, the buffer backing the string is no longer uniquely referenced. +- The append operation must therefore duplicate the buffer before mutating it. +- `GetSet.property { set }` writes this copy back over the top of `x`, destroying the original string. +- The original buffer's reference count drops to zero, and it's destroyed too. + +So, despite looking like in-place mutation, every mutating operation on `x` made through `property` is actually causing a full copy of `x`'s backing buffer. +This is a linear-time operation. +If we appended to this property in a loop, that loop would end up being quadratic in complexity. +This is likely very surprising to the developer and is a frequent performance pitfall. + +[An accessor](#design-modify) which _lends_ a value to the caller in place, without producing a temporary copy, is desirable to avoid this quadratic time complexity while preserving Swift's ability for computed properties to provide expressive and ergonomic APIs. + +#### Non-`Copyable` types + +When the value being mutated is noncopyable, the common `get`/`set` pattern often becomes impossible to implement: +the very first step of mutation makes a copy! +Thus, `get` and `set` can't be used to wrap access to a noncopyable value: + +```swift +struct UniqueString : ~Copyable {...} + +struct UniqueGetSet : ~Copyable { + var x: UniqueString + + var property: UniqueString { + get { // error: 'self' is borrowed and cannot be consumed + x + } + set { x = newValue } + } +} +``` + +`get` borrows `self` but then tries to _give_ ownership of `x` to its caller. Since `self` is only borrowed, it cannot give up ownership of its `x` value, and since `UniqueString` is not `Copyable`, `x` cannot be copied either, making `get` impossible to implement. +Therefore, to provide computed properties on noncopyable types with comparable expressivity, we again need [an accessor](#design-modify) that borrows `self` and _lends_ access to `x` to its caller without implying independent ownership. + +### `yielding borrow` + +For properties and subscripts of noncopyable type, the current official accessors are insufficient not only for mutating, but even for _inspecting_. +Even if we remove `set` from our `UniqueGetSet` type above, we still hit the same error in its getter: + +```swift +struct UniqueString : ~Copyable {...} + +struct UniqueGet : ~Copyable { + var x: UniqueString + + var property: UniqueString { + get { // error: 'self' is borrowed and cannot be consumed + return x + } + } +} +``` + +As discussed above, `UniqueGet.property { get }` borrows `self` but attempts to transfer ownership of `self.x` to the caller, which is impossible. + +This particular error could be addressed by marking the getter `consuming`: + +```swift +struct UniqueString : ~Copyable {...} + +struct UniqueConsumingGet : ~Copyable { + var x: UniqueString + + var property: UniqueString { + consuming get { + return x + } + } +} +``` + +This allows the getter to take ownership of the `UniqueConsumingGet`, +allowing it to destructively extract `x` and transfer ownership to the caller. +Here's how that looks in the caller: + +```swift +let container = UniqueConsumingGet() +let x = container.property // consumes container! +// container is no longer valid +``` + +This might sometimes be desirable, but for many typical uses of properties and subscripts, it is not. If a container holds a number of noncopyable fields, it should be possible to inspect each field in turn, but doing so wouldn't be possible if inspecting one consumes the container. + +Similar to the mutating case, what's needed here is [an accessor](#design-read) which _borrows_ `self` and which _lends_ `x`--this time immutably--to the caller. + +## Proposed solution + +We propose two new accessor kinds: +- `yielding mutate`, to enable mutating a value without first copying it +- `yielding borrow`, to enable inspecting a value without copying it. + +## Detailed design + +### `yielding borrow` + +[`UniqueGet`](#read-motivation) can now allow its clients to inspect its field non-destructively with `yielding borrow`: + +```swift +struct UniqueString : ~Copyable {...} + +struct UniqueBorrow : ~Copyable { + var x: UniqueString + + var property: UniqueString { + yielding borrow { + yield x + } + } +} +``` + +The `UniqueBorrow.property { yielding borrow }` accessor is a "yield-once coroutine". +When called, it borrows `self`, and then runs until reaching a `yield`, at which point it suspends, lending the yielded value back to the caller. +Once the caller is finished accessing the value, it resumes the accessor's execution. +The accessor continues running where it left off after the `yield`. + +If a a property or subscript provides a `yielding borrow`, it cannot also provide a `get`. + +#### `yielding borrow` as a protocol requirement + +Such accessors should be usable on values of generic and existential type. +To indicate that a protocol provides immutable access to a property or subscript via a `yielding borrow` coroutine, we propose allowing `yielding borrow` to appear where `get` does today: + +```swift +protocol Containing { + var property: UniqueString { yielding borrow } +} +``` + +A protocol requirement cannot specify both `yielding borrow` and `get`. + +A `yielding borrow` requirement can be witnessed by a stored property, a `yielding borrow` accessor, or a getter. + +#### `get` of noncopyable type as a protocol requirement + +Property and subscript requirements in protocols have up to this point only been able to express readability in terms of `get` requirements. +This becomes insufficient when the requirement has a noncopyable type: + +```swift +protocol Producing { + var property: UniqueString { get } +} +``` + +To fulfill such a requirement, the conformance must provide a getter, meaning its implementation must be able to produce a value whose ownership can be transferred to the caller. +In practical terms, this means that the requirement cannot be witnessed by a stored property or a `yielding borrow` accessor when the result is of noncopyable type,[^2] since the storage of a stored property is owned by the containing aggregate and the result of a `yielding borrow` is owned by the suspended coroutine, and it would be necessary to copy to provide ownership to the caller. +However, if the type of the `get` requirement is copyable, the compiler can synthesize the getter from the other accessor kinds by introducing copies as necessary. + +[^2]: While the compiler does currently accept such code, it does so by interpreting that `get` as a `yielding borrow`, which is a bug. + +### `yielding mutate` + +The `GetSet` type [above](#modify-motivation) could be implemented with `yielding mutate` as follows: + +```swift +struct GetMutate { + var x: String = "👋🏽 Hello" + + var property: String { + get { print("Getting", x); return x } + yielding mutate { + print("Yielding", x) + yield &x + print("Post yield", x) + } + } +} + +var getMutate = GetMutate() +getMutate.property.append(", 🌍!") +// prints: +// Yielding 👋🏽 Hello +// Post yield 👋🏽 Hello, 🌍! +``` + +Like `UniqueBorrow.property { yielding borrow }` above, `GetMutate.property { yielding mutate }` is a yield-once coroutine. +However, the `yielding mutate` accessor lends `x` to the caller _mutably_. + +Things to note about this example: +* `get` is never called — the property access is handled entirely by the `yielding mutate` accessor +* the `yield` is similar to a `return`, but control returns to the `yielding mutate` after the `append` completes +* there is no more `newValue` – the yielded value is modified by `append` +* because it's granting _mutable_ access to the caller, `yield` uses the `&` sigil, similar to passing an argument `inout` + +Unlike the `get`/`set` pair, the `yielding mutate` accessor is able to safely provide access to the yielded value without copying it. +This can be done safely because the accessor preserves ownership of the value until it has completely finished running: +When it yields the value, it only lends it to the caller. +The caller is exclusively borrowing the value yielded by the coroutine. + +`get` is still used when only fetching, not modifying, the property: + +```swift +_ = getMutate.property +// prints: +// Getting 👋🏽 Hello, 🌍! +``` + +`yielding mutate` is sufficient to allow assignment to a property: + +``` +getMutate.property = "Hi, 🌍, 'sup?" +// prints: +// Yielding 👋🏽 Hello, 🌍! +// Post yield Hi, 🌍, 'sup? +``` + +It is, however, also possible to supply _both_ a `yielding mutate` and a `set` accessor. +The `set` accessor will be favored in the case of a whole-value reassignment, which may be more efficient than preparing and yielding a value to be overwritten: + +```swift +struct GetSetMutate { + var x: String = "👋🏽 Hello" + + var property: String { + get { x } + yielding mutate { yield &x } + set { print("Setting",newValue); x = newValue } + } +} +var getSetMutate = GetSetMutate() +getSetMutate.property = "Hi 🌍, 'sup?" +// prints: +// Setting Hi 🌍, 'sup? +``` + +#### Pre- and post-processing in `yielding mutate` + +As with `set`, `yielding mutate` gives the property or subscript implementation an opportunity to perform some post-processing on the new value after assignment. Even in these cases, `yielding mutate` can often allow for a more efficient implementation than a traditional `get`/`set` pair would provide. +Consider the following implementation of an enhanced version of `Array.first` that allows the user to modify the first value of the array: + +```swift +extension Array { + var first: Element? { + get { isEmpty ? nil : self[0] } + yielding mutate { + var tmp: Optional + if isEmpty { + tmp = nil + yield &tmp + if let newValue = tmp { + self.append(newValue) + } + } else { + tmp = self[0] + yield &tmp + if let newValue = tmp { + self[0] = newValue + } else { + self.removeFirst() + } + } + } + } +} +``` + +This implementation takes a similar approach to `Swift.Dictionary`'s key-based subscript: +if there is no `first` element, the accessor appends one. +If `nil` is assigned, the element is removed. +Otherwise, the element is updated to the value from the assigned `Optional`'s payload. + +Because the fetch and update code are all contained in one block, the `isEmpty` check is not duplicated; if the same logic were implemented with a `get`/`set` pair, `isEmpty` would need to be checked independently in both accessors. With `yielding mutate`, whether the array was initially empty is part of the accessor's state, which remains present until the accessor is resumed and completed. + +#### Taking advantage of exclusive access + +The optional return value of `first` in the code above means that, despite using `yielding mutate`, we have reintroduced the problem of triggering copy-on-write when mutating through the `first` property. +That act of placing the value in the array inside of an `Optional` (namely, `tmp = self[0]`) creates a copy. + +Like a `set` accessor, a `yielding mutate` accessor in a `struct` or `enum` property is `mutating` by default (unless explicitly declared a `nonmutating yielding mutate`). +This means that the `yielding mutate` accessor has exclusive access to `self`, and that exclusive access extends for the duration of the accessor's execution, including its suspension after `yield`-ing. +Since the implementation of `Array.first { yielding mutate }` has exclusive access to its underlying buffer, it can move the value of `self[0]` directly into the `Optional`, yield it, and then move the result back after being resumed: + +```swift +extension Array { + var first: Element? { + yielding mutate { + var tmp: Optional + if isEmpty { + // Unchanged + } else { + // Illustrative code only, Array's real internals are fiddlier. + // _storage is an UnsafeMutablePointer to the Array's storage. + + // Move first element in _storage into a temporary, leaving that slot + // in the storage buffer as uninintialized memory. + tmp = _storage.move() + + // Yield that moved value to the caller + yield &tmp + + // Once the caller returns, restore the array to a valid state + if let newValue = tmp { + // Re-initialize the storage slot with the modified value + _storage.initialize(to: newValue) + } else { + // Element removed. Slide other elements down on top of the + // uninitialized first slot: + _storage.moveInitialize(from: _storage + 1, count: self.count - 1) + self.count -= 1 + } + } + } +} +``` + +While the `yielding mutate` coroutine is suspended after yielding, the `Array` is left in an invalid state: the memory location where the first element is stored is left uninitialized, and must not be accessed. +However, this is safe thanks to Swift's rules preventing conflicting access to memory. +Unlike a `get`, the `yielding mutate` is guaranteed to have an opportunity to put the element back (or to remove the invalid memory if the entry is set to `nil`) after the caller resumes it, restoring the array to a valid state in all circumstances before any other code can access it. + +### The "yield once" rule + +Notice that there are _two_ yields in this `Array.first { yielding mutate }` implementation, for the empty and non-empty branches. +Nonetheless, exactly one `yield` is executed on any path through the accessor. +In general, the rules for yields in yield-once coroutines are similar to those of deferred initialization of `let` variables: +it must be possible for the compiler to guarantee there is exactly one yield on every path. +In other words, there must not be a path through the yield-once coroutine's body with either zero[^1] or more than one yield. +The `Array.first` example is valid since there is a `yield` in both the `if` and the `else` branch. +In cases where the compiler cannot statically guarantee this, refactoring or use of `fatalError()` to assert unreachable code paths may be necessary. + +[^1]: Note that it is legal for a path without any yields to terminate in a `fatalError`. Such a path is not _through_ the function. + +### Throwing callers + +The `Array.first { yielding mutate }` implementation above is correct even if the caller throws while the coroutine is suspended. + +```swift +try? myArray.first?.throwingMutatingOp() +``` + +Thanks to Swift's rule that `inout` arguments be initialized at function exit, the element must be a valid value when `throwingMutatingOp` throws. +When `throwingMutatingOp` does throw, control returns back to the caller. +The body of `Array.first { yielding mutate }` is resumed, and `tmp` is a valid value. +Then the code after the `yield` executes. +The coroutine cleans up as usual, writing the updated temporary value in `tmp` back into the storage buffer. + +## Source compatibility + +The following code is legal today: + +```swift +func borrow(_ c : () -> T) -> T { c() } +var reader : Int { + borrow { + fatalError() + } +} +``` + +Currently, the code declares a property `reader` with an implicit getter. +The implicit getter has an implicit return. +The expression implicitly returned is a call to the function `borrow` with a trailing closure. + +An analogous situation exists for `mutate`. + +This proposal takes the identifiers `borrow` and `mutate` as contextual keywords +as part of the `yielding borrow` and `yielding mutate` accessor declarations. +By themselves, these two new accessor forms do not immediately require a source break; however, as the [accessors vision](https://github.com/rjmccall/swift-evolution/blob/accessors-vision/visions/accessors.md) lays out, we will more than likely want to introduce non-`yielding` variants of `borrow` and `mutate` in the near future as well. + +Therefore, we propose an alternate interpretation for this code: +that it declare a property `reader` with a `borrow` accessor. +Although such an accessor does not yet exist, making the syntax change now will prepare the language for subsequent accessors we introduce in the near future. + +If this presents an unacceptable source compatibility break, the change may have to be gated on a language version. + +## ABI compatibility + +Adding a `yielding mutate` accessor to an existing read-only subscript or computed property has the same ABI implications as adding a setter; it must be guarded by availability on ABI-stable platforms. +The stable ABI for mutable properties and subscripts provides `get`, `set`, and `_modify` coroutines, deriving `set` from `_modify` or vice versa if one is not explicitly implemented. +Therefore, adding or removing `yielding mutate`, adding or removing `set`, replacing a `set` with a `yielding mutate`, or replacing `yielding mutate` with `set` are all ABI-compatible changes, so long as the property or subscript keeps at least one of `set` or `yielding mutate`, and those accessors agree on whether they are `mutating` or `nonmutating` on `self`. + +By contrast, `yielding borrow` and `get` are mutually exclusive. Replacing one with the other is always an ABI-breaking change. +Replacing `get` with `yielding borrow` is also an API-breaking change for properties or subscripts of noncopyable type. + +Renaming the current `_modify` (as used by the standard library, e.g.) to `yielding mutate` is an ABI additive change: a new `yielding mutate` symbol will be added. +When the compiler sees a `yielding mutate` with an early enough availability, the compiler will synthesize a corresponding `_modify` whose body will just forward to `yielding mutate`. +This is required for ABI stability: code compiled against an older standard library which calls `_modify` will continue to do so. +Meanwhile, code compiled against a newer standard library will call the new `yielding mutate`. +The same applies to renaming `_read` to `yielding borrow`. + +## Implications on adoption + +### Runtime support + +The new ABI will require runtime support which would need to be back deployed in order to be used on older deployment targets. + +### When to favor coroutines over `get` and `set` + +Developers will need to understand when to take advantage of `yielding` accessors, and we should provide some guidance: + +#### `yielding mutate` + +`yielding mutate` is desirable when the value of a property or subscript element is likely to be subject to frequent in-place modifications, especially when the value's type uses copy-on-write and the temporary value created by a `get`-update-`set` sequence is likely to trigger full copies of the buffer that could otherwise be avoided. +Additionally, if the mutation operation involves complex setup and teardown that requires persisting state across the operation, `yielding mutate` allows for arbitrary state to be preserved across the access. + +For noncopyable types, implementing `yielding mutate` is necessary to provide in-place mutation support if the value of the property or subscript cannot be read using a `get`. + +Even in cases where `yielding mutate` is beneficial, it is often still profitable to also provide a `set` accessor for whole-value reassignment. `set` may be able to avoid setup necessary to support an arbitrary in-place mutation and can execute as a standard function call, which may be more efficient than a coroutine. + +#### `yielding borrow` + +For read-only operations on `Copyable` types, the performance tradeoffs between `get` and `yielding borrow` are more subtle. +`borrowing yield` can avoid copying a result value that is stored as part of the originating value. +On the other hand, `get` can execute as a normal function without the overhead of a coroutine. +(As part of the implementation work for this proposal, we are implementing a more efficient coroutine ABI that should be significantly faster than the implementation used by `_read` and `_modify` today; however, a plain function call is likely to remain somewhat faster.) +`yielding borrow` may nonetheless be preferable for providing access to large value types in memory that would be expensive to copy as part of returning from a `get`. + +`yielding borrow` also introduces the ability to perform both setup work before and teardown work after the caller has accessed the value, whereas `get` can at best perform setup work before returning control to the caller. +Particularly in conjunction with non-`Escapable` types, which could eventually be lifetime-bound to their access, a `yielding borrow` accessor producing a non-`Escapable` value could provide limited access to a resource in a similar manner to `withSomething { }` closure-based APIs, without the "pyramid of doom" or scoping limitations of closure-based APIs. + +For properties and subscripts with noncopyable types, the choice between `yielding borrow` and `get` will often be forced by whether the API contract specifies a borrowed or owned result. +An operation that provides temporary borrowed access to a component of an existing value without transferring ownership, when that value is expected to persist after the access is complete, must be implemented using `yielding borrow`. +An operation that computes a new value every time is best implemented using `get`. +(Such an operation can in principle also be implemented using `yielding borrow`; it would compute the temporary value, yield a borrow to the caller, and then destroy the temporary value when the coroutine is resumed. +This would be inefficient, and would furthermore prevent the caller from being able to perform consuming operations on the result.) + +For polymorphic interfaces such as protocol requirements and overridable class members that are properties or subscripts of noncopyable type, the most general requirement is `yielding borrow`, since a `yielding borrow` requirement can be satisfied by either a `yielding borrow` or a `get` implementation (in the latter case, by wrapping the getter in a synthesized `yielding borrow` that invokes the getter, yields a borrow the result, then destroys the value on resume). +Using `yielding borrow` in polymorphic or ABI-stable interfaces may thus desirable to allow for maximum evolution flexibility if being able to consume the result is never expected to be part of the API contract. + +## Future directions + +### Yield-once functions + +Further ergonomic enhancements to the language may be needed over time to make the most of this feature. +For example, coroutine accessors do not compose well with functions because functions cannot themselves currently yield values. +In the future, it may be desirable to also support yield-once functions: + +```swift +var value: C { yielding mutate { ... } } +yielding func updateValue(...) -> inout C { + yield &self.value + additionalWork(value) +} +``` + +### Forwarding both consuming and borrowing accesses to the same declaration + +When a property or subscript has a `consuming get`, a caller can take ownership of the field at the expense of also destroying the rest of object. +When a property or subscript has a `yielding borrow` accessor, a caller can borrow the field to inspect it without taking ownership of it. + +As proposed here, it's not yet possible for a single field to provide both of these behaviors to different callers. +Since both of these behaviors have their uses, and many interfaces want to provide "perfect forwarding" where any of consuming, mutating, or borrowing operations on a wrapper can be performed by passing through the same operation to an underlying value, it may be desirable in the future to allow a single field to provide both a `yielding borrow` and a `consuming get`: + +```swift +subscript(index: Int) -> Value { + consuming get {...} + yielding borrow {...} +} +``` + +The rules that Swift currently uses to determine what accessor(s) to use in order to evaluate an expression are currently only defined in terms of whether the access is mutable or not, so these rules would need further elaboration to distinguish non-mutating accesses by whether they are consuming or borrowing in order to determine which accessor to favor. + +### Transitioning between owned and borrowed accessors for API evolution + +When a noncopyable API first comes into existence, its authors may not want to commit to it producing an owned value. As discussed [above](#read-implications), providing a `yielding borrow` accessor is often the most conservative API choice for noncopyable properties and subscripts: + +```swift +subscript(index: Int) -> Value { + yielding borrow {...} +} +``` + +As their module matures, however, the authors may decide that committing to providing a consumable value is worthwhile. +To support this use-case, in the future, it may be desirable to allow for forward-compatible API and ABI evolution by promoting a `yielding borrow` accessor to `get`: + +```swift +subscript(index: Int) -> Value { + // Deprecate the `yielding borrow` + @available(*, deprecated) + yielding borrow {...} + + // Favor the `get` for new code + get {...} +} +``` + + +That would enable the module to evolve to a greater commitment while preserving ABI. + +For APIs of copyable type, the reverse evolution is also a possibility; an API could originally be published using a `get` and later decide that a `yielding borrow` accessor is overall more efficient. + +Since it would typically be ambiguous whether the `yielding borrow` or `get` should be favored at a use site, it would make sense to require that one or the other accessor be deprecated or have earlier availability than the other in order to show that one is unambiguously preferred over the other for newly compiled code with recent enough availability. + +### Non-`yielding` `borrow` and `mutate` accessors + +A `yielding` accessor lends the value it yields to its caller. +The caller only has access to that value until it resumes the coroutine. +After the coroutine is resumed, it has the opportunity to clean up. +This enables a `yielding borrow` or `mutate` to do interesting work, such as constructing a temporary aggregate from its base object's fields: + +```swift +struct Pair : ~Copyable { + var left: Left + var right: Right + + var reversed: Pair { + yielding mutate { + let result = Pair(left: right, right: left) + yield &result + self = .init(left: result.right, right: result.left) + } + } +} +``` + +That the access ends when the coroutine is resumed means that the lifetime of the lent value is strictly shorter than that of the base value. +In the example above, the lifetime of `reversed` is shorter than that of the `Pair` it is called on. + +As discussed in the [accessors vision](https://github.com/rjmccall/swift-evolution/blob/accessors-vision/visions/accessors.md), when a value is merely being projected from the base object and does not need any cleanup after being accessed, this is undesirably limiting: +a value projected from a base naturally has _the same_ lifetime as the base. + +This is especially problematic in the context of composition with non-`Escapable` types. +Consider the following wrapper type[^3]: + +[^3]: This example involves writing out a `yielding borrow` accessor. The same issue exists when the compiler synthesizes a `yielding borrow` accessor for a stored property exported from a resilient module. + +```swift +struct Wrapper : ~Copyable & ~Escapable { + var _stuffing: Stuffing + + var stuffing: Stuffing { + yielding borrow { + yield _stuffing + } + } +} +``` + +When the instance of `Wrapper` is local to a function, the strict nesting of lifetimes may not immediately be a problem: + +```swift +{ + let wrapper: Wrapper = ... + borrowStuffing(wrapper.stuffing) + // lifetime of wrapper.stuffing ends (at coroutine resumption) + // lifetime of wrapper ends +} +``` + +But when `Wrapper` is a parameter, or otherwise nonlocal to the function, it is natural to expect to return the `Stuffing` back to the source of the `Wrapper`, but the `yielding borrow` accessor artificially limits its lifetime: + +```swift +@lifetime(borrow wrapper) +func getStuffing(from wrapper: borrowing Wrapper) -> Stuffing { + return wrapper.stuffing // error +} +``` + +The issue is that the lifetime of `stuffing` ends _within_ `getStuffing`, when the `yielding borrow` coroutine is resumed, which prevents `stuffing` from being returned. + +To address use cases like this, in the future, it may be desirable to introduce a variant of `borrow` and `mutate` accessors that immediately `return`s the borrowed or mutable value without involving a coroutine: + +```swift +var stuffing: Stuffing { + borrow { + return _stuffing + } +} +``` + +A non-`yielding` `borrow` or `mutate` accessor would be limited to returning values that can be accessed immediately and which do not require any cleanup (implicit or explicit) when the access ends. As such, the `yielding` variants would still provide the most flexibility in implementation, at the cost of the additional lifetime constraint of the coroutine. Similar to the [discussion around evolving `yielding borrow` accessors into `get` accessors](#ownership-evolution), there is also likely to be a need to allow for APIs initially published in terms of `yielding` accessors to transition to their corresponding non-`yielding` accessors in order to trade stronger implementation guarantees for more flexibility in the lifetime of derived `~Escapable` values. + +## Alternatives considered + +### Unwinding the accessor when an error is thrown in the caller + +A previous version of this proposal specified that if an error is thrown in a coroutine caller while a coroutine is suspended, the coroutine is to "unwind" and the code after the `yield` is not to run. +In the [example above](#throwing-callers), the code after the `yield` would not run if `throwingMutatingOp` threw an error. + +This approach was tied up with the idea that a `yielding mutate` accessor might clean up differently if an error was thrown in the caller. +The intervening years of experience with the feature have not borne out the need for this. +If an error is thrown in a caller into which a value has been yielded, the _caller_ must put the yielded mutable value back into a consistent state. +As with `inout` function arguments, the compiler enforces this: +it is an error to consume the value yielded from a `yielding mutate` accessor without reinitializing it before resuming the `yielding mutate` accessor. +When there are higher-level invariants which the value being modified must satisfy, in general, only the caller will be in a position to ensure that they are satisfied on the throwing path. + +Once that basis has been removed, there is no longer a reason to enable a coroutine to "unwind" when an error was thrown in the caller. +It should always finish execution the same way. + +### Naming scheme + +These coroutine accessors have been a long-standing "unofficial" feature of the Swift compiler under the names `_read` and `_modify`. +Previous revisions of this proposal merely dropped the underscore and proposed the coroutine accessors be provided under the name `read` and `modify`. +However, as we have continued to build out Swift's support for ownership and lifetime dependencies with `~Copyable` and `~Escapable` types, we have since identified the need for non-coroutine accessors that can produce borrowed and/or mutable values without imposing a lifetime dependency on a coroutine access. + +In order to avoid a proliferation of unrelated-seeming accessor names, this revision of the proposal uses the name `yielding borrow` instead of `read` and `yielding mutate` instead of `modify`. +We feel these names better connect the accessors to what ownership of the result is given: +a `borrow` accessor gives `borrowing` access to its result, and a `mutate` accessor gives `mutating` (in other words, `inout`) access. +Using `yielding` as an added modifier relates these accessors to potential non-coroutine variants of the accessors that could exist in the future; a `borrow` accessor (without `yielding`) would in the future be an accessor that returns a borrow without involving a coroutine. +The same `yielding` modifier could also be used in other places in the future, such as `func` declarations, to allow for yield-once coroutines to be defined as regular functions outside of accessors. + +## Acknowledgments + +John McCall and Arnold Schwaighofer provided much of the original implementation of accessor coroutines. Tim Kientzle and John McCall authored the accessors vision document this proposal serves as part of the implementation of. + diff --git a/proposals/0475-observed.md b/proposals/0475-observed.md new file mode 100644 index 0000000000..7690394895 --- /dev/null +++ b/proposals/0475-observed.md @@ -0,0 +1,592 @@ +# Transactional Observation of Values + +* Proposal: [SE-0475](0475-observed.md) +* Authors: [Philippe Hausler](https://github.com/phausler) +* Review Manager: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Status: **Accepted** +* Implementation: https://github.com/swiftlang/swift/pull/79817 +* Review: ([pitch](https://forums.swift.org/t/pitch-transactional-observation-of-values/78315)) ([review](https://forums.swift.org/t/se-0475-transactional-observation-of-values/79224)) ([acceptance](https://forums.swift.org/t/accepted-se-0475-transactional-observation-of-values/80389)) + +## Introduction + +Observation was introduced to add the ability to observe changes in graphs of +objects. The initial tools for observation afforded seamless integration into +SwiftUI, however aiding SwiftUI is not the only intent of the module - it is +more general than that. This proposal describes a new safe, ergonomic and +composable way to observe changes to models using an AsyncSequence, starting +transactions at the first willSet and then emitting a value upon that +transaction end at the first point of consistency by interoperating with +Swift Concurrency. + +## Motivation + +Observation was designed to allow future support for providing an `AsyncSequence` +of values, as described in the initial [Observability proposal](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0395-observability.md). +This follow-up proposal offers tools for enabling asynchronous sequences of +values, allowing non-SwiftUI systems to have the same level of "just-the-right-amount-of-magic" +as when using SwiftUI. + +Numerous frameworks in the Darwin SDKs provide APIs for accessing an +`AsyncSequence` of values emitted from changes to a property on a given model +type. For example, DockKit provides `trackingStates` and Group Activities +provides `localParticipantStates`. These are much like other APIs that provide +`AsyncSequence` from a model type; they hand crafted to provide events from when +that object changes. These manual implementations are not trivial and require +careful book-keeping to get right. In addition, library and application code +faces the same burden to use this pattern for observing changes. Each of these +uses would benefit from having a centralized and easy mechanism to implement +this kind of sequence. + +Observation was built to let developers avoid the complexity inherent when +making sure the UI is updated upon value changes. For developers using SwiftUI +and the `@Observable` macro to mark their types, this principle is already +realized; directly using values over time should mirror this ease of use, +providing the same level of power and flexibility. That model of tracking changes +by a graph allows for perhaps the most compelling part of Observation; it +can track changes by utilizing naturally written Swift code that is written just +like the logic of other plain functions. In practice that means that any solution +will also follow that same concept even for disjoint graphs that do not share +connections. The solution will allow for iterating changed values for applications +that do not use UI as seamlessly as those that do. + +## Proposed solution + +This proposal adds a straightforward new tool: a closure-initialized `Observations` +type that acts as a sequence of closure-returned values, emitting new values +when something within that closure changes. + +This new type makes it easy to write asynchronous sequences to track changes +but also ensures that access is safe with respect to concurrency. + +The simple `Person` type declared here will be used for examples in the +remainder of this proposal: + +```swift +@Observable +final class Person { + var firstName: String + var lastName: String + + var name: String { firstName + " " + lastName } + + init(firstName: String, lastName: String) { + self.firstName = firstName + self.lastName = lastName + } +} +``` + +Creating an `Observations` asynchronous sequence is straightforward. This example +creates an asynchronous sequence that yields a value every time the composed +`name` property is updated: + +```swift +let names = Observations { person.name } +``` + +However if the example was more complex and the `Person` type in the previous +example had a `var pet: Pet?` property which was also `@Observable` then the +closure can be written with a more complex expression. + +```swift +let greetings = Observations { + if let pet = person.pet { + return "Hello \(person.name) and \(pet.name)" + } else { + return "Hello \(person.name)" + } +} +``` + +In that example it would track both the assignment of a new pet and then consequently +that pet's name. + +## Detailed design + +There are a few behaviors that are prerequisites to understanding the requirements +of the actual design. These two key behaviors are how the model handles tearing +and how the model handles sharing. + +Tearing is where a value that is expected to be assigned as a singular +transactional operation can potentially be observed in an intermediate and +inconsistent state. The example `Person` type shows this when a `firstName` is +set and then the `lastName` is set. If the observation was triggered just on the +trailing edge (the `didSet` operation) then an assignment to both properties +would garner an event for both properties and potentially get an inconsistent +value emitted from `name`. Swift has a mechanism for expressing the grouping of +changes together: isolation. When an actor or an isolated type is modified it is +expected (enforced by the language itself) to be in a consistent state at the +next suspension point. This means that if we can utilize the isolation that is +safe for the type then the suspensions on that isolation should result in safe +(and non torn values). This means that the implementation must be transactional +upon that suspension; starting the transaction on the first trigger of a leading +edge (the `willSet`) and then completing the transaction on the next suspension +of that isolation. + +The simple example of tearing would work as the following: + +```swift +let person = Person(firstName: "", lastName: "") +// willSet \.firstName - start a transaction +person.firstName = "Jane" +// didSet \.firstName +// willSet \.lastName - the transaction is still dirty +person.lastName = "Appleseed" +// didSet \.lastName +// the next suspension the `name` property will be valid +``` + +Suspensions are any point where a task can be calling out to something where +they `await`. Swift concurrency enforces safety around these by making sure that +isolation is respected. Any time a function has a suspension point data +associated with the type must be ready to be read by the definitions of actor +isolation. In the previous example of the `Person` instance the `firstName` and +`lastName` properties are mutated together in the same isolation, that means +that no other access in that isolation can read those values when they are torn +without the type being `Sendable` (able to be read from multiple isolations). +That means that in the case of a non-`Sendable` type the access must be +constrained to an isolation, and in the `Sendable` cases the mutation is guarded +by some sort of mechanism like a lock, In either case it means that the next +time one can read a safe value is on that same isolation of the safe access to +start with and that happens on that isolations next suspension. + +Observing at the next suspension point means that we can also address the second +issue too; sharing. The expectation of observing a property from a type as an +AsyncSequence is that multiple iterations of the same sequence from multiple +tasks will emit the same values at the same iteration points. The following code +is expected to emit the same values in both tasks. + +```swift + +let names = Observations { person.firstName + " " + person.lastName } + +Task.detached { + for await name in names { + print("Task1: \(name)") + } +} + +Task.detached { + for await name in names { + print("Task2: \(name)") + } +} +``` + +In this case both tasks will get consistently safe accessed values. This can +be achieved without needing an extra buffer since the suspension of each side of +the iteration are continuations resuming all together upon the accessor's +execution on the specified isolation. This facilitates subject-like behavior +such that the values are sent from the isolation for access to the iteration's +continuation. + +The previous initialization using the closure is a sequence of values of the computed +properties as a `String`. This has no sense of termination locally within the +construction. Making the return value of that closure be a lifted `Optional` suffers +the potential conflation of a terminal value and a value that just happens to be nil. +This means that there is a need for a second construction mechanism that offers a +way of expressing that the `Observations` sequence iteration will run until finished. + +For the example if `Person` then has a new optional field of `homePage` which +is an optional URL it then means that the construction can disambiguate +by returning the iteration as the `next` value or the `finished` value. + +``` +@Observable +final class Person { + var firstName: String + var lastName: String + var homePage: URL? + + var name: String { firstName + " " + lastName } + + init(firstName: String, lastName: String) { + self.firstName = firstName + self.lastName = lastName + } +} + +let hosts = Observations.untilFinished { [weak person] in + if let person { + .next(person.homePage?.host) + } else { + .finished + } +} +``` + +Putting this together grants a signature as such: + +```swift +public struct Observations: AsyncSequence, Sendable { + public init( + @_inheritActorContext _ emit: @escaping @isolated(any) @Sendable () throws(Failure) -> Element + ) + + public enum Iteration: Sendable { + case next(Element) + case finished + } + + public static func untilFinished( + @_inheritActorContext _ emit: @escaping @isolated(any) @Sendable () throws(Failure) -> Iteration + ) -> Observations +} +``` + +Picking the initializer apart first captures the current isolation of the +creation of the `Observations` instance. Then it captures a `Sendable` closure that +inherits that current isolation. This means that the closure may only execute on +the captured isolation. That closure is run to determine which properties are +accessed by using Observation's `withObservationTracking`. So any access to a +tracked property of an `@Observable` type will compose for the determination of +which properties to track. + +The closure is not run immediately it is run asynchronously upon the first call +to the iterator's `next` method. This establishes the first tracking state for +Observation by invoking the closure inside a `withObservationTracking` on the +implicitly specified isolation. Then upon the first `willSet` it will enqueue on +to the isolation a new execution of the closure and finishing the transaction to +prime for the next call to the iterator's `next` method. + +The closure has two other features that are important for common usage; firstly +the closure is typed-throws such that any access to that emission closure will +potentially throw an error if the developer specifies. This allows for complex +composition of potentially failable systems. Any thrown error will mean that the +`Observations` sequence is complete and loops that are currently iterating will +terminate with that given failure. Subsequent calls then to `next` on those +iterators will return `nil` - indicating that the iteration is complete. + +The type `Observations` will conform to `AsyncSequence`. This means that it +adheres to the cancellation behavior of other `AsyncSequence` types; if the task +is cancelled then the iterator will return nil, and any time it becomes +terminal for any reason that sequence will remain terminal and continue returning nil. +Termination by cancellation however is independent for each instance. + +## Behavioral Notes + +There are a number of scenarios of iteration that can occur. These can range from production rate to iteration rate differentials to isolation differentials to concurrent iterations. Enumerating all possible combinations is of course not possible but the following explanations should illustrate some key usages. `Observations` does not make unsafe code somehow safe - the concepts of isolation protection or exclusive access are expected to be brought to the table by the types involved. It does however require the enforcements via Swift Concurrency particularly around the marking of the types and closures being required to be `Sendable`. The following examples will only illustrate well behaved types and avoid fully unsafe behavior that would lead to crashes because the types being used are circumventing that language safety. + +The most trivial case is where a single produce and single consumer are active. In this case they both are isolated to the same isolation domain. For ease of reading; this example is limited to the `@MainActor` but could just as accurately be represented in some other actor isolation. + +```swift +@MainActor +func iterate(_ names: Observations) async { + for await name in names { + print(name) + } +} + +@MainActor +func example() async throws { + let person = Person(firstName: "", lastName: "") + + // note #2 + let names = Observations { + person.name + } + + Task { + await iterate(names) + } + + for i in 0..<5 { + person.firstName = "\(i)" + person.lastName = "\(i)" + try await Task.sleep(for: .seconds(0.1)) // note #1 + } +} + +try await example() + +``` + +The result of the observation will print the following output. + +``` +0 0 +1 1 +2 2 +3 3 +4 4 +``` + +The values are by the virtue of the suspension at `note #1` are all emitted, the first name and last name are conjoined because they are both mutated before the suspension. The type `Person` does not need to be `Sendable` because `note #2` is implicitly picking up the `@MainActor` isolation of the enclosing isolation context. That isolation means that the person is always safe to access in that scope. + +Next is the case where the mutation of the properties out-paces the iteration. Again the example is isolated to the same domain. + +```swift +@MainActor +func iterate(_ names: Observations) async { + for await name in names { + print(name) + try? await Task.sleep(for: .seconds(0.095)) + } +} + +@MainActor +func example() async throws { + let person = Person(firstName: "", lastName: "") + + // @MainActor is captured here as the isolation + let names = Observations { + person.name + } + + Task { + await iterate(names) + } + + for i in 0..<5 { + person.firstName = "\(i)" + person.lastName = "\(i)" + try await Task.sleep(for: .seconds(0.1)) + } +} + +try await example() + +``` + +The result of the observation may print the following output, but the primary property is that the values are conjoined to the same consistent view. It is expected that some values may not be represented during the iteration because the transaction has not yet been handled by the iteration. + +``` +0 0 +1 1 +2 2 +3 3 +``` + +The last value is never observed because the program ends before it would be. If the program did not terminate then another value would be observed. + +Observations can be used across boundaries of concurrency. This is where the iteration is done on a different isolation than the mutations. The types however are accessed always in the isolation that the creation of the Observations closure is executed. This means that if the `Observations` instance is created on the main actor then the subsequent calls to the closure will be done on the main actor. + +```swift +@globalActor +actor ExcplicitlyAnotherActor: GlobalActor { + static let shared = ExcplicitlyAnotherActor() +} + +@ExcplicitlyAnotherActor +func iterate(_ names: Observations) async { + for await name in names { + print(name) + } +} + +@MainActor +func example() async throws { + let person = Person(firstName: "", lastName: "") + + // @MainActor is captured here as the isolation + let names = Observations { + person.name + } + + Task.detached { + await iterate(names) + } + + for i in 0..<5 { + person.firstName = "\(i)" + person.lastName = "\(i)" + try await Task.sleep(for: .seconds(0.1)) + } +} + +``` + +The values still will be conjoined as expected for their changes, however just like the out-paced case there is a potential in which an alteration may slip between the isolations and only a subsequent value is represented during the iteration. However since is particular example has no lengthy execution (greater than 0.1 seconds) it means that it does not get out paced by production and returns all values. + +``` +0 0 +1 1 +2 2 +3 3 +4 4 +``` + +If the `iterate` function was altered to have a similar `sleep` call that exceeded the production then it would result in similar behavior of the previous producer/consumer rate case. + +The next behavioral illustration is the value distribution behaviors; this is where two or more copies of an `Observations` are iterated concurrently. + +```swift + +@MainActor +func iterate1(_ names: Observations) async { + for await name in names { + print("A", name) + } +} + + +@MainActor +func iterate2(_ names: Observations) async { + for await name in names { + print("B", name) + } +} + +@MainActor +func example() async throws { + let person = Person(firstName: "", lastName: "") + + // @MainActor is captured here as the isolation + let names = Observations { + person.name + } + + Task.detached { + await iterate1(names) + } + + Task.detached { + await iterate2(names) + } + + for i in 0..<5 { + person.firstName = "\(i)" + person.lastName = "\(i)" + try await Task.sleep(for: .seconds(0.1)) + } +} + +try await example() +``` + +This situation commonly comes up when the asynchronous sequence is stored as a property of a type. By vending these as a shared instance to a singular source of truth it can provide both a consistent view and reduce overhead for design considerations. However when the sequences are then combined with other isolations the previous caveats come in to play. + +``` +A 0 0 +B 0 0 +B 1 1 +A 1 1 +A 2 2 +B 2 2 +A 3 3 +B 3 3 +B 4 4 +A 4 4 +``` + +The same rate commentary applies here as before but an additional wrinkle is that the delivery between the A and B sides is non-determinstic (in some cases it can deliver as A then B and other cases B then A). + +There is one additional clarification of expected behaviors - the iterators should have an initial state to determine if that specific iterator is active yet or not. This means that upon the first call to next the value will be obtained by calling into the isolation of the constructing closure to "prime the pump" for observation and obtain a first value. This can be encapsulated into an exaggerated test example as the following: + +```swift + +@MainActor +func example() async { + let person = Person(firstName: "0", lastName: "0") + + // @MainActor is captured here as the isolation + let names = Observations { + person.name + } + Task { + try await Task.sleep(for: .seconds(2)) + person.firstName = "1" + person.lastName = "1" + + } + Task { + for await name in names { + print("A = \(name)") + } + } + Task { + for await name in names { + print("B = \(name)") + } + } + try? await Task.sleep(for: .seconds(10)) +} + +await example() +``` + +Which results in the following output: + +``` +A = 0 0 +B = 0 0 +B = 1 1 +A = 1 1 +``` + +This ensures the first value is produced such that every sequence will always be primed with a value and will eventually come to a mutual consistency to the values no matter the isolation. + +## Effect on ABI stability & API resilience + +This provides no alteration to existing APIs and is purely additive. However it +does have a few points of interest about future source compatibility; namely +the initializer does ferry the inherited actor context as a parameter and if +in the future Swift develops a mechanism to infer this without a user +overridable parameter then there may be a source breaking ambiguity that would +need to be disambiguated. + +## Notes to API authors + +This proposal does not change the fact that the spectrum of APIs may range from +favoring `AsyncSequence` properties to purely `@Observable` models. They both +have their place. However the calculus of determining the best exposition may +be slightly more refined now with `Observations`. + +If a type is representative of a model and is either transactional in that +some properties may be linked in their meaning and would be a mistake to read +in a disjoint manner (the tearing example from previous sections), or if the +model interacts with UI systems it now more so than ever makes sense to use +`@Observable` especially with `Observations` now as an option. Some cases may have +previously favored exposing those `AsyncSequence` properties and would now +instead favor allowing the users of those APIs compose things by using `Observations`. +The other side of the spectrum will still exist but now is more strongly +relegated to types that have independent value streams that are more accurately +described as `AsyncSequence` types being exposed. The suggestion for API authors +is that now with `Observations` favoring `@Observable` perhaps should take more +of a consideration than it previously did. + +## Alternatives Considered + +Both initialization mechanisms could potentially be collapsed into an optional, +however that creates potential ambiguity of valid nil elements versus termination. + +There have been many iterations of this feature so far but these are some of the +highlights of alternative mechanisms that were considered. + +Just expose a closure with `didSet`: This misses the mark with regards to concurrency +safety but also faces a large problem with regards to transactionality. This would also +be out sync with the expected behavior of existing observation uses like SwiftUI. +The one benefit of that approach is that each setter call would have a corresponding +callback and would be more simple to implement with the existing infrastructure. It +was ultimately rejected because that would fall prey to the issue of tearing and +the general form of composition was not as ergonomic as other solutions. + +Expose an AsyncSequence based on `didSet`: This also falls to the same issues with the +closure approach except is perhaps slightly more ergonomic to compose. This was also +rejected due to the tearing problem stated in the proposal. + +Expose an AsyncSequence property extension based on `KeyPath`: This could be adapted +to the `willSet` and perhaps transactional models, but faces problems when attempting +to use `KeyPath` across concurrency domains (since by default they are not Sendable). +The implementation of that approach would require considerable improvement to handling +of `KeyPath` and concurrency (which may be an optimization path that could be considered +in the future if the API merits it). As it stands however the `KeyPath` approach in +comparison to the closure initializer is considerably less easy to compose. + +The closure type passed to the initializer does not absolutely require @Sendable in the +cases where the initialization occurs in an isolated context, if the initializer had a +parameter of an isolation that was non-nullable this could be achieved for that restriction +however up-coming changes to Swift's Concurrency will make this approach less appealing. +If this route would be taken it would restrict the potential advanced uses cases where +the construction would be in an explicitly non-isolated context. + +A name of `Observed` was considered, however that type name led to some objections that +rightfully claimed it was a bit odd as a name since it is bending the "nouning" of names +pretty strongly. This lead to the alternate name `Observations` which strongly leans +into the plurality of the name indicating that it is more than one observation - lending +to the sequence nature. + +It was seriously considered during the feedback to remove the initializer methods and only +have construction by two global functions named `observe` and `observeUntilFinished` +that would act as the current initializer methods. Since the types must still be returned +to allow for storing that return into a property it does not offer a distinct advantage. diff --git a/proposals/0476-abi-attr.md b/proposals/0476-abi-attr.md new file mode 100644 index 0000000000..22caeee1a2 --- /dev/null +++ b/proposals/0476-abi-attr.md @@ -0,0 +1,945 @@ +# Controlling the ABI of a function, initializer, property, or subscript + +* Proposal: [SE-0476](0476-abi-attr.md) +* Authors: [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 6.2)** +* Review: ([pitch](https://forums.swift.org/t/pitch-controlling-the-abi-of-a-declaration/75123)) ([review](https://forums.swift.org/t/se-0476-controlling-the-abi-of-a-function-initializer-property-or-subscript/79233)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0476-controlling-the-abi-of-a-function-initializer-property-or-subscript/79644)) + +## Introduction + +We propose introducing the `@abi` attribute, which provides an alternate +version of the declaration used for name mangling. This feature would allow +developers of ABI-stable libraries to make minor changes, such as changing +the sendability of a parameter or renaming a declaration (so long as source +compatibility is preserved in a backwards-deployable way), without requiring +deep knowledge of compiler implementation details. + +## Motivation + +Maintainers of ABI-stable libraries sometimes need to update or correct +existing declarations for various reasons: + +1. To adopt new language features, like changing `@Sendable` to `sending`, + in an existing declaration. + +2. To replace an existing declaration with a source-compatible but ABI-breaking + equivalent, like replacing a `rethrows` method with one using typed + `throws`. + +3. To correct a mistake, like removing an unnecessary `@escaping` attribute or + adding a `Sendable` generic constraint. + +4. To rename an API whose name is felt to be catastrophically confusing. + +Many revisions will cause fundamental changes in how an API will be used at the +machine code level that clients must account for; for instance, changing `` +to `` requires callers to generate code that will pass the witness +table for `T`'s `Hashable` conformance. However, some features are designed to +have little or no impact on the code generated by the caller. For example, +these two declarations: + +```swift +// `T` must be `Sendable` +func fn(_: T) {} + +// `T` parameter must be `sending` +func fn(_: borrowing sending T) {} // note: 'borrowing sending' is currently banned, + // pending a decision on whether it should have the + // meaning we want it to have here +``` + +Have identical parameter signatures at the IR level, with one pointer to the +argument and another pointer to `T`'s value witness table, and use the same +result type and calling convention too: + +```text +define hidden swiftcc void @"$s4main2fnyyxs8SendableRzlF"(ptr noalias %0, ptr %T) + ^~~~~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~ +define hidden swiftcc void @"$s4main2fnyyxlF"(ptr noalias %0, ptr %T) + ^~~~~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~~~~ +``` + +Other details, such as the parameter ownership conventions, also line up to +make this work; suffice it to say, the function generated when you use +`borrowing sending` is perfectly capable of handling the arguments passed by +callers that think the parameter is `Sendable`. The only differences between +them are the compile-time checks applied by the compiler and the part of their +mangled names that indicates the feature being used: + +```text +define hidden swiftcc void @"$s4main2fnyyxs8SendableRzlF"(ptr noalias %0, ptr %T) + ^~~~~~~~~~~~~ 'T: Sendable' +define hidden swiftcc void @"$s4main2fnyyxlF"(ptr noalias %0, ptr %T) + ^ 'T' ('sending' is not indicated by the mangled name) +``` + +Thus, if there was a way to tell the compiler to continue using the mangled +name for `fn(_: T)`, a library designer could actually change the +declaration to be treated like `fn(_: borrowing sending T)` when compiling +with the new version of the library *without* breaking ABI compatibility. + +This is part of how the `@preconcurrency` attribute works. `@preconcurrency` +has two effects: It instructs the type checker to permit Swift 5 code to use +the declaration in ways that would violate the rules of certain concurrency +annotations, and it causes those annotations to be omitted from the +declaration's mangled name. That makes it perfect for retrofitting sendability +checking onto APIs that were created before Swift Concurrency was introduced. +However, it is designed specifically for that exact task, which makes it +inflexible: It cannot be used to suppress some concurrency features but not +others (for instance, to amend a mistake in one parameter without affecting +other parts of the declaration), and it cannot be applied to adopt +non-concurrency features which have the same property of being ABI-compatible +except for a different mangled name. + +For everything else, there's the compiler-internal `@_silgen_name` attribute. +`@_silgen_name` is an internal hack that overrides name mangling at specific +points in the compiler, replacing the mangled name with an arbitrary string. +If you know the original mangled name, therefore, you can use this attribute +to keep that name stable even if the declaration has evolved enough that it +would normally use a different name. That makes `@_silgen_name` enormously +flexible—it can be used to handle an arbitrary set of changes, and the +standard library uses it extensively for this purpose. For example, when the +standard library introduced a new `Collection.map(_:)` that used typed +`throws` in [SE-0413][], it continued to support clients expecting the old +`rethrows`-based `map` by using a `@_silgen_name` hack and +`@usableFromInline internal`: + +```swift +extension Collection { + // New `map(_:)` using typed `throws`: + @inlinable + @backDeployed(...) // slight lie, but that's irrelevant here + public func map( + _ transform: (Element) throws(E) -> T + ) throws(E) -> [T] { + // ...actual implementation of `map` omitted... + } + + // Wrapper with the same ABI as the old `map(_:)` which used `rethrows`: + @_silgen_name("$sSlsE3mapySayqd__Gqd__7ElementQzKXEKlF") + // ^-- func map<$T>(_: (Self.Element) throws -> $T) rethrows -> Swift.Array<$T> + // in Swift.Collection extension from module Swift + @usableFromInline + func __rethrows_map( + _ transform: (Element) throws -> T + ) throws -> [T] { // 'throws' and 'rethrows' have the same ABI + try map(transform) // calls through to the new `map(_:)` + } +} +``` + +This creates a declaration which is written in Swift source code as +`__rethrows_map(_:)`, but which has the mangled name of a function named +`map(_:)`. When a module is compiled against this new version of the standard +library, calls to `map(_:)` will use the new method directly; if a module is +compiled against an older standard library, though, it will end up calling the +`__rethrows_map(_:)` compatibility wrapper instead. + +Although it is a powerful tool, `@_silgen_name` has its own set of serious +drawbacks: + +* It has absolutely no compile-time safety checking. +* It works only with functions and is incompatible with certain function + features like opaque return types and `@backDeployed`.[1] +* It requires deep knowledge of the name mangling and calling convention to use + correctly. + +In practice, you basically need to be a Swift compiler or runtime engineer to +use it correctly. For this reason `@_silgen_name` has never been proposed to +Swift Evolution or recommended for general use. + +Library maintainers need a tool that is much more flexible than +`@preconcurrency` but also much safer and more ergonomic than `@_silgen_name`. + +> *[1] This is because the name mangling has facilities to create multiple +> symbols that are all related to the same declaration, but `@_silgen_name` +> only provides an override for the name of the main symbol. Any declaration +> that requires more than one symbol—such as a type declaration, a function +> with an opaque return type, or a function with a back-deployment +> thunk—would have no way to generate a mangled name for these additional +> symbols.* + + [SE-0413]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0413-typed-throws.md#effect-on-abi-stability + +## Proposed solution + +We propose a new attribute, called `@abi`, which specifies an alternate +declaration that provides its ABI for name mangling purposes. This alternate +declaration is enclosed within the argument parentheses; it has no body or +initializer expression but is otherwise a syntactically complete declaration. + +For example, the `@_silgen_name`-using `__rethrows_map(_:)` method shown in the +Motivation section could be written much more clearly by using `@abi`: + +```swift +extension Collection { + // Wrapper with the same ABI as the old `map(_:)` which used `rethrows`: + @abi( + func map( + _ transform: (Element) throws -> T + ) rethrows -> [T] + ) + @usableFromInline + func __rethrows_map( + _ transform: (Element) throws -> T + ) throws -> [T] { // 'throws' and 'rethrows' have the same ABI + try map(transform) // calls through to the new `map(_:)` + } +} +``` + +Notice how the `@abi` attribute basically contains the original version of the +declaration. When Swift is performing name mangling, this declaration is what +it will use; for all other functions, it will use the outer `__rethrows_map` +declaration. In particular, the `map(_:)` call in the body doesn't get resolved +to the `map(_:)` function in the `@abi` attribute; it looks for other +implementations and eventually finds the new typed-throws `map(_:)`. + +What's more, the ABI declaration can be checked against the original one to +make sure they're compatible. For example, at the ABI level `throws` and +`rethrows` are interchangeable, but a non-`throws`/`rethrows` method handles +its return values differently from them. If the maintainer accidentally dropped +the `throws` effect while implementing this function, the compiler would +complain about the mismatch: + +```swift +extension Collection { + @abi( + func map( + _ transform: (Element) throws -> T + ) rethrows -> [T] // error: 'rethrows' doesn't match API + ) + @usableFromInline + func __rethrows_map( + _ transform: (Element) throws -> T + ) -> [T] { // Whoops, should be 'throws' or 'rethrows'! + try map(transform) + } +} +``` + +This checking also makes sure that the details specified in the `@abi` +attribute are actually relevant. For example, the `@abi` attribute +automatically inherits the access control, availability, and `@objc`-ness of +the API it's attached to, so these are omitted from the `@abi` attribute. +Default arguments, too, are left out because they're irrelevant to ABI. The +compiler will diagnose this unnecessary information and suggest removing it. + +All sorts of precision changes are possible. Here's another use for a +`@_silgen_name` hack in the standard library: The maintainers discovered a data +race safety bug in an API that had already shipped and needed to add an +`@Sendable` attribute to prevent it, but `@preconcurrency` alone would have +also suppressed the `@Sendable` attribute on the parameter that had been +correctly annotated. `@abi` makes it easy to fix this sort of problem: + +```swift +public struct AsyncStream { + // ...other declarations omitted... + + @abi( + init( + unfolding produce: @escaping /* not @Sendable */ () async -> Element?, + onCancel: (@Sendable () -> Void)? = nil + ) + ) + @preconcurrency + public init( + unfolding produce: @escaping @Sendable () async -> Element?, + onCancel: (@Sendable () -> Void)? = nil + ) { + // Implementation omitted + } +} +``` + +Because `@preconcurrency` is applied to the outer declaration, but not to the +one inside the `@abi` attribute, its typechecking effects will be applied +(improving source compatibility for code written before the second `@Sendable` +was added) but its name mangling effects will not (keeping the mangled name +stable to preserve ABI compatibility). + +This feature goes beyond what `@_silgen_name` could do, however. For example, +it can be applied to `var` and `let` declarations: + +```swift +@abi(var oldName: Int) +public var newName: Int +``` + +The mangled name of an accessor includes the mangled name of the variable or +subscript it belongs to; thanks to `@abi`, the accessors for this variable will +have `oldName` mangled into their names. + +### Supported changes (and unsupported uses of them) + +This feature can be used to override the mangling of a declaration's: + +* Name, argument labels, and (for unary operator functions) fixity (`prefix` + vs. `postfix`) + +* Preconcurrency status, actor isolation (where this does not affect calling + convention), and execution environment + +* Generic constraints to marker protocols (`BitwiseCopyable`, `Copyable`, + `Escapable`, `Sendable`) + +* Certain aspects of parameter and `self` behavior (variadic (vs. `Array`); + `@autoclosure`; `sending`; ownership specifiers as long as the behavior is + compatible) + +* Certain aspects of argument, result, and thrown types (marker protocols in + existentials; tuple element labels; `@escaping`, `@Sendable`, and `sending` + results on closures) + +Note that some of these changes relate to safety properties of your code, such +as data race safety and escapability. When you use `@abi` to maintain ABI +compatibility with older versions of your library while tightening safety +constraints for new clients, you must take special care to remember that +clients compiled without those changes may violate the new constraints. In +practice, this means that you should probably only use `@abi` to make +retroactive changes to safety constraints when you know that violating the +constraint was *always* unsafe and it simply wasn't enforced until now. + +For instance, the `AsyncStream.init(unfolding:onCancel:)` example above adds +`@Sendable` to a closure parameter that previously didn't have the attribute. +This is appropriate because the closure was *always* run concurrently; code +that passed a non-`@Sendable` closure was already buggy, so this change merely +made the bug easier to detect. It would have been inappropriate if the closure +was originally run synchronously and was changed to run concurrently, because +code that previously worked fine would now have new data races. + +(`@abi` can still be used to help implement behavior changes, but the pattern +is different: you make the original version `@usableFromInline internal` and +change its API name to something you won't use by accident, applying `@abi` to +keep its mangled name the same as always. Then you create a new declaration +with the old API name and the behavior changes, using `@backDeployed` to ensure +that new binaries can interoperate with old versions of your library. +`__rethrows_map(_:)` is a good example of this pattern.) + +In short: Much like when `@inlinable` is used, **it is the developer's +responsibility to ensure that the current behavior of the declaration is +compatible with clients built against older versions of it. The compiler +doesn't understand the history of your codebase and cannot detect some +mistakes.** + +## Detailed design + +### Grammar + +An `@abi` attribute's argument list must have exactly one argument, which in +this proposal must be one of the following productions: + +* *function-declaration* +* *initializer-declaration* +* *constant-declaration* +* *variable-declaration* +* *subscript-declaration* + +This argument must *not* include any of the following sub-productions: + +* *code-block* +* *getter-setter-block* +* *getter-setter-keyword-block* +* *willSet-didSet-block* +* *initializer* (initial value expression) + +To that end, we amend the following productions in the Swift grammar to make +code blocks optional: + +```diff + initializer-declaration → initializer-head generic-parameter-clause? + parameter-clause async? throws-clause? +- generic-where-clause? initializer-body ++ generic-where-clause? initializer-body? + + initializer-declaration → initializer-head generic-parameter-clause? + parameter-clause async? 'rethrows' +- generic-where-clause? initializer-body ++ generic-where-clause? initializer-body? + + subscript-declaration → subscript-head subscript-result generic-where-clause? +- code-block ++ code-block? +``` + +We don't need to worry about ambiguity in terminating these productions because +the block-less forms always occur in `@abi` attributes; its closing parenthesis serves as a terminator for the declaration. + +> **Note**: If future development of the `@abi` attribute requires additional +> information to be added to it, this can be done by adding new productions at +> the beginning of the argument list, terminated by a comma or colon to +> distinguish them from declaration modifiers: +> +> ```swift +> @abi(unchecked, func liveDangerously(_: AnyObject)) // Future direction +> func liveDangerously(_ object: AnyObject?) { ... } +> ``` + +### Terminology and basic concepts + +Syntactically, an `@abi` attribute involves two declarations. The *ABI-only +declaration* is the one in the attribute's argument list; the *API-only +declaration* is the one the attribute is attached to. + +```swift +@abi(func abiOnlyDeclaration()) +func apiOnlyDeclaration() {} +``` + +A declaration which does not involve an `@abi` attribute at all—that is, which +is neither API-only nor ABI-only—is called a *normal declaration*. + +There are two *ABI roles*: + +* An *API-providing declaration* determines the behavior of the declaration in + source code: what name developers write to address it, what constraints and + behaviors are applied at use sites, how it is implemented (its body, + accessors, or members), etc. + +* An *ABI-providing declaration* determines how the declaration affects mangled + symbol names--both its own name and any names derived from it. + +Every declaration has at least one of these roles. Every declaration also has a +*counterpart* which fulfills the roles it does not. When the compiler wants to +compute some aspect of a declaration pertaining to a role that declaration does +not have, it automatically substitutes the declaration's counterpart. + +Roles and counterparts work as follows: + +| Declaration is… | ABI-providing | API-providing | Counterpart | +| --------------- | ------------- | ------------- | ------------------------------------------- | +| Normal | ✅ | ✅ | Is its own counterpart | +| ABI-only | ✅ | | Declaration `@abi` attribute is attached to | +| API-only | | ✅ | Declaration in `@abi` attribute | + +### Declaration checking + +When you use the `@abi` attribute, Swift validates various aspects of the +ABI-providing declaration in light of its API counterpart. An *aspect* is any +way in which the external appearance of a declaration might vary. Attributes +and modifiers are aspects, but so are the declaration's name, its result or +value types, its generic signature (if it has one), its parameter list (if it +has one), its effects (if it has any), and so on. + +#### Aspects with no ABI impact must be omitted + +Many aspects of a declaration only matter for an API-providing declaration; +they're irrelevant on a declaration that's ABI-only. These include: + +* Default arguments for parameters + +* Attributes which only affect compile-time checking or behavior, such as + `@unsafe`, `@discardableResult`, or result builder attributes + +* Certain attributes and modifiers which have ABI effects, but where the + compiler has been designed to inherit the ABI-providing declaration's + behavior from its API counterpart where needed: + + * `@objc` (and its ilk) and `dynamic`, including inference behaviors + * Access control modifiers and `@usableFromInline` + * `@inlinable` and other attributes controlling inlining + * `@available` and `@backDeployed` + * `override` + +These aspects are generally *forbidden* on an ABI-providing declaration. If +they are present, the compiler will diagnose an error and suggest they be +removed. + +In practice, this means that an `@abi` attribute is often significantly shorter +than the declaration it's attached to because it doesn't need to specify as +much information: + +```swift +@abi( + // Same signature as below, except `T` is not `Sendable`. + static + func assumeIsolated( + _ operation: @MainActor () throws -> T, + file: StaticString, + line: UInt + ) rethrows -> T +) +@available(macOS 14.0, iOS 17.0, watchOS 10.0, tvOS 17.0, *) // not needed in @abi +@usableFromInline // not needed in @abi +internal // not needed in @abi +static +func assumeIsolated( + _ operation: @MainActor () throws -> T, + file: StaticString = #fileID, // default argument not needed in @abi + line: UInt = #line // default argument not needed in @abi +) rethrows -> T { + ... +} +``` + +The intended workflow is that a developer can paste the entire original +declaration into an `@abi` attribute and the compiler will then tell them which +parts of it they should remove. + +#### Call compatibility + +For aspects which *do* have ABI impact, the compiler enforces that the +ABI-providing declaration is *call-compatible* with its API-providing +counterpart. Broadly, "call compatibility" means that, other than the mangled +names, the machine code generated to call the ABI-providing declaration would +be equally able to call its API counterpart. For instance: + +* Declarations are of the same fundamental kind (a `func` for a `func`, a `var` + for a `var`, etc.), so they expose the same basic capabilities and entry + points. + +* Effects match closely enough that the caller and callee will agree on which + stack should be used and which implicit parameters will be passed. + +* Inputs and outputs are passed similarly enough to ensure type and memory + safety, including compatible memory management behavior, and including the + implicit inputs and outputs used for generic parameters, `self`, and + throwing. + +However, call compatibility does *not* require aspects of the declaration +that only change the mangled name and/or compile-time checking to match: + +* Names, argument labels, or other name-like traits of a declaration (such + as the fixity modifiers for operator functions) may vary. + +* Aspects which affect only syntax (and possibly mangling) may vary. For + instance, a regular closure may be used instead of an `@autoclosure`; tuple + types with different element labels may be used; an array parameter may be + used instead of a variadic parameter; ordinary optionals may be used instead + of implicitly-unwrapped optionals; `throws` and `rethrows` may be used + interchangeably. + +* Concurrency safety and lifetime restrictions which don't affect the ability + to call the declaration may vary. For instance, a non-escaping closure may be + used instead of an `@escaping` closure; `sending` modifiers, `Sendable` + constraints, or neither may be used interchangeably so long as memory + management isn't affected; isolation may vary so long as extra data does not + need to be passed; `~Copyable` and `~Escapable` constraints may vary. + +#### Impact on redeclaration checking + +A declaration must have a unique signature in each of its roles. That is, an +API-only declaration is checked against API-providing declarations; an +ABI-only declaration is checked against ABI-providing declarations; a normal +declaration is checked twice, first against API-providing declarations and then +against ABI-providing declarations. + +In general, name lookup will return declarations with the API-providing role +and will ignore declarations with the ABI-providing role. Even when you're +writing an ABI-only declaration, you should use the API names of other +declarations, not the ABI names. + +#### Declaring multiple variables + +When `var` or `let` is used, the ABI-providing declaration must bind the same +number of patterns, each of which has the same number of variables, as its API +counterpart. That is, the first of these is valid, while the others are not: + +```swift +// OK: +@abi(var x, y: Int) +var a, b: Int + +// Mismatched: +@abi(var x, y, z: Int) +var a, b: Int + +// Also mismatched: +@abi(var x, y: Int) +var a: Int, (b1, b2): (Int, Int) + +// Mismatched even though the total adds up: +@abi(var x, y, z: Int) +var a: Int, (b1, b2): (Int, Int) +``` + +An ABI-providing declaration does *not* infer missing types from its API +counterpart. In practice, this means that an ABI-providing declaration +may need to explicitly declare types that its API counterpart infers from an +initial value expression. + +An ABI-providing `var` or `let` does not have a list of accessors or specify +anything about them; in a sense, it can be thought of as inferring its +accessors from its API counterpart. + +### Limitations on feature scope + +#### Supported declaration kinds + +In this proposal, `@abi` may be applied only to `func`, `init`, `var`, `let`, +and `subscript` declarations. Other declarations are less straightforward to +support in various ways; see the future directions section for details. + +#### Language features with auxiliary declarations + +`@abi` can neither contain, nor be applied alongside, `lazy` or a property +wrapper. These features implicitly create auxiliary declarations, and it isn't +clear how those should interact with `@abi`. + +#### Limited support for macros + +Neither attached nor freestanding macros can be used inside an `@abi` +attribute. None of the attached macro roles would be useful since ABI-providing +declarations do not have bodies, members, accessors, or extensions; the +freestanding macro roles, on the other hand, expand to complete declarations, +while some of the future directions involve supporting special stub syntax +which would be incompatible here. + +`@abi` can still be applied *alongside* an attached macro or *to* a +freestanding macro, although in practice many macros will need to handle `@abi` +attributes specially. + +### Non-normative: Precise rules as currently implemented + +To help evaluate how these principles will work in practice, we've listed the +current implementation's rules below. **However, we do not guarantee that the +rules listed here will exactly match the final behavior of the feature.** +Basically, we don't want to put every bug fix through an amendment or every +tiny, straightforward expansion of capabilities through a proposal. + +#### Must be omitted (no ABI impact or inheritance in place) + +* Default arguments on parameters +* Result builder attributes on parameters or declarations +* `@available` +* `@inlinable`, `@inline`, `@backDeployed`, `@usableFromInline`, + `@_alwaysEmitIntoClient`, `@_transparent` +* Objective-C opt-in attributes (`@objc`, `@IBAction`, `@IBDesignable`, + `@IBInspectable`, `@IBOutlet`, `@IBSegueAction`, `@GKInspectable`, + `@NSManaged`, `@nonobjc`) +* `optional` modifier in `@objc` protocols +* `@NSCopying` +* `@_expose` and `@_cdecl` +* `@LLDBDebuggerFunction` +* `dynamic` modifier and `@_dynamicReplacement` +* `@specialize` on functions and initializers +* `override` modifier +* Access control (`open`, `public`, `package`, `internal`, `fileprivate`, + `private`) +* Setter access control (`open(set)`, `public(set)`, `package(set)`, + `internal(set)`, `fileprivate(set)`, `private(set)`) +* `@_spi` and `@_spi_available` +* Reference ownership (`weak`, `unowned`, `unowned(unsafe)`) +* `@warn_unqualified_access` +* `@discardableResult` +* `@implementation` on functions +* `@differentiable`, `@derivative`, `@transpose` +* `@noDerivative` on declarations other than parameters +* `@exclusivity` +* `@safe` and `@unsafe` +* `@abi` +* Unsupported features (`lazy`, property wrapper attributes, attached macro + attributes) + +#### Must be specified and must match + +* Declaration kind (`func`, `var`, etc.) +* `convenience` and `required` modifiers on initializers +* `distributed` modifier +* Result type of functions +* Failability of initializers +* Value type of subscripts and variables +* Number of parameters on functions, initializers, and subscripts +* Parameter types +* `inout` on parameters and `mutating`/`nonmutating` modifiers on members +* `@noDerivative` on parameters +* `@_addressable` on parameters and `@_addressableSelf` on members +* `@lifetime` attributes (NOTE: this probably ought to be "vary with + constraints", but an interaction with `@_addressableForDependencies` needs + to be worked out) +* `async` effect +* Aspects of types which are not listed elsewhere + +#### Allowed to vary, but with constraints + +* `throws` effect and thrown type (`rethrows` is equivalent to `throws`) +* Generic signature of functions, initializers, and subscripts (marker + protocols may vary) +* Variadic parameter types (`T...` and `Array` are treated as equivalent) +* Parameter ownership specifiers and `self` ownership modifiers (ones with + equivalent memory management behavior may be substituted for one another) +* `sending` on parameter and result types (so long as its ownership behavior + is preserved) +* `static`, `class`, and `final` modifiers (`class final` is equivalent to + `static`) +* Actor isolation (other than `@isolated(any)`, which is incompatible with + the others) + +#### Allowed to vary arbitrarily + +* Base names of functions and variables +* Argument labels and parameter names +* `prefix` and `postfix` modifiers on operator functions +* Whether optionals are implicitly unwrapped +* Element labels in tuple types +* `@autoclosure` on parameters +* `@escaping` on closures +* `@Sendable` on closures +* `isolated` on parameters +* `_const` on parameters and variables +* Generic parameter names +* Use of type sugar (e.g. `Optional` and `T?` are equivalent) +* Use of generic types that have a same-type constraint (e.g. in the + presence of `T == Int`, `T` and `Int` are equivalent) +* Use of marker protocols in existential types +* `@Sendable` on functions and initializers +* `@preconcurrency` +* `@execution` +* Aspects of declarations which are not listed elsewhere + +## Source compatibility + +This feature is additive and affects only the ABI. However, many of the changes +that can be effected using it can be source-breaking unless done with care. For +example: + +* When renaming a declaration, make sure there's a declaration with the + original name that can be called in the same situations, and consider using + `@backDeployed` to ensure recompiled clients don't have to raise their + minimum deployment target. + +* When a type changes, make sure that it will either become more broad, or that + you are willing to accept any breakage that results. For example, switching + from a `Sendable` constraint to a `(borrowing) sending` parameter strictly + increases the set of valid callers, so that's probably always okay; switching + from no constraint to a `Sendable` constraint, on the other hand, will break + some callers, but might be acceptable if the missing `Sendable` constraint + created an opportunity for data races. + +## ABI compatibility + +This feature is intended to give libraries additional options to evolve APIs +without breaking the corresponding ABIs. + +We are currently evaluating adoption in `stdlib/public`. So far, it looks like +we can replace all uses of `@_silgen_name` to specify mangled Swift names with +uses of `@abi`, and in some cases remove hacks; this will be roughly 75 +declarations. + +Note that this feature does not subsume the use of `@_silgen_name` with an +arbitrary, C-style symbol name to either declare a function implemented in +C++ using the Swift calling convention, or to generate a symbol that's easy to +access from C-family code or a compiler intrinsic. About 200 of the uses of +`@_silgen_name` in `stdlib/public` are of this type; we expect these to remain +as-is. + +## Implications on adoption + +This feature is intended to help ease the adoption of other new features by +allowing a declaration's ABI to be "pinned" to its original form even as it +continues to evolve. Note that there is only ever a need to specify the +*original* form of the declaration, not any revisions that may have occurred +between then and the current form; there is therefore never a reason you would +need to specify more than one `@abi` attribute, nor to tie an `@abi` attribute +to a specific platform version. + +In module interfaces, the `@abi` attribute is partially suppressible. +Specifically, for `func`s that do not use `@backDeployed` and do not have +opaque result types, the compiler emits a module interface that falls back to +using an equivalent `@_silgen_name` attribute. For other declarations, however, +the compiler falls back to an `@available(*, unavailable)` attribute instead, +with a message indicating that the developer will need a newer compiler to use +the declaration. + +## Future directions + +### Unchecked mode + +There may be situations where a skilled engineer knows that a specific use of +`@abi` is compatible, but the compiler does not know how to prove that. While +that can often be considered a compiler bug—the checker *should* be able to +tell that the code is safe—it may be useful, either as a workaround or to +handle extreme edge cases, to be able to turn off `@abi`'s compatibility +checking: + +```swift +@abi(unchecked, func liveDangerously(_: AnyObject)) +func liveDangerously(_ object: AnyObject?) { ... } +``` + +### Support for types and extensions + +It ought to be possible to use `@abi` with types: + +```swift +@abi(struct Buffer: ~Copyable) +public struct FrameBuffer: ~Copyable { ... } + +@available(*, unavailable, renamed: "FrameBuffer") +@abi(typealias FrameBuffer) // keeps `typealias Buffer` from colliding with `struct Buffer` +typealias Buffer = FrameBuffer +``` + +Here, the library maintainer discovered after shipping that the name `Buffer` +is too vague—clients didn't understand what it meant, and some of them even +had another type named `Buffer`. When they rebuild with the new version of the +library, they will get an error with a fix-it to change `Buffer` to +`FrameBuffer`, but it will still use the name `Buffer` at the ABI level so +that existing binaries don't break. This will apply not only to the type +itself, but also to its members and even to functions with a `FrameBuffer` in +their overload signature. + +Type renaming may create challenges for module interface source stability, +since a module interfaces could refer to a type by an older or newer name than +its current one. It might be possible to address this by making module +interfaces always refer to types by their ABI name. (This should be +non-breaking as long as it's introduced at the same time as `@abi` for types.) + +This could also be used to affect the inference of properties of other +declarations. Consider this example: + +```swift +@abi(protocol Component) +@preconcurrency @MainActor // Added after shipping +public protocol Component { ... } + +extension Component { + public func onEvent(_ handler: @Sendable @escaping () -> Void) -> some Component { ... } +} +``` + +The library maintainer decided after the fact that `Component`s should be +isolated to the main actor, but that broke some Swift 5 code, so they added +`@preconcurrency` for its typechecking effects. However, `@preconcurrency` then +got applied to `onEvent(_:)`, suppressing its `@Sendable` attribute, which +changed its ABI. Using `@abi` on `Component` should override the ABI effects of +`@preconcurrency` not just for `Component` itself, but for every declaration +nested inside it. + +To support this kind of inference, the compiler may need to add an inferred +`@abi` attribute when a declaration with both roles depends on one which +provides only one role. For instance, if a type conforms to a protocol whose +`@abi` attribute specifies different actor isolation or different marker +protocols, Swift may need to add an inferred `@abi` attribute so the type's ABI +will be compatible with the protocol's ABI while its API will be compatible +with the protocol's ABI. + +### Support for enum cases + +It would probably be possible to allow `@abi` to be attached to a `case` +declaration, allowing it to backwards-compatibly rename or otherwise control +the ABI of enum cases. + +### Support for auxiliary declarations + +It might be possible to allow `@abi` to be used with `lazy` and property +wrappers either by coming up with rules to derive an `@abi` attribute for those +declarations, or by creating a syntax that can specify them: + +```swift +@abi(nonisolated var currentValue: Int) +@abi(for: projection, nonisolated var $currentValue: Binding) +@abi(for: storage, nonisolated var _currentValue: Binding) +@MainActor @Binding var currentValue: Int +``` + +### Support for accessors + +It might be possible to allow `@abi` to be attached to individual accessors. + +### Support for context changes + +It might be possible to allow an ABI-providing declaration to belong to a +different context than its counterpart—for instance, turning a global variable +into a static property, or moving a method to a single-property `@frozen` +wrapper struct. + +A particularly interesting one might be allowing an extension member to +be mangled as a member of the main type declaration, or vice versa, since users +may not be aware of the ABI impact of moving a declaration from one to the +other. + +### Equivalent type attribute + +Many uses of `@abi` only change one or two types in a complicated declaration. +It might be possible to provide an `@abi` *type* attribute that can be applied +on the spot as a shorthand: + +```swift +public func runConcurrently( + _ body: @escaping @abi(() -> Void) @Sendable () -> Void +) { ... } + +// Equivalent to: +@abi(func runConcurrently(_: @escaping () -> Void)) +public func runConcurrently( + _ body: @escaping @Sendable () -> Void +) { ... } +``` + +### Support for moving declarations to a different module + +It might be possible to roll some of the functionality of the compiler-internal +`@_originallyDefinedIn` attribute into this attribute. + +## Alternatives considered + +### Many narrow features + +The original motivation for this proposal involved fairly narrow cases where +`@preconcurrency` was too blunt an instrument, such as suppressing the ABI +impact of one sendability annotation while leaving others intact. One could +imagine designing individual type or decl attributes for each specific change +one might wish to make, but this would require both a lot of effort from +compiler engineers to design a specific tool for each problem, and a lot of +effort from library maintainers to figure out which tool to apply to a given +task and how to use it. + +### An argument that describes the differences + +Rather than creating many totally separate features, one could imagine an +`@abi` declaration attribute with an argument list which somehow described the +differences between the API and ABI. However, we see use cases for changing +virtually every aspect of an API—its name; adding or removing declaration +attributes, modifiers, and effects; adding or removing inherited protocols and +generic constraints; changing parameter, result, and error types; changing type +attributes and modifiers; even changing individual sub-types within generic +types, function types, and protocol compositions at any position in the +declaration—and a mini-language to address and edit all of these different +aspects of a declaration seems difficult to design and tedious to learn. +By contrast, re-specifying the entire declaration in the argument reuses the +developer's existing knowledge of how to read and write declarations and gives +them an easy way to adopt it (just copy the existing declaration into an `@abi` +attribute before you start editing in new features). + +### Syntax where the two declarations are peers + +We could design this differently such that the API-only and ABI-only +declarations are peers in the same context: + +```swift +// This declaration provides the ABI... +@abi(for: __rethrows_map(_:)) @usableFromInline func map( + _ transform: (Element) throws -> T +) rethrows -> [T] + +// ...for this declaration +@usableFromInline func __rethrows_map( + _ transform: (Element) throws -> T +) throws -> [T] { + try map(transform) +} +``` + +In theory, this design could simplify parsing, since the `@abi` attribute's +argument might just be an ordinary expression. However, it introduces several +complications: + +1. Merely giving the name of a declaration may not be specific enough in the + presence of overloads, or even when the API and ABI have the same name but + slight type differences. We might normally tell developers to work around + this by using more specific names, but that's not really an appropriate + answer for a tool which is designed to allow fine control of API and ABI + naming. +2. A lot of compiler logic would have to be modified to filter out ABI-only or + API-only declarations when it walked through lists of top-level decls or + members. The current design, where the ABI-only declarations are tucked away + in attributes, keeps them from being accessed by accident. +3. If the future direction for `@abi` on type declarations is taken, the + productions for full type declarations will not be suitable, as they require + member blocks. + +## Acknowledgments + +Thanks to Holly Borla for recognizing the need for an `@abi` attribute. diff --git a/proposals/0477-default-interpolation-values.md b/proposals/0477-default-interpolation-values.md new file mode 100644 index 0000000000..fc024885c9 --- /dev/null +++ b/proposals/0477-default-interpolation-values.md @@ -0,0 +1,171 @@ +# Default Value in String Interpolations + +* Proposal: [SE-0477](0477-default-interpolation-values.md) +* Authors: [Nate Cook](https://github.com/natecook1000) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift#80547](https://github.com/swiftlang/swift/pull/80547) +* Review: ([pitch](https://forums.swift.org/t/pitch-default-values-for-string-interpolations/69381)) ([review](https://forums.swift.org/t/se-0477-default-value-in-string-interpolations/79302)) ([acceptance](https://forums.swift.org/t/accepted-with-modification-se-0477-default-value-in-string-interpolations/79609)) + +## Introduction + +A new string interpolation syntax for providing a default string +when interpolating an optional value. + +## Motivation + +String interpolations are a streamlined and powerful way to include values within a string literal. +When one of those values is optional, however, +interpolating is not so simple; +in many cases, a developer must fall back to unpalatable code +or output that exposes type information. + +For example, +placing an optional string in an interpolation +yields an important warning and two suggested fixes, +only one of which is ideal: + +```swift +let name: String? = nil +print("Hello, \(name)!") +// warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit? +// print("Hello, \(name)!") +// ^~~~ +// note: use 'String(describing:)' to silence this warning +// print("Hello, \(name)!") +// ^~~~ +// String(describing: ) +// note: provide a default value to avoid this warning +// print("Hello, \(name)!") +// ^~~~ +// ?? <#default value#> + +``` + +The first suggestion, adding `String(describing:)`, +silences the warning but includes `nil` in the output of the string — +maybe okay for a quick shell script, +but not really appropriate result for anything user-facing. + +The second suggestion is good, +allowing us to provide whatever default string we'd like: + +```swift +let name: String? = nil +print("Hello, \(name ?? "new friend")!") +``` + +However, the nil-coalescing operator (`??`) +only works with values of the same type as the optional value, +making it awkward or impossible to use when providing a default for non-string types. +In this example, the `age` value is an optional `Int`, +and there isn't a suitable integer to use when it's `nil`: + +```swift +let age: Int? = nil +print("Your age: \(age)") +// warning, etc.... +``` + +To provide a default string when `age` is missing, +we have to write some gnarly code, +or split out the missing case altogether: + +```swift +let age: Int? = nil +// Optional.map +print("Your age: \(age.map { "\($0)" } ?? "missing")") +// Ternary expression +print("Your age: \(age != nil ? "\(age!)" : "missing")") +// if-let statement +if let age { + print("Your age: \(age)") +} else { + print("Your age: missing") +} +``` + +## Proposed solution + +The standard library should add a string interpolation overload +that lets you write the intended default as a string, +no matter what the type of value: + +```swift +let age: Int? = nil +print("Your age: \(age, default: "missing")") +// Prints "Your age: missing" +``` + +This addition will improve the clarity of code that uses string interpolations +and encourage developers to provide sensible defaults +instead of letting `nil` leak into string output. + +## Detailed design + +The implementation of this new interpolation overload looks like this, +added as an extension to the `DefaultStringInterpolation` type: + +```swift +extension DefaultStringInterpolation { + mutating func appendInterpolation( + _ value: T?, + default: @autoclosure () -> String + ) { + if let value { + self.appendInterpolation(value) + } else { + self.appendInterpolation(`default`()) + } + } +} +``` + +The new interpolation's `default:` parameter name +matches the one in the `Dictionary` subscript that has a similar purpose. + +You can try this out yourself by copy/pasting the snippet above into a project or playground, +or by experimenting with [this Swift Fiddle](https://swiftfiddle.com/nxttprythnfbvlm4hwjyt2jbjm). + +## Source compatibility + +This proposal adds one new API to the standard library, +which should not be source-breaking for any existing projects. +If a project or a dependency has added a similar overload, +it will take precedence over the new standard library API. + +## ABI compatibility + +This proposal is purely an extension of the ABI of the +standard library and does not change any existing features. + +## Implications on adoption + +The new API will be included in a new version of the Swift runtime, +and is marked as backward deployable. + +## Future directions + +There are [some cases][reflecting] where a `String(reflecting:)` conversion +is more appropriate than the `String(describing:)` normally used via string interpolation. +Additional string interpolation overloads could make it easier to use that alternative conversion, +and to provide a default when working with optional values. + +[reflecting]: https://forums.swift.org/t/pitch-default-values-for-string-interpolations/69381/58 + +## Alternatives considered + +**An interpolation like `"\(describing: value)"`** +This alternative would provide a shorthand for the first suggested fix, +using `String(describing:)`. +Unlike the solution proposed, +this kind of interpolation doesn't make it clear that you're working with an optional value, +so you could end up silently including `nil` in output without expecting it +(which is the original reason for the compiler warnings). + +**Extend `StringInterpolationProtocol` instead** +The proposed new interpolation works with _any_ optional value, +but some types only accept a limited or constrained set of types interpolations. +If the new `\(_, default:)` interpolation proves to be a useful pattern, +other types can add it as appropriate. + diff --git a/proposals/0478-default-isolation-typealias.md b/proposals/0478-default-isolation-typealias.md new file mode 100644 index 0000000000..590ea5fd7b --- /dev/null +++ b/proposals/0478-default-isolation-typealias.md @@ -0,0 +1,115 @@ +# Default actor isolation typealias + +* Proposal: [SE-0478](0478-default-isolation-typealias.md) +* Authors: [Holly Borla](https://github.com/hborla) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Active Review (April 21 ... May 5, 2025)** +* Vision: [Improving the approachability of data-race safety](/visions/approachable-concurrency.md) +* Implementation: [swiftlang/swift#80572](https://github.com/swiftlang/swift/pull/80572) +* Experimental Feature Flag: `DefaultIsolationTypealias` +* Previous Proposal: [SE-0466: Control default actor isolation inference][SE-0466] +* Review: ([pitch](https://forums.swift.org/t/pitch-a-typealias-for-per-file-default-actor-isolation/79150))([review](https://forums.swift.org/t/se-0478-default-actor-isolation-typealias/79436)) + +## Introduction + +[SE-0466: Control default actor isolation inference][SE-0466] introduced the ability to specify default actor isolation on a per-module basis. This proposal introduces a new typealias for specifying default actor isolation in individual source files within a module. This allows specific files to opt out of main actor isolation within a main-actor-by-default module, and opt into main actor isolation within a nonisolated-by-default module. + +## Motivation + +SE-0466 allows code to opt in to being “single-threaded” by default by isolating everything in the module to the main actor. When the programmer really wants concurrency, they can request it explicitly by marking a function or type as `nonisolated`, or they can define it in a module that does not default to main-actor isolation. However, it's very common to group multiple declarations used in concurrent code into one source file or a small set of source files. Instead of choosing between writing `nonisolated` on each individual declaration or splitting those files into a separate module, it's desirable to state that all declarations in those files default to `nonisolated`. + +## Proposed solution + +This proposal allows writing a private typealias named `DefaultIsolation` to specify the default actor isolation for a file. + +An underlying type of `MainActor` specifies that all declarations in the file default to main actor isolated: + +```swift +// main.swift + +private typealias DefaultIsolation = MainActor + +// Implicitly '@MainActor' +var global = 0 + +// Implicitly '@MainActor' +func main() { ... } + +main() +``` + +An underlying type of `nonisolated` specifies that all declarations in the file default to `nonisolated`: + +```swift +// Point.swift + +private typealias DefaultIsolation = nonisolated + +// Implicitly 'nonisolated' +struct Point { + var x: Int + var y: Int +} +``` + +## Detailed design + + A typealias named `DefaultIsolation` can specify the actor isolation to use for the source file it's written in under the following conditions: + +* The typealias is written at the top-level. +* The typealias is `private` or `fileprivate`; the `DefaultIsolation` typealias cannot be used to set the default isolation for the entire module, so its access level cannot be `internal` or above. +* The underlying type is either `MainActor` or `nonisolated`. + + It is not invalid to write a typealias called `DefaultIsolation` that does not meet the above conditions. Any typealias named `DefaultIsolation` that does not meet the above conditions will be skipped when looking up the default isolation for the source file. The compiler will emit a warning for any `DefaultIsolation` typealias that is not considered for default actor isolation along with the reason why: + +```swift +@globalActor +actor CustomGlobalActor { + static let shared = CustomGlobalActor() +} + +private typealias DefaultIsolation = CustomGlobalActor // warning: not used for default actor isolation +``` + +To allow writing `nonisolated` as the underlying type of a typealias, this proposal adds a typealias named `nonisolated` to the Concurrency library: + +```swift +public typealias nonisolated = Never +``` + +This typealias serves no purpose beyond specifying default actor isolation. To specify `nonisolated` using the `DefaultIsolation` typealias, the underlying type must be `nonisolated` exactly; it is invalid to write `private typealias DefaultIsolation = Never`. + +## Source compatibility + +Technically source breaking if someone happens to have written a private `DefaultIsolation` typealias with an underlying type of `MainActor`, which will start to infer every declaration in that file as `@MainActor`-isolated after this change. This seems extremely unlikely. + +## ABI compatibility + +This proposal has no ABI impact on existing code. + +## Implications on adoption + +This proposal does not change the adoption implications of adding `@MainActor` to a declaration that was previously nonisolated and vice versa. The source and ABI compatibility implications of changing actor isolation are documented in the Swift migration guide's [Library Evolution](https://github.com/apple/swift-migration-guide/blob/29d6e889e3bd43c42fe38a5c3f612141c7cefdf7/Guide.docc/LibraryEvolution.md#main-actor-annotations) article. + +## Alternatives considered + +Adding a typealias named `nonisolated` to `Never` to the Concurrency library to enable writing it as the underlying type of a typealias is pretty strange; this approach leverages the fact that `nonisolated` is a contextual keyword, so it's valid to use `nonisolated` as an identifier. This proposal uses a typealias instead of an empty struct or enum type to avoid the complications of having a new type be only available with the Swift 6.2 standard library. + +It's extremely valuable to have a consistent way to spell `nonisolated`. Introducing a type that follows standard naming conventions, such as `Nonisolated`, or using an existing type like `Never` is more consistent with recommended style, but overall complicates the concurrency model because it means you need to spell `nonisolated` differently when specifying it per file versus writing it on a declaration. And because the underlying type of this typealias is used to infer actor isolation, it's not used as a type in the same way that other typealiases are. + +Another alternative is to introduce a bespoke syntax such as `using MainActor` or `using nonisolated`. This approach preserves a consistent spelling for `nonisolated`, but at the cost of adding new language syntax that deviates from other defaulting rules such as the default literal types and the default actor system types. + +Having a `nonisolated` typealias may also allow us to improve the package manifest APIs for specifying default isolation, allowing us to move away from using `nil` to specify `nonisolated`: + +```swift +SwiftSetting.defaultIsolation(nonisolated.self) +``` + +We can also pursue allowing bare metatypes without `.self` to allow: + +```swift +SwiftSetting.defaultIsolation(nonisolated) +SwiftSetting.defaultIsolation(MainActor) +``` + +[SE-0466]: /proposals/0466-control-default-actor-isolation.md diff --git a/proposals/0479-method-and-initializer-keypaths.md b/proposals/0479-method-and-initializer-keypaths.md new file mode 100644 index 0000000000..bd1db9fe2e --- /dev/null +++ b/proposals/0479-method-and-initializer-keypaths.md @@ -0,0 +1,176 @@ +# Method and Initializer Key Paths + +* Proposal: [SE-0479](0479-method-and-initializer-keypaths.md) +* Authors: [Amritpan Kaur](https://github.com/amritpan), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Becca Royal-Gordon](https://github.com/beccadax) +* Status: **Active Review (April 22 ... May 5, 2025)** +* Implementation: [swiftlang/swift#78823](https://github.com/swiftlang/swift/pull/78823), [swiftlang/swiftsyntax#2950](https://github.com/swiftlang/swift-syntax/pull/2950), [swiftlang/swiftfoundation#1179](https://github.com/swiftlang/swift-foundation/pull/1179) +* Experimental Feature Flag: `KeyPathWithMethodMembers` +* Review: ([pitch](https://forums.swift.org/t/pitch-method-key-paths/76678)) ([review](https://forums.swift.org/t/se-0479-method-and-initializer-key-paths/79457)) + +## Introduction + +Swift key paths can be written to properties and subscripts. This proposal extends key path usage to include references to method members, such as instance and type methods, and initializers. + +## Motivation + +Key paths to method members and their advantages have been explored in several discussions on the Swift forum, specifically to [unapplied instance methods](https://forums.swift.org/t/allow-key-paths-to-reference-unapplied-instance-methods/35582) and to [partially and applied methods](https://forums.swift.org/t/pitch-allow-keypaths-to-represent-functions/67630). Extending key paths to include reference to methods and initializers and handling them similarly to properties and subscripts will unify instance and type member access for a more consistent API. While this does not yet encompass all method kinds, particularly those with effectful or non-hashable arguments, it lays the groundwork for more expressive, type-safe APIs. In doing so, it brings many of the benefits of existing key path components to supported methods and initializers, such as abstraction, reusability via generic functions and dynamic invocation with state type safety. + +## Proposed solution + +We propose the following usage: + +```swift +struct Calculator { + func square(of number: Int) -> Int { + return number * number * multiplier + } + + func cube(of number: Int) -> Int { + return number * number * number * multiplier + } + + init(multiplier: Int) { + self.multiplier = multiplier + } + + let multiplier: Int +} + +// Key paths to Calculator methods +let squareKeyPath = \Calculator.square +let cubeKeyPath = \Calculator.cube +``` + +These key paths can then be invoked dynamically with a generic function: + +```swift +func invoke(object: T, keyPath: KeyPath U>, param: U) -> U { + return object[keyPath: keyPath](param) +} + +let calc = Calculator(multiplier: 2) + +let squareResult = invoke(object: calc, keyPath: squareKeyPath, param: 3) +let cubeResult = invoke(object: calc, keyPath: cubeKeyPath, param: 3) +``` + +Or used to dynamically create a new instance of Calculator: + +```swift +let initializerKeyPath = \Calculator.Type.init(multiplier: 5) +``` + +This proposed feature homogenizes the treatment of member declarations by extending the expressive power of key paths to method and initializer members. + +## Detailed design + +Key path expressions can refer to instance methods, type methods and initializers, and imitate the syntax of non-key path member references. + +### Argument Application + +Key paths can reference methods in two forms: + +1. Without argument application: The key path represents the unapplied method signature. +2. With argument application: The key path references the method with arguments already applied. + +Continuing our `Calculator` example, we can write either: + +```swift +let squareWithoutArgs: KeyPath Int> = \Calculator.square +let squareWithArgs: KeyPath = \Calculator.square(of: 3) +``` + +If the member is a metatype (e.g., a static method, class method, initializer, or when referring to the type of an instance), you must explicitly include `.Type` in the key path root type. + +```swift +struct Calculator { + static func add(_ a: Int, _ b: Int) -> Int { + return a + b + } +} + +let addKeyPath: KeyPath = \Calculator.Type.add(4, 5) +``` + +Here, `addKeyPath` is a key path that references the add method of `Calculator` as a metatype member. The key path’s root type is `Calculator.Type`, and the value resolves to an applied instance method result type of`Int`. + +### Overloads + +Keypaths to methods with the same base name and distinct argument labels can be disambiguated with explicit argument labels: + +```swift +struct Calculator { + var subtract: (Int, Int) -> Int { return { $0 + $1 } } + func subtract(this: Int) -> Int { this + this} + func subtract(that: Int) -> Int { that + that } +} + +let kp1 = \Calculator.subtract // KeyPath Int +let kp2 = \Calculator.subtract(this:) // KeyPath Int> +let kp3 = \Calculator.subtract(that:) // KeyPath Int> +let kp4 = \Calculator.subtract(that: 1) // KeyPath +``` + +### Implicit closure conversion + +This feature also supports implicit closure conversion of key path methods, allowing them to used in expressions where closures are expected, such as in higher order functions: + +```swift +struct Calculator { + func power(of base: Int, exponent: Int) -> Int { + return Int(pow(Double(base), Double(exponent))) + } +} + +let calculators = [Calculator(), Calculator()] +let results = calculators.map(\.power(of: 2, exponent: 3)) +``` + +### Dynamic member lookups + +`@dynamicMemberLookup` can resolve method references through key paths, allowing methods to be accessed dynamically without explicit function calls: + +```swift +@dynamicMemberLookup +struct DynamicKeyPathWrapper { + var root: Root + + subscript(dynamicMember keyPath: KeyPath) -> Member { + root[keyPath: keyPath] + } +} + +let dynamicCalculator = DynamicKeyPathWrapper(root: Calculator()) +let power = dynamicCalculator.power +print(power(10, 2)) +``` + +### Effectful value types + +Methods annotated with `nonisolated` and `consuming` are supported by this feature. However, noncopying root and value types [are not supported](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0437-noncopyable-stdlib-primitives.md#additional-future-work). `mutating`, `throws` and `async` are not supported for any other component type and will similarly not be supported for methods. Additionally keypaths cannot capture closure arguments that are not `Hashable`/`Equatable`. + +### Component chaining + +Component chaining between methods or from method to other key path types is also supported with this feature and will continue to behave as `Hashable`/`Equatable` types. + +```swift +let kp5 = \Calculator.subtract(this: 1).signum() +let kp6 = \Calculator.subtract(this: 2).description +``` + +## Source compatibility + +This feature has no effect on source compatibility. + +## ABI compatibility + +This feature does not affect ABI compatibility. + +## Implications on adoption + +This feature has no implications on adoption. + +## Future directions + +The effectful value types that are unsupported by this feature will all require new `KeyPath` types and so have been left out of this proposal. Additionally, this lack of support impacts existing key path component kinds and could be addressed in a unified proposal that resolves this gap across all key path component kinds. diff --git a/proposals/0480-swiftpm-warning-control.md b/proposals/0480-swiftpm-warning-control.md new file mode 100644 index 0000000000..c0faff43e7 --- /dev/null +++ b/proposals/0480-swiftpm-warning-control.md @@ -0,0 +1,329 @@ +# Warning Control Settings for SwiftPM + +* Proposal: [SE-0480](0480-swiftpm-warning-control.md) +* Authors: [Dmitrii Galimzianov](https://github.com/DmT021) +* Review Manager: [John McCall](https://github.com/rjmccall), [Franz Busch](https://github.com/FranzBusch) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift-package-manager#8315](https://github.com/swiftlang/swift-package-manager/pull/8315) +* Review: ([pitch](https://forums.swift.org/t/pitch-warning-control-settings-for-swiftpm/78666)) ([review](https://forums.swift.org/t/se-0480-warning-control-settings-for-swiftpm/79475)) ([returned for revision](https://forums.swift.org/t/se-0480-warning-control-settings-for-swiftpm/79475/8)) ([acceptance](https://forums.swift.org/t/accepted-se-0480-warning-control-settings-for-swiftpm/80327)) +* Previous Proposal: [SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md) + +## Introduction + +This proposal adds new settings to SwiftPM to control how the Swift, C, and C++ compilers treat warnings during the build process. It builds on [SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md), which introduced warning control flags for the Swift compiler but left SwiftPM support as a future direction. + +## Motivation + +The Swift Package Manager currently lacks a unified way to control warnings across Swift, C, and C++ compilation. This limitation forces developers to either use `unsafeFlags` or accept the default warning settings. + +## Proposed solution + +This proposal introduces new methods to SwiftPM's build settings API, allowing fine-grained control over warnings. + +### API + +#### Cross-language API (Swift, C, and C++) + +```swift +/// The level at which a compiler warning should be treated. +public enum WarningLevel: String { + /// Treat as a warning. + /// + /// Warnings will be displayed during compilation but will not cause the build to fail. + case warning + + /// Treat as an error. + /// + /// Warnings will be elevated to errors, causing the build to fail if any such warnings occur. + case error +} + +extension SwiftSetting { // Same for CSetting and CXXSetting + public static func treatAllWarnings( + as level: WarningLevel, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting // or CSetting or CXXSetting + + public static func treatWarning( + _ name: String, + as level: WarningLevel, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting // or CSetting or CXXSetting +} +``` + +#### C/C++-specific API + +In C/C++ targets, we can also enable or disable specific warning groups, in addition to controlling their severity. + +```swift +extension CSetting { // Same for CXXSetting + public static func enableWarning( + _ name: String, + _ condition: BuildSettingCondition? = nil + ) -> CSetting // or CXXSetting + + public static func disableWarning( + _ name: String, + _ condition: BuildSettingCondition? = nil + ) -> CSetting // or CXXSetting +} +``` +_The necessity of these functions is also explained below in the Alternatives considered section._ + +### Example usage + +```swift +.target( + name: "MyLib", + swiftSettings: [ + .treatAllWarnings(as: .error), + .treatWarning("DeprecatedDeclaration", as: .warning), + ], + cSettings: [ + .enableWarning("all"), + .disableWarning("unused-function"), + + .treatAllWarnings(as: .error), + .treatWarning("unused-variable", as: .warning), + ], + cxxSettings: [ + .enableWarning("all"), + .disableWarning("unused-function"), + + .treatAllWarnings(as: .error), + .treatWarning("unused-variable", as: .warning), + ] +) +``` + +## Detailed design + +### Settings and their corresponding compiler flags + +| Method | Swift | C/C++ | +|--------|-------|-------| +| `treatAllWarnings(as: .error)` | `-warnings-as-errors` | `-Werror` | +| `treatAllWarnings(as: .warning)` | `-no-warnings-as-errors` | `-Wno-error` | +| `treatWarning("XXXX", as: .error)` | `-Werror XXXX` | `-Werror=XXXX` | +| `treatWarning("XXXX", as: .warning)` | `-Wwarning XXXX` | `-Wno-error=XXXX` | +| `enableWarning("XXXX")` | N/A | `-WXXXX` | +| `disableWarning("XXXX")` | N/A | `-Wno-XXXX` | + +### Order of settings evaluation + +The order in which warning control settings are specified in a target's settings array directly affects the order of the resulting compiler flags. This is critical because when multiple flags affect the same warning group, compilers apply them sequentially with the last flag taking precedence. + +For example, consider these two different orderings for C++ settings: + +```swift +// Example 1: "unused-variable" in front of "unused" +cxxSettings: [ + .treatWarning("unused-variable", as: .error), + .treatWarning("unused", as: .warning), +] + +// Example 2: "unused" in front of "unused-variable" +cxxSettings: [ + .treatWarning("unused", as: .warning), + .treatWarning("unused-variable", as: .error), +] +``` + +In Example 1, the compiler will receive flags in this order: +``` +-Werror=unused-variable -Wno-error=unused +``` +Since "unused-variable" is a specific subgroup of the broader "unused" group, and the "unused" flag is applied last, all unused warnings (including unused-variable) will be treated as warnings. + +In Example 2, the compiler will receive flags in this order: +``` +-Wno-error=unused -Werror=unused-variable +``` +Due to the "last one wins" rule, unused-variable warnings will be treated as errors, while other unused warnings remain as warnings. + +The same principle applies when combining any of the new build settings: + +```swift +cxxSettings: [ + .enableWarning("all"), // Enable the "all" warning group + .enableWarning("extra"), // Enable the "extra" warning group + .disableWarning("unused-parameter"), // Disable the "unused-parameter" warning group + .treatAllWarnings(as: .error), // Treat all warnings as errors + .treatWarning("unused", as: .warning), // Keep warnings of the "unused" group as warnings +] +``` + +This will result in compiler flags: +``` +-Wall -Wextra -Wno-unused-parameter -Werror -Wno-error=unused +``` + +When configuring warnings, be mindful of the order to achieve the desired behavior. + +### Remote targets behavior + +When a target is remote (pulled from a package dependency rather than defined in the local package), the warning control settings specified in the manifest do not apply to it. SwiftPM will strip all of the warning control flags for remote targets and substitute them with options for suppressing warnings (`-w` for Clang and `-suppress-warnings` for Swift). + +This behavior is already in place but takes into account only `-warnings-as-errors` (for Swift) and `-Werror` (for Clang) flags. We expand this list to include the following warning-related flags: + +**For C/C++:** +* `-Wxxxx` +* `-Wno-xxxx` +* `-Werror` +* `-Werror=xxxx` +* `-Wno-error` +* `-Wno-error=xxxx` + +**For Swift:** +* `-warnings-as-errors` +* `-no-warnings-as-errors` +* `-Wwarning xxxx` +* `-Werror xxxx` + +This approach ensures that warning control settings are applied only to the targets you directly maintain in your package, while dependencies remain buildable without warnings regardless of their warning settings. + +### Interaction with command-line flags + +SwiftPM allows users to pass additional flags to the compilers using the `-Xcc`, `-Xswiftc`, and `-Xcxx` options with the `swift build` command. These flags are appended **after** the flags generated from the package manifest. + +This ordering enables users to modify or override package-defined warning settings without modifying the package manifest. + +#### Example + +```swift +let package = Package( + name: "MyExecutable", + targets: [ + // C target with warning settings + .target( + name: "cfoo", + cSettings: [ + .enableWarning("all"), + .treatAllWarnings(as: .error), + .treatWarning("unused-variable", as: .warning), + ] + ), + // Swift target with warning settings + .executableTarget( + name: "swiftfoo", + swiftSettings: [ + .treatAllWarnings(as: .error), + .treatWarning("DeprecatedDeclaration", as: .warning), + ] + ), + ] +) +``` + +When built with additional command-line flags: + +```sh +swift build -Xcc -Wno-error -Xswiftc -no-warnings-as-errors +``` + +The resulting compiler invocations will include both sets of flags: + +``` +# C compiler invocation +clang ... -Wall -Werror -Wno-error=unused-variable ... -Wno-error ... + +# Swift compiler invocation +swiftc ... -warnings-as-errors -Wwarning DeprecatedDeclaration ... -no-warnings-as-errors -Xcc -Wno-error ... +``` + +Flags are processed from left to right, and since `-no-warnings-as-errors` and `-Wno-error` apply globally to all warnings, they override the warning treating flags defined in the package manifest. + +#### Limitations + +This approach has a limitation when used with `-suppress-warnings`, which is mutually exclusive with other warning control flags: + +```sh +swift build -Xswiftc -suppress-warnings +``` + +Results in compiler errors: + +``` +error: conflicting options '-warnings-as-errors' and '-suppress-warnings' +error: conflicting options '-Wwarning' and '-suppress-warnings' +``` + + +## Security + +This change has no impact on security, safety, or privacy. + +## Impact on existing packages + +The proposed API will only be available to packages that specify a tools version equal to or later than the SwiftPM version in which this functionality is implemented. + +## Alternatives considered + +### Disabling a warning via a treat level + +Clang allows users to completely disable a specific warning, so for C/C++ settings we could implement that as a new case in the `WarningLevel` enum: + +```swift +public enum WarningLevel { + case warning + case error + case ignored +} +``` + +_(Since Swift doesn't allow selective warning suppression, we would actually have to split the enum into two: `SwiftWarningLevel` and `CFamilyWarningLevel`)_ + +But some warnings in Clang are disabled by default. If we simply pass `-Wno-error=unused-variable`, the compiler won't actually produce a warning for an unused variable. It only makes sense to use it if we have enabled the warning: `-Wunused-variable -Werror -Wno-error=unused-variable`. + +This necessitates separate functions to enable and disable warnings. Therefore, instead of `case ignored`, we propose the functions `enableWarning` and `disableWarning`. + +## Future directions + +### Package-level settings + +It has been noted that warning control settings are often similar across all targets. It makes sense to declare them at the package level while allowing target-level customizations. However, many other settings would also likely benefit from such inheritance, and SwiftPM doesn't currently provide such an option. Therefore, it was decided to factor this improvement out and look at all the settings holistically in the future. + +### Support for other C/C++ Compilers + +The C/C++ warning control settings introduced in this proposal are initially implemented with Clang's warning flag syntax as the primary target. However, the API itself is largely compiler-agnostic, and there's potential to extend support to other C/C++ compilers in the future. + +For instance, many of the proposed functions could be mapped to flags for other compilers like MSVC: + +| SwiftPM Setting | Clang | MSVC (Potential Mapping) | +| :-------------------------------- | :---------------- | :----------------------- | +| `.treatAllWarnings(as: .error)` | `-Werror` | `/WX` | +| `.treatAllWarnings(as: .warning)` | `-Wno-error` | `/WX-` | +| `.treatWarning("name", as: .error)`| `-Werror=name` | `/we####` (where `####` is MSVC warning code) | +| `.treatWarning("name", as: .warning)`| `-Wno-error=name` | No direct equivalent | +| `.enableWarning("name")` | `-Wname` | `/wL####` (e.g., `/w4####` to enable at level 4) | +| `.disableWarning("name")` | `-Wno-name` | `/wd####` | + +Where direct mappings are incomplete (like `.treatWarning(as: .warning)` for MSVC, which doesn't have a per-warning equivalent to Clang's `-Wno-error=XXXX`), SwiftPM could emit diagnostics indicating the setting is not fully supported by the current compiler. If more fine-grained control is needed for a specific compiler (e.g., MSVC's warning levels `0-4` for `enableWarning`), future enhancements could introduce compiler-specific settings or extend the existing API. + +A key consideration is the handling of warning names or codes (the `"name"` parameter in the API). SwiftPM does not maintain a comprehensive list of all possible warning identifiers and their mapping across different compilers. Instead, package authors would be responsible for providing the correct warning name or code for the intended compiler. + +To facilitate this, if support for other C/C++ compilers is added, the existing `BuildSettingCondition` API could be extended to allow settings to be applied conditionally based on the active C/C++ compiler. For example: + +```swift +cxxSettings: [ + // Clang-specific warning + .enableWarning("unused-variable", .when(cxxCompiler: .clang)), + // MSVC-specific warning (using its numeric code) + .enableWarning("4101", .when(cxxCompiler: .msvc)), + // Common setting that maps well + .treatAllWarnings(as: .error) +] +``` + +This approach, combined with the existing behavior where remote (dependency) packages have their warning control flags stripped and replaced with suppression flags, would allow projects to adopt new compilers. Even if a dependency uses Clang-specific warning flags, it would not cause build failures when the main project is built with a different compiler like MSVC, as those flags would be ignored. + +### Formalizing "Development-Only" Build Settings + +The warning control settings introduced by this proposal only apply when a package is built directly and are suppressed when the package is consumed as a remote dependency. + +During the review of this proposal, it was suggested that this "development-only" characteristic could be made more explicit, perhaps by introducing a distinct category of settings (e.g., `devSwiftSettings`). This is an interesting avenue for future exploration. SwiftPM already has a few other settings that exhibit similar behavior. A dedicated future proposal for "development-only" settings could address all such use cases holistically, providing a clearer and more general mechanism for package authors to distinguish between "dev-only" settings and those that propagate to consumers. + +## Acknowledgments + +Thank you to [Doug Gregor](https://github.com/douggregor) for the motivation, and to both [Doug Gregor](https://github.com/douggregor) and [Holly Borla](https://github.com/hborla) for their guidance during the implementation of this API. diff --git a/proposals/0481-weak-let.md b/proposals/0481-weak-let.md new file mode 100644 index 0000000000..ca885adc1c --- /dev/null +++ b/proposals/0481-weak-let.md @@ -0,0 +1,123 @@ +# `weak let` + +* Proposal: [SE-0481](0481-weak-let.md) +* Authors: [Mykola Pokhylets](https://github.com/nickolas-pohilets) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Accepted** +* Implementation: [swiftlang/swift#80440](https://github.com/swiftlang/swift/pull/80440) +* Review: ([discussion](https://forums.swift.org/t/weak-captures-in-sendable-sending-closures/78498)) ([pitch](https://forums.swift.org/t/pitch-weak-let/79271)) ([review](https://forums.swift.org/t/se-0481-weak-let/79603)) ([acceptance](https://forums.swift.org/t/accepted-se-0481-weak-let/79895)) + +[SE-0302]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0302-concurrent-value-and-concurrent-closures.md + +## Introduction + +Swift provides weak object references using the `weak` modifier on variables and stored properties. Weak references become `nil` when the object is destroyed, causing the value of the variable to seem to change. Swift has therefore always required `weak` references to be declared with the `var` keyword rather than `let`. However, that causes unnecessary friction with [sendability checking][SE-0302]: because weak references must be mutable, classes and closures with such references are unsafe to share between concurrent contexts. This proposal lifts that restriction and allows `weak` to be combined with `let`. + +## Motivation + +Currently, Swift classes with weak stored properties cannot be `Sendable`, because weak properties have to be mutable, and mutable properties are not allowed in `Sendable` classes: + +```swift +final class C: Sendable {} + +final class VarUser: Sendable { + weak var ref1: C? // error: stored property 'ref1' of 'Sendable'-conforming class 'VarUser' is mutable +} +``` + +Similarly, closures with explicit `weak` captures cannot be `@Sendable`, because such captures are implicitly *made* mutable, and `@Sendable` closures cannot capture mutable variables. This is surprising to most programmers, because every other kind of explicit capture is immutable. It is extremely rare for Swift code to directly mutate a `weak` capture. + +```swift +func makeClosure() -> @Sendable () -> Void { + let c = C() + return { [weak c] in + c?.foo() // error: reference to captured var 'c' in concurrently-executing code + + c = nil // allowed, but surprising and very rare + } +} +``` + +In both cases, allowing the weak reference to be immutable would solve the problem, but this is not currently allowed: + +```swift +final class LetUser: Sendable { + weak let ref1: C? // error: 'weak' must be a mutable variable, because it may change at runtime +} +``` + +The restriction that weak references have to be mutable is based on the idea that the reference is mutated when the referenced object is destroyed. Since it's mutated, it must be kept in mutable storage, and hence the storage must be declared with `var`. This way of thinking about weak references is problematic, however; it does not work very well to explain the behavior of weak references that are components of other values, such as `struct`s. For example, a return value is normally an immutable value, but a `struct` return value can contain a weak reference that may become `nil` at any point. + +In fact, wrapping weak references in a single-property `struct` is a viable workaround to the `var` restriction in both properties and captures: + +```swift +struct WeakRef { + weak var ref: C? +} + +final class WeakStructUser: Sendable { + let ref: WeakRef // ok +} + +func makeClosure() -> @Sendable () -> Void { + let c = C() + return { [c = WeakRef(ref: c)] in + c.ref?.foo() // ok + } +} +``` + +The existence of this simple workaround is itself an argument that the prohibition of `weak let` is not enforcing some fundamentally important rule. + +It is true that the value of a `weak` variable can be observed to change when the referenced object is destroyed. However, this does not have to be thought of as a mutation of the variable. A different way of thinking about it is that the variable continues to hold the same weak reference to the object, but that the program is simply not allowed to observe the object through that weak reference after the object is destroyed. This better explains the behavior of weak references in `struct`s: it's not that the destruction of the object changes the `struct` value, it's that the weak reference that's part of the `struct` value will now return `nil` if you try to observe it. + +Note that all of this relies on the fact that the thread-safety of observing a weak reference is fundamentally different from the thread-safety of assigning `nil` into a `weak var`. Swift's weak references are thread-safe against concurrent destruction: well-ordered reads and writes to a `weak var` or `weak let` will always behave correctly even if the referenced object is concurrently destroyed. But they are not *atomic* in the sense that writing to a `weak var` will behave correctly if another context is concurrently reading or writing to that same `var`. In this sense, a `weak var` is like any other `var`: mutations need to be well-ordered with all other accesses. + +## Proposed solution + +`weak` can now be freely combined with `let` in any position that `weak var` would be allowed. +Similar to `weak var`, `weak let` declarations also must be of `Optional` type. + +This proposal maintains the status quo regarding `weak` on function arguments and computed properties: +* There is no valid syntax to indicate that function argument is a weak reference. +* `weak` on computed properties is allowed, but has no effect. + +An explicit `weak` capture is now immutable under this proposal, like any other explicit capture. If the programmer really needs a mutable capture, they must capture a separate `weak var`: + +```swift +func makeClosure() -> @Sendable () -> Void { + let c = C() + // Closure is @Sendable + return { [weak c] in + c?.foo() + c = nil // error: cannot assign to value: 'c' is an immutable capture + } +} + +func makeNonSendableClosure() -> () -> Void { + let c = C() + weak var explicitlyMutable: C? = c + // Closure cannot be @Sendable anymore + return { + explicitlyMutable?.foo() + explicitlyMutable = nil // ok + } +} +``` + +## Source compatibility + +Allowing `weak let` bindings is an additive change that makes previously invalid code valid. It is therefore perfectly source-compatible. + +Treating weak captures as immutable is a source-breaking change. Any code that attempts to write to the capture will stop compiling. +The overall amount of such code is expected to be small. + +Since the captures of a closure are opaque and cannot be observed outside of the closure, changing the mutability of weak captures has no impact on clients of the closure. + +## ABI compatibility + +There is no ABI impact of this change. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source code with no deployment constraints and without affecting source or ABI compatibility. diff --git a/proposals/0482-swiftpm-static-library-binary-target-non-apple-platforms.md b/proposals/0482-swiftpm-static-library-binary-target-non-apple-platforms.md new file mode 100644 index 0000000000..5c7d90c92c --- /dev/null +++ b/proposals/0482-swiftpm-static-library-binary-target-non-apple-platforms.md @@ -0,0 +1,164 @@ +# Binary Static Library Dependencies + +* Proposal: [SE-0482](0482-swiftpm-static-library-binary-target-non-apple-platforms.md) +* Authors: [Daniel Grumberg](https://github.com/daniel-grumberg), [Max Desiatov](https://github.com/MaxDesiatov), [Franz Busch](https://github.com/FranzBusch) +* Review Manager: [Kuba Mracek](https://github.com/kubamracek) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift-package-manager#8639](https://github.com/swiftlang/swift-package-manager/pull/8639) [swiftlang/swift-package-manager#8741](https://github.com/swiftlang/swift-package-manager/pull/8741) +* Review: ([discussion](https://forums.swift.org/t/se-0482-binary-static-library-dependencies/79634)) ([pitch](https://forums.swift.org/t/pitch-swiftpm-support-for-binary-static-library-dependencies/78619)) ([acceptance](https://forums.swift.org/t/accepted-se-0482-binary-static-library-dependencies/80042)) +* Bugs: [Swift Package Manger Issue](https://github.com/swiftlang/swift-package-manager/issues/7035) + +## Introduction + +Swift continues to grow as a cross-platform language supporting a wide variety of use cases from [programming embedded device](https://www.swift.org/blog/embedded-swift-examples/) to [server-side development](https://www.swift.org/documentation/server/) across a multitude of [operating systems](https://www.swift.org/documentation/articles/static-linux-getting-started.html). +However, currently SwiftPM supports linking against binary dependencies on Apple platforms only. +This proposal aims to make it possible to provide static library dependencies exposing a C interface on non-Apple platforms that depend only on the standard C library. +The scope of this proposal is C libraries only, distributing Swift libraries has additional challenges (see [Future directions](#future-directions). + +## Motivation + +The Swift Package Manager’s [`binaryTarget` type](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md) lets packages vend libraries that either cannot be built in Swift Package Manager for technical reasons, +or for which the source code cannot be published for legal or other reasons. + +In the current version of SwiftPM, binary targets support the following: + +* Libraries in an Xcode-oriented format called XCFramework, and only for Apple platforms, introduced in [SE-0272](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md). +* Executables through the use of artifact bundles introduced in [SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md). + +We aim here to bring a subset of the XCFramework capabilities to non-Apple platforms in a safe way. + +While this proposal is specifically focused on binary static library dependencies without unexpected unresolved external symbols on non-Apple platforms, +it tries to do so in a way that will not prevent broader future support for static libraries and dynamically linked libraries. + +## Proposed solution + +This proposal extends artifact bundles introduced by [SE-0305](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md) to include a new kind of artifact type to represent a binary library dependency: `staticLibrary`. +The artifact manifest would encode the following information for each variant: + +* The static library to pass to the linker. + On Apple and Linux platforms, this would be `.a` files and on Windows it would be a `.lib` file. +* Enough information to be able to use the library's API in the packages source code, + i.e., headers and module maps for libraries exporting a C-based interface. + +Additionally, we propose the addition of an auditing tool that can validate the library artifact is safe to use across the Linux-based platforms supported by the Swift project. +Such a tool would ensure that people do not accidentally distribute artifacts that require dependencies that are not met on the various deployment platforms. +However when an artifact isn't widely consumed and all dependent packages are known, +artifact vendors can provide artifacts with dependencies on other C libraries provided that each client target depends explicitly on all required dependencies of the artifact. + +## Detailed design + +This section describes the changes to artifact bundle manifests in detail, the semantic impact of the changes on SwiftPM's build infrastructure, and describes the operation of the auditing tool. + +### Artifact Manifest Semantics + +The artifact manifest JSON format for a static library is described below: + +```json +{ + "schemaVersion": "1.0", + "artifacts": { + "": { + "version": "", + "type": "staticLibrary", + "variants": [ + { + "path": "", + "supportedTriples": ["", ... ], + "staticLibraryMetadata": { + "headerPaths": [", ...], + "moduleMapPath": "" + } + }, + ... + ] + }, + ... + } +} + +``` + +The additions are: + +* The `staticLibrary` artifact `type` that indicates this binary artifact is not an executable but rather a static library to link against. +* The `headerPaths` field specifies directory paths relative to the root of the artifact bundle that contain the header interfaces to the static library. + These are forwarded along to the swift compiler (or the C compiler) using the usual search path arguments. +* The optional `moduleMapPath` field specifies the path relative to the root of the artifact bundle that contains a custom module map to use if the header paths do not contain the module definitions or to provide custom overrides. + This field is required if the library's API is to be imported into Swift code. + +As with executable binary artifacts, the `path` field represents the relative path to the binary from the root of the artifact bundle, +and the `supportedTriples` field provides information about the target triples supported by this variant. + +An example artifact might look like: + +```json +{ + "schemaVersion": "1.0", + "artifacts": { + "example": { + "type": "staticLibrary", + "version": "1.0.0", + "variants": [ + { + "path": "libExample.a", + "supportedTriples": ["aarch64-unknown-linux-gnu"], + "staticLibraryMetadata": { + "headerPaths": ["include"], + "moduleMapPath": "include/example.modulemap" + } + } + ] + } + } +} +``` + +### Auditing tool + +Without proper auditing it would be very easy to provide binary static library artifacts that call into unresolved external symbols that are not available on the runtime platform, e.g., due to missing linkage to a system dynamic library. + +We propose the introduction of a new tool that can validate the "safety" of a binary library artifact across the platforms it supports and the corresponding runtime environment. + +In this proposal we restrict ourselves to static libraries that do not have any external dependencies beyond the C standard library and runtime. +To achieve this we need to be able to detect validate this property across the three object file formats used in static libraries on our supported platforms: [Mach-O](https://developer.apple.com/library/archive/documentation/Performance/Conceptual/CodeFootprint/Articles/MachOOverview.html#//apple_ref/doc/uid/20001860-BAJGJEJC) on Apple platforms, [ELF](https://refspecs.linuxfoundation.org/elf/elf.pdf) on Linux-based platforms, and [COFF](https://learn.microsoft.com/en-us/windows/win32/debug/pe-format) on Windows. +All three formats express references to external symbols as _relocations_ which reside in a single section of each object file. + +We propose adding the `llvm-objdump` to the toolchain to provide the capability to inspect relocations across all three supported object file formats. The tool would use `llvm-objdump` every object file in the static library and construct a complete list of symbols defined and referenced across the entire library. +Additionally, the tool would construct a simple C compiler invocation to derive to generate a default linker invocation. +This would be used to derive the libraries linked by default in a C program, these libraries would then be scanned to contribute to the list of defined symbols. +The tool would then check that the referenced symbols list is a subset of the set of defined symbols and emit an error otherwise. + +This would be sufficient to guarantee that all symbols from the static library would be available at runtime for statically linked executables or for ones running on the build host. +To ensure maximum runtime compatibility we would also provide a Linux-based Docker image that uses the oldest supported `glibc` for a given Swift version. +As `glibc` is backwards compatible, a container running the audit on a given static library would ensure that the version of `glibc` on any runtime platform would be compatible with the binary artifact. +This strategy as been successfully employed in the Python community with [`manylinux`](https://peps.python.org/pep-0513/). + +## Security + +This proposal brings the security implications outlined in [SE-0272](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0272-swiftpm-binary-dependencies.md#security) to non-Apple platforms, +namely that a malicious attacker having access to both the server hosting the artifact and the git repository that vends the Package Manifest could provide a malicious library. +Users should exercise caution when onboarding binary dependencies. + +## Impact on existing packages + +No current package should be affected by this change since this is only an additive change in enabling SwiftPM to use binary target library dependencies on non-Apple platforms. + +## Future directions + +### Support Swift static libraries + +To do this we would extend the static library binary artifact manifest to provide a `.swiftinterface` file that can be consumed by the Swift compiler to import the Swift APIs. +Additionally we would extend the auditing tool to validate the usage of Swift standard library and runtime symbols, e.g., from `libSwiftCore`. + +### Extend binary compatibility guarantees + +This proposal limits itself to providing facilities for binary compatibility only with the C standard library and runtime. +In the future we could provide a system to allow binary artifact distributors to specify additional linkage dependencies for their binary artifacts. +These would be used to customize the operation of the audit tool and perform automatic linking of them in any client target that depends on the binary artifact, in the same way [CMake](https://cmake.org/cmake/help/v4.0/prop_tgt/INTERFACE_LINK_LIBRARIES.html) propagates link dependencies transitively. + +### Add support for dynamically linked dependencies + +On Windows dynamic linking requires an _import library_ which is a small static library that contains stubs for symbols exported by the dynamic library. +These stubs are roughly equivalent to a PLT entry in an ELF executable, but are generated during the build of the dynamic library and must be provided to clients of the library for linking purposes. +Similarly on Linux and Apple platforms binary artifact maintainers may wish to provide a dynamic library stub to improve link performance. +To support these use cases the library binary artifact manifest schema could be extended to provide facilities to provide both a link-time and runtime dependency. diff --git a/proposals/0483-inline-array-sugar.md b/proposals/0483-inline-array-sugar.md new file mode 100644 index 0000000000..507605ec34 --- /dev/null +++ b/proposals/0483-inline-array-sugar.md @@ -0,0 +1,205 @@ +# `InlineArray` Type Sugar + +* Proposal: [SE-0483](0483-inline-array-sugar.md) +* Authors: [Hamish Knight](https://github.com/hamishknight), [Ben Cohen](https://github.com/airspeedswift) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Implemented (Swift 6.2)** +* Review: ([pitch](https://forums.swift.org/t/pitch-inlinearray-type-sugar/79142)) ([first review](https://forums.swift.org/t/se-0483-inlinearray-literal-syntax/79643)) ([second review](https://forums.swift.org/t/second-review-se-0483-inlinearray-type-sugar/80337)) ([acceptance](https://forums.swift.org/t/accepted-se-0483-inlinearray-type-sugar/81509)) + +## Introduction + +We propose the introduction of type sugar for the `InlineArray` type, providing more succinct syntax for declaring an inline array. + +## Motivation + +[SE-0453](/proposals/0453-vector.md) introduced a new type, `InlineArray`, which includes a size parameter as part of its type: + +``` +let fiveIntegers: InlineArray<5, Int> = .init(repeating: 99) +``` + +Declaring this type is more cumbersome than its equivalent dynamically-sized array, which has sugar for the type syntax: + +```swift +let fiveIntegers: [Int] = .init(repeating: 99, count: 5) +``` + +This becomes more pronounced when dealing with multiple dimensions: + +```swift +let fiveByFive: InlineArray<5, InlineArray<5, Int>> = .init(repeating: .init(repeating: 99)) +``` + +Almost every other language in a similar category to Swift – C, C++, Objective-C, Pascal, Go, Rust, Zig, Java, C# – has a simple syntax for their fixed-size array type. The introduction of a fixed-size array type into Swift should also introduce a shorthand syntax, in keeping with Swift's general approach of low ceremony and concise syntax. Swift further deviates from its peer languages by giving its _dynamic_ array type, `Array` (known in many other languages as `vector`) a sugared form. This can lead to an assumption that `Array` should be used under almost all circumstances, despite it having significant downsides in many uses (see further discussion in alternatives considered). + +## Proposed solution + +A new sugared version of the `InlineArray` type is proposed: + +```swift +let fiveIntegers: [5 of Int] = .init(repeating: 99) +``` + +The choice of `of` forms something close to a grammatical phrase ("an array of five ints"). A short contextual keyword is also in keeping with Swift's tradition in other areas such as `in` or `let`. + +## Detailed design + +The new syntax consists of the value for the integer generic parameter and the type of the element generic parameter, separated by `of`. + +This will be added to the grammar alongside the current type sugar: + +> **Grammar of a type** +> _type → sized-array-type_ +> +> **Grammar of a sized array type** +> _sized-array-type → [ expression `of` type ]_ + +Note that while the grammar allows for any expression, this is currently limited to only integer literals or integer type parameters, as required by the current implementation of `InlineArray`. If that restriction changes, so would the value allowed in the expression in the sugar. + +The new sugar is equivalent to declaring a type of `InlineArray`, so all rules that can be applied to the generic placeholders for the unsugared version also apply to the sugared version: + +```swift +// Nesting +let fiveByFive: InlineArray<5, InlineArray<5, Int>> = .init(repeating: .init(repeating: 99)) +let fiveByFive: [5 of [5 of Int]] = .init(repeating: .init(repeating: 99)) + +// Inference from context: +let fiveIntegers: [5 of _] = .init(repeating: 99) +let fourBytes: [_ of Int8] = [1,2,3,4] +let fourIntegers: [_ of _] = [1,2,3,4] + +// use on rhs +let fiveDoubles = [5 of _](repeating: 1.23) +``` + +The sugar can also be used in place of the unsugared type wherever it might appear: + +```swift +[5 of Int](repeating: 99) +MemoryLayout<[5 of Int]>.size +unsafeBitCast((1,2,3), to: [3 of Int].self) +``` + +There must be whitespace on either side of the separator; i.e., you cannot write `[5of Int]`. There are no requirements to balance whitespace; `[5 of Int]` is permitted. A new line can appear after the `of` but not before it, as while this is not ambiguous, this aids with the parser recovery logic, leading to better syntax error diagnostics. + +## Source Compatibility + +Since it is not currently possible to write any form of the proposed syntax in Swift today, this proposal does not alter the meaning of any existing code. + +## Impact on ABI + +This is purely compile-time sugar for the existing type. It is resolved at compile time and does not appear in the ABI nor rely on any version of the runtime. + +## Future Directions + +### Repeated value equivalent + +Analogous to arrays, there is an equivalent *value* sugar for literals of a specific size: + +```swift +// type inferred to be [5 of Int] +let fiveInts = [5 of 99] + +// type inferred to be [5 of [5 of Int]] +let fiveByFive = [5 of [5 of 99]] +``` + +Unlike the sugar for the type, this would also have applicability for existing types: + +```swift +// equivalent to .init(repeating: 99, count: 5) +let dynamic: [Int] = [5 of 99] +``` + +This is a much bigger design space, potentially requiring a new expressible-by-literal protocol and a way to map the literal to an initializer. As such, it is left for a future proposal. + +However, the choice of syntax for the type sugar has a significant impact on the viability of this future direction (see alternatives considered). Given the potential benefit of such a value syntax, any choice for the type sugar should consider its future extension to value sugar. + +[^expressions]: It could also cause confusion once expressions are allowed for declaring an `InlineArray` i.e. `[5 * 5 * Int]` would be allowed. + +### Flattened multi-dimensional arrays + +For multi-dimensional arrays, `[5 of [5 of Int]]` could be flattened to `[5 of 5 of Int]` without any additional parsing issues. This could be an alternative considered but is in future directions as it could also be introduced as sugar for the former case at a later date. + +## Alternatives Considered + +### Indication of the "inline" nature via the sugar. + +The naming of `InlineArray` incorporates important information about the nature of the type – that it includes its values inline rather than indirectly via a pointer. This name was chosen over other alternatives such as `FixedSizeArray` because the "inline-ness" was considered the more fundamental property, and so a better driver for the name. + +This has led to suggestions that this inline nature is important to include in the sugar was well. However, the current state privileges the position of `Array` as the only array type that is sugared. This implies that `Array` is the right choice in all circumstances, with inline arrays being a rare micro-optimization. This is not the case. + +For example, consider a translation of this code from the popular [Ray Tracing in One Weekend](https://raytracing.github.io/books/RayTracingInOneWeekend.html#thevec3class) tutorial: + +```cpp +class vec3 { + public: + double e[3]; + + double x() const { return e[0]; } + double y() const { return e[1]; } + double z() const { return e[2]; } + // etc +} +``` + +The way in which Swift privileges `[Double]` with sugar strongly implies you should use that in this translation. Doing so would have significant performance downsides: +- Creating new instances of `Vec3` requires a heap allocation, and destroying them require a free operation. +- `Vec3` could no longer be `BitwiseCopyable`, instead requiring a reference counting operation to make a copy. +- Access to a coordinate would require pointer chasing, and a contiguous array of `Vec3` objects would not be guaranteed to exist in contiguous memory. +- Every access to those coordinate accessors would need to check the bounds (because while the author might ensure that the value of `e` will only ever have length 3, the compiler cannot easily know this) and, in the case of mutation, a check for uniqueness of the pointer. `InlineArray<3, Double>` has none of these problems. + +Other examples include the use of nested `Array` types i.e. using `[[Double]]` to represent a matrix of known size, which would also have noticeable negative performance impact depending on the use case, compared to using an `InlineArray>` (or perhaps `[InlineArray]`) to model the same values. Today, this is possible via custom subscripts that logically represent the inner array within a single `[Double]`, but the introduction of `InlineArray` introduces other potentially more ergonomic options. + +In other cases, the "copy on write" nature of `Array` can mislead users into thinking that copies are risk-free, when actually copying an array can lead to "defeating" copy on write in subtle ways that can cause difficult-to-hunt-down performance issues. In all these cases, you need to pick the right one of two options for the performance goals you are trying to achieve. + +Swift's choice (deviating from many of its peers) to only have a dynamic array type, and to emphasize the utility of this type through sugar, has led to a shaping of the culture of writing Swift code that favors the sugared dynamic array even when this leads to otherwise-avoidable negative performance impact. This is not intended to make the case that Swift should _not_ have this sugar. Dynamic arrays are widely useful and Swift's readability goals are improved by Swift having a concise syntax for creating them. But writing performant Swift code inevitably involves having an understanding of the underlying performance characteristics of _all_ the types you are using, and syntax or type naming alone cannot solve this. + +Of course, all this only matters when you are trying to write code that maximizes performance. But that is a really important use case for Swift. The goal for Swift is a language that is as safe and enjoyable to write as many high-level non-performant languages, but also can achieve peak performance when that is your goal. And the idea is that when you are targeting that level of performance, you don't have to go into "ugly, no longer nice swift" mode to do it, with nice sugared `[Double]` replaced with less pleasant full type name of `InlineArray` – something a user coming from Go or C++ or Rust might find a downgrade. Similarly, attempting to incorporate the word "inline" into the sugar e.g. `[5 inline Int]` creates a worst of both worlds solution that many would find offputting to use, without solving the fundamental issue. + +For these reasons, we should be considering `InlineArray` a peer of `Array` (even if the need for it is less common – just not "niche"), and providing a pleasant to use sugar for both types. + +### Choice of delimiter + +The most obvious alternative here is the choice of separator. Other options include: + +- `[5 by Int]` is similar to `of`, but is less applicable to the value syntax (`[5 by 5]` doesn't read as an array of 5 instances of 5), without being clearer for the type syntax. +- `[5 x Int]`, using the ascii letter `x`, as an approximation for multiplication, reflecting common uses such as "a 4x4 vehicle". This was the choice of a previous revision of this proposal. +- `[5 * Int]`, using the standard ASCII symbol for multiplication. +- `[5 ⨉ Int]`, the Unicode n-ary times operator. This looks nice but is impractical as not keyboard-accessible. +- `[5; Int]` is what Rust uses, but appears to have little association with "times" or "many". Similarly other arbitrary punctuation e.g. `,` or `/`. `:` is of course ruled out as it is used for dictionary literals. +- `#` does have an association with counts in some areas such as set theory, but is used as a prefix operator rather than infix i.e. `[#5 Int]`. This is less expected than the infix form, and could also be read as "the fifth `Int`". It is also unclear how this would work with expressions like an array of size `5*5`. +- No delimiter at all i.e. `[5 Int]`. While this might be made to parse, the lack of any separator is found unsettling by some users and is less visually clear, especially once expressions are allowed instead of the `5`. + +Note that `*` is an existing operator, and may lead to ambiguity in future when expressions can be used to determine the size: `[5 * N * Int]`. `of` is clearer in this case: `[5 * N of Int]`. It also avoids parsing ambiguity, as the grammar does not allow two identifiers in succession. This becomes more important if the future direction of a value equivalent is pursued. `[2 * 2 * 2]` could be interpreted as `[2, 2, 2, 2]`, `[4, 4,]`, or `[8]`. + +Since `of` cannot follow another identifier today, `[of of Int]` is unambiguous,[^type] but would clearly be hard to read. This is likely a hypothetical concern rather than a practical one since `of` is very rare as a variable name. The previous proposal's use of `x` involved a variable name that was more common. + +`x` is also less clear when used for the value version: `[5 x 5]` can be parsed unambiguously, but looks similar to five times five, and so visually has the same challenges as with the `*` operator, even if this isn't a problem for the compiler. + +[^type]: or even `[of of of]`, since `of` can be a type name, albeit one that defies Swift's naming conventions. + +Another thing to consider is how that separator looks in the fully inferred version, which tend to start to look a little like ascii diagrams: + +``` +[_ of _] +[_ x _] +[_ * _] +[_; _] +``` + +Of all these, the `of` choice is less susceptible to the ascii art problem. + +### Order of size and type + +The order of size first, then type is determined by the ordering of the unsugared type, and deviating from this for the sugared version is not an option. + +### Whitespace around the delimeter + +In theory, when using integer literals or `_` the whitespace could be omitted (`[5x_]` is unambiguously `[5 x _]`). However, special casing allowing whitespace omission is not desirable. + +### Choice of brackets + +`InlineArray` has a lot in common with tuples – especially in sharing "copy on copy" behavior, unlike regular `Array`. So `(5 of Int)` may be an appropriate alternative to the square brackets, echoing this similarity. However, tuples and `InlineArray`s remain very different types, and it could be misleading to imply that `InlineArray` is "just" a tuple. + +Beyond varying the separator, there may be other dramatically different syntax that moves further from the "like Array sugar, but with a size argument". For example, dropping the brackets altogether (i.e. `let a: 5 of Int`). However, these are probably too much of a departure from the current Swift idioms, so likely to cause further confusion without any real upside. diff --git a/proposals/0484-allow-additional-args-to-dynamicmemberlookup-subscripts.md b/proposals/0484-allow-additional-args-to-dynamicmemberlookup-subscripts.md new file mode 100644 index 0000000000..744cebb58b --- /dev/null +++ b/proposals/0484-allow-additional-args-to-dynamicmemberlookup-subscripts.md @@ -0,0 +1,281 @@ +# Allow Additional Arguments to `@dynamicMemberLookup` Subscripts + +* Proposal: [SE-0484](0484-allow-additional-args-to-dynamicmemberlookup-subscripts.md) +* Authors: [Itai Ferber](https://github.com/itaiferber) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Accepted** +* Implementation: [swiftlang/swift#81148](https://github.com/swiftlang/swift/pull/81148) +* Previous Proposals: [SE-0195](0195-dynamic-member-lookup.md), [SE-0252](0252-keypath-dynamic-member-lookup.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-allow-additional-arguments-to-dynamicmemberlookup-subscripts/79558)) ([review](https://forums.swift.org/t/se-0484-allow-additional-arguments-to-dynamicmemberlookup-subscripts/79853)) ([acceptance](https://forums.swift.org/t/accepted-se-0484-allow-additional-arguments-to-dynamicmemberlookup-subscripts/80167)) + +## Introduction + +SE-0195 and SE-0252 introduced and refined `@dynamicMemberLookup` to provide type-safe "dot"-syntax access to arbitrary members of a type by reflecting the existence of certain `subscript(dynamicMember:)` methods on that type, turning + +```swift +let _ = x.member +x.member = 42 +ƒ(&x.member) +``` + +into + +```swift +let _ = x[dynamicMember: ] +x[dynamicMember: ] = 42 +ƒ(&x[dynamicMember: ]) +``` + +when `x.member` doesn't otherwise exist statically. Currently, in order to be eligible to satisfy `@dynamicMemberLookup` requirements, a subscript must: + +1. Take _exactly one_ argument with an explicit `dynamicMember` argument label, +2. Whose type is non-variadic and is either + * A `{{Reference}Writable}KeyPath`, or + * A concrete type conforming to `ExpressibleByStringLiteral` + +This proposal intends to relax the "exactly one" requirement above to allow eligible subscripts to take additional arguments after `dynamicMember` as long as they have a default value (or are variadic, and thus have an implicit default value). + +## Motivation + +Dynamic member lookup is often used to provide expressive and succinct API in wrapping some underlying data, be it a type-erased foreign language object (e.g., a Python `PyVal` or a JavaScript `JSValue`) or a native Swift type. This (and [`callAsFunction()`](0253-callable.md)) allow a generalized API interface such as + +```swift +struct Value { + subscript(_ property: String) -> Value { + get { ... } + set { ... } + } + + func invoke(_ method: String, _ args: Any...) -> Value { + ... + } +} + +let x: Value = ... +let _ = x["member"] +x["member"] = Value(42) +x.invoke("someMethod", 1, 2, 3) +``` + +to be expressed much more naturally: + +```swift +@dynamicMemberLookup +struct Value { + struct Method { + func callAsFunction(_ args: Any...) -> Value { ... } + } + + subscript(dynamicMember property: String) -> Value { + get { ... } + set { ... } + } + + subscript(dynamicMember method: String) -> Method { ... } +} + +let x: Value = ... +let _ = x.member +x.member = Value(42) +x.someMethod(1, 2, 3) +``` + +However, as wrappers for underlying data, sometimes interfaces like this need to be able to "thread through" additional information. For example, it might be helpful to provide information about call sites for debugging purposes: + +```swift +struct Value { + subscript( + _ property: String, + function: StaticString = #function, + file: StaticString = #fileID, + line: UInt = #line + ) -> Value { + ... + } + + func invokeMethod( + _ method: String, + function: StaticString = #function, + file: StaticString = #fileID, + line: UInt = #line, + _ args: Any... + ) -> Value { + ... + } +} +``` + +When additional arguments like this have default values, they don't affect the appearance of call sites at all: + +```swift +let x: Value = ... +let _ = x["member"] +x["member"] = Value(42) +x.invoke("someMethod", 1, 2, 3) +``` + +However, these are not valid for use with dynamic member lookup subscripts, since the additional arguments prevent subscripts from being eligible for dynamic member lookup: + +```swift +@dynamicMemberLookup // error: @dynamicMemberLookupAttribute requires 'Value' to have a 'subscript(dynamicMember:)' method that accepts either 'ExpressibleByStringLiteral' or a key path +struct Value { + subscript( + dynamicMember property: String, + function: StaticString = #function, + file: StaticString = #fileID, + line: UInt = #line + ) -> Value { + ... + } + + subscript( + dynamicMember method: String, + function: StaticString = #function, + file: StaticString = #fileID, + line: UInt = #line + ) -> Method { + ... + } +} +``` + +## Proposed solution + +We can amend the rules for such subscripts to make them eligible. With this proposal, in order to be eligible to satisfy `@dynamicMemberLookup` requirements, a subscript must: + +1. Take an initial argument with an explicit `dynamicMember` argument label, +2. Whose parameter type is non-variadic and is either: + * A `{{Reference}Writable}KeyPath`, or + * A concrete type conforming to `ExpressibleByStringLiteral`, +3. And whose following arguments (if any) are all either variadic or have a default value + +## Detailed design + +Since compiler support for dynamic member lookup is already robust, implementing this requires primarily: + +1. Type-checking of `@dynamicMemberLookup`-annotated declarations to also consider `subscript(dynamicMember:...)` methods following the above rules as valid, and +2. Syntactic transformation of `T.` to `T[dynamicMember:...]` in the constraint system to fill in default arguments expressions for any following arguments + +## Source compatibility + +This is largely an additive change with minimal impact to source compatibility. Types which do not opt in to `@dynamicMemberLookup` are unaffected, as are types which do opt in and only offer `subscript(dynamicMember:)` methods which take a single argument. + +However, types which opt in to `@dynamicMemberLookup` and currently offer an overload of `subscript(dynamicMember:...)`—which today is not eligible for consideration for dynamic member lookup—_may_ now select this overload when they wouldn't have before. + +### Overload resolution + +Dynamic member lookups go through regular overload resolution, with an additional disambiguation rule that prefers keypath-based subscript overloads over string-based ones. Since the `dynamicMember` argument to dynamic member subscripts is implicit, overloads of `subscript(dynamicMember:)` are primarily selected based on their return type (and typically for keypath-based subscripts, how that return type is used in forming the type of a keypath parameter). + +With this proposal, all arguments to `subscript(dynamicMember:...)` are still implicit, so overloads are still primarily selected based on return type, with the additional disambiguation rule that prefers overloads with fewer arguments over overloads with more arguments. (This rule applies "for free" since it already applies to method calls, which dynamic member lookups are transformed into.) + +This means that if a type today offers a valid `subscript(dynamicMember:) -> T` and a (currently-unconsidered) `subscript(dynamicMember:...) -> U`, + +1. If `T == U` then the former will still be the preferred overload in all circumstances +2. If `T` and `U` are compatible (and equally-specific) at a callsite then the former will still be the preferred overload +3. If `T` and `U` are incompatible, or if one is more specific than the other, then the more specific type will be preferred + +For example: + +```swift +@dynamicMemberLookup +struct A { + /* (1) */ subscript(dynamicMember member: String) -> String { ... } + /* (2) */ subscript(dynamicMember member: String, _: StaticString = #function) -> String { ... } +} + +@dynamicMemberLookup +struct B { + /* (3) */ subscript(dynamicMember member: String) -> String { ... } + /* (4) */ subscript(dynamicMember member: String, _: StaticString = #function) -> Int { ... } +} + +@dynamicMemberLookup +struct C { + /* (5) */ subscript(dynamicMember member: String) -> String { ... } + /* (6) */ subscript(dynamicMember member: String, _: StaticString = #function) -> String? { ... } +} + +// T == U +let _ = A().member // (1) preferred over (2); no ambiguity +let _: String = A().member // (1) preferred over (2); no ambiguity + +// T and U are compatible +let _: Any = A().member // (1) preferred over (2); no ambiguity +let _: Any = B().member // (3) preferred over (4); no ambiguity +let _: Any = C().member // (5) preferred over (6); no ambiguity + +// T and U are incompatible/differently-specific +let _: String = B().member // (3) +let _: Int = B().member // (4);️ would not previously compile +let _: String = C().member // (5); no ambiguity +let _: String? = C().member // (6) preferred over (5); ⚠️ previously (5) ⚠️ +``` + +This last case is the only source of behavior change: (6) was previously not considered a valid candidate, but has a return type more specific than (5), and is now picked at a callsite. + +In practice, it is expected that this situation is exceedingly rare. + +## ABI compatibility + +This feature is implemented entirely in the compiler as a syntactic transformation and has no impact on the ABI. + +## Implications on adoption + +The changes in this proposal require the adoption of a new version of the Swift compiler. + +## Alternatives considered + +The main alternative to this proposal is to not implement it, as: +1. It was noted in [the pitch thread](https://forums.swift.org/t/pitch-allow-additional-arguments-to-dynamicmemberlookup-subscripts/79558) that allowing additional arguments to dynamic member lookup widens the gap in capabilities between dynamic members and regular members — dynamic members would be able to + + 1. Have caller side effects (i.e., have access to `#function`, `#file`, `#line`, etc.), + 2. Constrain themselves via generics, and + 3. Apply isolation to themselves via `#isolation` + + where regular members cannot. However, (i) and (iii) are not considered an imbalance in functionality but instead are the raison d'être of this proposal. (ii) is also already possible today as dynamic member subscripts can be constrained via generics (and this is often used with keypath-based lookup). +2. This is possible to work around using explicit methods such as `get()` and `set(_:)`: + + ```swift + @dynamicMemberLookup + struct Value { + struct Property { + func get( + function: StaticString = #function, + file: StaticString = #file, + line: UInt = #line + ) -> Value { + ... + } + + func set( + _ value: Value, + function: StaticString = #function, + file: StaticString = #file, + line: UInt = #line + ) { + ... + } + } + + subscript(dynamicMember member: String) -> Property { ... } + } + + let x: Value = ... + let _ = x.member.get() // x.member + x.member.set(Value(42)) // x.member = Value(42) + ``` + + However, this feels non-idiomatic, and for long chains of getters and setters, can become cumbersome: + + ```swift + let x: Value = ... + let _ = x.member.get().inner.get().nested.get() // x.member.inner.nested + x.member.get().inner.get().nested.set(Value(42)) // x.member.inner.nested = Value(42) + ``` + +### Source compatibility + +It is possible to avoid the risk of the behavior change noted above by adjusting the constraint system to always prefer `subscript(dynamicMember:) -> T` overloads over `subscript(dynamicMember:...) -> U` overloads (if `T` and `U` are compatible), even if `U` is more specific than `T`. However, + +1. This would be a departure from the normal method overload resolution behavior that Swift developers are familiar with, and +2. If `T` were a supertype of `U`, it would be impossible to ever call the more specific overload except by direct subscript access diff --git a/proposals/0485-outputspan.md b/proposals/0485-outputspan.md new file mode 100644 index 0000000000..76f467f6fa --- /dev/null +++ b/proposals/0485-outputspan.md @@ -0,0 +1,861 @@ +# OutputSpan: delegate initialization of contiguous memory + +* Proposal: [SE-0485](0485-outputspan.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Doug Gregor](https://github.com/DougGregor) +* Status: **Implemented (Swift 6.2)** ([Extensions to standard library types](#extensions) pending) +* Roadmap: [BufferView Roadmap](https://forums.swift.org/t/66211) +* Implementation: [swiftlang/swift#81637](https://github.com/swiftlang/swift/pull/81637) +* Review: [Pitch](https://forums.swift.org/t/pitch-outputspan/79473), [Review](https://forums.swift.org/t/se-0485-outputspan-delegate-initialization-of-contiguous-memory/80032), [Acceptance](https://forums.swift.org/t/accepted-with-modifications-se-0485-outputspan-delegate-initialization-of-contiguous-memory/80435) + +[SE-0446]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md +[SE-0447]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md +[SE-0456]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0456-stdlib-span-properties.md +[SE-0467]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0467-MutableSpan.md +[PR-LifetimeAnnotations]: https://github.com/swiftlang/swift-evolution/pull/2750 +[Forum-LifetimeAnnotations]: https://forums.swift.org/t/78638 +[SE-0453]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0453-vector.md + + +#### Table of Contents + +- [Introduction](#introduction) + +- [Motivation](#motivation) + +- [Proposed solution](#proposed-solution) + +- [Detailed Design](#design) + +- [Source compatibility](#source-compatibility) + +- [ABI compatibility](#abi-compatibility) + +- [Implications on adoption](#implications-on-adoption) + +- [Alternatives Considered](#alternatives-considered) + +- [Future directions](#future-directions) + +- [Acknowledgements](#acknowledgements) + + +## Introduction + +Following the introduction of [`Span`][SE-0447] and [`MutableSpan`][SE-0467], this proposal adds a general facility for initialization of exclusively-borrowed memory with the `OutputSpan` and `OutputRawSpan` types. The memory represented by `OutputSpan` consists of a number of initialized elements, followed by uninitialized memory. The operations of `OutputSpan` can change the number of initialized elements in memory, unlike `MutableSpan` which always represent initialized memory representing a fixed number of elements. + +## Motivation + +Some standard library container types can delegate initialization of some or all of their storage to user code. Up to now, it has only been possible to do so with explicitly unsafe functions, which have also proven error-prone. The standard library provides this unsafe functionality with the closure-taking initializers `Array.init(unsafeUninitializedCapacity:initializingWith:)` and `String.init(unsafeUninitializedCapacity:initializingUTF8With:)`. + +These functions have a few different drawbacks, most prominently their reliance on unsafe types, which makes them unpalatable in security-conscious environments. We continue addressing these issues with `OutputSpan` and `OutputRawSpan`, new non-copyable and non-escapable types that manage initialization of typed and untyped memory. + +In addition to the new types, we propose adding new API for some standard library types to take advantage of `OutputSpan` and `OutputRawSpan`. + +## Proposed solution + +#### OutputSpan + +`OutputSpan` allows delegating the initialization of a type's memory, by providing access to an exclusively-borrowed view of a range of contiguous memory. `OutputSpan`'s contiguous memory always consists of a prefix of initialized memory, followed by a suffix of uninitialized memory. `OutputSpan`'s operations manage the initialization state in order to preserve that invariant. The common usage pattern we expect to see for `OutputSpan` consists of passing it as an `inout` parameter to a function, allowing the function to produce an output by writing into a previously uninitialized region. + +Like `MutableSpan`, `OutputSpan` relies on two guarantees: (a) that it has exclusive access to the range of memory it represents, and (b) that the memory locations it represents will remain valid for the duration of the access. These guarantee data race safety and lifetime safety. `OutputSpan` performs bounds-checking on every access to preserve bounds safety. `OutputSpan` manages the initialization state of the memory in represents on behalf of the memory's owner. + +#### OutputRawSpan + +`OutputRawSpan` allows delegating the initialization of heterogeneously-typed memory, such as memory being prepared by an encoder. It makes the same safety guarantees as `OutputSpan`, but manages untyped memory. + +#### Extensions to standard library types + +The standard library will provide new container initializers that delegate to an `OutputSpan`. Delegated initialization generally requires a container to perform some operations after the initialization has happened. In the case of `Array` this is simply noting the number of initialized elements; in the case of `String` this consists of validating the input, then noting metadata about the input. This post-processing implies the need for a scope, and we believe that scope is best represented by a closure. The `Array` initializer will be as follows: + +```swift +extension Array { + public init( + capacity: Int, + initializingWith: (_ span: inout OutputSpan) throws(E) -> Void + ) throws(E) +} +``` + +We will also extend `String`, `UnicodeScalarView` and `InlineArray` with similar initializers, and add append-in-place operations where appropriate. + +#### `@_lifetime` attribute + +Some of the API presented here must establish a lifetime relationship between a non-escapable returned value and a callee binding. This relationship will be illustrated using the `@_lifetime` attribute recently [pitched][PR-LifetimeAnnotations] and [formalized][Forum-LifetimeAnnotations]. For the purposes of this proposal, the lifetime attribute ties the lifetime of a function's return value to one of its input parameters. + +Note: The eventual lifetime annotations proposal may adopt a syntax different than the syntax used here. We expect that the Standard Library will be modified to adopt an updated lifetime dependency syntax as soon as it is finalized. + +## Detailed Design + +#### OutputSpan + +`OutputSpan` is a simple representation of a partially-initialized region of memory. It is non-copyable in order to enforce exclusive access during mutations of its memory, as required by the law of exclusivity: + +````swift +@frozen +public struct OutputSpan: ~Copyable, ~Escapable { + internal let _start: UnsafeMutableRawPointer? + public let capacity: Int + internal var _count: Int +} +```` + +The memory represented by an `OutputSpan` instance consists of `count` initialized instances of `Element`, followed by uninitialized memory with storage space for `capacity - count` additional elements of `Element`. + +```swift +extension OutputSpan where Element: ~Copyable { + /// The number of initialized elements in this `OutputSpan`. + public var count: Int { get } + + /// A Boolean value indicating whether the span is empty. + public var isEmpty: Bool { get } + + /// A Boolean value indicating whether the span is full. + public var isFull: Bool { get } + + /// The number of additional elements that can be added to this `OutputSpan` + public var freeCapacity: Int { get } // capacity - count +} +``` + +##### Single-element operations + +The basic operation supported by `OutputSpan` is appending an element. When an element is appended, the correct amount of memory needed to represent it is initialized, and the `count` property is incremented by 1. If the `OutputSpan` has no available space (`capacity == count`), this operation traps. +```swift +extension OutputSpan where Element: ~Copyable { + /// Append a single element to this `OutputSpan`. + @_lifetime(self: copy self) + public mutating func append(_ value: consuming Element) +} + +extension OutputSpan { + /// Repeatedly append an element to this `OutputSpan`. + @_lifetime(self: copy self) + public mutating func append(repeating repeatedValue: Element, count: Int) +} +``` +The converse operation `removeLast()` is also supported, and returns the removed element if `count` was greater than zero. +```swift +extension OutputSpan where Element: ~Copyable { + /// Remove the last initialized element from this `OutputSpan`. + /// + /// Returns the last element. The `OutputSpan` must not be empty. + @discardableResult + @_lifetime(self: copy self) + public mutating func removeLast() -> Element +} +``` + +##### Bulk removals from an `OutputSpan`'s memory: + +Bulk operations to deinitialize some or all of an `OutputSpan`'s memory are also available: +```swift +extension OutputSpan where Element: ~Copyable { + /// Remove the last N elements, returning the memory they occupy + /// to the uninitialized state. + /// + /// `n` must not be greater than `count` + @_lifetime(self: copy self) + public mutating func removeLast(_ n: Int) + + /// Remove all this span's elements and return its memory to the uninitialized state. + @_lifetime(self: copy self) + public mutating func removeAll() +} +``` + +##### Accessing an `OutputSpan`'s initialized memory: + +The initialized elements are accessible for read-only or mutating access via the `span` and `mutableSpan` properties: + +```swift +extension OutputSpan where Element: ~Copyable { + /// Borrow the underlying initialized memory for read-only access. + public var span: Span { + @_lifetime(borrow self) borrowing get + } + + /// Exclusively borrow the underlying initialized memory for mutation. + public mutating var mutableSpan: MutableSpan { + @_lifetime(&self) mutating get + } +} +``` + +`OutputSpan` also provides the ability to access its individual initialized elements by index: +```swift +extension OutputSpan where Element: ~Copyable { + /// The type that represents an initialized position in an `OutputSpan`. + typealias Index = Int + + /// The range of initialized positions for this `OutputSpan`. + var indices: Range { get } + + /// Accesses the element at the specified initialized position. + subscript(_ index: Index) -> Element { borrow; mutate } + // accessor syntax from accessors roadmap (https://forums.swift.org/t/76707) + + /// Exchange the elements at the two given offsets + mutating func swapAt(_ i: Index, _ j: Index) +} +``` + +##### Interoperability with unsafe code + +We provide a method to process or populate an `OutputSpan` using unsafe operations, which can also be used for out-of-order initialization. + +```swift +extension OutputSpan where Element: ~Copyable { + /// Call the given closure with the unsafe buffer pointer addressed by this + /// OutputSpan and a mutable reference to its count of initialized elements. + /// + /// This method provides a way to process or populate an `OutputSpan` using + /// unsafe operations, such as dispatching to code written in legacy + /// (memory-unsafe) languages. + /// + /// The supplied closure may process the buffer in any way it wants; however, + /// when it finishes (whether by returning or throwing), it must leave the + /// buffer in a state that satisfies the invariants of the output span: + /// + /// 1. The inout integer passed in as the second argument must be the exact + /// number of initialized items in the buffer passed in as the first + /// argument. + /// 2. These initialized elements must be located in a single contiguous + /// region starting at the beginning of the buffer. The rest of the buffer + /// must hold uninitialized memory. + /// + /// This function cannot verify these two invariants, and therefore + /// this is an unsafe operation. Violating the invariants of `OutputSpan` + /// may result in undefined behavior. + @_lifetime(self: copy self) + public mutating func withUnsafeMutableBufferPointer( + _ body: ( + UnsafeMutableBufferPointer, + _ initializedCount: inout Int + ) throws(E) -> R + ) throws(E) -> R +} +``` + +##### Creating an `OutputSpan` instance: + +Creating an `OutputSpan` is an unsafe operation. It requires having knowledge of the initialization state of the range of memory being targeted. The range of memory must be in two regions: the first region contains initialized instances of `Element`, and the second region is uninitialized. The number of initialized instances is passed to the `OutputSpan` initializer through its `initializedCount` argument. + +```swift +extension OutputSpan where Element: ~Copyable { + /// Unsafely create an OutputSpan over partly-initialized memory. + /// + /// The memory in `buffer` must remain valid throughout the lifetime + /// of the newly-created `OutputSpan`. Its prefix must contain + /// `initializedCount` initialized instances, followed by uninitialized + /// memory. + /// + /// - Parameters: + /// - buffer: an `UnsafeMutableBufferPointer` to be initialized + /// - initializedCount: the number of initialized elements + /// at the beginning of `buffer`. + @unsafe + @_lifetime(borrow buffer) + public init( + buffer: UnsafeMutableBufferPointer, + initializedCount: Int + ) +} + +extension OutputSpan { + /// Unsafely create an OutputSpan over partly-initialized memory. + /// + /// The memory in `buffer` must remain valid throughout the lifetime + /// of the newly-created `OutputSpan`. Its prefix must contain + /// `initializedCount` initialized instances, followed by uninitialized + /// memory. + /// + /// - Parameters: + /// - buffer: an `UnsafeMutableBufferPointer` to be initialized + /// - initializedCount: the number of initialized elements + /// at the beginning of `buffer`. + @unsafe + @_lifetime(borrow buffer) + public init( + buffer: borrowing Slice>, + initializedCount: Int + ) +} +``` + +We also provide a default (no-parameter) initializer to create an empty, zero-capacity `OutputSpan`. Such an initializer is useful in order to be able to materialize an empty span for the `nil` case of an Optional, for example, or to exchange with another span in a mutable struct. + +```swift +extension OutputSpan where Element: ~Copyable { + /// Create an OutputSpan with zero capacity + @_lifetime(immortal) + public init() +} +``` + +Such an empty `OutputSpan` does not depend on a memory allocation, and therefore has the longest lifetime possible :`immortal`. This capability is important enough that we also propose to immediately define similar empty initializers for `Span`, `RawSpan`, `MutableSpan` and `MutableRawSpan`. + +##### Retrieving initialized memory from an `OutputSpan` + +Once memory has been initialized using `OutputSpan`, the owner of the memory must consume the `OutputSpan` in order to retake ownership of the initialized memory. The owning type must pass the memory used to initialize the `OutputSpan` to the `finalize(for:)` function. Passing the wrong buffer is a programmer error and the function traps; this requirement also ensures that user code does not wrongly replace the `OutputSpan` with an unrelated instance. The `finalize(for:)` function consumes the `OutputSpan` instance and returns the number of initialized elements. If `finalize(for:)` is not called, the initialized portion of `OutputSpan`'s memory will be deinitialized when the binding goes out of scope. + +```swift +extension OutputSpan where Element: ~Copyable { + /// Consume the OutputSpan and return the number of initialized elements. + /// + /// Parameters: + /// - buffer: The buffer being finalized. This must be the same buffer as used + /// to initialize the `OutputSpan` instance. + /// Returns: The number of elements that were initialized. + @unsafe + public consuming func finalize( + for buffer: UnsafeMutableBufferPointer + ) -> Int +} + +extension OutputSpan { + /// Consume the OutputSpan and return the number of initialized elements. + /// + /// Parameters: + /// - buffer: The buffer being finalized. This must be the same buffer as used + /// to initialize the `OutputSpan` instance. + /// Returns: The number of bytes that were initialized. + @unsafe + public consuming func finalize( + for buffer: Slice> + ) -> Int +} +``` + + +#### `OutputRawSpan` +`OutputRawSpan` is similar to `OutputSpan`, but its initialized memory is untyped. Its API supports appending the bytes of instances of `BitwiseCopyable` types, as well as a variety of bulk initialization operations. + +```swift +@frozen +public struct OutputRawSpan: ~Copyable, ~Escapable { + internal var _start: UnsafeMutableRawPointer? + public let capacity: Int + internal var _count: Int +} +``` + +The memory represented by an `OutputRawSpan` contains `byteCount` initialized bytes, followed by uninitialized memory. + +```swift +extension OutputRawSpan { + /// The number of initialized bytes in this `OutputRawSpan` + public var byteCount: Int { get } + + /// A Boolean value indicating whither the span is empty. + public var isEmpty: Bool { get } + + /// A Boolean value indicating whither the span is full. + public var isFull: Bool { get } + + /// The number of uninitialized bytes remaining in this `OutputRawSpan` + public var freeCapacity: Int { get } // capacity - byteCount +} +``` + + +##### Appending to `OutputRawSpan` + +The basic operation is to append the bytes of some value to an `OutputRawSpan`. Note that since the fundamental operation is appending bytes, `OutputRawSpan` does not concern itself with memory alignment. +```swift +extension OutputRawSpan { + /// Append a single byte to this span + @_lifetime(self: copy self) + public mutating func append(_ value: UInt8) + + /// Appends the given value's bytes to this span's initialized bytes + @_lifetime(self: copy self) + public mutating func append( + _ value: T, as type: T.Type + ) + + /// Appends the given value's bytes repeatedly to this span's initialized bytes + @_lifetime(self: copy self) + public mutating func append( + repeating repeatedValue: T, count: Int, as type: T.Type + ) +} +``` + +An `OutputRawSpan`'s initialized memory is accessible for read-only or mutating access via the `bytes` and `mutableBytes` properties: + +```swift +extension OutputRawSpan { + /// Borrow the underlying initialized memory for read-only access. + public var bytes: RawSpan { + @_lifetime(borrow self) borrowing get + } + + /// Exclusively borrow the underlying initialized memory for mutation. + public var mutableBytes: MutableRawSpan { + @_lifetime(&self) mutating get + } +} +``` + +Deinitializing memory from an `OutputRawSpan`: + +```swift +extension OutputRawSpan { + + /// Remove the last byte from this span + @_lifetime(self: copy self) + public mutating func removeLast() -> UInt8 { + + /// Remove the last N elements, returning the memory they occupy + /// to the uninitialized state. + /// + /// `n` must not be greater than `count` + @_lifetime(self: copy self) + public mutating func removeLast(_ n: Int) + + /// Remove all this span's elements and return its memory + /// to the uninitialized state. + @_lifetime(self: copy self) + public mutating func removeAll() +} +``` + +##### Interoperability with unsafe code + +We provide a method to process or populate an `OutputRawSpan` using unsafe operations, which can also be used for out-of-order initialization. + +```swift +extension OutputRawSpan { + /// Call the given closure with the unsafe buffer pointer addressed by this + /// OutputRawSpan and a mutable reference to its count of initialized bytes. + /// + /// This method provides a way to process or populate an `OutputRawSpan` using + /// unsafe operations, such as dispatching to code written in legacy + /// (memory-unsafe) languages. + /// + /// The supplied closure may process the buffer in any way it wants; however, + /// when it finishes (whether by returning or throwing), it must leave the + /// buffer in a state that satisfies the invariants of the output span: + /// + /// 1. The inout integer passed in as the second argument must be the exact + /// number of initialized bytes in the buffer passed in as the first + /// argument. + /// 2. These initialized elements must be located in a single contiguous + /// region starting at the beginning of the buffer. The rest of the buffer + /// must hold uninitialized memory. + /// + /// This function cannot verify these two invariants, and therefore + /// this is an unsafe operation. Violating the invariants of `OutputRawSpan` + /// may result in undefined behavior. + @_lifetime(self: copy self) + public mutating func withUnsafeMutableBytes( + _ body: ( + UnsafeMutableRawBufferPointer, + _ initializedCount: inout Int + ) throws(E) -> R + ) throws(E) -> R +} +``` + +##### Creating `OutputRawSpan` instances + +Creating an `OutputRawSpan` is an unsafe operation. It requires having knowledge of the initialization state of the range of memory being targeted. The range of memory must be in two regions: the first region contains initialized bytes, and the second region is uninitialized. The number of initialized bytes is passed to the `OutputRawSpan` initializer through its `initializedCount` argument. + +```swift +extension OutputRawSpan { + + /// Unsafely create an OutputRawSpan over partly-initialized memory. + /// + /// The memory in `buffer` must remain valid throughout the lifetime + /// of the newly-created `OutputRawSpan`. Its prefix must contain + /// `initializedCount` initialized bytes, followed by uninitialized + /// memory. + /// + /// - Parameters: + /// - buffer: an `UnsafeMutableBufferPointer` to be initialized + /// - initializedCount: the number of initialized elements + /// at the beginning of `buffer`. + @unsafe + @_lifetime(borrow buffer) + public init( + buffer: UnsafeMutableRawBufferPointer, + initializedCount: Int + ) + + /// Create an OutputRawSpan with zero capacity + @_lifetime(immortal) + public init() + + /// Unsafely create an OutputRawSpan over partly-initialized memory. + /// + /// The memory in `buffer` must remain valid throughout the lifetime + /// of the newly-created `OutputRawSpan`. Its prefix must contain + /// `initializedCount` initialized bytes, followed by uninitialized + /// memory. + /// + /// - Parameters: + /// - buffer: an `UnsafeMutableBufferPointer` to be initialized + /// - initializedCount: the number of initialized elements + /// at the beginning of `buffer`. + @unsafe + @_lifetime(borrow buffer) + public init( + buffer: Slice, + initializedCount: Int + ) +} +``` + +##### Retrieving initialized memory from an `OutputRawSpan` + +Once memory has been initialized using `OutputRawSpan`, the owner of the memory must consume the instance in order to retake ownership of the initialized memory. The owning type must pass the memory used to initialize the `OutputRawSpan` to the `finalize(for:)` function. Passing the wrong buffer is a programmer error and the function traps. `finalize()` consumes the `OutputRawSpan` instance and returns the number of initialized bytes. + +```swift +extension OutputRawSpan { + /// Consume the OutputRawSpan and return the number of initialized bytes. + /// + /// Parameters: + /// - buffer: The buffer being finalized. This must be the same buffer as used + /// to create the `OutputRawSpan` instance. + /// Returns: The number of initialized bytes. + @unsafe + public consuming func finalize( + for buffer: UnsafeMutableRawBufferPointer + ) -> Int + + /// Consume the OutputRawSpan and return the number of initialized bytes. + /// + /// Parameters: + /// - buffer: The buffer being finalized. This must be the same buffer as used + /// to create the `OutputRawSpan` instance. + /// Returns: The number of initialized bytes. + @unsafe + public consuming func finalize( + for buffer: Slice + ) -> Int +} +``` + + +#### Extensions to Standard Library types + +The standard library and Foundation will add a few initializers that enable initialization in place, intermediated by an `OutputSpan` instance, passed as a parameter to a closure: + +```swift +extension Array { + /// Creates an array with the specified capacity, then calls the given + /// closure with an OutputSpan to initialize the array's contents. + public init( + capacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) + + /// Grows the array to ensure capacity for the specified number of elements, + /// then calls the closure with an OutputSpan covering the array's + /// uninitialized memory. + public mutating func append( + addingCapacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) +} + +extension ContiguousArray { + /// Creates an array with the specified capacity, then calls the given + /// closure with an OutputSpan to initialize the array's contents. + public init( + capacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) + + /// Grows the array to ensure capacity for the specified number of elements, + /// then calls the closure with an OutputSpan covering the array's + /// uninitialized memory. + public mutating func append( + addingCapacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) +} + +extension ArraySlice { + /// Grows the array to ensure capacity for the specified number of elements, + /// then calls the closure with an OutputSpan covering the array's + /// uninitialized memory. + public mutating func append( + addingCapacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) +} + +extension String { + /// Creates a new string with the specified capacity in UTF-8 code units, and + /// then calls the given closure with a OutputSpan to initialize the string's + /// contents. + /// + /// This initializer replaces ill-formed UTF-8 sequences with the Unicode + /// replacement character (`"\u{FFFD}"`). This may require resizing + /// the buffer beyond its original capacity. + public init( + repairingUTF8WithCapacity capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) + + /// Creates a new string with the specified capacity in UTF-8 code units, and + /// then calls the given closure with a OutputSpan to initialize the string's + /// contents. + /// + /// This initializer does not try to repair ill-formed code unit sequences. + /// If any are found, the result of the initializer is `nil`. + public init?( + validatingUTF8WithCapacity capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) + + /// Grows the string to ensure capacity for the specified number + /// of UTF-8 code units, then calls the closure with an OutputSpan covering + /// the string's uninitialized memory. + public mutating func append( + addingUTF8Capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) +} + +extension UnicodeScalarView { + /// Creates a new string with the specified capacity in UTF-8 code units, and + /// then calls the given closure with a OutputSpan to initialize + /// the string's contents. + /// + /// This initializer replaces ill-formed UTF-8 sequences with the Unicode + /// replacement character (`"\u{FFFD}"`). This may require resizing + /// the buffer beyond its original capacity. + public init( + repairingUTF8WithCapacity capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) + + /// Creates a new string with the specified capacity in UTF-8 code units, and + /// then calls the given closure with a OutputSpan to initialize + /// the string's contents. + /// + /// This initializer does not try to repair ill-formed code unit sequences. + /// If any are found, the result of the initializer is `nil`. + public init?( + validatingUTF8WithCapacity capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) + + /// Grows the string to ensure capacity for the specified number + /// of UTF-8 code units, then calls the closure with an OutputSpan covering + /// the string's uninitialized memory. + public mutating func append( + addingUTF8Capacity: Int, + initializingUTF8With initializer: ( + inout OutputSpan + ) throws(E) -> Void + ) throws(E) +} + +extension InlineArray { + /// Creates an array, then calls the given closure with an OutputSpan + /// to initialize the array's elements. + /// + /// NOTE: The closure must initialize every element of the `OutputSpan`. + /// If the closure does not do so, the initializer will trap. + public init( + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) +} +``` + +#### Extensions to `Foundation.Data` + +While the `swift-foundation` package and the `Foundation` framework are not governed by the Swift evolution process, `Data` is similar in use to standard library types, and the project acknowledges that it is desirable for it to have similar API when appropriate. Accordingly, we plan to propose the following additions to `Foundation.Data`: +```swift +extension Data { + /// Creates a data instance with the specified capacity, then calls + /// the given closure with an OutputSpan to initialize the instances's + /// contents. + public init( + capacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) + + /// Creates a data instance with the specified capacity, then calls + /// the given closure with an OutputSpan to initialize the instances's + /// contents. + public init( + rawCapacity: Int, + initializingWith initializer: (inout OutputRawSpan) throws(E) -> Void + ) throws(E) + + /// Ensures the data instance has enough capacity for the specified + /// number of bytes, then calls the closure with an OutputSpan covering + /// the uninitialized memory. + public mutating func append( + addingCapacity: Int, + initializingWith initializer: (inout OutputSpan) throws(E) -> Void + ) throws(E) + + /// Ensures the data instance has enough capacity for the specified + /// number of bytes, then calls the closure with an OutputSpan covering + /// the uninitialized memory. + public mutating func append( + addingRawCapacity: Int, + initializingWith initializer: (inout OutputRawSpan) throws(E) -> Void + ) throws(E) +} +``` + +#### Changes to `MutableSpan` and `MutableRawSpan` + +This proposal considers the naming of `OutputSpan`'s bulk-initialization methods, and elects to defer their implementation until we have more experience with the various kinds of container we need to support. We also introduced bulk updating functions to `MutableSpan` and `MutableRawSpan` in [SE-0467][SE-0467]. It is clear that they have the same kinds of parameters as `OutputSpan`'s bulk-initialization methods, but the discussion has taken a [different direction](#contentsOf) in the latter case. We would like both of these sets of operations to match. Accordingly, we will remove the bulk `update()` functions proposedin [SE-0467][SE-0467], to be replaced with a better naming scheme later. Prototype bulk-update functionality will also be added via a package in the meantime. + +## Source compatibility + +This proposal is additive and source-compatible with existing code. + +## ABI compatibility + +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption + +The additions described in this proposal require a new version of the Swift standard library and runtime. + + +## Alternatives Considered + +#### Vending `OutputSpan` as a property + +`OutputSpan` changes the number of initialized elements in a container (or collection), and this requires some operation to update the container after the `OutputSpan` is consumed. Let's call that update operation a "cleanup" operation. The cleanup operation needs to be scheduled in some way. We could associate the cleanup with the `deinit` of `OutputSpan`, or the `deinit` of a wrapper of `OutputSpan`. Neither of these seem appealing; the mechanisms would involve an arbitrary closure executed at `deinit` time, or having to write a full wrapper for each type that vends an `OutputSpan`. We could potentially schedule the cleanup operation as part of a coroutine accessor, but these are not productized yet. The pattern established by closure-taking API is well established, and that pattern fits the needs of `OutputSpan` well. + +#### Container construction pattern + +A constrained version of possible `OutputSpan` use consists of in-place container initialization. This proposal introduces a few initializers in this vein, such as `Array.init(capacity:initializingWith:)`, which rely on closure to establish a scope. A different approach would be to use intermediate types to perform such operations: + +```swift +struct ArrayConstructor: ~Copyable { + @_lifetime(&self) mutating var outputSpan: OutputSpan + private let _ArrayBuffer + + init(capacity: Int) +} + +extension Array { + init(_: consuming ArrayConstructor) +} +``` + +## Future directions + +#### Helpers to initialize memory in an arbitrary order + +Some applications may benefit from the ability to initialize a range of memory in a different order than implemented by `OutputSpan`. This may be from back-to-front or even arbitrary order. There are many possible forms such an initialization helper can take, depending on how much memory safety the application is willing to give up in the process of initializing the memory. At the unsafe end, this can be delegating to an `UnsafeMutableBufferPointer` along with a set of requirements; this option is proposed here. At the safe end, this could be delegating to a data structure which keeps track of initialized memory using a bitmap. It is unclear how much need there is for this more heavy-handed approach, so we leave it as a future enhancement if it is deemed useful. + +#### Insertions + +A use case similar to appending is insertions. Appending is simply inserting at the end. Inserting at positions other than the end is an important capability. We expect to add insertions soon in a followup proposal if `OutputSpan` is accepted. Until then, a workaround is to append, then rotate the elements to the desired position using the `mutableSpan` view. + +#### Generalized removals + +Similarly to generalized insertions (i.e. not from the end), we can think about removals of one or more elements starting at a given position. We expect to add generalized removals along with insertions in a followup proposal after `OutputSpan` is accepted. + +#### Variations on `Array.append(addingCapacity:initializingWith:)` + +The function proposed here only exposes uninitialized capacity in the `OutputSpan` parameter to its closure. A different function (perhaps named `edit()`) could also pass the initialized portion of the container, allowing an algorithm to remove or to add elements. This could be considered in addition to `append()`. + +#### Methods to initialize or update in bulk + +The `RangeReplaceableCollection` protocol has a foundational method `append(contentsOf:)` for which this document does not propose a corresponding method. We expect to first add such bulk-copying functions as part of of a package. + +`OutputSpan` lays the groundwork for new, generalized `Container` protocols that will expand upon and succeed the `Collection` hierarchy while allowing non-copyability and non-escapability to be applied to both containers and elements. We hope to find method and property names that will be generally applicable. The `append(contentsOf:)` method we refer to above always represents copyable and escapable collections with copyable and escapable elements. The definition is as follows: `mutating func append(contentsOf newElements: __owned S)`. This supports copying elements from the source, while also destroying the source if we happen to hold its only copy. This is obviously not sufficient if the elements are non-copyable, or if we only have access to a borrowed source. + +When the elements are non-copyable, we must append elements that are removed from the source. Afterwards, there are two possible dispositions of the source: destruction (`consuming`), where the source can no longer be used, or mutation (`inout`), where the source has been emptied but is still usable. + +When the elements are copyable, we can simply copy the elements from the source. Afterwards, there are two possible dispositions of the source: releasing a borrowed source, or `consuming`. The latter is approximately the same behaviour as `RangeReplaceableCollection`'s `append(contentsOf:)` function shown above. + +In an ideal world, we would like to use the same name for all of these variants: + +```swift +extension OutputSpan { + mutating func append(contentsOf: consuming some Sequence) + mutating func append(contentsOf: borrowing some Container) +} +extension OutputSpan where Element: ~Copyable { + mutating func append(contentsOf: consuming some ConsumableContainer) + mutating func append(contentsOf: inout some RangeReplaceableContainer) +} +``` + +However, this would break down in particular for `UnsafeMutableBufferPointer`, since it would make it impossible to differentiate between just copying the elements out of it, or moving its elements out (and deinitializing its memory). Once the `Container` protocols exist, we can expect that the same issue would exist for any type that conforms to more than one of the protocols involved in the list above. For example if a type conforms to `Container` as well as `Sequence`, then there would be an ambiguity. + +We could fix this by extending the syntax of the language. It is already possible to overload two functions where they differ only by whether a parameter is `inout`, for example. This is a more advanced [future direction](#contentsOf-syntax). + +Instead of the "ideal" solution, we could propose `append()` functions in the following form: + +```swift +extension OutputSpan { + mutating func append(contentsOf: consuming some Sequence) + mutating func append(copying: borrowing some Container) +} +extension OutputSpan where Element: ~Copyable { + mutating func append(consuming: consuming some ConsumableContainer) + mutating func append(moving: inout some RangeReplaceableContainer) +} +``` + +In this form, we continue to use the `contentsOf` label for `Sequence` parameters, but use different labels for the other types of containers. The `update()` methods of `MutableSpan` could be updated in a similar manner, for the same reasons. + +We note that the four variants of `append()` are required for generalized containers. We can therefore expect that the names we choose will appear later on many types of collections that interact with the future `Container` protocols. Since this nomenclature could become ubiquitous when interacting with `Collection` and `Container` instances, we defer a formal proposal until we have more experience and feedback. + +In the meantime, many applications will work efficiently with repeated calls to `append()` in a loop. The bulk initialization functions are implementable by using `withUnsafeBufferPointer` as a workaround as well, if performance is an issue before a package-based solution is released. + +#### Language syntax to distinguish between ownership modes for function arguments + +In the previous "Future Direction" subsection about [bulk initialization methods](#contentsOf), we suggest a currently unachievable naming scheme: +```swift +extension OutputSpan { + mutating func append(contentsOf: consuming some Sequence) + mutating func append(contentsOf: borrowing some Container) +} +extension OutputSpan where Element: ~Copyable { + mutating func append(contentsOf: consuming some ConsumableContainer) + mutating func append(contentsOf: inout some RangeReplaceableContainer) +} +``` + +The language partially supports disambiguating this naming scheme, in that we can already distinguish functions over the mutability of a single parameter: + +```swift +func foo(_ a: borrowing A) {} +func foo(_ a: inout A) {} + +var a = A() +foo(a) +foo(&a) +``` + +We could expand upon this ability to disambiguate by using keywords or even new sigils: + +```swift +let buffer: UnsafeMutableBufferPointer = ... +let array = Array(capacity: buffer.count*2) { + (o: inout OutputSpan) in + o.append(contentsOf: borrow buffer) + o.append(contentsOf: consume buffer) +} +``` + +## Acknowledgements + +Thanks to Karoy Lorentey, Nate Cook and Tony Parker for their feedback. diff --git a/proposals/0486-adoption-tooling-for-swift-features.md b/proposals/0486-adoption-tooling-for-swift-features.md new file mode 100644 index 0000000000..81f3424f5e --- /dev/null +++ b/proposals/0486-adoption-tooling-for-swift-features.md @@ -0,0 +1,387 @@ +# Migration tooling for Swift features + +* Proposal: [SE-0486](0486-adoption-tooling-for-swift-features.md) +* Authors: [Anthony Latsis](https://github.com/AnthonyLatsis), [Pavel Yaskevich](https://github.com/xedin) +* Review Manager: [Franz Busch](https://github.com/FranzBusch) +* Status: **Implemented (Swift 6.2)** +* Implementation: https://github.com/swiftlang/swift-package-manager/pull/8613 +* Review: [Pitch](https://forums.swift.org/t/pitch-adoption-tooling-for-upcoming-features/77936), [Review](https://forums.swift.org/t/se-0486-migration-tooling-for-swift-features/80121) + +## Introduction + +Swift 5.8 introduced [upcoming features][SE-0362], which enabled piecemeal +adoption of individual source-incompatible changes that are included in a +language mode. +Many upcoming features have a mechanical migration, meaning the compiler can +determine the exact source changes necessary to allow the code to compile under +the upcoming feature while preserving the behavior of the code. +This proposal seeks to improve the experience of enabling individual Swift +features by providing an integrated mechanism for producing these source code +modifications automatically. + +## Motivation + +It is the responsibility of project maintainers to preserve source (and binary) +compatibility both internally and for library clients when enabling an upcoming +feature, which can be difficult or tedious without having tools to help detect +possibly inadvertent changes or perform monotonous migration shenanigans for +you. +*Our* responsibility is to make that an easier task for everybody. + +### User intent + +A primary limiting factor in how proactively and accurately the compiler can +assist developers with adopting a feature is a lack of comprehension of user +intent. +Is the developer expecting guidance on adopting an improvement? +All the compiler knows to do when a feature is enabled is to compile code +accordingly. +If an upcoming feature supplants an existing grammatical construct or +invalidates an existing behavior, the language rules alone suffice because +Swift can consistently infer the irrefutable need to diagnose certain code +patterns just by spotting them. + +Needless to say, not all upcoming features fall under these criteria (and not +all features are source-breaking in the first place). +Consider [`DisableOutwardActorInference`][SE-0401], which changes actor +isolation inference rules with respect to wrapped properties. +There is no way for the programmer to specify that they'd like compiler fix-its +to make the existing actor isolation inference explicit. +If they enable the upcoming feature, their code will simply behave differently. +This was a point of debate in the review of [SE-0401], and the Language +Steering Group concluded that automatic migration tooling is the right way to +address this particular workflow, as +[noted in the acceptance notes][SE-0401-acceptance]: + +> the Language Steering Group believes that separate migration tooling to +> help programmers audit code whose behavior will change under Swift 6 mode +> would be beneficial for all upcoming features that can change behavior +> without necessarily emitting errors. + +### Automation + +Many existing and prospective upcoming features account for simple and reliable +migration paths to facilitate adoption: + +* [`NonfrozenEnumExhaustivity`][SE-0192]: Restore exhaustivity with + `@unknown default:`. +* [`ConciseMagicFile`][SE-0274]: `#file` → `#filePath`. +* [`ForwardTrailingClosures`][SE-0286]: Disambiguate argument matching by + de-trailing closures and/or inlining default arguments. +* [`ExistentialAny`][SE-0335]: `P` → `any P`. +* [`ImplicitOpenExistentials`][SE-0352]: Suppress opening with `as any P` + coercions. +* [`BareSlashRegexLiterals`][SE-0354]: Disambiguate using parentheses, + e.g. `foo(/a, b/)` → `foo((/a), b/)`. +* [`DeprecateApplicationMain`][SE-0383]: `@UIApplicationMain` → `@main`, + `@NSApplicationMain` → `@main`. +* [`DisableOutwardActorInference`][SE-0401]: Specify global actor isolation + explicitly. +* [`InternalImportsByDefault`][SE-0409]: `import X` → `public import X`. +* [`GlobalConcurrency`][SE-0412]: Convert the global variable to a `let`, or + `@MainActor`-isolate it, or mark it with `nonisolated(unsafe)`. +* [`MemberImportVisibility`][SE-0444]: Add explicit imports appropriately. +* [`InferSendableFromCaptures`][SE-0418]: Suppress inference with coercions + and type annotations. +* [Inherit isolation by default for async functions][async-inherit-isolation-pitch]: + Mark nonisolated functions with the proposed attribute. + +Application of these adjustments can be fully automated in favor of preserving +behavior, saving time for more important tasks, such as identifying, auditing, +and testing code where a change in behavior is preferable. + +## Proposed solution + +Introduce the notion of a migration mode for individual experimental and +upcoming features. +The core idea behind migration mode is a declaration of intent that can be +leveraged to build better supportive adoption experiences for developers. +If enabling a feature communicates an intent to *enact* rules, migration mode +communicates an intent to migrate code so as to preserve compatibility once the +feature is enabled. + +This proposal will support the set of existing upcoming features that +have mechanical migrations, as described in the [Automation](#automation) +section. +All future proposals that intend to introduce an upcoming feature and +provide for a mechanical migration should include a migration mode and detail +its behavior alongside the migration paths in the *Source compatibility* +section. + +## Detailed design + +Upcoming features that have mechanical migrations will support a migration +mode, which is a new mode of building a project that will produce compiler +warnings with attached fix-its that can be applied to preserve the behavior +of the code under the feature. + +The action of enabling a previously disabled upcoming feature in migration +mode must not cause any new compiler errors or behavioral changes, and the +fix-its produced must preserve compatibility. +Compatibility here refers to both source and binary compatibility, as well as +to behavior. +Additionally, this action will have no effect if the mode is not supported +for a given upcoming feature, i.e., because the upcoming feature does not +have a mechanical migration. +A corresponding warning will be emitted in this case to avoid the false +impression that the impacted source code is compatible with the feature. +This warning will belong to the diagnostic group `StrictLanguageFeatures`. + +### Interface + +The `-enable-*-feature` frontend and driver command line options will start +supporting an optional mode specifier with `migrate` as the only valid mode: + +``` +-enable-upcoming-feature [:] +-enable-experimental-feature [:] + + := migrate +``` + +For example: + +``` +-enable-upcoming-feature InternalImportsByDefault:migrate +``` + +If the specified mode is invalid, the option will be ignored, and a warning will +be emitted. +This warning will belong to the diagnostic group `StrictLanguageFeatures`. +In a series of either of these options applied to a given feature, only the +last option will be honored. +If a feature is both implied by the effective language mode and enabled in +migration mode, the latter option will be disregarded. + +### Diagnostics + +Diagnostics emitted in relation to a specific feature in migration mode must +belong to a diagnostic group named after the feature. +The names of diagnostic groups can be displayed alongside diagnostic messages +using `-print-diagnostic-groups` and used to associate messages with features. + +### `swift package migrate` command + +To enable seamless migration experience for Swift packages, I'd like to propose a new Swift Package Manager command - `swift package migrate` to complement the Swift compiler-side changes. + +The command would accept one or more features that have migration mode enabled and optionally a set of targets to migrate, if no targets are specified the whole package is going to be migrated to use new features. + +#### Interface + +``` +USAGE: swift package migrate [] --to-feature ... + +OPTIONS: + --target The targets to migrate to specified set of features or a new language mode. + --to-feature + The Swift language upcoming/experimental feature to migrate to. + -h, --help Show help information. +``` + +#### Use case + +``` +swift package migrate --target MyTarget,MyTest --to-feature ExistentialAny +``` + +This command would attempt to build `MyTarget` and `MyTest` targets with `ExistentialAny:migrate` feature flag, apply any fix-its associated with +the feature produced by the compiler, and update the `Package.swift` to +enable the feature(s) if both of the previous actions are successful: + +``` +.target( + name: "MyTarget", + ... + swiftSettings: [ + // ... existing settings, + .enableUpcomingFeature("ExistentialAny") + ] +) +... +.testTarget( + name: "MyTest", + ... + swiftSettings: [ + // ... existing settings, + .enableUpcomingFeature("ExistentialAny") + ] +) +``` + +In the "whole package" mode, every target is going to be updated to include +new feature flag(s). This is supported by the same functionality as `swift package add-setting` command. + +If it's, for some reason, impossible to add the setting the diagnostic message would suggest what to add and where i.e. `...; please add 'ExistentialAny' feature to 'MyTarget' target manually`. + +#### Impact on Interface + +This proposal introduces a new command but does not interfere with existing commands. It follows the same pattern as `swift build` and `swift test` in a consistent manner. + +## Source compatibility + +This proposal does not affect language rules. +The described changes to the API surface are source-compatible. + +## ABI compatibility + +This proposal does not affect binary compatibility or binary interfaces. + +## Implications on adoption + +Entering or exiting migration mode can affect behavior and is therefore a +potentially source-breaking action. + +## Future directions + +### Producing source incompatible fix-its + +For some features, a source change that alters the semantics of +the program is a more desirable approach to addressing an error that comes +from enabling the feature. +For example, programmers might want to replace cases of `any P` with `some P`. +Migration tooling could support the option to produce source incompatible +fix-its in cases where the compiler can detect that a different behavior might +be more beneficial. + +### Applications beyond mechanical migration + +The concept of migration mode could be extrapolated to additive features, such +as [typed `throws`][SE-0413] or [opaque parameter types][SE-0341], by providing +actionable adoption tips. +Additive features are hard-enabled and become an integral part of the language +as soon as they ship. +Many recent additive features are already integrated into the Swift feature +model, and their metadata is kept around either to support +[feature availability checks][SE-0362-feature-detection] in conditional +compilation blocks or because they started off as experimental features. + +Another feasible extension of migration mode is promotion of best practices. + +### Augmented diagnostic metadata + +The current serialization format for diagnostics does not include information +about diagnostic groups or whether a particular fix-it preserves semantics. +There are several reasons why this data can be valuable for users, and why it +is essential for future tools built around migration mode: +* The diagnostic group name can be used to, well, group diagnostics, as well as + to communicate relationships between diagnostics and features and filter out + relevant diagnostics. + This can prove especially handy when multiple features are simultaneously + enabled in migration mode, or when similar diagnostic messages are caused by + distinct features. +* Exposing the purpose of a fix-it can help developers make quicker decisions + when offered multiple fix-its. + Furthermore, tools can take advantage of this information by favoring and + auto-applying source-compatible fix-its. + +## Alternatives considered + +### A distinct `-migrate` option + +This direction has a questionably balanced set of advantages and downsides. +On one hand, it would provide an adequate foundation for invoking migration +for a language mode in addition to individual features. +On the other hand, an independent option is less discoverable, has a steeper +learning curve, and makes the necessary relationships between it and the +existing `-enable-*-feature` options harder to infer. +Perhaps more notably, a bespoke option by itself would not scale to any future +modes, setting what might be an unfortunate example for further decentralization +of language feature control. + +### API for package manifests + +The decision around surfacing migration mode in the `PackageDescription` +library depends on whether there is a consensus on the value of enabling it as +a persistent setting as opposed to an automated procedure in the long run. + +Here is how an API change could look like for the proposed solution: + +```swift ++extension SwiftSetting { ++ @available(_PackageDescription, introduced: 6.2) ++ public enum SwiftFeatureMode { ++ case migrate ++ case on ++ } ++} +``` +```diff + public static func enableUpcomingFeature( + _ name: String, ++ mode: SwiftFeatureMode = .on, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting + + public static func enableExperimentalFeature( + _ name: String, ++ mode: SwiftFeatureMode = .on, + _ condition: BuildSettingCondition? = nil + ) -> SwiftSetting +``` + +It can be argued that both Swift modules and the volume of changes required for +migration can be large enough to justify spreading the review over several +sessions, especially if migration mode gains support for parallel +[source-incompatible fix-its][#producing-source-incompatible-fix-its]. +However, we also expect higher-level migration tooling to allow for +incremental progress. + +### Naming + +The next candidates in line per discussions are ***adopt***, ***audit***, +***stage***, and ***preview***, respectively. +* ***preview*** and ***stage*** can both be understood as to report on the + impact of a change, but are less commonly used in the sense of code + migration. +* ***audit*** best denotes a recurrent action in this context, which we believe + is more characteristic of the static analysis domain, such as enforcing a set + of custom compile-time rules on code. +* An important reservation about ***adoption*** of source-breaking features is + that it comprises both code migration and integration. + It may be more prudent to save this term for a future add-on mode that, + unlike migration mode, implies that the feature is enabled, and can be invoked + in any language mode to aid developers in making better use of new behaviors + or rules. + To illustrate, this mode could appropriately suggest switching from `any P` + to `some P` for `ExistentialAny`. + +### `swift package migrate` vs. `swift migrate` + +Rather than have migrate as a subcommand (ie. `swift package migrate`), another option is to add another top level command, ie. `swift migrate`. + +As the command applies to the current package, we feel a `swift package` sub-command fits better than a new top-level command. This also aligns with the recently added package refactorings (eg. `add-target`). + +## Acknowledgements + +This proposal was inspired by documents prepared by [Allan Shortlidge] and +[Holly Borla]. +Special thanks to Holly for her guidance throughout the draft stage. + + + +[Holly Borla]: https://github.com/hborla +[Allan Shortlidge]: https://github.com/tshortli + +[SE-0192]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0192-non-exhaustive-enums.md +[SE-0274]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0274-magic-file.md +[SE-0286]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0286-forward-scan-trailing-closures.md +[SE-0296]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0296-async-await.md +[SE-0335]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0335-existential-any.md +[SE-0337]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0337-support-incremental-migration-to-concurrency-checking.md +[SE-0341]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0341-opaque-parameters.md +[SE-0352]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0352-implicit-open-existentials.md +[SE-0354]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0354-regex-literals.md +[SE-0362]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md +[SE-0362-feature-detection]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md#feature-detection-in-source-code +[SE-0383]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0383-deprecate-uiapplicationmain-and-nsapplicationmain.md +[SE-0401]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0401-remove-property-wrapper-isolation.md +[SE-0401-acceptance]: https://forums.swift.org/t/accepted-with-modifications-se-0401-remove-actor-isolation-inference-caused-by-property-wrappers/66241 +[SE-0409]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0409-access-level-on-imports.md +[SE-0411]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0411-isolated-default-values.md +[SE-0413]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0413-typed-throws.md +[SE-0412]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0412-strict-concurrency-for-global-variables.md +[SE-0418]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0418-inferring-sendable-for-methods.md +[SE-0423]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0423-dynamic-actor-isolation.md +[SE-0434]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0434-global-actor-isolated-types-usability.md +[SE-0444]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0444-member-import-visibility.md +[async-inherit-isolation-pitch]: https://forums.swift.org/t/pitch-inherit-isolation-by-default-for-async-functions/74862 diff --git a/proposals/0487-extensible-enums.md b/proposals/0487-extensible-enums.md new file mode 100644 index 0000000000..a8e226c95b --- /dev/null +++ b/proposals/0487-extensible-enums.md @@ -0,0 +1,319 @@ +# Nonexhaustive enums + +* Proposal: [SE-0487](0487-extensible-enums.md) +* Authors: [Pavel Yaskevich](https://github.com/xedin), [Franz Busch](https://github.com/FranzBusch), [Cory Benfield](https://github.com/lukasa) +* Review Manager: [Ben Cohen](https://github.com/airspeedswift) +* Status: **Accepted** +* Bug: [apple/swift#55110](https://github.com/swiftlang/swift/issues/55110) +* Implementation: [apple/swift#80503](https://github.com/swiftlang/swift/pull/80503) +* Upcoming Feature Flag: `ExtensibleAttribute` +* Review: ([pitch](https://forums.swift.org/t/pitch-extensible-enums-for-non-resilient-modules/77649)) ([first review](https://forums.swift.org/t/se-0487-extensible-enums/80114)) ([second review](https://forums.swift.org/t/second-review-se-0487-extensible-enums/80837)) ([acceptance](https://forums.swift.org/t/accepted-se-0487-nonexhaustive-enums/81508)) + +Previously pitched in: + +- https://forums.swift.org/t/extensible-enumerations-for-non-resilient-libraries/35900 +- https://forums.swift.org/t/pitch-non-frozen-enumerations/68373 + +Revisions: +- Renamed the attribute to `@nonexhaustive` and `@nonexhaustive(warn)` respectively +- Re-focused this proposal on introducing a new `@extensible` attribute and + moved the language feature to a future direction +- Introduced a second annotation `@nonExtensible` to allow a migration path into + both directions +- Added future directions for adding additional associated values +- Removed both the `@extensible` and `@nonExtensible` annotation in favour of + re-using the existing `@frozen` annotation +- Added the high level goals that this proposal aims to achieve +- Expanded on the proposed migration path for packages with regards to their + willingness to break API +- Added future directions for exhaustive matching for larger compilation units +- Added alternatives considered section for a hypothetical + `@preEnumExtensibility` +- Added a section for `swift package diagnose-api-breaking-changes` + +## Introduction + +This proposal provides developers the capabilities to mark public enums in +non-resilient Swift libraries as extensible. This makes Swift `enum`s vastly +more useful in public API of such libraries. + +## Motivation + +When Swift was enhanced to add support for ABI-stable libraries that were built with +"library evolution" enabled ("resilient" libraries as we call them in this proposal), +the Swift language had to support these libraries vending enums that might have cases +added to them in a later version. Swift supports exhaustive switching over cases. +When binaries are compiled against a ABI-stable library they need to be able to handle the +addition of a new case by that library later on, without needing to be rebuilt. + +Consider the following simple library to your favorite pizza place: + +```swift +public enum PizzaFlavor { + case hawaiian + case pepperoni + case cheese +} +``` + +In the standard "non-resilient" mode, users of the library can write exhaustive switch +statements over the enum `PizzaFlavor`: + +```swift +switch pizzaFlavor { +case .hawaiian: + throw BadFlavorError() +case .pepperoni: + try validateNoVegetariansEating() + return .delicious +case .cheese: + return .delicious +} +``` + +Swift requires switches to be exhaustive i.e. the must handle every possibility. +If the author of the above switch statement was missing a case (perhaps they forgot +`.hawaiian` is a flavor), the compiler will error, and force the user to either add a +`default:` clause, or to add the missing case. + +If later a new case is added to the enum (maybe `.veggieSupreme`), exhaustive switches +over that enum might no longer be exhaustive. This is often _desirable_ within a single +codebase (even one split up into multiple modules). A case is added, and the compiler will +assist in finding all the places where this new case must be handled. + +But it presents a problem for authors of both resilient and non-resilient libraries: + +- For non-resilient libraries, adding a case is a source-breaking API change: clients +exhaustively switching over the enum will no longer compile. So can only be done with +a major semantic version bump. +- For resilient libraries, even that is not an option. An ABI-stable library cannot allow +a situation where a binary that has not yet been recompiled can no longer rely on its +switches over an enum are exhaustive. + +Because of the implications on ABI and the requirement to be able to evolve +libraries with public enumerations in their API, the resilient language dialect introduced +"non-exhaustive enums" in [SE-0192](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0192-non-exhaustive-enums.md). + +If the library was compiled with `-enable-library-evolution`, when a user attempts to +exhaustively switch over the `PizzaFlavor` enum the compiler will emit an error +(when in Swift 6 language mode, a warning in prior language modes), requiring users +to add an `@unknown default:` clause: + +```swift +switch pizzaFlavor { +case .hawaiian: + throw BadFlavorError() +case .pepperoni: + try validateNoVegetariansEating() + return .delicious +case .cheese: + return .delicious +@unknown default: + try validateNoVegetariansEating() + return .delicious +} +``` + +The user is forced to specify how cases are handled if they are introduced later. This +allows ABI-stable libraries to add cases without risking undefined behavior in client +binaries that haven't yet been recompiled. + +When a resilient library knows that an enumeration will never be extended, the author +can annotate the enum with `@frozen`, which in the case of enums is a guarantee that no +further cases can be added. For example, the `Optional` type in the standard library is +frozen, as no third option beyond `some` and `none` will ever be added. This brings +performance benefits, and also the convenience of not requiring an `@unknown default` case. + +`@frozen` is a powerful attribute that can be applied to both structs and enums. It has a +wide ranging number of effects, including exposing their size directly as part of the ABI +and providing direct access to stored properties. However, on enums it happens to +have source-level effects on the behavior of switch statements by clients of a library. +This difference was introduced late in the process of reviewing SE-0192. + +Extensibility of enums is also desirable for non-resilient libraries. Without it, there is no +way for a Swift package to be able to evolve a public enumeration without breaking the API. +However, in Swift today it is not possible for the default, "non-resilient" dialect to opt-in +to the extensible enumeration behavior. This is a substantial limitation, and greatly reduces +the utility of enumerations in non-resilient Swift. + +Over the past years, many packages have run into this limitation when trying to express APIs +using enums. As a non-exhaustive list of problems this can cause: + +- Using enumerations to represent `Error`s is inadvisable, as if new errors need + to be introduced they cannot be added to existing enumerations. This leads to + a proliferation of `Error` enumerations. "Fake" enumerations can be made using + `struct`s and `static let`s, but these do not work with the nice `Error` + pattern-match logic in catch blocks, requiring type casts. +- Using an enumeration to refer to a group of possible ideas without entirely + exhaustively evaluating the set is potentially dangerous, requiring a + deprecate-and-replace if any new elements appear. +- Using an enumeration to represent any concept that is inherently extensible is + tricky. For example, `SwiftNIO` uses an enumeration to represent HTTP status + codes. If new status codes are added, SwiftNIO needs to either mint new + enumerations and do a deprecate-and-replace, or it needs to force these new + status codes through the .custom enum case. + +This proposal plans to address these limitations on enumerations in +non-resilient Swift. + +## Proposed solution + +We propose to introduce a new `@nonexhaustive` attribute that can be applied to +enumerations to mark them as extensible. Such enums will behave the same way as +non-frozen enums from resilient Swift libraries. + +An example of using the new attribute is below: + +```swift +/// Module A +@nonexhaustive +public enum PizzaFlavor { + case hawaiian + case pepperoni + case cheese +} + +/// Module B +switch pizzaFlavor { // error: Switch covers known cases, but 'MyEnum' may have additional unknown values, possibly added in future versions +case .hawaiian: + throw BadFlavorError() +case .pepperoni: + try validateNoVegetariansEating() + return .delicious +case .cheese: + return .delicious +} +``` + +### Exhaustive switching inside same module/package + +Code inside the same module or package can be thought of as one co-developed +unit of code. Inside the same module or package, switching exhaustively over an +`@nonexhaustive` enum inside will not require an`@unknown default`, and using +one will generate a warning. + +### `@nonexhaustive` and `@frozen` + +An enum cannot be `@frozen` and `@nonexhaustive` at the same time. Thus, marking an +enum both `@nonexhaustive` and `@frozen` is not allowed and will result in a +compiler error. + +### API breaking checker + +The behavior of `swift package diagnose-api-breaking-changes` is also updated +to understand the new `@nonexhaustive` attribute. + +### Staging in using `@nonexhaustive(warn)` + +We also propose adding a new `@nonexhaustive(warn)` attribute that can be used +to mark enumerations as pre-existing to when they became extensible.This is +useful for developers that want to stage in changing an existing non-extensible +enum to be extensible over multiple releases. Below is an example of how this +can be used: + +```swift +// Package A +public enum Foo { + case foo +} + +// Package B +switch foo { +case .foo: break +} + +// Package A wants to make the existing enum extensible +@nonexhaustive(warn) +public enum Foo { + case foo +} + +// Package B now emits a warning downgraded from an error +switch foo { // warning: Enum might be extended later. Add an @unknown default case. +case .foo: break +} + +// Later Package A decides to extend the enum and releases a new major version +@nonexhaustive(warn) +public enum Foo { + case foo + case bar +} + +// Package B didn't add the @unknown default case yet. So now we we emit a warning and an error +switch foo { // error: Unhandled case bar & warning: Enum might be extended later. Add an @unknown default case. +case .foo: break +} +``` + +While the `@nonexhaustive(warn)` attribute doesn't solve the need of requiring +a new major when a new case is added it allows developers to stage in changing +an existing non-extensible enum to become extensible in a future release by +surfacing a warning about this upcoming break early. + +## Source compatibility + +### Resilient modules + +- Adding or removing the `@nonexhaustive` attribute has no-effect since it is the default in this language dialect. +- Adding the `@nonexhaustive(warn)` attribute has no-effect since it only downgrades the error to a warning. +- Removing the `@nonexhaustive(warn)` attribute is an API breaking since it upgrades the warning to an error again. + +### Non-resilient modules + +- Adding the `@nonexhaustive` attribute is an API breaking change. +- Removing the `@nonexhaustive` attribute is an API stable change. +- Adding the `@nonexhaustive(warn)` attribute has no-effect since it only downgrades the error to a warning. +- Removing the `@nonexhaustive(warn)` attribute is an API breaking since it upgrades the warning to an error again. + +## ABI compatibility + +The new attribute does not affect the ABI of an enum since it is already the +default in resilient modules. + +## Future directions + +### Aligning the language dialects + +In a previous iteration of this proposal, we proposed to add a new language +feature to align the language dialects in a future language mode. The main +motivation behind this is that the current default of non-extensible enums is a +common pitfall and results in tremendous amounts of unnoticed API breaks in the +Swift package ecosystem. We still believe that a future proposal should try +aligning the language dialects. This proposal is focused on providing a first +step to allow extensible enums in non-resilient modules. + +Regardless of whether a future language mode changes the default for non-resilient +libraries, a way of staging in this change will be required (similar to how the +`@preconcurency` attribute facilitated incremental adoption of Swift concurrency). + +### `@unknown catch` + +Enums can be used for errors. Catching and pattern matching enums could add +support for an `@unknown catch` to make pattern matching of typed throws align +with `switch` pattern matching. + +### Allow adding additional associated values + +Adding additional associated values to an enum can also be seen as extending it +and we agree that this is interesting to explore in the future. However, this +proposal focuses on solving the primary problem of the usability of public +enumerations in non-resilient modules. + +### Larger compilation units than packages + +During the pitch it was brought up that a common pattern for application +developers is to split an application into multiple smaller packages. Those +packages are versioned together and want to have the same exhaustive matching +behavior as code within a single package. As a future direction, build and +package tooling could allow to define larger compilation units to express this. +Until then developers are encouraged to use `@frozen` attributes on their +enumerations to achieve the same effect. + +## Alternatives considered + +### Different names for the attribute + +We considered different names for the attribute such as `@nonFrozen` or +`@extensible`; however, we felt that `@nonexhaustive` communicates the idea of +an extensible enum more clearly. diff --git a/proposals/0488-extracting.md b/proposals/0488-extracting.md new file mode 100644 index 0000000000..529b1d7e53 --- /dev/null +++ b/proposals/0488-extracting.md @@ -0,0 +1,146 @@ +# Apply the extracting() slicing pattern more widely + +* Proposal: [SE-0488](0488-extracting.md) +* Author: [Guillaume Lessard](https://github.com/glessard) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Implemented (Swift 6.2)** +* Implementation: underscored `_extracting()` members of `Span` and `RawSpan`, pending elsewhere. +* Review: ([pitch](https://forums.swift.org/t/pitch-apply-the-extracting-slicing-pattern-to-span-and-rawspan/80322)) ([review](https://forums.swift.org/t/se-0488-apply-the-extracting-slicing-pattern-more-widely/80854)) ([acceptance](https://forums.swift.org/t/accepted-se-0488-apply-the-extracting-slicing-pattern-more-widely/81235)) + +[SE-0437]: 0437-noncopyable-stdlib-primitives.md +[SE-0447]: 0447-span-access-shared-contiguous-storage.md +[SE-0467]: 0467-MutableSpan.md +[Forum-LifetimeAnnotations]: https://forums.swift.org/t/78638 + + +## Introduction and Motivation + +Slicing containers is an important operation, and non-copyable values have introduced a significant change in the spelling of that operation. When we [introduced][SE-0437] non-copyable primitives to the standard library, we allowed slicing `UnsafeBufferPointer` and related types via a family of `extracting()` methods. We expanded upon these when introducing [`MutableSpan`][SE-0467]. + +Now that we have a [supported spelling][Forum-LifetimeAnnotations] for lifetime dependencies, we propose adding the `extracting()` methods to `Span` and `RawSpan`, as well as members of the `UnsafeBufferPointer` family that were missed in [SE-0437][SE-0437]. + + +## Proposed solution + +As previously discussed in [SE-0437][SE-0437], the slicing pattern established by the `Collection` protocol cannot be generalized for either non-copyable elements or non-escapable containers. The solution is a family of functions named `extracting()`, with appropriate argument labels. + +The family of `extracting()` methods established by the [`MutableSpan` proposal][SE-0467] is as follows: +```swift +public func extracting(_ bounds: Range) -> Self +public func extracting(_ bounds: some RangeExpression) -> Self +public func extracting(_: UnboundedRange) -> Self +@unsafe public func extracting(unchecked bounds: Range) -> Self +@unsafe public func extracting(unchecked bounds: ClosedRange) -> Self + +public func extracting(first maxLength: Int) -> Self +public func extracting(droppingLast k: Int) -> Self +public func extracting(last maxLength: Int) -> Self +public func extracting(droppingFirst k: Int) -> Self +``` + +These will be provided for the following standard library types: +```swift +Span +RawSpan +UnsafeBufferPointer +UnsafeMutableBufferPointer +Slice> +Slice> +UnsafeRawBufferPointer +UnsafeMutableRawBufferPointer +Slice +Slice +``` +Some of the types in the list above already have a subset of the `extracting()` functions; their support will be rounded out to the full set. + + +## Detailed design + +The general declarations for these functions is as follows: +```swift +/// Returns an extracted slice over the items within +/// the supplied range of positions. +/// +/// Traps if any position within the range is invalid. +@_lifetime(copy self) +public func extracting(_ bounds: Range) -> Self + +/// Returns an extracted slice over the items within +/// the supplied range of positions. +/// +/// Traps if any position within the range is invalid. +@_lifetime(copy self) +public func extracting(_ bounds: some RangeExpression) -> Self + +/// Returns an extracted slice over all items of this container. +@_lifetime(copy self) +public func extracting(_: UnboundedRange) -> Self + +/// Returns an extracted slice over the items within +/// the supplied range of positions. +/// +/// This function does not validate `bounds`; this is an unsafe operation. +@unsafe @_lifetime(copy self) +public func extracting(unchecked bounds: Range) -> Self + +/// Returns an extracted slice over the items within +/// the supplied range of positions. +/// +/// This function does not validate `bounds`; this is an unsafe operation. +@unsafe @_lifetime(copy self) +public func extracting(unchecked bounds: ClosedRange) -> Self + +/// Returns an extracted slice over the initial elements +/// of this container, up to the specified maximum length. +@_lifetime(copy self) +public func extracting(first maxLength: Int) -> Self + +/// Returns an extracted slice excluding +/// the given number of trailing elements. +@_lifetime(copy self) +public func extracting(droppingLast k: Int) -> Self + +/// Returns an extracted slice containing the final elements +/// of this container, up to the given maximum length. +@_lifetime(copy self) +public func extracting(last maxLength: Int) -> Self + +/// Returns an extracted slice excluding +/// the given number of initial elements. +@_lifetime(copy self) +public func extracting(droppingFirst k: Int) -> Self +``` +For escapable types, the `@_lifetime` attribute is not applied. + + +### Usage hints + +The `extracting()` pattern, while not completely new, is still a departure over the slice pattern established by the `Collection` protocol. For `Span`, `RawSpan`, `MutableSpan` and `MutableRawSpan`, we can add unavailable subscripts and function with hints towards the corresponding `extracting()` function: + +```swift +@available(*, unavailable, renamed: "extracting(_ bounds:)") +public subscript(bounds: Range) -> Self { extracting(bounds) } + +@available(*, unavailable, renamed: "extracting(first:)") +public func droppingFirst(_ k: Int) -> Self { extracting(first: k) } +``` + +## Source compatibility +This proposal is additive and source-compatible with existing code. + +## ABI compatibility +This proposal is additive and ABI-compatible with existing code. + +## Implications on adoption +The additions described in this proposal require a new version of the Swift standard library. + +## Alternatives considered +This is an extension of an existing pattern. We are not considering a different pattern at this time. + +## Future directions +#### Disambiguation over ownership type +The `extracting()` functions proposed here are borrowing. `MutableSpan` has versions defined as mutating, but it could benefit from consuming ones as well. In general there could be a need for all three ownership variants of a given operation (`borrowing`, `consuming`, or `mutating`.) In order to handle these variants, we could establish a pattern for disambiguation by name, or we could invent new syntax to disambiguate by ownership type. This is a complex topic left to future proposals. + +## Acknowledgements +Thanks to Karoy Lorentey and Tony Parker. + diff --git a/proposals/0489-codable-error-printing.md b/proposals/0489-codable-error-printing.md new file mode 100644 index 0000000000..6dd70e71a5 --- /dev/null +++ b/proposals/0489-codable-error-printing.md @@ -0,0 +1,154 @@ +# Improve `EncodingError` and `DecodingError`'s printed descriptions + +* Proposal: [SE-0489](0489-codable-error-printing.md) +* Authors: [Zev Eisenberg](https://github.com/ZevEisenberg) +* Review Manager: [Xiaodi Wu](https://github.com/xwu) +* Status: **Accepted** +* Implementation: https://github.com/swiftlang/swift/pull/80941 +* Review: ([pitch](https://forums.swift.org/t/pitch-improve-encodingerror-and-decodingerror-s-printed-descriptions/79872)) ([review](https://forums.swift.org/t/se-0489-improve-encodingerror-and-decodingerrors-printed-descriptions/81021)) ([acceptance](https://forums.swift.org/t/accepted-se-0489-improve-encodingerror-and-decodingerrors-printed-descriptions/81380)) + +## Introduction + +`EncodingError` and `DecodingError` do not specify any custom debug description. The default descriptions bury the useful information in a format that is difficult to read. Less experienced developers may assume they are not human-readable at all, even though they contain useful information. The proposal is to conform `EncodingError` and `DecodingError` to `CustomDebugStringConvertible` and provide nicely formatted debug output. + +## Motivation + +Consider the following example model structs: + +```swift +struct Person: Codable { + var name: String + var home: Home +} + +struct Home: Codable { + var city: String + var country: Country +} + +struct Country: Codable { + var name: String + var population: Int +} +``` + +Now let us attempt to decode some invalid JSON. In this case, it is missing a field in a deeply nested struct. + +```swift +// Note missing "population" field +let jsonData = Data(""" +[ + { + "name": "Ada Lovelace", + "home": { + "city": "London", + "country": { + "name": "England" + } + } + } +] +""".utf8) + +do { + _ = try JSONDecoder().decode([Person].self, from: jsonData) +} catch { + print(error) +} +``` + +This outputs the following: + +`keyNotFound(CodingKeys(stringValue: "population", intValue: nil), Swift.DecodingError.Context(codingPath: [_CodingKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "home", intValue: nil), CodingKeys(stringValue: "country", intValue: nil)], debugDescription: "No value associated with key CodingKeys(stringValue: \"population\", intValue: nil) (\"population\").", underlyingError: nil))` + +All the information you need is there: +- The kind of error: a missing key +- Which key was missing: `"population"` +- The path of the value that had a missing key: index 0, then key `"home"`, then key `"country"` +- The underlying error: none, in this case + +However, it is not easy or pleasant to read such an error, particularly when dealing with large structures or long type names. It is common for newer developers to assume the above output is some kind of log spam and not even realize it contains exactly the information they are looking for. + +## Proposed solution + +Conform `EncodingError` and `DecodingError` to `CustomDebugStringConvertible` and provide a clean, readable debug description for each. + +Complete examples of the before/after diffs are available in the description of the [implementation pull request](https://github.com/swiftlang/swift/pull/80941) that accompanies this proposal. + +**Note 1:** This proposal is _not_ intended to specify an exact output format, and any examples are not a guarantee of current or future behavior. You are still free to inspect the contents of thrown errors directly if you need to detect specific problems. + +**Note 2:** The output could be further improved by modifying `JSONDecoder` to write a better debug description. See [Future Directions](#future-directions) for more. + +## Detailed design + +```swift +@available(SwiftStdlib 6.2, *) +extension EncodingError: CustomDebugStringConvertible { + public var debugDescription: String {...} +} + +@available(SwiftStdlib 6.2, *) +extension DecodingError: CustomDebugStringConvertible { + public var debugDescription: String {...} +} +``` + +## Source compatibility + +The new conformance changes the result of converting an `EncodingError` or `DecodingError` value to a string. This changes observable behavior: code that attempts to parse the result of `String(describing:)` or `String(reflecting:)` can be misled by the change of format. + +However, the documentation of these interfaces explicitly state that when the input type conforms to none of the standard string conversion protocols, then the result of these operations is unspecified. + +Changing the value of an unspecified result is not considered to be a source incompatible change. + +## ABI compatibility + +The proposal conforms two previously existing stdlib types to a previously existing stdlib protocol. This is technically an ABI breaking change: on ABI-stable platforms, we may have preexisting Swift binaries that implement a retroactive `CustomDebugStringConvertible` conformance, or binaries that assume that the existing error types do _not_ conform to the protocol. + +We do not expect this to be an issue in practice, since checking an arbitrary error for conformance to `CustomDebugStringConvertible` at run-time seems unlikely. In the event that it now conforms where it didn't before, it will presumably use the new implementation instead of whatever fallback was being provided previously. + +## Implications on adoption + +### Conformance to `CustomDebugStringConvertible` + +The conformance to `CustomDebugStringConvertible` is not backdeployable. As a result, code that runs on ABI-stable platforms with earlier versions of the standard library won't output the new debug descriptions. + +### `debugDescription` Property + +It is technically possible to backdeploy the `debugDescription` property, but without the protocol conformance, it is of limited utility. + +## Future directions + +### Better error generation from Foundation encoders/decoders + +The debug descriptions generated in Foundation sometimes contain the same information as the new debug descriptions from this proposal. A future change to the standard JSON and Plist encoders and decoders could provide more compact debug descriptions once they can be sure they have the new standard library descriptions available. They could also use a more compact description when rendering the description of a `CodingKey`. Take, for example: + +``` +Debug description: No value associated with key CodingKeys(stringValue: "population", intValue: nil) ("population"). +``` + +The `CodingKeys(stringValue: "population", intValue: nil) ("population")` part is coming from the default `description` of `CodingKey`, plus an extra parenthesized string value at the end for good measure. The Foundation (de|en)coders could construct a more compact description that does not repeat the key, just like we do within this proposal in the context of printing a coding path. + +### Print context of surrounding lines in source data + +When a decoding error occurs, in addition to printing the path, the error message could include some surrounding lines from the source data. This was explored in this proposal's antecedent, [UsefulDecode](https://github.com/ZevEisenberg/UsefulDecode). But more detailed messages would require passing more context data from the decoder and changing the public interface of `DecodingError` to carry more data. This option is best left as something to think about as [we design `Codable`'s successor](https://forums.swift.org/t/the-future-of-serialization-deserialization-apis/78585). But just to give an example of the _kind_ of context that could be provided (please do not read anything into the specifics of the syntax; this is a sketch, not a proposal): + +``` +Value not found: expected 'name' (String) at [0]/address/city/birds/[1]/name, got: +{ + "feathers" : "some", + "name" : null +} +``` + +## Alternatives considered + +We could conform `EncodingError` and `DecodingError` to `CustomStringConvertible` instead of `CustomDebugStringConvertible`. The use of the debug-flavored protocol emphasizes that the new descriptions aren't intended to be used outside debugging contexts. This is in keeping with the precedent set by [SE-0445](0445-string-index-printing.md). + +We could change `CodingKey.description` to return the bare string or int value, which would improve the formatting and reduce duplication as seen in [Proposed solution](#proposed-solution). But changing the exsting implementation of an existing public method seems needlessly risky, as existing code may (however inadvisably) be depending on the format of the current `description`. Additionally, the encoders and decoders in Foundation should not depend on implementation details of `CodingKey.description` that are not guaranteed. If we want the encoders/decoders to produce better formatting, they should be responsible for generating those strings directly. See [further discussion in the PR](https://github.com/swiftlang/swift/pull/80941#discussion_r2064277369). + +## Acknowledgments + +This proposal follows in the footsteps of [SE-0445](0445-string-index-printing.md). Thanks to [Karoy Lorentey](https://github.com/lorentey) for writing that proposal, and for flagging it as similar to this one. + +Thanks to Kevin Perry [for suggesting](https://forums.swift.org/t/the-future-of-serialization-deserialization-apis/78585/77) that this would make a good standalone change regardless of the direction of future serialization tools, and for engaging with the PR from the beginning. diff --git a/proposals/0490-environment-constrained-shared-libraries.md b/proposals/0490-environment-constrained-shared-libraries.md new file mode 100644 index 0000000000..d4d9447eb1 --- /dev/null +++ b/proposals/0490-environment-constrained-shared-libraries.md @@ -0,0 +1,163 @@ +# Environment Constrained Shared Libraries + +* Proposal: [SE-0490](0490-environment-constrained-shared-libraries.md) +* Authors: [tayloraswift](https://github.com/tayloraswift) +* Review Manager: [Alastair Houghton](https://github.com/al45tair) +* Status: **Active Review (Sep 5 - Sep 18, 2025)** +* Implementation: [swiftlang/swift-package-manager#8249](https://github.com/swiftlang/swift-package-manager/pull/8249) +* Documentation: [How to use Environment-Constrained Shared Libraries](https://github.com/swiftlang/swift-package-manager/blob/1eaf59d2facc74c88574f38395aa49983b2badcc/Documentation/ECSLs.md) +* Bugs: [SR-5714](https://github.com/swiftlang/swift-package-manager/issues/5714) +* Review: ([pitch](https://forums.swift.org/t/pitch-replaceable-library-plugins/77605)) ([review](https://forums.swift.org/t/se-0490-environment-constrained-shared-libraries/81975)) + +## Introduction + +SwiftPM currently has no support for non-system binary library dependencies on Linux. This proposal adds support for **Environment Constrained Shared Libraries**, which are a type of dynamic library that is shared across a fleet of machines and can be upgraded without recompiling and redeploying all applications running on those machines. We will distribute Environment Constrained Shared Libraries through the existing `.artifactbundle` format. + +Swift-evolution thread: [Discussion thread](https://forums.swift.org/t/pitch-replaceable-library-plugins/77605) + +Example Producer: [swift-dynamic-library-example](https://github.com/tayloraswift/swift-dynamic-library-example) + +Example Consumer: [swift-dynamic-library-example-client](https://github.com/tayloraswift/swift-dynamic-library-example-client) + +## Motivation + +Many of us in the Server World have a Big App with a small component that changes very rapidly, much more rapidly than the rest of the App. This component might be something like a filter, or an algorithm, or a plugin that is being constantly tuned. + +We could, for argument’s sake, try and turn this component into data that can be consumed by the Big App, which would probably involve designing a bytecode and an interpreter, and maybe even a whole interpreted domain-specific programming language. But that is very hard and we would rather just write this thing in Swift, and let Swift code call Swift code. + +While macOS has Dynamic Library support through XCFrameworks, on Linux we currently have to recompile the Big App from source and redeploy the Big App every time the filter changes, and we don’t want to do that. What we really want instead is to have the Big App link the filter as a Dynamic Library, and redeploy the Dynamic Library as needed. + + +## Proposed solution + +On Linux, there are a lot of obstacles to having fully general support for Dynamic Libraries. Swift is not ABI stable on Linux, and Linux itself is not a single platform but a wide range of similar platforms that provide few binary compatibility guarantees. This means it is pretty much impossible for a public Swift library to vend precompiled binaries that will Just Work for everyone, and we are not going to try to solve that problem in this proposal. + +Instead, we will focus on **Environment Constrained Shared Libraries** (ECSLs). We choose this term to emphasize the distinction between our use case and fully general Dynamic Libraries. + +### Target environment + +Unlike fully general Dynamic Libraries, you would distribute Environment Constrained Shared Libraries strictly for controlled consumption within a known environment, such as a fleet of servers maintained by a single organization. + +ECSLs are an advanced tool, and maintaining the prerequisite environment to deploy them safely is neither trivial nor recommended for most users. + +The organization that distributes an ECSL is responsible for defining what exactly constitutes a “platform” for their purposes. An organization-defined platform is not necessarily an operating system or architecture, or even a specific distribution of an operating system. A trivial example of two such platforms might be: + +1. Ubuntu 24.04 with the Swift 6.1.2 runtime installed at `/home/ubuntu/swift` +2. Ubuntu 24.04 with the Swift 6.1.2 runtime installed at `/home/ubuntu/swift-runtime` + +Concepts like Platform Triples are not sufficient to describe an ECSL deployment target. Even though both “platforms” above would probably share the Triple `aarch64-unknown-linux-gnu`, Swift code compiled (without `--static-swift-stdlib`) for one would never be able to run on the other. + +Organizations will add and remove environments as needed, and trying to define a global registry of all possible environments is a non-goal. + +The proposed ECSL distribution format does not support shipping multiple variants of ECSLs targeting multiple environments in the same Artifact Bundle, nor does it specify a standardized means for identifying the environment in which a particular ECSL is intended to execute in. +Users are responsible for computing the correct URL of the Artifact Bundle for the environment they are building for, possibly within the package manifest. Swift tooling will not, on its own, diagnose or prevent the installation of an incompatible ECSL. + +### Creating ECSLs + +To compile an ECSL, you just need to build an ordinary SwiftPM library product with the `-enable-library-evolution` flag. This requires no modifications to SwiftPM. + +You would package an ECSL as an `.artifactbundle` just as you would an executable, with the following differences: + +- The `info.json` must have `schemaVersion` set to `1.2` or higher. +- The artifact type must be `dynamicLibrary`, a new enum case introduced in this proposal. +- The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. +- The artifact payload must include the `.swiftinterface` file corresponding to the actual library object. + +Because SwiftPM is not (and cannot be) aware of a particular organization’s set of deployment environments, this enforces the requirement that each environment must have its own Artifact Bundle. + +The organization that distributes the ECSL is responsible for upholding ABI stability guarantees, including the exact Swift compiler and runtime versions needed to safely consume the ECSL. + + +### Consuming ECSLs + +To consume an ECSL, you would add a `binaryTarget` to your `Package.swift` manifest, just as you would for an executable. Because organizations are responsible for defining their set of supported environments, they are also responsible for defining the URLs that the Artifact Bundles for each environment are hosted under, so there are no new fields in the `PackageDescription` API. + +We expect that the logic for selecting the correct ECSL for a given environment would live within the `Package.swift` file, that it would be highly organization-specific, and that it would be manipulated using existing means such as environment variables. + + +### Deploying ECSLs + +Deploying ECSLs does not involve SwiftPM or Artifact Bundles at all. You would deploy an ECSL by copying the latest binaries to the appropriate `@rpath` location on each machine in your fleet. The `@rpath` location is part of the organization-specific environment definition, and is not modeled by SwiftPM. + +Some organizations might choose to forgo the `@rpath` mechanism entirely and simply install the ECSLs in a system-wide location. + + +## Detailed design + +### Schema extensions + +We will extend the `ArtifactsArchiveMetadata` schema to include a new `dynamicLibrary` case in the `ArtifactType` enum. + +```diff +public enum ArtifactType: String, RawRepresentable, Decodable { + case executable ++ case dynamicLibrary + case staticLibrary + case swiftSDK +} +``` + +This also bumps the latest `schemaVersion` to `1.2`. + + +### Artifact Bundle layout + +Below is an example of an `info.json` file for an Artifact Bundle containing a single library called `MyLibrary`. + +```json +{ + "schemaVersion": "1.2", + "artifacts": { + "MyLibrary": { + "type": "dynamicLibrary", + "version": "1.0.0", + "variants": [{ "path": "MyLibrary" }] + } + } +} +``` + +The artifact must have exactly one variant in the `variants` list, and the `supportedTriples` field is forbidden. An ECSL Artifact Bundle can contain multiple libraries at the top level. + +Below is an example of the layout of an Artifact Bundle containing a single library called `MyLibrary`. Only the `info.json` must appear at the root of the Artifact Bundle; all other files can appear at whatever paths are defined in the `info.json`, as long as they are within the Artifact Bundle. + +```text +📂 example.artifactbundle + 📂 MyLibrary + ⚙️ libMyLibrary.so + 📝 MyLibrary.swiftinterface + 📝 info.json +``` + +A macOS Artifact Bundle would contain a `.dylib` instead of a `.so`. ECSLs will be supported on macOS, although we expect this will be an exceedingly rare use case, as this need is already well-served by the XCFramework. + + +## Security + +ECSLs are not intended for public distribution, and are not subject to the same security concerns as public libraries. Organizations that distribute ECSLs are responsible for ensuring that the ECSLs are safe to consume. + + +## Impact on existing packages + +There will be no impact on existing packages. All Artifact Bundle schema changes are additive. + + +## Alternatives considered + +### Extending Platform Triples to model deployment targets + +SwiftPM currently uses Platform Triples to select among artifact variants when consuming executables. This is workable because it is usually feasible to build executables that are portable across the range of platforms encompassed by a single Platform Triple. + +We could extend Platform Triples to model ECSL deployment targets, but this would privilege a narrow set of predefined deployment architectures, and if you wanted to add a new environment, you would have to modify SwiftPM to teach it to recognize the new environment. + +### Supporting multiple variants of an ECSL in the same Artifact Bundle + +We could allow an Artifact Bundle to contain multiple variants of an ECSL, but we would still need to support a way to identify those variants, which in practice forces SwiftPM to become aware of organization-defined environments. + +We also don’t see much value in this feature, as you would probably package and upload ECSLs using one CI/CD workflow per environment anyway. Combining artifacts would require some kind of synchronization mechanism to await all pipelines before fetching and merging bundles. + +One benefit of merging bundles would be that it reduces the number of checksums you need to keep track of, but we expect that most organizations will have a very small number of supported environments, with new environments continously phasing out old environments. + +### Using a different `ArtifactType` name besides `dynamicLibrary` + +We intentionally preserved the structure of the `variants` list in the `info.json` file, despite imposing the current restriction of one variant per library, in order to allow this format to be extended in the future to support fully general Dynamic Libraries. diff --git a/proposals/0491-module-selectors.md b/proposals/0491-module-selectors.md new file mode 100644 index 0000000000..ad86323f3b --- /dev/null +++ b/proposals/0491-module-selectors.md @@ -0,0 +1,776 @@ +# Module selectors for name disambiguation + +* Proposal: [SE-0491](0491-module-selectors.md) +* Authors: [Becca Royal-Gordon](https://github.com/beccadax) +* Review Manager: [Freddy Kellison-Linn](https) +* Status: **Accepted** +* Bug: [swiftlang/swift#53580](https://github.com/swiftlang/swift/issues/53580) (SR-11183) +* Implementation: [swiftlang/swift#34556](https://github.com/swiftlang/swift/pull/34556) +* Review: ([pitch](https://forums.swift.org/t/pitch-module-selectors/80835)) ([review](https://forums.swift.org/t/se-0491-module-selectors-for-name-disambiguation/82124)) ([acceptance](https://forums.swift.org/t/accepted-se-0491-module-selectors-for-name-disambiguation/82589)) + +Previously pitched in: + +* [Pitch: Fully qualified name syntax](https://forums.swift.org/t/pitch-fully-qualified-name-syntax/28482) + +## Introduction + +We propose that Swift's grammar be extended so that, wherever an identifier +is written in source code to reference a declaration, it can be prefixed by +`ModuleName::` to disambiguate which module the declaration is expected to +come from. This syntax will provide a way to resolve several types of name +ambiguities and conflicts. + +## Motivation + +Swift's name lookup rules promote its goal of allowing code to be written in a +very clean, readable style. However, in some circumstances it can be very +difficult to unambiguously reference the declaration you want. + +### Background + +When Swift looks up a name in your source code to find the declaration it +refers to, that lookup can be either *qualified* or *unqualified*. Qualified +lookups are restricted to looking inside a certain declaration, while +unqualified lookups search more broadly. For example, in a chain of names such +as: + +```swift +mission().booster().launch() +``` + +`booster()` can only refer to members of whatever type is returned by +`mission()`, and `launch()` can only refer to members of whatever type is +returned by `booster()`, so Swift will find them using a qualified lookup. +`mission()`, on the other hand, does not have to be a member of some specific +type, so Swift will find that declaration using an unqualified lookup. + +> **Note**: Although the examples given here mostly concern uses in +> expressions, qualified and unqualified lookups are also used for names in +> type syntax, such as `Mission.Booster.Launch`. The exact lookup rules are +> slightly different but the principles are the same. + +Both kinds of lookups are slightly sensitive to context in that, since the +acceptance of [SE-0444 Member Import Visibility][SE-0444], +they are both limited to declarations imported in the current source file; +however, unqualified lookups take *much* more than just that into account. They +search through any enclosing scopes to find the "closest" use of that name. For +example, in code like: + +```swift +import RocketEngine +import IonThruster + +extension Mission { + struct Booster { + func launch(_ crew: Int) { + let attempt = 1 + ignite() + } + } +} +``` + +Swift will look for `ignite` in the following places: + +1. The local declarations inside `launch(_:)` +2. The parameters to `launch(_:)` +3. Instance members and generic parameters of the enclosing type `Booster` + (including its extensions, superclasses, conformances, etc.) +4. Static members and generic parameters of the enclosing type `Mission` +5. Top-level declarations in this module +6. Top-level declarations in other imported modules +7. The names of imported modules + +These rules are a little complicated when written out like this, but their +effect is pretty simple: Swift finds whichever `ignite` is in the "closest" +scope to the use site. If both `Booster` and `Mission` have an `ignite`, for +example, Swift will use the one in `Booster` and ignore the one in `Mission`. + +Of particular note is the last place Swift looks: the names of imported +modules. This is intended to help with situations where two modules have +declarations with the same name. For example, if both `RocketEngine` and +`IonThruster` declare an `ignite()`, `RocketEngine.ignite()` will find `ignite` +using a qualified lookup inside the module `RocketEngine`, filtering out the +one in `IonThruster`. This works in simple cases, but it breaks down in a +number of complicated ones. + +### Unqualified lookups are prone to shadowing + +Swift does not prevent declarations in different scopes from having the same +name. For example, there's nothing preventing you from having both a top-level +type and a nested type with the same name: + +```swift +struct Scrubber { ... } + +struct LifeSupport { + struct Scrubber { ... } +} +``` + +This means that the same name can have different meanings in different places: + +```swift +// This returns the top-level `Scrubber`: +func makeScrubber() -> Scrubber { ... } + +extension LifeSupport { + // This returns `LifeSupport.Scrubber`: + func makeScrubber() -> Scrubber { ... } +} +``` + +Specifically, we say that within the extension, `LifeSupport.Scrubber` +*shadows* the top-level `Scrubber`. + +This poses certain challenges—especially for mechanically-generated code, such +as module interface files—but it's usually not completely insurmountable +because you can qualify a top-level declaration with its module name. However, +it becomes a problem if *the module name itself* is shadowed by a type with the +same name: + +```swift +// Module RocketEngine +public struct RocketEngine { ... } +public struct Fuel { ... } + +// Another module +import RocketEngine + +_ = RocketEngine.Fuel() // Oops, this is looking for a nested type in the + // struct RocketEngine.RocketEngine! +``` + +In this situation, we can no longer qualify top-level declarations with module +names. That makes code generation *really* complicated, because there is no +syntax that works reliably—qualifying will help with some failures but cause +others. + +That may sound like a farfetched edge case, but it's surprisingly common for a +module to contain a type with the same name. For instance, the `XCTest` module +includes an `XCTest` class, which is a base class for `XCTestCase` and +`XCTestSuite`. To avoid this kind of trouble, developers must be careful to +give modules different names from the types inside them—the `Observation` +module, for example, might have been called `Observable` if it didn't have a +type with that name. + +### Qualified lookups can be unresolvably ambiguous + +Extensions create the possibility that a type may have two members with the +same name and similar or even outright conflicting overload signatures, +distinguished only by being in different modules. This is not a problem for +Swift's ABI because the mangled name of an extension member includes the +module it was declared in; however, there is no way to add a module name to +an already-qualified non-top-level lookup, so there's no way to express this +distinction in the surface language. Developers' only option may be to fiddle +with their imports in an attempt to make sure the desired member is the only +one that's visible. + +### Macros don't support module qualification + +Macros cannot have members--the grammar of a macro expansion allows only a +single identifier, and any subsequent `.` is taken to be a member lookup on +the expansion--so there is currently no way to qualify a macro expansion with +a module name. This limitation was discussed during the [second review of SE-0382][SE-0382-review-2] +and the author suggested the only viable solution was to add a new, +grammatically-distinguishable syntax for module qualification. + +### These problems afflict module interfaces, but aren't unique to them + +These issues show up most often in module interfaces because the compiler +needs to generate syntax that reliably resolves to a specific declaration, but +the rules' sensitivity to context and module contents (which might change over +time!) makes that very difficult. In practice, the compiler does not attempt to +fully account for shadowing and name conflicts--by default it qualifies names +as fully as the language allows (which works about 95% of the time) and offers +a number of (undocumented) workaround flags to adjust that which are added by a +maintainer when they discover that their module is in the remaining 5%. These +flags aren't enabled automatically, though, and they don't affect the module +interfaces of downstream modules which need to reference affected declarations. +In short, the situation is a mess. + +It's important to keep in mind, though, that this doesn't *just* affect module +interfaces and generated code. Code written by humans can also run into these +issues; it's just that a person will notice the build error and fiddle with +their code until they get something that works. It therefore makes sense to +introduce a new syntax that can be used by both machines and humans. + +### Separate modules make this uniquely severe + +While problematic conflicts can sometimes occur between two declarations in a +single module, the authors believe that per-module disambiguation is the right +approach because shadowing within a module is much easier to detect and +resolve. The developer will generally notice shadowing problems when they build +or test their code, and since they control both the declaration site and the +use site, they have options to resolve any problems that are not otherwise +available (like renaming declarations or tweaking their overload signatures). +The compiler also detects and prevents outright conflicts within a specific +module, such as two extensions declaring the exact same member, which it would +allow if the declarations were in different modules. + +## Proposed solution + +We propose adding *module selectors* to the language. A module selector is +spelled `::` and can be placed before an identifier to indicate +which module it is expected to come from: + +```swift +_ = RocketEngine::Fuel() // Picks up the `Fuel` in `RocketEngine`, bypassing + // any other `Fuel`s that might be in scope +``` + +On an unqualified lookup, a module selector also indicates that lookup should +start at the top level, skipping over the declarations in contextually-visible +scopes: + +```swift +// In module NASA + +struct Scrubber { ... } + +struct LifeSupport { + struct Scrubber { ... } +} + +extension LifeSupport { + // This returns the top-level `Scrubber` + func makeMissionScrubber() -> NASA::Scrubber { ... } +} +``` + +Module selectors may also be placed on qualified lookups to indicate which +module an extension member should belong to: + +```swift +// In module IonThruster +extension Spacecraft { + public struct Engine { ... } +} + +// In module RocketEngine +extension Spacecraft { + public struct Engine { ... } +} + +// In module NASA +import IonThruster +import RocketEngine + +func makeIonThruster() -> Spacecraft.IonThruster::Engine { ... } +``` + +Module selectors are permitted at locations in the type and expression syntax +where a declaration from elsewhere is referenced by name. However, it is +invalid to use one on the name of a *new* declaration: + +```swift +struct NASA::Scrubber { // Invalid--new declarations are always in the current module + ... +} +``` + +We chose this syntax—module name plus `::` operator prefixing the name they +qualify—because `::` is unused in Swift (it can't even be a custom operator) +and because using `::` in this fashion is highly precedented in other +languages. (C++, PHP, Java, and Rust all use it to indicate that the name on +the right should be looked up inside the scope on the left; Ruby and Perl use +it *specifically* to look up declarations inside modules.) + +## Detailed design + +### Grammar and parsing + +A module selector has the following grammar: + +> *module-selector* → *identifier* `::` + +The following productions may now optionally include a module selector (changes are in bold): + +> *type-identifier* → ***module-selector?*** *type-name* *generic-argument-clause?* | ***module-selector?*** *type-name* *generic-argument-clause?* `.` *type-identifier* +> +> *primary-expression* → ***module-selector?*** *identifier* *generic-argument-clause?* +> +> *implicit-member-expression* → `.` ***module-selector?*** *identifier*
+> *implicit-member-expression* → `.` ***module-selector?*** *identifier* `.` *postfix-expression* +> +> *macro-expansion-expression* → `#` ***module-selector?*** *identifier* *generic-argument-clause?* *function-call-argument-clause?* *trailing-closures?* +> +> *key-path-component* → ***module-selector?*** *identifier* *key-path-postfixes?* | *key-path-postfixes* +> +> *function-call-argument* → ***module-selector?*** *operator* | *identifier* `:` ***module-selector?*** *operator* +> +> *initializer-expression* → *postfix-expression* `.` ***module-selector?*** `init`
+> *initializer-expression* → *postfix-expression* `.` ***module-selector?*** `init` `(` *argument-names* `)` +> +> *explicit-member-expression* → *postfix-expression* `.` ***module-selector?*** *identifier* *generic-argument-clause?*
+> *explicit-member-expression* → *postfix-expression* `.` ***module-selector?*** *identifier* `(` *argument-names* `)` +> +> *attribute-name* → ***module-selector?*** *identifier* +> +> *enum-case-pattern* → *type-identifier?* `.` ***module-selector?*** *enum-case-name* *tuple-pattern?* + +Additionally, a new production allows a scoped `import` declaration to use a +module selector and identifier instead of an import path: + +> *import-declaration* → *attributes?* `import` *import-kind?* *import-path*
+> ***import-declaration* → *attributes?* `import` *import-kind* *module-selector* *identifier*** + +Note that this new *import-declaration* production does not allow a submodule +to be specified. Use the old `.`-operator-based syntax for submodules. + +#### Token-level behavior + +The `::` operator may be separated from its *identifier* by any whitespace, +including newlines. However, the `::` operator must *not* be separated from the +token after it by a newline: + +```swift +NationalAeronauticsAndSpaceAdministration:: + RocketEngine // Invalid +NationalAeronauticsAndSpaceAdministration + ::RocketEngine // OK +``` + +> **Note**: This restriction aids in recovery when parsing incomplete code; +> the member-lookup `.` operator follows a similar rule. + +If the token after the `::` operator is a keyword, it will be treated as an +ordinary identifier unless it would have special meaning: + +```swift +print(default) // Invalid; 'default' is a keyword and needs backticks +print(NASA.default) // OK under SE-0071 +print(NASA::default) // OK under this proposal +``` + +Depending on context, the following keywords may still be treated as special in +expressions: + +* `deinit` +* `init` +* `subscript` + +> **Note**: This behavior is analogous to [SE-0071 Allow (most) keywords in member references][SE-0071]. + +Similarly, attributes that use a module selector will always be treated as +custom attributes, not built-in attributes. (Put another way, built-in +attributes do not belong to *any* module—not even `Swift`.) Like all custom +attributes, any arguments must be valid expressions. + +```swift +@Swift::available(macOS 15.0.1, *) // Invalid; not parsed as the built-in `@available` +class X {} +``` + +#### Patterns + +Module selectors are allowed in *enum-case-pattern* and in *type* and +*expression* productions nested inside patterns. However, *identifier-pattern* +is unmodified and does *not* permit a module selector, even in shorthand +syntaxes designed to declare a shadow of an existing variable. If a module +selector is needed, you must use an explicit initializer expression. + +```swift +if let NASA::rocket { ... } // Invalid +if let rocket = NASA::rocket { ... } // OK + +Task { [NASA::rocket] in ... } // Invalid +Task { [rocket = NASA::rocket] in ... } // OK +``` + +#### Operator and precedence group declarations + +The *precedence-group-name* production is unmodified and does not permit +a module selector. Precedence group names exist in a separate namespace from +other identifiers and no need for this feature has been demonstrated. + +#### Parsed declaration names + +A parsed declaration name, such as the name in an `@available(renamed:)` +argument, may use module selectors on the declaration's base name and context +names. + +```swift +@available(*, deprecated, renamed: "NASA::launch(_:from:)") // OK +public func launch(_ mission: Mission) { + launch(mission, from: LaunchPad.default) +} +``` + +Module selectors are not valid on base names in clang `swift_name` and +`swift_async_name` attributes, since these specify the name of the current +declaration, rather than referencing a different declaration. + +> **Note**: Clang Importer currently cannot apply import-as-member `swift_name` +> or `swift_async_name` attributes that name a context in a different module, +> but if this limitation is ever lifted, module selectors ought to be supported +> on context names in these clang attributes. + +#### Syntaxes reserved for future directions + +It is never valid to write two module selectors in a row; if you want to access +a declaration which belongs to a clang submodule, you should just write the +top-level module name in the module selector. + +It is never valid to write a keyword, operator, or `_` in place of a module +name; if a module's name would be mistaken for one of these, it must be +wrapped in backticks to form an identifier. + +### Effects on lookup + +When a reference to a declaration is prefixed by a module selector, only +declarations declared in, or re-exported by, the indicated module will be +considered as candidates. All other declarations will be filtered out. + +For example, in the following macOS code: + +```swift +import Foundation + +class NSString {} + +func fn(string: Foundation::NSString) {} +``` + +`string` will be of type `Foundation.NSString`, rather than the `NSString` +class declared in the same file. Because the AppKit module +re-exports Foundation, this example would also behave the same way: + +```swift +import AppKit + +class NSString {} + +func fn(string: AppKit::NSString) {} +``` + +> **Note**: Allowing re-exports ensures that "hoisting" a type from its +> original module up to another module it imports is not a source-breaking +> change. It also helps with situations where developers don't realize where a +> given type is declared; for instance, many developers believe `NSObject` is +> declared in `Foundation`, not `ObjectiveC`. + +Additionally, when a reference to a declaration prefixed by a module selector +is used for an unqualified lookup, the lookup will begin at the module-level +scope, skipping any intervening enclosing scopes. That means a top-level +declaration will not be shadowed by local variables, parameters, generic +parameters, or members of enclosing types: + +```swift +// In module MyModule + +class Shadowed { + struct Shadowed { + let Shadowed = 42 + func Shadowed(Shadowed: () -> Void) { + let Shadowed = "str" + let x = MyModule::Shadowed() // refers to top-level `class Shadowed` + } + } +} +``` + +A module selector can only rule out declarations that might otherwise have been +chosen instead of the desired declaration; it cannot access a declaration which +some other language feature has ruled out. For example, if a declaration is +inaccessible because of access control or hasn't been imported into the current +source file, a module selector will not allow it to be accessed. + +#### Member types of type parameters + +A member type of a type parameter must not be qualified by a module selector. + +```swift +func fn(_: T) where T.Swift::ID == Int { // not allowed + ... +} +``` + +This is because, when a generic parameter conforms to two protocols that have +associated types with the same name, the member type actually refers to *both* +of those associated types. It doesn't make sense to use a module name to select +one associated type or the other--it will always encompass both of them. + +(In some cases, a type parameter's member type might end up referring to a +concrete type—typically a typealias in a protocol extension–which +theoretically *could* be disambiguated in this way. However, in these +situations you could always use the protocol instead of the generic parameter +as the base (and apply a module selector to it if needed), so we've chosen not +to make an exception for them.) + +## Source compatibility + +This change is purely additive; it only affects the behavior of code which uses +the new `::` token. In the current language, this sequence can only appear +in valid Swift code in the selector of an `@objc` attribute, and the parser +has been modified to split the token when it is encountered there. + +## ABI compatibility + +This change does not affect the ABI of existing code. The Swift compiler has +always resolved declarations to a specific module and then embedded that +information in the ABI's symbol names; this proposal gives developers new ways +to influence those resolution decisions but doesn't expand the ABI in any way. + +## Implications on adoption + +Older compilers will not be able to parse source code which uses module +selectors. This means package authors may need to increase their tools version +if they want to use the feature, and authors of inlinable code may need to +weigh backwards compatibility concerns. + +Similarly, when a newer compiler emits module selectors into its module +interfaces, older compilers won't be able to understand those files. This isn't +a dealbreaker since Swift does not guarantee backwards compatibility for module +interfaces, but handling it will require careful staging and there may be a +period where ABI-stable module authors must opt in to emitting module +interfaces that use the feature. + +## Future directions + +### Special syntax for the current module + +We could allow a special token, or no token, to be used in place of the module +name to force a lookup to start at the top level, but not restrict it to a +specific module. Candidates include: + +```swift +Self::ignite() +_::ignite() +*::ignite() +::ignite() +``` + +These syntaxes have all been intentionally kept invalid (a module named `Self`, +for instance, would have to be wrapped in backticks: `` `Self`::someName ``), +so one of them can be added later if there's demand for it. + +### Disambiguation for subscripts + +There is currently no way to add a module selector to a use of a subscript. We +could add support for a syntax like: + +```swift +myArray.Swift::[myIndex] +``` + +### Disambiguation for conformances + +Retroactive conformances have a similar problem to extension members—the ABI +distinguishes between otherwise identical conformances in different modules, +but the surface syntax has no way to resolve any ambiguity—so a feature which +addressed them might be nice. However, there is no visible syntax associated +with use of a conformance that can be qualified with a module selector, so it's +difficult to address as part of this proposal. + +It's worth keeping in mind that [SE-0364's introduction of `@retroactive`][SE-0364] +reflects a judgment that retroactive conformances should be used with care. The +absence of such a feature is one of the complications `@retroactive` is meant +to flag. + +### Support selecting conflicting protocol requirements + +Suppose that a single type conforms to two protocols with conflicting protocol +requirements: + +```swift +protocol Employable { + /// Terminate `self`'s employment. + func fire() +} + +protocol Combustible { + /// Immolate `self`. + func fire() +} + +struct Technician: Employable, Combustible { ... } +``` + +It'd be very useful to be able to unambiguously specify which protocol's +requirement you're trying to call: + +```swift +if myTechnician.isGoofingOff { + myTechnician.Employable::fire() +} +if myTechnician.isTooCloseToTheLaunch { + myTechnician.Combustible::fire() +} +``` + +However, allowing a protocol name—rather than a module name—to be written +before the `::` token re-introduces the same ambiguity this proposal seeks +to solve because a protocol name could accidentally shadow a module name. +We'll probably need a different feature with a distinct syntax to resolve +this use case—perhaps something like: + +```swift +if myTechnician.isGoofingOff { + (myTechnician as some Employable).fire() +} +``` + +### Support selecting default implementations + +Similarly, it would be useful to be able to specify that you want to call a +default implementation of a protocol requirement even if the conformance +provides another witness. (This could be used similarly to how `super` is used +in overrides.) However, this runs into similar problems with reintroducing +ambiguity, and it also just doesn't quite fit the shape of the syntax (there's +no name to uniquely identify the default implementation you want). Once again, +this probably requires a different feature with a distinct syntax—perhaps +something a little more like how `super` works. + +## Alternatives considered + +### Change lookup rules in module interfaces + +Some of the problems with module interfaces could be resolved by changing the +rules for qualified lookup *within module interface files specifically*. For +instance, we could decide that in a module interface file, unqualified lookups +can only find module names, and the compiler must always qualify every name +with a module name. + +This would probably be a viable solution with enough effort, but it has a +number of challenges: + +1. There are some declarations—generic parameters, for instance—which are + not accessible through any qualified lookup (they are neither top-level + declarations nor accessible through the member syntax). We would have to + invent some way to reference these. + +2. Existing module interfaces have already been produced which would be broken + by this change, so it would have to somehow be made conditional. + +3. Currently, inlinable function bodies are not comprehensively regenerated + for module interfaces; instead, the original textual source code is + inserted with minor programmatic edits to remove comments and `#if` blocks. + This means we would have to revert to the normal lookup rules within an + inlinable function body. + +It also would not help with ambiguous *qualified* lookups, such as when two +modules use extensions to add identically-named nested types to a top-level +type, and it would not give developers new options for handling ambiguity in +human-written code. + +### Add a fallback lookup rule for module name shadowing + +The issue with shadowing of module names could be addressed by adding a narrow +rule saying that, when a type has the same name as its enclosing module and a +qualified lookup inside it doesn't find any viable candidates, Swift will fall +back to looking in the module it shadowed. + +This would address the `XCTest.XCTestCase` problem, which is the most common +seen in practice, but it wouldn't help with more complicated situations (like +a nested type shadowing a top-level type, or a type in one module having the +same name as a different module). It's also not a very principled rule and +making it work properly in expressions might complicate the type checker. + +### Use a syntax involving a special prefix + +We considered creating a special syntax which would indicate unambiguously that +the next name must be a module name, such as `#Modules.FooKit.bar`. However, +this would only have helped with top-level declarations, not members of +extensions. + +### Use a syntax that avoids `::`'s shortcomings + +Although we have good reasons to propose using the `::` operator (see "Proposed +solution" above), we do not think it's a perfect choice. It appears visually +"heavier" than the `.` operator, which means developers reading the code might +mentally group the identifiers incorrectly: + +```swift +Mission.NASA::Booster.Exhaust // Looks like it means `(Mission.NASA) :: (Booster.Exhaust)` + // but actually means `Mission . (NASA::Booster) . Exhaust` +``` + +This is not unprecedented—in C++, `myObject.MyClass::myMember` means +`(myObject) . (MyClass::myMember)`—but it's awkward for developers without +a background in a language that works like this. + +We rejected a number of alternatives that would avoid this problem. + +#### Make module selectors qualify different names + +One alternative would be to have the module selector qualify the *rightmost* +name in the member chain, rather than the leftmost, so that a module selector +could only appear at the head of a member chain. The previous example would +then be written as: + +```swift +(NASA::Mission.Booster).Exhaust +``` + +We don't favor this design because we believe: + +1. Developers more frequently need to qualify top-level names (which exist in a + very crowded quasi-global namespace) than member names (which are already + limited by the type they're looking inside); a syntax that makes qualifying + the member the default is optimizing for the wrong case. + +2. The distance between the module and the identifier it qualifies increases + the cognitive burden of pairing up modules to the names they apply to. + +3. Subjectively, it's just *weird* that the selector applies to a name that's a + considerable distance from it, rather than the name immediately adjacent. + +A closely related alternative would be to have the module selector qualify +*all* names in the member chain, so that in `(NASA::Mission.Booster).Exhaust`, +both `Mission` and `Booster` must be in module `NASA`. We think point #1 from +the list above applies to this design too: `Mission` is a sparse enough +namespace that developers are more likely to be hindered by `Booster` being +qualified by the `NASA` module than helped by it. + +#### Use a totally different spelling + +We've considered and rejected a number of other spellings for this feature, +such as: + +```swift +Mission.#module(NASA).Booster.Exhaust // Much wordier, not implementable as a macro +Mission.::NASA.Booster.Exhaust // Looks weird in member position +Mission.Booster@NASA.Exhaust // Ignores Swift's left-to-right nesting convention +Mission.'NASA.Booster.Exhaust // Older compilers would mis-lex in inactive `#if` blocks +Mission.NASA'Booster.Exhaust // " +Mission.(NASA)Booster.Exhaust // Arbitrary; little connection to prior art +Mission.'NASA'.Booster.Exhaust // " +``` + +### Don't restrict whitespace to the right of the `::` + +Allowing a newline between `::` and the identifier following it would mean +that, when an incomplete line of code ended with a module selector, Swift might +interpret a keyword on the next line as a variable name, likely causing multiple +confusing syntax errors in the next statement. For instance: + +```swift +let x = NASA:: // Would be interpreted as `let x = NASA::if x { ... }`, +if x { ... } // causing several confusing syntax errors. +``` + +Forbidding it, however, has a cost: it restricts the code styles developers can +use. If a developer wants to put a line break in the middle of a name with a +module selector, they will not be able to format it like this: + +```swift +SuperLongAndComplicatedModuleName:: + superLongAndComplicatedFunctionName() +``` + +And will have to write it like this instead: + +```swift +SuperLongAndComplicatedModuleName + ::superLongAndComplicatedFunctionName() +``` + +The member-lookup `.` operator has a similar restriction, but developers may +not want to style them in exactly the same way, particularly since C++ +developers often split a line after a `::`. + + [SE-0071]: "Allow (most) keywords in member references" + [SE-0364]: "Warning for Retroactive Conformances of External Types" + [SE-0382-review-2]: "SE-0382 (second review): Expression Macros" + [SE-0444]: "SE-0444 Member Import Visibility" diff --git a/proposals/0492-section-control.md b/proposals/0492-section-control.md new file mode 100644 index 0000000000..9cb160550f --- /dev/null +++ b/proposals/0492-section-control.md @@ -0,0 +1,478 @@ +# Section Placement Control + +* Proposal: [SE-0492](0492-section-control.md) +* Authors: [Kuba Mracek](https://github.com/kubamracek) +* Status: **Active review (September 22 ... October 14, 2025)** +* Implementation: available in recent `main` snapshots under the experimental feature `SymbolLinkageMarkers` and with undercored attribute names `@_section` and `@_used`. +* Review: [review](https://forums.swift.org/t/se-0492-section-placement-control/82289) +* Discussion threads: + * Pitch #1: https://forums.swift.org/t/pitch-low-level-linkage-control-attributes-used-and-section/65877 + * Pitch #2: https://forums.swift.org/t/pitch-2-low-level-linkage-control/69752 + * Pitch #3: https://forums.swift.org/t/pitch-3-section-placement-control/77435 + +## Introduction + +This proposal adds `@section` and `@used` attributes which can be applied to global variables. These allow users to directly control which section of the resulting binary should global variables be emitted into, and give users the ability to disable DCE (dead code elimination) on those. The goal is to enable systems and embedded programming use cases like runtime discovery of test metadata from multiple modules, and also to serve as a low-level building block for higher-level features (e.g. linker sets, plugins). + +## Motivation + +**Testing frameworks** need to be able to produce test metadata about user’s types and other declarations (e.g. standalone test entrypoints) in a way that they are discoverable and enumerable at runtime. In dynamic languages like Objective-C, this is typically done at runtime using reflection, by querying the language runtime, and/or walking lists of types or exported symbols: + +```swift +// MyXCTestModule +@objc class TableValidationTests: XCTestCase { + func test1() { ... } +} + +// Testing framework, pseudo-code +let classList = objc_copyClassList(...) +for aClass in classList { + if aClass is XCTestCase { + let methodList = class_copyMethodList(aClass) + ... + } +} +``` + +A similarly dynamic approach was proposed in [SE-0385 (Custom Reflection Metadata)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0385-custom-reflection-metadata.md), but was rejected because for Swift, a more static approach would be a better fit: If Swift code had the ability to produce custom metadata directly into the resulting binaries in a well-understood way, we would be able to directly access the data at runtime via platform loader’s APIs and also use offline binary inspection tools (such as objdump, objcopy, otool). This would be more efficient and not require the language runtime and thus also be palatable in embedded use cases using Embedded Swift. + +Mainstream operating systems support dynamically loading modules from disk at runtime, and large applications tend to build **plugin systems** as a way to separate the development of subsystems, or to support separate compilation of 3rd party code. Loading a module from disk can be done using standard APIs like `dlopen`, but discovering and calling the interface in the plugin usually requires using unsafe C constructs (dlsym, casting of pointers) and/or querying the language runtime for type information, similarly to the testing enumerating approach mentioned above. A better approach could publish the information about a plugin in a structured way into the binary, and a runtime component could locate this metadata, and provide access to it in a type-safe way. + +This proposal recommends to use sections of the various object file formats as the vehicle for custom metadata produced by Swift code. This approach is a good fit to the above mentioned use cases, but also enables others: + +* “**Linker sets**” are an approach in systems programming that collects data from multiple source files or subsystems using a standard linker behavior of collocating symbols that belong to the same section. In principle, this is simply a generalization of the test enumeration and plugin discovery use cases mentioned above. The primary goal is decentralization of the information, for example, linker sets can be used to describe memory requirements of each subsystem (and a boot step at runtime can process those to figure out how much heap should be made available). +* Emitting custom metadata into binaries can be used to convey **information to the debugger**. The `@DebugDescription` macro generates a “summary string” for LLDB that summarizes the contents of the fields of a type without the need for runtime evaluation, but rather as a composition of the fields that LLDB assembles. To make those summary strings discoverable by LLDB, placing them into a custom section is a clean solution allowing LLDB to consume them in the case where LLDB has access to the binary on disk, or even without that. Embedded programs might need to rely on such a mechanism as the only way to get enhanced data visualization in the debugger, because runtime evaluation from a debugger is commonly not possible at all in firmware. + +```swift +@DebugDescription struct Student { + var name: String + var id: Int + + /* synthesized by the @DebugDescription macro, made discoverable by LLDB */ + let __Student_lldb_summary = ("PupilKit.Student", "${var.id}: ${var.name}") +} +``` + +* More embedded and systems programming use cases often require directly control of section placement as well, for example to adhere to a **startup contract** with platform libraries, SDK’s linker scripts or the hardware itself. Such contract can be pre-existing in the platform and require placing a specific data structure into a specific section. Enabling doing that directly in Swift will provide a more intuitive and safer implementation option, and users of Swift for embedded devices won’t need to reach for C/C++ as a workaround. + +## Proposed Solution + +The proposal is to add two new attributes `@section` and `@used` that will allow annotating global and static variables with directives to place the value into a custom section, and to require no-dead-stripping aka "attribute used". Using `@section` requires that the initializer expression is a constant expression (see [Constant expressions](#constant-expressions) below for the definition of that): + +```swift +// Place an entry into a section, mark as "do not dead strip". +// Initializer expression must be a constant expression. +// The global variable is implicitly made statically initialized. +@section(MachO: "__DATA,mysection") +@used +let myLinkerSetEntry: Int = 42 // ✅ + +// Non-constant or statically non-initializable expressions are disallowed +@section(MachO: "__DATA,mysection") +let myLinkerSetEntry: Int = Int.random(in: 0 ..< 10) // ❌ error + +// Section-placed globals can be "var", the initializer expression still must be constant +@section(MachO: "__DATA,mysection") +var mutableVariable: Int = 42 // ✅ + +// Some complex data types are allowed (tuples, function references) +typealias PluginData = (version: Int, identifier: UInt64, initializer: @convention(c) ()->()) + +@section(MachO: "__DATA,plugins") +@used +let myPlugin: PluginData = ( + version: 1, + initializer: { print("init") } +) +``` + +On top of specifying a custom section name with the `@section` attribute, marking a variable as `@used` is needed to prevent otherwise unused symbols from being removed by the compiler. When using section placement to e.g. implement linker sets, such values are typically going to have no usage at compile time, and at the same time they should not be exposed in public interface of libraries (not be made public), therefore we the need the `@used` attribute. + +Custom section names can be specified as a string literal. The string will be used directly, without any processing, as the section name for the symbol. Different object file formats (ELF, Mach-O, COFF) have different restrictions and rules on what are valid section names, and cross-platform code will have to use different names for different file formats. To support that, the `@section` attribute will allow specifying one or more section names per object file formats, optionally with a fallback: + +```swift +@section(MachO: "__DATA,mysection") var global = ... // ok when compiling for Mach-O, error if compiling for ELF +@section(MachO: "__DATA,mysection", ELF: "mysection") var global = ... +@section(MachO: "__DATA,mysection", default: "mysection") var global = ... +``` + +A new `#if objectFormat(...)` conditional compilation directive will be provided to support conditionalizing based on the file format: + +```swift +#if objectFormat(MachO) +@section(MachO: "__DATA,mysection") // Custom section on Mach-O, normal placement otherwise +#endif +var global = ... +``` + +For the ELF file format specifically, the compiler will also emit a “section index” into produced object files, containing an entry about each custom section used in the compilation. This is a solution to an ELF specific problem where the behavior of ELF linkers and loaders means that sections are not easily discoverable at runtime. + +> Note: The intention is that the `@section` and `@used` attributes are to be used rarely and only by specific use cases; high-level application code should not need to use them directly and instead should rely on libraries, macros and other abstractions over the low-level attributes. + +> The scope of this proposal is limited to compile-time behavior and compile-time control. We expect that full user-facing solutions for features like linker sets, test discovery or plugins will also require runtime implementations to discover and iterate the contents of custom sections, possibly from multiple modules. This proposal makes sure to provide the right building blocks and artifacts in binaries for the runtime components, but doesn’t prescribe the shape of those. However, it is providing a significant step towards generalized and safe high-level mechanisms for those use cases. See the discussion in [Runtime discovery of data in custom sections](#runtime-discovery-of-data-in-custom-sections) and [Linker sets, plugins as high-level APIs](#linker-sets-plugins-as-high-level-apis) in Future Directions. + +## Detailed design + +### Attributes @section and @used on global and static variables + +Two new attributes are to be added: `@section`, with a labeled list of arguments specifying the section name(s), and a argument-less `@used` attribute. The attributes can be used either together or independently. + +On `@section`, the arguments are labeled by object file format type (see [Cross-platform object file format support](#cross_platform_object_file_format_support) for the full list), there must be at least one labeled argument present, and optionally there can be a `default:` argument with a fallback section. The section names must be string literals. If compiling for a object file format that is not listed, and no `default:` argument is present, the compilation fails. + +```swift +// (1) +@section(MachO: "__DATA,mysection") +@used +let global = ... // ✅ + +// (2) +@section(MachO: "__DATA,mysection") +let global = ... // ✅ + +// (3) +@used +let global = ... // ✅ + +// (4) +@section let global = ... // ❌ no argument +@section(abc: "section") let global = ... // ❌ "abc" not a valid object file format +@section(ELF: "section") let global = ... // ok if compiling for ELF, compilation failure otherwise +@section(ELF: "section", default: "default_section") let global = ... // ✅ +``` + +The new attributes (`@section` and `@used`) can be used on variable declarations under these circumstances: + +* the variable must be a global variable or a static member variable (no local variables, no non-static member variables) +* the variable must not be declared inside a generic context (either directly in generic type or nested in a generic type) +* the variable must be a stored property (not a computed property) +* the variable must not have property observers (didSet, willSet) +* the initial expression assigned to the variable must be a constant expression, and it must be eligible for static initilization + +> *Note: These restrictions limit the `@section` and `@used` attributes to only be allowed on variables that are expected to be represented as exactly one statically-initialized global storage symbol (in the linker sense) for the variable’s content. This is generally true for all global and static variables in C and C++, but in Swift global variables might have zero global storage symbols (e.g. a computed property), or need non-trivial storage (e.g. lazily initialized variables with runtime code in the initializer expression).* + +```swift +@section(MachO: "__DATA,mysection") @used +let global = 42 // ✅ + +@section(MachO: "__DATA,mysection") @used +var global = 42 // ✅ + +@section(MachO: "__DATA,mysection") @used +var computed: Int { return 42 } // ❌ ERROR: @section cannot be used on computed properties + +struct MyStruct { + @section(MachO: "__DATA,mysection") @used + static let staticMemberLet = 42 // ✅ + + @section(MachO: "__DATA,mysection") @used + static var staticMemberVar = 42 // ✅ + + @section(MachO: "__DATA,mysection") @used + let member = 42 // ❌ ERROR: @section cannot be used on non-static members + + @section(MachO: "__DATA,mysection") @used + var member = 42 // ❌ ERROR: @section cannot be used on non-static members +} + +struct MyGenericStruct { + @section(MachO: "__DATA,mysection") @used + static let staticMember = 42 // ❌ ERROR: @section cannot be used in a generic context + + @section(MachO: "__DATA,mysection") @used + static var staticMember = 42 // ❌ ERROR: @section cannot be used in a generic context +} +``` + +When allowed, the `@section` attribute on a variable declaration has the following effects: + +1. The variable’s initializer expression is going to be constant folded at compile-time, and assigned as the initial value to the storage symbol for the variable, i.e. the variable will be **statically initialized**. The variable’s value will not be lazily computed at runtime, and it will not use the one-time initialization helper code and token. If that’s not possible, an error is diagnosed. +2. The storage symbol for the variable will be placed into a custom section with the specified name. + - Concretely, the section name string value will be set verbatim as a section specifier for the storage symbol at the LLVM IR level of the compiler. This means that any special behavior that the optimizer, the backend, the assembler or the linker applies based on known section names (or attributes specified as suffixes on the section name) will apply. +3. If applied to a global that is declared as part of top-level executable code (i.e. main.swift), the usual non-top-level-code initialization behavior is applied to the global. I.e. the variable is not sequentially initialized at startup. + +The custom section name specified in the `@section` attribute is not validated by the compiler, instead it’s passed directly as a string to the linker. + +When allowed, the `@used` attribute on a variable declaration has the following effect: + +1. The storage symbol for the variable will be marked as “do not dead-strip”. + +The effects described above are applied to the storage symbols and don’t generally affect optimizations and other transformations in the compiler. For example, the compiler is still allowed to propagate and copy a constant value of a `let` variable to code that uses the variable, therefore there’s no guarantee that a value stored into a global with a custom section will not be propagated and “leak” outside of the section. The `@used` annotation, however, does inform the optimizer that such a variable cannot be removed, even when it doesn’t have any observed users or even if it’s inaccessible due to language rules (e.g. if it’s a private static member on an otherwise empty type). + +### Constant expressions + +Swift currently does not have a formal notion of a **constant expression**, i.e. an expression with a syntactic form that *guarantees the ability to know it's value at compile-time*. This proposal provides a definition of a "bare minimum" constant expression, with the understanding that this does not cover the language needs in generality, and with the expectation that the Swift compiler and language will keep expanding the allowed forms of constant expressions in the future. See [Generalized constant values and expressions](#generalized-constant-values-and-expressions) in Future Directions for further discussion on this. + +This proposal defines a **constant expression** as being one of: + +- an integer literal using any of standard integer types (Int, UInt, Int8/16/32/64/128, UInt8/16/32/64/128) +- a floating-point literal of type Float or Double +- a boolean literal of type Bool +- a direct reference to a non-generic function using its name (the function itself is not generic, and also it must not be defined in a generic context) +- a direct reference to a non-generic metatype using the type name directly (the type itself is not generic, and also it must not be defined in a generic context), where the type is non-resilient +- a tuple composed of only other constant expressions +- an array literal of type InlineArray composed of only other constant expressions + +Explicitly, this definition currently does **not allow** any operators, using any user-defined named types, any other standard type (e.g. strings, dictionaries, sets), using closures, or referencing any variables by name. See below for examples of valid and invalid constant expressions: + +```swift +@section(...) let a = 42 // ✅ +@section(...) let b = 3.14 // ✅ +@section(...) let c = 1 + 1 // ❌ operators not allowed +@section(...) let d = Int.max // ❌ not a literal +@section(...) let e: UInt8 = 42 // ✅ +@section(...) let f = UInt8(42) // ❌ not a literal +@section(...) let g: MyCustomExpressibleByIntegerLiteral = 42 // ❌ not a standard type + +@section(...) let composition1 = (1, 2, 3, 2.718, true) // ✅ +@section(...) let composition2 = (1, 2, Int.max) // ❌ tuple component not constant +@section(...) let composition3: InlineArray = [1, 2, 3] // ✅ +@section(...) let composition4: InlineArray = [1, 2, Int.max] // ❌ array component not constant +@section(...) let composition5: (Int, [1 of Int], [1 of (Int, Int)]) = (1, [1], [(1, 1)]) // ✅ + +func foo() -> Int { return 42 } +@section(...) let func1 = foo // ✅ +@section(...) let func2 = foo() // ❌ not a function reference +@section(...) let func3 = Bool.random // ✅ +@section(...) let func4 = Bool.self.random // ❌ not a direct reference +@section(...) let func5 = (Bool.self as Bool.Type).random // ❌ not a direct reference +@section(...) let func6 = [Int].randomElement // ❌ generic +@section(...) let func7 = { } // ❌ not using name + +struct S { } +@section(...) let metatype1 = S.self // ✅ +@section(...) let metatype2 = Int.self // ✅ +@section(...) let metatype3 = Int.self.self // ❌ not a direct reference +import Foundation +@section(...) let metatype4 = URL.self // ❌ resilient +``` + +### Guaranteed static initialization + +Using attribute `@section` requires the initializer expression of the variable to be a **constant expression**. It's not required to separately annotate the expression for being a compile-time expression, instead this is implied from the `@section` attribute. On top of the constant-ness, `@section` on a global or static variable enforces **static initialization** on that variable. + +We consider the variable to be eligible for static initialization when: + +1. the initializer expression represents a valid compile-time constant, and +2. the initializer expression can be constant-folded into a representation that does not require any runtime initialization (pointer relocations/fixups done automatically by the loader are not considered runtime initialization for this purpose). + +Not all constant expressions are necessarily statically initializable. For section placement we require the stronger property (static initialization) because we expect the data to be readable without any runtime mechanisms (i.e. reading raw bytes from the section at runtime, or offline binary inspection). + +```swift +@section(MachO: "__DATA,mysection") +let a = 42 // ✅ + +@section(MachO: "__DATA,mysection") +let sectionPlaced = ...expression... // ✅, guaranteed to be statically initialized + +@section(MachO: "__DATA,mysection") +let notStaticallyInitializable = ...expression that cannot be statically initialized... // ❌ +``` + +> *Note: As of this writing, all valid constant values are also eligible to be statically initialized, but we don’t expect that to hold in the future. So it’s important to distinguish between (a) a global variable being initialized with a language-level constant value, and (b) a global variable that is guaranteed to be statically initialized. The difference can be subtle, and in some cases immaterial in practice, but future enhancements of constant values in Swift might take advantage of this difference — i.e. not all constant values are going to be statically initializable. Consider the following example: If a future language versions allows dictionary literals to be constant values, such values might not be statically initializable because of randomized hash seeding:* + +> ```swift +> let d1 = ["a": 42, "b": 777] // constant, but not statically initializable +> let d2 = d1.count // statically initializable +> ``` + +> *However, using a statically non-initializable value in an expression does not preclude the outer expression from being statically initialized either. In this example, `d1` would not be allowed to be placed into a custom section because it’s not statically initializable. But `d2` could still potentially be statically initializable (even though the definition of `d2` uses a sub-expression that is not statically initializable), as it’s simply an integer.* + +As described in [Swift Compile-Time Values](https://github.com/artemcm/swift-evolution/blob/const-values/proposals/0nnn-const-values.md), values of function types are eligible for being compile-time evaluable. Their concrete pointer value is not fully known until link-time or program load-time (depending on type of linking, ASLR, PAC, etc.). For the purposes of guaranteed static initialization, function values are statically initialized into a function pointer. This pointer is still subject to normal linking and loading resolutions and fixups. + +```swift +func foo() { ... } + +@section(MachO: "__DATA,mysection") +let a = (42, foo) // "foo" is statically initialized into a + // linkable/relocatable pointer +``` + +### Cross-platform object file format support + +In some cases, it’s useful to conditionalize code to support multiple object file formats, for example when using Embedded Swift to target baremetal systems without any OS. For that, a new `#if objectFormat(...)` conditional compilation directive will be provided. The allowed values in this directive will match the set of supported object file formats by the Swift compiler (and expand as needed in the future). Currently, they exact values will be (case sensitive): + +* COFF +* ELF +* MachO +* Wasm + +```swift +#if objectFormat(MachO) || objectFormat(ELF) +@section(MachO: "__DATA_CONST,mysection", ELF: "mysection") +#endif +let value = ... // the value gets a custom section on MachO and ELF, but not on other object file formats +``` + +### ELF section index + +The goal of placing metadata into custom section is to make them discoverable both via offline inspection (e.g. objdump or otool) and at runtime. The facilities for that are dependent on the type of linking (static vs dynamic), and platform’s linker and loader: + +* For **static linking**, the bounds of a section can be statically determined by the linker and on all supported platforms and their file formats (COFF, ELF, MachO, Wasm), the linker-provided “**encapsulation symbols**” can be used to retrieve those bounds. + * In ELF and Wasm formats, these symbols are `__start_
` / `__stop_
`. + * In Mach-O, these symbols are `section$start$$` / `section$end$$`. + * In COFF, these symbols need to be manually constructed by using “grouped sections” (section name is suffixed with a $ + string) which are automatically lexicographically ordered by the linker. For example, by manually placing a start symbol into `.section$A` , end symbol into `.section$C` and all actual section entries into `.section$B`, the two helper symbols’s addresses effectively describe the bounds of the section. +* For **dynamic linking**, the above mentioned encapsulation symbols are available too, but they always only describe the bounds of the section in the current module. Retrieving section content process-wide means collecting metadata from multiple images at runtime, which requires further assistance or support from the loader. + * In Mach-O (Darwin OS's), image headers are present in the address space, and they include section bounds information. The loader provides straightforward image iteration APIs (`_dyld_get_image_header`), as well as image load callbacks (`_dyld_register_func_for_add_image`), and an API to lookup section bounds by name from a particular image (getsectiondata). + * In COFF (Windows), image headers are present in the address space, and they include section bounds information. `Module32FirstW`/`Module32NextW` can be used to enumerate images, and structures such as `IMAGE_DOS_HEADER`, `IMAGE_NT_HEADERS`, `IMAGE_FILE_HEADER`, and `IMAGE_SECTION_HEADER` can be used to walk a module and find its section bounds. + * In Wasm, dynamic linking is work in progress and not generally available yet. + * In ELF, however, section bounds are not guaranteed to be present in the address space at runtime, and in practice they are typically not present. This creates a challenge for retrieving section data in this configuration (ELF + multiple modules with dynamic linking) at runtime. + +To solve this problem for the ELF object file format, the Swift compiler is going to emit a “**section index**” into every compilation that uses any symbols placed into a custom section. The index will be emitted only when producing ELF files, and consists of entries added into its own separate well-named section called `swift5_sections`. Each entry will have the following structure: + +```c +struct SectionIndexEntry { + const char *name; + const void *start; // effectively equal to __start_ + const void *stop; // effectively equal to __stop_ +}; +``` + +The section index will describe the bounds of all custom sections used in Swift code. When compiling into a single object file (e.g. in WMO mode without -num-threads), there will be only a single entry per distinct section name, but in compilation modes that produce multiple object files for a single module, there may be multiple entries for the same section. The entries are going to be “linkonce_odr”, i.e. duplicate entries will be collapsed at link time, so in a linked module, only one entry per section will remain. + +This way, runtime code present in the same module, for example SwiftRT-ELF helper code (swiftrt.o) which is currently being silently linked in to all modules, can walk over the section index using the encapsulation symbols, and register the section bounds in a globally maintained data structure in the Swift runtime. Implementation of that and exposing such a facility in an actual API from the Swift runtime is left as a future direction. + +## Source compatibility + +This proposal is purely additive (adding new attributes), without impact on source compatibility. + +## Effect on ABI stability + +This change does not impact ABI stability for existing code. + +Adding, removing, or changing the `@section` attribute on variables should generally be viewed as an ABI breaking change, section placement can affect linking behavior of that symbol. In some cases, it is possible to make careful non-ABI-breaking changes via the `@section` attribute. + +Adding `@used` does not affect ABI stability. Removing `@used` can be viewed as an ABI breaking change, however not in the traditional sense: The effect of `@used` only exists on symbols that would normally not be exported (e.g. private symbols), which shouldn’t be part of a ABI in the first place. However, dynamic lookups of such symbols are still possible, and if the behavior of those is considered ABI, then removing `@used` can be ABI breaking. + +## Effect on API resilience + +This change does not impact API resilience for existing code. + +Adding, removing, or changing the `@section` attribute on variables does not affect API resilience. + +Adding or removing `@used` does not affect API resilience. + +## Future Directions + +### Section placement for functions + +This proposal only allows data placement into custom sections, however, placing code into custom sections is a relatively useful and common approach in systems and embedded programming. In the future, the `@section` and `@used` attributes could be extended to apply to function declarations, and possibly other language constructs that generate executable code (e.g. closures). A prominent use case is firmware entry points and booting schemes, which often require startup code to be in a predefined section: + +```swift +// code for the function is placed into the custom section +@section(MachO: "__TEXT,boot") +func firmwareBootEntrypoint() { ... } +``` + +This will require some design decisions to be made around when should that be allowed, whether the attribute should be automatically inherited, and what exact behavior should we expect from the compiler around thunks, compiler-generated helper functions, getters and setters, etc. + +### Standalone attribute for required static initialization + +Static initialization of a global can be useful on its own, without placing data into a custom section, and a separate attribute for that could be added. This way, one can get the same effects as the `@section` attribute (static initialization, normal initalization behavior if top-level code) except the symbol would not be actually placed into any custom section. + +### Generalized constant values and expressions + +The notions of constant expressions and constant values is applicable to a much wider set of use cases that just section placement, and the set of allowed types and syntactical forms should be expanded in the future into a full-featured system for compile-time programming. A dedicated proposal, [Swift Compile-Time Values](https://github.com/artemcm/swift-evolution/blob/const-values/proposals/0nnn-const-values.md), is being pitched [on the forums](https://forums.swift.org/t/pitch-3-swift-compile-time-values/77434) and describes in detail the possible future of generalized constants, the relevant motivation and use cases. + +### Allowing a reference to a constant string declaration as a section name + +The requirement to only use string literals as the section names could be lifted in the future, and we might allow referring to a declaration of variable with a compile-time string. This would be useful to avoid repetition when placing multiple values into the same section without needing to use macros. + +```swift +#if DEBUG +let mySectionName = "__DATA,mysection" // required to be a compile-time value +#else +let mySectionName = "__DATA,myothersection" // required to be a compile-time value +#endif + +@section(MachO: mySectionName) +var global = ... +``` + +### Runtime discovery of data in custom sections + +As described in [ELF section index](#elf-section-index), accessing records in a custom section at runtime is heavily dependent on the object file format (ELF, Mach-O, Wasm, COFF), type of linking (static vs dynamic) and available APIs from the operating system. For a single configuration, users can directly use an appropriate method of accessing the section data, and e.g. in embedded firmwares this might be completely fine as projects are commonly avoiding any attempts to be multi-platform or portable. + +However, for multi-platform libraries and general purpose packages, supporting the full matrix of combinations would be very impractical. Because of that, it’s expected that a unified API for accessing the bounds and contents of a section (across multiple modules in presence of dynamic linking) is provided either as part of the Swift runtime, the standard library, or as a portable package. This API would likely still be a relatively low-level API, providing access to the raw bytes of sections across multiple loaded modules, but it would provide an shared abstraction across platforms, file formats, and linking types. + +### Linker sets, plugins as high-level APIs + +This proposal only builds the compile-time half of a user-facing “linker set” mechanism (placing structured data into sections). To access and enumerate the data at runtime, one can imagine a direct, still relatively low-level API like this: + +```swift +func enumerateLinkerSet(fromSectionNamed: String) -> Sequence { + // extract section data, assuming the raw data in the section are records of "T" + // probably built on top of a cross-platform section access API mentioned in previous section +} +``` + +But a solution based on macros could achieve a higher-level abstraction for the entire “linker set” mechanism: + +```swift +@DefineLinkerSet("name", type: Int) // other macros understand that linker set "name" + // has entries of type Int + +@LinkerSetEntry("name") let entry1: Int = 42 // ok +@LinkerSetEntry("name") let entry2: Float = 7.7 // error + +for entry in #enumerateLinkerSet("name") { + print(entry) +} +``` + +Similarly, a plugin registration and discovery mechanism based on macros could provide full type safety and hide the low-level aspects completely: + +```swift +// In PluginModule: +@PluginRecord(protocol: PluginProtocol, type: MyPluginType) +let plugin = PluginData(name: "myPlugin", version: 1, initialization: { ... }) + +// In MainModule: +... load available plugins via dlopen ... +for plugin in Plugin.enumerateLoadedPlugins(for: PluginProtocol.self) { + print(plugin.name) + let t = plugin.instantiateType() + ... +} +``` + +### Access to a stable address + +Generally, Swift values and variables do not have a stable address, and converting an inout reference to a UnsafePointer does not guarantee a stable address even on a global variable. However, statically initialized globals/static (either via the `@section` attribute, or via `@constInitialized`) do fundamentally have a stable address because they have an exact location in the binary on disk and in the module’s image at runtime. It’d be useful to provide direct access to that, in a way that adds the missing stable-address guarantees compared to inout-to-pointer conversions, and also allow fetching an address of a `let` variable (inout references only work on `var`): + +```swift +@constInitialized let x = 42 + +let address: UnsafePointer = #address(x) // would only work on statically initialized + // globals/statics +``` + +## Alternatives Considered + +### Requiring explicitly spelled out `@const` when using `@section` + +`@section` annotated globals/statics require their initializer expressions to be constant expressions, but the expression does not have to be marked as `@const` manually, it’s implied instead. An alternative of requiring the `@const` was considered: + +```swift +@section(...) let global: Int = @const 42 +``` + +Because `@const` does not affect parsing or type resolution of the expression, it’s not helpful to the compiler, and it doesn’t seem to improve readability for users either: If the expression is a constant expression or not statically initializable, it will be rejected from compilation with a clear explanation. Adding a `@const` does not convey any new information. + +### Umbrella attribute for linkage properties + +Instead of separate `@section` and `@used` attributes, a unified attribute with parameters to control individual linkage properties was considered, spelled for example `@linkage(section: ..., used)`. Further linkage control features would be added into this umbrella attribute. + +This, however, adds challenges on composability — one umbrella attribute would need to allow multiple occurrences and the design would need rules for merging of individual properties from multiple attributes. Separate standalone attributes compose trivially, and also they play nicely with the existing `#if hasAttribute(...)` conditional compilation mechanism. There is currently no mechanism for conditional compilation based on whether a sub-feature of a umbrella attribute is available in the compiler. + +Given the above, and also given that controlling symbol and linker level properties is not something that we expect normal application code to do directly, it’s more appropriate to keep the attribute system simple, and have individual orthogonal composable attributes. + +### `@section` implying `@used` + +In a lot of the list code snippets in this proposal, both `@section` and `@used` were used together, and so it may seem that it’s not necessary for those to be two separate attributes. However: + +* `@section` and `@used` represent separate concepts and all combinations of them can be useful. An example of using `@section` without `@used` is to place for example a large data table from a library into its own section for binary size accounting reasons (so that it shows up separately in per-section binary size listings), but where we’d still expect the data table to be dead-code removed if not used. +* It’s already common to have those attributes as separate options in existing popular systems programming languages (C, C++, Rust). + +### Blocking section placement into compiler reserved sections + +In most cases, placing data into one of Swift’s runtime reserved sections (e.g. `__swift5_types`, etc.) without relying on extreme details of the compiler and runtime would result in invalid binaries. It was considered to simply reject using `@section` to target one of these reserved sections, but ultimately that would introduce both false positives (*what if we at some point wanted to write compiler/runtime code in Swift to actually legitimately place data into these sections?*) and false negatives (*there are many other "reserved" sections that the Swift compiler and language cannot know about*), and is thus left out of this proposal. diff --git a/proposals/0493-defer-async.md b/proposals/0493-defer-async.md new file mode 100644 index 0000000000..c460b1cbd4 --- /dev/null +++ b/proposals/0493-defer-async.md @@ -0,0 +1,132 @@ +# Support `async` calls in `defer` bodies + +* Proposal: [SE-0493](0493-defer-async.md) +* Authors: [Freddy Kellison-Linn](https://github.com/Jumhyn) +* Review Manager: [Holly Borla](https://github.com/hborla) +* Status: **Active review (September 22 - October 6, 2025)** +* Implementation: [swiftlang/swift#83891](https://github.com/swiftlang/swift/pull/83891) +* Review: ([pitch](https://forums.swift.org/t/support-async-calls-in-defer-bodies/81790)) ([review](https://forums.swift.org/t/se-0493-support-async-calls-in-defer-bodies/82293)) + +## Introduction + +This is a targeted proposal to introduce support for asynchronous calls within `defer` statements. Such calls must be marked with `await` as any other asynchronous call would be, and `defer` statements which do asynchronous work will be implicitly awaited at any relevant scope exit point. + +## Motivation + +The `defer` statement was introduced in Swift 2 (before Swift was even open source) as the method for performing scope-based cleanup in a reliable way. Whenever a lexical scope is exited, the bodies of prior `defer` statements within that scope are executed (in reverse order, in the case of multiple `defer` statements). + +```swift +func sendLog(_ message: String) async throws { + let localLog = FileHandle("log.txt") + + // Will be executed even if we throw + defer { localLog.close() } + + localLog.appendLine(message) + try await sendNetworkLog(message) +} +``` + +This lets cleanup operations be syntactically colocated with the corresponding setup while also preventing the need to manually insert the cleanup along every possible exit path. + +While this provides a convenient and less-bug-prone way to perform important cleanup, the bodies of `defer` statements are not permitted to do any asynchronous work. If you attempt to `await` something in the body of a `defer` statement, you'll get an error even if the enclosing context is `async`: + +```swift +func f() async { + await setUp() + // error: 'async' call cannot occur in a defer body + defer { await performAsyncTeardown() } + + try doSomething() +} +``` + +If a particular operation *requires* asynchronous cleanup, then there aren't any great options today. An author can either resort to inserting the cleanup on each exit path manually (risking that they or a future editor will miss a path), or else spawn a new top-level `Task` to perform the cleanup: + +```swift +defer { + // We'll clean this up... eventually + Task { await performAsyncTeardown() } +} +``` + +## Proposed solution + +This proposal allows `await` statements to appear in `defer` bodies whenever the enclosing context is already `async`. Whenever a scope is exited, the bodies of all prior `defer` statements will be executed in reverse order of declaration, just as before. The bodies of any `defer` statements containing asynchronous work will be `await`ed, and run to completion before the function returns. + +Thus, the example from **Motivation** above will become valid code: +```swift +func f() async { + await setUp() + defer { await performAsyncTeardown() } // OK + + try doSomething() +} +``` + +## Detailed design + +When a `defer` statement contains asynchronous work, we will generate an implicit `await` when it is called on scope exit. See **Alternatives Considered** for further discussion. + +We always require that the parent context of the `defer` be explicitly or implicitly `async` in order for `defer` to contain an `await`. That is, the following is not valid: + +```swift +func f() { + // error: 'async' call in a function that does not support concurrency + defer { await g() } +} +``` + +In positions where `async` can be inferred, such as for the types of closures, an `await` within the body of a `defer` is sufficient to infer `async`: + +```swift +// 'f' implicitly has type '() async -> ()' +let f = { + defer { await g() } +} +``` + +The body of a `defer` statement will always inherit the isolation of its enclosing scope, so an asynchronous `defer` body will never introduce *additional* suspension points beyond whatever suspension points are introduced by the functions it calls. + +## Source compatibility + +This change is additive and opt-in. Since no `defer` bodies today can do any asynchronous work, the behavior of existing code will not change. + +## ABI compatibility + +This proposal does not have any impact at the ABI level. It is purely an implementation detail. + +## Implications on adoption + +Adoping asynchronous `defer` is an implementation-level detail and does not have any implications on ABI or API stability. + +## Alternatives considered + +### Require some statement-level marking such as `defer async` + +We do not require any more source-level annotation besides the `await` that will appear on the actual line within the `defer` which invokes the asynchronous work. We could go further and require one to write something like: +```swift +defer async { + await fd.close() +} +``` + +This proposal declines to introduce such requirement. Because `defer` bodies are typically small, targeted cleanup work, we do not believe that substantial clarity is gained by requiring another marker which would remain local *to the `defer`* statement itself. Moreover, the enclosing context of such `defer` statements will *already* be required to be `async`. In the case of `func` declarations, this will be explicit. In the case of closures, this may be inferred, but will be no less implicit than the inference that already happens from having an `await` in a closure body. + +### Require some sort of explicit `await` marking on scope exit + +The decision to implicltly await asyncrhonous `defer` bodies has the potential to introduce unexpected suspension points within function bodies. This proposal takes the position that the implicit suspension points introduced by asynchronous `defer` bodies is almost entirely analagous to the [analysis](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0317-async-let.md#requiring-an-awaiton-any-execution-path-that-waits-for-an-async-let) provided by the `async let` proposal. Both of these proposals would require marking every possible control flow edge which exits a scope. + +If anything, the analysis here is even more favorable to `defer`. In the case of `async let` it is possible to have an implicit suspension point without `await` appearing anywhere in the source—with `defer`, any suspension point within the body will be marked with `await`. + +### Suppress task cancellation within `defer` bodies + +During discussion of the behavior expected from asynchronous `defer` bodies, one point raised was whether we ought to remove the ability of code within a `defer` to observe the current task's cancellation state. Under this proposal, no such change is adopted, and if the current task is cancelled then all code called from the `defer` will observe that cancellation (via `Task.isCancelled`, `Task.checkCancellation()`, etc.) just as it would if called from within the main function body. + +The alternative suggestion here noted that because task cancellation can sometimes cause code to be skipped, it is not in the general case appropriate to run necessary cleanup code within an already-cancelled task. For instance, if one wishes to run some cleanup on a timeout (via `Task.sleep`) or via an HTTP request or filesystem operation, these operations could interpret running in a cancelled task as an indication that they _should not perform the requested work_. + +We could, instead, notionally 'un-cancel' the current task if we enter a `defer` body: all code called from within the `defer` would observe `Task.isCancelled == true`, `Task.checkCancellation()` would not throw, etc. This would allow timeouts to continue to function and ensure that any downstream cleanup would not misinterpret task cancellation as an indication that it should early-exit. + +This proposal does not adopt such behavior, for a combination of reasons: +1. Synchronous `defer` bodies already observe cancellation 'normally', i.e., `Task.isCancelled` can be accessed within the body of a synchronous `defer`, and it will reflect the actual cancellation status of the enclosing task. While it is perhaps less likely that existing synchronous code exhibits behavior differences with respect to cancellation status, it would be undesirable if merely adding `await` in one part of a `defer` body could result in behavior changes for other, unrelated code in the same `defer` body. +2. We do not want a difference in behavior that could occur merely from moving existing code into a `defer`. Existing APIs which are sensitive to cancellation must already be used with care even in straight-line code where `defer` may not be used (since cancellation can happen at any time), and this proposal takes the position that such APIs are more appropriately addressed by a general `withCancellationIgnored { ... }` feature (or similar) as discussed in the pitch thread. diff --git a/proposals/0494-add-is-identical-methods.md b/proposals/0494-add-is-identical-methods.md new file mode 100644 index 0000000000..076d35f9a5 --- /dev/null +++ b/proposals/0494-add-is-identical-methods.md @@ -0,0 +1,793 @@ +# Add `isIdentical(to:)` Methods for Quick Comparisons to Concrete Types + +* Proposal: [SE-0494](0494-add-is-identical-methods.md) +* Authors: [Rick van Voorden](https://github.com/vanvoorden), [Karoy Lorentey](https://github.com/lorentey) +* Review Manager: [John McCall](https://github.com/rjmccall) +* Status: **Active review (September 22nd...October 6th, 2025)** +* Implementation: ([String, Substring](https://github.com/swiftlang/swift/pull/82055)), ([Array, ArraySlice, ContiguousArray](https://github.com/swiftlang/swift/pull/82438)), ([Dictionary, Set](https://github.com/swiftlang/swift/pull/82439)) +* Review: ([prepitch](https://forums.swift.org/t/-/78792)) ([first pitch](https://forums.swift.org/t/-/79145)) ([second pitch](https://forums.swift.org/t/-/80496)) ([review](https://forums.swift.org/t/se-0494-add-isidentical-to-methods-for-quick-comparisons-to-concrete-types/82296)) + +### Table of Contents + + * [Introduction](#introduction) + * [Motivation](#motivation) + * [Prior Art](#prior-art) + * [Proposed Solution](#proposed-solution) + * [Detailed Design](#detailed-design) + * [`String`](#string) + * [`Substring`](#substring) + * [`Array`](#array) + * [`ArraySlice`](#arrayslice) + * [`ContiguousArray`](#contiguousarray) + * [`Dictionary`](#dictionary) + * [`Set`](#set) + * [Source Compatibility](#source-compatibility) + * [Impact on ABI](#impact-on-abi) + * [Future Directions](#future-directions) + * [Alternatives Considered](#alternatives-considered) + * [Exposing Identity](#exposing-identity) + * [Different Names](#different-names) + * [Generic Contexts](#generic-contexts) + * [Overload for Reference Comparison](#overload-for-reference-comparison) + * [Support for Optionals](#support-for-optionals) + * [Alternative Semantics](#alternative-semantics) + * [Acknowledgments](#acknowledgments) + +## Introduction + +We propose new `isIdentical(to:)` instance methods to concrete types for quickly determining if two instances must be equal by-value. + +## Motivation + +Suppose we need an algorithm that transforms an `Array` of `Int` values to select only the even numbers: + +```swift +func result(for input: [Int]) -> [Int] { + print("computing new result") + return input.filter { + $0 % 2 == 0 + } +} +``` + +This produces a correct answer… but what about performance? We expect our `result` function to run in `O(n)` time across the size of our `input` value. Suppose we need for this algorithm to be called *many* times over the course of our application. It might also be the case that we sometimes call this algorithm with the same `input` value more than once: + +```swift +let a = [1, 2, 3, 4] +print(result(for: a)) +// Prints "computing new result" +// Prints "[2, 4]" +let b = a +print(result(for: b)) +// Prints "computing new result" +// Prints "[2, 4]" +let c = [1, 2, 3, 4] +print(result(for: c)) +// Prints "computing new result" +// Prints "[2, 4]" +let d = [1, 2, 3, 4, 5, 6] +print(result(for: d)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +let e = d +print(result(for: e)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +let f = [1, 2, 3, 4, 5, 6] +print(result(for: f)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +``` + +If we call our `result` function with an `Array` of values and then pass the same `Array` of values again, we might want to return our previous `result` *without* performing another `O(n)` operation. Because our `result` function is a pure function and free of side-effects, we can check the new `input` value against the last `input` value we used to compute our `result`. If the `input` values have not changed, the `result` value *also* must not have changed. + +Here is an attempt to *memoize* our `result`: + +```swift +final class Memoizer { + private var input: [Int]? + private var result: [Int]? + + func result(for input: [Int]) -> [Int] { + if let result = self.result, + self.input == input { + return result + } else { + print("computing new result") + self.input = input + let result = input.filter { + $0 % 2 == 0 + } + self.result = result + return result + } + } +} +``` + +When we pass `input` values we can see that a new `result` is not computed if we already computed a `result` for those same `input` values: + +```swift +let memoizer = Memoizer() +let a = [1, 2, 3, 4] +print(memoizer.result(for: a)) +// Prints "computing new result" +// Prints "[2, 4]" +let b = a +print(memoizer.result(for: b)) +// Prints "[2, 4]" +let c = [1, 2, 3, 4] +print(memoizer.result(for: c)) +// Prints "[2, 4]" +let d = [1, 2, 3, 4, 5, 6] +print(memoizer.result(for: d)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +let e = d +print(memoizer.result(for: e)) +// Prints "[2, 4, 6]" +let f = [1, 2, 3, 4, 5, 6] +print(memoizer.result(for: f)) +// Prints "[2, 4, 6]" +``` + +This looks like a big improvement… until we begin to investigate a little closer. There’s a subtle performance bottleneck here now from a different direction. Our memoization algorithm depends on the value equality of our `input` values — and this is *also* an `O(n)` operation. So while it is true that we have reduced the amount of `O(n)` operations that take place to compute our `result` values, we have *added* `O(n)` operations to determine value equality. As the amount of time spent computing value equality grows, we might no longer see any performance wins from memoization: it would be cheaper to just go ahead and compute a new `result` every time. + +Let’s see another example. Suppose we are working on our SwiftUI app to display Contacts from [SE-0261](0261-identifiable.md). Let’s begin with our basic data model: + +```swift +struct Contact: Identifiable, Equatable { + let id: Int + var name: String + var isFavorite: Bool +} +``` + +We added an `isFavorite` property to indicate our user added this `Contact` value as one of their favorites. + +Here is a SwiftUI view component that displays our favorite `Contact` values in a `FavoriteContactList`: + +```swift +struct FavoriteContactList: View { + @State private var selection: Contact.ID? + + private let contacts: [Contact] + + init(_ contacts: [Contact]) { + self.contacts = contacts + } + + private var favorites: [Contact] { + self.contacts.filter { + $0.isFavorite + } + } + + var body: some View { + List(self.favorites, selection: self.$selection) { contact in + FavoriteCell(contact) + } + } +} +``` + +When we compute our `body` property we also compute our `favorites` property. The implication is that *every* time our `body` property is computed we perform *another* `O(n)` algorithm across our `contacts`. Because our `FavoriteContactList` supports selection, every time our user selects a `Contact` value we update our `State`. Updating our `State` computes our `body` which computes our `favorites` property. So even though our `contacts` values *have not changed*, we *still* pay the performance penalty of *another* `O(n)` operation just to support cell selection. + +This might look like a good opportunity for another attempt at memoization. Here is an approach using a dynamic property wrapper: + +```swift +@propertyWrapper struct Favorites: DynamicProperty { + @State private var storage: Storage + private let contacts: [Contact] + + init(_ contacts: [Contact]) { + self.storage = Storage(contacts) + self.contacts = contacts + } + + func update() { + self.storage.update(self.contacts) + } + + var wrappedValue: [Contact] { + self.storage.wrappedValue + } +} + +extension Favorites { + private final class Storage { + private var contacts: [Contact] + private var favorites: [Contact]? + + init(_ contacts: [Contact]) { + self.contacts = contacts + self.favorites = nil + } + + func update(_ contacts: [Contact]) { + if self.contacts != contacts { + self.contacts = contacts + self.favorites = nil + } + } + + var wrappedValue: [Contact] { + if let favorites = self.favorites { + return favorites + } + print("computing new result") + let favorites = self.contacts.filter { + $0.isFavorite + } + self.favorites = favorites + return favorites + } + } +} +``` + +Here is what that looks like used from our `FavoriteContactList`: + +```swift +struct FavoriteContactList: View { + @State private var selection: Contact.ID? + + @Favorites private var favorites: [Contact] + + init(_ contacts: [Contact]) { + self._favorites = Favorites(contacts) + } + + var body: some View { + List(self.favorites, selection: self.$selection) { contact in + FavoriteCell(contact) + } + } +} +``` + +When we build and run our app we see that we no longer compute our `favorites` values every time our user selects a new `Contact`. But similar to what we saw in our command line utility, we have traded performance in a different direction. The value equality operation we perform is *also* `O(n)`. As the amount of time we spend computing value equality grows, we can begin to spend more time computing value equality than we would have spent computing our `favorites`: we no longer see the performance benefits of memoization. + +This proposal introduces an advanced performance hook for situations like this: a set of `isIdentical(to:)` methods that are designed to return *faster* than an operation to determine value equality. The `isIdentical(to:)` methods can return `true` in `O(1)` to indicate two values *must* be equal. + +## Prior Art + +We said that the performance of the value equality operator on an `Array` value was `O(n)`. This is true in the *worst case*, but there does exist an important “fast path” that can return `true` in constant time. + +Many types in Standard Library are “copy-on-write” data structures. These types present as value types, but can leverage a reference to some shared state to optimize for performance. When we copy this value we copy a reference to shared storage. If we perform a mutation on a copy we can preserve value semantics by copying the storage reference to a unique value before we write our mutation: we “copy” on “write”. + +This means that many types in Standard Library already have some private reference that can be checked in constant time to determine if two values are identical. Because these types copy before writing, two values that are identical by their shared storage *must* be equal by value. What we propose here is a way to “expose” this fast path operation. + +Product engineers have evolved patterns over the years that can already come close to what we are proposing. Product engineers building on `Array` can use `withUnsafeBufferPointer` or `withContiguousStorageIfAvailable` to compare the “identity” of two `Array` values. One drawback here is that these are only guaranteed to return an identity in constant time if there already exists a contiguous storage. If there does *not* exist a contiguous storage, we might have to perform an `O(n)` algorithm — which defeats the purpose of us choosing this as a fast path. Another option might be `withUnsafeBytes`, but this carries some restrictions on the `Element` of our `Array` and also might require for a contiguous storage to be created: an `O(n)` algorithm. + +Even if we were able to use `withUnsafeBytes` for other data structures, a comparison using `memcmp` might compare “unnecessary” bits that do not affect the identity. This slows down our algorithm and also returns “false negatives”: returning `false` when these instances should be treated as identical. + +A solution for modern operating systems is the support we added from [SE-0456](0456-stdlib-span-properties.md) to bridge an `Array` to `Span`. We can then compare these instances using the `isIdentical(to:)` method on `Span`. One drawback here is that we are blocked on back-deploying support for bridging `Array` to `Span`: it is only available on the most modern operating systems. Another drawback is that if our `Array` does not have a contiguous storage, we have to copy one: an `O(n)` operation. We are also blocked on bringing support for `Span` to collection types like `Dictionary` that do not already implement contiguous storage. + +A new `isIdentical(to:)` method could work around all these restrictions. We could return in constant time *without* needing to copy memory to a contiguous storage. We could adopt this method on many types that might not *ever* have a contiguous storage. We could also work with our library maintainers to discuss a back-deployment strategy that could bring this method to legacy operating systems. + +Our proposal would not be the first example of `isIdentical(to:)` shipping across Swift. `String` already ships a public-but-underscored version of this API.[^1] + +```swift +extension String { + /// Returns a boolean value indicating whether this string is identical to + /// `other`. + /// + /// Two string values are identical if there is no way to distinguish between + /// them. + /// + /// Comparing strings this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// string storage object. Therefore, identical strings are guaranteed to + /// compare equal with `==`, but not all equal strings are considered + /// identical. + /// + /// - Performance: O(1) + @_alwaysEmitIntoClient + public func _isIdentical(to other: Self) -> Bool { + self._guts.rawBits == other._guts.rawBits + } +} +``` + +We don’t see this API currently being used in Standard Library, but it’s possible this API is already being used to optimize performance in private frameworks from Apple. + +Many more examples of `isIdentical(to:)` functions are currently shipping in `Swift-Collections`[^2][^3][^4][^5][^6][^7][^8][^9][^10][^11][^12][^13], `Swift-Markdown`[^14], and `Swift-CowBox`[^15]. We also support `isIdentical(to:)` on the upcoming `Span` and `RawSpan` types from Standard Library.[^16] + +## Proposed Solution + +Before we look at the concrete types in this proposal, let’s begin with some more general principles and ideas we would expect for *all* concrete types to follow when adopting this new method. While this specific proposal is not adding a new protocol to Standard Library, it could be helpful to think of an “informal” protocol that guides us in choosing the types to adopt this new method. This could then serve as a guide for library maintainers that might choose to adopt this method on *new* types in the future. + +Suppose we are proposing an `isIdentical(to:)` method on a type `T`. We propose the following axioms that library maintainers should adopt: +* `a.isIdentical(to: a)` is always `true` (Reflexivity) +* If `T` is `Equatable`: + * `a.isIdentical(to: b)` implies `a == b` (*or else `a` and `b` are exceptional values*) + * `isIdentical(to:)` is *meaningfully* faster than `==` + +Let’s look through these axioms a little closer: + +**`a.isIdentical(to: a)` is always `true` (Reflexivity)** + +* An implementation of `isIdentical(to:)` that always returns `false` would not be an impactful API. We must guarantee that `isIdentical(to:)` *can* return `true` at least *some* of the time. + +**If `T` is `Equatable` then `a.isIdentical(to: b)` implies `a == b`** + +* This is the “fast path” performance optimization that will speed up the memoization examples we saw earlier. One important side effect here is that when `a.isIdentical(to: b)` returns `false` we make *no* guarantees about whether or not `a` is equal to `b`. +* We assume this axiom holds only if `a` and `b` are not “exceptional” values. A example of an exceptional value would be if a container that is generic over `Float` contains `nan`. + +**If `T` is `Equatable` then `isIdentical(to:)` is *meaningfully* faster than `==`** + +* While we could implement `isIdentical(to:)` on types like `Int` or `Bool`, these types are not included in this proposal. Our proposal focuses on types that have the ability to return from `isIdentical(to:)` meaningfully faster than `==`. If a type would perform the same amount of work in `isIdentical(to:)` that takes place in `==`, our advice is that library maintainers should *not* adopt `isIdentical(to:)` on this type. There should exist some legit internal fast-path on this type: like a pointer to a storage buffer that can be compared by reference identity. + +This proposal focuses on concrete types that are `Equatable`, but it might also be the case that a library maintainer would adopt `isIdentical(to:)` on a type that is *not* `Equatable`: like `Span`. Our expectation is that a library maintainer adopting `isIdentical(to:)`on a type that is not `Equatable` has some strong and impactful real-world use-cases ready to make use of this API. Just because a library maintainer *can* adopt this API does not imply they *should*. A library maintainer should also be ready to document for product engineers exactly what is implied from `a.isIdentical(to: b)` returning `true`. What does it *mean* for `a` to be “identical” to `b` if we do not have the implication that `a == b`? We leave this decision to the library maintainers that have the most context on the types they have built. + +Suppose we had an `isIdentical(to:)` method available on `Array`. Let’s go back to our earlier example and see how we can use this as an alternative to checking for value equality from our command line utility: + +```swift +final class Memoizer { + ... + + func result(for input: [Int]) -> [Int] { + if let result = self.result, + self.input.isIdentical(to: input) { + return result + } else { + ... + } + } +} +``` + +We can run our previous example and confirm that we are not computing new results when the input has not changed: + +```swift +let memoizer = Memoizer() +let a = [1, 2, 3, 4] +print(memoizer.result(for: a)) +// Prints "computing new result" +// Prints "[2, 4]" +let b = a +print(memoizer.result(for: b)) +// Prints "[2, 4]" +let c = [1, 2, 3, 4] +print(memoizer.result(for: c)) +// Prints "computing new result" +// Prints "[2, 4]" +let d = [1, 2, 3, 4, 5, 6] +print(memoizer.result(for: d)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +let e = d +print(memoizer.result(for: e)) +// Prints "[2, 4, 6]" +let f = [1, 2, 3, 4, 5, 6] +print(memoizer.result(for: f)) +// Prints "computing new result" +// Prints "[2, 4, 6]" +``` + +When we return `true` from `isIdentical(to:)` we skip computing a new `result`. When `isIdentical(to:)` returns `false` we compute a new `result`. Because `isIdentical(to:)` *can* return `false` when two values are equal, we might be computing the same `result` more than once. The performance tradeoff is that because the operation to compute a new `result` is `O(n)` time, we might not *want* to perform another `O(n)` value equality operation to determine if we should compute a new `result`. Our `isIdentical(to:)` will return in constant time no matter how many elements are in `input` or how expensive this value equality operation would be. + +Let’s go back to our SwiftUI app for displaying `Contact` values. Here is what the change would look like to use `isIdentical(to:)` in place of value equality to memoize `favorites`: + +```swift +extension Favorites { + private final class Storage { + ... + + func update(_ contacts: [Contact]) { + if self.contacts.isIdentical(to: contacts) == false { + self.contacts = contacts + self.favorites = nil + } + } + + ... + } +} +``` + +When we build and run our SwiftUI app we confirm that we are not computing new `favorites` when the user selects new `Contact` values from `FavoriteContactList`. + +## Detailed Design + +We propose adding `isIdentical` methods to the following concrete types from Standard Library: +* `String` +* `String.UnicodeScalarView` +* `String.UTF16View` +* `String.UTF8View` +* `Substring` +* `Substring.UnicodeScalarView` +* `Substring.UTF16View` +* `Substring.UTF8View` +* `Array` +* `ArraySlice` +* `ContiguousArray` +* `Dictionary` +* `Set` + +For each type being presented we codify important semantics in our header documentation. + +### `String` + +```swift +extension String { + /// Returns a boolean value indicating whether this string is identical to + /// `other`. + /// + /// Two string values are identical if there is no way to distinguish between + /// them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isIdentical(to: b)` implies `b.isIdentical(to: a)`. (Symmetry) + /// - If `a.isIdentical(to: b)` and `b.isIdentical(to: c)` are both `true`, + /// then `a.isIdentical(to: c)` is also `true`. (Transitivity) + /// - `a.isIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, + /// will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing strings this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// string storage object. Therefore, identical strings are guaranteed to + /// compare equal with `==`, but not all equal strings are considered + /// identical. + /// + /// - Performance: O(1) + public func isIdentical(to other: Self) -> Bool { ... } +} +``` + +The following types will adopt the same semantic guarantees as `String`: +* `String.UnicodeScalarView` +* `String.UTF16View` +* `String.UTF8View` + +### `Substring` + +```swift +extension Substring { + /// Returns a boolean value indicating whether this substring is identical to + /// `other`. + /// + /// Two substring values are identical if there is no way to distinguish + /// between them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isIdentical(to: b)` implies `b.isIdentical(to: a)`. (Symmetry) + /// - If `a.isIdentical(to: b)` and `b.isIdentical(to: c)` are both `true`, + /// then `a.isIdentical(to: c)` is also `true`. (Transitivity) + /// - `a.isIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, + /// will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing substrings this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// substring storage object. Therefore, identical substrings are guaranteed + /// to compare equal with `==`, but not all equal substrings are considered + /// identical. + /// + /// - Performance: O(1) + public func isIdentical(to other: Self) -> Bool { ... } +} +``` + +The following types will adopt the same semantic guarantees as `Substring`: +* `Substring.UnicodeScalarView` +* `Substring.UTF16View` +* `Substring.UTF8View` + +### `Array` + +```swift +extension Array { + /// Returns a boolean value indicating whether this array is identical to + /// `other`. + /// + /// Two array values are identical if there is no way to distinguish between + /// them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isIdentical(to: b)` implies `b.isIdentical(to: a)`. (Symmetry) + /// - If `a.isIdentical(to: b)` and `b.isIdentical(to: c)` are both `true`, + /// then `a.isIdentical(to: c)` is also `true`. (Transitivity) + /// - If `a` and `b` are `Equatable`, then `a.isIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, + /// will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing arrays this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// array storage object. Therefore, identical arrays are guaranteed to + /// compare equal with `==`, but not all equal arrays are considered + /// identical. + /// + /// - Performance: O(1) + public func isIdentical(to other: Self) -> Bool { ... } +} +``` + +### `ArraySlice` + +```swift +extension ArraySlice { + /// Returns a boolean value indicating whether this array is identical to + /// `other`. + /// + /// Two array values are identical if there is no way to distinguish between + /// them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isIdentical(to: b)` implies `b.isIdentical(to: a)`. (Symmetry) + /// - If `a.isIdentical(to: b)` and `b.isIdentical(to: c)` are both `true`, + /// then `a.isIdentical(to: c)` is also `true`. (Transitivity) + /// - If `a` and `b` are `Equatable`, then `a.isIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, + /// will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing arrays this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// array storage object. Therefore, identical arrays are guaranteed to + /// compare equal with `==`, but not all equal arrays are considered + /// identical. + /// + /// - Performance: O(1) + public func isIdentical(to other: Self) -> Bool { ... } +} +``` + +### `ContiguousArray` + +```swift +extension ContiguousArray { + /// Returns a boolean value indicating whether this array is identical to + /// `other`. + /// + /// Two array values are identical if there is no way to distinguish between + /// them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isIdentical(to: b)` implies `b.isIdentical(to: a)`. (Symmetry) + /// - If `a.isIdentical(to: b)` and `b.isIdentical(to: c)` are both `true`, + /// then `a.isIdentical(to: c)` is also `true`. (Transitivity) + /// - If `a` and `b` are `Equatable`, then `a.isIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, + /// will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing arrays this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// array storage object. Therefore, identical arrays are guaranteed to + /// compare equal with `==`, but not all equal arrays are considered + /// identical. + /// + /// - Performance: O(1) + public func isIdentical(to other: Self) -> Bool { ... } +} +``` + +### `Dictionary` + +```swift +extension Dictionary { + /// Returns a boolean value indicating whether this dictionary is identical to + /// `other`. + /// + /// Two dictionary values are identical if there is no way to distinguish + /// between them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isIdentical(to: b)` implies `b.isIdentical(to: a)`. (Symmetry) + /// - If `a.isIdentical(to: b)` and `b.isIdentical(to: c)` are both `true`, + /// then `a.isIdentical(to: c)` is also `true`. (Transitivity) + /// - If `a` and `b` are `Equatable`, then `a.isIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, + /// will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing dictionaries this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying + /// dictionary storage object. Therefore, identical dictionaries are + /// guaranteed to compare equal with `==`, but not all equal dictionaries are + /// considered identical. + /// + /// - Performance: O(1) + public func isIdentical(to other: Self) -> Bool { ... } +} +``` + +### `Set` + +```swift +extension Set { + /// Returns a boolean value indicating whether this set is identical to + /// `other`. + /// + /// Two set values are identical if there is no way to distinguish between + /// them. + /// + /// For any values `a`, `b`, and `c`: + /// + /// - `a.isIdentical(to: a)` is always `true`. (Reflexivity) + /// - `a.isIdentical(to: b)` implies `b.isIdentical(to: a)`. (Symmetry) + /// - If `a.isIdentical(to: b)` and `b.isIdentical(to: c)` are both `true`, + /// then `a.isIdentical(to: c)` is also `true`. (Transitivity) + /// - `a.isIdentical(b)` implies `a == b` + /// - `a == b` does not imply `a.isIdentical(b)` + /// + /// Values produced by copying the same value, with no intervening mutations, + /// will compare identical: + /// + /// ```swift + /// let d = c + /// print(c.isIdentical(to: d)) + /// // Prints true + /// ``` + /// + /// Comparing sets this way includes comparing (normally) hidden + /// implementation details such as the memory location of any underlying set + /// storage object. Therefore, identical sets are guaranteed to compare equal + /// with `==`, but not all equal sets are considered identical. + /// + /// - Performance: O(1) + public func isIdentical(to other: Self) -> Bool { ... } +} +``` + +## Source Compatibility + +This proposal is additive and source-compatible with existing code. + +## Impact on ABI + +This proposal is additive and ABI-compatible with existing code. + +## Future Directions + +Any Standard Library types that are copy-on-write values could be good candidates to add `isIdentical` functions. Here are some potential types to consider for a future proposal: + +* `Character` +* `Dictionary.Keys` +* `Dictionary.Values` +* `KeyValuePairs` +* `StaticBigInt` +* `StaticString` +* `UTF8Span` + +This proposal focuses on what we see as the most high-impact types to support from Standard Library. This proposal *is not* meant to discourage adding `isIdentical(to:)` on any of these types at some point in the future. A follow-up “second-round” proposal could focus on these remaining types. + +## Alternatives Considered + +### Exposing Identity + +Our proposal introduces a new instance method on types that uses some underlying concept of “identity” to perform quick comparisons between two instances. A different approach would be to *return* the underlying identity to product engineers. If a product engineer wanted to test two instances for equality by identity they could perform that check themselves. + +There’s a lot of interesting directions to go with that idea… but we don’t think this is right approach for now. Introducing some concept of an “escapable” identity to value types like `Array` would require *a lot* of design. It’s overthinking the problem and solving for something we don’t need right now. + +### Different Names + +Multiple different names have been suggested for these operations. Including: + +* `hasSameRepresentation(as:)` +* `isKnownIdentical(to:)` + +We prefer `isIdentical(to:)`. This also has the benefit of being consistent with the years of prior art we have accumulated across the Swift ecosystem. + +### Generic Contexts + +We proposed an “informal” protocol for library maintainers adopting `isIdentical(to:)` on new types. Could we just build a new protocol in Standard Library? Maybe. We don’t see a big need for this right now. If product engineers would want for these types to conform to some common protocol to use across generic contexts, those product engineers can define that protocol in their own packages. If these protocols “incubate” in the community and become a common practice, we can consider proposing a new protocol in Standard Library. + +Instead of a new protocol, could we somehow add `isIdentical(to:)` on `Equatable`? Maybe. This would introduce some more tricky questions. If we adopt this on *all* `Equatable` types, what do we do about types like `Int` or `Bool` that do not have an ability to perform a fast check for identity? Similar to our last idea, we prefer to focus just on concrete types for now. If product engineers want to make `isIdentical(to:)` available on generic contexts across `Equatable`, we encourage them to experiment with their own extension for that. If this pattern becomes popular in the community, we can consider a new proposal to add this on `Equatable` in Standard Library. + +### Overload for Reference Comparison + +Could we “overload” the `===` operator from `AnyObject`? This proposal considers that question to be orthogonal to our goal of exposing identity equality with the `isIdentical` methods. We could choose to overload `===`, but this would be a larger “conceptual” and “philosophical” change because the `===` operator is currently meant for `AnyObject` types — not value types like `Array`. + +### Support for Optionals + +We can support `Optional` values with the following extension: + +```swift +extension Optional { + public func isIdentical(to other: Self) -> Bool + where Wrapped == Array { + switch (self, other) { + case let (value?, other?): + return value.isIdentical(to: other) + case (nil, nil): + return true + default: + return false + } + } +} +``` + +Because this extension needs no `private` or `internal` symbols from Standard Library, we can omit this extension from our proposal. Product engineers that want this extension can choose to implement it for themselves. + +### Alternative Semantics + +Instead of publishing an `isIdentical` method which implies two types *must* be equal, could we think of things from the opposite direction? Could we publish a `maybeDifferent` method which implies two types *might not* be equal? This then introduces some potential ambiguity for product engineers: to what extent does “maybe different” imply “probably different”? This ambiguity could be settled with extra documentation on the method, but `isIdentical` solves that ambiguity up-front. The `isIdentical` method is also consistent with the prior art in this space. + +In the same way this proposal exposes a way to quickly check if two values *must* be equal, product engineers might want a way to quickly check if two values *must not* be equal. This is an interesting idea, but this can exist as an independent proposal. We don’t need to block the review of this proposal on a review of `isNotIdentical` semantics. + +## Acknowledgments + +Thanks to [Ben Cohen](https://forums.swift.org/t/-/78792/7) for helping to think through and generalize the original use-case and problem-statement. + +Thanks to [David Nadoba](https://forums.swift.org/t/-/80496/61/) for proposing the formal equivalence relation semantics and axioms on concrete types. + +Thanks to [Xiaodi Wu](https://forums.swift.org/t/-/80496/67) for proposing that our equivalence relation semantics would carve-out for “exceptional” values like `Float.nan`. + +[^1]: https://github.com/swiftlang/swift/blob/swift-6.1.2-RELEASE/stdlib/public/core/String.swift#L397-L415 +[^2]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/DequeModule/Deque._Storage.swift#L223-L225 +[^3]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/HashTreeCollections/HashNode/_HashNode.swift#L78-L80 +[^4]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/HashTreeCollections/HashNode/_RawHashNode.swift#L50-L52 +[^5]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Conformances/BigString%2BEquatable.swift#L14-L16 +[^6]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigString%2BUnicodeScalarView.swift#L77-L79 +[^7]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigString%2BUTF8View.swift#L39-L41 +[^8]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigString%2BUTF16View.swift#L39-L41 +[^9]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigSubstring.swift#L100-L103 +[^10]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigSubstring%2BUnicodeScalarView.swift#L94-L97 +[^11]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigSubstring%2BUTF8View.swift#L64-L67 +[^12]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/BigString/Views/BigSubstring%2BUTF16View.swift#L87-L90 +[^13]: https://github.com/apple/swift-collections/blob/1.2.0/Sources/RopeModule/Rope/Basics/Rope.swift#L68-L70 +[^14]: https://github.com/swiftlang/swift-markdown/blob/swift-6.1.1-RELEASE/Sources/Markdown/Base/Markup.swift#L370-L372 +[^15]: https://github.com/Swift-CowBox/Swift-CowBox/blob/1.1.0/Sources/CowBox/CowBox.swift#L19-L27 +[^16]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md diff --git a/proposals/0495-cdecl.md b/proposals/0495-cdecl.md new file mode 100644 index 0000000000..debdcb3ed4 --- /dev/null +++ b/proposals/0495-cdecl.md @@ -0,0 +1,228 @@ +# C compatible functions and enums + +* Proposal: [SE-0495](0495-cdecl.md) +* Author: [Alexis Laferrière](https://github.com/xymus) +* Review Manager: [Steve Canon](https://github.com/stephentyrone) +* Status: **Active Review (Sept 25 ... Oct 9, 2025)** +* Implementation: On `main` with the experimental feature flags `CDecl` for `@c`, and `CImplementation` for `@c @implementation`. With the exception of the `@objc` support for global functions which is available under the name `@_cdecl`. +* Review: ([pitch](https://forums.swift.org/t/pitch-formalize-cdecl/79557))([review](https://forums.swift.org/t/se-0495-c-compatible-functions-and-enums/82365)) + +## Introduction + +Implementing a C function in Swift eases integration of Swift and C code. This proposal introduces `@c` to mark Swift functions as callable from C, and enums as representable in C. It provides the same behavior under the `@objc` attribute for Objective-C compatible global functions. + +To expose the function to C clients, this proposal adds a new C block to the compatibility header where `@c` functions are printed. As an alternative, this proposal extends `@implementation` support to global functions, allowing users to declare the function in a hand-written C header. + +> Note: This proposal aims to formalize and extend the long experimental `@_cdecl`. While experimental this attribute has been widely in use so we will refer to it as needed for clarity in this document. + +## Motivation + +Swift already offers some integration with C, notably it can import declarations from C headers and call C functions. Swift also already offers a wide integration with Objective-C: import headers, call methods, print the compatibility header, and implement Objective-C classes in Swift with `@implementation`. These language features have proven to be useful for integration with Objective-C. Offering a similar language support for C will further ease integrating Swift and C, and encourage incremental adoption of Swift in existing C code bases. + +Offering a C compatibility type-checking ensures `@c` functions only reference types representable in C. This type-checking helps cross-platform development as one can define a `@c` while working from an Objective-C compatible environment and still see the restrictions from a C only environment. + +Printing the C representation of `@c` functions in a C header will enable a mixed-source software to easily call the functions from C code. The current generated header is limited to Objective-C and C++ content. Adding a section for C compatible clients will extend its usefulness to this language. + +Extending `@implementation` to support global C functions will provide support to developers through type-checking by ensuring the C declaration matches the corresponding definition in Swift. + +## Proposed solution + +We propose to introduce the new `@c` attribute for global functions and enums, extend `@objc` for global functions, and support `@c @implementation`. + +### `@c` global functions + +Introduce the `@c` attribute to mark a global function as a C function implemented in Swift. That function uses the C calling convention and its signature can only reference types representable in C. Its body is implemented in Swift as usual. The signature of that function is printed in the compatibility header using C corresponding types, allowing C source code to import the compatibility header and call the function. + +A `@c` function is declared with an optional C function name, by default the Swift base name is used as C name: +```swift +@c func foo() {} + +@c(mirrorCName) +func mirror(value: CInt) -> CInt { return value } +``` + +### `@objc` global functions + +Extends the `@objc` attribute to be accepted on a global function. It offers the same behavior as `@c` while allowing the signature to reference types representable in Objective-C. The signature of a `@objc` function is printed in the compatibility header using corresponding Objective-C types. + +A `@objc` function is declared with an optional C compatible name without parameter labels: + +```swift +@objc func bar() {} + +@objc(mirrorObjCName) +func objectMirror(value: NSObject) -> NSObject { return value } +``` + +> Note: The attribute `@objc` can be used on a global function to replace `@_cdecl` as it preserves the behavior of the unofficial attribute. + +### `@c` enums + +Accept `@c` on enums to mark them as C compatible. These enums can be referenced from `@c` or `@objc` functions. They are printed in the compatibility header as a C enum or a similar type. + +A `@c` enum may declare a custom C name, and must declare an integer raw type compatible with C: + +```swift +@c +enum CEnum: CInt { + case a + case b +} +``` + +The attribute `@objc` is already accepted on enums. These enums qualify as an Objective-C representable type and are usable from `@objc` global function signatures but not from `@c` functions. + +### `@c @implementation` global functions + +Extend support for the `@implementation` attribute, introduced in [SE-0436](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0436-objc-implementation.md), to global functions marked with either `@c` or `@objc`. These functions are declared in an imported C or Objective-C header, while the Swift function provides their implementation. Type-checking ensures the declaration matches the implementation signature in Swift. Functions marked `@implementation` are not printed in the compatibility header. + +The declaration and implementation are distinct across languages and must have matching names and types: + +```c +// C header +int cImplMirror(int value); +``` + +```swift +// Swift sources +@c @implementation +func cImplMirror(_ value: CInt) -> CInt { return value } +``` + +## Detailed design + +This proposal extends the language syntax, type-checking for both global functions and enums, supporting logic for `@implementation`, and the content printed to the compatibility header. + +### Syntax + +Required syntax changes involve one new attribute and the reuse of two existing attributes. + +* Introduce the attribute `@c` accepted on global functions and enums. It accepts one optional parameter specifying the corresponding C name of the declaration. The C name defaults to the Swift base identifier of the declaration, it doesn't consider parameter names. + +* Extend `@objc` to be accepted on global functions, using the optional parameter to define the C function name instead of the Objective-C symbol. Again here, the C function name defaults to the base identifier of the Swift function. + +* Extend `@implementation` to be accepted on global functions marked with either `@c` or `@objc`. + +### Type-checking of global functions signatures + +Global functions marked with `@c` or `@objc` need type-checking to ensure types used in their signature are representable in the target language. + +The following types are accepted in the signature of `@c` global functions: + +- Primitive types defined in the standard library: Int, UInt, Int8, Float, Double, Bool, etc. +- Pointers defined in the standard library: OpaquePointer and the variants of Unsafe{Mutable}{Raw}Pointer. +- C primitive types defined in the standard library: CChar, CInt, CUnsignedInt, CLong, CLongLong, etc. +- Function references using the C calling convention, marked with `@convention(c)`. +- SIMD types where the scalar is representable in C. +- Enums marked with `@c`. +- Imported C types. + +In addition to the types above, the following types should be accepted in the signature of `@objc` global functions: + +- `@objc` classes, enums and protocols. +- Imported Objective-C types. + +For both `@c` and `@objc` global functions, type-checking should reject: + +- Optional non-pointer types. +- Non-`@objc` classes. +- Swift structs. +- Non-`@c` enums. +- Protocol existentials. + +### Type-checking of `@c` enums + +For `@c` enums to be representable in C, type-checking should ensure the raw type is defined to an integer value that is itself representable in C. This is the same check as already applied to `@objc` enums. + +### `@c @implementation` and `@objc @implementation` + +A global function marked with `@c @implementation` or `@objc @implementation` needs to be associated with the corresponding declaration from imported headers. The compiler should report uses without a corresponding C declaration or inconsistencies in the match. + +### Compatibility header printing + +The compiler should print a single compatibility header for all languages, adding a block specific to C as is currently done for Objective-C and C++. Printing the header is requested using the preexisting compiler flags: `-emit-objc-header`, `-emit-objc-header-path` or `-emit-clang-header-path`. + +This C block declares the `@c` global functions and enums using C types, while `@objc` functions are printed in the Objective-C block with Objective-C types. + +The C block should be printed in a way it's parseable by compilers targeting C, Objective-C and C++. To do so, ensure that only C types are printed, there is no unprotected use of non-standard C features, and the syntax is C compatible. + +### Type mapping to C + +When printing `@c` functions in the compatibility header, Swift types are mapped to their corresponding C representations. Here is a partial mapping: + +- Swift `Bool` maps to C `bool` from `stdbool.h`. +- Swift `Int` maps to `ptrdiff_t`, `UInt` to `size_t`, `Int8` to `int8_t`, `UInt32` to `uint32_t`, etc. +- Swift floating-point types `Float` and `Double` map to C `float` and `double` respectively. +- Swift's version of C primitive types map to their C equivalents: `CInt` to `int`, `CLong` to long, etc. +- Swift SIMD types map to vector types printed as needed in the compatibility header. +- Function references with `@convention(c)` map to C function pointers. + +## Source compatibility + +This proposal preserves all source compatibility as the new features are opt-in. + +Existing adopters of `@_cdecl` can replace the attribute with `@objc` to preserve the same behavior. Alternatively, they can update it to `@c` to get the more restrictive C compatibility check. Using `@c` will however change how the corresponding C function is printed in the compatibility header so it may be necessary to update sources calling into the function. + +## ABI compatibility + +Marking a global function with `@c` or `@objc` makes it use the C calling convention. Adding or removing these attributes on a function is an ABI breaking change. Updating existing `@_cdecl` to `@objc` or `@c` is ABI stable. + +Adding or removing the `@c` attribute on an enum is ABI stable, but changing its raw type is not. + +Moving the implementation of an existing C function to Swift using `@c` or `@objc @implementation` within the same binary is ABI stable. + +## Implications on adoption + +The changes proposed here are backwards compatible with older runtimes. + +## Future directions + +This work opens the door to closer interoperability with the C language. + +### `@c` struct support + +A valuable addition would be supporting C compatible structs declared in Swift. We could consider exposing them to C as opaque data, or produce structs with a memory layout representable in C. Both have different use cases and advantages: + +* Using an opaque data representation would hide Swift details from C. Hiding these details allows the Swift struct to reference any Swift types and language features, without the concern of finding an equivalent C representation. This approach should be enough for the standard references to user data in C APIs. + +* Producing a Swift struct with a C memory layout would give the C code direct access to the data. This struct could be printed in the compatibility header as a normal C struct. This approach would need to be more restrictive on the Swift types and features used in the struct, starting with accepting only C representable types. + +### Custom calling conventions + +Defining a custom calling convention on a function may be a requirement by some API for callback functions and such. + +With this proposal, it should be possible to declare a custom calling convention by using `@c @implementation`. This allows to apply any existing C attribute on the definition in the C header. + +We could allow specifying the C calling conventions from Swift code with further work. Either by extending `@convention` to be accepted on `@c` and `@objc` global functions, and have it accept a wider set of conventions. Or by adding an optional named parameter to the `@c` attribute in the style of `@c(customCName, convention: stdcall)`. + +## Alternatives considered + +### `@c` attribute name + +This proposal uses the `@c` attribute on functions to identify them as C functions implemented in Swift and on enums to identify them as C compatible. This concise attribute clearly references interoperability with the C language. Plus, having an attribute specific to this feature aligns with `@objc` which is already used on some functions, enums, and for `@objc @implementation`. + +We considered some alternatives: + +- An official `@cdecl` may be more practical for discoverability but the terms *cdecl* and *decl* are compiler implementation details we do not wish to surface in the language. + +- An official `@expose(c)`, formalizing the experimental `@_expose(Cxx)`, would align the global function use case with what has been suggested for the C++ interop. However, sharing an attribute for the features described here may add complexity to both compiler implementation and user understanding of the language. + + While `@_expose(Cxx)` supports enums, it doesn't have the same requirement as `@objc` and `@c` for the raw type. The generated representation in the compatibility header for the enums differs too. The attribute `@_expose(Cxx)` also supports structs, while we consider supporting `@c` structs in the future, we have yet to pick the best approach so it would likely differ from the C++ one. + + Although sharing an attribute avoids adding a new one to the language, it also implies a similar behavior between the language interops. However, these behaviors already diverge and we may want to have each feature evolve differently in the future. + +### `@objc` attribute on global functions + +We use the `@objc` attribute on global functions to identify them as C functions implemented in Swift that are callable from Objective-C. This was more of a natural choice as `@objc` is already widely used for interoperability with Objective-C. + +We considered using instead `@c @objc` to make it more explicit that the behavior is similar to `@c`, and extending it to Objective-C is additive. We went against this option as it doesn't add much useful information besides being closer to the compiler implementation. + +### Compatibility header + +We decided to extend the existing compatibility header instead of introducing a new one specific to C compatibility. This allows content printed for Objective-C to reference C types printed earlier in the same header. Plus this follows the current behavior of the C++ interop which prints its own block in the same compatibility header. + +Since we use the same compatibility header, we also use the same compiler flags to request it being emitted. We considered adding a C specific flag as the main one, `-emit-objc-header`, is Objective-C specific. In practice build systems tend to use the `-path` variant, in that case we already have `-emit-clang-header-path` that applies well to the C language. We could add a `-emit-clang-header` flag but the practical use of such a flag would be limited. + +## Acknowledgements + +A special thank you goes to Becca Royal-Gordon, Joe Groff and many others for the past work on `@_cdecl` on which this proposal is built. diff --git a/proposals/0496-inline-always.md b/proposals/0496-inline-always.md new file mode 100644 index 0000000000..f10d161433 --- /dev/null +++ b/proposals/0496-inline-always.md @@ -0,0 +1,511 @@ +# `@inline(always)` attribute + +* Proposal: [SE-0496](0496-inline-always.md) +* Authors: [Arnold Schwaighofer](https://github.com/aschwaighofer) +* Review Manager: [Tony Allevato](https://github.com/allevato) +* Status: **Active review (October 2–16, 2025)** +* Implementation: [swiftlang/swift#84178](https://github.com/swiftlang/swift/pull/84178) +* Review: ([pitch](https://forums.swift.org/t/pitch-inline-always-attribute/82040)) ([review](https://forums.swift.org/t/se-0496-inline-always-attribute/82480)) + +## Introduction + +The Swift compiler performs an optimization that expands the body of a function +into the caller called inlining. Inlining exposes the code in the callee to the +code in the caller. After inlining, the Swift compiler has more context to +optimize the code across caller and callee leading to better optimization in +many cases. Inlining can increase code size. To avoid unnecessary code size +increases, the Swift compiler uses heuristics (properties of the code) to +determine whether to perform inlining. Sometimes these heuristics tell the +compiler not to inline a function even though it would be beneficial to do so. +The proposed attribute `@inline(always)` instructs the compiler to always inline +the annotated function into the caller giving the author explicit control over +the optimization. + +## Motivation + +Inlining a function referenced by a function call enables the optimizer to see +across function call boundaries. This can enable further optimization. The +decision whether to inline a function is driven by compiler heuristics that +depend on the shape of the code and can vary between compiler versions. + +In the following example the decision to inline might depend on the number of +instructions in `callee` and on detecting that the call to callee is frequently +executed because it is surrounded by a loop. Inlining this case would be +beneficial because the compiler is able to eliminate a store to a stack slot in +the `caller` after inlining the `callee` because the function's `inout` calling +convention ABI that requires an address no longer applies and further +optimizations are enabled by the caller's function's context. + +```swift +func callee(_ result: inout SomeValue, _ cond: Bool) { + result = SomeValue() + if cond { + // many lines of code ... + } +} + +func caller() { + var cond: Bool = false + var x : SomeValue = SomeValue() + for i in 0 ..< 1 { + callee(&x, cond) + } +} + +func callerAfterInlining(_ cond: Bool { + var x : SomeValue = SomeValue() + var cond: Bool = false + for i in 0 ..< 1 { + // Inlined `callee()`: + // Can keep `SomeValue()` in registers because no longer + // passed as an `inout` argument. + x = SomeValue() // Can hoist `x` out of the loop and perform constant + // propagation. + if cond { // Can remove the code under the conditional because it is + // known not to execute. + // many lines of code ... + } + } +} +``` + +The heuristic might fail to detect that code is frequently executed (surrounding +loop structures might be several calls up in the call chain) or the number of +instructions in the callee might be to large for the heuristic to decide that +inlining is beneficial. +Heuristics might change between compiler versions either directly or indirectly +because some properties of the internal representation of the optimized code +changes. +To give code authors reliable control over the inlining process we propose to +add an `@inline(always)` function attribute. + +This optimization control should instruct the compiler to inline the referenced +function or emit an error when it is not possible to do so. + +```swift +@inline(always) +func callee(_ result: inout SomeValue, _ cond: Bool) { + result = SomeValue() + if cond { + // many lines of code ... + } +} +``` + +## Proposed solution + +We desire for the attribute to function as an optimization control. That means +that the proposed `@inline(always)` attribute should emit an error diagnostic if +inlining is not possible in all optimization modes. However, this gets +complicated by the fact that the value of the function at a call site might be +determined dynamically at runtime: + +- Calls through first class function values + ```swift + @inline(always) f() {...} + + func a() { + let fv = f + fv() + } + ``` +- Calls through protocol values and protocol constraint generic types + ```swift + protocol P { + func method() + } + struct S : P { + @inline(always) + func method() {...} + } + func a(_ t: T) { + t.method() + let p : P = S() + p.method() + } + ``` +- Calls through class instance values and the method referenced is not `final` + ```swift + class C { + @inline(always) + func method() {...} + } + func a(c: C) { + c.method() + } + ``` + +In such cases, the compiler cannot determine at a call site which function is +applied without doing non-local analysis: either dataflow, or class hiarchy +analysis. +These cases are in contrast to when the called function can statically be +determined purely by looking at the call site, we refer to this set as direct +function references in the following: + +- Calls to free standing functions +- Calls to methods of `actor`, `struct`, `enum` type +- Calls to final methods of `class` type, and type (`static/class`) methods of + `class` type + +Therefore, in cases where the value of the function at a usage site is +dynamically derived we don't emit an error even if the dynamic value of the +applied function was annotated with `@inline(always)`. We only emit an error if +the annotated function is directly referenced and something would cause it to be +not inlined or if some property at the declaration site of the function would +make it not possible in the common case. + +Listing the different scenarios that can occur for a function marked with +`@inline(always)`: + +1. A function can definitely be inlined at the use site: direct function + references barring recursion cycles +2. A function can never be always inlined at a use site and we diagnose an + error: cycles in `@inline(always)` functions calling each other and all + references are direct. +3. A function can not be inlined reliably and we diagnose an error at the + declaration site: non-final method declaration +4. A function can not be inlined and we don't diagnose an error: calls through + first class function values, protocol values, and protocol constraint generic + types. + +### Direct function references + +Calls to freestanding functions, methods of `enum`, `struct`, `actor` types, +final methods of `class` types, and type methods of `class` types don't +dynamically dispatch to different implementations. Calls to such methods can +always be inlined barring the recursion limitation (see later). (case 1) + +```swift +struct S { + @inline(always) + final func method() {} +} + +func f() { + let s: S = ... + s.method() // can definitely be inlined +} + +class C { + @inline(always) + final func finalMethod() {} + + @inline(always) + class func method() {} +} + +class Sub : C {} + +func f2() { + let c: C = ... + c.finalMethod() // can definitely be inlined + let c2: Sub = .. + c2.finalMethod() // can definitely be inlined + C.method() // can definitely be inlined +} + +@inline(always) +func freestanding() {} + +func f3() { + freestanding() // can definitely be inlined +} + +``` + +### Non final class methods + +Swift performs dynamic dispatch for non-final methods of classes based on the +dynamic receiver type of the class instance value at a use site. Inferring the +value of that dynamic computation at compile time is not possible in many cases +and the success of inlining cannot be ensured. We treat a non-final method +declaration with `@inline(always)` as an declaration site error because we +assume that the intention of the attribute is that the method will be inlined in +most cases and this cannot be guaranteed (case 3). + +```swift +class C { + @inline(always) // error: non-final method marked @inline(always) + func method() {} +} + +class C2 : C { + @inline(always) // error: non-final method marked @inline(always) + override func method() {} +} + +func f(c: C) { + c.method() // dynamic type of c might be C or C2, could not ensure success + // of inlining in general +} +``` + +### Recursion + +Repeatedly inlining `@inline(always)` functions calling each other would lead to +an infinite cycle of inlining. We can never follow the `@inline(always)` +semantics and diagnose an error (case 2). + +```swift +@inline(always) +func callee() { + ... + if cond2 { + caller() // error: caller is marked @inline(always) and would create an + // inlining cycle + } +} + +@inline(always) +func caller() { + ... + if cond { + callee() + } +} +``` + +### First class function values + +Swift allows for functions as first class objects. They can be assigned to +variables and passed as arguments. The reference function of a function value +cannot be reliably be determined at the usage and is therefore not diagnosed as +an error (case 4). + +```swift +@inline(always) +func callee() {} + +func use(_ f: () -> ()) { + f() +} +func useFunctionValue() { + let f = callee + ... + f() // function value use, may be inlined but not diagnosed if not + use(callee) // function value use, may be inlined in `use()` but not diagnosed + // if not +} +``` + +### Protocol methods + +Protocol constraint or protocol typed values require a dynamic computation to +determine the eventual method called. Inferring the value of the eventual method +called at compile time is not possible in general and the success of inlining +cannot be ensured. We don't diagnose a usage site error if the underlying method +is marked with `@inline(always)` (case 4) + +```swift +protocol P { + func method() +} +struct S : P { + @inline(always) + func method() {} +} +final class C : P { + @inline(always) + func method() {} +} + +@inline(always) +func generic (_ t: T) { + t.method() +} + +func f() { + let p: P = S() + p.method() // might not get inlined, not diagnosed + generic(S()) // might not get inlined, not diagnosed + let p2: P = C() + p2.method() // might not get inlined, not diagnosed + generic(C()) // might not get inlined, not diagnosed +} +``` + +### Optimization control as optimization hint + +A clever optimizer might be able to derive the dynamic value at the +call site, in such cases the optimizer shall respect the optimization control +and perform inlining. + +In the following example the functions will be inlined when build with higher +optimization levels than `-Onone`. + +```swift +@inline(always) +func binaryOp(_ left: T, _ right: T, _ op: (T, T) -> T) -> T { + op(left, right) +} + +@inline(always) +func add(_ left: Int, _ right: Int) -> Int { left + right } + +print(binaryOp(5, 10, add)) +print(binaryOp(5, 10) { add($0, $1) }) +``` + + +### Interaction with `@inlinable` + +`@inlinable` makes the function body available to clients (callers in other +modules) in library evolution mode. Functions with `open`, `public`, or +`package` level access cause emission of an ABI entry point for clients to call +but in the absence of aforementioned attributes do not make the body available +to the client. + +`@inline(always)` intention is to be able to guarantee that inlining will happen +for any caller inside or outside the defining module therefore it makes sense to +require the use of "@inlinable" attribute with them. This attribute could be +required to be explicitly stated. And for it to be an error when the attribute +is omitted. + +```swift +@inline(always) +@inlinable // or @_alwaysEmitIntoClient +public func caller() { ... } + +@inline(always) // error: a public function marked @inline(always) must be marked @inlinable +public func callee() { +} +``` + +Alternatively, the attribute could be implicitly implied by the usage of +`@inline(always)`. We take the position that it should be implied to avoid the +redundancy of spelling it out. + +For access levels equal and lower than `internal` `@inlinable` is not implied. + +As a consequence all the rules that apply to `@inlinable` also apply to +`public`/`open`/`package` declarations marked with `@inline(always)`. + +```swift +internal func g() { ... } + +@inline(always) +public func inlinableImplied() { + g() // error: global function 'g()' is internal and cannot be referenced from an + '@inlinable' function +} +``` + +### Interaction with `@usableFromInline` + +A `public` `@inlinable` function can reference a function with `internal` access +if it is either `@inlinable` (see above) or `@usableFromInline`. `@usableFromInline` +ensures that there is a public entry point to the `internal` level function but +does not ensure that the body of the function is available to external +modules. Therefore, it is an error to combine `@inline(always)` with a +`@usableFromInline` function as we cannot guarantee that the function can +always be inlined. + +```swift +@inline(always) // error: an internal function marked with `@inline(always)` and + `@usableFromInline` could be referenced from an + `@inlinable` function and must be marked inlinable +@usableFromInline +internal func callee() {} + +@inlinable +public func caller() { + callee() // could not inline callee into external module +} +``` + +### Module internal access levels + +To mark `internal`, `private` and `fileprivate` function declarations +with `@inline(always)` does not imply the `@inlinable` attribute's semantics. +They can only be referenced from within the module. `internal` declarations can +be marked with `@inlinable` if this is required by the presence of other +`@inlinable` (or public `@inline(always)`) functions that reference them. + + +```swift +public func caller() { + callee() +} + +@inline(always) // okay because caller would force either `@inlinable` or + // `@usableFromInline` if it was marked @inlinable itself +internal func callee() { +} + + +@inline(always) // okay can only referenced from within the module +private func callee2() { +} +``` + +## Source compatibility + +This proposal is additive. Existing code has not used the attribute. It has no +impact on existing code. Existing references to functions in libraries that are +now marked with `@inline(always)` will continue to compile successfully with the +added effect that functions will get inlined (that could have happened with +changes to inlining heuristic). + +## ABI compatibility + +The addition of the attribute has no effect on ABI compatibility. We chose to +imply `@inlinable` for `public` (et al.) declarations which will continue to +emit an entry point for existing binary clients. + +## Implications on adoption + +This feature can be freely adopted and un-adopted in source +code with no deployment constraints and without affecting source or ABI +compatibility. + +## Future directions + +`@inline(always)` can be too restrictive in cases where inlining is only +required within a module. For such cases we can introduce an `@inline(module)` +attribute in the future. + + +```swift +@inlinable +public caller() { + if coldPath { + callee() + } +} + +public otherCaller() { + if hotPath { + callee() + } +} + +@inline(module) +@usableFromInline +internal func callee() { +} +``` + +## Alternatives considered + +We could treat `@inline(always)` as an optimization hint that does not need to +be enforced or applied at all optimization levels similar to how the existing +`@inline(__always)` attribute functions and not emit errors if it cannot be +guaranteed to be uphold when the function is directly referenced. +This would deliver less predictable optimization behavior in cases where authors +overlooked requirements for inlining to happen such as not marking a public +function as `@inlinable`. + +With respect to `@inlinable` an initial draft of the proposal suggested to +require spelling the `@inlinable` attribute on `public` declarations or an error +would be displayed. The argument was made that this would ensure that authors +would be aware of the additional semantics implied by the attribute: the body is +exposed. This was juxtaposed by the argument that spelling both `@inlinable` and +`@inline(always)` is redundant. + +## Acknowledgments + +Thanks to [Jordan Rose](https://forums.swift.org/t/optimization-controls-and-optimization-hints/81612/7) for pointing out that inlining can't be always guaranteed, specifically the case of closures. +Thanks to [Xiaodi Wu](https://forums.swift.org/t/pitch-inline-always-attribute/82040/7) for proposing inferring `@inlinable`. +Thanks to [Tony Allevato](https://github.com/swiftlang/swift-evolution/pull/2958#discussion_r2379238582) for suggesting to error on on non-final methods and +providing editing feedback. +Thanks to [Doug Gregor](https://github.com/DougGregor), [Joe Groff](https://github.com/jckarter), [Tim Kientzle](https://github.com/tbkka), and [Allan Shortlidge](https://github.com/tshortli) for discussions related to the feature. diff --git a/proposals/testing/0001-refactor-bug-inits.md b/proposals/testing/0001-refactor-bug-inits.md new file mode 100644 index 0000000000..4d9a77b5e7 --- /dev/null +++ b/proposals/testing/0001-refactor-bug-inits.md @@ -0,0 +1,173 @@ +# Dedicated `.bug()` functions for URLs and IDs + +* Proposal: [ST-0001](0001-refactor-bug-inits.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Implemented (Swift 6.0)** +* Implementation: [swiftlang/swift-testing#401](https://github.com/swiftlang/swift-testing/pull/401) +* Review: ([pitch](https://forums.swift.org/t/pitch-dedicated-bug-functions-for-urls-and-ids/71842)) ([acceptance](https://forums.swift.org/t/swt-0001-dedicated-bug-functions-for-urls-and-ids/71842/2)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0001](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0001-refactor-bug-inits.md). + +## Introduction + +One of the features of swift-testing is a test traits system that allows +associating metadata with a test suite or test function. One trait in +particular, `.bug()`, has the potential for integration with development tools +but needs some refinement before integration would be practical. + +## Motivation + +A test author can associate a bug (AKA issue, problem, ticket, etc.) with a test +using the `.bug()` trait, to which they pass an "identifier" for the bug. The +swift-testing team's intent here was that a test author would pass the unique +identifier of the bug in the test author's preferred bug-tracking system (e.g. +GitHub Issues, Bugzilla, etc.) and that any tooling built around this trait +would be able to infer where the bug was located and how to view it. + +It became clear immediately that a generic system for looking up bugs by unique +identifier in an arbitrary and unspecified database wouldn't be a workable +solution. So we modified the description of `.bug()` to explain that, if the +identifier passed to it was a valid URL, then it would be "interpreted" as a URL +and that tools could be designed to open that URL as needed. + +This design change then placed the burden of parsing each `.bug()` trait and +potentially mapping it to a URL on tools. swift-testing itself avoids linking to +or using Foundation API such as `URL`, so checking for a valid URL inside the +testing library was not feasible either. + +## Proposed solution + +To solve the underlying problem and allow test authors to specify a URL when +available, or just an opaque identifier otherwise, we propose splitting the +`.bug()` function up into two overloads: + +- The first overload takes a URL string and additional optional metadata; +- The second overload takes a bug identifier as an opaque string or integer and, + optionally, a URL string. + +Test authors are then free to specify any combination of URL and opaque +identifier depending on the information they have available and their specific +needs. Tools authors are free to consume either or both of these properties and +present them where appropriate. + +## Detailed design + +The `Bug` trait type and `.bug()` trait factory function shall be refactored +thusly: + +```swift +/// A type representing a bug report tracked by a test. +/// +/// To add this trait to a test, use one of the following functions: +/// +/// - ``Trait/bug(_:_:)`` +/// - ``Trait/bug(_:id:_:)-10yf5`` +/// - ``Trait/bug(_:id:_:)-3vtpl`` +public struct Bug: TestTrait, SuiteTrait, Equatable, Hashable, Codable { + /// A URL linking to more information about the bug, if available. + /// + /// The value of this property represents a URL conforming to + /// [RFC 3986](https://www.ietf.org/rfc/rfc3986.txt). + public var url: String? + + /// A unique identifier in this bug's associated bug-tracking system, if + /// available. + /// + /// For more information on how the testing library interprets bug + /// identifiers, see . + public var id: String? + + /// The human-readable title of the bug, if specified by the test author. + public var title: Comment? +} + +extension Trait where Self == Bug { + /// Construct a bug to track with a test. + /// + /// - Parameters: + /// - url: A URL referring to this bug in the associated bug-tracking + /// system. + /// - title: Optionally, the human-readable title of the bug. + /// + /// - Returns: An instance of ``Bug`` representing the specified bug. + public static func bug(_ url: _const String, _ title: Comment? = nil) -> Self + + /// Construct a bug to track with a test. + /// + /// - Parameters: + /// - url: A URL referring to this bug in the associated bug-tracking + /// system. + /// - id: The unique identifier of this bug in its associated bug-tracking + /// system. + /// - title: Optionally, the human-readable title of the bug. + /// + /// - Returns: An instance of ``Bug`` representing the specified bug. + public static func bug(_ url: _const String? = nil, id: some Numeric, _ title: Comment? = nil) -> Self + + /// Construct a bug to track with a test. + /// + /// - Parameters: + /// - url: A URL referring to this bug in the associated bug-tracking + /// system. + /// - id: The unique identifier of this bug in its associated bug-tracking + /// system. + /// - title: Optionally, the human-readable title of the bug. + /// + /// - Returns: An instance of ``Bug`` representing the specified bug. + public static func bug(_ url: _const String? = nil, id: _const String, _ title: Comment? = nil) -> Self +} +``` + +The `@Test` and `@Suite` macros have already been modified so that they perform +basic validation of a URL string passed as input and emit a diagnostic if the +URL string appears malformed. + +## Source compatibility + +This change is expected to be source-breaking for test authors who have already +adopted the existing `.bug()` functions. This change is source-breaking for code +that directly refers to these functions by their signatures. This change is +source-breaking for code that uses the `identifier` property of the `Bug` type +or expects it to contain a URL. + +## Integration with supporting tools + +Tools that integrate with swift-testing and provide lists of tests or record +results after tests have run can use the `Bug` trait on tests to present +relevant identifiers and/or URLs to users. + +Tools that use the experimental event stream output feature of the testing +library will need a JSON schema for bug traits on tests. This work is tracked in +a separate upcoming proposal. + +## Alternatives considered + +- Inferring whether or not a bug identifier was a URL by parsing it at runtime + in tools. As discussed above, this option would require every tool that + integrates with swift-testing to provide its own URL-parsing logic. + +- Using different argument labels (e.g. the label `url` for the URL argument + and/or no label for the `id` argument.) We felt that URLs, which are + recognizable by their general structure, did not need labels. At least one + argument must have a label to avoid ambiguous resolution of the `.bug()` + function at compile time. + +- Inferring whether or not a bug identifier was a URL by parsing it at compile- + time or at runtime using `Foundation.URL` or libcurl. swift-testing actively + avoids linking to Foundation if at all possible, and libcurl would be a + platform-specific solution (Windows doesn't ship with libcurl, but does have + `InternetCrackUrlW()` whose parsing engine differs.) We also run the risk of + inappropriately interpreting some arbitrary bug identifier as a URL when it is + not meant to be parsed that way. + +- Removing the `.bug()` trait. We see this particular trait as having strong + potential for integration with tools and for use by test authors; removing it + because we can't reliably parse URLs would be unfortunate. + +## Acknowledgments + +Thanks to the swift-testing team and managers for their contributions! Thanks to +our community for the initial feedback around this feature. diff --git a/proposals/testing/0002-json-abi.md b/proposals/testing/0002-json-abi.md new file mode 100644 index 0000000000..516b986838 --- /dev/null +++ b/proposals/testing/0002-json-abi.md @@ -0,0 +1,428 @@ +# A stable JSON-based ABI for tools integration + +* Proposal: [ST-0002](0002-json-abi.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Implemented (Swift 6.0)** +* Implementation: [swiftlang/swift-testing#383](https://github.com/swiftlang/swift-testing/pull/383), + [swiftlang/swift-testing#402](https://github.com/swiftlang/swift-testing/pull/402) +* Review: ([pitch](https://forums.swift.org/t/pitch-a-stable-json-based-abi-for-tools-integration/72627)) ([acceptance](https://forums.swift.org/t/pitch-a-stable-json-based-abi-for-tools-integration/72627/4)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0002](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0002-json-abi.md). + +## Introduction + +One of the core components of Swift Testing is its ability to interoperate with +Xcode 16, VS Code, and other tools. Swift Testing has been fully open-sourced +across all platforms supported by Swift, and can be added as a package +dependency (or—eventually—linked from the Swift toolchain.) + +## Motivation + +Because Swift Testing may be used in various forms, and because integration with +various tools is critical to its success, we need it to have a stable interface +that can be used regardless of how it's been added to a package. There are a few +patterns in particular we know we need to support: + +- An IDE (e.g. Xcode 16) that builds and links its own copy of Swift Testing: + the copy used by the IDE might be the same as the copy that tests use, in + which case interoperation is trivial, but it may also be distinct if the tests + use Swift Testing as a package dependency. + + In the case of Xcode 16, Swift Testing is built as a framework much like + XCTest and is automatically linked by test targets in an Xcode project or + Swift package, but if the test target specifies a package dependency on Swift + Testing, that dependency will take priority when the test code is compiled. + +- An IDE (e.g. VS Code) that does _not_ link directly to Swift Testing (and + perhaps, as with VS Code, cannot because it is not natively compiled): such an + IDE needs a way to configure and invoke test code and then to read events back + as they occur, but cannot touch the Swift symbols used by the tests. + + In the case of VS Code, because it is implemented using TypeScript, it is not + able to directly link to Swift Testing or other Swift libraries. In order for + it to interpret events from a test run like "test started" or "issue + recorded", it needs to receive those events in a format it can understand. + +Tools integration is important to the success of Swift Testing. The more tools +provide integrations for it, the more likely developers are to adopt it. The +more developers adopt, the more tests are written. And the more tests are +written, the better our lives as software engineers will be. + +## Proposed solution + +We propose defining and implementing a stable ABI for using Swift Testing that +can be reliably adopted by various IDEs and other tools. There are two aspects +of this ABI we need to implement: + +- A stable entry point function that can be resolved dynamically at runtime (on + platforms with dynamic loaders such as Darwin, Linux, and Windows.) This + function needs a signature that will not change over time and which will take + input and pass back asynchronous output in a format that a wide variety of + tools will be able to interpret (whether they are written in Swift or not.) + + This function should be implemented in Swift as it is expected to be used by + code that can call into Swift, but which cannot rely on the specific binary + minutiae of a given copy of Swift Testing. + +- A stable format for input that can be passed to the entry point function and + which can also be passed at the command line; and a stable format for output + that can be consumed by tools to interpret test results. + + Some tools cannot directly link to Swift code and must instead rely on + command-line invocations of `swift test`. These tools will be able to pass + their test configuration and options as an argument in the stable format and + will be able to receive event information in the same stable format via a + dedicated channel such as a file or named pipe. + +> [!NOTE] +> This document proposes defining a stable format for input and output, but only +> actually defines the JSON schema for _output_. We intend to define the schema +> for input in a subsequent proposal. +> +> In the interim, early adopters can encode an instance of Swift Testing's +> `__CommandLineArguments_v0` type using `JSONEncoder`. + +## Detailed design + +We propose defining the stable input and output format using JSON as it is +widely supported across platforms and languages. The proposed JSON schema for +output is defined [here](https://github.com/swiftlang/swift-testing/blob/main/Documentation/ABI/JSON.md). + +### Example output + +The proposed schema is a sequence of JSON objects written to an event handler or +file stream. When a test run starts, Swift Testing first emits a sequence of +JSON objects representing each test that is part of the planned run. For +example, this is the JSON representation of Swift Testing's own `canGetStdout()` +test function: + +```json +{ + "kind": "test", + "payload": { + "displayName": "Can get stdout", + "id": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4", + "isParameterized": false, + "kind": "function", + "name": "canGetStdout()", + "sourceLocation": { + "column": 4, + "fileID": "TestingTests/FileHandleTests.swift", + "line": 33 + } + }, + "version": 0 +} +``` + +A tool that is observing this data stream can build a map or dictionary of test +IDs to comprehensive test details if needed. Once all tests in the planned run +have been written out, testing begins. Swift Testing writes a sequence of JSON +objects representing various events such as "test started" or "issue recorded". +For example, here is an abridged sequence of events generated for a test that +records a failed expectation: + +```json +{ + "kind": "event", + "payload": { + "instant": { + "absolute": 266418.545786299, + "since1970": 1718302639.76747 + }, + "kind": "testStarted", + "messages": [ + { + "symbol": "default", + "text": "Test \"Can get stdout\" started." + } + ], + "testID": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4" + }, + "version": 0 +} + +{ + "kind": "event", + "payload": { + "instant": { + "absolute": 266636.524236724, + "since1970": 1718302857.74857 + }, + "issue": { + "isKnown": false, + "sourceLocation": { + "column": 7, + "fileID": "TestingTests/FileHandleTests.swift", + "line": 29 + } + }, + "kind": "issueRecorded", + "messages": [ + { + "symbol": "fail", + "text": "Expectation failed: (EOF → -1) == (feof(fileHandle) → 0)" + } + ], + "testID": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4" + }, + "version": 0 +} + +{ + "kind": "event", + "payload": { + "instant": { + "absolute": 266636.524741106, + "since1970": 1718302857.74908 + }, + "kind": "testEnded", + "messages": [ + { + "symbol": "fail", + "text": "Test \"Can get stdout\" failed after 0.001 seconds with 1 issue." + } + ], + "testID": "TestingTests.FileHandleTests/canGetStdout()/FileHandleTests.swift:33:4" + }, + "version": 0 +} +``` + +Each event includes zero or more "messages" that Swift Testing intends to +present to the user. These messages contain human-readable text as well as +abstractly-specified symbols that correspond to the output written to the +standard error stream of the test process. Tools can opt to present these +messages in whatever ways are appropriate for their interfaces. + +### Invoking from the command line + +When invoking `swift test`, we propose adding three new arguments to Swift +Package Manager: + +| Argument | Value Type | Description | +|---|:-:|---| +| `--configuration-path` | File system path | Specifies a path to a file, named pipe, etc. containing test configuration/options. | +| `--event-stream-output-path` | File system path | Specifies a path to a file, named pipe, etc. to which output should be written. | +| `--event-stream-version` | Integer | Specifies the version of the stable JSON schema to use for output. | + +The process for adding arguments to Swift Package Manager is separate from the +process for Swift Testing API changes, so the names of these arguments are +speculative and are subject to change as part of the Swift Package Manager +review process. + +If `--configuration-path` is specified, Swift Testing will open it for reading +and attempt to decode its contents as JSON. If `--event-stream-output-path` is +specified, Swift Testing will open it for writing and will write a sequence of +[JSON Lines](https://jsonlines.org) to it representing the data and events +produced by the test run. `--event-stream-version` determines the stable schema +used for output; pass `0` to match the schema proposed in this document. + +> [!NOTE] +> If `--event-stream-output-path` is specified but `--event-stream-version` is +> not, the format _currently_ used is based on direct JSON encodings of the +> internal Swift structures used by Swift Testing. This format is necessary to +> support Xcode 16 Beta 1. In the future, the default value of this argument +> will be assumed to equal the newest available JSON schema version (`0` as of +> this document's acceptance, i.e. the JSON schema will match what we are +> proposing here until a new schema supersedes it.) +> +> Tools authors that rely on the JSON schema are strongly advised to specify a +> version rather than relying on this behavior to avoid breaking changes in the +> future. + +On platforms that support them, callers can use a named pipe with +`--event-stream-output-path` to get live results back from the test run rather +than needing to wait until the file is closed by the test process. Named pipes +can be created on Darwin or Linux with the POSIX [`mkfifo()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/mkfifo.2.html) +function or on Windows with the [`CreateNamedPipe()`](https://learn.microsoft.com/en-us/windows/win32/api/namedpipeapi/nf-namedpipeapi-createnamedpipew) +function. + +If `--configuration-path` is specified in addition to explicit command-line +options like `--no-parallel`, the explicit command-line options take priority. + +### Invoking from Swift + +Tools that can link to and call Swift directly have the option of instantiating +the tools-only SPI type `Runner`, however this is only possible if the tools and +the test target link to the exact same copy of Swift Testing. To support tools +that may link to a different copy (intentionally or otherwise), we propose +adding an exported symbol to the Swift Testing library with the following Swift +signature: + +```swift +@_spi(ForToolsIntegrationOnly) +public enum ABIv0 { + /* ... */ + + /// The type of the entry point to the testing library used by tools that want + /// to remain version-agnostic regarding the testing library. + /// + /// - Parameters: + /// - configurationJSON: A buffer to memory representing the test + /// configuration and options. If `nil`, a new instance is synthesized + /// from the command-line arguments to the current process. + /// - recordHandler: A JSON record handler to which is passed a buffer to + /// memory representing each record as described in `ABI/JSON.md`. + /// + /// - Returns: Whether or not the test run finished successfully. + /// + /// - Throws: Any error that occurred prior to running tests. Errors that are + /// thrown while tests are running are handled by the testing library. + public typealias EntryPoint = @convention(thin) @Sendable ( + _ configurationJSON: UnsafeRawBufferPointer?, + _ recordHandler: @escaping @Sendable (_ recordJSON: UnsafeRawBufferPointer) -> Void + ) async throws -> Bool + + /// The entry point to the testing library used by tools that want to remain + /// version-agnostic regarding the testing library. + /// + /// The value of this property is a Swift function that can be used by tools + /// that do not link directly to the testing library and wish to invoke tests + /// in a binary that has been loaded into the current process. The value of + /// this property is accessible from C and C++ as a function with name + /// `"swt_abiv0_getEntryPoint"` and can be dynamically looked up at runtime + /// using `dlsym()` or a platform equivalent. + /// + /// The value of this property can be thought of as equivalent to + /// `swift test --event-stream-output-path` except that, instead of streaming + /// JSON records to a named pipe or file, it streams them to an in-process + /// callback. + public static var entryPoint: EntryPoint { get } +} +``` + +The inputs and outputs to this function are typed as `UnsafeRawBufferPointer` +rather than `Data` because the latter is part of Foundation, and adding a public +dependency on a Foundation type would make it very difficult for Foundation to +adopt Swift Testing. It is a goal of the Swift Testing team to keep our Swift +dependency list as small as possible. + +### Invoking from C or C++ + +We expect most tools that need to make use of this entry point will not be able +to directly link to the exported Swift symbol and will instead need to look it +up at runtime using a platform-specific interface such as [`dlsym()`](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlsym.3.html) +or [`GetProcAddress()`](https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-getprocaddress). +The `ABIv0.entryPoint` property's getter will be exported to C and C++ as: + +```c++ +extern "C" const void *_Nonnull swt_abiv0_getEntryPoint(void); +``` + +The value returned from this C function is a direct representation of the value +of `ABIv0.entryPoint` and can be cast back to its Swift function type using +[`unsafeBitCast(_:to:)`](https://developer.apple.com/documentation/swift/unsafebitcast%28_%3Ato%3A%29). + +On platforms where data-pointer-to-function-pointer conversion is disallowed per +the C standard, this operation is unsupported. See §6.3.2.3 and §J.5.7 of +[the C standard](https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf). + +> [!NOTE] +> Swift Testing is statically linked into the main executable when it is +> included as a package dependency. On Linux and other platforms that use the +> ELF executable format, symbol information for the main executable may not be +> available at runtime unless the `--export-dynamic` flag is passed to the +> linker. + +## Source compatibility + +The changes proposed in this document are additive. + +## Integration with supporting tools + +Tools are able to use the proposed additions as described above. + +## Future directions + +- Extending the JSON schema to cover _input_ as well as _output_. As discussed, + we will do so in a subsequent proposal. + +- Extending the JSON schema to include richer information about events such as + specific mismatched values in `#expect()` calls. This information is complex + and we need to take care to model it efficiently and clearly. + +- Adding Markdown or other formats to event messages. Rich text can be used by + tools to emphasize values, switch to code voice, provide improved + accessibility, etc. + +- Adding additional entry points for different access patterns. We anticipate + that a Swift function and a command-line interface are sufficient to cover + most real-world use cases, but it may be the case that tools could use other + mechanisms for starting test runs such as: + - Pure C or Objective-C interfaces; + - A WebAssembly and/or JavaScript [`async`-compatible](https://github.com/WebAssembly/component-model/blob/2f447274b5028f54c549cb4e28ceb493a471dd4b/design/mvp/Async.md) + interface; + - Platform-specific interfaces; or + - Direct bindings to other languages like Rust, Go, C#, etc. + +## Alternatives considered + +- Doing nothing. If we made no changes, we would be effectively requiring + developers to use Xcode for all Swift Testing development and would be + requiring third-party tools to parse human-readable command-line output. This + approach would run counter to several of the Swift project's high-level goals + and would not represent a true cross-platform solution. + +- Using direct JSON encodings of Swift Testing's internal types to represent + output. We initially attempted this and you can see the results in the Swift + Testing repository if you look for "snapshot" types. A major downside became + apparent quickly: these data types don't make for particularly usable JSON + unless you're using `JSONDecoder` to convert back to them, and the default + JSON encodings produced with `JSONEncoder` are not stable if we e.g. add + enumeration cases with associated values or add non-optional fields to types. + +- Using a format other than JSON. We considered using XML, YAML, Apple property + lists, and a few other formats. JSON won out pretty quickly though: it is + widely supported across platforms and languages and it is trivial to create + Swift structures that encode to a well-designed JSON schema using + `JSONEncoder`. Property lists would be just as easy to create, but it is a + proprietary format and would not be trivially decodable on non-Apple platforms + or using non-Apple tools. + +- Exposing the C interface as a function that returns heap-allocated memory + containing a Swift function reference. This allows us to emit a "thick" Swift + function but requires callers to manually manage the resulting memory, and it + may be difficult to reason about code that requires an extra level of pointer + indirection. By having the C entry point function return a thin Swift function + instead, the caller need only bitcast it and can call it directly, and the + equivalent Swift interface can simply be a property getter rather than a + function call. + +- Exposing the C interface as a function that takes a callback and a completion + handler as might traditionally used by Objective-C callers, of the form: + + ```c++ + extern "C" void swt_abiv0_entryPoint( + __attribute__((__noescape__)) const void *_Nullable configurationJSON, + size_t configurationJSONLength, + void *_Null_unspecified context, + void (*_Nonnull recordHandler)( + __attribute__((__noescape__)) const void *recordJSON, + size_t recordJSONLength, + void *_Null_unspecified context + ), + void (*_Nonnull completionHandler)( + _Bool success, + void *_Null_unspecified context + ) + ); + ``` + + The known clients of the native entry point function are all able to call + Swift code and do not need this sort of interface. If there are other clients + that would need the entry point to use a signature like this one, it would be + straightforward to implement it in a future amendment to this proposal. + +## Acknowledgments + +Thanks much to [Dennis Weissmann](https://github.com/dennisweissmann) for his +tireless work in this area and to [Paul LeMarquand](https://github.com/plemarquand) +for putting up with my incessant revisions and nitpicking while he worked on +VS Code's Swift Testing support. + +Thanks to the rest of the Swift Testing team for reviewing this proposal and the +JSON schema and to the community for embracing Swift Testing! diff --git a/proposals/testing/0003-make-serialized-trait-api.md b/proposals/testing/0003-make-serialized-trait-api.md new file mode 100644 index 0000000000..488035bedf --- /dev/null +++ b/proposals/testing/0003-make-serialized-trait-api.md @@ -0,0 +1,154 @@ +# Make .serialized trait API + +* Proposal: [ST-0003](0003-make-serialized-trait-api.md) +* Authors: [Dennis Weissmann](https://github.com/dennisweissmann) +* Status: **Implemented (Swift 6.0)** +* Implementation: +[swiftlang/swift-testing#535](https://github.com/swiftlang/swift-testing/pull/535) +* Review: ([pitch](https://forums.swift.org/t/pitch-make-serialized-trait-public-api/73147)) ([acceptance](https://forums.swift.org/t/pitch-make-serialized-trait-public-api/73147/5)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0003](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0003-make-serialized-trait-api.md). + +## Introduction + +We propose promoting the existing `.serialized` trait to public API. This trait +enables developers to designate tests or test suites to run serially, ensuring +sequential execution where necessary. + +## Motivation + +The Swift Testing library defaults to parallel execution of tests, promoting +efficiency and isolation. However, certain test scenarios demand strict +sequential execution due to shared state or complex dependencies between tests. +The `.serialized` trait provides a solution by allowing developers to enforce +serial execution for specific tests or suites. + +While global actors ensure that only one task associated with that actor runs +at any given time, thus preventing concurrent access to actor state, tasks can +yield and allow other tasks to proceed, potentially interleaving execution. +That means global actors do not ensure that a specific test runs entirely to +completion before another begins. A testing library requires a construct that +guarantees that each annotated test runs independently and completely (in its +suite), one after another, without interleaving. + +## Proposed Solution + +We propose exposing the `.serialized` trait as a public API. This attribute can +be applied to individual test functions or entire test suites, modifying the +test execution behavior to enforce sequential execution where specified. + +Annotating just a single test in a suite does not enforce any serialization +behavior - the testing library encourages parallelization and the bar to +degrade overall performance of test execution should be high. +Additionally, traits apply inwards - it would be unexpected to impact the exact +conditions of a another test in a suite without applying a trait to the suite +itself. +Thus, this trait should only be applied to suites (to enforce serial execution +of all tests inside it) or parameterized tests. If applied to just a test this +trait does not have any effect. + +## Detailed Design + +The `.serialized` trait functions as an attribute that alters the execution +scheduling of tests. When applied, it ensures that tests or suites annotated +with `.serialized` run serially. + +```swift +/// A type that affects whether or not a test or suite is parallelized. +/// +/// When added to a parameterized test function, this trait causes that test to +/// run its cases serially instead of in parallel. When applied to a +/// non-parameterized test function, this trait has no effect. When applied to a +/// test suite, this trait causes that suite to run its contained test functions +/// and sub-suites serially instead of in parallel. +/// +/// This trait is recursively applied: if it is applied to a suite, any +/// parameterized tests or test suites contained in that suite are also +/// serialized (as are any tests contained in those suites, and so on.) +/// +/// This trait does not affect the execution of a test relative to its peers or +/// to unrelated tests. This trait has no effect if test parallelization is +/// globally disabled (by, for example, passing `--no-parallel` to the +/// `swift test` command.) +/// +/// To add this trait to a test, use ``Trait/serialized``. +public struct ParallelizationTrait: TestTrait, SuiteTrait {} + +extension Trait where Self == ParallelizationTrait { + /// A trait that serializes the test to which it is applied. + /// + /// ## See Also + /// + /// - ``ParallelizationTrait`` + public static var serialized: Self { get } +} +``` + +The call site looks like this: + +```swift +@Test(.serialized, arguments: Food.allCases) func prepare(food: Food) { + // This function will be invoked serially, once per food, because it has the + // .serialized trait. +} + +@Suite(.serialized) struct FoodTruckTests { + @Test(arguments: Condiment.allCases) func refill(condiment: Condiment) { + // This function will be invoked serially, once per condiment, because the + // containing suite has the .serialized trait. + } + + @Test func startEngine() async throws { + // This function will not run while refill(condiment:) is running. One test + // must end before the other will start. + } +} + +@Suite struct FoodTruckTests { + @Test(.serialized) func startEngine() async throws { + // This function will not run serially - it's not a parameterized test and + // the suite is not annotated with the `.serialized` trait. + } + + @Test func prepareFood() async throws { + // It doesn't matter if this test is `.serialized` or not, traits applied + // to other tests won't affect this test don't impact other tests. + } +} +``` + +## Source Compatibility + +Introducing `.serialized` as a public API does not have any impact on existing +code. Tests will continue to run in parallel by default unless explicitly +marked with `.serialized`. + +## Integration with Supporting Tools + +N/A. + +## Future Directions + +There might be asks for more advanced and complex ways to affect parallelization +which include ways to specify dependencies between tests ie. "Require `foo()` to +run before `bar()`". + +## Alternatives Considered + +Alternative approaches, such as relying solely on global actors for test +isolation, were considered. However, global actors do not provide the +deterministic, sequential execution required for certain testing scenarios. The +`.serialized` trait offers a more explicit and flexible mechanism, ensuring +that each designated test or suite runs to completion without interruption. + +Various more complex parallelization and serialization options were discussed +and considered but ultimately disregarded in favor of this simple yet powerful +implementation. + +## Acknowledgments + +Thanks to the swift-testing team and managers for their contributions! Thanks +to our community for the initial feedback around this feature. diff --git a/proposals/testing/0004-constrain-the-granularity-of-test-time-limit-durations.md b/proposals/testing/0004-constrain-the-granularity-of-test-time-limit-durations.md new file mode 100644 index 0000000000..ea00728fca --- /dev/null +++ b/proposals/testing/0004-constrain-the-granularity-of-test-time-limit-durations.md @@ -0,0 +1,205 @@ +# Constrain the granularity of test time limit durations + +* Proposal: [ST-0004](0004-constrain-the-granularity-of-test-time-limit-durations.md) +* Authors: [Dennis Weissmann](https://github.com/dennisweissmann) +* Status: **Implemented (Swift 6.0)** +* Implementation: +[swiftlang/swift-testing#534](https://github.com/swiftlang/swift-testing/pull/534) +* Review: ([pitch](https://forums.swift.org/t/pitch-constrain-the-granularity-of-test-time-limit-durations/73146)) ([acceptance](https://forums.swift.org/t/pitch-constrain-the-granularity-of-test-time-limit-durations/73146/3)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0004](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0004-constrain-the-granularity-of-test-time-limit-durations.md). + +## Introduction + +Sometimes tests might get into a state (either due the test code itself or due +to the code they're testing) where they don't make forward progress and hang. +Swift Testing provides a way to handle these issues using the TimeLimit trait: + +```swift +@Test(.timeLimit(.minutes(60)) +func testFunction() { ... } +``` + +Currently there exist multiple overloads for the `.timeLimit` trait: one that +takes a `Swift.Duration` which allows for arbitrary `Duration` values to be +passed, and one that takes a `TimeLimitTrait.Duration` which constrains the +minimum time limit as well as the increment to 1 minute. + +## Motivation + +Small time limit values in particular cause more harm than good due to tests +running in environments with drastically differing performance characteristics. +Particularly when running in CI systems or on virtualized hardware tests can +run much slower than at desk. +Swift Testing should help developers use a reasonable time limit value in its +API without developers having to refer to the documentation. + +It is crucial to emphasize that unit tests failing due to exceeding their +timeout should be exceptionally rare. At the same time, a spurious unit test +failure caused by a short timeout can be surprisingly costly, potentially +leading to an entire CI pipeline being rerun. Determining an appropriate +timeout for a specific test can be a challenging task. + +Additionally, when the system intentionally runs multiple tests simultaneously +to optimize resource utilization, the scheduler becomes the arbiter of test +execution. Consequently, the test may take significantly longer than +anticipated, potentially due to external factors beyond the control of the code +under test. + +A unit test should be capable of failing due to hanging, but it should not fail +due to being slow, unless the developer has explicitly indicated that it +should, effectively transforming it into a performance test. + +The time limit feature is *not* intended to be used to apply small timeouts to +tests to ensure test runtime doesn't regress by small amounts. This feature is +intended to be used to guard against hangs and pathologically long running +tests. + +## Proposed Solution + +We propose changing the `.timeLimit` API to accept values of a new `Duration` +type defined in `TimeLimitTrait` which only allows for `.minute` values to be +passed. +This type already exists as SPI and this proposal is seeking to making it API. + +## Detailed Design + +The `TimeLimitTrait.Duration` struct only has one factory method: +```swift +public static func minutes(_ minutes: some BinaryInteger) -> Self +``` + +That ensures 2 things: +1. It's impossible to create short time limits (under a minute). +2. It's impossible to create high-precision increments of time. + +Both of these features are important to ensure the API is self documenting and +conveying the intended purpose. + +For parameterized tests these time limits apply to each individual test case. + +The `TimeLimitTrait.Duration` struct is declared as follows: + +```swift +/// A type that defines a time limit to apply to a test. +/// +/// To add this trait to a test, use one of the following functions: +/// +/// - ``Trait/timeLimit(_:)`` +@available(macOS 13.0, iOS 16.0, tvOS 16.0, watchOS 9.0, *) +public struct TimeLimitTrait: TestTrait, SuiteTrait { + /// A type representing the duration of a time limit applied to a test. + /// + /// This type is intended for use specifically for specifying test timeouts + /// with ``TimeLimitTrait``. It is used instead of Swift's built-in `Duration` + /// type because test timeouts do not support high-precision, arbitrarily + /// short durations. The smallest allowed unit of time is minutes. + public struct Duration: Sendable { + + /// Construct a time limit duration given a number of minutes. + /// + /// - Parameters: + /// - minutes: The number of minutes the resulting duration should + /// represent. + /// + /// - Returns: A duration representing the specified number of minutes. + public static func minutes(_ minutes: some BinaryInteger) -> Self + } + + /// The maximum amount of time a test may run for before timing out. + public var timeLimit: Swift.Duration { get set } +} +``` + +The extension on `Trait` that allows for `.timeLimit(...)` to work is defined +like this: + +```swift +/// Construct a time limit trait that causes a test to time out if it runs for +/// too long. +/// +/// - Parameters: +/// - timeLimit: The maximum amount of time the test may run for. +/// +/// - Returns: An instance of ``TimeLimitTrait``. +/// +/// Test timeouts do not support high-precision, arbitrarily short durations +/// due to variability in testing environments. The time limit must be at +/// least one minute, and can only be expressed in increments of one minute. +/// +/// When this trait is associated with a test, that test must complete within +/// a time limit of, at most, `timeLimit`. If the test runs longer, an issue +/// of kind ``Issue/Kind/timeLimitExceeded(timeLimitComponents:)`` is +/// recorded. This timeout is treated as a test failure. +/// +/// The time limit amount specified by `timeLimit` may be reduced if the +/// testing library is configured to enforce a maximum per-test limit. When +/// such a maximum is set, the effective time limit of the test this trait is +/// applied to will be the lesser of `timeLimit` and that maximum. This is a +/// policy which may be configured on a global basis by the tool responsible +/// for launching the test process. Refer to that tool's documentation for +/// more details. +/// +/// If a test is parameterized, this time limit is applied to each of its +/// test cases individually. If a test has more than one time limit associated +/// with it, the shortest one is used. A test run may also be configured with +/// a maximum time limit per test case. +public static func timeLimit(_ timeLimit: Self.Duration) -> Self +``` + +And finally, the call site of the API looks like this: + +```swift +@Test(.timeLimit(.minutes(60)) +func serve100CustomersInOneHour() async { + for _ in 0 ..< 100 { + let customer = await Customer.next() + await customer.order() + ... + } +} +``` + +The `TimeLimitTrait.Duration` struct has various `unavailable` overloads that +are included for diagnostic purposes only. They are all documented and +annotated like this: + +```swift +/// Construct a time limit duration given a number of . +/// +/// This function is unavailable and is provided for diagnostic purposes only. +@available(*, unavailable, message: "Time limit must be specified in minutes") +``` + +## Source Compatibility + +This impacts clients that have adopted the `.timeLimit` trait and use overloads +of the trait that accept an arbitrary `Swift.Duration` except if they used the +`minutes` overload. + +## Integration with Supporting Tools + +N/A + +## Future Directions + +We could allow more finegrained time limits in the future that scale with the +performance of the test host device. +Or take a more manual approach where we detect the type of environment +(like CI vs local) and provide a way to use different timeouts depending on the +environment. + +## Alternatives Considered + +We have considered using `Swift.Duration` as the currency type for this API but +decided against it to avoid common pitfalls and misuses of this feature such as +providing very small time limits that lead to flaky tests in different +environments. + +## Acknowledgments + +The authors acknowledge valuable contributions and feedback from the Swift +Testing community during the development of this proposal. diff --git a/proposals/testing/0005-ranged-confirmations.md b/proposals/testing/0005-ranged-confirmations.md new file mode 100644 index 0000000000..7a53229ae2 --- /dev/null +++ b/proposals/testing/0005-ranged-confirmations.md @@ -0,0 +1,190 @@ +# Range-based confirmations + +* Proposal: [ST-0005](0005-ranged-confirmations.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Implemented (Swift 6.1)** +* Bug: rdar://138499457 +* Implementation: [swiftlang/swift-testing#598](https://github.com/swiftlang/swift-testing/pull/598), [swiftlang/swift-testing#689](https://github.com/swiftlang/swift-testing/pull689) +* Review: ([pitch](https://forums.swift.org/t/pitch-range-based-confirmations/74589)) ([acceptance](https://forums.swift.org/t/pitch-range-based-confirmations/74589/7)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0005](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0005-ranged-confirmations.md). + +## Introduction + +Swift Testing includes [an interface](https://swiftpackageindex.com/swiftlang/swift-testing/main/documentation/testing/confirmation(_:expectedcount:isolation:sourcelocation:_:)) +for checking that some asynchronous event occurs a given number of times +(typically exactly once or never at all.) This proposal enhances that interface +to allow arbitrary ranges of event counts so that a test can be written against +code that may not always fire said event the exact same number of times. + +## Motivation + +Some tests rely on fixtures or external state that is not perfectly +deterministic. For example, consider a test that checks that clicking the mouse +button will generate a `.mouseClicked` event. Such a test might use the +`confirmation()` interface: + +```swift +await confirmation(expectedCount: 1) { mouseClicked in + var eventLoop = EventLoop() + eventLoop.eventHandler = { event in + if event == .mouseClicked { + mouseClicked() + } + } + await eventLoop.simulate(.mouseClicked) +} +``` + +But what happens if the user _actually_ clicks a mouse button while this test is +running? That might trigger a _second_ `.mouseClicked` event, and then the test +will fail spuriously. + +## Proposed solution + +If the test author could instead indicate to Swift Testing that their test will +generate _one or more_ events, they could avoid spurious failures: + +```swift +await confirmation(expectedCount: 1...) { mouseClicked in + ... +} +``` + +With this proposal, we add an overload of `confirmation()` that takes any range +expression instead of a single integer value (which is still accepted via the +existing overload.) + +## Detailed design + +A new overload of `confirmation()` is added: + +```swift +/// Confirm that some event occurs during the invocation of a function. +/// +/// - Parameters: +/// - comment: An optional comment to apply to any issues generated by this +/// function. +/// - expectedCount: A range of integers indicating the number of times the +/// expected event should occur when `body` is invoked. +/// - isolation: The actor to which `body` is isolated, if any. +/// - sourceLocation: The source location to which any recorded issues should +/// be attributed. +/// - body: The function to invoke. +/// +/// - Returns: Whatever is returned by `body`. +/// +/// - Throws: Whatever is thrown by `body`. +/// +/// Use confirmations to check that an event occurs while a test is running in +/// complex scenarios where `#expect()` and `#require()` are insufficient. For +/// example, a confirmation may be useful when an expected event occurs: +/// +/// - In a context that cannot be awaited by the calling function such as an +/// event handler or delegate callback; +/// - More than once, or never; or +/// - As a callback that is invoked as part of a larger operation. +/// +/// To use a confirmation, pass a closure containing the work to be performed. +/// The testing library will then pass an instance of ``Confirmation`` to the +/// closure. Every time the event in question occurs, the closure should call +/// the confirmation: +/// +/// ```swift +/// let minBuns = 5 +/// let maxBuns = 10 +/// await confirmation( +/// "Baked between \(minBuns) and \(maxBuns) buns", +/// expectedCount: minBuns ... maxBuns +/// ) { bunBaked in +/// foodTruck.eventHandler = { event in +/// if event == .baked(.cinnamonBun) { +/// bunBaked() +/// } +/// } +/// await foodTruck.bakeTray(of: .cinnamonBun) +/// } +/// ``` +/// +/// When the closure returns, the testing library checks if the confirmation's +/// preconditions have been met, and records an issue if they have not. +/// +/// If an exact count is expected, use +/// ``confirmation(_:expectedCount:isolation:sourceLocation:_:)`` instead. +public func confirmation( + _ comment: Comment? = nil, + expectedCount: some RangeExpression & Sequence Sendable, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: (Confirmation) async throws -> sending R +) async rethrows -> R +``` + +### Ranges without lower bounds + +Certain types of range, specifically [`PartialRangeUpTo`](https://developer.apple.com/documentation/swift/partialrangeupto) +and [`PartialRangeThrough`](https://developer.apple.com/documentation/swift/partialrangethrough), +may have surprising behavior when used with this new interface because they +implicitly include `0`. If a test author writes `...10`, do they mean "zero to +ten" or "one to ten"? The programmatic meaning is the former, but some test +authors might mean the latter. If an event does not occur, a test using +`confirmation()` and this `expectedCount` value would pass when the test author +meant for it to fail. + +The unbounded range (`...`) type `UnboundedRange` is effectively useless when +used with this interface and any use of it here is almost certainly a programmer +error. + +`PartialRangeUpTo` and `PartialRangeThrough` conform to `RangeExpression`, but +not to `Sequence`, so they will be rejected at compile time. `UnboundedRange` is +a non-nominal type and will not match either. We will provide unavailable +overloads of `confirmation()` for these types with messages that explain why +they are unavailable, e.g.: + +```swift +@available(*, unavailable, message: "Unbounded range '...' has no effect when used with a confirmation.") +public func confirmation( + _ comment: Comment? = nil, + expectedCount: UnboundedRange, + isolation: isolated (any Actor)? = #isolation, + sourceLocation: SourceLocation = #_sourceLocation, + _ body: (Confirmation) async throws -> R +) async rethrows -> R +``` + +## Source compatibility + +This change is additive. Existing tests are unaffected. + +Code that refers to `confirmation(_:expectedCount:isolation:sourceLocation:_:)` +by symbol name may need to add a contextual type to disambiguate the two +overloads at compile time. + +## Integration with supporting tools + +The type of the associated value `expected` for the `Issue.Kind` case +`confirmationMiscounted(actual:expected:)` will change from `Int` to +`any RangeExpression & Sendable`[^1]. Tools that implement event handlers and +distinguish between `Issue.Kind` cases are advised not to assume the type of +this value is `Int`. + +## Alternatives considered + +- Doing nothing. We have identified real-world use cases for this interface + including in Swift Testing’s own test target. +- Allowing the use of any value as the `expectedCount` argument so long as it + conforms to a protocol `ExpectedCount` (we'd have range types and `Int` + conform by default.) It was unclear what this sort of flexibility would let + us do, and posed challenges for encoding and decoding events and issues when + using the JSON event stream interface. + +## Acknowledgments + +Thanks to the testing team for their help preparing this pitch! + +[^1]: In the future, this type will change to + `any RangeExpression & Sendable`. Compiler support is required + ([96960993](rdar://96960993)). diff --git a/proposals/testing/0006-return-errors-from-expect-throws.md b/proposals/testing/0006-return-errors-from-expect-throws.md new file mode 100644 index 0000000000..eee9037cd3 --- /dev/null +++ b/proposals/testing/0006-return-errors-from-expect-throws.md @@ -0,0 +1,272 @@ +# Return errors from `#expect(throws:)` + +* Proposal: [ST-0006](0006-return-errors-from-expect-throws.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Implemented (Swift 6.1)** +* Bug: rdar://138235250 +* Implementation: [swiftlang/swift-testing#780](https://github.com/swiftlang/swift-testing/pull/780) +* Review: ([pitch](https://forums.swift.org/t/pitch-returning-errors-from-expect-throws/75567)) ([acceptance](https://forums.swift.org/t/pitch-returning-errors-from-expect-throws/75567/5)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0006](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0006-return-errors-from-expect-throws.md). + +## Introduction + +Swift Testing includes overloads of `#expect()` and `#require()` that can be +used to assert that some code throws an error. They are useful when validating +that your code's failure cases are correctly detected and handled. However, for +more complex validation cases, they aren't particularly ergonomic. This proposal +seeks to resolve that issue by having these overloads return thrown errors for +further inspection. + +## Motivation + +We offer three variants of `#expect(throws:)`: + +- One that takes an error type, and matches any error of the same type; +- One that takes an error _instance_ (conforming to `Equatable`) and matches any + error that compares equal to it; and +- One that takes a trailing closure and allows test authors to write arbitrary + validation logic. + +The third overload has proven to be somewhat problematic. First, it yields the +error to its closure as an instance of `any Error`, which typically forces the +developer to cast it before doing any useful comparisons. Second, the test +author must return `true` to indicate the error matched and `false` to indicate +it didn't, which can be both logically confusing and difficult to express +concisely: + +```swift +try #require { + let potato = try Sack.randomPotato() + try potato.turnIntoFrenchFries() +} throws: { error in + guard let error = error as PotatoError else { + return false + } + guard case .potatoNotPeeled = error else { + return false + } + return error.variety != .russet +} +``` + +The first impulse many test authors have here is to use `#expect()` in the +second closure, but it doesn't return the necessary boolean value _and_ it can +result in multiple issues being recorded in a test when there's really only one. + +## Proposed solution + +I propose deprecating [`#expect(_:sourceLocation:performing:throws:)`](https://developer.apple.com/documentation/testing/expect(_:sourcelocation:performing:throws:)) +and [`#require(_:sourceLocation:performing:throws:)`](https://developer.apple.com/documentation/testing/require(_:sourcelocation:performing:throws:)) +and modifying the other overloads so that, on success, they return the errors +that were thrown. + +## Detailed design + +All overloads of `#expect(throws:)` and `#require(throws:)` will be updated to +return an instance of the error type specified by their arguments, with the +problematic overloads returning `any Error` since more precise type information +is not statically available. The problematic overloads will also be deprecated: + +```diff +--- a/Sources/Testing/Expectations/Expectation+Macro.swift ++++ b/Sources/Testing/Expectations/Expectation+Macro.swift ++@discardableResult + @freestanding(expression) public macro expect( + throws errorType: E.Type, + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R +-) ++) -> E? where E: Error + ++@discardableResult + @freestanding(expression) public macro require( + throws errorType: E.Type, + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R +-) where E: Error ++) -> E where E: Error + ++@discardableResult + @freestanding(expression) public macro expect( + throws error: E, + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R +-) where E: Error & Equatable ++) -> E? where E: Error & Equatable + ++@discardableResult + @freestanding(expression) public macro require( + throws error: E, + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R +-) where E: Error & Equatable ++) -> E where E: Error & Equatable + ++@available(swift, deprecated: 100000.0, message: "Examine the result of '#expect(throws:)' instead.") ++@discardableResult + @freestanding(expression) public macro expect( + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R, + throws errorMatcher: (any Error) async throws -> Bool +-) ++) -> (any Error)? + ++@available(swift, deprecated: 100000.0, message: "Examine the result of '#require(throws:)' instead.") ++@discardableResult + @freestanding(expression) public macro require( + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: () async throws -> R, + throws errorMatcher: (any Error) async throws -> Bool +-) ++) -> any Error +``` + +(More detailed information about the deprecations will be provided via DocC.) + +The `#expect(throws:)` overloads return an optional value that is `nil` if the +expectation failed, while the `#require(throws:)` overloads return non-optional +values and throw instances of `ExpectationFailedError` on failure (as before.) + +> [!NOTE] +> Instances of `ExpectationFailedError` thrown by `#require(throws:)` on failure +> are not returned as that would defeat the purpose of using `#require(throws:)` +> instead of `#expect(throws:)`. + +Test authors will be able to use the result of the above functions to verify +that the thrown error is correct: + +```swift +let error = try #require(throws: PotatoError.self) { + let potato = try Sack.randomPotato() + try potato.turnIntoFrenchFries() +} +#expect(error == .potatoNotPeeled) +#expect(error.variety != .russet) +``` + +The new code is more concise than the old code and avoids boilerplate casting +from `any Error`. + +## Source compatibility + +In most cases, this change does not affect source compatibility. Swift does not +allow forming references to macros at runtime, so we don't need to worry about +type mismatches assigning one to some local variable. + +We have identified two scenarios where a new warning will be emitted. + +### Inferred return type from macro invocation + +The return type of the macro may be used by the compiler to infer the return +type of an enclosing closure. If the return value is then discarded, the +compiler may emit a warning: + +```swift +func pokePotato(_ pPotato: UnsafePointer) throws { ... } + +let potato = Potato() +try await Task.sleep(for: .months(3)) +withUnsafePointer(to: potato) { pPotato in + // ^ ^ ^ ⚠️ Result of call to 'withUnsafePointer(to:_:)' is unused + #expect(throws: PotatoError.rotten) { + try pokePotato(pPotato) + } +} +``` + +This warning can be suppressed by assigning the result of the macro invocation +or the result of the function call to `_`: + +```swift +withUnsafePointer(to: potato) { pPotato in + _ = #expect(throws: PotatoError.rotten) { + try pokePotato(pPotato) + } +} +``` + +### Use of `#require(throws:)` in a generic context with `Never.self` + +If `#require(throws:)` (but not `#expect(throws:)`) is used in a generic context +where the type of thrown error is a generic parameter, and the type is resolved +to `Never`, there is no valid value for the invocation to return: + +```swift +func wrapper(throws type: E.Type, _ body: () throws -> Void) throws -> E { + return try #require(throws: type) { + try body() + } +} +let error = try #require(throws: Never.self) { ... } +``` + +We don't think this particular pattern is common (and outside of our own test +target, I'd be surprised if anybody's attempted it yet.) However, we do need to +handle it gracefully. If this pattern is encountered, Swift Testing will record +an "API Misused" issue for the current test and advise the test author to switch +to `#expect(throws:)` or to not pass `Never.self` here. + +## Integration with supporting tools + +N/A + +## Future directions + +- Adopting [typed throws](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0413-typed-throws.md) + to statically require that the error thrown from test code is of the correct + type. + + If we adopted typed throws in the signatures of these macros, it would force + adoption of typed throws in the code under test even when it may not be + appropriate. For example, if we adopted typed throws, the following code would + not compile: + + ```swift + func cook(_ food: consuming some Food) throws { ... } + + let error: PotatoError? = #expect(throws: PotatoError.self) { + var potato = Potato() + potato.fossilize() + try cook(potato) // 🛑 ERROR: Invalid conversion of thrown error type + // 'any Error' to 'PotatoError' + } + ``` + + We believe it may be possible to overload these macros or their expansions so + that the code sample above _does_ compile and behave as intended. We intend to + experiment further with this idea and potentially revisit typed throws support + in a future proposal. + +## Alternatives considered + +- Leaving the existing implementation and signatures in place. We've had + sufficient feedback about the ergonomics of this API that we want to address + the problem. + +- Having the return type of the macros be `any Error` and returning _any_ error + that was thrown even on mismatch. This would make the ergonomics of the + subsequent test code less optimal because the test author would need to cast + the error to the appropriate type before inspecting it. + + There's a philosophical argument to be made here that if a mismatched error is + thrown, then the test has already failed and is in an inconsistent state, so + we should allow the test to fail rather than return what amounts to "bad + output". + + If the test author wants to inspect any arbitrary thrown error, they can + specify `(any Error).self` instead of a concrete error type. + +## Acknowledgments + +Thanks to the team and to [@jakepetroules](https://github.com/jakepetroules) for +starting the discussion that ultimately led to this proposal. diff --git a/proposals/testing/0007-test-scoping-traits.md b/proposals/testing/0007-test-scoping-traits.md new file mode 100644 index 0000000000..619d1f48c6 --- /dev/null +++ b/proposals/testing/0007-test-scoping-traits.md @@ -0,0 +1,515 @@ +# Test Scoping Traits + +* Proposal: [ST-0007](0007-test-scoping-traits.md) +* Authors: [Stuart Montgomery](https://github.com/stmontgomery) +* Status: **Implemented (Swift 6.1)** +* Implementation: [swiftlang/swift-testing#733](https://github.com/swiftlang/swift-testing/pull/733), [swiftlang/swift-testing#86](https://github.com/swiftlang/swift-testing/pull/86) +* Review: ([pitch](https://forums.swift.org/t/pitch-custom-test-execution-traits/75055)) ([review](https://forums.swift.org/t/proposal-test-scoping-traits/76676)) ([acceptance](https://forums.swift.org/t/proposal-test-scoping-traits/76676/3)) + +> [!NOTE] +> This proposal was accepted before Swift Testing began using the Swift +> evolution review process. Its original identifier was +> [SWT-0007](https://github.com/swiftlang/swift-testing/blob/main/Documentation/Proposals/0007-test-scoping-traits.md). + +### Revision history + +* **v1**: Initial pitch. +* **v2**: Dropped 'Custom' prefix from the proposed API names (although kept the + word in certain documentation passages where it clarified behavior). +* **v3**: Changed the `Trait` requirement from a property to a method which + accepts the test and/or test case, and modify its default implementations such + that custom behavior is either performed per-suite or per-test case by default. +* **v4**: Renamed the APIs to use "scope" as the base verb instead of "execute". + +## Introduction + +This introduces API which enables a `Trait`-conforming type to provide a custom +execution scope for test functions and suites, including running code before or +after them. + +## Motivation + +One of the primary motivations for the trait system in Swift Testing, as +[described in the vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#trait-extensibility), +is to provide a way to customize the behavior of tests which have things in +common. If all the tests in a given suite type need the same custom behavior, +`init` and/or `deinit` (if applicable) can be used today. But if only _some_ of +the tests in a suite need custom behavior, or tests across different levels of +the suite hierarchy need it, traits would be a good place to encapsulate common +logic since they can be applied granularly per-test or per-suite. This aspect of +the vision for traits hasn't been realized yet, though: the `Trait` protocol +does not offer a way for a trait to customize the execution of the tests or +suites it's applied to. + +Customizing a test's behavior typically means running code either before or +after it runs, or both. Consolidating common set-up and tear-down logic allows +each test function to be more succinct with less repetitive boilerplate so it +can focus on what makes it unique. + +## Proposed solution + +At a high level, this proposal entails adding API to the `Trait` protocol +allowing a conforming type to opt-in to providing a custom execution scope for a +test. We discuss how that capability should be exposed to trait types below. + +### Supporting scoped access + +There are different approaches one could take to expose hooks for a trait to +customize test behavior. To illustrate one of them, consider the following +example of a `@Test` function with a custom trait whose purpose is to set mock +API credentials for the duration of each test it's applied to: + +```swift +@Test(.mockAPICredentials) +func example() { + // ... +} + +struct MockAPICredentialsTrait: TestTrait { ... } + +extension Trait where Self == MockAPICredentialsTrait { + static var mockAPICredentials: Self { ... } +} +``` + +In this hypothetical example, the current API credentials are stored via a +static property on an `APICredentials` type which is part of the module being +tested: + +```swift +struct APICredentials { + var apiKey: String + + static var shared: Self? +} +``` + +One way that this custom trait could customize the API credentials during each +test is if the `Trait` protocol were to expose a pair of method requirements +which were then called before and after the test, respectively: + +```swift +public protocol Trait: Sendable { + // ... + func setUp() async throws + func tearDown() async throws +} + +extension Trait { + // ... + public func setUp() async throws { /* No-op */ } + public func tearDown() async throws { /* No-op */ } +} +``` + +The custom trait type could adopt these using code such as the following: + +```swift +extension MockAPICredentialsTrait { + func setUp() { + APICredentials.shared = .init(apiKey: "...") + } + + func tearDown() { + APICredentials.shared = nil + } +} +``` + +Many testing systems use this pattern, including XCTest. However, this approach +encourages the use of global mutable state such as the `APICredentials.shared` +variable, and this limits the testing library's ability to parallelize test +execution, which is +[another part of the Swift Testing vision](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#parallelization-and-concurrency). + +The use of nonisolated static variables is generally discouraged now, and in +Swift 6 the above `APICredentials.shared` property produces an error. One way +to resolve that is to change it to a `@TaskLocal` variable, as this would be +concurrency-safe and still allow tests accessing this state to run in parallel: + +```swift +extension APICredentials { + @TaskLocal static var current: Self? +} +``` + +Binding task local values requires using the scoped access +[`TaskLocal.withValue()`](https://developer.apple.com/documentation/swift/tasklocal/withvalue(_:operation:isolation:file:line:)) +API though, and that would not be possible if `Trait` exposed separate methods +like `setUp()` and `tearDown()`. + +For these reasons, I believe it's important to expose this trait capability +using a single, scoped access-style API which accepts a closure. A simplified +version of that idea might look like this: + +```swift +public protocol Trait: Sendable { + // ... + + // Simplified example, not the actual proposal + func executeTest(_ body: @Sendable () async throws -> Void) async throws +} + +extension MockAPICredentialsTrait { + func executeTest(_ body: @Sendable () async throws -> Void) async throws { + let mockCredentials = APICredentials(apiKey: "...") + try await APICredentials.$current.withValue(mockCredentials) { + try await body() + } + } +} +``` + +### Avoiding unnecessarily lengthy backtraces + +A scoped access-style API has some potential downsides. To apply this approach +to a test function, the scoped call of a trait must wrap the invocation of that +test function, and every _other_ trait applied to that same test which offers +custom behavior _also_ must wrap the other traits' calls in a nesting fashion. +To visualize this, imagine a test function with multiple traits: + +```swift +@Test(.traitA, .traitB, .traitC) +func exampleTest() { + // ... +} +``` + +If all three of those traits provide a custom scope for tests, then each of them +needs to wrap the call to the next one, and the last trait needs to wrap the +invocation of the test, illustrated by the following: + +``` +TraitA.executeTest { + TraitB.executeTest { + TraitC.executeTest { + exampleTest() + } + } +} +``` + +Tests may have an arbitrary number of traits applied to them, including those +inherited from containing suite types. A naïve implementation in which _every_ +trait is given the opportunity to customize test behavior by calling its scoped +access API might cause unnecessarily lengthy backtraces that make debugging the +body of tests more difficult. Or worse: if the number of traits is great enough, +it could cause a stack overflow. + +In practice, most traits probably do _not_ need to provide a custom scope for +the tests they're applied to, so to mitigate these downsides it's important that +there be some way to distinguish traits which customize test behavior. That way, +the testing library can limit these scoped access calls to only traits which +need it. + +### Avoiding unnecessary (re-)execution + +Traits can be applied to either test functions or suites, and traits applied to +suites can optionally support inheritance by implementing the `isRecursive` +property of the `SuiteTrait` protocol. When a trait is directly applied to a +test function, if the trait customizes the behavior of tests it's applied to, it +should be given the opportunity to perform its custom behavior once for every +invocation of that test function. In particular, if the test function is +parameterized and runs multiple times, then the trait applied to it should +perform its custom behavior once for every invocation. This should not be +surprising to users, since it's consistent with the behavior of `init` and +`deinit` for an instance `@Test` method. + +It may be useful for certain kinds of traits to perform custom logic once for +_all_ the invocations of a parameterized test. Although this should be possible, +we believe it shouldn't be the default since it could lead to work being +repeated multiple times needlessly, or unintentional state sharing across tests, +unless the trait is implemented carefully to avoid those problems. + +When a trait conforms to `SuiteTrait` and is applied to a suite, the question of +when its custom scope (if any) should be applied is less obvious. Some suite +traits support inheritance and are recursively applied to all the test functions +they contain (including transitively, via sub-suites). Other suite traits don't +support inheritance, and only affect the specific suite they're applied to. +(It's also worth noting that a sub-suite _can_ have the same non-recursive suite +trait one of its ancestors has, as long as it's applied explicitly.) + +As a general rule of thumb, we believe most traits will either want to perform +custom logic once for _all_ children or once for _each_ child, not both. +Therefore, when it comes to suite traits, the default behavior should depend on +whether it supports inheritance: a recursive suite trait should by default +perform custom logic before each test, and a non-recursive one per-suite. But +the APIs should be flexible enough to support both, for advanced traits which +need it. + +## Detailed design + +I propose the following new APIs: + +- A new protocol `TestScoping` with a single required `provideScope(...)` method. + This will be called to provide scope for a test, and allows the conforming + type to perform custom logic before or after. +- A new method `scopeProvider(for:testCase:)` on the `Trait` protocol whose + result type is an `Optional` value of a type conforming to `TestScoping`. A + `nil` value returned by this method will skip calling the `provideScope(...)` + method. +- A default implementation of `Trait.scopeProvider(...)` which returns `nil`. +- A conditional implementation of `Trait.scopeProvider(...)` which returns `self` + in the common case where the trait type conforms to `TestScoping` itself. + +Since the `scopeProvider(...)` method's return type is optional and returns `nil` +by default, the testing library cannot invoke the `provideScope(...)` method +unless a trait customizes test behavior. This avoids the "unnecessarily lengthy +backtraces" problem above. + +Below are the proposed interfaces: + +```swift +/// A protocol that allows providing a custom execution scope for a test +/// function (and each of its cases) or a test suite by performing custom code +/// before or after it runs. +/// +/// Types conforming to this protocol may be used in conjunction with a +/// ``Trait``-conforming type by implementing the +/// ``Trait/scopeProvider(for:testCase:)-cjmg`` method, allowing custom traits +/// to provide custom scope for tests. Consolidating common set-up and tear-down +/// logic for tests which have similar needs allows each test function to be +/// more succinct with less repetitive boilerplate so it can focus on what makes +/// it unique. +public protocol TestScoping: Sendable { + /// Provide custom execution scope for a function call which is related to the + /// specified test and/or test case. + /// + /// - Parameters: + /// - test: The test under which `function` is being performed. + /// - testCase: The test case, if any, under which `function` is being + /// performed. When invoked on a suite, the value of this argument is + /// `nil`. + /// - function: The function to perform. If `test` represents a test suite, + /// this function encapsulates running all the tests in that suite. If + /// `test` represents a test function, this function is the body of that + /// test function (including all cases if it is parameterized.) + /// + /// - Throws: Whatever is thrown by `function`, or an error preventing this + /// type from providing a custom scope correctly. An error thrown from this + /// method is recorded as an issue associated with `test`. If an error is + /// thrown before `function` is called, the corresponding test will not run. + /// + /// When the testing library is preparing to run a test, it starts by finding + /// all traits applied to that test, including those inherited from containing + /// suites. It begins with inherited suite traits, sorting them + /// outermost-to-innermost, and if the test is a function, it then adds all + /// traits applied directly to that functions in the order they were applied + /// (left-to-right). It then asks each trait for its scope provider (if any) + /// by calling ``Trait/scopeProvider(for:testCase:)-cjmg``. Finally, it calls + /// this method on all non-`nil` scope providers, giving each an opportunity + /// to perform arbitrary work before or after invoking `function`. + /// + /// This method should either invoke `function` once before returning or throw + /// an error if it is unable to provide a custom scope. + /// + /// Issues recorded by this method are associated with `test`. + func provideScope(for test: Test, testCase: Test.Case?, performing function: @Sendable () async throws -> Void) async throws +} + +public protocol Trait: Sendable { + // ... + + /// The type of the test scope provider for this trait. + /// + /// The default type is `Never`, which cannot be instantiated. The + /// ``scopeProvider(for:testCase:)-cjmg`` method for any trait with this + /// default type must return `nil`, meaning that trait will not provide a + /// custom scope for the tests it's applied to. + associatedtype TestScopeProvider: TestScoping = Never + + /// Get this trait's scope provider for the specified test and/or test case, + /// if any. + /// + /// - Parameters: + /// - test: The test for which a scope provider is being requested. + /// - testCase: The test case for which a scope provider is being requested, + /// if any. When `test` represents a suite, the value of this argument is + /// `nil`. + /// + /// - Returns: A value conforming to ``Trait/TestScopeProvider`` which may be + /// used to provide custom scoping for `test` and/or `testCase`, or `nil` if + /// they should not have any custom scope. + /// + /// If this trait's type conforms to ``TestScoping``, the default value + /// returned by this method depends on `test` and/or `testCase`: + /// + /// - If `test` represents a suite, this trait must conform to ``SuiteTrait``. + /// If the value of this suite trait's ``SuiteTrait/isRecursive`` property + /// is `true`, then this method returns `nil`; otherwise, it returns `self`. + /// This means that by default, a suite trait will _either_ provide its + /// custom scope once for the entire suite, or once per-test function it + /// contains. + /// - Otherwise `test` represents a test function. If `testCase` is `nil`, + /// this method returns `nil`; otherwise, it returns `self`. This means that + /// by default, a trait which is applied to or inherited by a test function + /// will provide its custom scope once for each of that function's cases. + /// + /// A trait may explicitly implement this method to further customize the + /// default behaviors above. For example, if a trait should provide custom + /// test scope both once per-suite and once per-test function in that suite, + /// it may implement the method and return a non-`nil` scope provider under + /// those conditions. + /// + /// A trait may also implement this method and return `nil` if it determines + /// that it does not need to provide a custom scope for a particular test at + /// runtime, even if the test has the trait applied. This can improve + /// performance and make diagnostics clearer by avoiding an unnecessary call + /// to ``TestScoping/provideScope(for:testCase:performing:)``. + /// + /// If this trait's type does not conform to ``TestScoping`` and its + /// associated ``Trait/TestScopeProvider`` type is the default `Never`, then + /// this method returns `nil` by default. This means that instances of this + /// trait will not provide a custom scope for tests to which they're applied. + func scopeProvider(for test: Test, testCase: Test.Case?) -> TestScopeProvider? +} + +extension Trait where Self: TestScoping { + // Returns `nil` if `testCase` is `nil`, else `self`. + public func scopeProvider(for test: Test, testCase: Test.Case?) -> Self? +} + +extension SuiteTrait where Self: TestScoping { + // If `test` is a suite, returns `nil` if `isRecursive` is `true`, else `self`. + // Otherwise, `test` is a function and this returns `nil` if `testCase` is + // `nil`, else `self`. + public func scopeProvider(for test: Test, testCase: Test.Case?) -> Self? +} + +extension Trait where TestScopeProvider == Never { + // Returns `nil`. + public func scopeProvider(for test: Test, testCase: Test.Case?) -> Never? +} + +extension Never: TestScoping {} +``` + +Here is a complete example of the usage scenario described earlier, showcasing +the proposed APIs: + +```swift +@Test(.mockAPICredentials) +func example() { + // ...validate API usage, referencing `APICredentials.current`... +} + +struct MockAPICredentialsTrait: TestTrait, TestScoping { + func provideScope(for test: Test, testCase: Test.Case?, performing function: @Sendable () async throws -> Void) async throws { + let mockCredentials = APICredentials(apiKey: "...") + try await APICredentials.$current.withValue(mockCredentials) { + try await function() + } + } +} + +extension Trait where Self == MockAPICredentialsTrait { + static var mockAPICredentials: Self { + Self() + } +} +``` + +## Source compatibility + +The proposed APIs are purely additive. + +This proposal will replace the existing `CustomExecutionTrait` SPI, and after +further refactoring we anticipate it will obsolete the need for the +`SPIAwareTrait` SPI as well. + +## Integration with supporting tools + +Although some built-in traits are relevant to supporting tools (such as +SourceKit-LSP statically discovering `.tags` traits), custom test behaviors are +only relevant within the test executable process while tests are running. We +don't anticipate any particular need for this feature to integrate with +supporting tools. + +## Future directions + +### Access to suite type instances + +Some test authors have expressed interest in allowing custom traits to access +the instance of a suite type for `@Test` instance methods, so the trait could +inspect or mutate the instance. Currently, only instance-level members of a +suite type (including `init`, `deinit`, and the test function itself) can access +`self`, so this would grant traits applied to an instance test method access to +the instance as well. This is certainly interesting, but poses several technical +challenges that puts it out of scope of this proposal. + +### Convenience trait for setting task locals + +Some reviewers of this proposal pointed out that the hypothetical usage example +shown earlier involving setting a task local value while a test is executing +will likely become a common use of these APIs. To streamline that pattern, it +would be very natural to add a built-in trait type which facilitates this. I +have prototyped this idea and plan to add it once this new trait functionality +lands. + +## Alternatives considered + +### Separate set up & tear down methods on `Trait` + +This idea was discussed in [Supporting scoped access](#supporting-scoped-access) +above, and as mentioned there, the primary problem with this approach is that it +cannot be used with scoped access-style APIs, including (importantly) +`TaskLocal.withValue()`. For that reason, it prevents using that common Swift +concurrency technique and reduces the potential for test parallelization. + +### Add `provideScope(...)` directly to the `Trait` protocol + +The proposed `provideScope(...)` method could be added as a requirement of the +`Trait` protocol instead of being part of a separate `TestScoping` protocol, and +it could have a default implementation which directly invokes the passed-in +closure. But this approach would suffer from the lengthy backtrace problem +described above. + +### Extend the `Trait` protocol + +The original, experimental implementation of this feature included a protocol +named`CustomExecutionTrait` which extended `Trait` and had roughly the same +method requirement as the `TestScoping` protocol proposed above. This design +worked, provided scoped access, and avoided the lengthy backtrace problem. + +After evaluating the design and usage of this SPI though, it seemed unfortunate +to structure it as a sub-protocol of `Trait` because it means that the full +capabilities of the trait system are spread across multiple protocols. In the +proposed design, the ability to return a test scoping provider is exposed via +the main `Trait` protocol, and it relies on an associated type to conditionally +opt-in to custom test behavior. In other words, the proposed design expresses +custom test behavior as just a _capability_ that a trait may have, rather than a +distinct sub-type of trait. + +Also, the implementation of this approach within the testing library was not +ideal as it required a conditional `trait as? CustomExecutionTrait` downcast at +runtime, in contrast to the simpler and more performant Optional property of the +proposed API. + +### API names + +We first considered "execute" as the base verb for the proposed new concept, but +felt this wasn't appropriate since these trait types are not "the executor" of +tests, they merely customize behavior and provide scope(s) for tests to run +within. Also, the term "executor" has prior art in Swift Concurrency, and +although that word is used in other contexts too, it may be helpful to avoid +potential confusion with concurrency executors. + +We also considered "run" as the base verb for the proposed new concept instead +of "execute", which would imply the names `TestRunning`, `TestRunner`, +`runner(for:testCase)`, and `run(_:for:testCase:)`. The word "run" is used in +many other contexts related to testing though, such as the `Runner` SPI type and +more casually to refer to a run which occurred of a test, in the past tense, so +overloading this term again may cause confusion. + +## Acknowledgments + +Thanks to [Dennis Weissmann](https://github.com/dennisweissmann) for originally +implementing this as SPI, and for helping promote its usefulness. + +Thanks to [Jonathan Grynspan](https://github.com/grynspan) for exploring ideas +to refine the API, and considering alternatives to avoid unnecessarily long +backtraces. + +Thanks to [Brandon Williams](https://github.com/mbrandonw) for feedback on the +Forum pitch thread which ultimately led to the refinements described in the +"Avoiding unnecessary (re-)execution" section. diff --git a/proposals/testing/0008-exit-tests.md b/proposals/testing/0008-exit-tests.md new file mode 100644 index 0000000000..2a69716107 --- /dev/null +++ b/proposals/testing/0008-exit-tests.md @@ -0,0 +1,910 @@ +# Exit tests + +* Proposal: [ST-0008](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0008-exit-tests.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Maarten Engels](https://github.com/maartene) +* Status: **Implemented (Swift 6.2)** +* Bug: [apple/swift-testing#157](https://github.com/apple/swift-testing/issues/157) +* Implementation: [apple/swift-testing#324](https://github.com/swiftlang/swift-testing/pull/324) +* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/fdfc7867df4e35e29b2a24edee34ea4412ec15b0/proposals/testing/0008-exit-tests.md) +* Review: ([pitch](https://forums.swift.org/t/pitch-exit-tests/78071)) ([review](https://forums.swift.org/t/st-0008-exit-tests/78692)) ([second review](https://forums.swift.org/t/second-review-st-0008-exit-tests/79198)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-st-0008-exit-tests/79553)) + +## Introduction + +One of the first enhancement requests we received for Swift Testing was the +ability to test for precondition failures and other critical failures that +terminate the current process when they occur. This feature is also frequently +requested for XCTest. With Swift Testing, we have the opportunity to build such +a feature in an ergonomic way. + +> [!NOTE] +> This feature has various names in the relevant literature, e.g. "exit tests", +> "death tests", "death assertions", "termination tests", etc. We consistently +> use the term "exit tests" to refer to them. + +## Motivation + +Imagine a function, implemented in a package, that includes a precondition: + +```swift +func eat(_ taco: consuming Taco) { + precondition(taco.isDelicious, "Tasty tacos only!") + ... +} +``` + +Today, a test author can write unit tests for this function, but there is no way +to make sure that the function rejects a taco whose `isDelicious` property is +`false` because a test that passes such a taco as input will crash (correctly!) +when it calls `precondition()`. + +An exit test allows testing this sort of functionality. The mechanism by which +an exit test is implemented varies between testing libraries and languages, but +a common implementation involves spawning a new process, performing the work +there, and checking that the spawned process ultimately terminates with a +particular (possibly platform-specific) exit status. + +Adding exit tests to Swift Testing would allow an entirely new class of tests +and would improve code coverage for existing test targets that adopt them. + +## Proposed solution + +This proposal introduces new overloads of the `#expect()` and `#require()` +macros that take, as an argument, a closure to be executed in a child process. +When called, these macros spawn a new process using the relevant +platform-specific interface (`posix_spawn()`, `CreateProcessW()`, etc.), call +the closure from within that process, and suspend the caller until that process +terminates. The exit status of the process is then compared against a known +value passed to the macro, allowing the test to pass or fail as appropriate. + +The function from earlier can then be tested using either of the new +overloads: + +```swift +await #expect(processExitsWith: .failure) { + var taco = Taco() + taco.isDelicious = false + eat(taco) // should trigger a precondition failure and process termination +} +``` + +## Detailed design + +### New expectations + +We will introduce the following new overloads of `#expect()` and `#require()` to +the testing library: + +```swift +/// Check that an expression causes the process to terminate in a given fashion. +/// +/// - Parameters: +/// - expectedExitCondition: The expected exit condition. +/// - observedValues: An array of key paths representing results from within +/// the exit test that should be observed and returned by this macro. The +/// ``ExitTest/Result/exitStatus`` property is always returned. +/// - comment: A comment describing the expectation. +/// - sourceLocation: The source location to which recorded expectations and +/// issues should be attributed. +/// - expression: The expression to be evaluated. +/// +/// - Returns: If the exit test passed, an instance of ``ExitTest/Result`` +/// describing the state of the exit test when it exited. If the exit test +/// fails, the result is `nil`. +/// +/// Use this overload of `#expect()` when an expression will cause the current +/// process to terminate and the nature of that termination will determine if +/// the test passes or fails. For example, to test that calling `fatalError()` +/// causes a process to terminate: +/// +/// await #expect(processExitsWith: .failure) { +/// fatalError() +/// } +/// +/// - Note: A call to this expectation macro is called an "exit test." +/// +/// ## How exit tests are run +/// +/// When an exit test is performed at runtime, the testing library starts a new +/// process with the same executable as the current process. The current task is +/// then suspended (as with `await`) and waits for the child process to +/// terminate. `expression` is not called in the parent process. +/// +/// Meanwhile, in the child process, `expression` is called directly. To ensure +/// a clean environment for execution, it is not called within the context of +/// the original test. If `expression` does not terminate the child process, the +/// process is terminated automatically as if the main function of the child +/// process were allowed to return naturally. If an error is thrown from +/// `expression`, it is handed as if the error were thrown from `main()` and the +/// process is terminated. +/// +/// Once the child process terminates, the parent process resumes and compares +/// its exit status against `expectedExitCondition`. If they match, the exit +/// test has passed; otherwise, it has failed and an issue is recorded. +/// +/// ## Child process output +/// +/// By default, the child process is configured without a standard output or +/// standard error stream. If your test needs to review the content of either of +/// these streams, you can pass its key path in the `observedValues` argument: +/// +/// let result = await #expect( +/// processExitsWith: .failure, +/// observing: [\.standardOutputContent] +/// ) { +/// print("Goodbye, world!") +/// fatalError() +/// } +/// if let result { +/// #expect(result.standardOutputContent.contains(UInt8(ascii: "G"))) +/// } +/// +/// - Note: The content of the standard output and standard error streams may +/// contain any arbitrary sequence of bytes, including sequences that are not +/// valid UTF-8 and cannot be decoded by [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). +/// These streams are globally accessible within the child process, and any +/// code running in an exit test may write to it including the operating +/// system and any third-party dependencies you have declared in your package. +/// +/// The actual exit condition of the child process is always reported by the +/// testing library even if you do not specify it in `observedValues`. +/// +/// ## Runtime constraints +/// +/// Exit tests cannot capture any state originating in the parent process or +/// from the enclosing lexical context. For example, the following exit test +/// will fail to compile because it captures an argument to the enclosing +/// parameterized test: +/// +/// @Test(arguments: 100 ..< 200) +/// func sellIceCreamCones(count: Int) async { +/// await #expect(processExitsWith: .failure) { +/// precondition( +/// count < 10, // ERROR: A C function pointer cannot be formed from a +/// // closure that captures context +/// "Too many ice cream cones" +/// ) +/// } +/// } +/// +/// An exit test cannot run within another exit test. +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +@discardableResult +@freestanding(expression) public macro expect( + processExitsWith expectedExitCondition: ExitTest.Condition, + observing observedValues: [any PartialKeyPath & Sendable] = [], + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @escaping @Sendable () async throws -> Void +) -> ExitTest.Result? = #externalMacro(module: "TestingMacros", type: "ExitTestExpectMacro") + +/// Check that an expression causes the process to terminate in a given fashion +/// and throw an error if it did not. +/// +/// [...] +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +@discardableResult +@freestanding(expression) public macro require( + processExitsWith expectedExitCondition: ExitTest.Condition, + observing observedValues: [any PartialKeyPath & Sendable] = [], + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @escaping @Sendable () async throws -> Void +) -> ExitTest.Result = #externalMacro(module: "TestingMacros", type: "ExitTestRequireMacro") +``` + +> [!NOTE] +> These interfaces are currently implemented and available on **macOS**, +> **Linux**, **FreeBSD**, **OpenBSD**, and **Windows**. If a platform does not +> support exit tests (generally because it does not support spawning or awaiting +> child processes), then we define `SWT_NO_EXIT_TESTS` when we build it. +> +> `SWT_NO_EXIT_TESTS` is not defined during test target builds and is presented +> here for illustrative purposes only. + +### Representing an exit test in Swift + +A new type, `ExitTest`, represents an exit test: + +```swift +/// A type describing an exit test. +/// +/// Instances of this type describe exit tests you create using the +/// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or +/// ``require(processExitsWith:observing:_:sourceLocation:performing:)`` macro. +/// You don't usually need to interact directly with an instance of this type. +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +public struct ExitTest: Sendable, ~Copyable { + /// The exit test that is running in the current process, if any. + /// + /// If the current process was created to run an exit test, the value of this + /// property describes that exit test. If this process is the parent process + /// of an exit test, or if no exit test is currently running, the value of + /// this property is `nil`. + /// + /// The value of this property is constant across all tasks in the current + /// process. + public static var current: ExitTest? { get } +} +``` + +### Exit conditions + +These macros take an argument of the new type `ExitTest.Condition`. This type +describes how the child process is expected to have exited: + +- With a specific exit code (as passed to the C standard function `exit()` or a + platform-specific equivalent); +- With a specific signal (on platforms that support signal handling[^winsig]); +- With any successful status; or +- With any failure status. + +[^winsig]: Windows nominally supports signal handling as it is part of the C + standard, but not to the degree that signals are supported by POSIX-like or + UNIX-derived operating systems. Swift Testing makes a "best effort" to emulate + signal-handling support on Windows. See [this](https://forums.swift.org/t/swift-on-windows-question-about-signals-and-exceptions/76640/2) + Swift forum message for more information. + +The type is declared as: + +```swift +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +extension ExitTest { + /// The possible conditions under which an exit test will complete. + /// + /// Values of this type are used to describe the conditions under which an + /// exit test is expected to pass or fail by passing them to + /// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or + /// ``require(processExitsWith:observing:_:sourceLocation:performing:)``. + /// + /// ## Topics + /// + /// ### Successful exit conditions + /// + /// - ``success`` + /// + /// ### Failing exit conditions + /// + /// - ``failure`` + /// - ``exitCode(_:)`` + /// - ``signal(_:)`` + public struct Condition: Sendable, CustomStringConvertible { + /// A condition that matches when a process terminates successfully with exit + /// code `EXIT_SUCCESS`. + /// + /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), + /// `EXIT_SUCCESS` and `EXIT_FAILURE` as well as `0` (as a synonym for + /// `EXIT_SUCCESS`.) + public static var success: Self { get } + + /// A condition that matches when a process terminates abnormally with any + /// exit code other than `EXIT_SUCCESS` or with any signal. + public static var failure: Self { get } + + public init(_ exitStatus: ExitStatus) + + /// Creates a condition that matches when a process terminates with a given + /// exit code. + /// + /// - Parameters: + /// - exitCode: The exit code yielded by the process. + /// + /// The C programming language defines two [standard exit codes](https://en.cppreference.com/w/c/program/EXIT_status), + /// `EXIT_SUCCESS` and `EXIT_FAILURE`. Platforms may additionally define their + /// own non-standard exit codes: + /// + /// | Platform | Header | + /// |-|-| + /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/_Exit.3.html), [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/sysexits.3.html) | + /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Exit-Status.html), `` | + /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?exit(3)), [``](https://man.freebsd.org/cgi/man.cgi?sysexits(3)) | + /// | OpenBSD | [``](https://man.openbsd.org/exit.3), [``](https://man.openbsd.org/sysexits.3) | + /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/exit-success-exit-failure) | + /// + /// On macOS, FreeBSD, OpenBSD, and Windows, the full exit code reported by + /// the process is yielded to the parent process. Linux and other POSIX-like + /// systems may only reliably report the low unsigned 8 bits (0–255) of + /// the exit code. + public static func exitCode(_ exitCode: CInt) -> Self + + /// Creates a condition that matches when a process terminates with a given + /// signal. + /// + /// - Parameters: + /// - signal: The signal that terminated the process. + /// + /// The C programming language defines a number of [standard signals](https://en.cppreference.com/w/c/program/SIG_types). + /// Platforms may additionally define their own non-standard signal codes: + /// + /// | Platform | Header | + /// |-|-| + /// | macOS | [``](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/signal.3.html) | + /// | Linux | [``](https://sourceware.org/glibc/manual/latest/html_node/Standard-Signals.html) | + /// | FreeBSD | [``](https://man.freebsd.org/cgi/man.cgi?signal(3)) | + /// | OpenBSD | [``](https://man.openbsd.org/signal.3) | + /// | Windows | [``](https://learn.microsoft.com/en-us/cpp/c-runtime-library/signal-constants) | + public static func signal(_ signal: CInt) -> Self + } +} +``` + +### Exit status + +The set of possible status codes reported by the child process are represented +by the `ExitStatus` enumeration: + +```swift +/// An enumeration describing possible status a process will yield on exit. +/// +/// You can convert an instance of this type to an instance of +/// ``ExitTest/Condition`` using ``ExitTest/Condition/init(_:)``. That value +/// can then be used to describe the condition under which an exit test is +/// expected to pass or fail by passing it to +/// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or +/// ``require(processExitsWith:observing:_:sourceLocation:performing:)``. +#if SWT_NO_PROCESS_SPAWNING +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +public enum ExitStatus: Sendable, Equatable, CustomStringConvertible { + /// The process terminated with the given exit code. + /// + /// [...] + case exitCode(_ exitCode: CInt) + + /// The process terminated with the given signal. + /// + /// [...] + case signal(_ signal: CInt) +} +``` + +### Exit test results + +These macros return an instance of the new type `ExitTest.Result`. This type +describes the results of the process including its reported exit condition and +the contents of its standard output and standard error streams, if requested. + +```swift +#if SWT_NO_EXIT_TESTS +@available(*, unavailable, message: "Exit tests are not available on this platform.") +#endif +extension ExitTest { + /// A type representing the result of an exit test after it has exited and + /// returned control to the calling test function. + /// + /// Both ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` + /// and ``require(processExitsWith:observing:_:sourceLocation:performing:)`` + /// return instances of this type. + public struct Result: Sendable { + /// The status of the process hosting the exit test at the time it exits. + /// + /// When the exit test passes, the value of this property is equal to the + /// exit status reported by the process that hosted the exit test. + public var exitStatus: ExitStatus { get set } + + /// All bytes written to the standard output stream of the exit test before + /// it exited. + /// + /// The value of this property may contain any arbitrary sequence of bytes, + /// including sequences that are not valid UTF-8 and cannot be decoded by + /// [`String.init(cString:)`](https://developer.apple.com/documentation/swift/string/init(cstring:)-6kr8s). + /// Consider using [`String.init(validatingCString:)`](https://developer.apple.com/documentation/swift/string/init(validatingcstring:)-992vo) + /// instead. + /// + /// When checking the value of this property, keep in mind that the standard + /// output stream is globally accessible, and any code running in an exit + /// test may write to it including including the operating system and any + /// third-party dependencies you have declared in your package. Rather than + /// comparing the value of this property with [`==`](https://developer.apple.com/documentation/swift/array/==(_:_:)), + /// use [`contains(_:)`](https://developer.apple.com/documentation/swift/collection/contains(_:)) + /// to check if expected output is present. + /// + /// To enable gathering output from the standard output stream during an + /// exit test, pass `\.standardOutputContent` in the `observedValues` + /// argument of ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` + /// or ``require(processExitsWith:observing:_:sourceLocation:performing:)``. + /// + /// If you did not request standard output content when running an exit test, + /// the value of this property is the empty array. + public var standardOutputContent: [UInt8] { get set } + + /// All bytes written to the standard error stream of the exit test before + /// it exited. + /// + /// [...] + public var standardErrorContent: [UInt8] { get set } + } +} +``` + +### Usage + +These macros can be used within a test function: + +```swift +@Test func `We only eat delicious tacos`() async { + await #expect(processExitsWith: .failure) { + var taco = Taco() + taco.isDelicious = false + eat(taco) + } +} +``` + +Given the definition of `eat(_:)` above, this test can be expected to hit a +precondition failure and crash the process; because `.failure` was the specified +exit condition, this is treated as a successful test. + +It is often interesting to examine what is written to the standard output and +standard error streams by code running in an exit test. Callers can request that +either or both stream be captured and included in the result of the call to +`#expect(processExitsWith:)` or `#require(processExitsWith:)`. Capturing these +streams can be a memory-intensive operation, so the caller must explicitly opt +in: + +```swift +@Test func `We only eat delicious tacos`() async throws { + let result = try await #require( + processExitsWith: .failure, + observing: [\.standardErrorContent]) + ) { ... } + let stdout = result.standardOutputContent + #expect(stdout.contains("ERROR: This taco tastes terrible!".utf8)) +} +``` + +There are some constraints on valid exit tests: + +1. Because exit tests are run in child processes, they cannot capture any state + from the calling context. See the **Future directions** for further + discussion. +1. Exit tests cannot recursively invoke other exit tests; this is a constraint + that could potentially be lifted in the future, but it would be technically + complex to do so. + +If a Swift Testing issue such as an expectation failure occurs while running an +exit test, it is reported to the parent process and to the user as if it +happened locally. If an error is thrown from an exit test and not caught, it +behaves the same way a Swift program would if an error were thrown from its +`main()` function (that is, the program terminates abnormally.) + +## Source compatibility + +This is a new interface that is unlikely to collide with any existing +client-provided interfaces. The typical Swift disambiguation tools can be used +if needed. + +## Integration with supporting tools + +SPI is provided to allow testing environments other than Swift Package Manager +to detect and run exit tests: + +```swift +@_spi(ForToolsIntegrationOnly) +extension ExitTest { + /// A type whose instances uniquely identify instances of ``ExitTest``. + public struct ID: Sendable, Equatable, Codable { /* ... */ } + + /// A value that uniquely identifies this instance. + public var id: ID { get set } + + /// Key paths representing results from within this exit test that should be + /// observed and returned to the caller. + /// + /// The testing library sets this property to match what was passed by the + /// developer to the `#expect(processExitsWith:)` or `#require(processExitsWith:)` + /// macro. If you are implementing an exit test handler, you can check the + /// value of this property to determine what information you need to preserve + /// from your child process. + /// + /// The value of this property always includes ``ExitTest/Result/exitStatus`` + /// even if the test author does not specify it. + /// + /// Within a child process running an exit test, the value of this property is + /// otherwise unspecified. + public var observedValues: [any PartialKeyPath & Sendable] { get set } + + /// Call the exit test in the current process. + /// + /// This function invokes the closure originally passed to + /// `#expect(processExitsWith:)` _in the current process_. That closure is + /// expected to terminate the process; if it does not, the testing library + /// will terminate the process as if its `main()` function returned naturally. + public consuming func callAsFunction() async -> Never + + /// Find the exit test function at the given source location. + /// + /// - Parameters: + /// - id: The unique identifier of the exit test to find. + /// + /// - Returns: The specified exit test function, or `nil` if no such exit test + /// could be found. + public static func find(identifiedBy id: ExitTest.ID) -> Self? + + /// A handler that is invoked when an exit test starts. + /// + /// - Parameters: + /// - exitTest: The exit test that is starting. + /// + /// - Returns: The result of the exit test including the condition under which + /// it exited. + /// + /// - Throws: Any error that prevents the normal invocation or execution of + /// the exit test. + /// + /// This handler is invoked when an exit test (i.e. a call to either + /// ``expect(processExitsWith:observing:_:sourceLocation:performing:)`` or + /// ``require(processExitsWith:observing:_:sourceLocation:performing:)``) is + /// started. The handler is responsible for initializing a new child + /// environment (typically a child process) and running the exit test + /// identified by `sourceLocation` there. + /// + /// In the child environment, you can find the exit test again by calling + /// ``ExitTest/find(at:)`` and can run it by calling + /// ``ExitTest/callAsFunction()``. + /// + /// The parent environment should suspend until the results of the exit test + /// are available or the child environment is otherwise terminated. The parent + /// environment is then responsible for interpreting those results and + /// recording any issues that occur. + public typealias Handler = @Sendable (_ exitTest: borrowing ExitTest) async throws -> ExitTest.Result +} + +@_spi(ForToolsIntegrationOnly) +extension Configuration { + /// A handler that is invoked when an exit test starts. + /// + /// For an explanation of how this property is used, see ``ExitTest/Handler``. + /// + /// When using the `swift test` command from Swift Package Manager, this + /// property is pre-configured. Otherwise, the default value of this property + /// records an issue indicating that it has not been configured. + public var exitTestHandler: ExitTest.Handler { get set } +} +``` + +Any tools that use `swift build --build-tests`, `swift test`, or equivalent to +compile executables for testing will inherit the functionality provided for +`swift test` and do not need to implement their own exit test handlers. Tools +that directly compile test targets or otherwise do not leverage Swift Package +Manager will need to provide an implementation. + +## Future directions + +### Support for iOS, WASI, etc. + +The need for exit tests on other platforms is just as strong as it is on the +supported platforms (macOS, Linux, FreeBSD/OpenBSD, and Windows). These +platforms do not support spawning new processes, so a different mechanism for +running exit tests would be needed. + +Android _does_ have `posix_spawn()` and related API and may be able to use the +same implementation as Linux. Android support is an ongoing area of research for +Swift Testing's core team. + +> [!NOTE] +> In the event we can add support for exit tests on a new platform _without_ any +> changes to the feature's public interface, the Testing Workgroup has agreed +> that an additional Swift Evolution proposal will not be necessary. + +### Recursive exit tests + +The technical constraints preventing recursive exit test invocation can be +resolved if there is a need to do so. However, we don't anticipate that this +constraint will be a serious issue for developers. + +### Support for passing state + +Arbitrary state is necessarily not preserved between the parent and child +processes, but there is little to prevent us from adding a variadic `arguments:` +argument and passing values whose types conform to `Codable`. + +The blocker right now is that there is no type information during macro +expansion, meaning that the testing library can emit the glue code to _encode_ +arguments, but does not know what types to use when _decoding_ those arguments. +If generic types were made available during macro expansion via the macro +expansion context, then it would be possible to synthesize the correct logic. + +Alternatively, if the language gained something akin to C++'s `decltype()`, we +could leverage closures' capture list syntax. Subjectively, capture lists ought +to be somewhat intuitive for developers in this context: + +```swift +let (lettuce, cheese) = taco.addToppings() +await #expect(processExitsWith: .failure) { [taco, plant = lettuce, cheese] in + try taco.removeToppings(plant, cheese) +} +``` + +### More nuanced support for throwing errors from exit test bodies + +Currently, if an error is thrown from an exit test without being caught, the +test behaves the same way a program does when an error is thrown from an +explicit or implicit `main() throws` function: the process terminates abnormally +and control returns to the test function that is awaiting the exit test: + +```swift +await #expect(processExitsWith: .failure) { + throw TacoError.noTacosFound +} +``` + +If the test function is expecting `.failure`, this means the test passes. +Although this behavior is consistent with modelling an exit test as an +independent program (i.e. the exit test acts like its own `main()` function), it +may be surprising to test authors who aren't thinking about error handling. In +the future, we may want to offer a compile-time diagnostic if an error is thrown +from an exit test body without being caught, or offer a distinct exit condition +(i.e. `.errorNotCaught(_ error: Error & Codable)`) for these uncaught errors. +For error types that conform to `Codable`, we could offer rethrowing behavior, +but this is not possible for error types that cannot be sent across process +boundaries. + +### Exit-testing customized processes + +The current model of exit tests is that they run in approximately the same +environment as the test process by spawning a copy of the executable under test. +There is a very real use case for allowing testing other processes and +inspecting their output. In the future, we could provide API to spawn a process +with particular arguments and environment variables, then inspect its exit +condition and standard output/error streams: + +```swift +let result = try await #require( + executableAt: "/usr/bin/swift", + passing: ["build", "--package-path", ...], + environment: [:], + exitsWith: .success +) +#expect(result.standardOutputContent.contains("Build went well!").utf8) +``` + +We could also investigate explicitly integrating with [`Foundation.Process`](https://developer.apple.com/documentation/foundation/process) +or the proposed [`Foundation.Subprocess`](https://github.com/swiftlang/swift-foundation/blob/main/Proposals/0007-swift-subprocess.md) +as an alternative: + +```swift +let process = Process() +process.executableURL = URL(filePath: "/usr/bin/swift", directoryHint: .notDirectory) +process.arguments = ["build", "--package-path", ...] +let result = try await #require(process, exitsWith: .success) +#expect(result.standardOutputContent.contains("Build went well!").utf8) +``` + +### Conformance of ExitStatus to ExpressibleByIntegerLiteral + +A contributor on the Swift forums suggested having `ExitStatus` conform to +[`ExpressibleByIntegerLiteral`](https://developer.apple.com/documentation/swift/expressiblebyintegerliteral) +and interpreting an integer literal as an exit code, such that a test author +could write: + +```swift +await #expect(processExitsWith: EX_CANTCREAT) { + ... +} +``` + +This would be convenient for test authors who are dealing with a variety of exit +codes, but is beyond the scope of this proposal. Adding conformance to this +protocol also requires some care to ensure that signal constants such as +`SIGABRT` cannot be accidentally interpreted as exit codes. + +## Alternatives considered + +- Doing nothing. + +- Marking exit tests using a trait rather than a new `#expect()` overload: + + ```swift + @Test(.exits(with: .failure)) + func `We only eat delicious tacos`() { + var taco = Taco() + taco.isDelicious = false + eat(taco) + } + ``` + + This syntax would require separate test functions for each exit test, while + reusing the same function for relatively concise tests may be preferable. + + It would also potentially conflict with parameterized tests, as it is not + possible to pass arbitrary parameters to the child process. It would be + necessary to teach the testing library's macro target about the + `.exits(with:)` trait so that it could produce a diagnostic when used with a + parameterized test function. + +- Inferring exit tests from test functions that return `Never`: + + ```swift + @Test func `No seafood for me, thanks!`() -> Never { + var taco = Taco() + taco.toppings.append(.shrimp) + eat(taco) + fatalError("Should not have eaten that!") + } + ``` + + There's a certain synergy in inferring that a test function that returns + `Never` must necessarily be a crasher and should be handled out of process. + However, this forces the test author to add a call to `fatalError()` or + similar in the event that the code under test does _not_ terminate, and there + is no obvious way to express that a specific exit code, signal, or other + condition is expected (as opposed to just "it exited".) + + We might want to support that sort of inference in the future (i.e. "don't run + this test in-process because it will terminate the test run"), but without + also inferring success or failure from the process' exit status. + +- Naming the macro something else such as: + + - `#exits(with:_:)`; + - `#exits(because:_:)`; + - `#expect(exitsBecause:_:)`; + - `#expect(terminatesBecause:_:)`; etc. + + While "with" is normally avoided in symbol names in Swift, it sometimes really + is the best preposition for the job. "Because", "due to", and others don't + sound "right" when the entire expression is read out loud. For example, you + probably wouldn't say "exits due to success" in English. + + A contributor in the Swift forums suggested `#expect(crashes:)`: + + ```swift + await #expect(crashes: { + ... + }) + ``` + + This would preclude the possibility of writing an exit test that is expected + to exit successfully—a scenario for which we have real-world use cases. It was + also not clear that the word "crash" applied to every failing exit status. For + example, a process that exits with the POSIX-defined exit code `EX_TEMPFAIL` + likely has not _crashed_; it has just reported that the requested operation + has failed. + + This signature would also be subject to label elision when used with trailing + closure syntax, resulting in: + + ```swift + await #expect { + ... + } + ``` + + The lack of any distinguishing label here would unacceptably impact the test's + readability as it gives no indication that the code is running out-of-process + or is expected to terminate its process. + +- Combining `ExitStatus` and `ExitTest.Condition` into a single type: + + ```swift + enum ExitCondition { + case failure // any failure + case exitCode(CInt) + case signal(CInt) + } + ``` + + This simplified the set of types used for exit tests, but made comparing two + exit conditions complicated and necessitated a `==` operator that did not + satisfy the requirements of the `Equatable` protocol. + +- Naming `ExitStatus` something else such as: + + - `StatusAtExit`, which might avoid some confusion with exit _codes_ but which + is not idiomatic Swift; + - `ProcessStatus`, but we don't say "process" in our API surface elsewhere; + - `Status`, which is too generic, + - `ExitReason`, but "status" is a more widely-used term of art for this + concept; or + - `TerminationStatus` (which Foundation uses to represent approximately the + same concept), but we don't use "termination" in Swift Testing's API + anywhere. + + In particular, there was some interest in using "termination" instead of + "exit" for consistency with Foundation. Foundation and the upcoming + `Subprocess` package use both terms interchangeably, so there is precedent for + either. "Exit" is more concise; "terminate" may be read to imply that the + process was _forced_ to stop running. + +- Naming `ExitStatus.exitCode(_:)` just `.code(_:)`. Some contributors on the + forums felt that the use of "exit" here was redundant given the proposed + `exitsWith:` and `processExitsWith:` labels. However, "code" is potentially + ambiguous: does it refer to an exit code, a signal code, the code the test + author is writing, etc.? + + We certainly don't want the exit test interface to be redundant. However, + given that: + + - We _expect_ (no pun intended) most uses of exit tests will check for + `.failure` rather than a specific exit code; + - "Exit code" is an established term of art; and + - `.exitCode(_:)` may appear in other contexts (not just as an argument to + `#expect(processExitsWith:)`) + + We have opted to keep the full case name. + +- Using parameter packs to specify observed values and return types: + + ```swift + @freestanding(expression) public macro require( + processExitsWith expectedExitCondition: ExitTest.Condition, + observing observedValues: (repeat (KeyPath)) = (), + _ comment: @autoclosure () -> Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation, + performing expression: @escaping @Sendable () async throws -> Void + ) -> (repeat each T) + ``` + + Using a parameter pack in this way would make it impossible to access + properties of the returned `ExitTest.Result` value that weren't observed, and + in general would mean developers wouldn't even need to use `ExitTest.Result`: + + ```swift + let (status, stderr) = try await #expect( + processExitsWith: .failure, + observing: (\.exitStatus, \.standardErrorContent) + ) { ... } + #expect(status == ...) + #expect(stderr.contains(...)) + ``` + + Unfortunately, the `#expect(processExitsWith:)` and `#require(processExitsWith:)` + macros do not have enough information at compile time to correctly infer the + types of the key paths passed as `observedValues` above, so we end up with + rather obscure errors: + + > 🛑 Cannot convert value of type 'KeyPath<_, _>' to expected argument type + > 'KeyPath' + + If, in the future, this error is resolved, we may wish to revisit this option, + so it can also be considered a "future direction" for the feature. + +- Changing the implementation of `precondition()`, `fatalError()`, etc. in the + standard library so that they do not terminate the current process while + testing, thus removing the need to spawn a child process for an exit test. + + Most of the functions in this family return `Never`, and changing their return + types would be ABI-breaking (as well as a pessimization in production code.) + Even if we did modify these functions in the Swift standard library, other + ways to terminate the process exist and would not be covered: + + - Calling the C standard function `exit()`; + - Throwing an uncaught Objective-C or C++ exception; + - Sending a signal to the process; or + - Misusing memory (e.g. trying to dereference a null pointer.) + + Modifying the C or C++ standard library, or modifying the Objective-C runtime, + would be well beyond the scope of this proposal. + +- Skipping test functions containing exit tests on platforms that do not support + exit tests. + + This would avoid the need to write `if os(...)`, `@available(...)`, or + `if #available(...)` in a cross-platform test function before using exit + tests. Swift Testing does not currently support skipping a test that has + already started executing, and the implementation of such a feature is beyond + the scope of this proposal. + + Even if the library supported this sort of action, it would likely be + surprising to test authors that they could write a test that compiles for e.g. + iOS but doesn't run and doesn't report any problems. + + Further, in general this is not a pattern that is used in the Swift ecosystem + for platform-specific functionality; instead, `#if os(...)` and availability + checks are the normal way to mark code as platform-specific. + +## Acknowledgments + +Many thanks to the XCTest and Swift Testing team. Thanks to @compnerd for his +help with the Windows implementation. Thanks to my colleagues Coops, +Danny N., David R., Drew Y., and Robert K. at Apple for +their help with the nuances of crash reporting on macOS. diff --git a/proposals/testing/0009-attachments.md b/proposals/testing/0009-attachments.md new file mode 100644 index 0000000000..e9e23102ee --- /dev/null +++ b/proposals/testing/0009-attachments.md @@ -0,0 +1,475 @@ +# Attachments + +* Proposal: [ST-0009](0009-attachments.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Rachel Brindle](https://github.com/younata) +* Status: **Implemented (Swift 6.2)** +* Bug: [swiftlang/swift-testing#714](https://github.com/swiftlang/swift-testing/issues/714) +* Implementation: [swiftlang/swift-testing#973](https://github.com/swiftlang/swift-testing/pull/973) +* Review: ([pitch](https://forums.swift.org/t/pitch-attachments/78072)) ([review](https://forums.swift.org/t/st-0009-attachments/78698)) ([acceptance](https://forums.swift.org/t/accepted-with-modifications-st-0009-attachments/79193)) + +## Introduction + +Test authors frequently need to include out-of-band data with tests that can be +used to diagnose issues when a test fails. This proposal introduces a new API +called "attachments" (analogous to the same-named feature in XCTest) as well as +the infrastructure necessary to create new attachments and handle them in tools +like VS Code. + +## Motivation + +When a test fails, especially in a remote environment like CI, it can often be +difficult to determine what exactly has gone wrong. Data that was produced +during the test can be useful, but there is currently no mechanism in Swift +Testing to output arbitrary data other than via `stdout`/`stderr` or via an +artificially-generated issue. A dedicated interface for attaching arbitrary +information to a test would allow test authors to gather relevant information +from a test in a structured way. + +## Proposed solution + +We propose introducing a new type to Swift Testing, `Attachment`, that represents +some arbitrary "attachment" to associate with a test. Along with `Attachment`, +we will introduce a new protocol, `Attachable`, to which types can conform to +indicate they can be attached to a test. + +Default conformances to `Attachable` will be provided for standard library types +that can reasonably be attached. We will also introduce a [cross-import overlay](https://forums.swift.org/t/cross-import-overlays/36710) +with Foundation—that is, a tertiary module that is automatically imported when +a test target imports both Foundation _and_ Swift Testing—that includes +additional conformances for Foundation types such as `Data` and `URL` and +provides support for attaching values that also conform to `Encodable` or +`NSSecureCoding`. + +## Detailed design + +The `Attachment` type is defined as follows: + +```swift +/// A type describing values that can be attached to the output of a test run +/// and inspected later by the user. +/// +/// Attachments are included in test reports in Xcode or written to disk when +/// tests are run at the command line. To create an attachment, you need a value +/// of some type that conforms to ``Attachable``. Initialize an instance of +/// ``Attachment`` with that value and, optionally, a preferred filename to use +/// when writing to disk. +public struct Attachment: ~Copyable where AttachableValue: Attachable & ~Copyable { + /// A filename to use when writing this attachment to a test report or to a + /// file on disk. + /// + /// The value of this property is used as a hint to the testing library. The + /// testing library may substitute a different filename as needed. If the + /// value of this property has not been explicitly set, the testing library + /// will attempt to generate its own value. + public var preferredName: String { get } + + /// The value of this attachment. + public var attachableValue: AttachableValue { get } + + /// Initialize an instance of this type that encloses the given attachable + /// value. + /// + /// - Parameters: + /// - attachableValue: The value that will be attached to the output of the + /// test run. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the testing library attempts to + /// derive a reasonable filename for the attached value. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + public init( + _ attachableValue: consuming AttachableValue, + named preferredName: String? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) + + /// Attach an attachment to the current test. + /// + /// - Parameters: + /// - attachment: The attachment to attach. + /// - sourceLocation: The source location of the call to this function. + /// + /// When attaching a value of a type that does not conform to both + /// [`Sendable`](https://developer.apple.com/documentation/swift/sendable) and + /// [`Copyable`](https://developer.apple.com/documentation/swift/copyable), + /// the testing library encodes it as data immediately. If the value cannot be + /// encoded and an error is thrown, that error is recorded as an issue in the + /// current test and the attachment is not written to the test report or to + /// disk. + /// + /// An attachment can only be attached once. + public static func record(_ attachment: consuming Self, sourceLocation: SourceLocation = #_sourceLocation) + + /// Attach a value to the current test. + /// + /// - Parameters: + /// - attachableValue: The value to attach. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the testing library attempts to + /// derive a reasonable filename for the attached value. + /// - sourceLocation: The source location of the call to this function. + /// + /// When attaching a value of a type that does not conform to both + /// [`Sendable`](https://developer.apple.com/documentation/swift/sendable) and + /// [`Copyable`](https://developer.apple.com/documentation/swift/copyable), + /// the testing library encodes it as data immediately. If the value cannot be + /// encoded and an error is thrown, that error is recorded as an issue in the + /// current test and the attachment is not written to the test report or to + /// disk. + /// + /// This function creates a new instance of ``Attachment`` and immediately + /// attaches it to the current test. + /// + /// An attachment can only be attached once. + public static func record(_ attachableValue: consuming AttachableValue, named preferredName: String? = nil, sourceLocation: SourceLocation = #_sourceLocation) + + /// Call a function and pass a buffer representing the value of this + /// instance's ``attachableValue-2tnj5`` property to it. + /// + /// - Parameters: + /// - body: A function to call. A temporary buffer containing a data + /// representation of this instance is passed to it. + /// + /// - Returns: Whatever is returned by `body`. + /// + /// - Throws: Whatever is thrown by `body`, or any error that prevented the + /// creation of the buffer. + /// + /// The testing library uses this function when writing an attachment to a + /// test report or to a file on disk. This function calls the + /// ``Attachable/withUnsafeBytes(for:_:)`` function on this attachment's + /// ``attachableValue-2tnj5`` property. + @inlinable public borrowing func withUnsafeBytes( + _ body: (UnsafeRawBufferPointer) throws -> R + ) throws -> R +} + +extension Attachment: Copyable where AttachableValue: Copyable {} +extension Attachment: Sendable where AttachableValue: Sendable {} +``` + +With `Attachment` comes `Attachable`, a protocol to which "attachable values" +conform: + +```swift +/// A protocol describing a type that can be attached to a test report or +/// written to disk when a test is run. +/// +/// To attach an attachable value to a test, pass it to ``Attachment/record(_:named:sourceLocation:)``. +/// To further configure an attachable value before you attach it, use it to +/// initialize an instance of ``Attachment`` and set its properties before +/// passing it to ``Attachment/record(_:sourceLocation:)``. An attachable +/// value can only be attached to a test once. +/// +/// The testing library provides default conformances to this protocol for a +/// variety of standard library types. Most user-defined types do not need to +/// conform to this protocol. +/// +/// A type should conform to this protocol if it can be represented as a +/// sequence of bytes that would be diagnostically useful if a test fails. If a +/// type cannot conform directly to this protocol (such as a non-final class or +/// a type declared in a third-party module), you can create a wrapper type +/// that conforms to ``AttachableWrapper`` to act as a proxy. +public protocol Attachable: ~Copyable { + /// An estimate of the number of bytes of memory needed to store this value as + /// an attachment. + /// + /// The testing library uses this property to determine if an attachment + /// should be held in memory or should be immediately persisted to storage. + /// Larger attachments are more likely to be persisted, but the algorithm the + /// testing library uses is an implementation detail and is subject to change. + /// + /// The value of this property is approximately equal to the number of bytes + /// that will actually be needed, or `nil` if the value cannot be computed + /// efficiently. The default implementation of this property returns `nil`. + /// + /// - Complexity: O(1) unless `Self` conforms to `Collection`, in which case + /// up to O(_n_) where _n_ is the length of the collection. + var estimatedAttachmentByteCount: Int? { get } + + /// Call a function and pass a buffer representing this instance to it. + /// + /// - Parameters: + /// - attachment: The attachment that is requesting a buffer (that is, the + /// attachment containing this instance.) + /// - body: A function to call. A temporary buffer containing a data + /// representation of this instance is passed to it. + /// + /// - Returns: Whatever is returned by `body`. + /// + /// - Throws: Whatever is thrown by `body`, or any error that prevented the + /// creation of the buffer. + /// + /// The testing library uses this function when writing an attachment to a + /// test report or to a file on disk. The format of the buffer is + /// implementation-defined, but should be "idiomatic" for this type: for + /// example, if this type represents an image, it would be appropriate for + /// the buffer to contain an image in PNG format, JPEG format, etc., but it + /// would not be idiomatic for the buffer to contain a textual description of + /// the image. + borrowing func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R + + /// Generate a preferred name for the given attachment. + /// + /// - Parameters: + /// - attachment: The attachment that needs to be named. + /// - suggestedName: A suggested name to use as the basis of the preferred + /// name. This string was provided by the developer when they initialized + /// `attachment`. + /// + /// - Returns: The preferred name for `attachment`. + /// + /// The testing library uses this function to determine the best name to use + /// when adding `attachment` to a test report or persisting it to storage. The + /// default implementation of this function returns `suggestedName` without + /// any changes. + borrowing func preferredName(for attachment: borrowing Attachment, basedOn suggestedName: String) -> String +} +``` + +Default conformances to `Attachable` are provided for: + +- `Array`, `ContiguousArray`, and `ArraySlice` +- `String` and `Substring` +- `Data` (if Foundation is also imported) + +Default _implementations_ are provided for types when they conform to +`Attachable` and either `Encodable` or `NSSecureCoding` (or both.) To use these +conformances, Foundation must be imported because `JSONEncoder` and +`PropertyListEncoder` are members of Foundation, not the Swift standard library. + +Some types cannot conform directly to `Attachable` because they require +additional information to encode correctly, or because they are not directly +`Sendable` or `Copyable`. A second protocol, `AttachableWrapper`, is provided +that refines `Attachable`: + +```swift +/// A protocol describing a type that can be attached to a test report or +/// written to disk when a test is run and which wraps another value that it +/// stands in for. +/// +/// To attach an attachable value to a test, pass it to ``Attachment/record(_:named:sourceLocation:)``. +/// To further configure an attachable value before you attach it, use it to +/// initialize an instance of ``Attachment`` and set its properties before +/// passing it to ``Attachment/record(_:sourceLocation:)``. An attachable +/// value can only be attached to a test once. +/// +/// A type can conform to this protocol if it represents another type that +/// cannot directly conform to ``Attachable``, such as a non-final class or a +/// type declared in a third-party module. +public protocol AttachableWrapper: Attachable, ~Copyable { + /// The type of the underlying value represented by this type. + associatedtype Wrapped + + /// The underlying value represented by this instance. + var wrappedValue: Wrapped { get } +} + +extension Attachment where AttachableValue: AttachableWrapper & ~Copyable { + /// The value of this attachment. + /// + /// When the attachable value's type conforms to ``AttachableWrapper``, the + /// value of this property equals the wrappers's underlying attachable value. + /// To access the attachable value as an instance of `T` (where `T` conforms + /// to ``AttachableWrapper``), specify the type explicitly: + /// + /// ```swift + /// let attachableValue = attachment.attachableValue as T + /// ``` + public var attachableValue: AttachableValue.Wrapped { get } +} +``` + +The cross-import overlay with Foundation also provides the following convenience +interface for attaching the contents of a file or directory on disk: + +```swift +extension Attachment where AttachableValue == _AttachableURLWrapper { + /// Initialize an instance of this type with the contents of the given URL. + /// + /// - Parameters: + /// - url: The URL containing the attachment's data. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the name of the attachment is + /// derived from the last path component of `url`. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + /// + /// - Throws: Any error that occurs attempting to read from `url`. + public init( + contentsOf url: URL, + named preferredName: String? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) async throws +} +``` + +`_AttachableURLWrapper` is a type that conforms to `AttachableWrapper` and +encloses the URL and corresponding mapped data. As an implementation detail, it +is omitted from this proposal for brevity. + +## Source compatibility + +This proposal is additive and has no impact on existing code. + +## Integration with supporting tools + +We will add a new command-line argument to the `swift test` command in Swift +Package Manager: + +```sh +--attachments-path Path where attachments should be saved. +``` + +If specified, an attachment will be written to that path when the attachment is +passed to one of the `Attachment.attach(_:sourceLocation:)` methods. If not +specified, attachments are not saved to disk. Tools that indirectly use Swift +Testing through `swift test` can specify a path (e.g. to a directory created +inside the system's temporary directory), then move or delete the created files +as needed. + +The JSON event stream ABI will be amended correspondingly: + +```diff +--- a/Documentation/ABI/JSON.md ++++ b/Documentation/ABI/JSON.md + ::= { + "kind": , + "instant": , ; when the event occurred + ["issue": ,] ; the recorded issue (if "kind" is "issueRecorded") ++ ["attachment": ,] ; the attachment (if kind is "valueAttached") + "messages": , + ["testID": ,] + } + + ::= "runStarted" | "testStarted" | "testCaseStarted" | + "issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" | +- "runEnded" ; additional event kinds may be added in the future ++ "runEnded" | "valueAttached"; additional event kinds may be added in the future + ++ ::= { ++ "path": , ; the absolute path to the attachment on disk ++} +``` + +As these changes are additive only, the JSON schema version does not need to be +incremented to support them. We are separately planning to increment the JSON +schema version to support other features; these changes will apply to the newer +version too. + +## Future directions + +- Attachment lifetime management: XCTest's attachments allow for specifying a + "lifetime", with two lifetimes currently available: + + ```objc + typedef NS_ENUM(NSInteger, XCTAttachmentLifetime) { + XCTAttachmentLifetimeKeepAlways = 0, + XCTAttachmentLifetimeDeleteOnSuccess = 1 + }; + ``` + + If a test passes, it is probably not necessary to keep its attachments saved + to disk. The exact "shape" this feature should take in Swift Testing is not + yet clear. + +- Image attachments: it is often useful to be able to attach images to tests, + however there is no cross-platform solution for this functionality. An + experimental implementation that allows attaching an instance of `CGImage` (on + Apple platforms) is available in Swift Testing's repository and shows what it + might look like for us to provide this functionality. + +- Additional conformances for types in other modules: in order to keep Swift + Testing's dependency graph as small as possible, we cannot link it to + arbitrary packages such as (for example) swift-collections even if it would be + useful to do so. That means we can't directly provide conformances to + `Attachable` for types in those modules. Adding additional cross-import + overlays would allow us to provide those conformances when both Swift Testing + and those packages are imported at the same time. + + This functionality may require changes in Swift Package Manager that are + beyond the scope of this proposal. + +- Adopting `RawSpan` instead of `UnsafeRawBufferPointer`: `RawSpan` represents a + safer alternative to `UnsafeRawBufferPointer`, but it is not yet available + everywhere we'd need it in the standard library, and our minimum deployment + targets on Apple's platforms do not allow us to require the use of `RawSpan` + (as no shipping version of Apple's platforms includes it.) + +- Adding an associated `Metadata` type to `Attachable` allowing for inclusion of + arbitrary out-of-band data to attachments: we see several uses for such a + feature: + + - Fine-grained control of the serialization format used for `Encodable` types; + - Metrics (scaling factor, rotation, etc.) for images; and + - Compression algorithms to use for attached files and directories. + + The exact shape of this interface needs further consideration, but it could be + added in the future without disrupting the interface we are proposing here. + [swiftlang/swift-testing#824](https://github.com/swiftlang/swift-testing/pull/824) + includes an experimental implementation of this feature. + +- Attaching attachments to issues or to activities: XCTest supports attachments + on `XCTIssue`; Swift Testing does not currently allow developers to create an + issue without immediately recording it, so there is no opportunity to attach + anything to one. XCTest also supports the concept of activities as subsections + of tests; they remain a future direction for Swift Testing. + +## Alternatives considered + +- Doing nothing: there's sufficient demand for this feature that we know we want + to address it. + +- Reusing the existing `XCTAttachment` API from XCTest: while this would + _probably_ have saved me a lot of typing, `XCTAttachment` is an Objective-C + class and is only available on Apple's platforms. The open-source + swift-corelibs-xctest package does not include it or an equivalent interface. + As well, this would create a dependency on XCTest in Swift Testing that does + not currently exist. + +- Implementing `Attachment` as a non-generic type and eagerly serializing + non-sendable or move-only attachable values: an earlier implementation did + exactly this, but it forced us to include an existential box in `Attachment` + to store the attachable value, and that would preclude ever supporting + attachments in Embedded Swift. + +- Having `Attachment` take a byte buffer rather than an attachable value, or + having it take a closure that returns a byte buffer: this would just raise the + problem of attaching arbitrary values up to the test author's layer, and that + would no doubt produce a lot of duplicate implementations of "turn this value + into a byte buffer" while also worsening the interface's ergonomics. + +- Adding a `var contentType: UTType { get set }` property to `Attachment` or to + `Attachable`: `XCTAttachment` lets you specify a Uniform Type Identifier that + tells Xcode the type of data. Uniform Type Identifiers are proprietary and not + available on Linux or Windows, and adding that property would force us to also + add a public dependency on the `UniformTypeIdentifiers` framework and, + indirectly, on Foundation, which would prevent Foundation from authoring tests + using Swift Testing in the future due to the resulting circular dependency. + + We considered using a MIME type instead, but there is no portable mechanism + for turning a MIME type into a path extension, which is ultimately what we + need when writing an attachment to persistent storage. + + Instead, `Attachable` includes the function `preferredName(for:basedOn:)` that + allows an implementation (such as that of `Encodable & Attachable`) to add a + path extension to the filename specified by the test author if needed. + +- Making the `Attachment.record(_:[named:]sourceLocation:)` methods a single + instance method of `Attachment` named `attach()`: this was in the initial + pitch but the community discussed several more ergonomic options and we chose + `Attachment.record(_:sourceLocation:)` instead. + +## Acknowledgments + +Thanks to Stuart Montgomery and Brian Croom for goading me into finally writing +this proposal! + +Thanks to Wil Addario-Turner for his feedback, in particular around `UTType` and +MIME type support. + +Thanks to Honza Dvorsky for his earlier work on attachments in XCTest and his +ideas on how to improve Swift Testing's implementation. diff --git a/proposals/testing/0010-evaluate-condition.md b/proposals/testing/0010-evaluate-condition.md new file mode 100644 index 0000000000..d6a7336ff1 --- /dev/null +++ b/proposals/testing/0010-evaluate-condition.md @@ -0,0 +1,65 @@ +# Public API to evaluate ConditionTrait + +* Proposal: [ST-0010](0010-evaluate-condition.md) +* Authors: [David Catmull](https://github.com/Uncommon) +* Review Manager: [Stuart Montgomery](https://github.com/stmontgomery) +* Status: **Implemented (Swift 6.2)** +* Bug: [swiftlang/swift-testing#903](https://github.com/swiftlang/swift-testing/issues/903) +* Implementation: [swiftlang/swift-testing#909](https://github.com/swiftlang/swift-testing/pull/909), [swiftlang/swift-testing#1097](https://github.com/swiftlang/swift-testing/pull/1097) +* Review: ([pitch](https://forums.swift.org/t/pitch-introduce-conditiontrait-evaluate/77242)) ([review](https://forums.swift.org/t/st-0010-public-api-to-evaluate-conditiontrait/79232)) ([acceptance](https://forums.swift.org/t/accepted-st-0010-public-api-to-evaluate-conditiontrait/79577)) + +## Introduction + +This adds an `evaluate()` method to `ConditionTrait` to evaluate the condition +without requiring a `Test` instance. + +## Motivation + +Currently, the only way a `ConditionTrait` is evaluated is inside the +`prepare(for:)` method. This makes it difficult for third-party libraries to +utilize these traits because evaluating a condition would require creating a +dummy `Test` to pass to that method. + +## Proposed solution + +The proposal is to add a `ConditionTrait.evaluate()` method which returns the +result of the evaluation. The existing `prepare(for:)` method is updated to call +`evaluate()` so that the logic is not duplicated. + +## Detailed design + +The `evaluate()` method is as follows, containing essentially the same logic +as was in `prepare(for:)`: + +```swift +extension ConditionTrait { + /// Evaluate this instance's underlying condition. + /// + /// - Returns: The result of evaluating this instance's underlying condition. + /// + /// The evaluation is performed each time this function is called, and is not + /// cached. + public func evaluate() async throws -> Bool +} +``` + +## Source compatibility + +This change is purely additive. + +## Integration with supporting tools + +This change allows third-party libraries to apply condition traits at other +levels than suites or whole test functions, for example if tests are broken up +into smaller sections. + +## Future directions + +This change seems sufficient for third party libraries to make use of +`ConditionTrait`. Changes for other traits can be tackled in separate proposals. + +## Alternatives considered + +Exposing `ConditionTrait.Kind` and `.kind` was also considered, but it seemed +unnecessary to go that far, and it would encourage duplicating the logic that +already exists in `prepare(for:)`. diff --git a/proposals/testing/0011-issue-handling-traits.md b/proposals/testing/0011-issue-handling-traits.md new file mode 100644 index 0000000000..4d16da9c70 --- /dev/null +++ b/proposals/testing/0011-issue-handling-traits.md @@ -0,0 +1,557 @@ +# Issue Handling Traits + +* Proposal: [ST-0011](0011-issue-handling-traits.md) +* Authors: [Stuart Montgomery](https://github.com/stmontgomery) +* Review Manager: [Paul LeMarquand](https://github.com/plemarquand) +* Status: **Implemented (Swift 6.2)** +* Implementation: [swiftlang/swift-testing#1080](https://github.com/swiftlang/swift-testing/pull/1080), + [swiftlang/swift-testing#1121](https://github.com/swiftlang/swift-testing/pull/1121), + [swiftlang/swift-testing#1136](https://github.com/swiftlang/swift-testing/pull/1136), + [swiftlang/swift-testing#1198](https://github.com/swiftlang/swift-testing/pull/1198) +* Review: ([pitch](https://forums.swift.org/t/pitch-issue-handling-traits/80019)) ([review](https://forums.swift.org/t/st-0011-issue-handling-traits/80644)) ([acceptance](https://forums.swift.org/t/accepted-st-0011-issue-handling-traits/81112)) + +## Introduction + +This proposal introduces a built-in trait for handling issues in Swift Testing, +enabling test authors to customize how expectation failures and other issues +recorded by tests are represented. Using a custom issue handler, developers can +transform issue details, perform additional actions, or suppress certain issues. + +## Motivation + +Swift Testing offers ways to customize test attributes and perform custom logic +using traits, but there's currently no way to customize how issues (such as +`#expect` failures) are handled when they occur during testing. + +The ability to handle issues using custom logic would enable test authors to +modify, supplement, or filter issues based on their specific requirements before +the testing library processes them. This capability could open the door to more +flexible testing approaches, improve integration with external reporting systems, +or improve the clarity of results in complex testing scenarios. The sections +below discuss several potential use cases for this functionality. + +### Adding information to issues + +#### Comments + +Sometimes test authors want to include context-specific information to certain +types of failures. For example, they might want to automatically add links to +documentation for specific categories of test failures, or include supplemental +information about the history of a particular expectation in case it fails. An +issue handler could intercept issues after they're recorded and add these +details to the issue's comments before the testing library processes them. + +#### Attachments + +Test failures often benefit from additional diagnostic data beyond the basic +issue description. Swift Testing now supports attachments (as of +[ST-0009](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0009-attachments.md)), +and the ability to add an attachment to an indiviual issue was mentioned as a +future direction in that proposal. The general capability of adding attachments +to issues is outside the scope of this proposal, but if such a capability were +introduced, an issue handler could programmatically attach log files, +screenshots, or other diagnostic artifacts when specific issues occur, making it +easier to diagnose test failures. + +### Suppressing warnings + +Recently, a new API was [pitched][severity-proposal] which would introduce the +concept of severity to issues, along with a new _warning_ severity level, making +it possible to record warnings that do not cause a test to be marked as a +failure. If that feature is accepted, there may be cases where a test author +wants to suppress certain warnings entirely or in specific contexts. + +For instance, they might choose to suppress warnings recorded by the testing +library indicating that two or more arguments to a parameterized test appear +identical, or for one of the other scenarios listed as potential use cases for +warning issues in that proposal. An issue handler would provide a mechanism to +filter issues. + +### Raising or lowering an issue's severity + +Beyond suppressing issues altogether, a test author might want to modify the +severity of an issue (again, assuming the recently pitched +[Issue Severity][severity-proposal] proposal is accepted). They might wish to +either _lower_ an issue with the default error-level severity to a warning (but +not suppress it), or conversely _raise_ a warning issue to an error. + +The Swift compiler now allows control over warning diagnostics (as of +[SE-0443](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0443-warning-control-flags.md)). +An issue handling trait would offer analogous functionality for test issues. + +### Normalizing issue details + +Tests that involve randomized or non-deterministic inputs can generate different +issue descriptions on each run, making it difficult to identify duplicates or +recognize patterns in failures. For example, a test verifying random number +generation might produce an expectation failure with different random values +each time: + +``` +Expectation failed: (randomValue → 0.8234) > 0.5 +Expectation failed: (randomValue → 0.6521) > 0.5 +``` + +An issue handler could normalize these issues to create a more consistent +representation: + +``` +Expectation failed: (randomValue → 0.NNNN) > 0.5 +``` + +The original numeric value could be preserved via a comment after being +obfuscated—see [Comments](#comments) under [Adding information to issues](#adding-information-to-issues) +above. + +> [!NOTE] +> This example involves an expectation failure. The value of the `kind` property +> for such an issue would be `.expectationFailed(_:)` and it would have an +> associated value of type `Expectation`. To transform the issue in the way +> described above would require modifying details of the associated `Expectation` +> and its substructure, but these details are currently SPI so test authors +> cannot modify them directly. +> +> Exposing these details is out of scope for this proposal, but a test author +> could still transform this issue to achieve a similar result by changing the +> issue's kind from `.expectationFailed(_:)` to `.unconditional`. This +> experience could be improved in the future in subsequent proposals if desired. + +This normalization can significantly improve the ability to triage failures, as +it becomes easier to recognize when multiple test failures have the same root +cause despite different specific values. + +## Proposed solution + +This proposal introduces a new trait type that can customize how issues are +processed during test execution. + +Here's one contrived example showing how this could be used to add a comment to +each issue recorded by a test: + +```swift +@Test(.compactMapIssues { issue in + var issue = issue + issue.comments.append("Checking whether two literals are equal") + return issue +}) +func literalComparisons() { + #expect(1 == 1) // ✅ + #expect(2 == 3) // ❌ Will invoke issue handler + #expect("a" == "b") // ❌ Will invoke issue handler again +} +``` + +Here's an example showing how warning issues matching a specific criteria could +be suppressed using `.filterIssues`. It also showcases a technique for reusing +an issue handler across multiple tests, by defining it as a computed property in +an extension on `Trait`: + +```swift +extension Trait where Self == IssueHandlingTrait { + static var ignoreSensitiveWarnings: Self { + .filterIssues { issue in + let description = String(describing: issue) + + // Note: 'Issue.severity' has been pitched but not accepted. + return issue.severity <= .warning && SensitiveTerms.all.contains { description.contains($0) } + } + } +} + +@Test(.ignoreSensitiveWarnings) func exampleA() { + ... +} + +@Test(.ignoreSensitiveWarnings) func exampleB() { + ... +} +``` + +The sections below discuss some of the proposed new trait's behavioral details. + +### Precedence order of handlers + +If multiple issue handling traits are applied to or inherited by a test, they +are executed in trailing-to-leading, innermost-to-outermost order. For example, +given the following code: + +```swift +@Suite(.compactMapIssues { ... /* A */ }) +struct ExampleSuite { + @Test(.filterIssues { ... /* B */ }, + .compactMapIssues { ... /* C */ }) + func example() { + ... + } +} +``` + +If an issue is recorded in `example()`, it's processed first by closure C, then +by B, and finally by A. (Unless an issue is suppressed, in which case it will +not advance to any subsequent handler's closure.) This ordering provides +predictable behavior and allows more specific handlers to process issues before +more general ones. + +### Accessing task-local context from handlers + +The closure of an issue handler is invoked synchronously at the point where an +issue is recorded. This means the closure can access task local state from that +context, and potentially use that to augment issues with extra information. +Here's an example: + +```swift +// In module under test: +actor Session { + @TaskLocal static var current: Session? + + let id: String + func connect() { ... } + var isConnected: Bool { ... } + ... +} + +// In test code: +@Test(.compactMapIssues { issue in + var issue = issue + if let session = Session.current { + issue.comments.append("Current session ID: \(session.id)") + } + return issue +}) +func example() async { + let session = Session(id: "ABCDEF") + await Session.$current.withValue(session) { + await session.connect() + #expect(await session.isConnected) // ❌ Expectation failed: await session.isConnected + // Current session ID: ABCDEF + } +} +``` + +### Recording issues from handlers + +Issue handling traits can record additional issues during their execution. These +newly recorded issues will be processed by any later issue handling traits in +the processing chain (see [Precedence order of handlers](#precedence-order-of-handlers)). +This capability allows handlers to augment or provide context to existing issues +by recording related information. + +For example: + +```swift +@Test( + .compactMapIssues { issue in + // This closure will be called for any issue recorded by the test function + // or by the `.filterIssues` trait below. + ... + }, + .filterIssues { issue in + guard let terms = SensitiveTerms.all else { + Issue.record("Cannot determine the set of sensitive terms. Filtering issue by default.") + return true + } + + let description = String(describing: issue).lowercased() + return terms.contains { description.contains($0) } + } +) +func example() { + ... +} +``` + +### Handling issues from other traits + +Issue handling traits process all issues recorded in the context of a test, +including those generated by other traits applied to the test. For instance, if +a test uses the `.enabled(if:)` trait and the condition closure throws an error, +that error will be recorded as an issue, the test will be skipped, and the issue +will be passed to any issue handling traits for processing. + +This comprehensive approach ensures that all issues related to a test, +regardless of their source, are subject to the same customized handling. It +provides a unified mechanism for issue processing that works consistently across +the testing library. + +### Effects in issue handler closures + +The closure of an issue handling trait must be: + +- **Non-`async`**: This reflects the fact that events in + Swift Testing are posted synchronously, which is a fundamental design decision + that, among other things, avoids the need for `await` before every `#expect`. + + While this means that issue handlers cannot directly perform asynchronous work + when processing an individual issue, future enhancements could offer + alternative mechanisms for asynchronous issue processing work at the end of a + test. See the [Future directions](#future-directions) section for more + discussion about this. + +- **Non-`throws`**: Since these handlers are already + being called in response to a failure (the recorded issue), allowing them to + throw errors would introduce ambiguity about how such errors should be + interpreted and reported. + + If an issue handler encounters an error, it can either: + + - Return a modified issue that includes information about the problem, or + - Record a separate issue using the standard issue recording mechanisms (as + [discussed](#recording-issues-from-handlers) above). + +### Handling of non-user issues + +Issue handling traits are applied to a test by a user, and are only intended for +handling issues recorded by tests written by the user. If an issue is recorded +by the testing library itself or the underlying system, not due to a failure +within the tests being run, such an issue will not be passed to an issue +handling trait. Similarly, an issue handling trait should not return an issue +which represents a problem they could not have caused in their test. + +Concretely, this policy means that issues for which the value of the `kind` +property is `.system` will not be passed to the closure of an issue handling +trait. Also, it is not supported for a closure passed to +`compactMapIssues(_:)` to return an issue for which the value of `kind` is +either `.system` or `.apiMisused` (unless the passed-in issue had that kind, +which should only be possible for `.apiMisused`). + +## Detailed design + +This proposal includes the following: + +* A new `IssueHandlingTrait` type that conforms to `TestTrait` and `SuiteTrait`. + * An instance method `handleIssue(_:)` which can be called directly on a + handler trait. This may be useful for composing multiple issue handling + traits. +* Static functions on `Trait` for creating instances of this type with the + following capabilities: + * A function `compactMapIssues(_:)` which returns a trait that can transform + recorded issues. The function takes a closure which is passed an issue and + returns either a modified issue or `nil` to suppress it. + * A function `filterIssues(_:)` which returns a trait that can filter recorded + issues. The function takes a predicate closure that returns a boolean + indicating whether to keep (`true`) or suppress (`false`) an issue. + +Below are the proposed interfaces: + +```swift +/// A type that allows transforming or filtering the issues recorded by a test. +/// +/// Use this type to observe or customize the issue(s) recorded by the test this +/// trait is applied to. You can transform a recorded issue by copying it, +/// modifying one or more of its properties, and returning the copy. You can +/// observe recorded issues by returning them unmodified. Or you can suppress an +/// issue by either filtering it using ``Trait/filterIssues(_:)`` or returning +/// `nil` from the closure passed to ``Trait/compactMapIssues(_:)``. +/// +/// When an instance of this trait is applied to a suite, it is recursively +/// inherited by all child suites and tests. +/// +/// To add this trait to a test, use one of the following functions: +/// +/// - ``Trait/compactMapIssues(_:)`` +/// - ``Trait/filterIssues(_:)`` +public struct IssueHandlingTrait: TestTrait, SuiteTrait { + /// Handle a specified issue. + /// + /// - Parameters: + /// - issue: The issue to handle. + /// + /// - Returns: An issue to replace `issue`, or else `nil` if the issue should + /// not be recorded. + public func handleIssue(_ issue: Issue) -> Issue? +} + +extension Trait where Self == IssueHandlingTrait { + /// Constructs an trait that transforms issues recorded by a test. + /// + /// - Parameters: + /// - transform: A closure called for each issue recorded by the test + /// this trait is applied to. It is passed a recorded issue, and returns + /// an optional issue to replace the passed-in one. + /// + /// - Returns: An instance of ``IssueHandlingTrait`` that transforms issues. + /// + /// The `transform` closure is called synchronously each time an issue is + /// recorded by the test this trait is applied to. The closure is passed the + /// recorded issue, and if it returns a non-`nil` value, that will be recorded + /// instead of the original. Otherwise, if the closure returns `nil`, the + /// issue is suppressed and will not be included in the results. + /// + /// The `transform` closure may be called more than once if the test records + /// multiple issues. If more than one instance of this trait is applied to a + /// test (including via inheritance from a containing suite), the `transform` + /// closure for each instance will be called in right-to-left, innermost-to- + /// outermost order, unless `nil` is returned, which will skip invoking the + /// remaining traits' closures. + /// + /// Within `transform`, you may access the current test or test case (if any) + /// using ``Test/current`` ``Test/Case/current``, respectively. You may also + /// record new issues, although they will only be handled by issue handling + /// traits which precede this trait or were inherited from a containing suite. + /// + /// - Note: `transform` will never be passed an issue for which the value of + /// ``Issue/kind`` is ``Issue/Kind/system``, and may not return such an + /// issue. + public static func compactMapIssues(_ transform: @escaping @Sendable (Issue) -> Issue?) -> Self + + /// Constructs a trait that filters issues recorded by a test. + /// + /// - Parameters: + /// - isIncluded: The predicate with which to filter issues recorded by the + /// test this trait is applied to. It is passed a recorded issue, and + /// should return `true` if the issue should be included, or `false` if it + /// should be suppressed. + /// + /// - Returns: An instance of ``IssueHandlingTrait`` that filters issues. + /// + /// The `isIncluded` closure is called synchronously each time an issue is + /// recorded by the test this trait is applied to. The closure is passed the + /// recorded issue, and if it returns `true`, the issue will be preserved in + /// the test results. Otherwise, if the closure returns `false`, the issue + /// will not be included in the test results. + /// + /// The `isIncluded` closure may be called more than once if the test records + /// multiple issues. If more than one instance of this trait is applied to a + /// test (including via inheritance from a containing suite), the `isIncluded` + /// closure for each instance will be called in right-to-left, innermost-to- + /// outermost order, unless `false` is returned, which will skip invoking the + /// remaining traits' closures. + /// + /// Within `isIncluded`, you may access the current test or test case (if any) + /// using ``Test/current`` ``Test/Case/current``, respectively. You may also + /// record new issues, although they will only be handled by issue handling + /// traits which precede this trait or were inherited from a containing suite. + /// + /// - Note: `isIncluded` will never be passed an issue for which the value of + /// ``Issue/kind`` is ``Issue/Kind/system``. + public static func filterIssues(_ isIncluded: @escaping @Sendable (Issue) -> Bool) -> Self +} +``` + +## Source compatibility + +This new trait is additive and should not affect source compatibility of +existing test code. + +If any users have an existing extension on `Trait` containing a static function +whose name conflicts with one in this proposal, the standard technique of +fully-qualifying its callsite with the relevant module name can be used to +resolve any ambiguity, but this should be rare. + +## Integration with supporting tools + +Most tools which integrate with the testing library interpret recorded issues in +some way, whether by writing them to a persistent data file or presenting them +in UI. These mechanisms will continue working as before, but the issues they act +on will be the result of any issue handling traits. If an issue handler +transforms an issue, the integrated tool will only receive the transformed issue, +and if a trait suppresses an issue, the tool will not be notified about the +issue at all. + +## Future directions + +### "Test ended" trait + +The current proposal does not allow `await` in an issue handling closure--see +[Non-`async`](#non-async) above. In addition to not allowing concurrency, the +proposed behavior is that the issue handler is called once for _each_ issue +recorded. + +Both of these policies could be problematic for some use cases. Some users may +want to collect additional diagnostics if a test fails, but only do so once per +per test (typically after it finishes) instead of once per _issue_, since the +latter may lead to redundant or wasteful work. Also, collecting diagnostics may +require calling `async` APIs. + +In the future, a new trait could be added which offers a closure that is +unconditionally called once after a test ends. The closure could be provided the +result of the test (e.g. pass/fail/skip) and access to all the issues it +recorded. This hypothetical trait's closure could be safely made `async`, since +it wouldn't be subject to the same limitations as event delivery, and this could +complement the APIs proposed above. + +### Comprehensive event observation API + +As a more generalized form of the ["Test ended" trait](#test-ended-trait) idea +above, Swift Testing could offer a more comprehensive suite of APIs for +observing test events of all kinds. This would a much larger effort, but was +[mentioned](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md#flexibility) +as a goal in the +[Swift Testing vision document](https://github.com/swiftlang/swift-evolution/blob/main/visions/swift-testing.md). + +### Standalone function + +It could be useful to offer the functionality of an issue handling trait as a +standalone function (similar to `withKnownIssue { }`) so that it could be +applied to a narrower section of code than an entire test or suite. This idea +came up during the pitch phase, and we believe that sort of pattern may be +useful more broadly for other kinds of traits. Accordingly, it may make more +sense to address this in a separate proposal and design it in a way that +encompasses any trait. + +## Alternatives considered + +### Allow issue handler closures to throw errors + +The current proposal does not allow throwing an error from an issue handling +closure--see [Non-`throws`](#non-throws) above. This artificial restriction +could be lifted, and errors thrown by issue handler closures could be caught +and recorded as issues, matching the behavior of test functions. + +As mentioned earlier, allowing thrown errors could make test results more +confusing. We expect that most often, a test author will add an issue handler +because they want to make failures easier to interpret, and they generally won't +want an issue handler to record _more_ issues while doing so even if it can. Not +allowing errors to be thrown forces the author of the issue handler to make an +explicit decision about whether they want an additional issue to be recorded if +the handler encounters an error. + +### Make the closure's issue parameter `inout` + +The closure parameter of `compactMapIssues(_:)` currently has one parameter of +type `Issue` and returns an optional `Issue?` to support returning `nil` in +order to suppress an issue. If an issue handler wants to modify an issue, it +first needs to copy it to a mutable variable (`var`), mutate it, then return the +modified copy. These copy and return steps require extra lines of code within +the closure, and they could be eliminated if the parameter was declared `inout`. + +The most straightforward way to achieve this would be for the closure to instead +have a `Void` return type and for its parameter to become `inout`. However, in +order to _suppress_ an issue, the parameter would also need to become optional +(`inout Issue?`) and this would mean that all usages would first need to be +unwrapped. This feels non-ergonomic, and would differ from the standard +library's typical pattern for `compactMap` functions. + +Another way to achieve this ([suggested](https://forums.swift.org/t/st-0011-issue-handling-traits/80644/3) +by [@Val](https://forums.swift.org/u/Val) during proposal review) could be to +declare the return type of the closure `Void?` and the parameter type +`inout Issue` (non-optional). This alternative would not require unwrapping the +issue first and would still permit suppressing issues by returning `nil`. It +could also make one of the alternative names (such as `transformIssues` +discussed below) more fitting. However, this is a novel API pattern which isn't +widely used in Swift, and may be confusing to users. There were also concerns +raised by other reviewers that the language's implicit return for `Void` may not +be intentionally applied to `Optional` and that this mechanism could break +in the future. + +### Alternate names for the static trait functions + +We could choose different names for the static `compactMapIssues(_:)` or +`filterIssues(_:)` functions. Some alternate names considered were: + +- `transformIssues` instead of `compactMapIssues`. "Compact map" seemed to align + better with "filter" of `filterIssues`, however. +- `handleIssues` instead of `compactMapIssues`. The word "handle" is in the name + of the trait type already; it's a more general word for what all of these + usage patterns enable, so it felt too broad. +- Using singular "issue" rather than plural "issues" in both APIs. This may not + adequately convey that the closure can be invoked more than once. + +## Acknowledgments + +Thanks to [Brian Croom](https://github.com/briancroom) for feedback on the +initial concept, and for making a suggestion which led to the +["Test ended" trait](#test-ended-trait) idea mentioned in Alternatives +considered. + +[severity-proposal]: https://forums.swift.org/t/pitch-test-issue-warnings/79285 diff --git a/proposals/testing/0012-exit-test-value-capturing.md b/proposals/testing/0012-exit-test-value-capturing.md new file mode 100644 index 0000000000..c31fd772ae --- /dev/null +++ b/proposals/testing/0012-exit-test-value-capturing.md @@ -0,0 +1,276 @@ +# Capturing values in exit tests + +* Proposal: [ST-0012](0012-exit-test-value-capturing.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Paul LeMarquand](https://github.com/plemarquand) +* Status: **Implemented (Swift 6.3)** +* Bug: [swiftlang/swift-testing#1157](https://github.com/swiftlang/swift-testing/issues/1157) +* Implementation: [swiftlang/swift-testing#1040](https://github.com/swiftlang/swift-testing/pull/1040), [swiftlang/swift-testing#1165](https://github.com/swiftlang/swift-testing/pull/1165) _et al._ +* Review: ([pitch](https://forums.swift.org/t/pitch-capturing-values-in-exit-tests/80494)) ([review](https://forums.swift.org/t/st-0012-capturing-values-in-exit-tests/80963)) ([acceptance](https://forums.swift.org/t/accepted-st-0012-capturing-values-in-exit-tests/81250)) + +## Introduction + +In Swift 6.2, we introduced the concept of an _exit test_: a section of code in +a test function that would run in an independent process and allow test authors +to test code that terminates the process. For example: + +```swift +enum Fruit: Sendable, Codable, Equatable { + case apple, orange, olive, tomato + var isSweet: Bool { get } + + consuming func feed(to bat: FruitBat) { + precondition(self.isSweet, "Fruit bats don't like savory fruits!") + ... + } +} + +@Test func `Fruit bats don't eat savory fruits`() async { + await #expect(processExitsWith: .failure) { + let fruit = Fruit.olive + let bat = FruitBat(named: "Chauncey") + fruit.feed(to: bat) // should trigger a precondition failure and process termination + } +} +``` + +This proposal extends exit tests to support capturing state from the enclosing +context (subject to several practical constraints.) + +## Motivation + +Exit tests in their current form are useful, but there is no reliable way to +pass non-constant information from the parent process to the child process, +which makes them difficult to use with parameterized tests. Consider: + +```swift +@Test(arguments: [Fruit.olive, .tomato]) +func `Fruit bats don't eat savory fruits`(_ fruit: Fruit) async { + await #expect(processExitsWith: .failure) { + let bat = FruitBat(named: "Chauncey") + fruit.feed(to: bat) // 🛑 can't capture 'fruit' from enclosing scope + } +} +``` + +In the above example, the test function's argument cannot be passed into the +exit test. In a trivial example like this one, it wouldn't be difficult to write +two tests that differ only in the case of `Fruit` they use in their exit test +bodies, but this approach doesn't scale very far and is generally an +anti-pattern when using Swift Testing. + +## Proposed solution + +We propose allowing the capture of values in an exit test when they are +specified in a closure capture list on the exit test's body. + +## Detailed design + +The signatures of the exit test macros `expect(processExitsWith:)` and +`require(processExitsWith:)` are unchanged. A test author may now add a closure +capture list to the body of an exit test: + +```swift +@Test(arguments: [Fruit.olive, .tomato]) +func `Fruit bats don't eat savory fruits`(_ fruit: Fruit) async { + await #expect(processExitsWith: .failure) { [fruit] in + let bat = FruitBat(named: "Chauncey") + fruit.feed(to: bat) + } +} +``` + +This feature has some necessary basic constraints: + +### Captured values must be explicitly listed in a closure capture list + +Swift Testing needs to know what values need to be encoded, sent to the child +process, and decoded. Swift macros including `#expect(processExitsWith:)` must +rely solely on syntax—that is, the code typed by a test author. An implicit +capture within an exit test body is indistinguishable from any other identifier +or symbol name. + +Hence, only values listed in the closure's capture list will be captured. +Implicitly captured values will produce a compile-time diagnostic as they do +today. + +### Captured values must conform to Sendable and Codable + +Captured values will be sent across process boundaries and, in order to support +that operation, must conform to `Codable`. As well, captured values need to make +their way through the various internal mechanisms of Swift Testing and its host +infrastructure, and so must conform to `Sendable`. Conformance to `Copyable` and +`Escapable` is implied. + +If a value that does _not_ conform to the above protocols is specified in an +exit test body's capture list, a diagnostic is emitted: + +```swift +let bat: FruitBat = ... +await #expect(processExitsWith: .failure) { [bat] in + // 🛑 Type of captured value 'bat' must conform to 'Sendable' and 'Codable' + ... +} +``` + +### Captured values' types must be visible to the exit test macro + +In order for us to successfully _decode_ captured values in the child process, +we must know their Swift types. Type information is not readily available during +macro expansion and we must, in general, rely on the parsed syntax tree for it. + +The type of `self` and the types of arguments to the calling function are, +generally, known and can be inferred from context[^shadows]. The types of other +values, including local variables and global state, are not visible in the +syntax tree and must be specified explicitly in the capture list using an `as` +expression: + +```swift +await #expect(processExitsWith: .failure) { [fruit = fruit as Fruit] in + ... +} +``` + +Finally, the types of captured literals (e.g. `[x = 123]`) are known at compile +time and can always be inferred as `IntegerLiteralType` etc., although we don't +anticipate this will be particularly useful in practice. + +If the type of a captured value cannot be resolved from context, the test author +will see an error at compile time: + +```swift +await #expect(processExitsWith: .failure) { [fruit] in + // 🛑 Type of captured value 'fruit' is ambiguous + // Fix-It: Add '= fruit as T' + ... +} +``` + +See the **Future directions** section of this proposal for more information on +how we hope to lift this constraint. If we are able to lift this constraint in +the future, we expect it will not require (no pun intended) a second Swift +Evolution proposal. + +[^shadows]: If a local variable is declared that shadows `self` or a function + argument, we may incorrectly infer the type of that value when captured. When + this occurs, Swift Testing emits a diagnostic of the form "🛑 Type of captured + value 'foo' is ambiguous". + +## Source compatibility + +This change is additive and relies on syntax that would previously be rejected +at compile time. + +## Integration with supporting tools + +Xcode, Swift Package Manager, and the Swift VS Code plugin _already_ support +captured values in exit tests as they use Swift Testing's built-in exit test +handling logic. + +Tools that implement their own exit test handling logic will need to account for +captured values. The `ExitTest` type now has a new SPI property: + +```swift +extension ExitTest { + /// The set of values captured in the parent process before the exit test is + /// called. + /// + /// This property is automatically set by the testing library when using the + /// built-in exit test handler and entry point functions. Do not modify the + /// value of this property unless you are implementing a custom exit test + /// handler or entry point function. + /// + /// The order of values in this array must be the same between the parent and + /// child processes. + @_spi(ForToolsIntegrationOnly) + public var capturedValues: [CapturedValue] { get set } +} +``` + +In the parent process (that is, for an instance of `ExitTest` passed to +`Configuration.exitTestHandler`), this property represents the values captured +at runtime by the exit test. In the child process (that is, for an instance of +`ExitTest` returned from `ExitTest.find(identifiedBy:)`), the elements in this +array do not have values associated with them until the hosting tool provides +them. + +## Future directions + +- Supporting captured values without requiring type information + + We need the types of captured values in order to successfully decode them, but + we are constrained by macros being syntax-only. In the future, the compiler + may gain a language feature similar to `decltype()` in C++ or `typeof()` in + C23, in which case we should be able to use it and avoid the need for explicit + types in the capture list. ([rdar://153389205](rdar://153389205)) + +- Explicitly marking the body closure as requiring explicit captures + + Currently, if the body closure implicitly captures a value, the diagnostic the + compiler provides is a bit opaque: + + > 🛑 A C function pointer cannot be formed from a closure that captures context + + In the future, it may be possible to annotate the body closure with an + attribute, keyword, or other decoration that tells the compiler we need an + explicit capture list, which would allow it to provide a clearer diagnostic if + a value is implicitly captured. + +- Supporting capturing values that do not conform to `Codable` + + Alternatives to `Codable` exist or have been proposed, such as + [`NSSecureCoding`](https://developer.apple.com/documentation/foundation/nssecurecoding) + or [`JSONCodable`](https://forums.swift.org/t/the-future-of-serialization-deserialization-apis/78585). + In the future, we may want to extend support for values that conform to these + protocols instead of `Codable`. + +## Alternatives considered + +- Doing nothing. There is sufficient motivation to support capturing values in + exit tests and it is within our technical capabilities. + +- Passing captured values as arguments to `#expect(processExitsWith:)` and its + body closure. For example: + + ```swift + await #expect( + processExitsWith: .failure, + arguments: [fruit, bat] + ) { fruit, bat in + ... + } + ``` + + This is technically feasible, but: + + - It requires that the caller state the capture list twice; + - Type information still isn't available for captured values, so you'd still + need to _actually_ write `{ (fruit: Fruit, bat: Bat) in ... }` (or otherwise + specify the types somewhere in the macro invocation); and + - The language already has a dedicated syntax for specifying lists of values + that should be captured in a closure. + +- Supporting non-`Sendable` or non-`Codable` captured values. Since exit tests' + bodies are, by definition, in separate isolation domains from the caller, and + since they, by nature, run in separate processes, conformance to these + protocols is fundamentally necessary. + +- Implicitly capturing `self`. This would require us to statically detect during + macro expansion whether `self` conformed to the necessary protocols _and_ + would preclude capturing any state from static or free test functions. + +- Forking the exit test process such that all captured values are implicitly + copied by the kernel into the new process. Forking, in the UNIX fashion, is + fundamentally incompatible with the Swift runtime and the Swift thread pool. + On Darwin, you [cannot fork a process that links to Core Foundation without + immediately calling `exec()`](https://duckduckgo.com/?q=__THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__), + and `fork()` isn't even present on Windows. + +## Acknowledgments + +Thanks to @rintaro for assistance investigating swift-syntax diagnostic support +and to @xedin for humouring my questions about `decltype()`. + +Thanks to the Swift Testing team and the Testing Workgroup as always. And thanks +to those individuals, who shall remain unnamed, who nerd-sniped me into building +this feature. diff --git a/proposals/testing/0013-issue-severity-warning.md b/proposals/testing/0013-issue-severity-warning.md new file mode 100644 index 0000000000..ffa7c04514 --- /dev/null +++ b/proposals/testing/0013-issue-severity-warning.md @@ -0,0 +1,244 @@ +# Test Issue Severity + +- Proposal: [ST-0013](0013-issue-severity-warning.md) +- Authors: [Suzy Ratcliff](https://github.com/suzannaratcliff) +- Review Manager: [Maarten Engels](https://github.com/maartene) +- Status: **Implemented (Swift 6.3)** +- Implementation: [swiftlang/swift-testing#1075](https://github.com/swiftlang/swift-testing/pull/1075), + [swiftlang/swift-testing#1247](https://github.com/swiftlang/swift-testing/pull/1247) +- Review: ([pitch](https://forums.swift.org/t/pitch-test-issue-warnings/79285)) ([review](https://forums.swift.org/t/st-0013-test-issue-warnings/80991)) ([acceptance](https://forums.swift.org/t/accepted-st-0013-test-issue-severity/81385)) + +## Introduction + +I propose introducing a new API to Swift Testing that allows developers to record issues with a specified severity level. By default, all issues will have severity level “error”, and a new “warning” level will be added to represent less severe issues. The effects of the warning recorded on a test will not cause a failure but will be included in the test results for inspection after the run is complete. + +## Motivation + +Currently, when an issue arises during a test, the only possible outcome is to mark the test as failed. This presents a challenge for users who want a deeper insight into the events occurring within their tests. By introducing a dedicated mechanism to record issues that do not cause test failure, users can more effectively inspect and diagnose problems at runtime and review results afterward. This enhancement provides greater flexibility and clarity in test reporting, ultimately improving the debugging and analysis process. + +### Use Cases + +- Warning about a Percentage Discrepancy in Image Comparison: + - Scenario: When comparing two images to assess their similarity, a warning can be triggered if there's a 95% pixel match, while a test failure is set at a 90% similarity threshold. + - Reason: In practices like snapshot testing, minor changes (such as a timestamp) might cause a discrepancy. Setting a 90% match as a pass ensures test integrity. However, a warning at 95% alerts testers that, although the images aren't identical, the test has passed, which may warrant further investigation. +- Warning for Duplicate Argument Inputs in Tests: + - Scenario: In a test library, issue a warning if a user inputs the same argument twice, rather than flagging an error. + - Reason: Although passing the same argument twice might not be typical, some users may have valid reasons for doing so. Thus, a warning suffices, allowing flexibility without compromising the test's execution. +- Warning for Recoverable Unexpected Events: + - Scenario: During an integration test where data is retrieved from a server, a warning can be issued if the primary server is down, prompting a switch to an alternative server. Usually mocking is the solution for this but may not test everything needed for an integration test. + - Reason: Since server downtime might happen and can be beyond the tester's control, issuing a warning rather than a failure helps in debugging and understanding potential issues without impacting the test's overall success. +- Warning for a retry during setup for a test: + - Scenario: During test setup part of your code may be configured to retry, it would be nice to notify in the results that a retry happened + - Reason: This makes sense to be a warning and not a failure because if the retry succeeds the test may still verify the code correctly + +## Proposed solution + +We propose introducing a new property on `Issue` in Swift Testing called `severity`, that represents if an issue is a `warning` or an `error`. +The default Issue severity will still be `error` and users can set the severity when they record an issue. + +Test authors will be able to inspect if the issue is a failing issue and will be able to check the severity. + +## Detailed design + +### Severity Enum + +We introduce a Severity enum to categorize issues detected during testing. This enum is crucial for distinguishing between different levels of test issues and is defined as follows: + +The `Severity` enum: + +```swift +extension Issue { + // ... + + public enum Severity: Codable, Comparable, CustomStringConvertible, Sendable { + /// The severity level for an issue which should be noted but is not + /// necessarily an error. + /// + /// An issue with warning severity does not cause the test it's associated + /// with to be marked as a failure, but is noted in the results. + case warning + + /// The severity level for an issue which represents an error in a test. + /// + /// An issue with error severity causes the test it's associated with to be + /// marked as a failure. + case error + } +} +``` + +### Recording Non-Failing Issues + +To enable test authors to log non-failing issues without affecting test results, we provide a method for recording such issues: + +```swift +Issue.record("My comment", severity: .warning) +``` + +Here is the `Issue.record` method definition with severity as a parameter. + +```swift +extension Issue { + // ... + + /// Record an issue when a running test and an issue occurs. + /// + /// - Parameters: + /// - comment: A comment describing the expectation. + /// - severity: The severity level of the issue. This factor impacts whether the issue constitutes a failure. + /// - sourceLocation: The source location to which the issue should be + /// attributed. + /// + /// - Returns: The issue that was recorded. + /// + /// Use this function if, while running a test, an issue occurs that cannot be + /// represented as an expectation (using the ``expect(_:_:sourceLocation:)`` + /// or ``require(_:_:sourceLocation:)-5l63q`` macros.) + @discardableResult public static func record( + _ comment: Comment? = nil, + severity: Severity = .error, + sourceLocation: SourceLocation = #_sourceLocation + ) -> Self +} +``` + +### Issue Type Enhancements + +The Issue type is enhanced with two new properties to better handle and report issues: + +- `severity`: This property allows access to the specific severity level of an issue, enabling more precise handling of test results. + +```swift +extension Issue { + // ... + + /// The severity of the issue. + public var severity: Severity { get set } +} + +``` + +- `isFailure`: A boolean computed property to determine if an issue results in a test failure, thereby helping in result aggregation and reporting. + +```swift +extension Issue { + // ... + + /// Whether or not this issue should cause the test it's associated with to be + /// considered a failure. + /// + /// The value of this property is `true` for issues which have a severity level of + /// ``Issue/Severity/error`` or greater and are not known issues via + /// ``withKnownIssue(_:isIntermittent:sourceLocation:_:when:matching:)``. + /// Otherwise, the value of this property is `false.` + /// + /// Use this property to determine if an issue should be considered a failure, instead of + /// directly comparing the value of the ``severity`` property. + public var isFailure: Bool { get } +} +``` + +Example usage of `severity` and `isFailure`: + +```swift +withKnownIssue { + // ... +} matching: { issue in + issue.isFailure || issue.severity > .warning +} +``` + +For more details on `Issue`, refer to the [Issue Documentation](https://developer.apple.com/documentation/testing/issue). + +This revision aims to clarify the functionality and usage of the `Severity` enum and `Issue` properties while maintaining consistency with the existing Swift API standards. + +## Source compatibility + +The aspect of this proposal which adds a new `severity:` parameter to the +`Issue.record` function introduces the possibility of a source breakage for any +clients who are capturing a reference to the function. Existing code could break +despite the fact that the new parameter specifies a default value of `.error`. +Here's a contrived example: + +``` +// ❌ Source breakage due to new `Issue.Severity` parameter +let myRecordFunc: (Comment?, SourceLocation) -> Issue = Issue.record +``` + +To avoid source breakage, we will maintain the existing overload and preserve +its signature, but mark it deprecated, disfavored, and hidden from documentation: + +```swift +extension Issue { + // ... + + @available(*, deprecated, message: "Use record(_:severity:sourceLocation:) instead.") + @_disfavoredOverload + @_documentation(visibility: private) + @discardableResult public static func record( + _ comment: Comment? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) -> Self +} +``` + +## Integration with supporting tools + +### Event stream + +Issue severity will be in the event stream output when a `issueRecorded` event occurs. This will be a breaking change because some tools may assume that all `issueRecorded` events are failing. Due to this we will be bumping the event stream version and v1 will maintain it's behavior and not output any events for non failing issues. We will also be adding `isFailure` to the issue so that clients will know if the issue should be treated as a failure. `isFailure` is a computed property. + +The JSON event stream ABI will be amended correspondingly: + +```diff + ::= { + "isKnown": , ; is this a known issue or not? ++ "severity": , ; the severity of the issue ++ "isFailure": , ; if the issue is a failing issue + ["sourceLocation": ,] ; where the issue occurred, if known + } +``` + +Example of an `issueRecorded` event in the json output: + +``` +{"kind":"event","payload":{"instant":{"absolute":302928.100968,"since1970":1751305230.364087},"issue":{"_backtrace":[{"address":4437724864},{"address":4427566652},{"address":4437724280},{"address":4438635916},{"address":4438635660},{"address":4440823880},{"address":4437933556},{"address":4438865080},{"address":4438884348},{"address":11151272236},{"address":4438862360},{"address":4438940324},{"address":4437817340},{"address":4438134208},{"address":4438132164},{"address":4438635048},{"address":4440836660},{"address":4440835536},{"address":4440834989},{"address":4438937653},{"address":4438963225},{"address":4438895773},{"address":4438896161},{"address":4438891517},{"address":4438937117},{"address":4438962637},{"address":4439236617},{"address":4438936181},{"address":4438962165},{"address":4438639149},{"address":4438935045},{"address":4438935513},{"address":11151270653},{"address":11151269797},{"address":4438738225},{"address":4438872065},{"address":4438933417},{"address":4438930265},{"address":4438930849},{"address":4438909741},{"address":4438965489},{"address":11151508333}],"_severity":"error","isFailure":true, "isKnown":false,"sourceLocation":{"_filePath":"\/Users\/swift-testing\/Tests\/TestingTests\/EntryPointTests.swift","column":23,"fileID":"TestingTests\/EntryPointTests.swift","line":46}},"kind":"issueRecorded","messages":[{"symbol":"fail","text":"Issue recorded"},{"symbol":"details","text":"Unexpected issue Issue recorded (warning) was recorded."}],"testID":"TestingTests.EntryPointTests\/warningIssues()\/EntryPointTests.swift:33:4"},"version":0} +``` + +### Console output + +When there is an issue recorded with severity warning, such as using the following code: + +```swift +Issue.record("My comment", severity: .warning) +``` + +the console output will look like the following: + +``` +◇ Test "All elements of two ranges are equal" started. +� Test "All elements of two ranges are equal" recorded a warning at ZipTests.swift:32:17: Issue recorded +↳ My comment +✔ Test "All elements of two ranges are equal" passed after 0.001 seconds with 1 warning. +``` + +## Alternatives considered + +- Separate Issue Creation and Recording: We considered providing a mechanism to create issues independently before recording them, rather than passing the issue details directly to the `record` method. This approach was ultimately set aside in favor of simplicity and directness in usage. + +- Naming of `isFailure` vs. `isFailing`: We evaluated whether to name the property `isFailing` instead of `isFailure`. The decision to use `isFailure` was made to adhere to naming conventions and ensure clarity and consistency within the API. + +- Severity-Only Checking: We deliberated not exposing `isFailure` and relying solely on `severity` checks. However, this was rejected because it would require test authors to overhaul their code should we introduce additional severity levels in the future. By providing `isFailure`, we offer a straightforward way to determine test outcome impact, complementing the severity feature. +- Naming `Severity.error` `Severity.failure` instead because this will always be a failing issue and test authors often think of test failures. Error and warning match build naming conventions and XCTest severity naming convention. + +## Future directions + +- In the future I could see the warnings being able to be promoted to errors in order to run with a more strict testing configuration + +- In the future I could see adding other levels of severity such as Info and Debug for users to create issues with other information. + +## Acknowledgments + +Thanks to Stuart Montgomery for creating and implementing severity in Swift Testing. + +Thanks to Joel Middendorf, Dorothy Fu, Brian Croom, and Jonathan Grynspan for feedback on severity along the way. diff --git a/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md b/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md new file mode 100644 index 0000000000..b4c95d641a --- /dev/null +++ b/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md @@ -0,0 +1,416 @@ +# Image attachments in Swift Testing (Apple platforms) + +* Proposal: [ST-0014](0014-image-attachments-in-swift-testing-apple-platforms.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Maarten Engels](https://github.com/maartene/) +* Status: **Accepted** +* Bug: rdar://154869058 +* Implementation: [swiftlang/swift-testing#827](https://github.com/swiftlang/swift-testing/pull/827), _et al._ +* Review: ([pitch](https://forums.swift.org/t/pitch-image-attachments-in-swift-testing/80867)) ([review](https://forums.swift.org/t/st-0014-image-attachments-in-swift-testing-apple-platforms/81507)) ([acceptance](https://forums.swift.org/t/accepted-st-0014-image-attachments-in-swift-testing-apple-platforms/81868)) + +## Introduction + +We introduced the ability to add attachments to tests in Swift 6.2. This +proposal augments that feature to support attaching images on Apple platforms. + +## Motivation + +It is frequently useful to be able to attach images to tests for engineers to +review, e.g. if a UI element is not being drawn correctly. If something doesn't +render correctly in a CI environment, for instance, it is very useful to test +authors to be able to download the failed rendering and examine it at-desk. + +Today, Swift Testing offers support for **attachments** which allow a test +author to save arbitrary files created during a test run. However, if those +files are images, the test author must write their own code to encode them as +(for example) JPEG or PNG files before they can be attached to a test. + +## Proposed solution + +We propose adding support for images as a category of Swift type that can be +encoded using standard graphics formats such as JPEG or PNG. Image serialization +is beyond the purview of the testing library, so Swift Testing will defer to the +operating system to provide the relevant functionality. As such, this proposal +covers support for **Apple platforms** only. Support for other platforms such as +Windows is discussed in the **Future directions** section of this proposal. + +## Detailed design + +A new protocol is introduced for Apple platforms: + +```swift +/// A protocol describing images that can be converted to instances of +/// ``Testing/Attachment``. +/// +/// Instances of types conforming to this protocol do not themselves conform to +/// ``Testing/Attachable``. Instead, the testing library provides additional +/// initializers on ``Testing/Attachment`` that take instances of such types and +/// handle converting them to image data when needed. +/// +/// The following system-provided image types conform to this protocol and can +/// be attached to a test: +/// +/// - [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage) +/// - [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage) +/// - [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) +/// (macOS) +/// - [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) +/// (iOS, watchOS, tvOS, visionOS, and Mac Catalyst) +/// +/// You do not generally need to add your own conformances to this protocol. If +/// you have an image in another format that needs to be attached to a test, +/// first convert it to an instance of one of the types above. +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +public protocol AttachableAsCGImage { + /// An instance of `CGImage` representing this image. + /// + /// - Throws: Any error that prevents the creation of an image. + var attachableCGImage: CGImage { get throws } +} +``` + +And conformances are provided for the following types: + +- [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage) +- [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage) +- [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) + (macOS) +- [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) + (iOS, watchOS, tvOS, visionOS, and Mac Catalyst) + +The implementation of `CGImage.attachableCGImage` simply returns `self`, while +the other implementations extract an underlying `CGImage` instance if available +or render one on-demand. + +> [!NOTE] +> The list of conforming types may be extended in the future. The Testing +> Workgroup will determine if additional Swift Evolution reviews are needed. + +### Attaching a conforming image + +New overloads of `Attachment.init()` and `Attachment.record()` are provided: + +```swift +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +extension Attachment { + /// Initialize an instance of this type that encloses the given image. + /// + /// - Parameters: + /// - attachableValue: The value that will be attached to the output of + /// the test run. + /// - preferredName: The preferred name of the attachment when writing it + /// to a test report or to disk. If `nil`, the testing library attempts + /// to derive a reasonable filename for the attached value. + /// - imageFormat: The image format with which to encode `attachableValue`. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + /// + /// The following system-provided image types conform to the + /// ``AttachableAsCGImage`` protocol and can be attached to a test: + /// + /// - [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage) + /// - [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage) + /// - [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) + /// (macOS) + /// - [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) + /// (iOS, watchOS, tvOS, visionOS, and Mac Catalyst) + /// + /// The testing library uses the image format specified by `imageFormat`. Pass + /// `nil` to let the testing library decide which image format to use. If you + /// pass `nil`, then the image format that the testing library uses depends on + /// the path extension you specify in `preferredName`, if any. If you do not + /// specify a path extension, or if the path extension you specify doesn't + /// correspond to an image format the operating system knows how to write, the + /// testing library selects an appropriate image format for you. + public init( + _ attachableValue: T, + named preferredName: String? = nil, + as imageFormat: AttachableImageFormat? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) where AttachableValue == _AttachableImageWrapper + + /// Attach an image to the current test. + /// + /// - Parameters: + /// - image: The value to attach. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the testing library attempts to + /// derive a reasonable filename for the attached value. + /// - imageFormat: The image format with which to encode `attachableValue`. + /// - sourceLocation: The source location of the call to this function. + /// + /// This function creates a new instance of ``Attachment`` wrapping `image` + /// and immediately attaches it to the current test. + /// + /// The following system-provided image types conform to the + /// ``AttachableAsCGImage`` protocol and can be attached to a test: + /// + /// - [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage) + /// - [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage) + /// - [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) + /// (macOS) + /// - [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) + /// (iOS, watchOS, tvOS, visionOS, and Mac Catalyst) + /// + /// The testing library uses the image format specified by `imageFormat`. Pass + /// `nil` to let the testing library decide which image format to use. If you + /// pass `nil`, then the image format that the testing library uses depends on + /// the path extension you specify in `preferredName`, if any. If you do not + /// specify a path extension, or if the path extension you specify doesn't + /// correspond to an image format the operating system knows how to write, the + /// testing library selects an appropriate image format for you. + public static func record( + _ image: T, + named preferredName: String? = nil, + as imageFormat: AttachableImageFormat? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) where AttachableValue == _AttachableImageWrapper +} +``` + +> [!NOTE] +> `_AttachableImageWrapper` is an implementation detail required by Swift's +> generic type system and is not itself part of this proposal. For completeness, +> its public interface is: +> +> ```swift +> @available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +> public struct _AttachableImageWrapper: Sendable, AttachableWrapper where Image: AttachableAsCGImage { +> public var wrappedValue: Image { get } +> } +> ``` + +### Specifying image formats + +A test author can specify the image format to use with `AttachableImageFormat`. +This type abstractly represents the destination image format and, where +applicable, encoding quality: + +```swift +/// A type describing image formats supported by the system that can be used +/// when attaching an image to a test. +/// +/// When you attach an image to a test, you can pass an instance of this type to +/// ``Attachment/record(_:named:as:sourceLocation:)`` so that the testing +/// library knows the image format you'd like to use. If you don't pass an +/// instance of this type, the testing library infers which format to use based +/// on the attachment's preferred name. +/// +/// The PNG and JPEG image formats are always supported. The set of additional +/// supported image formats is platform-specific: +/// +/// - On Apple platforms, you can use [`CGImageDestinationCopyTypeIdentifiers()`](https://developer.apple.com/documentation/imageio/cgimagedestinationcopytypeidentifiers()) +/// from the [Image I/O framework](https://developer.apple.com/documentation/imageio) +/// to determine which formats are supported. +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +public struct AttachableImageFormat: Sendable { + /// The encoding quality to use for this image format. + /// + /// The meaning of the value is format-specific with `0.0` being the lowest + /// supported encoding quality and `1.0` being the highest supported encoding + /// quality. The value of this property is ignored for image formats that do + /// not support variable encoding quality. + public var encodingQuality: Float { get } +} +``` + +Conveniences for the PNG and JPEG formats are provided as they are very widely +used and supported across almost all modern platforms, Web browsers, etc.: + +```swift +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +extension AttachableImageFormat { + /// The PNG image format. + public static var png: Self { get } + + /// The JPEG image format with maximum encoding quality. + public static var jpeg: Self { get } + + /// The JPEG image format. + /// + /// - Parameters: + /// - encodingQuality: The encoding quality to use when serializing an + /// image. A value of `0.0` indicates the lowest supported encoding + /// quality and a value of `1.0` indicates the highest supported encoding + /// quality. + /// + /// - Returns: An instance of this type representing the JPEG image format + /// with the specified encoding quality. + public static func jpeg(withEncodingQuality encodingQuality: Float) -> Self +} +``` + +For instance, to save an image in the JPEG format with 50% image quality, you +can use `.jpeg(withEncodingQuality: 0.5)`. + +On Apple platforms, a convenience initializer that takes an instance of `UTType` +is also provided and lets you select any format supported by the underlying +Image I/O framework: + +```swift +@available(macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0, *) +extension AttachableImageFormat { + /// The content type corresponding to this image format. + /// + /// The value of this property always conforms to [`UTType.image`](https://developer.apple.com/documentation/uniformtypeidentifiers/uttype-swift.struct/image). + public var contentType: UTType { get } + + /// Initialize an instance of this type with the given content type and + /// encoding quality. + /// + /// - Parameters: + /// - contentType: The image format to use when encoding images. + /// - encodingQuality: The encoding quality to use when encoding images. For + /// the lowest supported quality, pass `0.0`. For the highest supported + /// quality, pass `1.0`. + /// + /// If the target image format does not support variable-quality encoding, + /// the value of the `encodingQuality` argument is ignored. + /// + /// If `contentType` does not conform to [`UTType.image`](https://developer.apple.com/documentation/uniformtypeidentifiers/uttype-swift.struct/image), + /// the result is undefined. + public init(_ contentType: UTType, encodingQuality: Float = 1.0) +} +``` + +### Example usage + +A developer may then easily attach an image to a test by calling +`Attachment.record()` and passing the image of interest. For example, to attach +a rendering of a SwiftUI view as a PNG file: + +```swift +import Testing +import UIKit +import SwiftUI + +@MainActor @Test func `attaching a SwiftUI view as an image`() throws { + let myView: some View = ... + let image = try #require(ImageRenderer(content: myView).uiImage) + Attachment.record(image, named: "my view", as: .png) + // OR: Attachment.record(image, named: "my view.png") +} +``` + +## Source compatibility + +This change is additive only. + +## Integration with supporting tools + +None needed. + +## Future directions + +- Adding support for [`SwiftUI.Image`](https://developer.apple.com/documentation/swiftui/image) + and/or [`SwiftUI.GraphicsContext.ResolvedImage`](https://developer.apple.com/documentation/swiftui/graphicscontext/resolvedimage). + These types do not directly wrap an instance of `CGImage`. + + Since `SwiftUI.Image` conforms to [`SwiftUI.View`](https://developer.apple.com/documentation/swiftui/view), + it is possible to convert an instance of that type to an instance of `CGImage` + using [`SwiftUI.ImageRenderer`](https://developer.apple.com/documentation/swiftui/imagerenderer). + This approach is generalizable to all `SwiftUI.View`-cnforming types, and the + correct approach here may be to provide an `_AttachableViewWrapper` + type similar to the described `_AttachableImageWrapper` type. + +- Adding support for Windows image types. Windows has several generations of + imaging libraries: + + - Graphics Device Interface (GDI), which shipped with the original Windows in + 1985; + - GDI+, which was introduced with Windows XP in 2001; + - Windows Imaging Component (WIC) with Windows Vista in 2006; and + - Direct2D with Windows 7 in 2008. + + Of these libraries, only the original GDI provides a C interface that can be + directly referenced from Swift. The GDI+ interface is written in C++ and the + WIC and Direct2D interfaces are built on top of COM (a C++ abstraction layer.) + This reliance on C++ poses challenges for Swift Testing. Swift/C++ interop is + still a young technology and is not yet able to provide abstractions for + virtual C++ classes. + + None of these Windows' libraries are source compatible with Apple's Core + Graphics API, so support for any of them will require a different protocol. As + of this writing, [an experimental](https://github.com/swiftlang/swift-testing/pull/1245) + GDI- and (partially) GDI+-compatible protocol is available in Swift Testing + that allows a test author to attach an image represented by an `HBITMAP` or + `HICON` instance. Further work will be needed to make this experimental + Windows support usable with the newer libraries' image types. + +- Adding support for X11-compatible image types such as Qt's [`QImage`](https://doc.qt.io/qt-6/qimage.html) + or GTK's [`GdkPixbuf`](https://docs.gtk.org/gdk-pixbuf/class.Pixbuf.html). + We're also interested in implementing something here, but GUI-level libraries + aren't guaranteed to be present on Linux systems, so we cannot rely on their + headers or modules being accessible while building the Swift toolchain. It may + be appropriate to roll such functionality into a hypothetical `swift-x11`, + `swift-wayland`, `swift-qt`, `swift-gtk`, etc. package if one is ever created. + +- Adding support for Android's [`android.graphics.Bitmap`](https://developer.android.com/reference/android/graphics/Bitmap) + type. The Android NDK includes the [`AndroidBitmap_compress()`](https://developer.android.com/ndk/reference/group/bitmap#androidbitmap_compress) + function, but proper support for attaching an Android `Bitmap` may require a + dependency on [`swift-java`](https://github.com/swiftlang/swift-java) in some + form. Going forward, we hope to work with the new [Android Workgroup](https://www.swift.org/android-workgroup/) + to enhance Swift Testing's Android support. + +- Adding support for rendering to a PDF instead of an image. While technically + feasible using [existing](https://developer.apple.com/documentation/coregraphics/cgcontext/init(consumer:mediabox:_:)) + Core Graphics API, we haven't identified sufficient demand for this + functionality. + +## Alternatives considered + +- Doing nothing. Developers would need to write their own image conversion code. + Since this is a very common operation, it makes sense to incorporate it into + Swift Testing directly. + +- Making `CGImage` etc. conform directly to `Attachable`. Doing so would + prevent us from including sidecar data such as the desired `UTType` or + encoding quality as these types do not provide storage for that information. + As well, `NSImage` does not conform to `Sendable` and would be forced down a + code path that eagerly serializes it, which could pessimize its performance + once we introduce attachment lifetimes in a future proposal. + +- Designing a platform-agnostic solution. This would likely require adding a + dependency on an open-source image package such as [ImageMagick](https://github.com/ImageMagick/ImageMagick). + While we appreciate the value of such libraries and we want Swift Testing to + be as portable as possible, that would be a significant new dependency for the + testing library and the Swift toolchain at large. As well, we expect a typical + use case to involve an instance of `NSImage`, `CGImage`, etc. + +- Designing a solution that does not require `UTType` so as to support earlier + Apple platforms. The implementation is based on Apple's Image I/O framework + which requires a Uniform Type Identifier as input anyway, and the older + `CFString`-based interfaces we would need to use have been deprecated for + several years now. The `AttachableImageFormat` type allows us to abstract away + our platform-specific dependency on `UTType` so that, in the future, other + platforms can reuse `AttachableImageFormat` instead of implementing their own + equivalent solution. (As an example, the experimental Windows support + mentioned previously allows a developer to specify an image codec's `CLSID`.) + +- Designing a solution based around _drawing_ into a `CGContext` rather than + acquiring an instance of `CGImage`. If the proposed protocol looked like: + + ```swift + protocol AttachableByDrawing { + func draw(in context: CGContext, for attachment: Attachment) throws + } + ``` + + It would be easier to support alternative destination contexts (primarily PDF + contexts), but we would need to make a complete copy of an image in memory + before serializing it. If you start with an instance of `CGImage` or an object + that wraps an instance of `CGImage`, you can pass it directly to Image I/O. + +- Including convenience getters for additional image formats in + `AttachableImageFormat`. The set of formats we provide up-front support for is + intentionally small and limited to formats that are universally supported by + the various graphics libraries in use today. If we provided a larger set of + formats that are supported on Apple's platforms, developers may run into + difficulties porting their test code to platforms that _don't_ support those + additional formats. + +## Acknowledgments + +Thanks to Apple's testing teams and to the Testing Workgroup for their support +and advice on this project. diff --git a/proposals/testing/0015-image-attachments-in-swift-testing-windows.md b/proposals/testing/0015-image-attachments-in-swift-testing-windows.md new file mode 100644 index 0000000000..1b26056068 --- /dev/null +++ b/proposals/testing/0015-image-attachments-in-swift-testing-windows.md @@ -0,0 +1,393 @@ +# Image attachments in Swift Testing (Windows) + +* Proposal: [ST-0015](0015-image-attachments-in-swift-testing-windows.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Review Manager: [Stuart Montgomery](https://github.com/stmontgomery) +- Status: **Implemented (Swift 6.3)** +* Implementation: [swiftlang/swift-testing#1245](https://github.com/swiftlang/swift-testing/pull/1245), [swiftlang/swift-testing#1254](https://github.com/swiftlang/swift-testing/pull/1254), [swiftlang/swift-testing#1333](https://github.com/swiftlang/swift-testing/pull/1333), _et al_. +* Review: ([pitch](https://forums.swift.org/t/pitch-image-attachments-in-swift-testing-windows/81871)) ([review](https://forums.swift.org/t/st-0015-image-attachments-in-swift-testing-windows/82241)) ([acceptance](https://forums.swift.org/t/accepted-st-0015-image-attachments-in-swift-testing-windows/82575)) + +## Introduction + +In [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md), +we added to Swift Testing the ability to attach images (of types `CGImage`, +`NSImage`, `UIImage`, and `CIImage`) on Apple platforms. This proposal builds on +that one to add support for attaching images on Windows. + +## Motivation + +It is frequently useful to be able to attach images to tests for engineers to +review, e.g. if a UI element is not being drawn correctly. If something doesn't +render correctly in a CI environment, for instance, it is very useful to test +authors to be able to download the failed rendering and examine it at-desk. + +In [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#integration-with-supporting-tools), +we introduced the ability to attach images to tests on Apple's platforms. Swift +Testing is a cross-platform testing library, so we should extend this +functionality to other platforms too. This proposal covers Windows in +particular. + +## Proposed solution + +We propose adding the ability to automatically encode images to standard +graphics formats such as JPEG or PNG using Windows' built-in Windows Image +Component library, similar to how we added support on Apple platforms using Core +Graphics. + +## Detailed design + +### Some background about Windows' image types + +Windows has several generations of API for representing and encoding images. The +earliest Windows API of interest to this proposal is the Graphics Device +Interface (GDI) which dates back to the earliest versions of Windows. Image +types in GDI that are of interest to us are `HBITMAP` and `HICON`, which are +_handles_ (pointers-to-pointers) and which are not reference-counted. Both types +are projected into Swift as typealiases of `UnsafeMutablePointer`. + +Windows' latest[^direct2d] graphics API is the Windows Imaging Component (WIC) +which uses types based on the Component Object Model (COM). COM types (including +those implemented in WIC) are C++ classes that inherit from `IUnknown`. + +[^direct2d]: There is an even newer API in this area, Direct2D, but it is beyond + the scope of this proposal. A developer who has an instance of e.g. + `ID2D1Bitmap` can use WIC API to convert it to a WIC bitmap source before + attaching it to a test. + +`IUnknown` is conceptually similar to Cocoa's `NSObject` class in that it +provides basic reference-counting and reflection functionality. As of this +proposal, the Swift C/C++ importer is not aware of COM classes and does not +project them into Swift as reference-counted classes. Rather, they are projected +as `UnsafeMutablePointer`, and developers who use them must manually manage +their reference counts and must use `QueryInterface()` to cast them to other COM +classes. + +In short: the types we need to support are all specializations of +`UnsafeMutablePointer`, but we do not need to support all specializations of +`UnsafeMutablePointer` unconditionally. + +### Defining a new protocol for Windows image attachments + +A new protocol is introduced for Windows, similar to the `AttachableAsCGImage` +protocol we introduced for Apple's platforms: + +```swift +/// A protocol describing images that can be converted to instances of +/// [`Attachment`](https://developer.apple.com/documentation/testing/attachment). +/// +/// Instances of types conforming to this protocol do not themselves conform to +/// [`Attachable`](https://developer.apple.com/documentation/testing/attachable). +/// Instead, the testing library provides additional initializers on [`Attachment`](https://developer.apple.com/documentation/testing/attachment) +/// that take instances of such types and handle converting them to image data when needed. +/// +/// You can attach instances of the following system-provided image types to a +/// test: +/// +/// | Platform | Supported Types | +/// |-|-| +/// | macOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) | +/// | iOS, watchOS, tvOS, and visionOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) | +/// | Windows | [`HBITMAP`](https://learn.microsoft.com/en-us/windows/win32/gdi/bitmaps), [`HICON`](https://learn.microsoft.com/en-us/windows/win32/menurc/icons), [`IWICBitmapSource`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapsource) (including its subclasses declared by Windows Imaging Component) | +/// +/// You do not generally need to add your own conformances to this protocol. If +/// you have an image in another format that needs to be attached to a test, +/// first convert it to an instance of one of the types above. +public protocol AttachableAsIWICBitmapSource: SendableMetatype { + /// Create a WIC bitmap source representing an instance of this type. + /// + /// - Returns: A pointer to a new WIC bitmap source representing this image. + /// The caller is responsible for releasing this image when done with it. + /// + /// - Throws: Any error that prevented the creation of the WIC bitmap source. + func copyAttachableIWICBitmapSource() throws -> UnsafeMutablePointer +} +``` + +Conformance to this protocol is added to `UnsafeMutablePointer` when its +`Pointee` type is one of the following types: + +- [`HBITMAP.Pointee`](https://learn.microsoft.com/en-us/windows/win32/gdi/bitmaps) +- [`HICON.Pointee`](https://learn.microsoft.com/en-us/windows/win32/menurc/icons) +- [`IWICBitmapSource`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapsource) + (including its subclasses declared by Windows Imaging Component) + +> [!NOTE] +> The list of conforming types may be extended in the future. The Testing +> Workgroup will determine if additional Swift Evolution reviews are needed. + +A type in Swift can only conform to a protocol with **one** set of constraints, +so we need a helper protocol in order to make `UnsafeMutablePointer` +conditionally conform for all of the above types. This protocol must be `public` +so that Swift Testing can refer to it in API, but it is an implementation detail +and not part of this proposal: + +```swift +public protocol _AttachableByAddressAsIWICBitmapSource {} + +extension HBITMAP.Pointee: _AttachableByAddressAsIWICBitmapSource {} +extension HICON.Pointee: _AttachableByAddressAsIWICBitmapSource {} +extension IWICBitmapSource: _AttachableByAddressAsIWICBitmapSource {} + +extension UnsafeMutablePointer: AttachableAsIWICBitmapSource + where Pointee: _AttachableByAddressAsIWICBitmapSource {} +``` + +See the **Future directions** section (specifically the point about COM and C++ +interop) for more information on why the helper protocol is excluded from this +proposal. + +### Attaching a conforming image + +New overloads of `Attachment.init()` and `Attachment.record()` are provided: + +```swift +extension Attachment { + /// Initialize an instance of this type that encloses the given image. + /// + /// - Parameters: + /// - image: A pointer to the value that will be attached to the output of + /// the test run. + /// - preferredName: The preferred name of the attachment when writing it + /// to a test report or to disk. If `nil`, the testing library attempts + /// to derive a reasonable filename for the attached value. + /// - imageFormat: The image format with which to encode `image`. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + /// + /// You can attach instances of the following system-provided image types to a + /// test: + /// + /// | Platform | Supported Types | + /// |-|-| + /// | macOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) | + /// | iOS, watchOS, tvOS, and visionOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) | + /// | Windows | [`HBITMAP`](https://learn.microsoft.com/en-us/windows/win32/gdi/bitmaps), [`HICON`](https://learn.microsoft.com/en-us/windows/win32/menurc/icons), [`IWICBitmapSource`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapsource) (including its subclasses declared by Windows Imaging Component) | + /// + /// The testing library uses the image format specified by `imageFormat`. Pass + /// `nil` to let the testing library decide which image format to use. If you + /// pass `nil`, then the image format that the testing library uses depends on + /// the path extension you specify in `preferredName`, if any. If you do not + /// specify a path extension, or if the path extension you specify doesn't + /// correspond to an image format the operating system knows how to write, the + /// testing library selects an appropriate image format for you. + public init( + _ image: T, + named preferredName: String? = nil, + as imageFormat: AttachableImageFormat? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) where T: AttachableAsIWICBitmapSource, AttachableValue == _AttachableImageWrapper + + /// Attach an image to the current test. + /// + /// - Parameters: + /// - image: The value to attach. + /// - preferredName: The preferred name of the attachment when writing it + /// to a test report or to disk. If `nil`, the testing library attempts + /// to derive a reasonable filename for the attached value. + /// - imageFormat: The image format with which to encode `image`. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + /// + /// This function creates a new instance of ``Attachment`` wrapping `image` + /// and immediately attaches it to the current test. You can attach instances + /// of the following system-provided image types to a test: + /// + /// | Platform | Supported Types | + /// |-|-| + /// | macOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`NSImage`](https://developer.apple.com/documentation/appkit/nsimage) | + /// | iOS, watchOS, tvOS, and visionOS | [`CGImage`](https://developer.apple.com/documentation/coregraphics/cgimage), [`CIImage`](https://developer.apple.com/documentation/coreimage/ciimage), [`UIImage`](https://developer.apple.com/documentation/uikit/uiimage) | + /// | Windows | [`HBITMAP`](https://learn.microsoft.com/en-us/windows/win32/gdi/bitmaps), [`HICON`](https://learn.microsoft.com/en-us/windows/win32/menurc/icons), [`IWICBitmapSource`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapsource) (including its subclasses declared by Windows Imaging Component) | + /// + /// The testing library uses the image format specified by `imageFormat`. Pass + /// `nil` to let the testing library decide which image format to use. If you + /// pass `nil`, then the image format that the testing library uses depends on + /// the path extension you specify in `preferredName`, if any. If you do not + /// specify a path extension, or if the path extension you specify doesn't + /// correspond to an image format the operating system knows how to write, the + /// testing library selects an appropriate image format for you. + public static func record( + _ image: T, + named preferredName: String? = nil, + as imageFormat: AttachableImageFormat? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) where T: AttachableAsIWICBitmapSource, AttachableValue == _AttachableImageWrapper +} +``` + +> [!NOTE] +> `_AttachableImageWrapper` was described in [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#attaching-a-conforming-image). +> The only difference on Windows is that its associated `Image` type is +> constrained to `AttachableAsIWICBitmapSource` instead of `AttachableAsCGImage`. + +### Specifying image formats + +As on Apple platforms, a test author can specify the image format to use with +`AttachableImageFormat`. See [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#specifying-image-formats) +for more information about that type. + +Windows does not use Uniform Type Identifiers, so those `AttachableImageFormat` +members that use `UTType` are not available here. Instead, Windows uses a +variety of COM classes that implement codecs for different image formats. +Conveniences over those COM classes' `CLSID` values are provided: + +```swift +extension AttachableImageFormat { + /// The `CLSID` value of the Windows Imaging Component (WIC) encoder class + /// that corresponds to this image format. + /// + /// For example, if this image format equals ``png``, the value of this + /// property equals [`CLSID_WICPngEncoder`](https://learn.microsoft.com/en-us/windows/win32/wic/-wic-guids-clsids#wic-guids-and-clsids). + public var encoderCLSID: CLSID { get } + + /// Construct an instance of this type with the `CLSID` value of a Windows + /// Imaging Component (WIC) encoder class and the desired encoding quality. + /// + /// - Parameters: + /// - encoderCLSID: The `CLSID` value of the Windows Imaging Component + /// encoder class to use when encoding images. + /// - encodingQuality: The encoding quality to use when encoding images. For + /// the lowest supported quality, pass `0.0`. For the highest supported + /// quality, pass `1.0`. + /// + /// If the target image encoder does not support variable-quality encoding, + /// the value of the `encodingQuality` argument is ignored. + /// + /// If `clsid` does not represent an image encoder class supported by WIC, the + /// result is undefined. For a list of image encoder classes supported by WIC, + /// see the documentation for the [`IWICBitmapEncoder`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapencoder) + /// class. + public init(encoderCLSID: CLSID, encodingQuality: Float = 1.0) +} +``` + +For convenience, an initializer is provided that takes a path extension and +tries to map it to the appropriate codec's `CLSID` value: + +```swift +extension AttachableImageFormat { + /// Construct an instance of this type with the given path extension and + /// encoding quality. + /// + /// - Parameters: + /// - pathExtension: A path extension corresponding to the image format to + /// use when encoding images. + /// - encodingQuality: The encoding quality to use when encoding images. For + /// the lowest supported quality, pass `0.0`. For the highest supported + /// quality, pass `1.0`. + /// + /// If the target image format does not support variable-quality encoding, + /// the value of the `encodingQuality` argument is ignored. + /// + /// If `pathExtension` does not correspond to a recognized image format, this + /// initializer returns `nil`: + /// + /// - On Apple platforms, the content type corresponding to `pathExtension` + /// must conform to [`UTType.image`](https://developer.apple.com/documentation/uniformtypeidentifiers/uttype-swift.struct/image). + /// - On Windows, there must be a corresponding subclass of [`IWICBitmapEncoder`](https://learn.microsoft.com/en-us/windows/win32/api/wincodec/nn-wincodec-iwicbitmapencoder) + /// registered with Windows Imaging Component. + public init?(pathExtension: String, encodingQuality: Float = 1.0) +} +``` + +For consistency, `init(pathExtension:encodingQuality:)` is provided on Apple +platforms too. (This is the only part of this proposal that affects platforms +other than Windows.) + +### Example usage + +A developer may then easily attach an image to a test by calling +`Attachment.record()` and passing the image of interest. For example, to attach +an icon to a test as a PNG file: + +```swift +import Testing +import WinSDK + +@MainActor @Test func `attaching an icon`() throws { + let hIcon: HICON = ... + defer { + DestroyIcon(hIcon) + } + Attachment.record(hIcon, named: "my icon", as: .png) + // OR: Attachment.record(hIcon, named: "my icon.png") +} +``` + +## Source compatibility + +This change is additive only. + +## Integration with supporting tools + +Tools that handle attachments created by Swift Testing will gain support for +this functionality automatically and do not need to make any changes. + +## Future directions + +- Adding support for projecting COM classes as foreign-reference-counted Swift + classes. The C++ interop team is interested in implementing this feature, but + it is beyond the scope of this proposal. **If this feature is implemented in + the future**, it will cause types like `IWICBitmapSource` to be projected + directly into Swift instead of as `UnsafeMutablePointer` specializations. This + would be a source-breaking change for Swift Testing, but it would make COM + classes much easier to use in Swift. + + In the context of this proposal, `IWICBitmapSource` would be able to directly + conform to `AttachableAsIWICBitmapSource` and we would no longer need the + `_AttachableByAddressAsIWICBitmapSource` helper protocol. The + `AttachableAsIWICBitmapSource` protocol's `copyAttachableIWICBitmapSource()` + requirement would likely change to a property (i.e. + `var attachableIWICBitmapSource: IWICBitmapSource { get throws }`) as it would + be able to participate in Swift's automatic reference counting. + + The Swift team is tracking COM interop with [swiftlang/swift#84056](https://github.com/swiftlang/swift/issues/84056). + +- Adding support for managed (.NET or C#) image types. Support for managed types + on Windows would first require a new Swift/.NET or Swift/C# interop feature + and is therefore beyond the scope of this proposal. + +- Adding support for WinRT image types. WinRT is a thin wrapper around COM and + has C++ and .NET projections, neither of which are readily accessible from + Swift. It may be possible to add support for WinRT image types if COM interop + is implemented. + +- Adding support for other platforms. See [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#future-directions) + for further discussion about supporting additional platforms. + +## Alternatives considered + +- Doing nothing. We have already added support for attaching images on Apple's + platforms, and Swift Testing is meant to be a cross-platform library, so we + should make a best effort to provide the same functionality on Windows and, + eventually, other platforms. + +- Using more Windows-/COM-like terminology and spelling, e.g. + `CloneAttachableBitmapSource()` instead of `copyAttachableIWICBitmapSource()`. + Swift API should follow Swift API guidelines, even when extending types and + calling functions implemented under other standards. + +- Making `IWICBitmapSource` conform directly to `Attachable`. As with `CGImage` + in [ST-0014](https://github.com/swiftlang/swift-evolution/blob/main/proposals/testing/0014-image-attachments-in-swift-testing-apple-platforms.md#alternatives-considered), + this would prevent us from including additional information (i.e. an instance + of `AttachableImageFormat`). Further, it would be difficult to correctly + manage the lifetime of Windows' 'image objects as they do not participate in + automatic reference counting. + +- Using the GDI+ type [`Gdiplus.Image`](https://learn.microsoft.com/en-us/windows/win32/api/gdiplusheaders/nl-gdiplusheaders-image) + as our currency type instead of `IWICBitmapSource`. This type is a C++ class + but is not a COM class, and so it is not projected into Swift except as + `OpaquePointer` which makes it unsafe to extend it with protocol conformances. + As well, GDI+ is a much older API than WIC and is not recommended by Microsoft + for new development. + +- Designing a platform-agnostic solution. This would likely require adding a + dependency on an open-source image package such as [ImageMagick](https://github.com/ImageMagick/ImageMagick). + Such a library would be a significant new dependency for the testing library + and the Swift toolchain at large. + +## Acknowledgments + +Thank you to @compnerd and the C++ interop team for their help with Windows and +the COM API. diff --git a/releases/swift-2_2.md b/releases/swift-2_2.md index 6a0990b4ff..509795952a 100644 --- a/releases/swift-2_2.md +++ b/releases/swift-2_2.md @@ -14,10 +14,10 @@ as practical with Swift 2.0. ## Evolution proposals included in Swift 2.2 -* [SE-0001: Allow (most) keywords as argument labels](https://github.com/apple/swift-evolution/blob/master/proposals/0001-keywords-as-argument-labels.md) -* [SE-0015: Tuple comparison operators](https://github.com/apple/swift-evolution/blob/master/proposals/0015-tuple-comparison-operators.md) -* [SE-0014: Constraining `AnySequence.init`](https://github.com/apple/swift-evolution/blob/master/proposals/0014-constrained-AnySequence.md) -* [SE-0011: Replace `typealias` keyword with `associatedtype` for associated type declarations](https://github.com/apple/swift-evolution/blob/master/proposals/0011-replace-typealias-associated.md) -* [SE-0021: Naming Functions with Argument Labels](https://github.com/apple/swift-evolution/blob/master/proposals/0021-generalized-naming.md) -* [SE-0022: Referencing the Objective-C selector of a method](https://github.com/apple/swift-evolution/blob/master/proposals/0022-objc-selectors.md) -* [SE-0020: Swift Language Version Build Configuration](https://github.com/apple/swift-evolution/blob/master/proposals/0020-if-swift-version.md) +* [SE-0001: Allow (most) keywords as argument labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0001-keywords-as-argument-labels.md) +* [SE-0015: Tuple comparison operators](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0015-tuple-comparison-operators.md) +* [SE-0014: Constraining `AnySequence.init`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0014-constrained-AnySequence.md) +* [SE-0011: Replace `typealias` keyword with `associatedtype` for associated type declarations](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0011-replace-typealias-associated.md) +* [SE-0021: Naming Functions with Argument Labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0021-generalized-naming.md) +* [SE-0022: Referencing the Objective-C selector of a method](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0022-objc-selectors.md) +* [SE-0020: Swift Language Version Build Configuration](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0020-if-swift-version.md) diff --git a/releases/swift-3_0.md b/releases/swift-3_0.md index a766f0df72..ece2b1d352 100644 --- a/releases/swift-3_0.md +++ b/releases/swift-3_0.md @@ -52,92 +52,92 @@ make everything feel nicer. ## Evolution proposals included in Swift 3.0 -* [SE-0002: Removing currying `func` declaration syntax](https://github.com/apple/swift-evolution/blob/master/proposals/0002-remove-currying.md) -* [SE-0003: Removing `var` from Function Parameters](https://github.com/apple/swift-evolution/blob/master/proposals/0003-remove-var-parameters.md) -* [SE-0004: Remove the `++` and `--` operators](https://github.com/apple/swift-evolution/blob/master/proposals/0004-remove-pre-post-inc-decrement.md) -* [SE-0005: Better Translation of Objective-C APIs Into Swift](https://github.com/apple/swift-evolution/blob/master/proposals/0005-objective-c-name-translation.md) -* [SE-0006: Apply API Guidelines to the Standard Library](https://github.com/apple/swift-evolution/blob/master/proposals/0006-apply-api-guidelines-to-the-standard-library.md) -* [SE-0007: Remove C-style for-loops with conditions and incrementers](https://github.com/apple/swift-evolution/blob/master/proposals/0007-remove-c-style-for-loops.md) -* [SE-0008: Add a Lazy flatMap for Sequences of Optionals](https://github.com/apple/swift-evolution/blob/master/proposals/0008-lazy-flatmap-for-optionals.md) -* [SE-0016: Adding initializers to Int and UInt to convert from UnsafePointer and UnsafeMutablePointer](https://github.com/apple/swift-evolution/blob/master/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md) -* [SE-0017: Change `Unmanaged` to use `UnsafePointer`](https://github.com/apple/swift-evolution/blob/master/proposals/0017-convert-unmanaged-to-use-unsafepointer.md) -* [SE-0019: Swift Testing](https://github.com/apple/swift-evolution/blob/master/proposals/0019-package-manager-testing.md) -* [SE-0023: API Design Guidelines](https://github.com/apple/swift-evolution/blob/master/proposals/0023-api-guidelines.md) -* [SE-0025: Scoped Access Level](https://github.com/apple/swift-evolution/blob/master/proposals/0025-scoped-access-level.md) -* [SE-0029: Remove implicit tuple splat behavior from function applications](https://github.com/apple/swift-evolution/blob/master/proposals/0029-remove-implicit-tuple-splat.md) -* [SE-0031: Adjusting inout Declarations for Type Decoration](https://github.com/apple/swift-evolution/blob/master/proposals/0031-adjusting-inout-declarations.md) -* [SE-0032: Add `first(where:)` method to `SequenceType`](https://github.com/apple/swift-evolution/blob/master/proposals/0032-sequencetype-find.md) -* [SE-0033: Import Objective-C Constants as Swift Types](https://github.com/apple/swift-evolution/blob/master/proposals/0033-import-objc-constants.md) -* [SE-0034: Disambiguating Line Control Statements from Debugging Identifiers](https://github.com/apple/swift-evolution/blob/master/proposals/0034-disambiguating-line.md) -* [SE-0035: Limiting `inout` capture to `@noescape` contexts](https://github.com/apple/swift-evolution/blob/master/proposals/0035-limit-inout-capture.md) -* [SE-0036: Requiring Leading Dot Prefixes for Enum Instance Member Implementations](https://github.com/apple/swift-evolution/blob/master/proposals/0036-enum-dot.md) -* [SE-0037: Clarify interaction between comments & operators](https://github.com/apple/swift-evolution/blob/master/proposals/0037-clarify-comments-and-operators.md) -* [SE-0038: Package Manager C Language Target Support](https://github.com/apple/swift-evolution/blob/master/proposals/0038-swiftpm-c-language-targets.md) -* [SE-0039: Modernizing Playground Literals](https://github.com/apple/swift-evolution/blob/master/proposals/0039-playgroundliterals.md) -* [SE-0040: Replacing Equal Signs with Colons For Attribute Arguments](https://github.com/apple/swift-evolution/blob/master/proposals/0040-attributecolons.md) -* [SE-0043: Declare variables in 'case' labels with multiple patterns](https://github.com/apple/swift-evolution/blob/master/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md) -* [SE-0044: Import as Member](https://github.com/apple/swift-evolution/blob/master/proposals/0044-import-as-member.md) -* [SE-0046: Establish consistent label behavior across all parameters including first labels](https://github.com/apple/swift-evolution/blob/master/proposals/0046-first-label.md) -* [SE-0047: Defaulting non-Void functions so they warn on unused results](https://github.com/apple/swift-evolution/blob/master/proposals/0047-nonvoid-warn.md) -* [SE-0048: Generic Type Aliases](https://github.com/apple/swift-evolution/blob/master/proposals/0048-generic-typealias.md) -* [SE-0049: Move @noescape and @autoclosure to be type attributes](https://github.com/apple/swift-evolution/blob/master/proposals/0049-noescape-autoclosure-type-attrs.md) -* [SE-0052: Change IteratorType post-nil guarantee](https://github.com/apple/swift-evolution/blob/master/proposals/0052-iterator-post-nil-guarantee.md) -* [SE-0053: Remove explicit use of `let` from Function Parameters](https://github.com/apple/swift-evolution/blob/master/proposals/0053-remove-let-from-function-parameters.md) -* [SE-0054: Abolish `ImplicitlyUnwrappedOptional` type](https://github.com/apple/swift-evolution/blob/master/proposals/0054-abolish-iuo.md) -* [SE-0055: Make unsafe pointer nullability explicit using Optional](https://github.com/apple/swift-evolution/blob/master/proposals/0055-optional-unsafe-pointers.md) -* [SE-0057: Importing Objective-C Lightweight Generics](https://github.com/apple/swift-evolution/blob/master/proposals/0057-importing-objc-generics.md) -* [SE-0059: Update API Naming Guidelines and Rewrite Set APIs Accordingly](https://github.com/apple/swift-evolution/blob/master/proposals/0059-updated-set-apis.md) -* [SE-0060: Enforcing order of defaulted parameters](https://github.com/apple/swift-evolution/blob/master/proposals/0060-defaulted-parameter-order.md) -* [SE-0061: Add Generic Result and Error Handling to autoreleasepool()](https://github.com/apple/swift-evolution/blob/master/proposals/0061-autoreleasepool-signature.md) -* [SE-0062: Referencing Objective-C key-paths](https://github.com/apple/swift-evolution/blob/master/proposals/0062-objc-keypaths.md) -* [SE-0063: SwiftPM System Module Search Paths](https://github.com/apple/swift-evolution/blob/master/proposals/0063-swiftpm-system-module-search-paths.md) -* [SE-0064: Referencing the Objective-C selector of property getters and setters](https://github.com/apple/swift-evolution/blob/master/proposals/0064-property-selectors.md) -* [SE-0065: A New Model For Collections and Indices](https://github.com/apple/swift-evolution/blob/master/proposals/0065-collections-move-indices.md) -* [SE-0066: Standardize function type argument syntax to require parentheses](https://github.com/apple/swift-evolution/blob/master/proposals/0066-standardize-function-type-syntax.md) -* [SE-0067: Enhanced Floating Point Protocols](https://github.com/apple/swift-evolution/blob/master/proposals/0067-floating-point-protocols.md) -* [SE-0069: Mutability and Foundation Value Types](https://github.com/apple/swift-evolution/blob/master/proposals/0069-swift-mutability-for-foundation.md) -* [SE-0070: Make Optional Requirements Objective-C-only](https://github.com/apple/swift-evolution/blob/master/proposals/0070-optional-requirements.md) -* [SE-0071: Allow (most) keywords in member references](https://github.com/apple/swift-evolution/blob/master/proposals/0071-member-keywords.md) -* [SE-0072: Fully eliminate implicit bridging conversions from Swift](https://github.com/apple/swift-evolution/blob/master/proposals/0072-eliminate-implicit-bridging-conversions.md) -* [SE-0076: Add overrides taking an UnsafePointer source to non-destructive copying methods on UnsafeMutablePointer](https://github.com/apple/swift-evolution/blob/master/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md) -* [SE-0077: Improved operator declarations](https://github.com/apple/swift-evolution/blob/master/proposals/0077-operator-precedence.md) -* [SE-0081: Move `where` clause to end of declaration](https://github.com/apple/swift-evolution/blob/master/proposals/0081-move-where-expression.md) -* [SE-0085: Package Manager Command Names](https://github.com/apple/swift-evolution/blob/master/proposals/0085-package-manager-command-name.md) -* [SE-0086: Drop NS Prefix in Swift Foundation](https://github.com/apple/swift-evolution/blob/master/proposals/0086-drop-foundation-ns.md) -* [SE-0088: Modernize libdispatch for Swift 3 naming conventions](https://github.com/apple/swift-evolution/blob/master/proposals/0088-libdispatch-for-swift3.md) -* [SE-0089: Renaming `String.init(_: T)`](https://github.com/apple/swift-evolution/blob/master/proposals/0089-rename-string-reflection-init.md) -* [SE-0091: Improving operator requirements in protocols](https://github.com/apple/swift-evolution/blob/master/proposals/0091-improving-operators-in-protocols.md) -* [SE-0092: Typealiases in protocols and protocol extensions](https://github.com/apple/swift-evolution/blob/master/proposals/0092-typealiases-in-protocols.md) -* [SE-0093: Adding a public `base` property to slices](https://github.com/apple/swift-evolution/blob/master/proposals/0093-slice-base.md) -* [SE-0094: Add sequence(first:next:) and sequence(state:next:) to the stdlib](https://github.com/apple/swift-evolution/blob/master/proposals/0094-sequence-function.md) -* [SE-0095: Replace `protocol` syntax with `P1 & P2` syntax](https://github.com/apple/swift-evolution/blob/master/proposals/0095-any-as-existential.md) -* [SE-0096: Converting dynamicType from a property to an operator](https://github.com/apple/swift-evolution/blob/master/proposals/0096-dynamictype.md) -* [SE-0099: Restructuring Condition Clauses](https://github.com/apple/swift-evolution/blob/master/proposals/0099-conditionclauses.md) -* [SE-0101: Reconfiguring `sizeof` and related functions into a unified `MemoryLayout` struct](https://github.com/apple/swift-evolution/blob/master/proposals/0101-standardizing-sizeof-naming.md) -* [SE-0102: Remove `@noreturn` attribute and introduce an empty `Never` type](https://github.com/apple/swift-evolution/blob/master/proposals/0102-noreturn-bottom-type.md) -* [SE-0103: Make non-escaping closures the default](https://github.com/apple/swift-evolution/blob/master/proposals/0103-make-noescape-default.md) -* [SE-0106: Add a `macOS` Alias for the `OSX` Platform Configuration Test](https://github.com/apple/swift-evolution/blob/master/proposals/0106-rename-osx-to-macos.md) -* [SE-0107: UnsafeRawPointer API](https://github.com/apple/swift-evolution/blob/master/proposals/0107-unsaferawpointer.md) -* [SE-0109: Remove the `Boolean` protocol](https://github.com/apple/swift-evolution/blob/master/proposals/0109-remove-boolean.md) -* [SE-0111: Remove type system significance of function argument labels](https://github.com/apple/swift-evolution/blob/master/proposals/0111-remove-arg-label-type-significance.md) -* [SE-0112: Improved NSError Bridging](https://github.com/apple/swift-evolution/blob/master/proposals/0112-nserror-bridging.md) -* [SE-0113: Add integral rounding functions to FloatingPoint](https://github.com/apple/swift-evolution/blob/master/proposals/0113-rounding-functions-on-floatingpoint.md) -* [SE-0114: Updating Buffer "Value" Names to "Header" Names](https://github.com/apple/swift-evolution/blob/master/proposals/0114-buffer-naming.md) -* [SE-0115: Rename Literal Syntax Protocols](https://github.com/apple/swift-evolution/blob/master/proposals/0115-literal-syntax-protocols.md) -* [SE-0116: Import Objective-C `id` as Swift `Any` type](https://github.com/apple/swift-evolution/blob/master/proposals/0116-id-as-any.md) -* [SE-0117: Allow distinguishing between public access and public overridability](https://github.com/apple/swift-evolution/blob/master/proposals/0117-non-public-subclassable-by-default.md) -* [SE-0118: Closure Parameter Names and Labels](https://github.com/apple/swift-evolution/blob/master/proposals/0118-closure-parameter-names-and-labels.md) -* [SE-0120: Revise `partition` Method Signature](https://github.com/apple/swift-evolution/blob/master/proposals/0120-revise-partition-method.md) -* [SE-0121: Remove `Optional` Comparison Operators](https://github.com/apple/swift-evolution/blob/master/proposals/0121-remove-optional-comparison-operators.md) -* [SE-0124: `Int.init(ObjectIdentifier)` and `UInt.init(ObjectIdentifier)` should have a `bitPattern:` label](https://github.com/apple/swift-evolution/blob/master/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md) -* [SE-0125: Remove `NonObjectiveCBase` and `isUniquelyReferenced`](https://github.com/apple/swift-evolution/blob/master/proposals/0125-remove-nonobjectivecbase.md) -* [SE-0127: Cleaning up stdlib Pointer and Buffer Routines](https://github.com/apple/swift-evolution/blob/master/proposals/0127-cleaning-up-stdlib-ptr-buffer.md) -* [SE-0128: Change failable UnicodeScalar initializers to failable](https://github.com/apple/swift-evolution/blob/master/proposals/0128-unicodescalar-failable-initializer.md) -* [SE-0129: Package Manager Test Naming Conventions](https://github.com/apple/swift-evolution/blob/master/proposals/0129-package-manager-test-naming-conventions.md) -* [SE-0130: Replace repeating `Character` and `UnicodeScalar` forms of String.init](https://github.com/apple/swift-evolution/blob/master/proposals/0130-string-initializers-cleanup.md) -* [SE-0131: Add `AnyHashable` to the standard library](https://github.com/apple/swift-evolution/blob/master/proposals/0131-anyhashable.md) -* [SE-0133: Rename `flatten()` to `joined()`](https://github.com/apple/swift-evolution/blob/master/proposals/0133-rename-flatten-to-joined.md) -* [SE-0134: Rename two UTF8-related properties on String](https://github.com/apple/swift-evolution/blob/master/proposals/0134-rename-string-properties.md) -* [SE-0135: Package Manager Support for Differentiating Packages by Swift version](https://github.com/apple/swift-evolution/blob/master/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md) -* [SE-0136: Memory Layout of Values](https://github.com/apple/swift-evolution/blob/master/proposals/0136-memory-layout-of-values.md) -* [SE-0137: Avoiding Lock-In to Legacy Protocol Designs](https://github.com/apple/swift-evolution/blob/master/proposals/0137-avoiding-lock-in.md) +* [SE-0002: Removing currying `func` declaration syntax](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0002-remove-currying.md) +* [SE-0003: Removing `var` from Function Parameters](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0003-remove-var-parameters.md) +* [SE-0004: Remove the `++` and `--` operators](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0004-remove-pre-post-inc-decrement.md) +* [SE-0005: Better Translation of Objective-C APIs Into Swift](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0005-objective-c-name-translation.md) +* [SE-0006: Apply API Guidelines to the Standard Library](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0006-apply-api-guidelines-to-the-standard-library.md) +* [SE-0007: Remove C-style for-loops with conditions and incrementers](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0007-remove-c-style-for-loops.md) +* [SE-0008: Add a Lazy flatMap for Sequences of Optionals](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0008-lazy-flatmap-for-optionals.md) +* [SE-0016: Adding initializers to Int and UInt to convert from UnsafePointer and UnsafeMutablePointer](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0016-initializers-for-converting-unsafe-pointers-to-ints.md) +* [SE-0017: Change `Unmanaged` to use `UnsafePointer`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0017-convert-unmanaged-to-use-unsafepointer.md) +* [SE-0019: Swift Testing](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0019-package-manager-testing.md) +* [SE-0023: API Design Guidelines](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0023-api-guidelines.md) +* [SE-0025: Scoped Access Level](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0025-scoped-access-level.md) +* [SE-0029: Remove implicit tuple splat behavior from function applications](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0029-remove-implicit-tuple-splat.md) +* [SE-0031: Adjusting inout Declarations for Type Decoration](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0031-adjusting-inout-declarations.md) +* [SE-0032: Add `first(where:)` method to `SequenceType`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0032-sequencetype-find.md) +* [SE-0033: Import Objective-C Constants as Swift Types](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0033-import-objc-constants.md) +* [SE-0034: Disambiguating Line Control Statements from Debugging Identifiers](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0034-disambiguating-line.md) +* [SE-0035: Limiting `inout` capture to `@noescape` contexts](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0035-limit-inout-capture.md) +* [SE-0036: Requiring Leading Dot Prefixes for Enum Instance Member Implementations](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0036-enum-dot.md) +* [SE-0037: Clarify interaction between comments & operators](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0037-clarify-comments-and-operators.md) +* [SE-0038: Package Manager C Language Target Support](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0038-swiftpm-c-language-targets.md) +* [SE-0039: Modernizing Playground Literals](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0039-playgroundliterals.md) +* [SE-0040: Replacing Equal Signs with Colons For Attribute Arguments](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0040-attributecolons.md) +* [SE-0043: Declare variables in 'case' labels with multiple patterns](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0043-declare-variables-in-case-labels-with-multiple-patterns.md) +* [SE-0044: Import as Member](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0044-import-as-member.md) +* [SE-0046: Establish consistent label behavior across all parameters including first labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0046-first-label.md) +* [SE-0047: Defaulting non-Void functions so they warn on unused results](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0047-nonvoid-warn.md) +* [SE-0048: Generic Type Aliases](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0048-generic-typealias.md) +* [SE-0049: Move @noescape and @autoclosure to be type attributes](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0049-noescape-autoclosure-type-attrs.md) +* [SE-0052: Change IteratorType post-nil guarantee](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0052-iterator-post-nil-guarantee.md) +* [SE-0053: Remove explicit use of `let` from Function Parameters](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0053-remove-let-from-function-parameters.md) +* [SE-0054: Abolish `ImplicitlyUnwrappedOptional` type](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0054-abolish-iuo.md) +* [SE-0055: Make unsafe pointer nullability explicit using Optional](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0055-optional-unsafe-pointers.md) +* [SE-0057: Importing Objective-C Lightweight Generics](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0057-importing-objc-generics.md) +* [SE-0059: Update API Naming Guidelines and Rewrite Set APIs Accordingly](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0059-updated-set-apis.md) +* [SE-0060: Enforcing order of defaulted parameters](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0060-defaulted-parameter-order.md) +* [SE-0061: Add Generic Result and Error Handling to autoreleasepool()](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0061-autoreleasepool-signature.md) +* [SE-0062: Referencing Objective-C key-paths](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0062-objc-keypaths.md) +* [SE-0063: SwiftPM System Module Search Paths](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0063-swiftpm-system-module-search-paths.md) +* [SE-0064: Referencing the Objective-C selector of property getters and setters](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0064-property-selectors.md) +* [SE-0065: A New Model For Collections and Indices](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0065-collections-move-indices.md) +* [SE-0066: Standardize function type argument syntax to require parentheses](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0066-standardize-function-type-syntax.md) +* [SE-0067: Enhanced Floating Point Protocols](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0067-floating-point-protocols.md) +* [SE-0069: Mutability and Foundation Value Types](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0069-swift-mutability-for-foundation.md) +* [SE-0070: Make Optional Requirements Objective-C-only](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0070-optional-requirements.md) +* [SE-0071: Allow (most) keywords in member references](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0071-member-keywords.md) +* [SE-0072: Fully eliminate implicit bridging conversions from Swift](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0072-eliminate-implicit-bridging-conversions.md) +* [SE-0076: Add overrides taking an UnsafePointer source to non-destructive copying methods on UnsafeMutablePointer](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0076-copying-to-unsafe-mutable-pointer-with-unsafe-pointer-source.md) +* [SE-0077: Improved operator declarations](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0077-operator-precedence.md) +* [SE-0081: Move `where` clause to end of declaration](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0081-move-where-expression.md) +* [SE-0085: Package Manager Command Names](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0085-package-manager-command-name.md) +* [SE-0086: Drop NS Prefix in Swift Foundation](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0086-drop-foundation-ns.md) +* [SE-0088: Modernize libdispatch for Swift 3 naming conventions](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0088-libdispatch-for-swift3.md) +* [SE-0089: Renaming `String.init(_: T)`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0089-rename-string-reflection-init.md) +* [SE-0091: Improving operator requirements in protocols](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0091-improving-operators-in-protocols.md) +* [SE-0092: Typealiases in protocols and protocol extensions](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0092-typealiases-in-protocols.md) +* [SE-0093: Adding a public `base` property to slices](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0093-slice-base.md) +* [SE-0094: Add sequence(first:next:) and sequence(state:next:) to the stdlib](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0094-sequence-function.md) +* [SE-0095: Replace `protocol` syntax with `P1 & P2` syntax](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0095-any-as-existential.md) +* [SE-0096: Converting dynamicType from a property to an operator](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0096-dynamictype.md) +* [SE-0099: Restructuring Condition Clauses](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0099-conditionclauses.md) +* [SE-0101: Reconfiguring `sizeof` and related functions into a unified `MemoryLayout` struct](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0101-standardizing-sizeof-naming.md) +* [SE-0102: Remove `@noreturn` attribute and introduce an empty `Never` type](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0102-noreturn-bottom-type.md) +* [SE-0103: Make non-escaping closures the default](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0103-make-noescape-default.md) +* [SE-0106: Add a `macOS` Alias for the `OSX` Platform Configuration Test](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0106-rename-osx-to-macos.md) +* [SE-0107: UnsafeRawPointer API](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0107-unsaferawpointer.md) +* [SE-0109: Remove the `Boolean` protocol](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0109-remove-boolean.md) +* [SE-0111: Remove type system significance of function argument labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0111-remove-arg-label-type-significance.md) +* [SE-0112: Improved NSError Bridging](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0112-nserror-bridging.md) +* [SE-0113: Add integral rounding functions to FloatingPoint](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0113-rounding-functions-on-floatingpoint.md) +* [SE-0114: Updating Buffer "Value" Names to "Header" Names](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0114-buffer-naming.md) +* [SE-0115: Rename Literal Syntax Protocols](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0115-literal-syntax-protocols.md) +* [SE-0116: Import Objective-C `id` as Swift `Any` type](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0116-id-as-any.md) +* [SE-0117: Allow distinguishing between public access and public overridability](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0117-non-public-subclassable-by-default.md) +* [SE-0118: Closure Parameter Names and Labels](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0118-closure-parameter-names-and-labels.md) +* [SE-0120: Revise `partition` Method Signature](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0120-revise-partition-method.md) +* [SE-0121: Remove `Optional` Comparison Operators](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0121-remove-optional-comparison-operators.md) +* [SE-0124: `Int.init(ObjectIdentifier)` and `UInt.init(ObjectIdentifier)` should have a `bitPattern:` label](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0124-bitpattern-label-for-int-initializer-objectidentfier.md) +* [SE-0125: Remove `NonObjectiveCBase` and `isUniquelyReferenced`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0125-remove-nonobjectivecbase.md) +* [SE-0127: Cleaning up stdlib Pointer and Buffer Routines](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0127-cleaning-up-stdlib-ptr-buffer.md) +* [SE-0128: Change failable UnicodeScalar initializers to failable](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0128-unicodescalar-failable-initializer.md) +* [SE-0129: Package Manager Test Naming Conventions](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0129-package-manager-test-naming-conventions.md) +* [SE-0130: Replace repeating `Character` and `UnicodeScalar` forms of String.init](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0130-string-initializers-cleanup.md) +* [SE-0131: Add `AnyHashable` to the standard library](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0131-anyhashable.md) +* [SE-0133: Rename `flatten()` to `joined()`](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0133-rename-flatten-to-joined.md) +* [SE-0134: Rename two UTF8-related properties on String](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0134-rename-string-properties.md) +* [SE-0135: Package Manager Support for Differentiating Packages by Swift version](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0135-package-manager-support-for-differentiating-packages-by-swift-version.md) +* [SE-0136: Memory Layout of Values](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0136-memory-layout-of-values.md) +* [SE-0137: Avoiding Lock-In to Legacy Protocol Designs](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0137-avoiding-lock-in.md) diff --git a/releases/swift-4_0.md b/releases/swift-4_0.md index 3880a72769..a133a1160b 100644 --- a/releases/swift-4_0.md +++ b/releases/swift-4_0.md @@ -14,7 +14,7 @@ The high-priority features supporting Stage 1's source and ABI stability goals were: * Source stability features: The Swift language will need [some - accommodations](https://github.com/apple/swift-evolution/blob/master/proposals/0141-available-by-swift-version.md) + accommodations](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0141-available-by-swift-version.md) to support code bases that target different language versions, to help Swift deliver on its source-compatibility goals while still enabling rapid progress. @@ -38,11 +38,11 @@ stability goals were: of which manifest as extraneous underscored protocols and workarounds. If the underlying language deficiencies remain, they become a permanent part of the stable ABI. [Conditional - conformances](https://github.com/apple/swift-evolution/blob/master/proposals/0143-conditional-conformances.md), + conformances](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0143-conditional-conformances.md), [recursive protocol requirements](https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#recursive-protocol-constraints-), and [where clauses for associated - types](https://github.com/apple/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md) + types](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0142-associated-types-constraints.md) are known to be in this category, but it's plausible that other features will be in scope if they would be used in the standard library. diff --git a/releases/swift-5_0.md b/releases/swift-5_0.md index 8f980b3da9..fdd12922e0 100644 --- a/releases/swift-5_0.md +++ b/releases/swift-5_0.md @@ -8,11 +8,11 @@ ABI stability is only one of two pieces needed to support binary frameworks. The The need to achieve ABI stability in Swift 5 will guide most of the priorities for the release. In addition, there are important goals to complete that carry over from Swift 4 that are prerequisites to locking down the ABI of the standard library: -- **Generics features needed for standard library**. We will finish implementing [conditional conformances](https://github.com/apple/swift-evolution/blob/master/proposals/0143-conditional-conformances.md) and [recursive protocol requirements](https://github.com/apple/swift-evolution/blob/master/proposals/0157-recursive-protocol-constraints.md), which are needed for the standard library to achieve ABI stability. Both of these have gone through the evolution proposal process and there are no known other generics enhancements needed for ABI stability. +- **Generics features needed for standard library**. We will finish implementing [conditional conformances](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0143-conditional-conformances.md) and [recursive protocol requirements](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0157-recursive-protocol-constraints.md), which are needed for the standard library to achieve ABI stability. Both of these have gone through the evolution proposal process and there are no known other generics enhancements needed for ABI stability. - **API resilience**. We will implement the essential pieces needed to support API resilience, in order to allow public APIs for a library to evolve over time while maintaining a stable ABI. -- **Memory ownership model**. An (opt-in) Cyclone/Rust-inspired memory [ownership model](https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md) is strongly desirable for systems programming and for other high-performance applications that require predictable and deterministic performance. Part of this model was introduced in Swift 4 when we began to [ enforce exclusive access to memory](https://github.com/apple/swift-evolution/blob/master/proposals/0176-enforce-exclusive-access-to-memory.md). In Swift 5 our goal is to tackle the [pieces of the ownership model that are key to ABI stability](https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md#priorities-for-abi-stability). +- **Memory ownership model**. An (opt-in) Cyclone/Rust-inspired memory [ownership model](https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md) is strongly desirable for systems programming and for other high-performance applications that require predictable and deterministic performance. Part of this model was introduced in Swift 4 when we began to [ enforce exclusive access to memory](https://github.com/swiftlang/swift-evolution/blob/master/proposals/0176-enforce-exclusive-access-to-memory.md). In Swift 5 our goal is to tackle the [pieces of the ownership model that are key to ABI stability](https://github.com/apple/swift/blob/master/docs/OwnershipManifesto.md#priorities-for-abi-stability). ## Other Improvements diff --git a/visions/approachable-concurrency.md b/visions/approachable-concurrency.md new file mode 100644 index 0000000000..25e6e6b725 --- /dev/null +++ b/visions/approachable-concurrency.md @@ -0,0 +1,198 @@ +# Improving the approachability of data-race safety + +[SE-0434]: https://github.com/swiftlang/swift-evolution/blob/main/proposals/0434-global-actor-isolated-types-usability.md + +> This document is an official feature [vision document](https://forums.swift.org/t/the-role-of-vision-documents-in-swift-evolution/62101). The Language Steering Group has endorsed the goals and basic approach laid out in this document. This endorsement is not a pre-approval of any of the concrete proposals that may come out of this document. All proposals will undergo normal evolution review, which may result in rejection or revision from how they appear in this document. + +## Introduction + +Swift's built-in support for concurrency has three goals: + +1. Extend memory safety guarantees to low-level data races. +2. Maintain progressive disclosure for non-concurrent code, and make basic use of concurrency simple and easy. +3. Make advanced uses of concurrency to improve performance natural to accomplish and reason about. + +The Swift 6 language mode provides a baseline of correctness that meets the first goal, but sometimes it comes at the cost of the second, and it can be frustrating to adopt. Now that we have a lot more user experience under our belt as a community, it’s reasonable to ask what we can do in the language to address that problem. This document lays out several potential paths for improving the usability of Swift 6, focusing on two primary use cases: + +1. Simple situations where programmers aren’t intending to use concurrency at all. +2. Adapting an existing code base that uses concurrency libraries which predate Swift's native concurrency model. + +While performance is not our immediate focus, it’s something we need to keep in mind during this exercise: we don’t want these usability wins to create pervasive regressions or to make it frustratingly difficult to achieve a high level of performance. + +A key tenet of our thinking in this vision is that we want to drastically reduce the number of explicit concurrency annotations necessary in projects that aren’t trying to leverage parallelism for performance. This is important for many kinds of programming, such as UI programming and scripts, where concurrency is often localized and large swathes of the code are generally expected to be constrained to the main actor. At the same time, we want to maintain a smooth path for experienced programmers to opt in to concurrency and maintain the safety of complete data-race checking. + +As we see it, there should be three phases on the progressive disclosure path for concurrency: + +1. **Write sequential, single-threaded code**. By default, programmers writing executable projects will write sequential code; there is no runtime parallelism, and therefore no data-race safety errors are surfaced to the programmer. +2. **Write asynchronous code without data-race safety errors**. When programmers need functionality that can suspend, they can start introducing basic uses of async/await. Programmers won’t have to confront data-race safety at this point, because they aren’t yet introducing parallelism into their code. This is an important distinction, because there are many library APIs that perform work asynchronously, but they don’t need to use the programmer’s shared mutable state from a concurrent task. In these cases, programmers don’t have to understand data-race safety just to call an async API. +3. **Introduce parallelism to improve performance.** When the programmer is ready to embrace concurrency to get better performance, they can explicitly offload work from the main actor to the cooperative thread pool, leverage tasks and structured concurrency, etc, all while relying on the compiler to prevent mistakes that risk a data race. + +## Mitigating false positive data-race safety errors in sequential code + +A lot of code is effectively “single-threaded”. For example, most executables, such as apps, command-line tools, and scripts, start running on the main actor and just stay there unless some part of the code actually does something concurrent (like creating a `Task`). If there isn’t any use of concurrency, the entire program will run sequentially, and there’s no risk of data races — every concurrency diagnostic is necessarily a false positive! It would be good to be able to take advantage of that in the language, both to avoid annoying programmers with unnecessary diagnostics and to reinforce progressive disclosure. Many people get into Swift by writing these kinds of programs, and if we can avoid needing to teach them about concurrency straight away, we’ll make the language much more approachable. + +Now, “If nothing in the program uses concurrency, suppress all the concurrency diagnostics” requires what compiler writers call a *whole-program analysis*, and rules like that tend not to work out well on multiple levels. For one, it would require the compiler to look at all of the code in the program all at once; this might be okay for small scripts, but it would scale poorly as the program got more complex. More importantly, it would make the first adoption of concurrency extremely painful: programmers would be hit by a tidal wave of errors in code they haven’t changed. And, of course, many libraries do use concurrency behind the scenes; importing even a single library like that would force concurrency-safety diagnostics everywhere. + +A better approach is to locally state our assumption that the sequential parts of the program are “single-threaded”. Rather than having to assume the possibility of concurrency, Swift would know that these parts of the code will all run sequentially, which it can use to prove that there aren’t any data races. There can still be concurrent parts of the program elsewhere, but Swift would stop them from accessing the single-threaded bits. Fortunately, this is something that Swift can already model quite well! + +### Single-threaded code and its challenges under Swift 6 + +The easiest and best way to model single-threaded code is with a global actor. Everything on a global actor runs sequentially, and code that isn’t isolated to that actor can’t access its data. All programs start running on the global actor `MainActor`, and if everything in the program is isolated to the main actor, there shouldn’t be any concurrency errors. + +Unfortunately, it’s not quite that simple right now. Writing a single-threaded program is surprisingly difficult under the Swift 6 language mode. This is because Swift 6 defaults to a presumption of concurrency: if a function or type is not annotated or inferred to be isolated, it is treated as non-isolated, meaning it can be used concurrently. This default often leads to conflicts with single-threaded code, producing false positive diagnostics in cases such as: + +* global and static variables, +* conformances of main-actor-isolated types to non-isolated protocols, +* class deinitializers, +* overrides of non-isolated superclass methods in a main-actor-isolated subclass, and +* calls to main-actor-isolated functions from the platform SDK. + +To see this, let’s explore the first of those cases in more detail. A mutable global variable (or an immutable one that stores a non-`Sendable` value) is only memory-safe if it’s used in a single-threaded way. If the whole program is single-threaded, there’s no problem, and the variable is always safe to use. But since Swift 6 presumes concurrency by default, it requires a variable like this to be explicitly isolated to a global actor, like `@MainActor`. A function that uses that variable is then also required to be statically isolated to `@MainActor`: + +```swift +class AudioManager { + @MainActor + static let shared = AudioManager() + + func playSound() { ... } +} + +class Model { + func play() { + AudioManager.shared.playSound() // error: Main actor-isolated static property 'shared' can not be referenced from a nonisolated context + } +} +``` + +And this in turn means that functions that call those functions must also be `@MainActor`, and so on until the `@MainActor` annotation has been laboriously propagated throughout the entire transitive tree of callers. Because main actor isolation is so common, many programmers have resorted to reflexively writing `@MainActor` everywhere, an onerous annotation burden that goes against Swift’s goals of making the simplest things easy. + +Because the default programming model presumes concurrency, it is also hard on programmers who haven’t yet learned about concurrent programming, because they are confronted with the concept of data-race safety and actor isolation too early simply by using these basic language features: + +```swift +class AudioManager { + static let shared = AudioManager() // error: Static property 'shared' is not concurrency-safe because non-'Sendable' type 'AudioManager' may have shared mutable state +} +``` + +Analogous problems arise with all the other kinds of false positives listed above. For example, when using values from generic code, the value’s type usually must conform to one or more protocols. However, actor-isolated types cannot easily conform to protocols that aren’t aware of that isolation: they can declare the conformance, but it’s often impossible to write a useful implementation because the value’s properties will not be available. This is exactly the same kind of conflict as with global variables, where we have generally single-threaded code but a presumption of concurrency from the protocol, except that *this* conflict usually can’t be solved with annotations at all — the only fixes are to change the protocol, avoid all the isolated storage, or dangerously assert (with `assumeIsolated`) that the method is only used dynamically from the right actor. + +### Allowing modules to default to being “single-threaded” + +We believe that the right solution to these problems is to allow code to opt in to being “single-threaded” by default, on a module-by-module basis. This would change the default isolation rule for unannotated code in the module: rather than being non-isolated, and therefore having to deal with the presumption of concurrency, the code would instead be implicitly isolated to `@MainActor`. Code imported from other modules would be unaffected by the current module’s choice of default. When the programmer really wants concurrency, they can request it explicitly by marking a function or type as `nonisolated` (which can be used on any declaration as of [SE-0449](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0449-nonisolated-for-global-actor-cutoff.md)), or they can define it in a module that doesn’t default to main-actor isolation. This doesn’t fundamentally change anything about Swift’s isolation model; it just flips the default, effectively creating a model in which code is single-threaded except where it explicitly requests concurrency. Modules that don’t want this could of course continue to use the current rules. + +Making a module be isolated to the main actor by default would directly fix several of the false positive problems listed above for single-threaded code. Global variables would default to being isolated to the main actor, avoiding the diagnostic when they’re declared. Functions in the module would also default to being isolated to the main actor, allowing them to freely use both those isolated global variables and any main-actor-isolated functions and variables imported from the platform SDK. Class overrides and protocol conformances aren’t quite so easy, but we think we can extend them in ways that allow a natural solution with main actor isolation. We’ll get to how later in this document. + +### Default concurrency rules for executable and library modules + +As mentioned above, executable targets tend to center around the main actor. Command-line tools and scripts all start on the main actor and continue to run there unless they explicitly do something that introduces concurrency. Similarly, most UI programs make heavy use of single-threaded UI frameworks that privilege the main actor. This kind of code would be greatly improved by adopting a single-threaded model by default. Furthermore, since many new Swift programmers find themselves first writing this kind of code, this would also be a significant improvement to Swift’s progressive disclosure: programmers writing code in this mode should not run into data-race safety issues and diagnostics until they intentionally introduce concurrency. We feel that this amounts to a compelling argument that executable targets should default to inferring main actor isolation. + +The same argument does not apply to libraries. Most library functions are meant to be usable from any context, and libraries usually avoid using any global or shared mutable state. Swift also already asks a little more of library authors in general; for example, access control is usually a more significant concern for library authors than for app developers. It would be reasonable for library targets to default to `nonisolated` the same way they do today in Swift 6. + +Specific libraries could still decide to default to the main actor, such as when they’re libraries of UI widgets, or if a library is used for code organization within an executable project. + +### Risks of a language dialect + +Adding a per-module setting to specify the default isolation would introduce a new permanent language dialect. In a sense, Swift adds a new language dialect whenever it adds an upcoming language feature flag, but these are seen as “temporary” because it’s expected that those features will eventually be rolled into a future language mode. Permanent language dialects can be problematic for a variety of reasons: + +* They can harm readability if readers have to know which dialect the code uses before they can understand the code. +* They can harm usability if programmers have to consciously program differently based on the dialect in use in the code or if code cannot be easily moved between projects using different dialects. +* They can harm tools such as IDEs if the tool has to know which dialect the code uses before it can work correctly; this is particularly challenging for code files that may be used in multiple dialects, such as a `.h` file in a C/C++/Objective-C IDE. +* When dialects are platform-dependent, they can harm portability and basic workflows (such as testing) if it is difficult to make the same code build as multiple dialects. + +Some of these problems do not seem to apply to this proposed dialect because it only affects the isolation of declarations. Most IDE services, such as syntax highlighting and code completion, do not need to know the isolation of the surrounding context. And there’s no good reason for a module to build with a different default isolation on different platforms, so the dialect does not seem to introduce any portability concerns. + +Readability does seem to be a fair concern. It is often useful to know what isolation a function will run under, and with this change, that would be dialect-specific. However, it is worth noting that the dynamic isolation of a function is already not always statically knowable because of the way that e.g. synchronous `nonisolated` functions inherit their callers’ isolation. And it would be reasonable for IDEs to be able to present isolation information for the current context: even without this dialect change, it is not always easy to understand how Swift’s isolation inference rules will apply to any particular declaration. + +Programmers will probably not consciously program differently under these different dialects, and the compiler should provide reasonable guidance if they make a mistake. Moving code from a single-threaded module to a nonisolated module might be somewhat more arduous, however. To some degree, this is inherent and arguably even good: the programmer may be moving this code in an effort to generalize it to work concurrently, and any new diagnostics represent real problems that the programmer didn’t have to deal with before this generalization. But when the programmer is not trying to generalize the code to use concurrency, this could be frustrating, and it might be good for IDEs to offer assistance, such as tools to make the single-threaded assumptions of a piece of code explicit. + +On balance, we feel that the costs of this particular dialect are modest and manageable. + +## Isolated conformances + +When checking a conformance to a protocol, Swift 6 often requires implementations to be nonisolated when the requirement is, including when the requirement is synchronous or the parameters are not Sendable. This makes it difficult to implement nonisolated protocols with any kind of isolated type: global-actor-isolated types, certainly, but also actors themselves. This restriction is very important when writing concurrent code, but it's a common source of false positives in single-threaded programs. Even worse, there's often no good solution to the problem: a correct implementation of the protocol for an isolated type usually requires access to the isolated data, and the only way to get that is to assert that the calling context is actually isolated, which completely subverts the static isolation safety that Swift 6 tries to provide. + +In many ways, isolation's interaction with protocol requirements is similar to its interaction with function values. In both situations, we have an abstract signature that doesn't express isolation by default, which Swift wants to interpret as an affirmative statement that the isolation of the implementation doesn't matter. Over the last few years, Swift has gradually added more ways to handle isolated function values: + +- A function value can have a type like `() -> Bool` that says it's not sendable. These functions can have any kind of isolation as long as it's the same as the current concurrency domain. Since the function can't be used from a different context, there's no need for it to spell out its isolation explicitly in its type; Swift just checks that the isolation actually matches whenever it makes a new non-`Sendable` function value. + +- A function value can have a type like `@MainActor () -> Bool` that says it's isolated to a specific global actor. These functions can either be non-isolated or isolated to that actor, and Swift just treats them as the latter unconditionally when calling them. + +- A function value can have a type like `@isolated(any) () -> Bool` that says it might be isolated to a specific actor that it carries around with it dynamically. These functions can be isolated to anything, and the caller has to be prepared to handle it. + +Each of these ideas also works with protocol conformances. If we know that we only intend to use a protocol conformance from the current concurrency domain, we don't really care whether the implementation requires some sort of isolation as long as the current context has that isolation. Similarly, if we know that the implementation of a protocol might be isolated to a specific global actor, we can handle that whenever we use the conformance exactly as if the protocol requirements were declared isolated to that actor. And we could even do that dynamically with a statically-unknown isolation, the same way Swift already does with `@isolated(any)` function values. + +The most important of these for our model of single-threaded code is to be able to express global-actor-isolated conformances. When a type is isolated to a global actor, its methods will be isolated by default. Normally, these methods would not be legal implementations of nonisolated protocol requirements. When Swift recognizes this, it can simply treat the conformance as isolated to that global actor. This is a kind of *isolated conformance*, which will be a new concept in the language. + +Now, an isolated conformance is less flexible than a nonisolated conformance. For example, generic types and functions from nonisolated modules (including all current declarations) will still be interpreted as requiring nonisolated conformances. This will mean that they can't be called with a type that only has an isolated conformance, but it will also allow them to freely use the conformance from any concurrency domain, the same way they can today. Generic types and functions in "single-threaded" modules will default to allowing conformances isolated to the module's default global actor. Since those functions will themselves be isolated to that global actor, they won't have any problem using those conformances. + +A generic function that can work with both isolated and nonisolated conformances should be able to declare that it can accept an isolated conformance. It would then be restricted to only use the conformance from the current concurrency domain, as if it were a sort of "non-sendable conformance". This is an important tool for generic libraries such as the standard library, many of which will never use conformances concurrently and so are fine with accepting isolated conformances. + +This design is still being developed, and there are a lot of details that will have to be figured out. Nonetheless, we are tentatively very excited about the potential for this feature to fix problems with how Swift's generics system interacts with isolated types, especially global-actor-isolated types. + +## Isolated subclasses and overrides + +To achieve data race safety, Swift 6 has to diagnose a variety of problems that are specific to classes: + +- The sendability of a class must match the sendability of its superclass[^1]. +- If a class is sendable and not isolated to a global actor, its stored properties must be sendable. +- If a class is isolated to a global actor, its superclass must either be non-isolated or isolated to the same global actor. +- An override must have the same isolation as the declaration it overrides. + +[^1]: There is a natural exception to this rule: a sendable class can have a non-sendable superclass if the superclass and all of its ancestors only have sendable stored properties. Currently, Swift only implements this exception for the exact class `NSObject`. + +All of these diagnostics are false positives in purely sequential programs. Fortunately, many of them don't apply to global-actor-isolated classes in the first place. The restriction that isolated classes can't inherit from classes isolated to a different global actor is very reasonable, but it's extremely unlikely to affect programmers in practice because they rarely use global actors other than the main actor at all. The only significant false positive here, then, is the restriction on overriding. + +[SE-0434][] changed the rules for global-actor-isolated classes to allow them to inherit from non-sendable classes; the subclass just remains non-sendable. Since the class is non-sendable and isolated to a global actor, it's tempting to say that override restrictions shouldn't be necessary for it: the object reference should only be usable on the global actor in the first place. This isn't quite true today, but we think we can revise SE-0434 to make it true by imposing restrictions on initializers in the subclass. This would be a small source break, but it would greatly improve the usability of these classes. + +Unfortunately, global-actor-isolated classes that inherit from nonisolated sendable classes can't benefit from the same idea. We can only see one way to make it safe to allow isolated overrides of non-isolated methods for these classes without a whole-program prohibition of concurrency: we would have to prevent any reference to the subclass from being converted to its sendable superclass type. This is feasible to implement, but we suspect it would be too restrictive to be widely useful; we can revisit this with more information. + +## Easing the introduction of basic async code + +[SE-0338](https://github.com/hborla/swift-evolution/blob/async-function-isolation/proposals/0338-clarify-execution-non-actor-async.md) specifies that nonisolated async functions never run on an actor's executor. This design decision was made to prevent unnecessary serialization and contention for the actor by switching off of the actor to run the nonisolated async function, and any new tasks it creates that inherit isolation. The actor is then free to make forward progress on other work. This behavior is especially important for preventing unexpected overhang on the main actor. + +Over time, we have learned that this design decision undermines progressive disclosure, because it prioritizes main actor responsiveness at the expense of making basic asynchronous code difficult to write. Always switching off of an actor to run a nonisolated async function imposes data-race safety errors on programmers when they call the API with non-sendable arguments from the main actor. + +Many library APIs have transitioned to using isolated parameters to ensure that an async API runs on the caller by default, because it’s a much easier default to work with in client code. + +The current execution semantics of async functions also impede programmer’s understanding of the concurrency model because there is a significant difference in what `nonisolated` means on synchronous and asynchronous functions. Nonisolated synchronous functions always run in the isolation domain of the caller, while nonisolated async functions always switch off of the caller's actor (if there is one). It's confusing that `nonisolated` does not have a consistent meaning when applied to functions, and the current behavior conflates the concept of actor isolation with the ability for a function to suspend. + +Changing the default execution semantics of nonisolated async functions to run wherever they are called better facilitates progressive disclosure of concurrency. This default allows functions to leverage suspension without forcing callers to cross an isolation boundary and imposing data-race safety checks on arguments and results. A lot of basic asynchronous code can be written correctly and efficiently with only the ability to suspend. When an async function needs to always run off of an actor, the API author can still specify that with a new `@execution(concurrent)` annotation on the function. This provides a better default for most cases while still maintaining the ease of specifying that an async function switches off of an actor to run. + +Many programmers have internalized the SE-0338 semantics, and making this change several years after SE-0338 was accepted creates an unfortunate intermediate state where it's difficult to understand the semantics of a nonisolated async function without understanding the build settings of the module you're writing code in. We can alleviate some of these consequences with a careful migration design. There are more details about migration in the [automatic migration](#automatic-migration) section of this document, and in the [source compatibility](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0461-async-function-isolation.md#source-compatibility) section of the proposal for this change. + +This idea has already been pitched on the forums, and you can read the full proposal for it [here](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0461-async-function-isolation.md). + +## Easing incremental migration to data-race safety + +Many programmers are not new to concurrency as a concept and have extensive experience with multi-threaded programming. However, the Swift 6 data-race safety model is a significant shift for programmers with experience using concurrency libraries that predate Swift's native concurrency features. Moreover, there are many existing, large codebases that were built on such libraries, and migrating these codebases to both leverage modern concurrency features and enable static data-race safety takes significant engineering effort. An explicit goal of improving the approachability of data-race safety is lessening the amount of effort it takes to enable data-race safety in existing codebases. + +### Bridging between synchronous and asynchronous code + +Introducing async/await into an existing codebase is difficult to do incrementally, because the language does not provide tools to bridge between synchronous and asynchronous code. Sometimes programmers can kick off a new unstructured task to perform the async work, and other times that is not suitable, e.g. because the synchronous code needs a result from the async operation. It’s also not always possible to propagate `async` throughout callers, because the function signature might be declared in a library dependency that you don’t own. + +Notably, using actors in programs that make heavy use of the main actor forces programmers to use async/await, because all interactions with an actor must be done asynchronously from outside the actor. This significantly restricts the utility of actors, especially in existing codebases. + +Other concurrency libraries like Dispatch provide a limited tool set to wait on asynchronous work, such as `DispatchQueue.asyncAndWait`. These tools come with serious tradeoffs, including tying up limited system resources and introducing the possibility for deadlocks, but they provide critical functionality that is sometimes necessary in a project. It’s important for programmers to have the ability to express this in the language, and because the language model allows actor re-entrancy and doesn’t have a strict FIFO guarantee for tasks enqueued on an actor, there’s opportunity to mitigate the risk of deadlocks that these tools come with. + +### Mitigating runtime assertions due to isolation mismatches + +[SE-0423: Dynamic actor isolation enforcement from non-strict-concurrency contexts](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0423-dynamic-actor-isolation.md) introduced dynamic actor isolation assertions that are injected by the compiler at the boundaries between data-race safe and unsafe code. This dynamic checking catches actor isolation violations in library dependencies that have not yet migrated to the Swift 6 language mode, and may have data-race safety issues in their implementation. These checks are effective at identifying missing `@Sendable` annotations, but they also make Swift 6 adoption painful for clients when they migrate before their dependencies. + +Some of these runtime crashes are false positives; the runtime checks are inserted based on the static isolation of the function, but the function might not access any mutable state that’s isolated to or derived from the actor. In these cases, the dynamic checks can simply be elided based on analysis of the function implementation. + +In other cases, the dynamic assertion indicates the presence of a runtime data race, because isolated state is being accessed from outside the actor. The correct way to resolve this data race is either to run the function on the actor, or change the function to eliminate access to actor-isolated state. There are two possible avenues for the language to aid programmers in resolving the data race: + +* Instead of directly calling the function in the wrong isolation domain, enqueue a job that calls the function on the actor. This only works if the function does not return a result. +* Use the import-as-async heuristic from [SE-0297: Concurrency Interoperability with Objective-C](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0297-concurrency-objc.md) to automatically import completion handlers of asynchronous functions as `@Sendable`, which will allow the compiler to diagnose access to actor-isolated state in completion handlers. + +## Automatic migration + +Unlike the Swift 6 migration, all language changes with source compatibility impact described in this vision can be automatically migrated to while preserving the semantics of existing code. The source incompatible portions of this vision will be gated behind [upcoming feature flags](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0362-piecemeal-future-features.md), which will be enabled by default in a future 6.x language mode, except for the per-module setting to infer main actor by default. + +Compiler tooling can automatically migrate existing projects when they choose to enable each of these upcoming features, either individually or as part of a future language mode migration. Programmers will be able to perform a “migration build” with one or more upcoming language features enabled, or with a specific language mode that enables a set of upcoming features, and be offered source code changes that would allow the compiler to build the project without errors and without changing semantics. + +## What’s not in this vision + +This vision does not cover existing pain points with task ordering and actor re-entrancy. These are important problems, but they are more prevalent in more advanced uses of concurrency and warrant a separate, dedicated exploration. + +Improving concurrency diagnostics and documentation is also not covered in this document. All language proposals should consider diagnostics to the extent that language design decisions prevent precise and actionable error messages. Beyond that, diagnostics and documentation changes are not governed by the Swift evolution process, because these changes don't have long-term source compatibility and ABI constraints, so gating improvements behind a heavy weight review process isn't necessary. However, diagnostics and documentation are an extremely important tool for making the concurrency model more approachable, and they will be included in the implementation effort behind this vision. diff --git a/visions/embedded-swift.md b/visions/embedded-swift.md new file mode 100644 index 0000000000..7bd8d7aa50 --- /dev/null +++ b/visions/embedded-swift.md @@ -0,0 +1,165 @@ +# A Vision for Embedded Swift + +## Introduction + +Swift is a general purpose programming language suitable for both high-level application software and for low-level systems software. The existing major supported deployments of Swift are primarily targeting “large” operating systems (Linux, Windows, macOS, iOS), where storage and memory are relatively plentiful, multiple applications share code via dynamic linking, and the system can be expected to provide a number of common libraries (such as the C and C++ standard libraries). The typical size of the Swift runtime and standard library in these environments is around 5MB of binary size. + +However, lots of embedded platforms and low-level environments have constraints that make the standard Swift distribution impractical. These constraints have been described and discussed in existing prior work in this area, and there have been great past discussions in the Swift forums ([link](https://forums.swift.org/t/introduce-embedded-development-using-swift/56573)) and in last year’s video call ([link](https://forums.swift.org/t/call-for-interest-video-call-kickoff-using-swift-for-embedded-bare-metal-low-resources-programming/56911)), which shows there is a lot of interest and potential for Swift in this space. The motivation of “Embedded Swift” is to achieve a first class support for embedded platforms and unblock porting and using Swift in small environments. In particular the targets are: + +* (1) Platforms that have very limited memory + * Microcontrollers and embedded systems with limited memory + * Popular MCU board families and manufacturers (Arduino, STM32, ESP32, NXP, etc.) commonly offer boards that only have an order of 10’s or 100’s of kB of memory available. + * Firmware, and especially firmware projects that are run from SRAM, or ROM +* (2) Environments where runtime dependencies, implicit runtime calls, and heap allocations are restricted + * Low-level environments without an underlying operating system, such as bootloaders, hypervisors, firmware + * Operating system kernels, kernel extensions, and other non-userspace software components + * Userspace components that are too low-level in terms of dependencies, namely anything that the Swift runtime depends on. + * A special case here is the Swift runtime itself, which is today written in C++. The concepts described further in this document allow Swift to become the implementation language instead. + +A significant portion of the current Swift runtime and standard library supports Swift’s more dynamic features, particularly those that involve metadata. These features include: + +* Dynamic reflection facilities (such as mirrors, `as?` downcasts, and printing arbitrary values) +* Existentials +* ABI stability with support for library evolution +* Separately-compiled generics +* Dynamic code loading (plug-ins) + +On “smaller” operating systems, and in restricted environments with limited binary and memory size, the size of a full Swift standard library (with all public types and APIs present) and a Swift runtime, as well as the metadata required to support dynamic features, can be so large as to prevent the usage of Swift completely. In such environments, it may be reasonable to trade away some of the flexibility from the more dynamic features to achieve a significant reduction in code size, while still retaining the character of Swift. + +The following diagram summarizes the existing approaches to shrink down the size of the Swift runtime and standard library, and how “Embedded Swift” is tackling the problems with a new approach: + +diagram + +This document presents a vision for “Embedded Swift”, a new compilation model of Swift that can produce extremely small binaries without external dependencies, suitable for restricted environments including embedded (microcontrollers) and baremetal setups (no operating system at all), and low-level environments (firmware, kernels, device drivers, low-level components of userspace OS runtimes). + +Embedded Swift limits the use of language and standard library features that would require a larger Swift runtime, while maintaining most of Swift’s feature set. It is important that Embedded Swift not become a separate dialect of Swift. Rather, it should remain an easy-to-explain subset of Swift that admits the same code and idioms as the full Swift language, where any restrictions on the language model are flagged by the Swift compiler. The subset itself should also be useful beyond low-level environments, for example, in high-performance runtimes and kernels embedded within a larger Swift application. The rest of this document describes exactly which language features are impacted, as well as the compilation model used for restricted environments. + +## Goals + +There are several goals of this new compilation mode: + +* **Eliminate the “large codesize cost of entry”** for Swift. Namely, the size of the supporting libraries (Swift runtime and standard library) must not dominate the binary size compared to application code. +* **Simplify the code generated by the compiler** to make it easier to reason about it, and about its performance and runtime cost. In particular, complex runtime mechanisms such as runtime generic instantiation are undesirable. +* **Allow effective and intuitive dead-code stripping** on application, library and standard library code. +* **Support environments with and without a dynamic heap**. Effectively, there will be two bottom layers of Swift, and the lower one, “non-allocating” Embedded Swift, will necessarily be a more restricted compilation mode (e.g. classes will be disallowed as they fundamentally require heap allocations) and likely to be used only in very specialized use cases. “Allocating” Embedded Swift should allow classes and other language facilities that rely on the heap (e.g. indirect enums). +* **Remove or reduce the amount of implicit runtime calls**. Specifically, runtime calls supporting heavyweight runtime facilities (type metadata lookups, generic instantiation) should not exist in Embedded Swift programs. Lightweight runtime calls (e.g. refcounting) are permissible but should only happen when the application uses a language feature that needs them (e.g. refcounted types). +* **Introduce a way of producing minimal statically-linked binaries** without external dependencies, namely without the need to link with a non-dead-strippable large Swift runtime/stdlib library, and without the need to link full libc and libc++ libraries. The Swift standard library contains essential facilities for writing Swift code, and must be available to write code against, but it should “fold” into the application and/or be intuitively dead-strippable. +* **Define a language subset, not a dialect.** Any code of Embedded Swift should always compile in regular Swift and behave the same. + * **The Embedded Swift language subset should stay very close to “full” Swift**, even if it means adding alternative ABIs to the compiler to support some of them. Users should expect minimal porting effort to get code to work in Embedded Swift. + +## Embedded Swift Language Subset + +In order to achieve the goals listed above, Embedded **Swift** will impose limitations on certain language features: + +* Library Evolution will be limited in some way, and there’s no expectation of ABI stability or separate distribution of libraries in binary form. +* Objective-C interoperability will not be available. C and C++ interoperability is not affected. +* Reflection and Mirrors APIs will not be available. +* The standard library’s print() function in its current form will not be available, and an alternative will be provided instead. +* Metatypes will be restricted in some way, and code patterns where a metatype value is actually needed at runtime will be disallowed (but at a minimum using a metatype function argument as a type hint will be allowed, as well as calling class methods and initializers on concrete types). + Examples: + ```swift + func foo(t: T.Type) { ... `t` used in a downcast ... } // not OK + extension UnsafeRawPointer { + func load(as type: T.Type) -> T { ... `type` unused ... } // OK + } + MyGenericClass.classFunc() // OK + ``` +* Existentials and dynamic downcasting of existentials will be disallowed. For example: + ```swift + func foo(t: Any.Type) {} // not OK + var e: any Comparable = 42 // not OK + var a: [Any] = [1, "string", 3.5] // not OK + ``` +* The types of thrown errors will be restricted in some manner, because thrown errors are of existential type `any Error` (which is disallowed by the prior item). +* Classes will have restrictions, for example they cannot have non-final generic functions. For example: + ```swift + class MyClass { + func member() { } // OK + func genericMember { } // not OK + } + ``` + It’s an open question whether class metatypes are allowed to be used as runtime values and whether classes will allow dynamic downcasting. +* KeyPaths will be restricted, but at a minimum it will be allowed to use keypath literals to form closures returning a field from a type, and it will be allowed to use keypaths that are compile-time references to inlined stored properties (so that `MemoryLayout.offset(of: ...)` will work on those). +* String APIs requiring Unicode data tables will be unavailable by default (to avoid paying the associated codesize cost), and will require opting in. For example, string iteration, comparing two strings, hashing a string, string splitting are features needing Unicode data tables. These operations should become available on UTF8View instead with the proposal to add Equatable and Hashable conformances to String views ([link](https://forums.swift.org/t/pitch-add-equatable-and-hashable-conformance-to-string-views/60449)). + +**Non-allocating Embedded Swift** will add further restrictions on top of the ones listed above: + +* Classes cannot be instantiated, indirect enums cannot be constructed. +* Escaping closures are not allowed. +* Standard library features and API that rely on classes, indirect enums, escaping closures are not available. This includes for example dynamic containers (arrays, dictionaries, sets) and strings. + +The listed restrictions (for both “allocating” and “non-allocating” Embedded Swift) are not necessarily fundamental, and we might be able to (fully or partially) lift some of them in the future, by adding alternative compile-time implementations (as opposed to their current runtime implementations) of the language features. + +## Implementation of Embedded Compilation Mode + +The following describes the high-level points in the approach to implement Embedded Swift in the compiler: + +* **Specialization is required on all uses of generics and protocols** at compile-time, and libraries are compiled in a way that allows cross-module specialization (into clients of the libraries). + * Required specialization (also known as monomorphization in other compilers/languages) needs type parameters of generic types and functions to always be compile-time known at the caller site, and then the compiler creates a specialized instantiation of the generic type/function that is no longer generic. The result is that the compiled code does not need access to any type metadata at runtime. + * This compilation mode will not support separate compilation of generics, as that makes specialization not possible. Instead, library code providing generic types and functions will be required to provide function bodies as serialized SIL (effectively, “source code”) to clients via the mechanism described below. +* **Library code is built as always inlinable and “emitIntoClient”** to support the specialization of generics/protocols in use sites that are outside of the library. + * **This applies to the standard library, too**, and we shall distribute the standard library built this way with the toolchain. + * This effectively provides the source code of libraries to application builds. +* **The need for type metadata at runtime is completely eliminated**, by further ignoring ABI stability, disabling resilience, and disallowing reflection mirrors APIs. Classes with subclasses get a simple vtable (similar to C++ virtual classes). Classes without subclasses become final and don’t need a vtable. Witness tables (which describe a conformance of a type to a protocol) are only used at compile-time and not present at runtime. + * **Type metadata is not emitted into binaries at all.** This causes code emitted by the compiler to become dead-strippable in the intuitive way given that metadata records (concretely type metadata, protocol conformance records, witness tables) are not present in compiler outputs. + * **Runtime facilities to process metadata are removed** (runtime generic instantiation, runtime protocol conformance lookups) because there is no metadata present at runtime. + +## Enabling Embedded Swift Mode + +The exact mechanics of turning on Embedded Swift compilation mode are an open question and subject to further discussion and refinement. There are different use cases that should be covered: + +* the entire platform / system is using Embedded Swift as a platform level decision +* a single component / library is built using Embedded Swift for an environment that otherwise has other code built with other compilation modes or compilers +* for testing purposes, it’s highly desirable to be able to build a library using Embedded Swift and then exercise that library with a test harness that is built with regular Swift + +A possible solution here would be to have a top-level compiler flag, e.g. `-embedded`, but we could also make environments default to Embedded Swift mode where it makes sense to do so, based on the target triple that’s used for the compilation. Specifically, the existing “none” OS already has the meaning of “baremetal environment”, and e.g. `-target arm64-apple-none` could imply Embedded Swift mode. + +Building firmware using `-target arm64-apple-none` would highlight that we’re producing binaries that are “independent“ and not built for any specific OS. The standard library will be pre-built in the baremetal mode and available in the toolchain for common set of CPU architectures. (It does not need to be built “per OS”.) + +To support writing code that’s compiled under both regular Swift and also Embedded Swift, we should provide facilities to manage availability of APIs and conditional compilation of code. The concrete syntax for that is subject to discussion, the following snippet is presented only as a straw-man proposal: + +```swift +@available(embedded, unavailable, "not available in Embedded Swift mode") +public func notAvailableOnEmbedded() + +#if !mode(embedded) +... code not compiled under Embedded Swift mode ... +#endif + +@available(noAllocations, unavailable, "not available in no allocations mode") +public func notAvailableInNonAllocatingMode() + +#if !mode(noAllocations) +... code not compiled under no allocations mode ... +#endif +``` + +## Dependencies of Embedded Swift Programs + +The expectation is that for “non-allocating” Embedded Swift, the user should only need a working Swift toolchain, and be able to pass a (set of) .swift file(s) to the compiler and receive a .o file that is just as simple to work with (e.g. to be linked into any library, app, firmware binary, etc.) as if it was produced by Clang on source code written in C: + +``` +$ swiftc *.swift -target arm64-apple-none -no-allocations -wmo -c -o a.o +$ nm -gm a.o +... shows no dependencies beyond memset/memcpy ... +memset +memcpy +``` + +A similar situation is expected even for "allocating" Embedded Swift, except that there will be a need for a small runtime library (significantly smaller compared to the existing Swift runtime written in C++) to support object instantiation and refcounting: + +``` +$ swiftc *.swift -target arm64-apple-none -wmo -c -o a.o +$ nm -gm a.o +... only very limited dependencies ... +malloc +calloc +free +swift_allocObject +swift_initStackObject +swift_initStaticObject +swift_retain +swift_release +``` + +The malloc/calloc/free APIs are expected to be provided by the platform. The Swift runtime APIs will be provided as an implementation that’s optimized for small codesize and will be available as a static library in the toolchain for common CPU architectures. Interestingly, it’s possible to write that implementation in “non-allocating” Baremetal Swift. diff --git a/visions/macros.md b/visions/macros.md new file mode 100644 index 0000000000..90917903c9 --- /dev/null +++ b/visions/macros.md @@ -0,0 +1,515 @@ +# A Vision for Macros in Swift + +As Swift evolves, it gains new language features and capabilities. There are different categories of features: some fill in gaps, taking existing syntax that is not permitted and giving it a semantics that fit well with the existing language, for example conditional conformance or allowing existential values for protocols with `Self` or associated type requirements. Others introduce new capabilities or paradigms to the language, such as the addition of concurrency, ownership types, or comprehensive reflection. + +There is another large category of language features that provide syntactic sugar to eliminate common boilerplate, taking something that can be written out in long-form and making it more concise. Such features don't technically add any expressive power to the language, because you can always write the long-form version, but their effect can be transformational if it enables use cases that would otherwise have been unwieldy. The synthesis of `Codable` conformances, for example, is sheer boilerplate reduction, but it makes `Codable` support ubiquitous throughout the Swift ecosystem. Property wrappers allow one to factor out logic for property access, and have enabled a breadth of powerful libraries. New language features in this category are hard to evaluate, because there is a fundamental question of whether the feature is "worth it": does the set of use cases made better by this feature outweigh the cost of making the language larger and more complicated? + + + +## Democratizing syntactic sugar with macros + +Macros are a feature present in a number of languages that allow one to perform some kind of transformation on the program's input source code to produce a different program. The mechanism of transformation varies greatly, from lexical expansion in C macros, to custom rules that rewrite one syntax into other syntax, to programs that arbitrarily manipulate the abstract syntax tree (AST) of the program. Macro systems exist in C, LISP, Scheme, Scala, Racket, Rust, and a number of other languages, and each design has its own tradeoffs. + +In all of these languages, macros have the effect of democratizing syntactic sugar. Many tasks that would have required a new language feature or an external source-generating tool could, instead, be implemented as a macro. Doing so has trade-offs: many more people can implement a macro than can take a feature through the language's evolution process, but the macro implementation will likely have some compromises---non-ideal syntax, worse diagnostics, worse compile-time performance. Overall, the hope is that a macro system can keep the language smaller and more focused, yet remain expressive because it is extensible enough to support libraries for many different domains. As a project, a macro system should reduce the desire for new syntactic-sugar features, leaving more time for more transformative feature work. Even in the cases where a new language feature is warranted, a macro system can allow more experimentation with the feature to best understand how it should work, and then be "promoted" to a full language feature once we've gained experience from the macro version. + +### Use cases for macros + +There are many use cases for macros, but before we look forward to the new use cases that become possible with macros, let's take a look backward at existing Swift language features that might have been macros had this feature existed before: + +* **`Codable`**: What we think of as `Codable` is mostly a library feature, including the `Encodable` and `Decodable` protocols and the various encoding and decoding implementations. The language part of `Codable` is in the synthesis of `init(from:)`, `encode(to:)`, and `CodingKeys` definitions for structs, enums, and classes. A macro that is given information about the stored properties of a type, and the superclass of a class type, could generate the same implementations---and would be easier to implement, improve, and reason about than a bespoke implementation in the compiler. Synthesis for `Equatable`, `Comparable`, and `Hashable` conformances are similar. +* **String interpolation**: String interpolation is implemented as a series of calls into the string interpolation "builder." While the actual parsing of a string interpolation and matching of it to a type that is `ExpressibleByStringInterpolation` is outside the scope of most macro systems, the syntactic transformation into a set of `appendXXX` calls on the builder is something that could be implemented via a macro. +* **Property wrappers**: Property wrappers are integrated into the language syntax via a custom attribute approach (e.g., `@Clamped(0, 100) var percent: Double`), but the actual implementation of the feature is entirely a syntactic transformation that introduces new properties (e.g., the backing storage property `_percent`) and adds accessors to existing properties (e.g., `percent`'s getter becomes `_percent.wrappedValue`). Other built-in language features like `lazy`, `willSet`, and `didSet` use similar syntactic transformations. +* **Result builders**: Result builders are also integrated into the language syntax via a custom attribute, but the actual transformation applied to a closure is entirely syntactic: the compiler introduces calls into the builder's `buildExpression`, `buildBlock`, `buildOptional`, and so on. That syntactic transformation could be expressed via some form of macro. + +### When is a language feature better than a macro? + +As noted above, a macro system has the potential to replace large parts of existing Swift language features, and enable many new ones. But a macro system is not necessarily a good replacement for a special-built feature: + +* A special-built feature might benefit from custom ABI rules. +* A special-built feature might benefit from analyses that would be infeasible to apply in a macro, such as those dependent on data or control flow. +* A special-built feature might be able to offer substantially better diagnostics. +* A special-built feature might be substantially more efficient to apply because it can rely on information and data structures already in the compiler. +* A special-built feature might have capabilities that we need to deny to macros, lest the mere possibility of a macro applying incur massive compile-time costs. + +The goal of a macro system should be to be general enough to cover a breadth of potential language features, while still providing decent tooling support and discouraging abuse that makes Swift code hard to reason about. + + + +## Design questions for macros + +At a very high level, a macro takes part of the program's source code at compile time and translates it into other source code that is then compiled into the program. There are three fundamental questions about the use of macros: + +* What kind of translation can the macro perform? +* When is a macro expanded? +* How does the compiler expand the macro? + +### What kind of translation can the macro perform? + +A program's source code goes through several different representations as it is compiled, and a macro system can choose at what point in this translation it operates. We consider three different possibilities: + +* **Lexical**: a macro could operate directly on the program text (as a string) or a stream of tokens, and produce a new stream of tokens. The inputs to such a macro would not even have to be valid Swift syntax, which might allow for arbitrary sub-languages to be embedded within a macro. C macros are lexical in nature, and most lexical approaches would inherit the familiar problems of C macros: tooling (such as code completion and syntax highlighting) cannot reason about the inputs to lexical macros, and it's easy for such a macro to produce ill-formed output that results in poor diagnostics. +* **Syntactic**: a macro could operate on a syntax tree and produce a new syntax tree. The inputs to such a macro would be a parsed syntax tree, which is strictly less flexible than a lexical approach because it means the macros can only operate within the bounds of the existing Swift grammar. However, this restriction means that tooling based on the grammar (such as syntax highlighting) would apply to the inputs without having to expand the macro, and macro-using Swift code would follow the basic grammatical structure of Swift. The output of a macro should be a well-formed syntax tree, which will be type-checked by the compiler and integrated into the program. +* **Semantic**: a macro could operate on a type-checked representation of the program, such as an Abstract Syntax Tree (AST) with annotations providing types for expressions, information about which specific declarations are referenced in a function call, any implicit conversions applied to expressions, and so on. Semantic macros have a wealth of additional information that is not provided to lexical or syntactic information, but unlike lexical or syntactic macros, their inputs are restricted to well-typed Swift code. This limits the ability of macros to change the meaning of the code provided to them, which can be viewed both as a negative (less freedom to implement interesting macros) or as a positive (less chance of a macro doing something that confounds the expectations of a Swift programmer). A semantic macro could be required to itself produce a well-typed Abstract Syntax Tree that is incorporated into the program. + +Whichever kind of translation we choose, we will need some kind of language or library that is suitable for working with the program at that level. A lexical translation needs to be able to work with program text, whereas a syntactic translation also needs a representation of the program's syntax tree. Semantic translation requires a much larger part of the compiler, including a representation of the type system and the detailed results of fully type-checked code. + +### When is a macro expanded? + +The expansion of a macro could be initiated in a number of ways, including explicit macro-expansion syntax in the source code or implicit macro expansions that might depend on type checker behavior, e.g., as part of a conversion. The best way for macro expansion to be initiated may depend on the kind of translation that the macro performs: the expansion of a purely lexical or syntactic macro probably needs to be explicitly marked in the source code, because it can change the program structure in surprising ways, whereas a semantic macro might be implicitly expanded as part of type checking because it's working in concert with the type checker. + +Swift already has a syntactic pattern that could be used for explicit macro expansion in the form of the `#` prefix, e.g., as a generalization of language features like `#filePath`, `#line`, `#colorLiteral(red: 0.292, green: 0.081, blue: 0.6, alpha: 255)`, and `#warning("unknown platform")`. The general syntax of `#macroName(macro-arguments)` could be used to expand a macro with the given name and arguments. Doing so provides a clear indication of where macros are used, and would support lexical and/or syntactic macros that need to alter the fundamental syntactic structure of their arguments. We refer to macros written with the prefix `#` syntax as *freestanding macros*, because they act as an expression, declaration, or statement on their own, depending on context. For example, one could build a "stringify" macro that produces both its argument value and also a string representation of the argument's source code: + +```swift +let (value, code) = #stringify(x + y) // produces a tuple containing the result of x + y, and the string "x + y" +``` + +Similarly, Swift's attribute syntax provides an extension point that is already used for features such as property wrappers and result builders. This attribute syntax could be used to expand a macro whose expansion depends on the entity to which the attribute is attached. Therefore, we call these *attached macros*, and they can do things such as create a memberwise initializer for a struct: + +```swift +@MemberwiseInit +struct Point { + var x: Double + var y: Double +} +``` + +A `MemberwiseInit` attached macro would need access to the stored properties of the type it is applied to, as well as the ability to create a new declaration `init(x:y:)`. Such a macro would have to tie in to the compiler at an appropriate point where stored properties are known but the set of initializers has not been finalized. + +A similar approach could be used to synthesize parts of conformances to a protocol. For example, one could imagine that one could write a declaration whose body is implemented by a macro, e.g., + +```swift +protocol Equatable { + @SynthesizeEquatable + func ==(lhs: Self, rhs: Self) -> Bool +} +``` + +The `@SynthesizeEquatable` attribute could trigger a macro expansion when a particular type that conforms to `Equatable` is missing a suitable implementation of `==`. It could access the stored properties in the type used for `Self` so it can synthesize an `==` whose body is, e.g., `lhs.x == rhs.x && lhs.y == rhs.y`. + +There are likely many other places where a macro could be expanded, and the key points for any of them are: + +* What are the macro arguments and how are they evaluated (tokenized, parsed, type-checked, etc.)? +* What other information is available to macro expansion? +* What can the macro produce (statements, expressions, declarations, attributes, etc.) and how is that incorporated into the resulting program? + +### How does the compiler expand the macro? + +The prior design sections have focused on what the inputs and outputs of a macro are and where macros can be triggered, but not *how* the macro operates. Again, there are a number of possibilities: + +* Macros could use a limited textual expansion mechanism, like the C preprocessor. +* Macros could provide a set of pattern-matching rewrite rules, to identify specific syntax and rewrite it into other syntax, like Rust's [`macro_rules!`](https://doc.rust-lang.org/rust-by-example/macros.html). +* Macros could be arbitrary (Swift) code that manipulates a representation of the program (source code, syntax tree, typed AST, etc.) and produces a new program, like [Scala 3 macros](https://docs.scala-lang.org/scala3/guides/macros/macros.html) , [LISP macros](https://lisp-journey.gitlab.io/blog/common-lisp-macros-by-example-tutorial/), or [Rust procedural macros](https://doc.rust-lang.org/reference/procedural-macros.html). + +These options are ordered roughly in terms of increasing expressive power, where the last is the most flexible because one can write arbitrary Swift code to transform the program. The first two options have the benefit of being able to easily evaluate within the compiler, because they are fundamentally declarative in nature. This means that any tool built on the compiler can show the results of expanding a macro, e.g., within an IDE. + +The last option is more complicated, because it involves running arbitrary Swift code. The Swift compiler could conceivably include a complete interpreter for the Swift language, and so long as all of the code that is used in the macro definition is visible to that interpreter (e.g., it does not reference any code for which Swift source is not available), the Swift compiler could interpret the macro definition to produce the expanded result. LISP macros effectively work this way, because LISP is interpreted and can treat the executing program as data. + +Alternatively, the macro definition could be compiled separately from the program that uses the macro, for example into a standalone executable or a compiler plugin. The compiler would then invoke that executable or plugin to perform macro expansion each time it is necessary. This approach is taken both by Scala (which uses the JVM's JIT facilities to be able to compile the macro definition and load it into the compiler) and Rust procedural macros (which use a [`proc-macro`](https://doc.rust-lang.org/reference/linkage.html) crate type for specifically this purpose). A significant benefit of this approach is that the full source code of the macro need not be available as Swift code (so one can use system libraries), macro expansion can be faster (because it's compiled code), and it's easy to test macro definitions outside of the compiler. On the other hand, it means having the Swift compiler run arbitrary code, which opens up questions about security and sandboxing that need to be considered. + +### (Un)hygienic macros + +A [hygienic macro](https://en.wikipedia.org/wiki/Hygienic_macro#cite_note-hygiene-3) is a macro whose expansion cannot change the binding of names used in its macro arguments. For example, imagine the given use of `myMacro`: + +```swift +let x = 3.14159 +let y = 2.71828 +#myMacro(x + y) +``` + +The expression `x + y` is type-checked, and `x` and `y` are bound to local variables immediately above. With a hygienic macro, nothing the macro does can change the declarations to which `x` and `y` are bound. A non-hygienic macro could change these bindings. For example, imagine the macro use above expanded to the following: + +```swift +{ + let x = 42 + return x + y +}() +``` + +Here, the macro introduced a new local variable named `x`. With a hygienic macro, the newly-declared `x` is not found by the `x` in `x + y`: it is a different declaration (or it is not permitted to be introduced). With a non-hygienic macro, the `x` in `x + y` will now refer to the local variable introduced by the macro. In this case, the macro expansion for a non-hygienic macro will fail to type-check because one cannot add an `Int` and a `Double`. + +Hygienic macros do make some macros harder (or impossible) to write, if the macro intentionally wants to take over some of the names used in its arguments. For example, if one wanted to have a macro intercept access to local variables to (say) record the number of times `x` was dynamically accessed. As such, systems that provide hygienic macros often have a way to intentionally provide names from the environment in which the macro is used, such as Racket's [syntax parameters](https://docs.racket-lang.org/reference/stxparam.html). + +A standard approach to dealing with the problem of unintentional name collision in an unhygienic macro is to provide a way to generate unique names within the macro implementation. This approach has been used in LISP macros for decades via [`gensym`](http://clhs.lisp.se/Body/f_gensym.htm), and requires some discipline on the part of the macro implementer to create unique names whenever the macro creates a new declaration. + +## An approach to macros in Swift + +Based on the menu of design choices above, we propose a macro approach characterized by syntactic translation on already-type-checked Swift code that is implemented via a separate package. The intent here is to allow macros a lot of flexibility to implement interesting transformations, while not giving up the benefits of type-checking the code that the user wrote prior to translation. + +### Macro declarations + +A macro declaration indicates how the macro can be used in source code, much like a function declaration indicates the arguments and result type of a function. We declare macros as a new kind of entity, introduced with `macro`, that indicates that it's a macro definition and provides additional information about the interface to the macro. For example, consider the `stringify` macro described early, which could be defined as follows: + +```swift +@freestanding(expression) +macro stringify(_ value: T) -> (T, String) = #externalMacro(module: "MyMacros", type: "StringifyMacro") +``` + +The `macro` introducer indicates that this is a macro, named `stringify`. The `@freestanding` attribute notes that this is a freestanding macro (used with the `#` syntax) and that it is usable as an expression. The macro is defined (after the `=`) to have an externally-provided macro expansion operation that is the type named `MyMacros.StringifyMacro`. Because the definition is external, the `stringify` macro function doesn't need a function body. If Swift were to grow a way to implement macros directly here (rather than via a separate package), such macros could have a body but not an `#externalMacro` argument. + +A given macro can inhabit several different macro *roles*, each of which can expand in different ways. For example, consider a `Clamping` macro that implements behavior similar to the [property wrapper by the same name](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0258-property-wrappers.md#clamping-a-value-within-bounds): + +```swift +@attached(peer, prefixed(_)) +@attached(accessor) +macro Clamping(min: T, max: T) = #externalMacro(module: "MyMacros", type: "ClampingMacro") +``` + +The attribute specifies that this is an attached macro, so it can be used as an attribute as, e.g., + +```swift +@Clamping(min: 0, max: 255) var red: Int = 127 +``` + +The `Clamping` macro would be expanded in two different but complementary ways: + +* A *peer* declaration `_red` that provides the backing storage: + + ```swift + private var _red: Int = 127 + ``` + +* A set of *accessor*s that guard access to this storage, turning the `red` property into a computed property: + + ```swift + get { _red } + + set(__newValue) { + let __minValue = 0 + let __maxValue = 255 + if __newValue < __minValue { + _red = __minValue + } else if __newValue > __maxValue { + _red = __maxValue + } else { + _red = __newValue + } + } + ``` + + +### Macro definitions via a separate program + +Macro definitions would be provided in a separate program that performs a syntactic transformation. A macro definition would be implemented using [swift-syntax](https://github.com/apple/swift-syntax), by providing a type that conforms to one of the "macro" protocols in a new library, `SwiftSyntaxMacros`. For example, the `MyMacros` package we're using as an example might look like this: + +```swift +import SwiftDiagnostics +import SwiftParser +import SwiftSyntax +import SwiftSyntaxBuilder +import SwiftSyntaxMacros + +public struct StringifyMacro: ExpressionMacro { + static func expansion( + of node: some FreestandingMacroExpansionSyntax, + in context: some MacroExpansionContext + ) -> ExprSyntax { + guard let argument = node.argumentList.first?.expression else { + fatalError("compiler bug: the macro does not have any arguments") + } + + return "(\(argument), \(literal: argument.description))" + } +} +``` + +Conformance to `ExpressionMacro` indicates a macro definition for an expression macro, and corresponds to `@freestanding(expression)`. There will be several protocols, corresponding to the various roles that macros inhabit. Each protocol has an `expansion` method that will be called with the syntax nodes that are involved in the macro expansion, along with a `context` instance that provides more information about how the macro is being invoked. + +The implementation of these functions makes extensive use of Swift syntax manipulation via the `swift-syntax` package. The inputs and outputs are in terms of syntax nodes: `ExprSyntax` describes the syntax for any kind of expression in Swift, whereas `MacroExpansionExprSyntax` is the syntax for an explicitly-written macro expansion. The `expansion` operation will return a new syntax node that will replace the ones it was given in the program. We use string interpolation as a form of quasi-quoting: the return of `StringifyMacro.expansion` forms a tuple `(\(argument), "\(literal: argument.description)")` where the first argument is the expression itself and the second is the source code translated into a string literal. The resulting string will be parsed into an expression that is returned to the compiler. + +Macro implementations are "host" programs that are completely separate from the program in which macros are used. This distinction is most apparent in cross-compilation scenarios, where the host platform (where the compiler is run) differs from the target platform (where the compiled program will run), and these could use different operating systems and processor architectures. Macro implementations are compiled for and executed on the host platform, whereas the results of expanding a macro will be compiled for and executed on the target platform. Therefore, macro implementations are defined as their own kind of target in the Swift package manifest. For example, a package that ties together the macro declaration for `#stringify` and its implementation as `StringifyMacro` follows: + +```swift +import PackageDescription +import CompilerPluginSupport + +let package = Package( + name: "MyMacros", + dependencies: [ + .package( + url: "https://github.com/apple/swift-syntax.git", + branch: "main" + ), + ], + targets: [ + // Macro implementation target contains the StringifyMacro type. + // Always built for the host platform. + .macro(name: "StringifyImpl", + dependencies: [.product(name: "SwiftSyntaxMacros", package: "swift-syntax")]), + + // Library target provides the macro declaration (public macro stringify) that is + // used by client code. + // Built for the target platform. + .target(name: "StringifyLib", dependencies: ["StringifyImpl"]), + + // Clients of the macro will depend on the library target. + .executableTarget(name: "StringifyClient", dependencies: ["StringifyLib"]), + ] +) +``` + +Conceptually, the separation of `macro` targets into separate programs (for the host platform) from other targets (for the target platform) means that the individual macros could be built completely separately from each other and from other targets, even if they happen to be for the same platform. In the extreme, this could mean that each `macro` would be allowed to build against a different version of `swift-syntax`, and other targets could choose to also use `swift-syntax` with a different version. Given that `swift-syntax` is modeling the Swift language (which evolves over time), it does not promise a stable API, so having the ability to have different macro implementations depend on different versions of `swift-syntax` is a feature: it would prevent conflicting version requirements in macros from causing problems. + +Note that this separation of dependencies for distinct targets is currently not possible in the Swift Package Manager. In the interim, macro implementations will need to adapt to be built with different versions of the `swift-syntax` package. + +#### Diagnostics + +A macro implementation can be used to produce diagnostics (e.g., warnings and errors) to indicate problems encountered during macro expansion. The `stringify` macro described above doesn't really have a failure case, but imagine an `#embed("somefile.txt")` macro that takes the contents of a file at build time and turns them into an array of bytes. The macro could have several different failure modes: + +* The macro argument isn't a string literal, so it doesn't know what the file name is. +* The file might not be available for reading because it is missing, inaccessible, etc. + +These failures would be reported as errors by providing [`Diagnostic`](https://github.com/apple/swift-syntax/blob/main/Sources/SwiftDiagnostics/Diagnostic.swift) instances to the context that specify the underlying problem. The diagnostics would refer to the syntax nodes provided to the macro definition, and the compiler would provide those diagnostics to the user. + +In its limit, a macro might perform no translation whatsoever on the syntax tree it is given, but instead be there only to provide diagnostics---for example, as a context-specific, custom lint-like rule that enforces additional constraints on the program. + +### Macro roles + +The `@freestanding` and `@attached` attributes for macro declarations specify the roles that the macro can inhabit, each of which corresponds to a different place in the source code where the macro can be expanded. Here is a potential set of roles where macro expansion could be warranted. The set of roles could certainly grow over time to enable new capabilities in the language: + +* **Expression**: A freestanding macro that can occur anywhere that an expression can occur, and must produce an expression. `#colorLiteral` could fall into this category: + + ```swift + // In library + @freestanding(expression) + macro colorLiteral(red: Double, green: Double, blue: Double, alpha: Double) -> _ColorLiteralType = + #externalMacro(module: "MyMacros", type: "ColorLiteral") + + // In macro definition package + public struct ColorLiteral: ExpressionMacro { + public static func expansion( + expansion: MacroExpansionExprSyntax, + in context: MacroExpansionContext + ) -> ExprSyntax { + return ".init(\(expansion.argumentList))" + } + } + ``` + + With this, an expression like `#colorLiteral(red: 0.5, green: 0.5, blue: 0.25, alpha: 1.0)` would produce a value of the `_ColorLiteralType` (presumably defined by a framework), and would be rewritten by the macro into `.init(red: 0.5, green: 0.5, blue: 0.25, alpha: 1.0)` and type-checked with the `_ColorLiteralType` as context so it would initialize a value of that type. + +* **Declaration**: A freestanding macro that can occur anywhere a declaration can occur, such as at the top level, in the definition of a type or extension thereof, or in a function or closure body. The macro can expand to zero or more declarations. These macros could be used to subsume the `#warning` and `#error` directives from [SE-0196](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0196-diagnostic-directives.md): + + ```swift + /// Emits the given message as a warning, as in SE-0196. + @freestanding(declaration) macro warning(_ message: String) + + /// Emits the given message as an error, as in SE-0196. + @freestanding(declaration) macro error(_ message: String) + ``` + +* **Code item**: A freestanding macro that can occur within a function or closure body and can produce a mix of zero or more statements, expressions, and declarations. + +* **Accessor**: An attached macro that adds accessors to a stored property or subscript, as shown by the `Clamping` macro example earlier. The inputs would be the arguments provided to the macro in the attribute, along with the property or subscript declaration to which the accessors will be attached. The output would be a set of accessor declarations, i.e., a getter and setter. A `Clamping` macro could be implemented as follows: + + ```swift + extension ClampingMacro: AccessorMacro { + static func expansion( + of node: CustomAttributeSyntax, + providingAccessorsOf declaration: DeclSyntax, + in context: MacroExpansionContext + ) throws -> [AccessorDeclSyntax] { + let originalName = /* get from declaration */, + minValue = /* get from custom attribute node */, + maxValue = /* get from custom attribute node */ + let storageName = "_\(originalName)", + newValueName = context.getUniqueName(), + maxValueName = context.getUniqueName(), + minValueName = context.getUniqueName() + return [ + """ + get { \(storageName) } + """, + """ + set(\(newValueName)) { + let \(minValueName) = \(minValue) + let \(maxValueName) = \(maxValue) + if \(newValueName) < \(minValueName) { + \(storageName) = \(minValueName) + } else if \(newValueName) > \(maxValueName) { + \(storageName) = \(maxValueName) + } else { + \(storageName) = \(newValueName) + } + } + """ + ] + } + } + ``` + +* **Witness**: An attached macro that can be expanded to provide a "witness" that satisfies a requirement of a protocol for a given concrete type's conformance to that protocol. Such a macro would take as input the conforming type, the protocol, and a declaration (without a body) that will be created in the conforming type. The output would be that declaration with a body added and (potentially) other modifications. For this to work well, we would almost certainly need to expose a lot of information about the conforming type, such as the set of stored properties. Assuming that exists, let's implement the `synthesizeEquatable` macro referenced earlier in this document: + + ```swift + // In the standard library + @attached(witness) + macro SynthesizeEquatable() = #externalMacro(module: "MyMacros", type: "EquatableSynthesis") + + protocol Equatable { + @SynthesizeEquatable + static func ==(lhs: Self, rhs: Self) -> Bool + } + + // In the macro definition library + struct EquatableSynthesis: AttachedMacro { + /// Expand a macro described by the given custom attribute to + /// produce a witness definition for the requirement to which + /// the attribute is attached. + static func expansion( + of node: CustomAttributeSyntax, + witness: DeclSyntax, + conformingType: TypeSyntax, + storedProperties: [StoredProperty], + in context: MacroExpansionContext + ) throws -> DeclSyntax { + let comparisons: [ExprSyntax] = storedProperties.map { property in + "lhs.\(property.name) == rhs.\(property.name)" + } + let comparisonExpr: ExprSyntax = comparisons.map { $0.description }.joined(separator: " && ") + return witness.withBody( + """ + { + return \(comparisonExpr) + } + """ + ) + } + } + ``` + +* **Member**: An attached macro that can be applied on a type or extension that expands to one or more declarations that will be inserted as members into that type or extension. As with a conformance macro, a member macro would probably want access to the stored properties of the enclosing type, and potentially other information. As an example, let's create a macro to synthesize a memberwise initializer: + + ```swift + // In the standard library + @attached(member) + macro memberwiseInit(access: Access = .public) = #externalMacro(module: "MyMacros", type: "MemberwiseInit") + + // In the macro definition library + struct MemberwiseInit: MemberMacro { + static func expansion( + of node: AttributeSyntax, + attachedTo declaration: DeclSyntax, + in context: inout MacroExpansionContext + ) throws -> [DeclSyntax] {} + let parameters: [FunctionParameterSyntax] = declaration.storedProperties.map { property in + let paramDecl: FunctionParameterSyntax = "\(property.name): \(property.type)" + guard let initializer = property.initializer else { + return paramDecl + } + return paramDecl.withDefaultArgument( + InitializerClauseSyntax( + equal: TokenSyntax(.equal, presence: .present), + value: "\(initializer)" + ) + ) + } + + let assignments: [ExprSyntax] = conformingType.storedProperties.map { property in + "self.\(property.name) = \(property.name)" + } + + return + #""" + public init(\(parameters.map { $0.description }.joined(separator: ", "))) { + \(assignments.map { $0.description }.joined(separator: "\n")) + } + """# + } + } + ``` + + Using this macro on a type, e.g., + + ```swift + @MemberwiseInit + class Point { + var x, y: Int + var z: Int = 0 + } + ``` + + would produce code like the following: + + ```swift + public init(x: Int, y: Int, z: Int = 0) { + self.x = x + self.y = y + self.z = z + } + ``` + +* **Body**: A body macro would allow one to create or replace the body of a function, initializer, or closure through syntactic manipulation. Body macros are attached to one of these entities, e.g., + + ```swift + @Traced(logLevel: 2) + func myFunction(a: Int, b: Int) { ... } + ``` + + where the `Traced` macro is declared as something like: + + ```swift + @attached(body) macro Traced(logLevel: Int = 0) + ``` + + and can introduce new code into the body to, e.g., perform logging. + +* **Conformance**: Conformance macros could introduce protocol conformances to the type or extension to which they are attached. For example, this could be useful when composed with macro roles that create other members, such as a macro that both adds a protocol conformance and also a stored property required by that conformance. + +## Tools for using and developing macros + +Macros introduce novel problems for tooling, because the macro expansion process replaces (or augments) code that is explicitly written with other source code that makes it into the final program. The design of a macro system has a large impact on the ability to build good tools, and a poor design can directly impact discoverability, predictability, debuggability, and compile-time efficiency. C macros demonstrate nearly all of these problems: + +* C macros are bare identifiers that can be used anywhere in source code, so it is hard to discover where macros are being applied in the source code. C programmers have adopted the `UPPERCASE_MACRO_NAME` convention to try to understand which names are macros and which aren't. +* C macros can expand to an arbitrary sequence of tokens in a manner that destroys program structure, for example, one can close a `struct` or function definition with a C macro, making it hard to predict the scope of effects a macro can have. +* C macros are expanded via logic within the compiler's preprocessor, and therefore offer no debugging capabilities. The only way to see the effect of a macro is to generate preprocessed output for an entire translate unit, then inspect the original code. +* C macros are rarely persisted after a program is built, so debugging a program that has made heavy use of macros requires one to manually map between the original source code (pre-macro) and the generated machine code, with no record of the expansion itself. + +The design proposed here for Swift makes it possible to build good tooling despite the challenges macros pose: + +* Uses of macros are indicated in the source (with `#` or `@`) to make the use of macros clear. +* Expansions of macros have their effects restricted to the scope in which the macro is used (e.g., augmenting or adding declarations locally), and any effects visible from other parts of the program are declared up front by the macro (e.g., the names it introduces), so one can reason about the effects of a macro expansion. +* Implementations of macros are normal Swift programs, so they can be developed and tested using the normal tools for Swift. Much of the development and testing of a macro can be done outside of the compiler, with unit tests that (for example) test the syntactic transformation on isolated examples that translate Swift code into different Swift code. +* The localized nature of macro effects, and the fact that all macro-expanded code is itself normal Swift code, make it possible to record the results of macro expansion in a way that can reconstitute the effects of macro expansion without rerunning the compiler, allowing useful debugging and diagnostics flows. + +Early implementations of Swift macros already provide macro-expansion information in a manner that is amenable to existing tooling. For example, the result of expanding a macro fits into several existing workflows: + +* If a warning or error message refers into code generated by macro expansion, the compiler writes the macro-expanded code into a separate Swift file and refers to that file within the diagnostic message. Users can open that file to see the results of that macro expansion to understand the problem. Each such diagnostic provides a stack of notes that refers back to the point where the macro expansion occurred, which may itself be within other macro-expanded code, all the way back to the original source code that triggered the outermost macro expansion. For example: + + ``` + /tmp/swift-generated-sources/@__swiftmacro_9MacroUser14testAddBlocker1a1b1c2oaySi_S2iAA8OnlyAddsVtF03addE0fMf1_.swift:1:4: error: binary operator '-' cannot be applied to two 'OnlyAdds' operands + oa - oa + ~~ ^ ~~ + macro_expand.swift:200:7: note: in expansion of macro 'addBlocker' here + _ = #addBlocker(oa + oa) + ^~~~~~~~~~~~~~~~~~~~ + ``` + +* Debug information for macro-expanded code uses a similar scheme to diagnostics, allowing one to see the macro-expanded code while debugging (e.g., in a backtrace), set breakpoints in it, step through it, and so on. Freestanding macros are treated like inline functions, so one can "step into" a macro expansion from the place it occurs in the source code. + +* SourceKit, which is used for IDE integration features, provides a new "Expand Macro" refactoring that can expand any single use of a macro right in the source code. This can be used both to see the effects of the macro, as well as to eliminate the use of the macro in favor of (say) a customized version where the macro is used as a starting point. + +* Macro implementations can be tested using `swift-syntax` and existing testing tooling. For example, the following test checks the expansion behavior of the `stringify` macro by performing syntactic expansion on the input source code (`sf`) and checking that against the expected result of expansion (in the `XCTAssertEqual` at the end): + + ```swift + final class MyMacroTests: XCTestCase { + func testStringify() { + let sf: SourceFileSyntax = + #""" + let a = #stringify(x + y) + let b = #stringify("Hello, \(name)") + """# + let context = BasicMacroExpansionContext.init( + sourceFiles: [sf: .init(moduleName: "MyModule", fullFilePath: "test.swift")] + ) + let transformedSF = sf.expand(macros: ["stringify": StringifyMacro.self], in: context) + XCTAssertEqual( + transformedSF.description, + #""" + let a = (x + y, "x + y") + let b = ("Hello, \(name)", #""Hello, \(name)""#) + """# + ) + } + } + ``` + +All of the above work with existing tooling, creating a baseline development experience that provides discoverability, predictability, and debuggability. Over time, more tooling can be made aware of macros to provide a more polished experience. diff --git a/visions/memory-safety.md b/visions/memory-safety.md new file mode 100644 index 0000000000..69a206bc79 --- /dev/null +++ b/visions/memory-safety.md @@ -0,0 +1,255 @@ +# Optional Strict Memory Safety for Swift + +Swift is a memory-safe language *by default* , meaning that the major language features and standard library APIs are memory-safe. However, it is possible to opt out of memory safety when it’s pragmatic using certain “unsafe” language or library constructs. This document proposes a path toward an optional “strict” subset of Swift that prohibits any unsafe features. This subset is intended to be used for Swift code bases where memory safety is an absolute requirement, such as security-critical libraries. + +This document is an official feature [vision document](https://forums.swift.org/t/the-role-of-vision-documents-in-swift-evolution/62101). The Language Steering Group has endorsed the goals and basic approach laid out in this document. This endorsement is not a pre-approval of any of the concrete proposals that may come out of this document. All proposals will undergo normal evolution review, which may result in rejection or revision from how they appear in this document. + +## Introduction + +[Memory safety](https://en.wikipedia.org/wiki/Memory_safety) is a popular topic in programming languages nowadays. Essentially, memory safety is a property that prevents programmer errors from manifesting as [undefined behavior](https://en.wikipedia.org/wiki/Undefined_behavior) at runtime. Undefined behavior effectively breaks the semantic model of a language, with unpredictable results including crashes, data corruption, and otherwise-impossible program states. Much of the recent focus on memory safety is motivated by security, because memory safety issues offer a fairly direct way to compromise a program: in fact, the lack of memory safety in C and C++ has been found to be the root cause for ~70% of reported security issues in various analyses [[1](https://msrc.microsoft.com/blog/2019/07/a-proactive-approach-to-more-secure-code/)][[2](https://www.chromium.org/Home/chromium-security/memory-safety/)]. + +### Memory safety in Swift + +While there are a number of potential definitions for memory safety, the one provided by [this blog post](https://security.apple.com/blog/towards-the-next-generation-of-xnu-memory-safety/) breaks it down into five dimensions of safety: + +* **Lifetime safety** : all accesses to a value are guaranteed to occur during its lifetime. Violations of this property, such as accessing a value after its lifetime has ended, are often called use-after-free errors. +* **Bounds safety**: all accesses to memory are within the intended bounds of the memory allocation, such as accessing elements in an array. Violations of this property are called out-of-bounds accesses. +* **Type safety** : all accesses to a value use the type to which it was initialized, or a type that is compatible with that type. For example, one cannot access a `String` value as if it were an `Array`. Violations of this property are called type confusions. +* **Initialization safety** : all values are initialized prior to being used, so they cannot contain unexpected data. Violations of this property often lead to information disclosures (where data that should be invisible becomes available) or even other memory-safety issues like use-after-frees or type confusions. +* **Thread safety:** all values are accessed concurrently in a manner that is synchronized sufficiently to maintain their invariants. Violations of this property are typically called data races, and can lead to any of the other memory safety problems. + +Since its inception, Swift has provided memory safety for the first four dimensions. Lifetime safety is provided for reference types by automatic reference counting and for value types via [memory exclusivity](https://www.swift.org/blog/swift-5-exclusivity/); bounds safety is provided by bounds-checking on `Array` and other collections; type safety is provided by safe features for casting (`as?` , `is` ) and `enum` s; and initialization safety is provided by “definite initialization”, which doesn’t allow a variable to be accessed until it has been defined. Swift 6’s strict concurrency checking extends Swift’s memory safety guarantees to the last dimension. + +Providing memory safety does not imply the absence of run-time failures. Good language design often means defining away runtime failures in the type system. However, memory safely requires only that an error in the program cannot be escalated into a violation of one of the safety properties. For example, having reference types be non-nullable by default defines away most problems with NULL pointers. With explicit optional types, the force-unwrap operator (postfix `!` ) meets the definition of memory safety by trapping at runtime if the unwrapped optional is `nil` . The standard library also provides the [`unsafelyUnwrapped` property](https://developer.apple.com/documentation/swift/optional/unsafelyunwrapped) that does not check for `nil` in release builds: this does not meet the definition of memory safety because it admits violations of initialization and lifetime safety that could be exploited. + +### Unsafe code + +Swift is a memory-safe language *by default* , meaning that the major language features and standard library APIs are memory-safe. However, there exist opt-outs that allow one to write memory-unsafe code in Swift: + +* Language features like `unowned(unsafe)` and `nonisolated(unsafe)` that disable language safety features locally. +* Library constructs like `UnsafeMutableBufferPointer` or `unsafeBitCast(to:)` that provide lower-level access than existing language constructs provide. +* Interoperability with C-family APIs, which are implemented in a non-memory-safe language and tend to traffic in unsafe pointer types. + +The convention of using `unsafe` or `unchecked` in the names of unsafe constructs works fairly well in practice: memory-unsafe code in Swift tends to sticks out because of the need for `withUnsafe<...>` operations, and for large swaths of Swift code there is no need to reach down for the unsafe APIs. + +However, the convention is not entirely sufficient for identifying all Swift code that makes use of unsafe constructs. For example, it is possible to call the C `memcpy` directly from Swift as, e.g., `memcpy(&to, &from, numBytes)` , which can easily violate memory-safety along any dimension: `to` and `from` might be arrays with incompatible types, the number of bytes might be incorrect, etc. However, “unsafe” or “unchecked” do not appear in this code except as the (unseen) type of the parameters to `memcpy` . + +Moreover, some tasks require lower-level access to memory that is only expressible today via the unsafe pointer types, meaning that one must choose between using only safe constructs, or having access to certain APIs and optimizations. For example, all access to contiguous memory requires an `UnsafeMutableBufferPointer` , which compromises on both lifetime and bounds safety. However, it fulfills a vital role for various systems-programming tasks, including interacting directly with specialized hardware or using lower-level system libraries written in the C family. + +## Strictly-safe subset of Swift + +Swift’s by-default memory safety is a pragmatic choice that provides the benefits of memory safety to most Swift code while not requiring excessive ceremony for those places where some code needs to drop down to use unsafe constructs. However, there are code bases where memory safety is more important than programmer convenience, such as in security-critical subsystems handling untrusted data or that are executing with elevated privileges in an OS. + +For such code bases, it’s important to ensure that the code is staying within the strictly-safe subset of Swift. This can be accomplished with a compiler option that produces an error for any use of unsafe code, whether it’s an unsafe language feature or unsafe library construct. Any code written within this strictly-safe subset also works as “normal” Swift and can interoperate with existing Swift code. + +The compiler would flag any use of the following unsafe language features: + +* `@unchecked Sendable` +* `unowned(unsafe)` +* `nonisolated(unsafe)` +* `unsafeAddressor`, `unsafeMutableAddressor` + +In addition, an `@unsafe` attribute would be added to the language and would be used to mark any declaration that is unsafe to use. In the standard library, the following functions and types would be marked `@unsafe` : + +* `Unsafe(Mutable)(Raw)(Buffer)Pointer` +* `(Closed)Range.init(uncheckedBounds:)` +* `OpaquePointer` +* `CVaListPointer` +* `Unmanaged` +* `unsafeBitCast`, `unsafeDowncast` +* `Optional.unsafelyUnwrapped` +* `UnsafeContinuation`, `withUnsafe(Throwing)Continuation` +* `UnsafeCurrentTask` +* `Mutex`'s `unsafeTryLock`, `unsafeLock`, `unsafeUnlock` +* `VolatileMappedRegister.init(unsafeBitPattern:)` +* The `subscript(unchecked:)` introduced by the `Span` proposal. + +Any use of these APIs would be flagged by the compiler as a use of an unsafe construct. In addition to the direct `@unsafe` annotation, any API that uses an `@unsafe` type is considered to itself be unsafe. This includes C-family APIs that use unsafe types, such as the aforementioned `memcpy` that uses `Unsafe(Mutable)RawPointer` in its signature: + +```swift +func memcpy( + _: UnsafeMutableRawPointer?, + _: UnsafeRawPointer?, + _: Int +) -> UnsafeMutableRawPointer? +``` + +The rules described above make it possible to detect and report the use of unsafe constructs in Swift. + +An `@unsafe` function is allowed to use other unsafe constructs. As such, a Swift module compiled in the strictly-safe subset can contain both safe and unsafe code, but all unsafe code is marked by `@unsafe`. A client of the module can opt to use only the safe parts of that module, potentially using the strict safety checking to ensure this. + +### Wrapping unsafe behavior in safe APIs + +There should also be a way to wrap unsafe behavior into safe APIs. For example, the standard library's `Array` and [`Span`](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md) are necessarily implemented from unsafe primitives, such as `UnsafeRawPointer`, but expose primarily safe APIs. For example, the `Span` type could be defined like this: + +```swift +public struct Span: ~Escapable, Copyable, BitwiseCopyable { + internal let buffer: UnsafeBufferPointer +} +``` + +The subscript operation is safe, but necessarily uses `buffer`, which has an `@unsafe` type. Its implementation must acknowledge that it is using unsafe constructs internally, but that it does so in a manner that preserves safety for clients. There are several potential syntaxes, including an `unsafe { ... }` code block, which could look like this: + +```swift +public subscript(_ position: Int) -> Element { + get { + unsafe { + precondition(position >= 0 && position < buffer.count) + return buffer[position] + } + } +} +``` + +Alternatively, Swift could provide a `@safe(unchecked)` attribute that states that a particular API is safe, but that its safety cannot be checked by the compiler, akin to `@unchecked Sendable` conformances: + +```swift +public subscript(_ position: Int) -> Element { + @safe(unchecked) get { + precondition(position >= 0 && position < buffer.count) + return buffer[position] + } +} +``` + +The specific syntax chosen will be the subject of a specific proposal, and need not be determined by this document. Regardless, a Swift module that enables strict safety checking must limit its use of unsafe constructs to `@unsafe` declarations or those parts of the code that have acknowledged local use of unsafe constructs. + +### Auditability + +The aim of optional strict memory safety for Swift is to make it possible to write Swift that avoids unintentional use of unsafe constructs while not preventing their use entirely. To aid projects that wish to set a higher bar for memory safety, such as permitting no unsafe constructs outside of the standard library or requiring additional code review for any uses of unsafe constructs, Swift tooling should provide a way to audit the uses of unsafe constructs within an entire project (including its dependencies). An auditing tool should be able to identify and report Swift modules that were compiled without strict memory safety as well as all of the places where the opt-out mechanism (e.g., `unsafe { ... }` blocks or `@safe(unchecked)`) is used in modules that do opt in to strict memory safety. + +## Improving the expressibility of strictly-safe Swift + +The following sections describe language features and library constructs that improve on what can be expressed within the strictly-safe subset of Swift. These improvements will also benefit Swift in general, making it easier to correctly work with contiguous memory and interoperate with APIs from the C-family on languages. + +### Accessing contiguous memory + +Nearly every “unsafe” language feature and standard library API described in the previous section already has safe counterparts in the language: safe concurrency patterns via actors and `Mutex` , safe casting via `as?` , runtime-checked access to optionals (via `!` ) and continuations (`withChecked(Throwing)Continuation` ), and so on. + +One of the primary places where this doesn’t hold is with low-level access to contiguous memory. Even with `ContiguousArray` , which stores its elements contiguously, the only way to access elements is either one-by-one (e.g., subscripting) or to use an operation like `withUnsafeBufferPointer` that provides temporary access the storage via an `Unsafe(Mutable)BufferPointer` argument to a closure. These APIs are memory-unsafe along at least two dimensions: + +* **Lifetime safety**: the unsafe buffer pointer should only be used within the closure, but there is no checking to establish that the pointer does not escape the closure. If it does escape, it could be used after the closure has returned and the pointer could have effectively been “freed.” +* **Bounds safety**: the unsafe buffer pointer types do not perform bounds checking in release builds. + +[Non-escapable types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0446-non-escapable.md) provide the ability to create types whose instances cannot escape out of the context in which they were created with no runtime overhead. Non-escapable types allow the creation of a [memory-safe counterpart to the unsafe buffer types](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md), `Span` . With `Span` , it becomes possible to access contiguous memory in an array in a manner that maintains memory safety. For example: + +```swift +let span = myInts.span + +globalSpan = span // error: span value cannot escape the scope of myInts +print(span[myInts.count]) // runtime error: out-of-bounds access +print(span.first ?? 0) +``` + +[Lifetime dependencies](https://github.com/swiftlang/swift-evolution/pull/2305) can greatly improve the expressiveness of non-escaping types, making it possible to build more complex data structures while maintaining memory safety. + +### Expressing memory-safe interfaces for the C family of languages + +The C family of languages do not provide memory safety along any of the dimensions described in this document. As such, a Swift program that makes use of C APIs is never fully “memory safe” in the strict sense, because any C code called from Swift could undermine the memory safety guarantees Swift is trying to provide. Requiring that all such C code be rewritten in Swift would go against Swift’s general philosophy of incremental adoption into existing ecosystems. Therefore, this document proposes a different strategy: code written in Swift will be auditably memory-safe so long as the C APIs it uses follow reasonable conventions with respect to memory safety. As such, writing new code (or incrementally rewriting code from the C family) will not introduce new memory safety bugs, so that adopting Swift in an existing code base will incrementally improve on memory safety. This approach is complementary to any improvements made to memory safety within the C family of languages, such as [bounds-safety checks for C](https://clang.llvm.org/docs/BoundsSafety.html) or [C++ standard library hardening](https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2024/p3471r0.html). + +In the C family of languages, the primary memory safety issue for APIs is the widespread use of pointers that have neither lifetime annotations (who owns the pointer?) nor bounds annotations (how many elements does it point to?). As such, the pointers used in C APIs are reflected in Swift as unsafe pointer types, as shown above with `memcpy` . + +Despite the lack of this information, C APIs often follow a reasonable set of conventions that make them usable in Swift without causing memory-safety problems. Swift has a long history of utilizing annotations in C headers to describe these conventions and improve the projection of C APIs into Swift, including: + +* Nullability annotations (`_Nullable`, `_Nonnull`) that describe what values can be NULL, and affects whether a C type is reflected as optional in Swift. +* Non-escaping annotations (e.g., `__attribute__((noescape))`) on block pointer parameters, which results in them being imported as non-escaping function parameters. +* `@MainActor` and `Sendable` annotations on C APIs that support Swift 6’s data-race safety model. + +To provide safer interoperability with C APIs, additional annotations can be provided in C that Swift can use to project those C APIs into Swift APIs without any use of unsafe pointers. For example, the Clang [bounds-safety attributes](https://clang.llvm.org/docs/BoundsSafety.html) allow one to express when a C pointer’s size is described by another value: + +```cpp +double average(const double *__counted_by(N) ptr, int N); +``` + +Today, this function would be projected into a Swift function like the following: + +```swift +/*@unsafe*/ func average(_ ptr: UnsafePointer!, _ N: CInt) -> Double +``` + +However, Swift could use the `__counted_by` attribute to provide a more convenient API that bundles the count and length together, e.g., + +```swift +/*@unsafe*/ func average(_ ptr: UnsafeBufferPointer) -> Double +``` + +Now, a Swift caller that passes a local `Double` array would not need to pass the count separately, and cannot get it wrong: + +```swift +var values = [3.14159, 2.71828] +average(values) // ok, no need to pass count separately +``` + +This call is still technically unsafe, because we’re passing a temporary pointer into the array’s storage down to the `average` function. That function could save that pointer into some global variable that gets accessed some time after the call, causing a memory safety violation. The actual implementation of `average` is unlikely to do so, and could express this constraint using the existing `noescape` attribute as follows: + +```cpp +double average(const double *__counted_by(N) __attribute__((noescape)) ptr, int N); +``` + +The `average` function is now expressing that it takes in a `double` pointer referencing `count` values but will not retain the pointer beyond the call. These are the semantic requirements needed to provide a memory-safe Swift projection as follows: + +```swift +func average(_ ptr: Span) -> Double +``` + +More expressive Swift lifetime features can also have corresponding C annotations, allowing more C APIs to be reflected into safe APIs in Swift. For example, consider a C function that finds the minimal element in an array and returns a pointer to it: + +```cpp +const double *min_element(const double *__counted_by(N) __attribute__((noescape)) ptr, int N); +``` + +The returned pointer will point into the buffer passed in, so its lifetime is tied to that of the pointer argument. The aforementioned [lifetime dependencies proposal](https://github.com/swiftlang/swift-evolution/pull/2305) allows this kind of dependency to be expressed in Swift, where the resulting non-escaping value (e.g., a `Span` containing one element) has its lifetime tied to the input argument. Clang provides a [`lifetimebound`](https://clang.llvm.org/docs/AttributeReference.html#id11) attribute that expresses when a return value refers into memory associated with one of the parameters, which offers one way to express this lifetime relationship for C APIs: + +```c +const double * _Nullable __counted_by(1) +min_element(const double *__counted_by(N) __attribute__((noescape)) __attribute__((lifetimebound)) ptr, int N); +``` + +The result could be the following memory-safe Swift API: + +```swift +@lifetime(ptr) func min_element(_ ptr: Span) -> Span? +``` + +### Affordances for C++ interoperability + +C++ offers a number of further opportunities for improved safety by modeling lifetimes. For example, `std::vector` has a `front()` method that returns a reference to the element at the front of the vector: + +```cpp +const T& front() const; +``` + +The returned reference is valid so long as the vector instance still exists and has not been modified since the call to `front()`. Describing that lifetime dependency in C++ (for example, with the aforementioned `lifetimebound` attribute) would lead to a safe mapping of this API into Swift without the need to introduce an extra copy of the returned element, improving both safety and, potentially, performance. + +The C++ [`std::span`](https://en.cppreference.com/w/cpp/container/span) type is similar to the Swift `Span` type, in that it also carries both a pointer and bounds to describe a region of memory. However, `std::span` doesn't provide lifetime safety, so it is essentially an unsafe type from the Swift perspective. The same C attributes that provide lifetime safety for C pointers and references could be applied to `std::span` instances to provide safe Swift projections of C++ APIs. For example, the following annotated C++ API: + +```c++ +std::span substring_match( + std::span sequence [[clang::lifetimebound]], + std::span subsequence [[clang::noescape]] +); +``` + +could be imported into Swift as: + +```swift +@lifetime(sequence) +func substring_match(_ sequence: Span, _ subsequence: Span) -> Span +``` + +## Incremental adoption + +The introduction of any kind of additional checking into Swift requires a strategy that accounts for the practicalities of adoption within the Swift ecosystem. Different developers adopt new features on their own schedules, and some Swift code will never enable new checking features. Therefore, it is important that a given Swift module can adopt the proposed strict safety checking without requiring any module it depends on to have already done so, and without breaking any of its own clients that have not enabled strict safety checking. + +The optional strict memory safety model proposed by this lends itself naturally to incremental adoption. The proposed `@unsafe` attribute is not part of the type of the declaration it is applied to, and therefore does not propagate through the type system in any manner. Additionally, any use of an unsafe construct can be addressed locally, either by encapsulating it (e.g., via `@safe(unchecked)`) or propagating it (with `@unsafe`). This means that a module that has not adopted strict safety checking will not see any diagnostics related to this checking, even when modules it depends on adopt strict safety checking. + +The strict memory safety checking does not require any changes to the binary interface of a module, so it can be retroactively enabled (including `@unsafe` annotations) with no ABI or back-deployment concerns. Additionally, it is independent of other language subsetting approaches, such as Embedded Swift. + +## Should strict memory safety checking become the default? + +This proposes that the strict safety checking described be an opt-in feature with no path toward becoming the default behavior in some future language mode. There are several reasons why this checking should remain an opt-in feature for the foreseeable future: + +* The various `Unsafe` pointer types are the only way to work with contiguous memory in Swift today, and the safe replacements (e.g., `Span`) are new constructs that will take a long time to propagate through the ecosystem. Some APIs depending on these `Unsafe` pointer types cannot be replaced because it would break existing clients (either source, binary, or both). +* Interoperability with the C family of languages is an important feature for Swift. Most C(++) APIs are unlikely to ever adopt the safety-related attributes described above, which means that enabling strict safety checking by default would undermine the usability of C(++) interoperability. +* Swift's current (non-strict) memory safety by default is likely to be good enough for the vast majority of users of Swift, so the benefit of enabling stricter checking by default is unlikely to be worth the disruption it would cause. +* The auditing facilities described in this vision should be sufficient for Swift users who require strict memory safety, to establish where unsafe constructs are used and prevent "backsliding" where their use grows in an existing code base. These Swift users are unlikely to benefit much from strict safety being enabled by default in a new language mode, aside from any additional social pressure that would create on Swift programmers to adopt it. diff --git a/visions/platform-support.md b/visions/platform-support.md new file mode 100644 index 0000000000..0c38d50c52 --- /dev/null +++ b/visions/platform-support.md @@ -0,0 +1,285 @@ +# Swift Platform Support: A Vision for Evolution + +The Swift programming language has evolved into a versatile and powerful tool +for developers across a wide range of platforms. As the ecosystem continues to +grow, it is essential to establish a clear and forward-looking vision for +platform support. This vision has two main goals: + +1. To establish common terminology and definitions for platform support. + +2. To document a process for platforms to become officially supported in Swift. + +## Understanding Platforms in the Swift Ecosystem + +The term "platform" carries multiple interpretations. For our purposes, a +platform represents the confluence of operating system, architecture, and +environment where Swift code executes. Each platform is identified using a +version-stripped LLVM `Triple`—a precise technical identifier that captures the +essential characteristics of a host environment (e.g., +`x86_64-unknown-windows-msvc`). + +## The Anatomy of a `Triple` + +At its core, a `Triple` comprises 11 distinct elements arranged in a specific +pattern: + +``` +[architecture][sub-architecture][extensions][endian]-[vendor]-[kernel/OS][version]-[libc/environment][abi][version]-[object format] +``` + +This naming convention might initially appear complex, but it offers remarkable +precision. When a public entity isn't associated with a toolchain, the +placeholder `unknown` is used for the vendor field. Similarly, bare-metal +environments—those without an operating system—employ `none` as their OS/kernel +designation. + +While many of these fields may be elided, for use in Swift, the vendor and OS +fields are always included, even if they are placeholder values. + +Consider these illustrative examples: + +- `armv7eb-unknown-linux-uclibceabihf-coff`: A Linux system running on ARMv7 in big-endian mode, with the µClibc library and PE/COFF object format. +- `aarch64-unknown-windows-msvc-macho`: Windows NT on the ARM64 architecture using the MSVC runtime with Mach-O object format. +- `riscv64gcv-apple-ios14-macabi`: An iOS 14 environment running on a RISC-V processor with specific ISA extensions. + +This nomenclature creates a shared language for discussing platform capabilities +and constraints—an essential foundation for our support framework. + +## Distributions within Platforms + +A platform and distribution, while related, serve distinct roles in the Swift +ecosystem. A platform refers to the broader combination of Operating System, +architecture, and environment where Swift code executes and establishes the +foundational compatibility and functionality of Swift. + +A distribution, on the other hand, represents a specific implementation or +variant within a platform. For example, while Linux as a platform is supported, +individual distributions such as Ubuntu, Fedora, or Amazon Linux require +additional work to ensure that Swift integrates seamlessly. This includes +addressing distribution-specific configurations, dependencies, and conventions. + +Distributions are treated similarly to platforms in that they require a +designated owner. This owner is responsible for ensuring that Swift functions +properly on the distribution, adheres to the distribution's standards, and +remains a responsible citizen within that ecosystem. By assigning ownership, the +Swift community ensures that each distribution receives the attention and +stewardship necessary to maintain a high-quality experience for developers. + +## Platform Stewardship + +The health of each platform depends on active stewardship. Every platform in the +Swift ecosystem requires a designated owner who reviews platform-specific +changes and manages release activities. Platforms without active owners enter a +dormant state, reverting to exploratory status until new leadership emerges. + +This ownership model ensures that platform support remains intentional rather +than accidental—each supported environment has an advocate invested in its +success. + +The Platform Steering Group will regularly review the list of supported +platforms against the tier criteria below. While the Platform Steering Group +reserves the right to update the list or the tier criteria at any time, it is +expected that most such changes will be aligned with the Swift release cycle. + +## A Tiered Approach to Platform Support + +Swift's platform support strategy employs three distinct tiers, each +representing a different level of maturity. The requirements for each tier build +upon those of the previous tier. + +### Tier 1: "Supported" Platforms + +These are Swift's most mature environments, where the language must consistently +build successfully and pass comprehensive test suites. Swift on these platforms +offers the strongest guarantees of stability and performance. + +Platforms that are in Tier 1 should: + +- [ ] Digitally sign their release artifacts. +- [ ] Include a Software Bill of Materials (SBOM). + +- [ ] Include at a minimum the following Swift libraries: + + - [ ] Swift Standard Library + - [ ] Swift Supplemental Libraries + - [ ] Swift Core Libraries + - [ ] Swift Testing Frameworks (if applicable) + + (See [the Swift Runtime Libraries + document](https://github.com/swiftlang/swift/blob/main/Runtimes/Readme.md) + in [the Swift repository](https://github.com/swiftlang/swift).) + +- [ ] Maintain a three-version window of support, including: + + - [ ] At least one stable release. + - [ ] The next planned release. + - [ ] The development branch (`main`). + +- [ ] Have a clear, documented, story for debugging, to allow users to set up + an environment where their products can be executed on a device or + simulator and be debugged. + +- [ ] Have testing in CI, including PR testing. + +- [ ] Ship SDKs as regular release from [swift.org](https://swift.org) + +- [ ] Ensure that instructions needed to get started on the platform + are publicly available, ideally on or linked to from + [swift.org](https://swift.org). + +An important aspect of Tier 1 platforms is that maintenance of support +of these platforms is the collective responsibility of the Swift +project as a whole, rather than falling entirely on the platform +owner. This means: + +- Contributions should not be accepted if they break a Tier 1 platform. + +- If a Tier 1 platform does break, whoever is responsible for the code + that is breaking must work with the platform owner on some kind of + resolution, which may mean backing out the relevant changes. + +- New features should aim to function on all Tier 1 + platforms, subject to the availability of appropriate supporting + functionality on each platform. + +- There is a presumption that a release of Swift will be blocked if a + Tier 1 platform is currently broken. This is not a hard and fast + rule, and can be overridden if it is in the interests of the Swift + project as a whole. + +### Tier 1: "Supported" Toolchain Hosts + +Each toolchain host is an expensive addition to the testing matrix. +In addition to the requirements above, a toolchain host platform should: + +- [ ] Have CI coverage for the toolchain, including PR testing. + +- [ ] Offer toolchain distributions from + [swift.org](https://swift.org) as an official source, though + other distributions may also be available. + +- [ ] Include the following toolchain components: + + - [ ] Swift compiler (`swiftc`). + - [ ] C/C++ compiler (`clang`, `clang++`). + - [ ] Assembler (LLVM integrated assembler, built into `clang`). + - [ ] Linker (_typically_ `lld`). + - [ ] Debugger (`lldb`). + - [ ] Swift Package Manager (SwiftPM). + - [ ] Language Server (`sourcekit-lsp`). + - [ ] Debug Adapter (`lldb-dap`). + +- [ ] Code-sign individual tools as appropriate for the platform. + +Note that the bar for accepting a platform as a toolchain host is somewhat +higher than the bar for accepting a non-toolchain-host platform. + +### Tier 2: "Experimental" Platforms + +Experimental platforms occupy the middle ground—they must maintain the ability +to build but may experience occasional test failures. These platforms +represent Swift's expanding frontier. + +Platforms in this tier should: + +- [ ] Ensure that dependencies beyond the platform SDK can build from source. + +- [ ] Provide provenance information to validate the software supply chain. + +- [ ] Include at a minimum the following Swift libraries: + + - [ ] Swift Standard Library + - [ ] Swift Supplemental Libraries + - [ ] Swift Core Libraries + - [ ] Swift Testing Frameworks (if applicable) + + (See [the Swift Runtime Libraries + document](https://github.com/swiftlang/swift/blob/main/Runtimes/Readme.md) + in [the Swift repository](https://github.com/swiftlang/swift).) + +- [ ] Maintain at least a two-version window of support, including + + - [ ] The next planned release. + - [ ] The development branch (`main`). + +Unlike Tier 1, the Swift project does not assume collective +responsibility for experimental platforms. Platform owners should +work with individual contributors to keep their platform in a +buildable state. + +### Tier 3: "Exploratory" Platforms + +At the boundary of Swift's reach are exploratory platforms. +Exploratory status offers an entry point for platforms taking their +first steps into the Swift ecosystem. + +Platforms in this tier should: + +- [ ] Support reproducible builds without requiring external + patches, though there is no requirement that these build completely + or consistently. + +- [ ] Maintain support in the current development branch (`main`). + +The Swift Project does not assume collective responsibility for +exploratory platforms. Platform owners are responsible for keeping +their platform in a buildable state. + +## Platform Inclusion Process and Promotion + +Adding platform support begins with a formal request to the Platform Steering +Group, accompanied by a platform owner nomination. This structured yet +accessible approach balances Swift's need for stability with its aspiration for +growth. + +The request should include: + +- [ ] The preferred name of the platform. + +- [ ] The name and contact details of the platform owner. + +- [ ] The support tier into which the platform should be placed. + +- [ ] Instructions to build Swift for the platform, assuming someone + is starting from scratch, including any requirements for the + build system. + +- [ ] A list of tier requirements that are currently _not_ met by the + platform, including an explanation as to _why_ they are not met + and what the proposal is to meet them, if any. + +- [ ] Whether there has been any discussion about provisioning of CI + resources, and if so a copy of or link to that discussion. This + is particularly relevant for a Tier 1 platform request. + +Note that it is _not_ the case that a platform _must_ meet every +requirement of the requested tier in order to be placed into that +tier. The Platform Steering Group will consider each case on its +merits, and will make a decision based on the information at hand as +well as the overall benefit to the Swift Project. It should be +emphasized that the Platform Steering Group reserves the right to +consider factors other than those listed here when making decisions +about official platform support. + +The same process should be used to request a promotion to a higher +tier. + +## Existing Platforms and Demotion + +The following existing platforms are in Tier 1 regardless of any +text in this document: + +- All Apple platforms (macOS, iOS and so on). +- Linux +- Windows + +The Platform Steering Group reserves the right to demote any +platform to a lower tier, but regards demotion as a last resort +and will by preference work with platform owners to maintain +support appropriate for their platform's existing tier. + +Note that if your platform is one of the above special cases, and +there is some requirement in this document that is not being met, it +is expected that either there is a very good reason for the +requirement not being met, or that there is some plan to meet it in +future. diff --git a/visions/swift-testing.md b/visions/swift-testing.md new file mode 100644 index 0000000000..23879f437c --- /dev/null +++ b/visions/swift-testing.md @@ -0,0 +1,1051 @@ +# A New Direction for Testing in Swift + +## Introduction + +A key requirement for the success of any developer platform is a way to use +automated testing to identify software defects. Better APIs and tools for +testing can greatly improve a platform’s quality. Below, we propose a new +direction for testing in Swift. + +We start by defining our basic principles and describe specific features that +embody those principles. We then discuss several design considerations +in-depth. Finally, we present specific ideas for delivering an all-new testing +solution for Swift, and weigh them against alternatives considered. + +## Principles + +Testing in Swift should be **approachable** by both new programmers and +seasoned engineers. There should be few APIs to learn, and they should feel +ergonomic and modern. It should be easy to incrementally add new tests +alongside legacy ones. Testing should integrate seamlessly into the tools and +workflows that people know and use every day. + +A good test should be **expressive** and automatically include actionable +information when it fails. It should have a clear name and purpose, and there +should be facilities to customize a test’s representation and metadata. Test +details should be specified on the test, in code, whenever possible. + +A testing library should be **flexible** and capable of accommodating many +needs. It should allow grouping related tests when beneficial, or letting them +be standalone. There should be ways to customize test behaviors when necessary, +while having sensible defaults. Storing data temporarily during a test should +be possible and safe. + +A modern testing system should have **scalability** in mind and gracefully +handle large test suites. It should run tests in parallel by default, but allow +some tests to opt-out. It should be effortless to repeat a test with different +inputs and see granular results. The library should be lightweight and +efficient, imposing minimal overhead on the code being tested. + +## Features of a great testing system + +Guided by these principles, there are many specific features we believe are +important to consider when designing a new testing system. + +### Approachability + +* **Be easy to learn and use**: There should be few individual APIs to + memorize, they should have thorough documentation, and using them to write a + new test should be fast and seamless. Its APIs should be egonomic and adhere + to Swift’s [design guidelines](https://www.swift.org/documentation/api-design-guidelines/). +* **Validate expected behaviors or outcomes**: The most important job of any + testing library is checking that code meets specific expectations—for example, + by confirming that a function returns an expected result or that two values + are equal. There are many interesting variations on this, such as comparing + whole collections or checking for errors. A robust testing system should cover + all these needs, while using progressive disclosure to remain simple for + common cases. +* **Enable incremental adoption:** It should gracefully coexist with projects + that use XCTest or other testing libraries and allow incremental adoption so + that users can transition at their own pace. This is especially important + because this new system may take time to achieve feature parity. +* **Integrate with tools, IDEs, and CI systems:** A useful testing library + requires supporting tools for functionality such as listing and selecting + tests to run, launching runner processes, and collecting results. These + features should integrate seamlessly with common IDEs, SwiftPM’s `swift test` + command, and continuous integration (CI) systems. + +### Expressivity + +* **Include actionable failure details**: Tests provide the most value when they + fail and catch bugs, but for a failure to be actionable it needs to be + sufficiently detailed. When a test fails, it should collect and show as much + relevant information as reasonably possible, especially since it may not + reproduce reliably. +* **Offer flexible naming, comments, and metadata:** Test authors should be able + to customize the way tests are presented by giving them an informative name, + comments, or assigning metadata like labels to tests which have things in + common. +* **Allow customizing behaviors:** Some tests share common set-up or tear-down + logic, which need to be performed once for each test or group. Other times, a + test may begin failing for an irrelevant reason and must be temporarily + disabled. Some tests only make sense to run under certain conditions, such as + on specific device types or when an external resource is available. A modern + testing system should be flexible enough to satisfy all these needs, without + complicating simpler use cases. + +### Flexibility + +* **Allow organizing tests into groups (or not):** Oftentimes a component will + have several related tests that would make sense to group together. It should + be possible to group tests into hierarchies, while allowing simpler tests to + remain standalone. +* **Support per-test storage:** Tests often need to store data while they are + running and local variables are not always sufficient. For example, set up + logic for a test may create a value the test needs to access, but these are in + different scopes. There must be a way to carefully store per-test data, to + ensure it is isolated to a single test and initialized deterministically to + avoid unexpected dependencies or failures. +* **Allow observing test events:** Some use cases require an ability to observe + test events—for example, to perform custom reporting or analysis of results. A + testing library should offer hooks for event handling. + +### Scalability + +* **Parallelize execution:** Many tests can be run in parallel to improve + execution time, either using multiple threads in a single process or multiple + runner processes. A testing library should offer flexible parallelization + options for eligible tests, encourage parallelizing whenever possible, and + offer granular control over this behavior. It should also leverage Swift’s + data race safety features (such as `Sendable` enforcement) to the fullest + extent possible to avoid concurrency bugs. +* **Repeat a test multiple times with different arguments:** Many tests consist + of a template with minor variations—for example, invoking a function multiple + times with different arguments each time and validating the result of each + invocation. A testing library should make this pattern easy to apply, and + include detailed reporting so a failure during a single argument is + represented clearly. +* **Behave consistently across platforms:** Any new testing solution should be + cross-platform from its inception and support every platform Swift supports. + Its observable behaviors should be as consistent as possible across those + platforms, especially for core responsibilities such as discovering and + executing tests. + +## Design considerations + +Several areas deserve close examination when designing a new testing API. Some, +because they may benefit from language or compiler toolchain enhancements to +deliver the ideal experience, and others because they have non-obvious +reasoning or requirements. + +### Expectations + +Testing libraries typically offer APIs to compare values—for example, to +confirm that a function returns an expected result—and report a test failure if +a comparison does not succeed. Depending on the library, these APIs may be +called “assertions”, “expectations”, “checks”, “requirements”, “matchers“, or +other names. In this document we refer to them as **expectations**. + +For test failures to be actionable, they need to include enough details to +understand the problem, ideally without a human manually reproducing the +failure and debugging. The most important details relevant to expectation +failures are the values being compared or checked and the kind of expectation +being performed (e.g. equal, not-equal, less-than, is-not-nil, etc.). Also, if +any error was caught while evaluating an expression passed to an expectation, +that should be included. + +Beyond the values of evaluated expressions, there are other pieces of +information that may be useful to capture and include in expectations: + +* The **source code location** of the expectation, typically using the format + `#fileID:#line:#column`. This helps test authors jump quickly to the line of + code to view context, and lets IDEs present the failure in their UI at that + location. +* The **source code text of expression(s)** passed to the expectation. In an + example expectation API call `myAssertEqual(subject.label == "abc")`, the + source code text would be the string `"subject.label == \"abc\""` . + + Even though source code text may not be necessary when viewing failures in + an IDE since the code is present, this can still be helpful to confirm the + expected source code was evaluated in case it changed recently. It’s even + more useful when the failure is shown on a CI website or anywhere without + source, since a subexpression (such as `subject.label` in this example) may + give helpful clues about the failure. +* **Custom user-specified comments**. Comments can be helpful to allow test + authors to add context or information only needed if there was a failure. They + are typically short and included in the textual log output from the test + library. +* **Custom data or file attachments.** Some tests involve files or data + processing and may benefit from allowing expectations to save arbitrary data + or files in the results for later analysis. + +#### Powerful, yet simple + +Since the most important details to include in expectation failure messages are +the expression(s) being compared and the kind of expression, some testing +libraries offer a large number of specialized APIs for detailed reporting. Here +are some examples from other prominent testing libraries: + +| | Java (JUnit) | Ruby (RSpec) | XCTest | +|----|----|----|----| +| Equal | `assertEquals(result, 3);` | `expect(result).to eq(3)` | `XCTAssertEqual(result, 3)` | +| Identical | `assertSame(result, expected);` | `expect(result).to be(expected)` | `XCTAssertIdentical(result, expected)` | +| Less than or equal | N/A | `expect(result).to be <= 5` | `XCTAssertLessThanOrEqual(result, 5)` | +| Is null/nil | `assertNull(actual);` | `expect(actual).to be_nil` | `XCTAssertNil(actual)` | +| Throws | `assertThrows(E.class, () -> { ... });` | `expect {...}.to raise_error(E)` | `XCTAssertThrowsError(...) { XCTAssert($0 is E) }` | + +Offering a large number of specialized expectation APIs is a common practice +among testing libraries: XCTest has 40+ functions in its +[`XCTAssert` family](https://developer.apple.com/documentation/xctest/boolean_assertions); +JUnit has +[several dozen](https://junit.org/junit5/docs/5.0.1/api/org/junit/jupiter/api/Assertions.html); +RSpec has a +[large DSL](https://relishapp.com/rspec/rspec-expectations/docs/built-in-matchers) +of test matchers. + +Although this approach allows straightforward reporting, it is not scalable: + +* It increases the learning curve for new users by requiring them to learn many + new APIs and remember to use the correct one in each circumstance, or risk + having unclear test results. +* More complex use cases may not be supported—for example, if there is no + expectation for testing that a `Sequence` starts with some prefix using + `starts(with:)`, the user may need a workaround such as adding a custom + comment which includes the sequence for the results to be actionable. +* It requires testing library maintainers add bespoke APIs supporting many use + cases which creates a maintenance burden. +* Depending on the exact function signatures, it may require additional + overloads that complicate type checking. + +We believe expectations should strive to be as simple as possible and involve +few distinct APIs, but be powerful enough to include detailed results for every +expression. Instead of offering a large number of specialized expectations, +there should only be a few basic expectations and they should rely on ordinary +expressions, built-in language operators, and the standard library to cover all +use cases. + +#### Evaluation rules + +Expectations have certain rules which must be followed carefully when handling +arguments: + +* The primary expression(s) being checked should be evaluated exactly once. In + particular, if the expectation failed, showing the value of any evaluated + expression should not cause the expression to be evaluated a second time. This + is to avoid any undesirable or unexpected side effects of multiple + evaluations. +* Custom comments or messages should only be evaluated if the expectation + failed, and at most once, to similarly avoid undesirable side effects and + prevent unnecessary work. + +#### Continuing after a failure + +A single test may include multiple expectations, and a testing library must +decide whether to continue executing a test after one of its expectations +fails. Some tests benefit from always running to completion, even if an earlier +expectation failed, since they validate different things and early expectations +are unrelated to later ones. Other tests are structured such that later logic +depends heavily on the results of earlier expectations, so terminating the test +after any expectation fails may save time. Still other tests take a hybrid +approach, where only certain expectations are required and should terminate +test execution upon failure. + +This is a policy decision, and is something a testing library could allow users +to control on a global, per-test, or per-expectation basis. + +#### Rich representation of evaluated values + +Often, expectation APIs do not preserve raw expression values when reporting a +failure, and instead generate a string representation of those values for +reporting purposes. Although a string representation is often sufficient, +failure presentation could be improved if an expectation were able to keep +values of certain, known data types. + +As an example, imagine a hypothetical expectation API call +`ExpectEqual(image.height, 100)`, where `image` is a value of some well-known +graphical image type `UILibrary.Image`. Since this uses a known data type, the +expectation could potentially keep `image` upon failure and include it in test +results, and then an IDE or other tool could present the image graphically for +easier diagnosis. This capability could be extensible and cross-platform by +using a protocol to describe how to convert arbitrary values into one of the +testing library’s known data types, delivering much richer expectation results +presentation for commonly-used types. + +### Test traits + +A recurring theme in several of the features discussed above is a need to +express additional information or options about individual tests or groups of +tests. A few examples: + +* Describing test requirements or marking a test disabled. +* Assigning a tag or label to a test, to locate or run those which have + something in common. +* Declaring argument values for a parameterized or “data-driven” test. +* Performing common logic before or after a test. + +Collectively, these are referred to in this document as **traits**. The traits +for an individual test _could_ be stored in a standalone file, separate from +the test definition, but relying on a separate file has known downsides: it can +get out of sync if a test name changes, and it’s easy to overlook important +details—such as whether a test is disabled or has specific requirements—when +they’re stored separately. + +We believe that the traits for a single test should preferably be declared in +code placed as close to the test they describe as possible to avoid these +problems. However, global settings may still benefit from configuring via +external files, as there may not be a canonical location in code to place them. + +#### Trait inheritance + +When grouping related tests together, if a test trait is specified both for an +individual test and one of its containing groups, it may be ambiguous which +option takes precedence. The testing library must establish policies for how to +resolve this. + +Test traits may fall into different categories in terms of their inheritance +behavior. Some semantically represent multiple values that a user would +reasonably expect to be added together. One example is test requirements: if a +group specifies one requirement, while one of its test functions specifies +another, the test function should only run if both requirements are satisfied. +The order these requirements are evaluated are worth considering and formally +specifying, so that a user could be assured that requirements are always +evaluated “outermost-to-innermost” or vice-versa. + +Another example is test tags: they are also considered multi-value, but items +with tags are typically expected to have `Set` rather than `Array` semantics +and ignore duplicates, so for this type of trait the evaluation order is +insignificant. + +Other test traits semantically represent a single value and conflicts between +them may be more challenging to resolve. As a hypothetical example, imagine a +test trait spelled `.enabled(Bool)` which includes a `Bool` that determines +whether a test should run. If a group specifies `.enabled(false)` but one of +its test functions specifies `.enabled(true)`, which value should be honored? +Arguments could be made for either policy. + +When possible, it may be easier to avoid ambiguity: in the previous example, +this may be solved by only offering a `.disabled` option and not the opposite. +But the inheritance semantics of each option should be considered, and when +ambiguity is unavoidable, a policy for resolving it should be established and +documented. + +#### Trait extensibility + +A flexible test library should allow certain behaviors to be extended by test +authors. A common example is running logic before or after a test: if every +test in a certain group requires the same steps beforehand, those steps could +be placed in a single method in that group rather than expressed as an option +on a particular test. However, if only a few tests within a group require those +steps, it may make sense to leverage a test trait to mark those tests +individually. + +Test traits should provide the ability to extend behaviors to support this +workflow. For example, it should be possible to define a custom test trait, and +implement hooks that allow it to run custom code before or after a test or +group. + +### Test identity + +Some features require the ability to uniquely identify a test, such as +selecting individual tests to run or serializing results. It may also be useful +to access the name of a test inside its own body or for an entity observing +test events to query test names. + +A testing library should include a robust mechanism to uniquely identify tests +and identifiers should be stable across test runs. If it is possible to +customize a test’s display name, the testing library should decide which name +is authoritative and included in the unique identifier. Also, function +overloading could make certain test function names ambiguous without additional +type information. + +### Test discovery + +A frequent challenge for testing libraries in all languages is the need to +locate tests in order to run them. Users typically expect tests to be +discovered automatically, without needing to provide a comprehensive list since +that would be a maintenance burden. + +There are three types of test discovery worth considering in particular, since +they serve different purposes: + +* **At runtime:** When a test runner process is launched, the testing library + needs to locate tests so it can execute them. +* **After a build:** After compilation of all test code has completed + successfully, but before a test runner process has been launched, it may be + useful for a tool to introspect the test build products and print the list of + tests or extract other metadata about them without running them. +* **While authoring:** After tests have been written or edited, but before a + build has completed, it is common for an IDE or other tool to statically + analyze or index the code and locate tests so it can list them in a UI and + allow running them. + +Each of these are important to support, and may require different solutions. + +#### Non-runtime discovery + +Two of the above test discovery types—_After a build_ and _While authoring_— +require the ability to discover tests without launching a runner process, and +thus without using the testing library’s runtime logic and models to represent +tests. In addition to the IDE use case mentioned above, another reason +discovering tests statically may be useful is so CI systems can extract +information about tests and use it to optimize test execution scheduling on +physical devices. It is common for CI systems to run a different host OS than +the platform they are targeting—for example, an Intel Mac building tests for an +iOS device—and in those situations it may be impractical or expensive for the +CI system to launch a runner process to gather this information. + +Note that not _all_ test details are eligible to extract statically: those that +enable runtime test behaviors may not be, but trivial metadata (such as a +test’s name or whether it is disabled) should be extractable, especially with +further advances in Swift’s support for +[Build-Time Constant Values](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0359-build-time-constant-values.md). +While designing a new testing API, it is important to consider which test +metadata should be statically extractable to support these non-runtime discovery +use cases. + +### Parameterized testing + +Repeating a test multiple times with different arguments—formally referred to as +[Parameterized or Data-Driven Testing](https://en.wikipedia.org/wiki/Data-driven_testing) +—can allow expanding test coverage to cover more scenarios with minimal code +repetition. Although a user could approximate this using a simple loop such as +`for...in` in the body of a test, it’s often better to let testing libraries +handle this task. A testing library can automatically keep track of the +argument(s) for each invocation of a test and record them in the results. It can +also provide a way to selectively re-run individual argument combinations for +fine-grained debugging in case only one instance failed. + +Note that recording individual parameterized tests’ arguments in results and +re-running them requires some way to uniquely represent those arguments, which +overlaps with some of the considerations discussed in +[Test identity](#test-identity). + +### Parallelization and concurrency + +A modern testing system should make efficient use of the machine it runs on. +Many tests can safely run in parallel, and the testing system should encourage +this by enabling per-test parallelization by default. In addition to faster +results and shorter iteration time, running tests in parallel can help identify +bugs due to hidden dependencies between tests and encourage better state +isolation. + +However, some tests may need to disable parallelization and run one at a time. +It should be possible to opt-out, and this may be especially useful while +migrating from older testing systems which don't support parallelization. +Although opting-out of this behavior should be possible, it should be narrowly +scoped to not sacrifice other tests' ability to run in parallel. + +In addition to running tests in parallel relative to each other, tests +themselves should seamlessly support Swift's concurrency features. In particular, +this means: + +* Tests should be able to use async/await whenever necessary. +* Tests should support isolation to a global actor such as `@MainActor`, but be + nonisolated by default. (Isolation by default would undermine the goal of + running tests in parallel by default.) +* Values passed as arguments to parameterized tests should be `Sendable`, since + they may cross between isolation domains within the testing system's execution + machinery. +* Types containing tests functions and their stored properties need not be + `Sendable`, since they are only used from a single isolation domain while each + test function is run. + +### Tools integration + +A well-rounded testing library should be integrated with popular tools used by +the community. This integration should include some essential functionality such +as: + +* Building tests into products which can be be executed. +* Running all built tests. +* Showing per-test results, including details of each individual failure during + a test. +* Showing an aggregate summary of a test run, including failure statistics. + +Beyond the essentials, tools may offer other useful features, such as: + +* Filtering tests by name, specific traits (e.g. custom tags), or other criteria. +* Outputting results to a standard format such as JUnit XML for importing into + other tools. +* Controlling runtime options such as whether parallel execution is enabled, + whether failed tests should be reattempted, etc. +* Relaunching a test executable after an unexpected crash. +* Reporting "live" progress as each test finishes. + +In order to deliver the functionality above, a testing system needs to track +significant events which occur during execution, such as when each test starts +and finishes or encounters a failure. There needs to be a structured, +machine-readable representation of each event, and to provide granular progress +updates, there should be an option for tools to observe a stream of such events. +Any serialization formats involved should be versioned and provide some level of +stability over time, to ensure compatibility with tools which may evolve on +differing release schedules. + +Some of the features outlined above are much easier to achieve if there are at +least two processes involved: One which executes the tests themselves (which may +be unstable and terminate unexpectedly), and another which launches and monitors +the first process. In this document, we'll refer to this second process as the +**harness**. A two-process architecture involving a supervisory harness helps +enable functionality such as relaunching after a crash, live progress reporting, +or outputting results to a standard format. Some cases effectively _require_ a +harness, such as when launching tests on a remotely-connected device. A modern +testing library should provide any necessary APIs and runtime support to +facilitate integration with a harness. + +## Today’s solution: XCTest + +[XCTest](https://developer.apple.com/documentation/xctest) has historically +served as the de facto standard testing library in Swift. It was originally +written [in 1998](https://www.sente.ch/ocunit/?lang=en) in Objective-C, and +heavily embraced that language’s idioms throughout its APIs. It relies on +subclassing (reference semantics), dynamic message passing, NSInvocation, and +the Objective-C runtime for things like test discovery and execution. In the +2010s, it was integrated deeply into Xcode and given many more capabilities and +APIs which have helped Apple platform developers deliver industry-leading +software. + +When Swift was introduced, XCTest was extended further to support the new +language while maintaining its core APIs and overall approach. This allowed +developers familiar with using XCTest in Objective-C to quickly get up to +speed, but certain aspects of its design no longer embody modern best practices +in Swift, and some have become problematic and prevented enhancements. Examples +include its dependence on the Objective-C runtime for test discovery; its +reliance on APIs like NSInvocation which are unavailable in Swift; the frequent +need for implicitly-unwrapped optional (IUO) properties in test subclasses; and +its difficulty integrating seamlessly with Swift Concurrency. + +It is time to chart a new course for testing in Swift, and in proposing a new +direction, this ultimately represents a successor to XCTest. This transition +will likely span several years, and we aim to thoughtfully design and deliver a +solution that will be even more powerful while bearing in mind the many lessons +learned from maintaining it over the years. + +## A new direction + +> [!NOTE] +> The approach described below is not meant to include a solution for _every_ +> consideration or feature discussed in this document. It describes a starting +> point for this new direction, and covers many of the topics, but leaves some +> to be pursued as part of follow-on work. + +The new direction includes 3 major components exposed via a new module named +`Testing`: + +1. `@Test` and `@Suite` attached macros: These declare test functions and suite + types, respectively. +1. Traits: Values passed to `@Test` or `@Suite` which customize the behavior of + test functions or suite types. +1. Expectations `#expect` and `#require`: expression macros which validate + expected conditions and report failures. + +### Test and Suite declaration + +To declare test functions and suites (types containing tests), we will leverage +[Attached Macros (SE-0389)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md). +At a high level, this will consist of several attached macros +which may be placed on a test type or test function, defined in a new module +named `Testing`: + +```swift +/// Declare a test function +@attached(peer) +public macro Test( + // ...Parameters described later +) + +/// Declare a test suite. +@attached(member) @attached(peer) +public macro Suite( + // ...Parameters described later +) +``` + +Then, test authors may attach these macros to functions or types in a test +target. Here are some usage examples: + +```swift +import Testing + +// A test implemented as a global function +@Test func example1() { + // ... +} + +@Suite struct BeginnerTests { + // A test implemented as an instance method + @Test func example2() { ... } +} + +// Implicitly treated as a suite type, due to containing @Test functions. +actor IntermediateTests { + private var count: Int + + init() async throws { + // Runs before every @Test instance method in this type + self.count = try await fetchInitialCount() + } + + deinit { + // Runs after every @Test instance method in this type + print("count: \(count), delta: \(delta)") + } + + // A test implemented as an async and throws instance method + @Test func example3() async throws { + delta = try await computeDelta() + count += delta + // ... + } +} +``` + +**Test functions** may be defined as global functions or as either instance or +static methods in a type. They must always be explicitly annotated as `@Test`, +they need not follow any naming convention (such as beginning with “test”), and +they may include `async`, `throws`, or `mutating`. + +**Suite types**, or simply “suites”, are types containing `@Test` functions or +other nested suite types. Suite types may include the `@Suite` attribute +explicitly, although it is optional and only required when specifying traits +(described below). A suite type must have a zero-parameter `init()` if it +contains instance `@Test` methods. + +**Per-test storage:** The `IntermediateTests` example demonstrates per-test +set-up and tear-down as well as per-test storage: A unique instance of +`IntermediateTests` is created for every `@Test`-annotated instance method it +contains, which means that its `init` and `deinit` are run once before and +after each respectively, and they may contain set-up or tear-down logic. Since +`count` is an instance stored property, it acts as per-test storage, and since +`example3()` is isolated to its enclosing actor type it is allowed to mutate +`count`. + +**Sendability:** Note that the test functions and suite types in these examples +are not required to be `Sendable`. At runtime, if the `@Test` function is an +instance method, the testing library creates a thunk which instantiates the +suite type and invokes the `@Test` function on that instance. The suite type +instance is only accessed from a single Task. + +**Actor isolation:** `@Test` functions or types may be annotated with a global +actor (such as `@MainActor`), in accordance with standard language and type +system rules. This allows tests to match the global actor of their subject and +reduce the need for suspension points. + +#### Test discovery + +To facilitate test discovery, the attached macros above will eventually use a +feature such as `@linkage`, an attribute for controlling low-level symbol +linkage (see +[pitch](https://forums.swift.org/t/pitch-2-low-level-linkage-control/69752)). +It will allow placing variables and functions in a special section of the binary, +where they can be retrieved and at runtime or inspected statically at build time. + +However, before that feature lands, the testing library will use a temporary +approach of iterating through types conforming to a known protocol and +gathering their tests by calling a static property. The attached macros will +emit code which generates the types to be discovered using this mechanism. Once +more permanent support lands, the attached macros will be adjusted to adopt it +instead. + +Regardless of which technique the attached macros above use to facilitate test +discovery, the APIs called by their expanded code need not be considered public +or stable. In particular, code emitted by these macros may call +underscore-prefixed APIs declared in the `Testing` module, which are marked +`public` to facilitate use by these macros but are generally considered a +private implementation detail. + +### Traits + +As discussed earlier, it is important to support specifying traits for a test. +[SE-0389](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0389-attached-macros.md) +allows including parameters in an attached macro declaration, and this allows +users to pass arguments to a `@Test` attribute on a test function or type. + +The `Testing` module will offer an extensible mechanism for specifying per-test +traits via types conforming to protocols such as `TestTrait` and `SuiteTrait`: + +```swift +/// A protocol describing traits that can be added to a test function or +/// to a test suite. +public protocol Trait: Sendable { ... } + +/// A protocol describing traits that can be added to a test function. +public protocol TestTrait: Trait { ... } + +/// A protocol describing traits that can be added to a test suite. +public protocol SuiteTrait: Trait { ... } +``` + +Using these protocols, the attached macros `@Test` and `@Suite` shown earlier +will gain parameters accepting traits: + +```swift +/// Declare a test function. +/// +/// - Parameter traits: Zero or more traits to apply to this test. +@attached(peer) +public macro Test( + _ traits: any TestTrait... +) + +/// Declare a test function. +/// +/// - Parameters: +/// - displayName: The customized display name of this test. +/// - traits: Zero or more traits to apply to this test. +@attached(peer) +public macro Test( + _ displayName: _const String, + _ traits: any TestTrait... +) + +/// Declare a test suite. +/// +/// - Parameter traits: Zero or more traits to apply to this test suite. +@attached(member) @attached(peer) +public macro Suite( + _ traits: any SuiteTrait... +) + +/// Declare a test suite. +/// +/// - Parameters: +/// - displayName: The customized display name of this test suite. +/// - traits: Zero or more traits to apply to this test suite. +@attached(member) @attached(peer) +public macro Suite( + _ displayName: _const String, + _ traits: any SuiteTrait... +) +``` + +The specifics of the `Trait` protocols and the built-in types conforming to +them will be left to subsequent proposals. But to illustrate the general +pattern they will follow, here is an example showing how a hypothetical option +for marking a test disabled could be structured: + +```swift +/// A test trait which marks a test as disabled. +public struct DisabledTrait: TestTrait { + /// An optional comment related to this option. + public var comment: String? +} + +extension TestTrait where Self == DisabledTrait { + /// Construct a test trait which marks a test disabled, + /// with an optional comment. + public static func disabled(_ comment: String? = nil) -> Self +} + +// Usage example: +@Test(.disabled("Currently causing a crash: see #12345")) +func example4() { + // ... +} +``` + +#### Nesting / subgrouping tests + +Earlier examples showed how related tests may be grouped together by placing +them within a type. This technique also allows forming sub-groups by nesting +one type containing tests inside another: + +```swift +struct OuterTests { + @Test func outerExample() { /* ... */ } + + @Suite(.tags("edge-case")) + struct InnerTests { + @Test func innerExample1() { /* ... */ } + @Test func innerExample2() { /* ... */ } + } +} +``` + +When using this technique, test traits may be specified on nested types and +inherited by all tests they contain. For example, the `.tags("edge-case")` +trait shown here on `InnerTests` would have the effect of adding the tag +`edge-case` to both `innerExample1()` and `innerExample2()`, as well as to +`InnerTests`. + +#### Parameterized tests + +Parameterized testing is easy to support using this API: The `@Test` functions +shown earlier do not accept any parameters, making them non-parameterized, but +if a `@Test` function includes a parameter, then a different overload of the +`@Test` macro can be used which accepts a `Collection` whose associated +`Element` type matches the type of the parameter: + +```swift +/// Declare a test function parameterized over a collection of values. +/// +/// - Parameters: +/// - traits: Zero or more traits to apply to this test. +/// - collection: A collection of values to pass to the associated test +/// function. +/// +/// During testing, the associated test function is called once for each element +/// in `collection`. +@attached(peer) +public macro Test( + _ traits: any TestTrait..., + arguments collection: C +) where C: Collection & Sendable, C.Element: Sendable + +// Usage example: +@Test(arguments: ["a", "b", "c"]) +func example5(letter: String) { + // ... +} +``` + +Once Swift’s support for +[Variadic Generics](https://github.com/hborla/swift-evolution/blob/variadic-generics-vision/vision-documents/variadic-generics.md) +gains more functionality, the signature of these `@Test` macros may be revised +to accept more than one collection of arguments. This will expand the feature by +allowing a test function with arity _N_ to be repeated once for each combination +of elements from _N_ collections. + +### Expectations + +In existing test solutions available to Swift developers, there is limited +diagnostic information available for a failed expectation such as +`assert(2 < 1)`. The expression is reduced at runtime to a simple boolean value +with no context (such as the original source code) available to include in a +test’s output. + +By adopting +[Expression Macros (SE-0382)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0382-expression-macros.md), +we can give developers _implicitly expressive_ test +expectations. The expectation shown below, upon failure, can capture not just +the boolean value `false`, but also the left-hand and right-hand operands and +the operator itself (that is, `x`, `1`, and `<` respectively) and expand any +sub-expressions to their evaluated values, such as `x → 2`: + +```swift +let x = 2 +#expect(x < 1) // failed: (x → 2) < 1 +``` + +#### Handling optionals + +Some expectations _must_ pass for a test to proceed—these would be expressed +with a separate macro `#require()`. Because `#require()` _must_ pass, we can +infer additional behaviors based on its argument that we cannot do with +`#expect()`. For example, if an optional value is passed to `#require()`, we +can infer that `#require()` should return the optional value or fail if it is +`nil`: + +```swift +let x: Int? = 10 +let y: String? = nil +let z = try #require(x) // passes, z == 10 +let w = try #require(y) // fails, test ends early with a thrown error +``` + +#### Handling complex expressions + +We can also extract the components of an expression like `a.contains(b)` and, on +failure, report the value of `a` and `b`: + +```swift +let a = [1, 2, 3] +let b = 4 +#expect(a.contains(b)) // failed: (a → [1, 2, 3]).contains(b → 4) +``` + +#### Handling collections + +We can also leverage built-in language features for yet more expressiveness. +Consider the following test logic: + +```swift +let a = [1, 2, 3, 4, 5] +let b = [1, 2, 3, 3, 4, 5] +#expect(a == b) +``` + +This expectation will fail because of the extra element `3` in `b`. We can +leverage +[Ordered Collection Diffing (SE-0240)](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0240-ordered-collection-diffing.md) +to capture exactly how these arrays differ and present that information to the +developer as part of the test output or in the IDE. + +### API and ABI stability + +Once this project reaches its first major release, its API will be considered +stable and will follow the same general rules as the Swift standard library. +Source-breaking API changes will be considered a last resort and be preceded by +a generous deprecation period. + +However, unlike the standard library, this project will generally not guarantee +ABI stability. Tests are typically only run during the development cycle and +aren't distributed to end users, so the runtime testing library will not be +included in OS distributions and thus its interfaces do not need to maintain +ABI stability. + +### Project governance + +For this testing solution to stand the test of time, it needs to be well +maintained and any new features added to it should be thoughtfully designed and +undergo community review. + +The codebase for this testing system will be open source. Any significant +additions to its feature set or changes to its API or behavior will follow a +process inspired by, but separate from, +[Swift Evolution](https://www.swift.org/swift-evolution/). Changes will be +described in writing using a standard +[proposal template](https://github.com/apple/swift-testing/blob/main/Documentation/Proposals/0000-proposal-template.md) and discussed in the +[swift-testing](https://forums.swift.org/c/related-projects/swift-testing/103) +category of the Swift Forums. The process for pitching and submitting proposals +for review will be formalized separately and documented in the project repo. + +A new group—tentatively named the _Swift Testing Workgroup_—will be formed to +act as the primary governing body for this project. This group will be +considered a sub-group of the planned +[Ecosystem Steering Group](https://www.swift.org/blog/evolving-swift-project-workgroups/) +once it has been formed. The group will retroactively assume responsibility for +the [swift-corelibs-xctest](https://github.com/apple/swift-corelibs-xctest) +project as well. The responsibilities of members of this workgroup will include: + +* defining and approving roadmaps for the project; +* scheduling proposal reviews; +* guiding community discussion; +* making decisions about proposals; and +* working with members of related workgroups, such as the + [Platform](https://www.swift.org/platform-steering-group/), + [Server](https://www.swift.org/sswg/), or + [Documentation](https://www.swift.org/documentation-workgroup/) workgroups, on + topics which intersect with their areas of focus. + +The membership of this workgroup, its charter, and more specifics about its role +in the project will be formalized separately. + +### Distribution + +As mentioned in [Approachability](#approachability) above, testing should be +easy to use. The easier it is to write a test, the more tests will be written, +and software quality generally improves with more automated tests. + +Most open source Swift software is distributed as source code and built by +clients, using tools such as Swift Package Manager. Although this is very common, +if we took this approach, every client would need to download the source for +this project and its dependencies. A test target is included in SwiftPM's +standard New Package template, so if this project became the default testing +library used by that template, this would mean a newly-created package would +require internet access to build its tests. In addition to downloading source, +clients would also need to _build_ this project along with its dependencies. +Since this project relies on Swift Macros, it depends on +[swift-syntax](https://github.com/apple/swift-syntax) and that is a large +project known to have lengthy build times as of this writing. + +Due to these practical concerns, this project will be distributed as part of the +Swift toolchain, at least initially. Being in the toolchain means clients will +not need to download or build this project or its dependencies, and this +project's macros can use the copy of swift-syntax in the toolchain. Longer-term, +if the practical concerns described above are resolved, the library could be +removed from the toolchain, and doing so could yield other benefits such as more +explicit dependency tracking and more portable toolchains. + +#### Package support + +Although the Swift toolchain will be the primary method of distribution, the +project will support building as a Swift package. This will facilitate +development of the package itself by its maintainers and outside contributors. + +Clients will also have the option of declaring an explicit package dependency on +this project rather than relying on the built-in copy in the toolchain. If a +client does this, it will be possible for tools to integrate with the client's +specified copy of the project, as long as it is a version of the package the +tool supports. + +### Platform support + +Testing should be broadly available across platforms. The long-term goal is for +this project to be available on all platforms Swift itself supports. + +Whenever the +[Swift Platform Steering Group](https://www.swift.org/platform-steering-group/) +declares intention to support a new platform, this project will be considered +one of the highest-priority components to get working on the new platform (after +dependencies such as the standard library) since this project will enable +qualification of many other components in the stack. The maintainers of this +project will work with other Swift workgroups or steering groups to help enable +support on new platforms. + +One reason why broad platform support is important is so that this project can +eventually support testing the Swift standard library. The standard library +currently uses a custom library for testing +([StdlibUnittest](https://github.com/apple/swift/tree/main/stdlib/private/StdlibUnittest)) +but many of this project's benefits would be useful in that context as well, so +eventually we would like to rebase StdlibUnitTest on this project. + +While the goal is for this project to work on every platform Swift does, it +currently does not work on Embedded Swift due to its reliance on existentials. +It is possible this limitation may be overcome through project changes, but if +it cannot be, the project maintainers will work with other Swift work/steering +groups to identify a solution. + +## Alternatives considered + +### Declarative test definition using Result Builders + +While exploring new testing directions, we considered and thoroughly prototyped +an approach relying heavily on +[Result Builders](https://docs.swift.org/swift-book/LanguageGuide/AdvancedOperators.html#ID630). +At a high level, the idea involved a few pieces: + +* Types like `TestCase` and `TestSuite` representing individual tests and + groups, respectively. +* A `@resultBuilder` type `TestBuilder` allowing declarative creation of test + hierarchies. +* A protocol named e.g. `TestProvider` with a requirement + `@TestBuilder static var tests: TestSuite` which suite types would + implement in order to define their tests. +* Tests defined as closures in the `static var tests` result builder above, + accepting an instance of a type named `TestContext` which allowed accessing + per-test instance storage. + +This approach seemed promising at first and satisfied many of the goals +described in the beginning of this document. But we discovered several +significant drawbacks: + +* **Type-checking performance:** Certain Result Builder usage patterns are + known to lead to poor type-checking performance, especially when the + expression is long. When describing an entire suite of tests, which may be + nested arbitrarily, the work can become exponential and lead to a noticeable + increase in build time or, in the extreme case, compiler timeouts. +* **Accessing test state:** Because tests are defined in a `static` context, + per-test state must be accessed indirectly, via a `TestContext` wrapper type. + This made accessing per-test storage more verbose than necessary, and + introduced the need for synchronization on the wrapper type. +* **Global actor isolation:** It is difficult, or perhaps impossible, to use a + global actor (most often `@MainActor`) in both the test body and the type + enclosing the test which stored its per-test state, and ensure they match. In + practice, this means that tests whose subjects include global actors are + challenging to write without lots of `await` suspension points. +* **Build-time test discovery**: It is difficult, or perhaps impossible, to + discover tests comprehensively at build time since the definition of tests + happens in result builder functions. +* **Discovery:** Using Result Builders did not fully solve the problem of + runtime test discovery; it is still necessary to locate types conforming to + `TestProvider` protocol, even though the tests within each conforming type are + trivial to gather by calling its static `tests` property. + +### Imperative test definition + +Another approach for defining tests is using a builder pattern with an +imperative style API. A good example of this is Swift’s own +[StdlibUnittest](https://github.com/apple/swift/tree/main/stdlib/private/StdlibUnittest) +library which is used to test the standard library. To define tests, a user +first creates a `TestSuite` and then calls `.test("Some name") { /* body */ }` +one or more times to add a closure containing each test. + +One problem with generalizing this approach is that it doesn’t have a way to +deterministically discover tests either after a build or while authoring +tests in an IDE (see [Test discovery](#test-discovery)). Because tests are +defined using imperative code, it may contain arbitrary control flow logic +static analysis may not be able to reason about. As a contrived example, imagine +the following: + +```swift +import StdlibUnittest + +var myTestSuite = TestSuite("My tests") +if Bool.random() { + myTestSuite.test("Foo") { /* ... */ } +} +``` + +There may be arbitrary logic (such as `if Bool.random()` here) which influences +the test suite construction, and this makes important features like IDE +discovery impossible in the general case. diff --git a/visions/using-c++-from-swift.md b/visions/using-c++-from-swift.md new file mode 100644 index 0000000000..4b43b13acc --- /dev/null +++ b/visions/using-c++-from-swift.md @@ -0,0 +1,394 @@ +# Using C++ from Swift + +## Introduction + +This document lays out a vision for the development of the "forward" half of C++ and Swift interoperability: using C++ APIs from Swift. It sets overarching goals that drive the project’s design decisions, outlines some high-level topics related to C++ interoperability, and, finally, investigates a collection of specific API patterns and potential ways for the compiler to import them. This vision is a sketch, rather than a final design for C++ and Swift interoperability. Towards the end, this document suggests a process for evolving C++ interoperability over time, and it lays out the path for finalizing the designs discussed here. + +“Reverse” interoperability (using Swift APIs from C++) is another extremely important part of the interoperability story. However, reverse interoperability has largely different goals and constraints, which necessarily mean a different design and therefore a different vision document. The [vision for reverse interoperability](https://github.com/swiftlang/swift-evolution/blob/main/visions/using-swift-from-c%2B%2B.md) has already been [accepted](https://forums.swift.org/t/accepted-a-vision-for-using-swift-from-c/62102) by the Language Workgroup. + +This document is an official feature vision document, as described in the [draft review management guidelines](https://github.com/rjmccall/swift-evolution/blob/057b2383102f34c3d0f5b257f82bba0f5b94683d/review_management.md#future-directions-and-roadmaps) of the Swift evolution process. The Language Workgroup has endorsed the goals and basic approach laid out in this document. This endorsement is not a pre-approval of any of the concrete proposals that may come out of this document. All proposals will undergo normal evolution review, which may result in rejection or revision from how they appear in this document. + +## Goals + +There are many reasons for programmers to use C++ from Swift. They might work mostly in Swift but need to take advantage of some code written in C++, anything from a small snippet to a large library. On the other end of the spectrum, they might be C++ programmers looking to adopt Swift as a memory-safe successor language, with a goal of gradually rewriting their codebases into Swift. The foremost goal of Swift's C++ interoperation is to work well for all of these use cases, removing barriers to writing Swift instead of C++, without compromising Swift as a language. + +To do this, **Swift must import C++ APIs safely and idiomatically**. Swift's memory safety is a major feature of its design, and C++'s lack of safety is a major defect. If C++'s unsafety is fully inherited when using C++ APIs from Swift, interoperability will have made Swift a worse language, and it will have undermined one of the reasons to migrate to Swift in the first place. But Swift must also make C++ APIs feel natural to use and fit into Swift's strong language idioms. Often these goals coincide, because the better Swift understands how a C++ API is meant to be used, the more unsafety and boilerplate it can eliminate from use sites. If the Swift compiler does not understand how to import an API safely or idiomatically, it should decline to import it, requesting more information from the user (likely through the use of annotations) so that the API can be imported in a way that meets Swift’s standards. + +For example, many C++ APIs traffic in iterators. Direct uses of C++ iterators are difficult to make safe: iterators are unsafe unless used correctly, and that correctness relies on complex properties (such as the lifetime or consistency of the underlying data) that are impossible to statically enforce. Iterators are also not very idiomatic in Swift because iterator values can only be meaningfully interpreted in pairs (that violate Swift's exclusivity by definition). And iterator properties are often inconsistently defined, making them hard to use. So, Swift should recognize common C++ patterns like ranges (pairs of iterators) and containers and map them into Swift `Collection`s, making them automatically work with Swift's library of safe and idiomatic collections algorithms. For example, Swift code should be able to filter and map the contents of a `std::vector`: + +```Swift +images // "images" is of type std::vector + .filter { $0.size > 256 } + .map(UIImage.init) +``` + +This level of idiomatic interoperation allows programmers to immediately see the benefits of adopting Swift, even when using C++ APIs. It makes the two languages work cleanly together. It removes the need for extensive C or Objective-C bridging layers between C++ libraries and their Swift clients, which are often the source of bugs, performance problems, and expressivity restrictions. And when combined with "reverse" interop that exposes Swift APIs to C++, it allows Swift to be added incrementally to an existing C++ codebase and interoperate on a file-by-file basis, enabling it to function as a viable successor language for programmers looking to move past C++. + +Swift has had great success as a successor language to Objective-C with this approach of bidirectional, file-by-file interoperation. While the constraints and trade-offs of interoperating with C++ are vastly different from Objective-C, the same overall philosophy can largely be applied to make Swift an excellent C++ successor, because it permits the incremental adoption of Swift in a codebase rather than relying on all-at-once rewrites. To make this viable, interoperation must not rely on radically changing interfaces on either side of the language barrier. For Objective-C, Swift takes the approach of largely incorporating Objective-C by inclusion: most major Objective-C features have corresponding Swift features that are at least as expressive. This is not desirable for C++, most importantly because most of the unsafety of C++ arises from its widespread and idiomatic use of unmanaged pointer and reference types; naively translating these all to `UnsafePointer`s would create an unidiomatic and unsafe mess. So successful import of C++ to Swift must rely on recognizing patterns of how these features are used, perhaps with user guidance, and mapping them to more idiomatic Swift constructs. This idea has proven successful with Objective-C, such as methods with `NSError**` parameters being translated to `throws`; C++ will just need to use it more pervasively. + +Because of this, importing C++ APIs into Swift is a difficult task that must be handled with care. Almost every goal of C++ interoperability will be in tension with Swift's safety requirements. Swift must strike a careful balance in order to maintain Swift's safety without reintroducing the development, performance, or expressivity costs of an intermediate wrapper API. + +Safety is a top priority for the Swift programming language, which creates a tension with C++. While Swift enforces strong rules around things like memory safety, mutability, and nullability, C++ largely makes the programmer responsible for handling them correctly, on pain of undefined behavior. Simply using C++ APIs should not completely undermine Swift's language guarantees, especially guarantees around safety. At a minimum, imported C++ APIs should generally not be less safe to use from Swift than they would be in C++, and C++ interoperability should strive to make imported APIs *safer* in Swift than they are in C++ by providing safe API interfaces for common, unsafe C++ API patterns (such as iterators). When it is possible for the Swift compiler to statically derive safety properties and API semantics (i.e., how to safely use an API in Swift) from the C++ API interface, C++ interoperability should take advantage of this information. When that is not possible, C++ interoperability should provide annotations to communicate the necessary information to use these APIs safely in Swift. When APIs cannot be used safely or need careful management, Swift should make that clear to the programmer. As a last resort, Swift should make an API unavailable if there's no reasonable path to a sufficiently safe Swift interface for it. + +C++ interoperability should strive to have good diagnostics. Diagnostics that report source locations for a C++ API should refer to the API's original declaration in a C++ header, not to a location in a synthesized interface file. When a C++ API can be imported into Swift, diagnostics from misusing it (e.g. type errors when passing it an argument of the wrong type) should be similar to the diagnostics for analogous misuses of a Swift API. When a C++ API cannot be imported, attempts to use it should result in a clear error indicating why the API could not be imported, and the diagnostics should suggest specific ways that the programmer could make it importable (for example, by adding annotations). + +C++ provides tools to create high-performance APIs. The Swift compiler should embrace this. Interop should not be a significant source of overhead, and performance concerns should not be a reason to continue using C++ to call C++ APIs rather than Swift. + +C++ is a multi-paradigm language, designed to fit many use cases and allow many different programming styles. Different codebases often express the same concept in different ways. There is no prevailing consensus among C++ programmers about the right way to express specific concepts: how to name types and methods, how much to use templates, when to use heap allocation, how to propagate and handle errors, and so on. This creates problems for importing C++ APIs into Swift, which tends to have stronger conventions, some of which are backed by language rules. For instance, it is a common pattern in some C++ codebases to have classes that are only (or at least mostly) intended to be heap-allocated and passed around by pointer; consider this example: + +```cpp +// StatefulObject has object identity and reference semantics: +// it should be constructed with "create" and used via a pointer. +struct StatefulObject { + StatefulObject(const StatefulObject&) = delete; + StatefulObject() = delete; + + StatefulObject *create() { return new StatefulObject(); } +}; +``` + +This type is not intended to be used directly as the type of a local variable or a `std::vector` element. Values of the type are allocated on the heap by the `create` method and passed around as a pointer. This is weakly enforced by the way the type hides its constructors, but mostly it's communicated in the documentation and by the overall shape of the API. There is no C++ language feature or programming pattern that directly expresses these semantics. + +If `StatefulObject` were written idiomatically in Swift, it would be defined as a `class` to make it a reference type. This is an example of how Swift defines clear patterns for naming, generic programming, value categories, error handling, and so on, which codebases are encouraged to use as standard practices. These well-defined programming patterns make using Swift APIs a cohesive experience, and C++ interoperability should stive to maintain this experience for Swift programmers using C++ APIs. + +To achieve that, the compiler should map C++ APIs to one of these specific Swift programming patterns. In cases where the most appropriate Swift pattern can be inferred by the Swift compiler, it should map the API automatically. Otherwise, Swift should ask programmers to annotate their C++ APIs to guide how they are imported. For example, Swift imports C++ types as structs with value semantics by default. Because `StatefulObject` cannot be copied, Swift cannot import it via the default approach. To be able to use `StatefulObject`, the user should annotate it as a reference type so that the compiler can import it as a Swift `class`. Information on how to import APIs, such as `StatefulObject`, cannot always be statically determined (for example, `StatefulObject` might have been a move-only type, a singleton, or RAII-style API). The Swift compiler should not import APIs like `StatefulObject` for which it does not have sufficient semantic information. It is not a goal to import every C++ API into Swift, especially without additional, required information to present the API in an idiomatic way that promotes a cohesive Swift experience. + +Because of the difference in idioms between the two languages, and because of the safety concerns when exposing certain APIs to Swift, a C++ API might look quite different in Swift than it does in C++. It is a goal of C++ interoperability to provide a clear, well-defined mapping for whether and how APIs are imported into Swift. Users should be able to read the C++ interoperability documentation to have a good idea of how much of their API will be able to imported and what it will look like. Swift should also provide tools for inspecting what a C++ API will look like in Swift, and these tools should call out notable parts of the API that were not imported. + +## The approach + +Many C++ constructs have a clear, analogous mapping in Swift. These constructs can be easily and automatically imported to their corresponding Swift constructs. For example, C++ `enum`s and `enum class`es can be mapped to Swift `enum`s, and C++ operators can usually be mapped to similar Swift operators. Sometimes, to promote Swift’s idioms, operators should be imported "semantically" rather than directly. For example, `operator++` should map to a `successor` method in Swift, and `operator*` should map to a `pointee` property. Another example is a C++ `namespace`, which can be mapped to an empty `enum`, a common pattern in Swift. + +Swift and C++ both support object-oriented programming, and most C++ object-oriented features can be mapped trivially onto Swift counterparts: a member function in C++ translates to a method in Swift, and so on. Like Swift, C++ programming is often focused around types with value semantics, and the natural default when importing a C++ `class` or `struct` is to map it to a Swift `struct`. Copying and destroying this `struct` can simply invoke the corresponding C++ special members as appropriate. This fits nicely into Swift’s existing object model and allows most types to automatically work in Swift in an idiomatic, natural way. + +However, not all C++ `struct`s and `class`es are intended to be used as value types. The `StatefulObject` example in the Goals section shows how the basic tools provided by C++ are often used in idiomatically different ways, and this can be hard to automatically detect when importing the type. This is one example of a deeper conundrum when importing C++ APIs. A more fundamental place to see this is with memory management. + +In Objective-C, it's fairly straightforward for ARC to ensure that data is valid when it's used. Almost all data in Objective-C is represented with either a fundamental type (such as `double` or `BOOL`) or a reference-counted object type (such as `NSString *`). Values of fundamental types can be safely used without any concern about memory management, and reference-counted objects can be safely managed by ensuring that they're retained while they're still potentially used. The compiler will sometimes be more conservative about reference counts than a human would be, extending the lifetime of an object longer than is strictly necessary, but this usually doesn't change the semantics of the program. ARC doesn't manage C pointers, but it's relatively rare to work with C pointers in Objective-C, and it's rarer still to work with a C pointer that has a dependency on a managed object (although exceptions do exist, such as [`NSData`'s `-bytes` method](https://developer.apple.com/documentation/foundation/nsdata/1410616-bytes?language=objc)). This kind of dependency is problematic for safe memory management because the language often does not know about it and cannot ensure that the backing object stays valid while the pointer is being used. Swift's Objective-C interop has treated these as special cases (with an attribute) and dealt with them individually; while this solution is imperfect in several ways, its rarity has made it a low priority to fix. + +In contrast, it's very common for C++ APIs to work with unmanaged pointers, references, and views into other objects. The lifetime rules for using these correctly are inconsistent and sometimes unique to an API. As an example, consider three values: a value of type `std::vector`, a reference returned from that vector's `operator[]`, and an iterator returned from that vector's `begin()` method. At first glance, these values look similar to the compiler: they are all either pointers or class objects containing pointers. But each has its own semantics and expected use (especially concerning lifetime), and these differences are not conveyed explicitly in the source. The vector is a value type that can be copied, but copies can be expensive, and iterators and references into the vector are only valid for a specific copy. The result of `operator[]` is a mutable projection of a specific element, dependent on the vector for validity; but the value of that element can be copied out of the reference to get an independent value, and that is often how the operator is used. The iterator is also a projection, dependent on the vector for validity, but it must be used in conjunction with other iterators or with the vector itself in certain careful ways, and some operations will invalidate it completely. + +So there is a conundrum where superficially similar language constructs in C++ are used to express idiomatic patterns that are vastly different in their impact. The only viable approach for addressing this problem is to pick off these patterns one at a time. The Swift compiler will know about many possible C++ API patterns. If a C++ API has semantic annotations telling Swift that it follows a certain pattern, Swift will try to create a Swift interface for it following the rules of that pattern. In the absence of those annotations, Swift will try to use heuristics to recognize an appropriate pattern. If this fails, Swift will make the API unavailable. + +Consider how this applies to the `std::vector` example. `std::vector` maps over well as a Swift value type. Its `operator[]` can be imported as a Swift `subscript`, and the importer can take advantage of the fact that it returns a reference to allow elements to be efficiently borrowed. And while C++ iterators in general pose serious lifetime safety problems in Swift, Swift can recognize the common `begin()`/`end()` pattern and import it as a safe Swift iterator that encapsulates the unsafety internally. The following sections will go into detail explaining how each of these specific API patterns can be recognized in a C++ codebase. + +### Importing types + +One of the most common uses of this "API patterns" concept concerns the import of types. Swift types fall into two categories: value types and reference types. Copying a value of a reference type produces a new reference to the same underlying object, similar to an intrusive `std::shared_ptr` in C++ or a class type in Java. Copying a value of a value type recursively copies the components of the type to produce an independent value, similar to the behavior of a struct in C or the default behavior of a class type in C++. Furthermore, both kinds of types must always be copyable, although there are plans in the works to allow types to restrict this. + +Types in C++ do not always fit cleanly into this model, and they cannot always be automatically mapped to it even when they do. Many C++ classes are meant to be used as value types, but there are also quite a few C++ classes that Swift programmers would think of as reference types. The difference is not necessarily obvious in source. The closest bit of information that C++ provides directly is whether and how a class has changed its value operations (its copy and move constructors, its assignment operators, and its destructor). A reference type is more likely to delete its copy operations, while a value type is likely to still provide them. But this is not a reliable signal, because some value types are meant to be uncopyable (or even unmovable), while some reference types leave their copy operations intact, either by neglect or to enable objects to be easily cloned when necessary. Furthermore, C++ types sometimes have a more hybrid semantics: iterators, for example, can be used like values, but they're not independent from their underlying collection and in some ways act like references. And some C++ types aren't meant to be used as normal values at all; instead they fill specific idiomatic purposes, like the proxy element references used by `std::vector`, or scoped-destructor types like `std::lock_guard`. + +### Reference types + +Reference types generally fit well into the existing Swift model, and there is little need to restrict them. The safety properties of managed reference types imported from C++ are generally similar to both Swift's own classes and classes imported from Objective-C. The design below also includes unmanaged reference types, which are less safe than managed types, but not more unsafe than writing the code in C++. Overall, this allows C++ interoperability to offer a clear, native-feeling mapping for several common C++ API patterns. + +#### Criteria for importing as a reference type + +Whether a C++ class type is appropriate to import as a reference type is a complex question, and there are several criteria that go into answering it. + +The first criterion is whether object identity is part of the "value" of the type. Is comparing the address of two objects just asking whether they're stored at the same location, or it is deciding whether they represent the "same object" in a more significant sense? For example, consider a computer game that uses a world model where each object of the `GameObject` class represents a different game object. Copying an object actually means making a second object in the game world, one which initially shares the same internal data as another. This is a classic use of reference semantics, and `GameObject` is clearly a reference type. In contrast, a different game might use a world model where the `GameObjectState` class holds a snapshot of the current state of a game object. The actual game object is identified as part of that snapshot, but it's not synonymous with the snapshot, and copying the snapshot just produces an equivalent snapshot of the same object. This design does not rely on object identity; if `GameObjectState` is a reference type, it is because of some other factor. + +The second criterion is whether the C++ class is polymorphic. Does the class have subclasses whose objects contain additional data or behave differently from objects of the parent class? Swift value types cannot be directly polymorphic, so if polymorphism is an important part of a C++ class, it must be imported as a reference type. The most common indicator of a polymorphic C++ class is having `virtual` methods. More rarely, some C++ classes behave polymorphically but intentionally avoid having `virtual` methods to eliminate the memory overhead of a v-table pointer in every object. + +The third and final criterion whether objects of the C++ class are always passed around by reference. Are objects predominantly passed around using a pointer or reference type, such as a raw pointer (`*`), raw reference (`&` or `&&`), or smart pointer (like `std::unique_ptr` or `std::shared_ptr`)? When passed by raw pointer or reference, is there an expectation that that memory is stable and will continue to stay valid, or are receivers expected to copy the object if they need to keep the value alive independently? If objects are generally allocated and remain at a stable address, even if that address is not semantically part of the "value" of an object, the class may be idiomatically a reference type. This will sometimes be a judgment call for the programmer. + +Most of these criteria are not possible for a compiler to answer automatically by just looking at the code. A compiler cannot know the semantic meaning of object identity for a class type. Nor can can it know whether it is looking at a representative sample of how a type is passed around in a project. Classes satisfying these criteria will have to be annotated somehow to tell the compiler to import them as Swift classes. The one exception is that it might be reasonable to assume that a C++ class with `virtual` functions should be imported as a reference type. + +#### Object management + +Swift generally promises to make sure that objects are valid when used. This is an important part of Swift's core language goal of memory safety. Ideally, when Swift imports a C++ class as a reference type, it will import it as an appropriately managed type that receives the same guarantees as native Swift and imported Objective-C classes. + +It's useful to split the object-management problem into two questions: how objects are managed and whether they can be managed automatically. + +There are three common patterns for managing reference object lifetimes in C++. Swift should endeavor to support all three of them: + + - **Immortal** reference types are not designed to be managed individually by the program. Objects of these types are allocated and then intentionally "leaked" without tracking their uses. Sometimes these objects are not truly immortal: for example, they may be arena-allocated, with an expectation that they will only be referenced from other objects within the arena. Nonetheless, they aren't expected to be individually managed. + + The only reasonable thing Swift can do with immortal reference types is import them as unmanaged classes. This is perfectly fine when objects are truly immortal. If the object is arena-allocated, this is unsafe, but it's essentially an unavoidable level of unsafety given the choices of the C++ API. + + - **Unique** reference types are owned by a single context at once, which must ultimately either destroy it or pass ownership of it to a different context. There are two common idioms for unique ownership in C++. The first is that the object is passed around using a raw pointer (or sometimes a reference) and eventually destroyed using the `delete` operator. The second is that this is automated using a move-only smart pointer such as `std::unique_ptr`. This kind of use of `std::unique_ptr` is often paired with "borrowed" uses that traffic in raw pointers temporarily extracted from the smart pointer; in particular, method calls on the class via `operator->` implicitly receive a raw pointer as `this`. + + The introduction of [non-copyable types](https://forums.swift.org/t/pitch-noncopyable-or-move-only-structs-and-enums/61903) will allow Swift to directly support unique reference types as managed types. The main challenge in doing this will be understanding the ownership conventions for different C++ APIs. If ownership of a class is known to be passed around with a smart pointer like `std::unique_ptr`, then APIs trafficking in raw pointers can be assumed to be working with a borrow; that would support importing as a managed type. Otherwise, Swift will have to put the programmer in charge and either import as an unmanaged type or use a wrapper type like `Unmanaged` to mediate APIs with unknown conventions. + + - **Shared** reference types are reference-counted with custom retain and release operations. In C++, this is nearly always done with a smart pointer like `std::shared_ptr` rather than expecting programmers to manually use retain and release. This is generally compatible with being imported as a managed type. Shared pointer types are either "intrusive" or "non-intrusive", which unfortunately ends up being relevant to semantics. `std::shared_ptr` is a non-intrusive shared pointer, which supports pointers of any type without needing any cooperation. Intrusive shared pointers require cooperation but support some additional operations. Swift should endeavor to support both. + + As with unique reference types, shared reference types in C++ often have APIs that take raw pointers, such as methods on the class type. Unlike unique references, these cannot necessarily be thought of as borrows. Shared reference types are copyable types, and as a general rule, borrowed copyable values can be copied to produce owned values. However, in C++ terms, this would require constructing a shared pointer value from a raw pointer, which in general is not possible to do correctly for non-intrusive shared pointers. It's fine for Swift to *call* APIs that take raw pointers for shared reference types, but it cannot *implement* them without having a way to prevent copying the reference. + + `std::shared_ptr` uses atomic reference-counting in both its intrusive and non-intrusive modes. Non-atomic smart pointer types are supportable, but the imported class type must not be `Sendable`. + +[Examples of each of these are given below.](## Examples and Definitions) + +If a type is annotated as using one of these reference-type patterns, uses of the type in C++ that do not have a consistent interpretation under the pattern will be impossible to import. For example, suppose that `GameObject` is annotated as a shared reference type that uses `std::shared_ptr`. A C++ API that takes a parameter of type `std::unique_ptr`, or a different shared pointer class from the annotated one, must be made unavailable. (This would also include differences in secondary template arguments, such as the `Deleter` template argument of `std::unique_ptr`.) Similarly, a C++ API that takes a `GameObject` as an r-value must be made unavailable. + +Swift doesn't have to force programmers to pick one of these patterns specifically. Without further information, foreign reference types can be imported as an unmanaged class type. There would be an operation on an object to delete it, but if that's not safe to use, the programmer could simply not use it. This behavior would allow types with reference semantics to be expressed with little effort at the cost of complete safety. For some types this may be acceptable or even necessary. However, Swift should strive to make it easy to import types as managed class types, especially for APIs that already make extensive use of smart pointers. + +### Value types + +For the purposes of this document, a value type is any type that doesn't make sense to import as a reference type. Value types can be copied and destroyed, and the copies will be independent from each other, at least at the direct level. That is, part of the value of a value type might be a reference to an object (e.g. if it has a stored property of class type), and different copies of the value will share the same reference, but this reference can still be replaced in one copy without affecting other copies. + +Swift expresses value types using `struct`s and `enum`s. Copying a Swift `struct` normally does the same thing that copying a C or C++ `struct` does by default: it recursively copies all of the stored properties of the type. In C++, of course, that behavior can be customized with user-defined copy/move constructors and destructors; while Swift doesn't have an equivalent feature, it does still honor the operations specified in C++, so that destroying a value of the imported type in Swift calls the C++ destructor and so on. + +It's useful to call out three categories of value types. These categories don't necessarily change how the type is actually used in Swift, but they are essential to describing the interop story, safety and performance properties, potential API restrictions, and the user model more generally. + +#### Simple data types + +This document will refer to C++'s trivially-copyable value types that do not contain pointers as “simple data types.” This category includes fundamental types, such as integers and floating-point types, as well as aggregate types composed only of other simple data types. Simple data types have trivial value operations and never carry lifetime dependencies on other values. Simple data types and operations on them generally don't need any special restrictions in Swift. + +Swift will assume by default that lifetime dependencies aren't carried in integer types even though technically pointers can be reinterpreted into integer types. It's very uncommon for C++ APIs to violate this assumption: reinterpeting pointers as integers is important to a fair amount of code, but usually it's localized and transient and the pointer doesn't get passed around as an integer long-term. + +#### Self-contained types + +This category is a superset of the simple data types which also includes types with internally-managed pointers. Like simple data types, these types and their operations usually don't need any special restrictions in Swift. However, the fact that they can be non-trivial types can complicate some things. + +Swift will assume by default that a C++ `class` type which contains pointers but also provides user-defined special members is self-contained. This is a fairly reliable heuristic, but more consideration may be required in order to handle cases such as `std::vector`, which can carry lifetime dependencies indirectly even though it does manage the pointers it stores directly. In any case, Swift must provide annotations that allow the default to be corrected. + +#### View types + +This document will refer to value types that are not self-contained types as "view types". These types include pointers themselves as well as types which are recursively composed of other view types. The pointers held by view types refer to memory that is *not owned* by the pointer type (making view types a “view” into that memory rather than a value that encapsulates it). View types usually carry a dependency on some other value and must be used carefully to be safe. + +While trivially-copyable view types are very similar to simple data types with respect to their trivial value operations, they differ in the fact that, while they themselves are not inherently unsafe, they may be used in unsafe APIs (discussed later). + +### Projections + +The safety problem posed by view types is very broad. If we apply this categorization to the types offered natively by Swift, most types are self-contained; only the unsafe pointer types are view types. Swift also encourages view types to be encapsulated within types and exposed in only carefully-scoped ways, like with a `with...` API. These properties are not true for many C++ APIs, which offer a broad spectrum of novel view types and ways to project them out of managed types. Swift does not currently offer strong language tools for dealing with these projections. + +Probably the most common pattern of projection in C++ APIs is a method that returns a reference or pointer to memory that depends on `this`. Consider this example: + +```cpp +const std::string &getName() const { return this->_name; } +``` + +There are several possibilities for dealing with this pattern. For example, Swift could wrap it in a `_read` accessor, implicitly encoding that the reference is only available during an access to the containing object. Swift could also add explicit lifetime-dependency features, allowing this to be treated as a return of a borrowed value. Alternatively, Swift could simply force the return value to be immediately copied after return, as if the call actually returned an owned value. It's unclear which of these would be the best approach; perhaps a combination would. This is something that will need to be investigated over time, incorporating the experience of the community with using this feature. + +But there are also many projections in C++ that don't match the above pattern. Consider the following API which returns a vector of internal pointers: +```cpp +std::vector OwnedType::projectsInternalStorage(); +``` + +Or this API which fills in a pointer that has two levels of indirection: +```cpp +void VectorLike::begin(int **out) { *out = data(); } +``` + +Or even this global function that projects one of its parameters: +```cpp +int *begin(std::vector *v) { return v->data(); } +``` + +Swift will need to decide how to handle projections, and more generally the use of view types, that it doesn't recognize how to make safe. This may come with difficult trade-offs between usefulness and safety. Consider the `projectsInternalStorage` API above, and pretend that we aren't able to recognize a pattern here --- we don't know that the pointers just depend on the `OwnedType` object. Often, uses of this kind of API can be made safe in practice: when there's a `std::vector` of pointers, there's probably some straightforward thing making those pointers valid. If Swift decides not to import this API just because it might be used unsafely, that could do serious damage to the usability of C++ interop. At the very least, it should be possible to wrap such an API in a safer Swift abstraction, which will be impossible if it isn't imported at all. + +The trade-offs here are an open question for the Swift evolution process to eventually determine. + +### Iterators + +Both Swift and C++ have powerful libraries for algorithms and iterators. The standard C++ iterator API interface lends itself to the Swift model, allowing C++ iterators and ranges to be mapped to Swift iterators and sequences with relative ease. These mapped APIs are idiomatic, native Swift iterators and sequences; their semantics match the rest of the Swift language and Swift APIs compose around them nicely. By taking on Swift iterator semantics, iterators that are imported in this way are able to side-step most or all of the issues that other projects have (described above). + +Swift's powerful suite of algorithms match and go beyond the standard library algorithms provided by C++. These algorithms compose on top of protocols such as Sequence, which C++ ranges should automatically conform to. These Swift APIs and algorithms that operate on Swift iterators and sequences should be preferred to their C++ analogous, as they fit into the rest of the language naturally. However, algorithms are not the only API which operate on iterators and sequences and other C++ APIs must still be useable from Swift. The best way to represent C++ APIs that take one or many iterators (potentially pointing at the same range) is not clear and will need to be explored during the evolution processes. + +### Mutability + +Swift and C++ use very different models for controlling mutability. C++ defaults to treating methods, pointers, and references as non-`const`, and any `const`-ness can be easily cast away, which together mean that `const`-ness is not always a very reliable signal. C++ also encodes mutability into the type system in a first-class way, allowing functions to be overloaded bsaed on whether an argument is `const` or not. These decisions don't always align well with Swift, which defaults to immutability and relies on local information to strictly enforce it. + +One consequence is that C++ codebases which haven't adopted `const` correctness can be confusing to awkward to use from Swift because many operations which are not actually mutating appear to require mutability. Swift should encourage C++ codebases to adopt `const` on methods and values that are used in Swift. + +Overloaded functions should be clearly be disambiguated in Swift through naming. Mutability is a place where programmers may need to intervene and provide Swift with more information to help promote idiomatic APIs that are expressive and feel natural in Swift. + +Swift should also assume that C++ will not mutate values through `const` pointers and references, even though technically `const`-ness can cast away. It is reasonable for Swift to assume that C++ APIs will obey their type signatures, and the alternative would be very onerous for interop users. + +As discussed in the "View types" section, the Swift compiler must make assumptions about the C++ APIs that it is importing, and mutability is another place where Swift will need to make reasonable (not conservative) assumptions about the APIs that it is importing, promoting C++‘s weak notion of `const` to Swift’s much stricter ideal. + +Programmers will see some benefits from Swift's stronger mutability model immediately. Consider this example: +```cpp +// C++ +void append_n_times(std::string& s, const std::string& m, size_t n) { + for (size_t i = 0; i < n; ++i) + s += m; +} +``` +```swift +// Swift +var local: std.string = "a" +append_n_times(&local, local, 5) +``` +`append_n_times` misbehaves if `s` and `m` alias because `m` will be modified by the append: `s` will contain `2^n` copies of the original string instead of `n + 1`. This is not possible when called from Swift because Swift does not permit mutable arguments to be aliased, so the argument for `m` will be copied. (And if the programmer requests that this copy not happen, e.g. by explicitly borrowing that argument, the call will be statically diagnosed as ill-formed.) + +### Computed properties + +Value vs. reference types and mutability may require user input to map correctly in Swift, but constructs like iterators, getters, and setters can largely be imported automatically. Getters, setters, and subscripts can all be imported into Swift as computed properties. While many C++ codebases define a getter and setter pair, computed properties are the idiomatic way to handle this API pattern in Swift. And computed properties are not just about syntax, they also help promote safety and performance. For example, a C++ getter that returns a reference can be mapped into a generalized accessor in Swift that leverages coroutines to safely yield out its storage to the caller. This generalized accessor pattern allows safe and efficient access to C++ references in Swift and is another example of the more general philosophy for importing APIs: when Swift understands the semantics of an API it can map that API pattern to a strict Swift idiom that is safe, performant, and feels native, so that users get most of the benefits of Swift, even when calling C++ APIs. + +### Templates and generic APIs + +C++ and Swift use very different models for generic programming. C++ templates are eagerly instantiated for each set of template arguments they're used with, with type checking done separately for each instantiation based on the exact types in use. In contrast, Swift generics are type-checked once based on the requirements they impose on their type parameters, and while they can be specialized for a particular set of type arguments, that is not required or even always possible. + +This difference makes using generic C++ APIs in Swift difficult. Generic code in Swift will not be able to use C++ templates generically without substantial new language features and a lot of implementation work. Allowing C++ templates to be used on concrete Swift types is theoretically more feasible but still a major project because of the *ad hoc* nature of type constraints in templates. If this feature is ever pursued, it will likely require substantial user guidance through annotations or wrappers around imported APIs. + +Fortunately, these limitations do not apply when using C++ types with Swift generics. Unconstrained generics can be used with C++ types without any further work, and programmers can simply add protocol conformances to concrete C++ types in order to use them with constrained generics. + +[This forum post](https://forums.swift.org/t/bridging-c-templates-with-interop/55003) (Bridging C++ Templates with Interop) goes into depth on the issue of importing C++ templates into Swift. + +## The standard library + +Swift should provide an overlay for the C++ standard library to assist in the import of commonly used APIs, such as containers. This overlay should also provide helpful bridging utilities, such as protocols for handling imported ranges and iterators, or explicit conversions from C++ types to standard Swift types. + +C++ aims to provide sufficient tools to implement many features in its standard library rather than the compiler. While the Swift compiler also attempts to do this, it is not a goal in and of itself, resulting in many of C++'s analogous features being implemented in the compiler: tuples, pairs, reference counting, ownership, casting support, optionals, and so on. In these cases, the Swift compiler will need to work with both the C++ standard library and the Swift overlay for the C++ standard library to import these APIs correctly. + +The reverse is also true: C++ interop may require library-level Swift utilities to assist in the import of various C++ language concepts, such as iterators. To support this case, a set of Swift APIs specific to C++ interop will be imported implicitly whenever a C++ module is imported. These APIs should not have a dependency on the distinct C++ standard library or its overlay. + +## Evolution + +C++ interoperability is a huge feature that derives most of its benefit from the combination of its component features; for example, methods can't be used without types. C++ interop should be made useful to programmers before all component pieces have necessarily gone through evolution, both for the benefit of programmers wanting to use this feature, and for compiler developers designing and implementing the feature. + +C++ interoperability should bring in as many APIs as possible, even if they haven't gone through evolution. Swift evolution will progressively work through these APIs, formalizing them, and eventually interop will become a stable feature. Until a critical mass of APIs have been brought Swift's evolution process, a versioning scheme will allow C++ interoperability to be adopted and remain source stable while being evolved. Versions may be rapidly deprecated, but will be independent of Swift compiler versions, allowing source breaks even in minor compiler updates without disturbing adopters. + +This document allows specific, focused, and self contained evolution proposals to be created for individual pieces of the language and specific programming patterns by providing goals that lend themself to this kind of incremental design and evolution (by not importing everything and requiring specific mappings for specific API patterns) and by framing interop in a larger context that these individual evolution proposals can fit into. + +## Tooling and build process + +As a supported language feature, C++ and Swift interoperability must work well on every platform supported by Swift. In a similar vein, tools in the Swift ecosystem should be updated to support interoperability features. For example, SourceKit should provide autocompletion, jump-to-definition, etc. for C++ functions, methods, and types, and lldb should be able to print C++ types even in Swift frames. Finally, the Swift package manager should be updated with the necessary features to support building C++ dependencies. + +This document outlines a strategy for importing APIs that rely on semantic information from the user. In order to make this painless for users across a variety of projects, Swift will need to provide both inline annotation support for C++ APIs and side-file support for APIs that cannot be updated. For Objective-C, this side-file is an APINotes file. As part of Swift and C++ interoperability, APINotes will either need to be updated to support C++ APIs, or another kind of side-file will need to be created. + +## Appendix 1: Examples and Definitions + +**Reference Types** have reference semantics and object identity. A reference type is a pointer (or “reference”) to some object which means there is a layer of indirection. When a reference type is copied, the pointer’s value is copied rather than the object’s storage. This means reference types can be used to represent non-copyable types in C++. For real-world examples of C++ reference types, consider LLVM's [`Instruction` class](https://llvm.org/doxygen/IR_2Instruction_8h_source.html) or Qt's [`QWidget` class](https://github.com/qt/qtbase/blob/dev/src/widgets/kernel/qwidget.h). + +**Manually Managed Reference Types** + +Here a programmer has written a very large `StatefulObject` which contains many fields: + +```cpp +struct StatefulObject { + std::array names; + std::array places; + // ... + + StatefulObject(const StatefulObject&) = delete; + StatefulObject() = delete; + + StatefulObject *create() { return new StatefulObject(); } +}; +``` + + +Because this object is so expensive to copy, the programmer decided to delete the copy constructor. The programmer also decided that this object should be allocated on the heap, so they decided to delete the default constructor, and provide a create method in its place. + +In Swift, this `StatefulObject` should be imported as a reference type, as it has reference semantics. + +**API Incorrectly Using Reference Types** + +Here someone has written an API that uses `StatefulObject` as a value type. + +```cpp +StatefulObject makeAppState(); +``` + +This will invoke a copy of `StatefulObject` which violates the semantics that the API was written with. To be usable from Swift, this API needs to be updated to pass the object indirectly (by reference): + +```cpp +StatefulObject *makeAppState(); // OK +const StatefulObject *makeAppState(); // OK +StatefulObject &makeAppState(); // OK +const StatefulObject &makeAppState(); // OK +``` + +**Immortal Reference Types** + +Instances of `StatefulObject` above are manually managed by the programmer, they create it with the create method and are responsible for destroying it once it is no longer needed. However, some reference types need to exist for the duration of the program, these reference types are known as “immortal.” Examples of these immortal reference types might be pool allocators or app contexts. Let’s look at a `GameContext` object which allocates (and owns) various game elements: + +```cpp +struct GameContext { + // ... + + GameContext(const GameContext&) = delete; + + Player *createPlayer(); + Scene *createScene(); + Camera *createCamera(); +}; +``` + +Here the `GameContext` is meant to last for the entire game as a global allocator/state. Because the context will never be deallocated, it is known as an “immortal reference type” and the Swift compiler can make certain assumptions about it. + +**Automatically Managed Reference Types** + +While the `GameContext` will live for the duration of the program, individual `GameObject` should be released once they’re done being used. One such object is Player: + +```cpp +struct GameObject { + int referenceCount; + + GameObject(const GameObject&) = delete; +}; + +void gameObjectRetain(GameObject *obj); +void gameObjectRelease(GameObject *obj); + +struct Player : GameObject { + // ... +}; +``` + +Here Player uses the `gameObjectRetain` and `gameObjectRelease` function to manually manage its reference count in C++. Once the `referenceCount` hits `0`, the Player will be destroyed. Manually managing the reference count is prone to errors, as programmers may forget to retain or release the object. Fortunately, this kind of reference counting is something that Swift is very good at. To enable automatic reference counting, the user can specify the retain and release operations via attributes directly on the `GameObject`. This means the programmer no longer needs to manually call `gameObjectRetain` and `gameObjectRelease`; Swift will do this for them. They will also benefit from the suite of ARC optimizations that Swift has built up over the years. + +**Owned types** “own” some storage which can be copied and destroyed. An owned type must be copyable and destructible. The copy constructor must copy any storage that is owned by the type and the destructor must destroy that storage. Copies and destroys must balance out and these operations must not have side effects. Examples of owned types include `std::vector` and `std::string`. + +**Trivial types** are a subset of owned types. They can be copied by copying the bits of a value of the trivial type and do not need any special destruction logic. Examples of trivial types are `std::array` and `std::pair`. + +**Pointer types** are trivial types that hold pointers or references to some un-owned storage (storage that is not destroyed when the object is destroyed). Pointer types are *not* a subset of trivial types or owned types. Examples of pointer types include `std::string_view` and `std::span` and raw pointer types such as `int *` or `void *`. + +**Projections** are values rather than types. An example of a method which yields a projection is the `c_str` method on `std::string`. + +```cpp +struct string { // String is an owned type. + char *storage; + size_t size; + + char *c_str() { return storage; } // Projects internal storage +``` + +Iterators are also projections: + +```cpp + char *begin() { return storage; } // Projects internal storage + char *end() { return storage + size; } // Projects internal storage +``` + +Because `string` is an owned type, the Swift compiler cannot represent a projection of its storage, so the `begin`, `end`, and `c_str` APIs are not imported. A projection is only valid as long as the storage it points to is valid. Projections of reference types are usually safe because reference types have storage with long, stable lifetimes, but projections of owned types are more dangerous because the storage associated with a specific copy usually has a much shorter lifetime (therefore most of these projections of owned storage cannot yet be imported). + + +## Appendix 2: Lifetime and safety of self-contained types and projections + +The following section will go further into depth on the issues with using projections of self contained types in Swift, rather than proposing a solution on how to import them. Let’s start with an example Swift program that naively imports some self-contained type and returns a projections of it: + +```swift +var v = vector(1) +let start = v.begin() +doSomething(start) +fixLifetime(v) +``` + +To understand the problem with this code, the following snippet highlights where an implicit copy is created and destroyed: + +```swift +var v = vector(1) +let copy = copy(v) +let start = copy.begin() +destroy(copy) +doSomething(start) +fixLifetime(v) +``` + +Here, because Swift copies `v` into a temporary with a tight lifetime before the call to `begin`, `v` projects a dangling reference. This is an example of how subtly different lifetime models make using C++ types from Swift hard, if their semantics aren’t understood by the compiler. + +To make these APIs safe and usable, Swift cannot import unsafe projections of types that own memory, because they don’t fit the Swift model. Instead, the Swift compiler can try to infer what, semantically, the API is trying to do, or the library author can provide this information via annotations. In this case, the Swift compiler can infer that begin returns an iterator, which Swift can represent through the existing, safe Swift iterator interface. In the example above, “start” is a pointer type. Using this pointer returned by the “begin” method is unsafe, but the type of start itself is not unsafe. In other words, safety restrictions need not be applied to pointer types themselves but rather their unsafe uses. + +C++ often projects the storage of owned types. C++ is able to tie the lifetime of the projection to the source using lexcal scopes. Because there is a well-defined, lexical point in which objects are destroyed, C++ users can reason about projection’s lifetimes. While these safety properties are less formal than Swift, they are safety properties none-the-less, and form a model that works in C++. + +This model cannot be adopted in Swift, however, because the the same lexical lifetime model does not exist. Further, projections of self-contained types are completely foreign concept in Swift, meaning users aren’t familiar with programming in terms of this lexical model, and may not be aware of the added (implicit) constraints (that is, when objects are destroyed). Swift’s language model is such that returning projections from a copied value, even in smaller lexical scope, should be safe. In order to allow projections of self-contained types, this assumption must be broken, or C++ interoperability must take advantage of Swift ownership features to associate the lifetime of the projection to the source. + +The following example highlights the case described above: + +```swift +func getCString(str: std.string) -> UnsafePointer { str.c_str() } +``` + +The above function returns a dangling reference to `str`‘s inner storage. In C++, it is assumed that the programmer understands this is a bug, and generally would be expected to take `str` by reference. This is not the case in Swift. To represent this idiomatically in Swift, the lifetimes must be associated through a projection. Using the tools provided in the ownership manifesto this would mean yielding the value returned by `c_str` out of a [generalized accessor](https://github.com/apple/swift/blob/main/docs/OwnershipManifesto.md#generalized-accessors)(resulting in an error when the pointer is returned). diff --git a/visions/using-swift-from-c++.md b/visions/using-swift-from-c++.md new file mode 100644 index 0000000000..65583f2ec5 --- /dev/null +++ b/visions/using-swift-from-c++.md @@ -0,0 +1,139 @@ +# Using Swift from C++ + +This vision document presents a high level overview of the "reverse" (i.e. Swift-to-C++) half of the C++ interoperability Swift language feature. It highlights the key principles and goals that determine how Swift APIs are exposed to C++ users. It also outlines the evolution process for this feature. This document does not present the final design for how Swift APIs get mapped to C++ language constructs. The final design of this feature will evolve as this feature goes through the Swift evolution process. This document does not cover the "forward" (i.e. using C++ APIs from Swift) aspect of the C++ interoperability, as it’s covered by a [sibling document](https://github.com/apple/swift/pull/60501/files). + +This document is an official feature vision document, as described in the [draft review management guidelines](https://github.com/rjmccall/swift-evolution/blob/057b2383102f34c3d0f5b257f82bba0f5b94683d/review_management.md#future-directions-and-roadmaps) of the Swift evolution process. The Language Workgroup has endorsed the goals and basic approach laid out in this document. This endorsement is not a pre-approval of any of the concrete proposals that may come out of this document. All proposals will undergo normal evolution review, which may result in rejection or revision from how they appear in this document. + +## Introduction + +Swift currently provides support for interoperability with C and Objective-C. Swift can import C and Objective-C APIs, and it can also expose its `@objc` types and functions to Objective-C. However, Swift currently does not provide support for interoperability with C++. Supporting bidirectional C++ interoperability in Swift would allow it to call into C++ APIs, and would make Swift APIs accessible to C++. Making Swift APIs accessible to C++ would simplify the process of adoption of Swift in existing C++ codebases, as it would allow them to incrementally adopt Swift. Furthermore, it would make Swift-only libraries available to C++ codebases. + +## Overview of Swift's interoperability support + +Swift uses two very different approaches for interoperability with C and Objective-C. For "forward" interoperability, Swift embeds a copy of the Clang compiler, which it uses to directly load C and Objective-C Clang modules and translate declarations into a native Swift representation. For "reverse" interoperability, Swift does not want to require Clang to embed Swift, and so Swift generates an Objective-C header that can be used to call into Swift APIs that are exposed in that header. Because Objective-C is restricted in what kinds of types it can work with, and because it requires code to be emitted in a special way that doesn't match the regular Swift ABI, methods and types must be marked as being exposed to Objective-C with the `@objc` attribute. + +Swift's support for C++ interoperability is modeled after its support for C and Objective-C interoperability. For "forward" interoperability, the embedded Clang compiler is used to directly load C++ Clang modules and translate declarations into a native Swift representation. The "forward" interoperability [vision document](https://github.com/apple/swift/pull/60501/files) provides more details about this model. For "reverse" interoperability, Swift generates a header that uses C++ language constructs to represent Swift APIs that are exposed by the Swift module. Because of the relatively high expressivity of C++ headers, the generated header is able to provide representation for most native Swift functions, methods, initializers, accessors and types without needing any extra code to be generated in the Swift module. This allows C++ programmers to call into Swift APIs using the familiar C++ function and member function call syntax. + +## C++ interoperability does not require Swift opt in + +The user model employed by Swift's C++ interoperability is different than the user model employed by Swift's Objective-C interoperability. All else being equal, it's best if supporting interoperation with a language doesn't require changing source code or emitting extra code eagerly from the compiler. This doesn't work for Objective-C interoperability because Objective-C requires specially-compatible classes and methods, the code for which cannot be emitted from an Objective-C header. Swift chose to require programs to opt in to Objective-C interoperability with the `@objc` attribute, both to make export more predictable and to avoid emitting extra code and metadata for all classes. In contrast, as long as the C++ compiler supports the Swift calling convention, a C++ header can call native Swift functions directly, and the C++ type system can be used to wrap most Swift types in a safe C++ representation. Because of this, there is no reason to require Swift module authors to opt in into C++ interoperability. Instead, any Swift module that can be imported into Swift can also be imported into C++, and most APIs will come across automatically. + +Some API authors will desire explicit control over the C++ API. Swift will provide an annotation such as the proposed [`@expose` attribute](https://forums.swift.org/t/formalizing-cdecl/40677/50) to allow precise control over which APIs get exposed to C++. The API authors will be able to specify that their module should only expose the annotated APIs to C++ when they desire to do so. + +## Goals + +The ultimate goal of allowing Swift APIs to be used from C++ is to remove a barrier to writing more code in Swift rather than C++. Some C++ programmers would like to adopt a memory-safe language like Swift in their codebase. These programmers would typically like to adopt Swift gradually instead of relying on rewriting all of their code in Swift at once. For instance, they might write a new component in Swift, which then needs to be made usable from the rest of their C++ codebase. Today this could be done via an Objective-C wrapper that relies on `@objc` annotations. However, maintaining `@objc` wrapper code is annoying and error-prone, and it often adds significant performance costs and expressivity restrictions because of the limitations of C and Objective-C. Allowing C++ to directly use Swift APIs with minimal restrictions or performance overhead would lower the burden of adding Swift to an existing C++ codebase. It also makes Swift an even better language for stable system libraries and APIs, as it allows C++ projects to consume most such libraries without restrictions. These libraries would also have the option of not providing an Objective-C interface for Objective-C clients, as these clients could use the Swift APIs from Objective-C++ instead. + +The primary goal of Swift-to-C++ interoperability is to expose every language feature of Swift for which a reasonable C++ API can be created to represent that feature. Swift and C++ are both feature rich programming languages. They share many similar features, but Swift has some features that lack close analogues in C++. For example, C++ does not have a concept of argument labels, but the argument labels can be integrated into the base name of a function to get a roughly similar effect. On the other hand, C++ does not have any feature that can represent a Swift feature like Swift macros in a reasonable manner. Therefore, macros must be dropped in the C++ interface to the module. + +It is a goal that Swift-to-C++ interoperability can be used by both mixed Swift/C++ language projects that want to interoperate within themselves, and by C++ projects that need to use a Swift library that wasn't developed with C++ interoperability in mind. + +It is a goal of Swift-to-C++ interoperability to expose Swift APIs in a safe, performant and ergonomic manner. The Swift compiler and the language specification should follow several key related principles that are presented below. + +### Safety + +Safety is a top priority for the Swift programming language. Swift code expects its callers to adhere to Swift’s type rules and Swift’s memory model, regardless of whether it’s called from Swift or C++. Thus, the C++ code that calls Swift should properly enforce Swift’s expected language contracts. The enforcement should be done automatically in the generated header. This kind of enforcement does not prevent all possible issues though, as it does not change C++'s safety model. C++ is unsafe by default, and the user is able to use regular C++ pointers to write to memory as if they were using Swift's `UnsafeMutablePointer` type. This means that bugs in C++ code can easily lead to violations of Swift's invariants. For instance, a bug in user’s C++ code could accidentally overwrite a Swift object stored on the heap, which could cause unexpected behavior (such as a segfault) in Swift code. The user is expected to obey Swift's memory model and type rules when calling into Swift from C++, and thus they bear the ultimate responsibility for avoiding bugs like this one. The user can use certain program analysis tools, such as Address Sanitizer, to help them catch bugs that violate Swift's memory model rules. + +Swift expects that values of correct types are passed to Swift APIs. For instance, for calls to generic functions, Swift expects the caller to pass a value of type that conforms to all of the required generic requirements. Type safety should be enforced from C++ as well. The C++ compiler should verify that correct types are passed to the Swift APIs as they’re invoked from C++. A program that tries to pass an incorrect Swift type into a Swift function should not compile. Select uses of specific Swift types might be incompatible with static type validation. For example, verification of generic requirements for a Swift opaque type value returned from a Swift function in C++ requires run-time validation. The program should report a fatal error when such run-time type validation fails, to ensure that the type invariants that Swift expects are not violated. The reported fatal error should clearly indicate why type validation failed and point to the location in the user's code which caused the run-time check. + +Memory safety is paramount for Swift code. Swift automatically manages the lifetime of value and reference types in Swift. These principles should translate to C++ as well. The lifetime of Swift values that are created or passed around in C++ should be managed automatically. For instance, the generated C++ code should increment the retain count of a Swift class instance when a Swift class type value is copied in C++, and it should decrement the retain count of a Swift class instance when a Swift class type value is destroyed in C++. The default convention for using Swift types in C++ should discourage dangling references and any other patterns that could lead to an invalid use of a Swift value. + +Swift’s memory safety model also requires exclusive access to a value in order to modify that value. For instance, the same value can not be passed to two `inout` parameters in the same function call. Swift enforces exclusivity using both compile-time and run-time checks. The generated run-time checks trap when an exclusivity violation is detected at runtime. Calls into Swift APIs from C++ should verify exclusivity for Swift values as well. + +### Performance + +Swift-to-C++ bridging should be as efficient as possible. The Swift compiler should avoid unnecessary overhead for calling into Swift functions from C++, or using Swift types from C++. Generally, the bridging code should not convert Swift types into their C++ counterparts automatically, as that can add a lot of overhead. For instance, a Swift function returning a `String` should still return a Swift `String` in C++, and not a `std::string`. Some specific "primitive" Swift types should be converted into their C++ counterparts automatically in order to improve their ergonomics in C++. The conversion for such primitive types should be zero-cost as their ABI should match in Swift and C++. For instance, a Swift function returning a `Float` can return a `float` in C++ as they use the same underlying primitive LLVM type to represent the floating point value. + +Some Swift features require additional overhead to be used in C++. Resilient value types are a good example of this; C++ expects types to have a statically-known layout, but Swift's resilient value types do not satisfy this, and so the generated C++ types representing those types may need to dynamically allocate memory internally. In cases like these, the C++ interface should at least strive to minimize the dynamic overhead, for example by avoiding allocations for sufficiently small types. + +### Achieving safety with performance in mind + +Certain aspects of Swift’s memory model impose certain restrictions that create tension between the goal of achieving safety and the goal of avoiding unnecessary overhead for calling into Swift from C++. Checking for exclusivity violations is a good example of this. The C++ compiler does not have a notion of exclusivity it can verify, so it is difficult to prove that a value is accessed exclusively in the C++ code that calls into Swift. This means that the C++ code that calls into Swift APIs will most likely require more run-time checks to validate exclusivity than similar Swift code that calls the same Swift APIs. + +The adherence to Swift’s type and memory safety rules should be prioritized ahead of performance when C++ calls into Swift, even if this means more run-time checks are required. For users seeking maximum performance, Swift provides additional compiler flags that avoid certain run-time checks. Those flags should be taken into account when the C++ header is generated. An example of such flag is `-enforce-exclusivity`. When `-enforce-exclusivity=none` is passed to Swift, the Swift compiler does not emit any run-time checks that check for exclusivity violations. A flag like this should also affect the generated C++ header, and the Swift compiler should not emit any run-time checks for exclusivity in the generated C++ header when this flag is used. + +Correct modeling of certain aspects of Swift type semantics in C++ requires additional overhead. This issue is most prominent when modeling Swift's `consume` operation, as it performs a destructive move, which C++ does not support. Thus, the `consume` operation has to be modeled using a non-destructive move in C++, which requires additional storage and runtime checks when a Swift value type is used in C++. In cases like this one, the design of how Swift language constructs are mapped to C++ should strive to be as ergonomic and as intuitive as possible, provided there's still a way to achieve appropriate performance using compiler optimizations or future extensions to the C++ language standard. + +### Ergonomics + +Swift APIs should be mapped over to C++ language features that have a direct correspondence to the Swift language feature. In cases where a direct correspondence does not exist, the Swift compiler should provide a reasonable approximation to the original Swift language feature using other C++ constructs. For example, Swift’s enum type can contain methods and nested types, and such constructs can’t be represented by a single C++ enum type in an idiomatic manner. Swift’s enum type can be mapped to a C++ class instead that allows both enum-like `switch` statement behavior and also enables the C++ user to invoke member functions on the Swift enum value and access its nested types from C++. + +The C++ representation of certain Swift types should be appropriately enhanced to allow them to be used in an idiomatic manner. For instance, it should be possible to use Swift’s `Array` type (or any type that conforms to `Collection`) in a ranged-based for loop in C++. Such enhancements should be done with safety in mind, to ensure that Swift’s memory model is not violated. + +There should be no differences on the C++ side between using libraries that opt-in into library evolution and libraries that don’t, except in specific required cases, like checking the unknown default case of a resilient enum. + +### Clear language mapping rules + +C++ is a very expressive language and it can provide representation for a lot of Swift language constructs. Not every Swift language construct will map to its direct C++ counterpart. For instance, Swift initializers might get bridged to static `init` member functions instead of constructors in C++, to allow C++ code to call failable initializers in a way that’s consistent with other initializer calls. Therefore, it’s important to provide documentation that describes how Swift language constructs get mapped to C++. It is a goal of C++ interoperability to provide a clear and well-defined mapping for how Swift language constructs are mapped to C++. Additionally, it is also a goal to clearly document which language constructs are not bridged to C++. In addition to documentation, compiler diagnostics should inform the user about types or functions that can’t be exposed to C++, when the user wants to expose them explicitly. It is a goal of C++ interoperability to add a set of clear diagnostics that let the user know when a certain Swift declaration is not exposed. It is not a goal to diagnose such cases when the user did not instruct the compiler to expose a declaration explicitly. For example, the Swift compiler might not diagnose when an exposed Swift type does not expose one of its public methods to C++ due to its return type not being exposed, if such method does not have an explicit annotation that instructs the compiler that this method must be exposed to C++. + +Some Swift APIs patterns will map to distinct C++ language constructs or patterns. For instance, an empty Swift enum with static members is commonly used in a namespace-like manner in Swift. This kind of enum can be mapped to a C++ namespace. It is a goal of C++ interoperability to provide a clear mapping for how Swift API patterns like this one are bridged to C++. + +The semantics of how Swift types behave should be preserved when they’re mapped to C++. For instance, in C++, there should still be a semantic difference between Swift value and reference types. Additionally, Swift’s copy-on-write data types like `Array` should still obey the copy-on-write semantics in C++. + +### Swift language evolution and API design should be unaffected + +Importing Swift APIs into C++ should not degrade the experience of programming in Swift. That means that Swift as a language should continue to evolve based on its [current goals](https://www.swift.org/about/). New additions to Swift should not restrict their expressivity or safety in order to provide a better experience for Swift APIs in C++. Such new features do not have to be exposed to C++ when it doesn't make sense to do so. + +Swift API authors should not change the way they write Swift code and design Swift APIs based on how specific Swift language constructs are exposed to C++. They should use the most effective Swift constructs for the task. It is a key goal of C++ interoperability that the exposed C++ interfaces are safe, performant, and ergonomic enough that programmers will not be tempted to make their Swift code worse just to make the C++ interfaces better. + +### Objective-C support + +The existing Swift to Objective-C bridging layer should still be supported even when C++ bindings are generated in the generated header. Furthermore, the generated C++ bindings should use appropriate Objective-C++ types or constructs in the generated C++ declarations where it makes sense to do so. For instance, a Swift function that returns an Objective-C class instance should be usable from an Objective-C++ compilation unit. + +## The approach + +The Swift compiler exposes Swift APIs to C++ by generating a header file that contains C++ declarations that wrap around the underlying calls into Swift functions that use the native Swift calling convention. The header also provides a suitable representation for Swift types. This generation can be done retroactively, starting from a Swift module interface file, and does not need to be requested when the Swift module is built from source. + +Currently the generated header file depends on several LLVM and Clang compiler features for the Swift calling convention support, and thus it can only be compiled by Clang. The header does not depend on any other Clang-specific C++ language extensions. The header can use some other optional Clang-only features that improve developer experience for the C++ users, like specific attributes that improve diagnostics. These Clang-only features are enabled only when the Clang that is being used supports them, so older versions of Clang would still be able to consume the generated header. + +The generated header file uses advanced C++ features that require a recent C++ language standard. C++20 is the recommended language standard to use for Swift APIs, however, the generated header should also be compatible with C++17 and C++14. + +The generated header file also contains the C and the Objective-C declarations that Swift exposes to C and Objective-C on platforms that support C or Objective-C interoperability. Thus a single header file can be used by codebases that mix C, Objective-C and C++. + +For the majority of Swift libraries, the generated header file is a build artifact generated by the client. This is different from Objective-C interoperability, where the generated header is always generated by the library owner and distributed with it. However, Swift library owners can also generate the header and distribute it with the library if they want the client to consume the C++ interface directly without having to run the Swift compiler on their end. + +On platforms with ABI stability, the generated C++ code for an ABI stable Swift interface is not tied to the Swift compiler that compiled the Swift code for that Swift module, as ABI stability is respected by the C++ code in the header. In all other cases the generated C++ code in the header is assumed to be tied to the Swift compiler that compiled the Swift module, and thus the header should be regenerated when the compiler changes. + + The next few sections provide a high level overview of how Swift types and some other language constructs get bridged to C++. The exact details of how Swift language constructs are bridged will be covered by Swift evolution proposals, and additional documentation, such as this preliminary [user guide document](https://github.com/apple/swift/blob/main/docs/CppInteroperability/UserGuide-CallingSwiftFromC%2B%2B.md). + +### Bridging Swift types + +The generated header contains C++ class types that represent the Swift struct, enum, and class types that are exposed by the Swift module. These types provide access to methods, properties and other members using either idiomatic C++ constructs or, in some cases, non-idiomatic C++ constructs that allow C++ to access more functionality in a consistent manner. The basic operations on these types follow the corresponding Swift semantics. For instance, a C++ value for a Swift class type is essentially a shared pointer that owns a reference to a Swift object, whereas a C++ value for a Swift struct type stores a Swift value of that type, and copying or destroying the C++ value copies or destroys the underlying Swift value. The C++ types also support C++ move semantics and translate them as appropriate to Swift's [`consume` operation](https://github.com/swiftlang/swift-evolution/blob/main/proposals/0366-move-function.md). This enables high-performance use of Swift types from C++ and allows Swift's [non-copyable types](https://forums.swift.org/t/pitch-noncopyable-or-move-only-structs-and-enums/61903) to be exposed to C++. + +Protocol types also get exposed to C++. They provide access to their protocol interface to C++. The generated header also provides facilities to combine protocol types into a protocol composition type. The protocol composition type provides access to the combined protocol interface in C++. + +### Bridging generics + +Swift generic functions and types get bridged to C++ as C++ function and class templates. A generated C++ template instantiates a type-checked generic interface that uses the underlying polymorphic semantics that generics require when Swift APIs are called from the generated header. Type-checking is performed using the `requires` clause introduced in C++20. When C++17 and earlier is used, type-checking is performed using other legacy methods, like `enable_if` and `static_assert`. The two type-checking methods are compatible with the delayed template parsing compiler feature that Clang uses when building for Windows. + +To help achieve the performance goals outlined in the prior section, the generated class templates specialize the storage for the underlying Swift generic type when the Swift API that is exposed to C++ contains such a bounded Swift generic type. This ensures that non-resilient bounded generic values can be stored inline in a C++ type that represents the underlying Swift type, instead of being boxed on the heap. + +### Standard library support + +The Swift standard library contains a lot of useful functions and types that get bridged to C++. The generated standard library bindings enhance various Swift types like `Optional` and `Array` to provide idiomatic C++ APIs that allow the user to use such types in an idiomatic manner from C++. + +### Using Swift types in C++ templates + +The generated header is useful in mixed-language projects as C++ sources can include it, allowing C++ to call Swift APIs. However the C++ interoperability project also provides support for calling C++ APIs from Swift. In certain cases such C++ APIs contain function or class templates that need to be instantiated in Swift. Swift gives the user the ability to use Swift types in such cases, so the C++ templates have to be instantiated with Swift types. This means that the Swift types need to be translated into their C++ counterparts, which could then be used inside the instantiated C++ template specialization. This Swift-to-C++ type translation is performed using the same mechanism that’s used by the Swift compiler to generate the header with C++ interface for a Swift module. This means that a C++ template instantiated with a Swift type will see the same C++ representation of that Swift type regardless of whether it was instantiated from C++, or from Swift. + +## Evolution process + +The approach and the goals for how Swift APIs get bridged to C++ are outlined above. Each distinct Swift language construct that’s bridged to C++ will need to be covered by a detailed evolution proposal. These evolution proposals can refer to this vision document as a general context document for how Swift APIs should be bridged to C++. Every Swift API pattern that has a distinct mapping in C++ will also need a detailed and self-contained evolution proposal as well. The design for how each district language feature or API pattern is bridged to C++ is ratified only once its respective proposal goes through the Swift evolution process and is accepted by the Swift community. + +## The Swift ecosystem + +As a supported language feature, C++ and Swift interoperability must work well on every platform supported by Swift. In similar vein, tools in the Swift ecosystem should be updated to support C++ interoperability. The next few sections of this document provide specific recommendations for how various tools in the Swift ecosystem should be adapted to support C++ interoperability. The C++ interoperability workgroup intends to work on supporting most of these recommendations in the tools that are bundled with the Swift toolchain while working on the initial version of interoperability that could be adopted for production use cases. + +### Build tool support + +The Swift package manager (SwiftPM) is one of the most commonly used ways to build Swift code. Swift package manager can also build C and C++ code. SwiftPM should provide good support for bridging Swift APIs to C++ out of the box. It should generate a compatibility header for a Swift package when it determines that the C++ code in the same or dependent package includes such a header, and it should ensure that this header can be found when the C++ code is compiled. SwiftPM already has some support for generating a header file with Objective-C interface for a Swift module, and the C++ interoperability workgroup intends to reuse that support for supporting the generation of C++ bindings as well. + +CMake is a widely used tool used for configuring and building C++ code. CMake should provide good support for adding Swift code to C++ CMake targets. Swift’s ecosystem as a whole should ensure that it should be as straightforward as possible to add support for bridging Swift APIs to C++ within the same CMake project. The C++ interoperability workgroup should provide an example CMake project that shows how Swift and C++ can interoperate between each other, once the appropriate support for mixed-language Swift and C++ targets lands in CMake. + +### Debugging support + +Debugging support is critical for great user experience. LLDB should understand that C++ types in the generated header are just wrappers around Swift values. It should be able to display the underlying Swift value in the debugger when the user tries to inspect the C++ value that stores the Swift value in the debugger. In addition to that, the generated compatibility header should be correctly annotated to ensure that C++ inline thunks that call Swift APIs can be skipped when a user steps in or steps out into or from a call when debugging a program. + +### IDE support + +SourceKit-LSP is a language server that provides cross-platform IDE support for Swift code in the Swift ecosystem. It can also act as a language server for C, Objective-C and C++ as it can wrap around and redirect queries to clangd. This in turn allows SourceKit-LSP to provide support for mixed-language IDE queries, like jump-to-definition, that allows the IDE client to jump from an Objective-C method to call to its underlying Swift implementation. The C++ interoperability workgroup intends to reuse the current support for mixed-language Swift and Objective-C queries to add similar functionality for mixed-language Swift and C++ queries. This would allow IDE clients that use SourceKit-LSP to use features that can operate across the Swift/C++ language boundary. For instance, a client IDE would be able to support jump-to-definition from a Swift call expression to the called C++ function using SourceKit-LSP. This mixed-language query support in SourceKit-LSP is powered by the indexing data emitted by Clang. The C++ interoperability workgroup intends to extend Clang's indexing support to represent references to the wrapper C++ declarations from the generated header as references to the underlying Swift declarations. diff --git a/visions/webassembly.md b/visions/webassembly.md new file mode 100644 index 0000000000..95aa29f884 --- /dev/null +++ b/visions/webassembly.md @@ -0,0 +1,196 @@ +# A Vision for WebAssembly Support in Swift + +## Introduction + +WebAssembly (abbreviated [Wasm](https://webassembly.github.io/spec/core/intro/introduction.html#wasm)) is a virtual +machine instruction set focused on portability, security, and high performance. It is vendor-neutral, designed and +developed by [W3C](https://w3.org). An implementation of a WebAssembly virtual machine is usually called a +*WebAssembly runtime*. + +One prominent spec-compliant implementation of a Wasm runtime in Swift is [WasmKit](https://github.com/swiftwasm/WasmKit). +It is available as a Swift package, supports multiple host platforms, and has a simple API for interaction with guest +Wasm modules. + +An application compiled to a Wasm module can run on any platform that has a Wasm runtime available. Despite its origins +in the browser, it is a general-purpose technology that has use cases in client-side and server-side applications and +services. WebAssembly support in Swift makes the language more appealing in those settings, and also brings it to the +browser where it previously wasn't available at all[^1]. It facilitates a broader adoption of Swift in more environments +and contexts. + +The WebAssembly instruction set has useful properties from a security perspective, as it has no interrupts or +peripheral access instructions. Access to the underlying system is always done by calling explicitly imported +functions, implementations for which are provided by an imported WebAssembly module or a WebAssembly runtime itself. +The runtime has full control over interactions of the virtual machine with the outside world. + +WebAssembly code and data live in completely separate address spaces, with all executable code in a given module loaded +and validated by the runtime upfront. Combined with the lack of "jump to address" and a limited set of control flow +instructions that require explicit labels in the same function body, this makes a certain class of attacks impossible to +execute in a correctly implemented spec-compliant WebAssembly runtime. + +### WebAssembly System Interface and the Component Model + +The WebAssembly virtual machine has no in-built support for I/O; instead, a Wasm module's access to I/O is dependent +entirely upon the runtime that executes it. + +A standardized set of APIs implemented by a Wasm runtime for interaction with the host operating system is called +[WebAssembly System Interface (WASI)](https://wasi.dev). [WASI libc](https://github.com/WebAssembly/wasi-libc) is a +layer on top of WASI that Swift apps compiled to Wasm can already use thanks to C interop. The current implementation +of Swift stdlib and runtime for `wasm32-unknown-wasi` triple is based on this C library. It is important for WASI +support in Swift to be as complete as possible to ensure portability of Swift code in the broader Wasm ecosystem. + +In the last few years, the W3C WebAssembly Working Group considered multiple proposals for improving the WebAssembly +[type system](https://github.com/webassembly/interface-types) and +[module linking](https://github.com/webassembly/module-linking). These were later subsumed into a combined +[Component Model](https://component-model.bytecodealliance.org) proposal thanks to the ongoing work on +[WASI Preview 2](https://github.com/WebAssembly/WASI/blob/main/wasip2/README.md), which served as playground for +the new design. + +The Component Model defines these core concepts: + +- A *component* is a composable container for one or more WebAssembly modules that have a predefined interface; +- *WebAssembly Interface Types (WIT) language* allows defining contracts between components; +- *Canonical ABI* is an ABI for types defined by WIT and used by component interfaces in the Component Model. + +Preliminary support for WIT has been implemented in +[the `wit-tool` subcommand](https://github.com/swiftwasm/WasmKit/blob/0.0.3/Sources/WITTool/WITTool.swift) of the +WasmKit CLI. Users of this tool can generate `.wit` files from Swift declarations, and vice versa: Swift bindings from +`.wit` files. + +## Use Cases + +We can't anticipate every possible application Swift developers are going to create with Wasm, but we can provide a few +examples of its possible adoption in the Swift toolchain itself. To quote +[a GSoC 2024 idea](https://www.swift.org/gsoc2024/#building-swift-macros-with-webassembly): + +> WebAssembly could provide a way to build Swift macros into binaries that can be distributed and run anywhere, +> eliminating the need to rebuild them continually. + +This can be applicable not only to Swift macros, but also for the evaluation of SwiftPM manifests and plugins. + +In the context of Swift developer tools, arbitrary code execution during build time can be virtualized with Wasm. +While Swift macros, SwiftPM manifests, and plugins are sandboxed on Darwin platforms, with Wasm we can provide stronger +security guarantees on other platforms that have a compatible Wasm runtime available. + +The WebAssembly instruction set is designed with performance in mind. A WebAssembly module can be JIT-compiled or +compiled on a client machine to an optimized native binary ahead of time. With recently accepted proposals to the Wasm +specification it now supports features such as SIMD, atomics, multi-threading, and more. A WebAssembly runtime can +generate a restricted subset of native binary code that implements these features with little performance overhead. + +Adoption of Wasm in developer tools does not imply unavoidable performance overhead. With security guarantees that +virtualization brings, there's no longer a need to spawn a separate process for each Swift compiler and SwiftPM +plugin/manifest invocation. Virtualized Wasm binaries can run in the host process of a Wasm runtime, removing the +overhead of new process setup and IPC infrastructure. + +## Goals + +As of March 2024 all patches necessary for basic Wasm and WASI Preview 1 support have been merged to the Swift +toolchain and core libraries. Based on this, we propose a high-level roadmap for WebAssembly support and adoption in the Swift +ecosystem: + +1. Make it easier to evaluate and adopt Wasm with increased API coverage for this platform in the Swift core libraries. + Main prerequisite for that is setting up CI jobs for those libraries that run tests for WASI and also Embedded Wasm, where + possible. As a virtualized embeddable platform, not all system APIs are always available or easy to port to WASI. For example, + multi-threading, file system access, networking and localization need special support in Wasm runtimes and a certain amount of + consideration from a developer adopting these APIs. + +2. Improve support for cross-compilation in Swift and SwiftPM. We can simplify versioning, installation, and overall + management of Swift SDKs for cross-compilation in general, which is beneficial not only for WebAssembly, but for all + platforms. + +3. Continue work on Wasm Component Model support in Swift as the Component Model proposal is stabilized. Ensure + that future versions of WASI are available to Swift developers targeting Wasm. + +4. Make interoperability with Wasm components as smooth as C and C++ interop already is for Swift. With a formal + specification for Canonical ABI progressing, this will become more achievable with time. This includes consuming + components from, and building components with Swift. + +5. Improve debugging experience of Swift code compiled to Wasm. While rudimentary support for debugging + exists in some Wasm runtimes, we aim to improve it and, where possible, make it as good as debugging Swift code + compiled to other platforms. + +### Proposed Language Features + +In our work on Wasm support in Swift, we experimented with a few function attributes that could be considered +as pitches and eventually Swift Evolution proposals, if the community is interested in their wider adoption. +These attributes allow easier interoperation between Swift code and other Wasm modules linked with it by a Wasm +runtime. + +## Platform-specific Considerations + +### Debugging + +Debugging Wasm modules is challenging because Wasm does not expose ways to introspect and control the execution of +a Wasm module instance, so a debugger cannot be built on top of Wasm itself. Special support from the Wasm execution +engine is necessary for debugging. + +The current state of debugging tools in the Wasm ecosystem is not as mature as other platforms, but there are two +main directions: + +1. [LLDB debugger with Wasm runtime](https://github.com/llvm/llvm-project/pull/77949) supporting GDB Remote Serial Protocol; +2. [Wasm runtime with a built-in debugger](https://book.swiftwasm.org/getting-started/debugging.html#enhanced-dwarf-extension-for-swift). + +The first approach provides an almost equivalent experience to existing debugging workflows on other platforms. It +can utilize LLDB's Swift support, remote metadata inspection, and serialized Swift module information. However, since +Wasm is a Harvard architecture and has no way to allocate executable memory space at runtime, implementing expression +evaluation with JIT in user space is challenging. In other words, GDB stub in Wasm engines need tricky implementations +or need to extend the GDB Remote Serial Protocol. + +The second approach embeds the debugger within the Wasm engine. In scenarios where the Wasm engine is embedded as a +guest in another host engine (e.g. within a Web Browser), this approach allows seamless debugging experiences with the +host language by integrating with the host debugger. For example, in cases where JavaScript and Wasm call frames +are interleaved, the debugger works well in both contexts without switching tools. Debugging tools like Chrome DevTools +can use DWARF information embedded in Wasm file to provide debugging support. However, supporting Swift-specific +metadata information and JIT-based expression evaluation will require integrating LLDB's Swift plugin with these +debuggers in some way. + +In summary, debugging in the browser and outside of the browser context are sufficiently different activities to +require separate implementation approaches. + +### Multi-threading and Concurrency + +WebAssembly has [atomic operations in the instruction set](https://github.com/WebAssembly/threads) (only sequential +consistency is supported), but it does not have a built-in way to create threads. Instead, it relies on the host +environment to provide multi-threading support. This means that multi-threading in Wasm is dependent on the Wasm runtime +that executes a module. There are two proposals to standardize ways to create threads in Wasm: + +(1) [wasi-threads](https://github.com/WebAssembly/wasi-threads), which is already supported by some toolchains, +runtimes, and libraries but has been superseded; + +(2) The new [shared-everything-threads](https://github.com/WebAssembly/shared-everything-threads) proposal is still +in the early stages, but is expected to be the future of multi-threading in Wasm. + +Swift currently supports two threading models in Wasm: single-threaded (`wasm32-unknown-wasi`) and multi-threaded +using wasi-threads (`wasm32-unknown-wasip1-threads`). Despite the latter supporting multi-threading, Swift Concurrency +defaults to a cooperative single-threaded executor due to the lack of wasi-threads support in libdispatch. Preparing +for the shared-everything-threads proposal is crucial to ensure that Swift Concurrency can adapt to future +multi-threading standards in Wasm. + +### 64-bit address space + +WebAssembly currently uses a 32-bit address space, but [64-bit address space](https://github.com/WebAssembly/memory64/) +proposal is already in the implementation phase. + +Swift supports 64-bit pointers on other platforms where available, however WebAssembly is the first platform where +relative reference from data to code is not allowed. Alternative solutions like image-base relative addressing or +"small code model" for fitting 64-bit pointer in 32-bit are unavailable, at least for now. This means that we need +cooperation from the WebAssembly toolchain side or different memory layout in Swift metadata to support 64-bit linear +memory support in WebAssembly. + +### Shared libraries + +There are two approaches to using shared libraries in the WebAssembly ecosystem: + +1. [Emscripten-style dynamic linking](https://emscripten.org/docs/compiling/Dynamic-Linking.html) +2. [Component Model-based "ahead-of-time" linking](https://github.com/WebAssembly/component-model/blob/main/design/mvp/Linking.md) + +Emscripten-style dynamic linking is a traditional way to use shared libraries in WebAssembly, where the host +environment provides non-standard dynamic loading capabilities. + +The latter approach cannot fully replace the former, as it is unable to handle dynamic loading of shared libraries at +runtime, but it is more portable way to distribute programs linked with shared libraries, as it does not require the +host environment to provide any special capabilities except for Component Model support. + +Support for shared libraries in Swift means ensuring that Swift programs can be compiled in +position-independent code mode and linked with shared libraries by following the corresponding dynamic linking ABI. + +[^1]: We aim to address browser-specific use cases in a separate future document.