.cargo/root, CARGO_ROOT, --root limit config + workspace searches

If a root directory is specified then cargo will only search ancestors
looking for `.cargo` config files and workspaces up until the root
directory is reached (instead of walking to the root of the filesystem).

A root can be specified in three ways:

1. The existence of a `.cargo/root` file (discovered by checking parents
   up towards the root of the filesystem)
2. By setting the `CARGO_ROOT` environment variable
3. By passing `--root` on the command line

If more than one is specified then the effective root is the one that's
most-specific / closest to the current working directory.

### What does this PR try to resolve?

Fixes: #5418
(for my use case then #6805 isn't a practical workaround)

This goes some way to allow nesting of workspaces, by adding a way to
limit the directories that Cargo searches when looking for config files
and workspace manifests.

This does not enable nesting in the form of workspace inheritance - but
does provide the means to technically have a filesystem with nested
workspaces (that aren't aware of each other) and be able to hide any
outer (unrelated) workspace while building a nested workspace.

This gives more flexibility for tools that want to use cargo as an
implementation detail. In particular this allows you to sandbox the
build of nested third-party workspaces that may be (unknowingly)
dynamically unpacked within an outer workspace, in situations where
neither the workspace being built and the outer workspace are owned by
the tool that is managing the build.

For example a tool based on rustdoc-json should be able to fetch and
build documentation for third-party crates under any user-specified
build/target directory without having to worry about spooky action at a
distance due to config files and workspaces in ancestor directories.

In my case, I have a runtime for coding with LLMs that is given a repo
to work on and is expected to keep its artifacts contained to a `.ai/`
directory. This runtime supports building markdown documentation for
Rust crates, which involves using cargo to generate rustdoc-json data.
That tool is expected to keep its artifacts contained within
`.ai/docs/rust/build/`. It's possible that the project itself is Rust
based and could define a workspace or `.cargo/config.toml` but from the
pov of this toolchain those have nothing to do with the crate whose
documentation is being generated (which are packages downloaded from
crates.io).

### How to test and review this PR?

TODO: write tests

src/bin/cargo/cli.rs

@@ -2,6 +2,7 @@ use anyhow::{anyhow, Context as _};
 use cargo::core::{features, CliUnstable};
 use cargo::util::context::TermConfig;
 use cargo::{drop_print, drop_println, CargoResult};
+use cargo_util::paths;
 use clap::builder::UnknownArgumentValueParser;
 use itertools::Itertools;
 use std::collections::HashMap;
@@ -17,6 +18,34 @@ use crate::util::is_rustup;
 use cargo::core::shell::ColorChoice;
 use cargo::util::style;
 
+fn closest_valid_root<'a>(
+    cwd: &std::path::Path,
+    config_root: Option<&'a std::path::Path>,
+    env_root: Option<&'a std::path::Path>,
+    cli_root: Option<&'a std::path::Path>,
+) -> anyhow::Result<Option<&'a std::path::Path>> {
+    for (name, root) in [
+        (".cargo/root", config_root),
+        ("CARGO_ROOT", env_root),
+        ("--root", cli_root),
+    ] {
+        if let Some(root) = root {
+            if !cwd.starts_with(root) {
+                return Err(anyhow::format_err!(
+                    "the {} `{}` is not a parent of the current working directory `{}`",
+                    name,
+                    root.display(),
+                    cwd.display()
+                ));
+            }
+        }
+    }
+    Ok([config_root, env_root, cli_root]
+        .into_iter()
+        .flatten()
+        .max_by_key(|root| root.components().count()))
+}
+
 #[tracing::instrument(skip_all)]
 pub fn main(gctx: &mut GlobalContext) -> CliResult {
     // CAUTION: Be careful with using `config` until it is configured below.
@@ -25,6 +54,7 @@ pub fn main(gctx: &mut GlobalContext) -> CliResult {
 
     let args = cli(gctx).try_get_matches()?;
 
+    let mut need_reload = false;
     // Update the process-level notion of cwd
     if let Some(new_cwd) = args.get_one::<std::path::PathBuf>("directory") {
         // This is a temporary hack.
@@ -46,6 +76,45 @@ pub fn main(gctx: &mut GlobalContext) -> CliResult {
             .into());
         }
         std::env::set_current_dir(&new_cwd).context("could not change to requested directory")?;
+        need_reload = true;
+    }
+
+    // A root directory can be specified via CARGO_ROOT, --root or the existence of a `.cargo/root` file.
+    // If more than one is specified, the effective root is the one closest to the current working directory.
+
+    let cwd = std::env::current_dir().context("could not get current working directory")?;
+    // Windows UNC paths are OK here
+    let cwd = cwd
+        .canonicalize()
+        .context("could not canonicalize current working directory")?;
+    let config_root = paths::ancestors(&cwd, gctx.search_stop_path())
+        .find(|current| current.join(".cargo").join("root").exists());
+    let env_root = gctx
+        .get_env_os("CARGO_ROOT")
+        .map(std::path::PathBuf::from)
+        .map(|p| {
+            p.canonicalize()
+                .context("could not canonicalize CARGO_ROOT")
+        })
+        .transpose()?;
+    let env_root = env_root.as_deref();
+
+    let cli_root = args
+        .get_one::<std::path::PathBuf>("root")
+        .map(|p| {
+            p.canonicalize()
+                .context("could not canonicalize requested root directory")
+        })
+        .transpose()?;
+    let cli_root = cli_root.as_deref();
+
+    if let Some(root) = closest_valid_root(&cwd, config_root, env_root, cli_root)? {
+        tracing::debug!("root directory: {}", root.display());
+        gctx.set_search_stop_path(root);
+        need_reload = true;
+    }
+
+    if need_reload {
         gctx.reload_cwd()?;
     }
 
@@ -640,6 +709,14 @@ See '<cyan,bold>cargo help</> <cyan><<command>></>' for more information on a sp
                 .value_parser(["auto", "always", "never"])
                 .ignore_case(true),
         )
+        .arg(
+            Arg::new("root")
+                .help("Define a root that limits searching for workspaces and .cargo/ directories")
+                .long("root")
+                .value_name("ROOT")
+                .value_hint(clap::ValueHint::DirPath)
+                .value_parser(clap::builder::ValueParser::path_buf()),
+        )
         .arg(
             Arg::new("directory")
                 .help("Change to DIRECTORY before doing anything (nightly-only)")

src/cargo/core/workspace.rs

@@ -2028,7 +2028,7 @@ fn find_workspace_root_with_loader(
         let roots = gctx.ws_roots.borrow();
         // Iterate through the manifests parent directories until we find a workspace
         // root. Note we skip the first item since that is just the path itself
-        for current in manifest_path.ancestors().skip(1) {
+        for current in paths::ancestors(manifest_path, gctx.search_stop_path()).skip(1) {
             if let Some(ws_config) = roots.get(current) {
                 if !ws_config.is_excluded(manifest_path) {
                     // Add `Cargo.toml` since ws_root is the root and not the file
@@ -2061,7 +2061,7 @@ fn find_root_iter<'a>(
     manifest_path: &'a Path,
     gctx: &'a GlobalContext,
 ) -> impl Iterator<Item = PathBuf> + 'a {
-    LookBehind::new(paths::ancestors(manifest_path, None).skip(2))
+    LookBehind::new(paths::ancestors(manifest_path, gctx.search_stop_path()).skip(2))
         .take_while(|path| !path.curr.ends_with("target/package"))
         // Don't walk across `CARGO_HOME` when we're looking for the
         // workspace root. Sometimes a package will be organized with

src/cargo/util/context/mod.rs

@@ -567,6 +567,11 @@ impl GlobalContext {
         }
     }
 
+    /// Gets the path where ancestor config file and workspace searching will stop.
+    pub fn search_stop_path(&self) -> Option<&Path> {
+        self.search_stop_path.as_deref()
+    }
+
     /// Sets the path where ancestor config file searching will stop. The
     /// given path is included, but its ancestors are not.
     pub fn set_search_stop_path<P: Into<PathBuf>>(&mut self, path: P) {

src/doc/src/reference/config.md

@@ -44,6 +44,19 @@ those configuration files if it is invoked from the workspace root
 > and is the preferred form. If both files exist, Cargo will use the file
 > without the extension.
 
+The root of the search hierarchy can be constrained in three ways:
+
+1. By creating a `.cargo/root` file (empty)
+2. By setting the `CARGO_ROOT` environment variable
+3. Passing `--root`.
+
+If a root directory is given then Cargo will search parent directories up until
+it reaches the root directory, instead of searching all the way up to the root
+of the filesystem. Cargo will still check `$CARGO_HOME/config.toml` even if it
+is outside of the root directory. If multiple paths are specified then the
+effective root is the one that's most-specific (closest to the current working
+directory).
+
 ## Configuration format
 
 Configuration files are written in the [TOML format][toml] (like the

src/doc/src/reference/environment-variables.md

@@ -20,6 +20,10 @@ system:
   location of this directory. Once a crate is cached it is not removed by the
   clean command.
   For more details refer to the [guide](../guide/cargo-home.md).
+* `CARGO_ROOT` --- Instead of letting Cargo search every ancestor directory, up
+  to the root of the filesystem, looking for `.cargo` config directories or
+  workspace manifests, this limits how far Cargo can search. It doesn't stop
+  Cargo from reading `$CARGO_HOME/config.toml`, even if it's outside the root.
 * `CARGO_TARGET_DIR` --- Location of where to place all generated artifacts,
   relative to the current working directory. See [`build.target-dir`] to set
   via config.