DOC-522: Fix non-terminating traffic policy in robots.txt example

[Copilot]

The robots.txt traffic policy example included a forward-internal rule which is non-terminating, causing policy evaluation to continue after forwarding non-robots.txt requests — incorrect behavior.

Changes

• Removed forward-internal rule from the "deny bots/crawlers via robots.txt" example. Non-robots.txt requests naturally pass through to the upstream; no explicit forwarding rule is needed.
• Standardized expressions to use req.url.path.contains('/robots.txt') consistently across all robots.txt examples for consistency.
• Result: All robots.txt examples now use terminating custom-response actions with consistent expression syntax.

Before:

on_http_request:
  - expressions:
    - "!req.url.path.contains('/robots.txt')"
    actions:
    - type: forward-internal        # non-terminating — policy continued after this
      config:
        url: <Internal endpoint URL Here>
  - expressions:
    - "req.url.path.contains('/robots.txt')"
    actions:
    - type: custom-response
      config:
        body: "User-agent: *\r\nDisallow: /"
        headers:
          content-type: text/plain
        status_code: 200

After:

on_http_request:
  - expressions:
      - "req.url.path.contains('/robots.txt')"
    actions:
      - type: custom-response       # terminating
        config:
          body: "User-agent: *\r\nDisallow: /"
          headers:
            content-type: text/plain
          status_code: 200

Updated in traffic-policy/examples/block-unwanted-requests.mdx, snippets/traffic-policy/gallery/AddRobotsTxt.mdx, and snippets/traffic-policy/gallery/AddRobotsTxtSpecific.mdx.

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[github-actions]

Style linting summary

Vale found no style issues in this PR. Great job!

[sg-writer]

@copilot update your pr description to reflect the changes I just pushed

[Copilot]

@sg-writer The model claude-sonnet-4.6 is not available for your account. This can happen if the model was disabled by your organization's policy or if your Copilot plan doesn't include access to it.

You can try again without specifying a model (just @copilot) to use the default, or choose a different model from the model picker.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: 615eb21d-d491-4224-a11e-de72b3cbe8b0

[sg-writer]

@copilot see above

[Copilot]

@copilot see above

Updated PR description:

---

The robots.txt traffic policy example included a forward-internal rule which is non-terminating, causing policy evaluation to continue after forwarding non-robots.txt requests — incorrect behavior.

Changes

1. Removed forward-internal rule from the "deny bots/crawlers via robots.txt" example. Non-robots.txt requests naturally pass through to the upstream; no explicit forwarding rule is needed.
2. Standardized expressions to use req.url.path.contains('/robots.txt') consistently across all robots.txt examples (commit 9b9e4b8).

Result: All robots.txt examples now use terminating custom-response actions with consistent expression syntax.

Files Modified

• traffic-policy/examples/block-unwanted-requests.mdx
• snippets/traffic-policy/gallery/AddRobotsTxt.mdx
• snippets/traffic-policy/gallery/AddRobotsTxtSpecific.mdx

Before:

on_http_request:
  - expressions:
    - "!req.url.path.contains('/robots.txt')"
    actions:
    - type: forward-internal        # non-terminating
      config:
        url: <Internal endpoint URL Here>
  - expressions:
    - "req.url.path.contains('/robots.txt')"
    actions:
    - type: custom-response
      config:
        body: "User-agent: *\r\nDisallow: /"
        headers:
          content-type: text/plain
        status_code: 200

After:

on_http_request:
  - expressions:
      - "req.url.path.contains('/robots.txt')"
    actions:
      - type: custom-response       # terminating
        config:
          body: "User-agent: *\r\nDisallow: /"
          headers:
            content-type: text/plain
          status_code: 200

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/graphql
  • Triggering command: /usr/bin/gh gh pr view --json title,body (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[Copilot]

Pull request overview

Updates Traffic Policy documentation examples to avoid a non-terminating forward-internal action in the robots.txt pattern, ensuring only robots.txt requests receive the custom response while all other traffic proceeds normally.

Changes:

• Removed the forward-internal rule from the “deny bots/crawlers via robots.txt” example (YAML + JSON).
• Standardized the robots matcher to req.url.path.contains('/robots.txt') in the “specific bots/crawlers” examples.
• Kept the examples focused on terminating custom-response behavior for robots.txt.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
traffic-policy/examples/block-unwanted-requests.mdx	Simplifies robots.txt example to only intercept /robots.txt; updates matcher in the “specific bots” variant.
snippets/traffic-policy/gallery/AddRobotsTxt.mdx	Removes non-terminating forward rule from the standalone robots.txt snippet (YAML + JSON).
snippets/traffic-policy/gallery/AddRobotsTxtSpecific.mdx	Updates matcher to use req.url.path.contains('/robots.txt') in YAML + JSON snippet.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[S3Prototype]

Need to double check with the team here, maybe ask Jen or Michael latouf. According to docs forward-internal is a terminating action

• https://ngrok.com/docs/traffic-policy/concepts/actions#non-terminating-actions
• https://ngrok.com/docs/traffic-policy/actions/forward-internal#terminating-action

[sg-writer]

Pretty sure this was actually already resolved whoops! 😬