Add API Profiles feature design documentation for DMS

[Copilot]

Comprehensive architectural design for adapting Ed-Fi API Profiles from AdminAPI-2.x to DMS. Profiles enable fine-grained data access control via XML-defined inclusion/exclusion rules on resource properties, collections, and references.

Design Documentation

• API-PROFILES-DESIGN.md (40KB) - Complete architecture specification

  • Database schema: 4 normalized tables for profile metadata and rules
  • Pipeline integration: Profile resolution → validation → filtering middleware
  • Caching strategy: Distributed cache with >95% target hit rate, <10ms overhead
  • Performance targets: <5ms resolution (cached), 5-15ms validation, 10-20ms filtering
  • 5 implementation tickets with story points (total ~8-12 weeks)
  • REST API specs for profile CRUD operations
  • Mermaid diagrams: integration flow, resolution algorithm, database schema, enforcement pipeline

• API-PROFILES-QUICKSTART.md (12KB) - Developer onboarding guide

  • 5-minute setup with import/usage examples
  • Profile syntax: memberSelection strategies (IncludeOnly, ExcludeOnly, IncludeAll, ExcludeAll)
  • HTTP headers: Accept: application/json;profile=name (reads), Content-Type: application/json;profile=name (writes)
  • Common patterns and troubleshooting

• API-PROFILES-MIGRATION.md (18KB) - AdminAPI-2.x migration guide

  • 100% backward compatible XML format
  • 5-phase migration plan with validation procedures
  • Compatibility matrix showing feature parity
  • Rollback procedures

Example Profiles

Five production-ready XML profiles with inline documentation:

<!-- student-read-only.xml -->
<Profile name="Student-ReadOnly">
  <Resource name="Student">
    <ReadContentType memberSelection="IncludeOnly">
      <Property name="StudentUniqueId" />
      <Property name="FirstName" />
      <Property name="LastSurname" />
      <Reference name="SchoolReference">
        <Property name="SchoolId" />
      </Reference>
    </ReadContentType>
  </Resource>
</Profile>

Additional examples: write-limited student access, restricted assessment data, public school directory, full descriptor access.

profile-schema.xsd included for XML validation.

Database Schema

-- Profile metadata
CREATE TABLE dms.Profile (
  ProfileId BIGINT PRIMARY KEY,
  ProfileName VARCHAR(255) UNIQUE NOT NULL,
  ResourceName VARCHAR(255) NOT NULL,
  IsActive BOOLEAN DEFAULT TRUE
);

-- Property/collection/reference rules in separate tables
CREATE TABLE dms.ProfilePropertyRule (
  ProfilePropertyRuleId BIGINT PRIMARY KEY,
  ProfileId BIGINT REFERENCES dms.Profile(ProfileId),
  PropertyPath VARCHAR(1024) NOT NULL,
  RuleType VARCHAR(50) NOT NULL  -- Include/Exclude
);

Implementation Tickets

1. Profile Table & XML Import (8-10 pts, 2 weeks) - Schema, parser, import API
2. Enforcement Pipeline (13-15 pts, 3 weeks) - Middleware, cache, rule evaluation
3. Export & Management (8-10 pts, 2 weeks) - XML serialization, CRUD UI
4. Selection Mechanism (5-8 pts, 1-2 weeks) - Header parsing, defaults
5. Documentation & Examples (5-8 pts, 1-2 weeks) - Tutorials, testing frameworks

Architecture

Profile resolution flow:

Request → Extract header → Cache lookup → Load rules (if miss) → 
  Write: Validate body → Reject violations | Accept
  Read: Fetch data → Filter response → Return

Storage: XML profiles imported to database, cached in-memory
Selection: HTTP media type parameters (application/json;profile=name)
Enforcement: Pipeline middleware intercepts requests/responses
Management: REST API + Admin UI for CRUD operations

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• 4vyvsblobprodcus361.blob.core.windows.net
  • Triggering command: /usr/bin/dotnet dotnet restore EdFi.DataManagementService.sln (dns block)
• kh4vsblobprodcus325.blob.core.windows.net
  • Triggering command: /usr/bin/dotnet dotnet restore EdFi.DataManagementService.sln (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Design: Adaptation of Ed-Fi API Profiles Feature (XML) into Data Management Service (DMS)</issue_title>
<issue_description># Architecture Design: API Profiles Feature Adaptation from AdminAPI-2.x to Data Management Service (DMS)

Problem Definition

The Ed-Fi API Profiles feature enables the enforcement of data policies on API resources in support of specific operational or regulatory scenarios. Profiles are rulesets defined in XML, specifying which Resource properties, references, and collections are exposed or excluded for each API consumer role.

Currently, this feature exists in the AdminAPI-2.x codebase and enforces rules at the API layer. Moving this capability into the Data Management Service (DMS) is needed for:

• Centralized policy enforcement
• Support for dynamic, database-defined Profiles
• Consistency in the Ed-Fi ecosystem

Conceptual Design

Main Principles

• Profiles control resource visibility and update paths via explicit inclusion/exclusion, as specified in the XML profile documents.
• Profile enforcement is integrated into DMS's pipeline, ensuring all reads and writes are checked against active policies before returning data or performing updates.
• Database-backed/dynamic Profiles are supported, with rules imported from XML documents and stored in database tables for runtime evaluation.
• _Profile selection: _ API clients can specify the profile via the HTTP header (Accept for reads, Content-Type for writes); DMS resolves the effective data policy and enforces.
• Legacy compatibility: XML format is retained for import/export. End users benefit from no format migration.

Integration Strategy

graph TD
    A[API Client] -->|Profile via HTTP Header| B(DMS API Gateway)
    B --> C[Profile Resolver]
    C --> D[Profile Rules Engine]
    D --> E[Resource Data Filter]
    E --> F[Database]
    C & D --> G[Profile Table (DB)]

Loading

• Profiles are managed through DMS Admin, stored in Profile Table, and used by Rules Engine at runtime.
• Profile XML import/export is provided in Admin UI, so XMLs remain the source of truth/user interface.
• Rules Engine leverages existing logic for JSON schema manipulation and overposted data removal, reusing most AdminAPI-2.x parsing algorithms.
• Requests are filtered dynamically; profile applied is selected by header or pipeline logic (default assignment).

Existing XML Logic Utilization

• XML parsing: Use/adapt AdminAPI-2.x XML deserializer. Parsing produces object graph for Profile rules.
• Rule translation: This object graph is imported to DB tables. Use existing code for exclusion/inclusion resolution.
• Policy enforcement: For each request, DMS matches the resolved profile, retrieves rules from the DB, and applies inclusion/exclusion logic.
• Schema manipulation: Profile rules filter resource shape dynamically at runtime.

Implementation Tickets

Ticket 1: DMS Profile Table and XML Import

• User Story: As a DMS admin, I can import API Profile XML documents, and have them stored as dynamic profiles in the DMS database.
• Acceptance Criteria:
  • XML profile documents can be imported via DMS admin.
  • All profile rules are parsed and stored in Profile Table.
  • Profile Table schema supports all XML rule types (property, reference, collection, descriptor, etc).
  • Admin can view a list of profiles and their effective rules.
• Technical Details:
  • Migrate/adapt XML deserialization from AdminAPI-2.x.
  • Design DB schema for profiles and rules.
  • Provide Admin UI import/upload endpoint.
  • Support legacy XML format; migrations must retain format compatibility.

Ticket 2: DMS Profiles Enforcement Pipeline

• User Story: When a user performs data operations through the DMS, the system enforces the rules defined in the associated Profile, restricting properties, references, and collections as defined.
• Acceptance Criteria:
  • DMS request pipeline resolves the active profile per request.
  • Profile rules filter resource data shape before read/write completes.
  • Profile enforcement occurs regardless of source (imported XML, DB-edited profiles).
  • Enforcement supports exclusion/inclusion logic for all resource aspects.
• Technical Details:
  • Integrate rule engine in DMS pipeline.
  • Leverage/extend JSON schema filtering, reusing AdminAPI-2.x exclusion/inclusion logic.
  • Unit/integration test coverage for all resource types.

Ticket 3: Admin API - Profile Export and Management

• User Story: As an admin, I can export profiles from the DMS database back into XML files, and manage their lifecycle through an easy-to-use UI.
• Acceptance Criteria:
  • Profiles can be exported into valid XML, matching legacy files.
  • UI allows viewing, exporting, deleting, and duplicating profiles.
  • XML format is normalized and huma...

• Fixes Design: Adaptation of Ed-Fi API Profiles Feature (XML) into Data Management Service (DMS) #763

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[stephenfuqua]

@jleiva-gap if you're not going to merge this PR, please go ahead and close it.