diff --git a/README.md b/README.md index 36695ce80..93368c517 100644 --- a/README.md +++ b/README.md @@ -45,6 +45,13 @@ See the [docs folder](./docs/) for additional developer-oriented documentation. - [Setting Up Development Environment](./docs/SETUP-DEV-ENVIRONMENT.md) - [School Year Loader](./docs/SCHOOL-YEAR-LOADER.md) +## API Profiles Documentation + +- [API Profiles Design](./docs/API-PROFILES-DESIGN.md) - Comprehensive architecture and design +- [API Profiles Quick Start](./docs/API-PROFILES-QUICKSTART.md) - Getting started guide +- [API Profiles Migration](./docs/API-PROFILES-MIGRATION.md) - Migration from AdminAPI-2.x +- [Example Profiles](./docs/examples/profiles/) - Sample profile XML documents + ## Legal Information Copyright (c) 2025 Ed-Fi Alliance, LLC and contributors. diff --git a/docs/API-PROFILES-DESIGN.md b/docs/API-PROFILES-DESIGN.md new file mode 100644 index 000000000..7d3b8e87b --- /dev/null +++ b/docs/API-PROFILES-DESIGN.md @@ -0,0 +1,1242 @@ +# API Profiles Feature Design - Ed-Fi Data Management Service + +## Executive Summary + +This document describes the architecture and implementation strategy for adapting the Ed-Fi API Profiles feature from AdminAPI-2.x to the Data Management Service (DMS). API Profiles enable fine-grained data policy enforcement on API resources, controlling which properties, references, and collections are visible or modifiable based on specific operational or regulatory scenarios. + +## Table of Contents + +1. [Problem Definition](#problem-definition) +2. [Conceptual Design](#conceptual-design) +3. [Integration Strategy](#integration-strategy) +4. [Database Schema Design](#database-schema-design) +5. [Profile Resolution Flow](#profile-resolution-flow) +6. [Enforcement Pipeline Architecture](#enforcement-pipeline-architecture) +7. [Implementation Tickets](#implementation-tickets) +8. [Migration Strategy](#migration-strategy) +9. [API Specifications](#api-specifications) +10. [Example Profiles](#example-profiles) + +## Problem Definition + +### Background + +The Ed-Fi API Profiles feature is a critical capability that enables educational organizations to: + +- **Enforce data policies** on API resources based on operational or regulatory requirements +- **Control visibility** of sensitive data elements to different API consumers +- **Restrict update paths** to prevent unauthorized modifications +- **Support multi-tenant scenarios** where different clients have different data access rights + +### Current State + +The Profiles feature currently exists in the AdminAPI-2.x codebase where: +- Profiles are defined in XML format +- Rules are evaluated at the API layer during request processing +- Profile enforcement is tightly coupled with the API implementation +- Profile management is manual and file-based + +### Target State + +Moving this capability into DMS will provide: +- **Centralized policy enforcement** - consistent rules across all API operations +- **Dynamic profile management** - database-backed profiles with runtime updates +- **Scalability** - better performance with cached profile rules +- **Maintainability** - separation of concerns between policy and business logic +- **Consistency** - unified approach across the Ed-Fi ecosystem + +## Conceptual Design + +### Main Principles + +1. **Profile-Based Access Control** + - Profiles act as named policy documents that define resource access patterns + - Each profile specifies inclusion/exclusion rules for properties, references, and collections + - Multiple profiles can exist, supporting different consumer roles or scenarios + +2. **Database-Backed Storage** + - Profiles are imported from XML and stored in database tables + - Rules are normalized and indexed for efficient runtime evaluation + - Profile metadata (name, description, version) is maintained separately from rules + +3. **Runtime Enforcement** + - Profile resolution happens early in the request pipeline + - Rules are applied dynamically to filter request/response data + - Both read and write operations are subject to profile enforcement + +4. **Header-Based Selection** + - Clients specify profiles via HTTP headers + - `Accept` header for GET operations (response filtering) + - `Content-Type` header for POST/PUT operations (request validation) + - Default profile assignment when header is not specified + +5. **Legacy Compatibility** + - XML remains the canonical format for profile definition + - Import/export maintains format compatibility with AdminAPI-2.x + - Existing profile documents can be migrated without modification + +### Key Components + +```mermaid +graph TB + subgraph "API Layer" + A[API Client Request] + B[HTTP Headers
Accept/Content-Type] + end + + subgraph "DMS Pipeline" + C[Profile Resolver] + D[Profile Cache] + E[Rules Engine] + F[Request Validator] + G[Response Filter] + end + + subgraph "Data Layer" + H[Profile Repository] + I[Profile Tables] + J[Document Store] + end + + subgraph "Admin Interface" + K[Profile Import/Export] + L[Profile Management UI] + end + + A -->|1. Request| B + B -->|2. Extract Profile| C + C -->|3. Lookup| D + D -->|Cache Miss| H + H -->|4. Load| I + C -->|5. Rules| E + E -->|6a. Validate Write| F + E -->|6b. Filter Read| G + F -->|7. Store| J + G -->|7. Retrieve| J + K -->|Manage| I + L -->|Manage| I +``` + +## Integration Strategy + +### Architecture Overview + +The Profiles feature integrates into DMS through several layers: + +1. **Storage Layer** - PostgreSQL/MSSQL tables for profile data +2. **Repository Layer** - Data access abstractions for profile CRUD +3. **Service Layer** - Business logic for profile resolution and enforcement +4. **Middleware Layer** - Pipeline integration for request/response processing +5. **Admin Layer** - Management API for profile lifecycle operations + +### Request Processing Flow + +```mermaid +sequenceDiagram + participant Client + participant Gateway as API Gateway + participant Resolver as Profile Resolver + participant Cache as Profile Cache + participant DB as Profile DB + participant Pipeline as Request Pipeline + participant Backend as Document Store + + Client->>Gateway: HTTP Request + Profile Header + Gateway->>Resolver: Extract Profile Name + Resolver->>Cache: Get Profile Rules + alt Cache Hit + Cache-->>Resolver: Return Rules + else Cache Miss + Resolver->>DB: Query Profile + DB-->>Resolver: Profile Data + Resolver->>Cache: Update Cache + end + Resolver->>Pipeline: Attach Profile Context + + alt Write Operation + Pipeline->>Pipeline: Validate Against Profile + Pipeline->>Backend: Store if Valid + else Read Operation + Pipeline->>Backend: Retrieve Data + Pipeline->>Pipeline: Filter Against Profile + Pipeline-->>Client: Filtered Response + end +``` + +### Pipeline Integration Points + +The profile enforcement integrates into the existing DMS pipeline at specific stages: + +1. **Early Pipeline** - Profile resolution (after authentication) +2. **Validation Stage** - Write operation validation (before schema validation) +3. **Processing Stage** - Data filtering (before/after backend operations) +4. **Response Stage** - Response filtering (before serialization) + +## Database Schema Design + +### Profile Metadata Table + +```sql +CREATE TABLE dms.Profile ( + ProfileId BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + ProfileName VARCHAR(255) NOT NULL UNIQUE, + Description TEXT NULL, + ResourceName VARCHAR(255) NOT NULL, + ContentType VARCHAR(255) NOT NULL, + ReadContentType VARCHAR(255) NULL, + WriteContentType VARCHAR(255) NULL, + IsActive BOOLEAN NOT NULL DEFAULT TRUE, + CreatedAt TIMESTAMP NOT NULL DEFAULT NOW(), + LastModifiedAt TIMESTAMP NOT NULL DEFAULT NOW(), + CONSTRAINT UK_Profile_Name_Resource UNIQUE (ProfileName, ResourceName) +); + +CREATE INDEX IX_Profile_Name ON dms.Profile (ProfileName); +CREATE INDEX IX_Profile_Resource ON dms.Profile (ResourceName); +CREATE INDEX IX_Profile_Active ON dms.Profile (IsActive) WHERE IsActive = TRUE; +``` + +### Profile Property Rules Table + +```sql +CREATE TABLE dms.ProfilePropertyRule ( + ProfilePropertyRuleId BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + ProfileId BIGINT NOT NULL, + PropertyPath VARCHAR(1024) NOT NULL, + RuleType VARCHAR(50) NOT NULL, -- 'Include', 'Exclude' + ChildPolicy VARCHAR(50) NULL, -- 'IncludeAll', 'IncludeOnly', 'ExcludeAll', 'ExcludeOnly' + CONSTRAINT FK_ProfilePropertyRule_Profile + FOREIGN KEY (ProfileId) REFERENCES dms.Profile(ProfileId) + ON DELETE CASCADE +); + +CREATE INDEX IX_ProfilePropertyRule_ProfileId ON dms.ProfilePropertyRule (ProfileId); +CREATE INDEX IX_ProfilePropertyRule_PropertyPath ON dms.ProfilePropertyRule (PropertyPath); +``` + +### Profile Collection Rules Table + +```sql +CREATE TABLE dms.ProfileCollectionRule ( + ProfileCollectionRuleId BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + ProfileId BIGINT NOT NULL, + CollectionPath VARCHAR(1024) NOT NULL, + RuleType VARCHAR(50) NOT NULL, -- 'Include', 'Exclude' + ChildPolicy VARCHAR(50) NULL, + CONSTRAINT FK_ProfileCollectionRule_Profile + FOREIGN KEY (ProfileId) REFERENCES dms.Profile(ProfileId) + ON DELETE CASCADE +); + +CREATE INDEX IX_ProfileCollectionRule_ProfileId ON dms.ProfileCollectionRule (ProfileId); +CREATE INDEX IX_ProfileCollectionRule_CollectionPath ON dms.ProfileCollectionRule (CollectionPath); +``` + +### Profile Reference Rules Table + +```sql +CREATE TABLE dms.ProfileReferenceRule ( + ProfileReferenceRuleId BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + ProfileId BIGINT NOT NULL, + ReferencePath VARCHAR(1024) NOT NULL, + RuleType VARCHAR(50) NOT NULL, -- 'Include', 'Exclude' + CONSTRAINT FK_ProfileReferenceRule_Profile + FOREIGN KEY (ProfileId) REFERENCES dms.Profile(ProfileId) + ON DELETE CASCADE +); + +CREATE INDEX IX_ProfileReferenceRule_ProfileId ON dms.ProfileReferenceRule (ProfileId); +CREATE INDEX IX_ProfileReferenceRule_ReferencePath ON dms.ProfileReferenceRule (ReferencePath); +``` + +### Schema Rationale + +- **Normalized Structure**: Separate tables for properties, collections, and references allow efficient querying +- **Indexing Strategy**: Indexes on ProfileId and paths support fast rule lookup +- **Cascade Deletes**: Simplifies profile removal by automatically cleaning up rules +- **Path Storage**: VARCHAR(1024) accommodates deeply nested property paths +- **Extensibility**: Schema supports future rule types without structural changes + +## Profile Resolution Flow + +### Resolution Algorithm + +```mermaid +flowchart TD + Start([Incoming Request]) --> CheckHeader{Profile Header
Present?} + CheckHeader -->|Yes| ExtractName[Extract Profile Name] + CheckHeader -->|No| UseDefault{Default Profile
Configured?} + + ExtractName --> LookupCache{Profile in Cache?} + UseDefault -->|Yes| ExtractName + UseDefault -->|No| NoProfile[No Profile Applied] + + LookupCache -->|Yes| ValidateCache{Cache Valid?} + LookupCache -->|No| QueryDB[Query Profile DB] + + ValidateCache -->|Yes| LoadFromCache[Load from Cache] + ValidateCache -->|No| QueryDB + + QueryDB --> ProfileExists{Profile Exists?} + ProfileExists -->|Yes| LoadProfile[Load Profile & Rules] + ProfileExists -->|No| Error404[404 Profile Not Found] + + LoadProfile --> UpdateCache[Update Cache] + UpdateCache --> BuildContext[Build Profile Context] + LoadFromCache --> BuildContext + + BuildContext --> AttachToPipeline[Attach to Request Context] + NoProfile --> Continue[Continue Without Profile] + AttachToPipeline --> Continue + Continue --> End([Pipeline Processing]) + + Error404 --> End +``` + +### Cache Strategy + +- **Cache Key**: `profile:{profileName}:{resourceName}` +- **Cache Duration**: Configurable (default: 15 minutes) +- **Invalidation**: Manual or automatic on profile updates +- **Size Limits**: LRU eviction when memory threshold reached +- **Warming**: Pre-load frequently used profiles on startup + +## Enforcement Pipeline Architecture + +### Middleware Stack + +```mermaid +graph LR + subgraph "Request Flow" + A[Incoming Request] --> B[Authentication] + B --> C[Profile Resolution] + C --> D[Profile Validation] + D --> E[Schema Validation] + E --> F[Business Logic] + F --> G[Data Access] + end + + subgraph "Response Flow" + G --> H[Profile Filtering] + H --> I[Response Formation] + I --> J[Outgoing Response] + end + + style C fill:#ffeb3b + style D fill:#ffeb3b + style H fill:#ffeb3b +``` + +### Write Operation Enforcement + +For POST/PUT/PATCH operations: + +1. **Profile Resolution**: Extract profile from `Content-Type` header +2. **Rule Evaluation**: Load property/collection/reference rules +3. **Input Validation**: + - Check for excluded properties in request body + - Validate included properties are not missing (if required) + - Verify collection items conform to profile rules + - Ensure references are allowed +4. **Rejection**: Return 400 Bad Request if validation fails +5. **Processing**: Continue to backend if validation passes + +### Read Operation Enforcement + +For GET operations: + +1. **Profile Resolution**: Extract profile from `Accept` header +2. **Rule Loading**: Retrieve filtering rules for resource +3. **Data Retrieval**: Fetch data from backend (unfiltered) +4. **Response Filtering**: + - Remove excluded properties from response + - Filter collections based on collection rules + - Apply reference filtering + - Maintain data structure integrity +5. **Serialization**: Return filtered response + +### Error Handling + +- **Profile Not Found**: HTTP 404 with descriptive error message +- **Invalid Profile Rules**: HTTP 500 with error details logged +- **Validation Failures**: HTTP 400 with specific rule violations +- **Ambiguous Profile**: HTTP 400 when multiple profiles match + +## Implementation Tickets + +### Ticket 1: DMS Profile Table and XML Import + +**Epic**: Profile Management Infrastructure + +**User Story**: As a DMS administrator, I can import API Profile XML documents, and have them stored as dynamic profiles in the DMS database, so that I can centrally manage data access policies. + +**Acceptance Criteria**: +- [ ] Database schema created for profiles and rules (all tables with indexes) +- [ ] XML parser implemented for profile document deserialization +- [ ] Profile import API endpoint created (`POST /management/profiles/import`) +- [ ] XML validation enforced (schema compliance) +- [ ] Profile rules correctly parsed and stored in normalized tables +- [ ] Support for all rule types: properties, collections, references, descriptors +- [ ] Profile listing endpoint available (`GET /management/profiles`) +- [ ] Profile detail view endpoint available (`GET /management/profiles/{id}`) +- [ ] Error handling for malformed XML +- [ ] Transaction support for atomic import operations +- [ ] Unit tests for XML parsing (90%+ coverage) +- [ ] Integration tests for import API + +**Technical Implementation Details**: + +1. **Database Migration Script** (0026_Create_Profile_Tables.sql): + - Create all profile-related tables + - Add appropriate indexes and constraints + - Support both PostgreSQL and MSSQL + +2. **XML Parser Implementation** (ProfileXmlParser.cs): + - Leverage System.Xml.Serialization + - Define DTOs matching XML structure + - Handle nested elements for properties/collections/references + - Validate against expected schema + +3. **Repository Layer** (IProfileRepository.cs, ProfileRepository.cs): + - CRUD operations for profiles + - Batch insert for rules (performance) + - Query methods for efficient rule retrieval + +4. **Service Layer** (ProfileImportService.cs): + - Orchestrate XML parsing and database insertion + - Transaction management + - Validation and error handling + - Duplicate profile checking + +5. **API Controller** (ProfileManagementController.cs): + - File upload endpoint + - List/detail endpoints + - Authorization (admin only) + - Request/response DTOs + +**Dependencies**: +- Database access framework (Npgsql/Dapper) +- XML parsing libraries (System.Xml) +- File upload handling (ASP.NET Core) + +**Estimated Effort**: 8-10 story points (2 weeks) + +--- + +### Ticket 2: DMS Profiles Enforcement Pipeline + +**Epic**: Profile Enforcement + +**User Story**: When a user performs data operations through the DMS, the system enforces the rules defined in the associated Profile, restricting properties, references, and collections as defined, ensuring data governance policies are maintained. + +**Acceptance Criteria**: +- [ ] Profile resolver middleware implemented and integrated +- [ ] Profile resolution from HTTP headers (Accept/Content-Type) +- [ ] Write operation validation enforces profile rules +- [ ] Read operation filtering enforces profile rules +- [ ] Profile caching implemented for performance +- [ ] Support for default profile assignment +- [ ] Error responses for profile violations +- [ ] All resource types supported (resources, descriptors) +- [ ] Nested property filtering works correctly +- [ ] Collection filtering preserves data integrity +- [ ] Reference filtering applied correctly +- [ ] Pipeline performance impact <10ms per request +- [ ] Unit tests for all enforcement logic (90%+ coverage) +- [ ] Integration tests for read/write operations with profiles +- [ ] Performance tests validate caching effectiveness + +**Technical Implementation Details**: + +1. **Profile Resolution Middleware** (ProfileResolutionMiddleware.cs): + - Extract profile name from headers + - Cache lookup with fallback to database + - Attach profile context to HttpContext + - Handle missing/invalid profiles + +2. **Profile Cache Service** (ProfileCacheService.cs): + - In-memory cache using IMemoryCache + - Cache key generation + - Invalidation strategies + - Preloading mechanism + +3. **Write Enforcement Middleware** (ProfileWriteValidationMiddleware.cs): + - Integrate after ParseBodyMiddleware + - Traverse JSON document against profile rules + - Check for excluded properties + - Validate collection items + - Verify reference compliance + +4. **Read Enforcement Middleware** (ProfileReadFilteringMiddleware.cs): + - Integrate before response serialization + - Filter response JSON based on rules + - Remove excluded properties + - Filter collections + - Apply reference filters + +5. **Rule Evaluation Engine** (ProfileRuleEvaluator.cs): + - Recursive property path matching + - Include/exclude logic resolution + - Collection item filtering + - Reference filtering + - Performance optimizations (compiled expressions) + +6. **Profile Context** (ProfileContext.cs): + - Container for profile metadata and rules + - Efficient rule lookup structures (dictionaries/sets) + - Serializable for caching + +**Pipeline Integration**: +``` +Existing Pipeline: +[Auth] -> [ParsePath] -> [BuildResourceInfo] -> [ParseBody] -> [Validation] -> [Handler] + +With Profiles: +[Auth] -> [ProfileResolution] -> [ParsePath] -> [BuildResourceInfo] -> [ParseBody] + -> [ProfileWriteValidation] -> [Validation] -> [Handler] -> [ProfileReadFiltering] +``` + +**Dependencies**: +- Profile repository (Ticket 1) +- Request pipeline infrastructure +- JSON manipulation libraries (System.Text.Json) + +**Estimated Effort**: 13-15 story points (3 weeks) + +--- + +### Ticket 3: Admin API - Profile Export and Management + +**Epic**: Profile Management + +**User Story**: As an administrator, I can export profiles from the DMS database back into XML files, and manage their lifecycle through an easy-to-use UI, enabling profile portability and version control. + +**Acceptance Criteria**: +- [ ] Profile export endpoint implemented (`GET /management/profiles/{id}/export`) +- [ ] XML generation produces valid, well-formed documents +- [ ] Exported XML is compatible with AdminAPI-2.x format +- [ ] Profile update endpoint available (`PUT /management/profiles/{id}`) +- [ ] Profile delete endpoint available (`DELETE /management/profiles/{id}`) +- [ ] Profile activation/deactivation supported +- [ ] Profile duplication endpoint (`POST /management/profiles/{id}/duplicate`) +- [ ] XML format is human-readable with proper indentation +- [ ] Management UI allows profile CRUD operations +- [ ] UI supports XML preview before export +- [ ] Bulk export capability (multiple profiles) +- [ ] Export includes profile metadata (name, description, dates) +- [ ] Unit tests for XML serialization (90%+ coverage) +- [ ] Integration tests for all management endpoints +- [ ] UI/UX tests for management interface + +**Technical Implementation Details**: + +1. **XML Serializer** (ProfileXmlSerializer.cs): + - Convert database entities to XML DTOs + - Generate well-formed XML with proper namespaces + - Handle nested structures correctly + - Format output for readability + +2. **Export Service** (ProfileExportService.cs): + - Load profile with all rules from database + - Transform to XML DTO structure + - Serialize to XML string + - Validate exported XML against schema + +3. **Management API Extensions** (ProfileManagementController.cs): + - Export endpoint with file download + - Update/delete endpoints with authorization + - Activation toggle endpoint + - Duplication endpoint + +4. **Management UI** (ProfileManagementPage.razor): + - Profile list view with search/filter + - Profile detail/edit view + - XML preview modal + - Export/import buttons + - Delete confirmation dialogs + +5. **Validation Service** (ProfileValidationService.cs): + - Validate profile consistency before export + - Check for circular dependencies + - Ensure rule completeness + +**Admin UI Mockup**: +``` ++----------------------------------------------------------+ +| Profile Management [+ New Profile] | ++----------------------------------------------------------+ +| Search: [____________] Filter: [All ▼] | ++----------------------------------------------------------+ +| Name | Resource | Status | Actions | ++----------------------------------------------------------+ +| Test-Read-Only | Student | Active | [Export] ... | +| Assessment-Limited| Assessment | Active | [Export] ... | +| Minimal-Access | School | Inactive| [Export] ... | ++----------------------------------------------------------+ +``` + +**Dependencies**: +- Profile repository (Ticket 1) +- XML serialization libraries +- Admin UI framework (Blazor/React) + +**Estimated Effort**: 8-10 story points (2 weeks) + +--- + +### Ticket 4: API Profile Selection Mechanism + +**Epic**: Profile Runtime Features + +**User Story**: As an API client, I can specify which Profile to use for data requests using HTTP headers, or rely on default profile assignment, giving me control over data access patterns. + +**Acceptance Criteria**: +- [ ] `Accept` header parsing for GET operations +- [ ] `Content-Type` header parsing for POST/PUT operations +- [ ] Profile name extraction from media type parameters +- [ ] Default profile resolution when header absent +- [ ] Support for multiple profiles per consumer (future) +- [ ] Profile precedence rules documented +- [ ] Error handling for ambiguous profiles +- [ ] Profile selection works for all HTTP methods +- [ ] Selection logic handles edge cases (OPTIONS, HEAD) +- [ ] Documentation includes header format examples +- [ ] Client libraries updated with profile support +- [ ] Unit tests for header parsing (95%+ coverage) +- [ ] Integration tests for all selection scenarios +- [ ] Performance tests validate selection overhead + +**Technical Implementation Details**: + +1. **Header Parser** (ProfileHeaderParser.cs): + - Parse `Accept` header: `application/json;profile=student-read-only` + - Parse `Content-Type` header: `application/json;profile=student-write-limited` + - Extract profile name from media type parameters + - Handle multiple profiles (priority order) + - Validate profile name format + +2. **Profile Selection Service** (ProfileSelectionService.cs): + - Determine effective profile for request + - Apply default profile rules + - Handle profile conflicts + - Support profile hierarchy (future) + +3. **Configuration** (appsettings.json): + ```json + { + "Profiles": { + "DefaultProfile": "standard", + "EnableMultipleProfiles": false, + "RequireProfileForWrites": true + } + } + ``` + +4. **Client Examples**: + ```http + GET /data/v5/ed-fi/students + Accept: application/json;profile=read-only + + POST /data/v5/ed-fi/students + Content-Type: application/json;profile=write-limited + ``` + +5. **Documentation Updates**: + - Profile selection guide + - Header format specification + - Example requests with profiles + - Profile naming conventions + +**Error Scenarios**: +- Profile not found: 404 with message "Profile 'xyz' not found" +- Ambiguous profile: 400 with message "Multiple profiles match, specify one" +- Invalid profile name: 400 with message "Profile name 'xyz' is invalid" + +**Dependencies**: +- Profile resolution middleware (Ticket 2) +- HTTP header parsing utilities + +**Estimated Effort**: 5-8 story points (1-2 weeks) + +--- + +### Ticket 5: Enhanced Documentation & Example Profiles + +**Epic**: Documentation and Examples + +**User Story**: As a developer, I can review sample profile XMLs and clear documentation, enabling rapid onboarding and accurate implementation of profile-based data governance. + +**Acceptance Criteria**: +- [ ] Profile feature documentation completed +- [ ] Architecture diagrams included (Mermaid format) +- [ ] Profile XML schema documented +- [ ] 5+ example profiles provided with descriptions +- [ ] Migration guide from AdminAPI-2.x created +- [ ] API reference documentation updated +- [ ] Tutorial: Creating your first profile +- [ ] Tutorial: Migrating existing profiles +- [ ] Best practices guide for profile design +- [ ] Performance tuning guide +- [ ] Troubleshooting guide +- [ ] All examples tested and validated +- [ ] Documentation reviewed by technical writers +- [ ] Published to Ed-Fi documentation site + +**Technical Implementation Details**: + +1. **Core Documentation** (docs/API-PROFILES.md): + - Feature overview + - Use cases and scenarios + - Architecture and integration + - Configuration options + - Performance considerations + +2. **Tutorial Series**: + - Getting started with profiles + - Creating custom profiles + - Testing profile rules + - Deploying profiles to production + - Monitoring profile usage + +3. **Example Profiles** (examples/profiles/*.xml): + - student-read-only.xml + - student-write-limited.xml + - assessment-limited.xml + - school-minimal.xml + - descriptor-full-access.xml + +4. **Migration Guide** (docs/PROFILES-MIGRATION.md): + - Differences from AdminAPI-2.x + - Migration steps + - Common pitfalls + - Validation checklist + +5. **API Reference** (OpenAPI specification): + - Profile management endpoints + - Request/response schemas + - Example requests + - Error responses + +6. **Mermaid Diagrams**: + - Integration architecture + - Profile resolution flow + - Database schema + - Request pipeline + +**Documentation Structure**: +``` +docs/ +├── API-PROFILES.md (This document) +├── API-PROFILES-QUICKSTART.md +├── API-PROFILES-MIGRATION.md +├── API-PROFILES-REFERENCE.md +└── examples/ + └── profiles/ + ├── README.md + ├── student-read-only.xml + ├── student-write-limited.xml + └── assessment-limited.xml +``` + +**Dependencies**: +- All previous tickets (for accurate documentation) +- Example profile testing infrastructure + +**Estimated Effort**: 5-8 story points (1-2 weeks) + +## Migration Strategy + +### Migration from AdminAPI-2.x + +#### Phase 1: Assessment (Week 1-2) + +1. **Inventory Existing Profiles**: + - Identify all profile XML files in current deployment + - Document profile usage patterns + - Map profiles to API consumers + +2. **Compatibility Analysis**: + - Compare XML schema differences + - Identify deprecated features + - Note new capabilities + +3. **Test Environment Setup**: + - Deploy DMS in parallel with existing system + - Configure profile import pipeline + - Set up validation infrastructure + +#### Phase 2: Import and Validation (Week 3-4) + +1. **Batch Import**: + - Use profile import API to load existing XMLs + - Validate successful parsing and storage + - Compare rule counts and structure + +2. **Functional Testing**: + - Execute test cases with each profile + - Verify identical behavior between old and new systems + - Document any discrepancies + +3. **Performance Testing**: + - Benchmark profile resolution overhead + - Measure enforcement impact on throughput + - Optimize as needed + +#### Phase 3: Cutover (Week 5-6) + +1. **Gradual Rollout**: + - Start with non-production environments + - Enable profiles for specific API consumers + - Monitor for issues + +2. **Production Migration**: + - Schedule maintenance window + - Import all profiles to production DMS + - Activate profile enforcement + - Verify functionality + +3. **Decommission Old System**: + - Run systems in parallel for 1-2 weeks + - Gradually shift traffic to DMS + - Retire AdminAPI-2.x profile enforcement + +### Backward Compatibility + +- **XML Format**: Maintain 100% compatibility with AdminAPI-2.x format +- **Profile Names**: No naming changes required +- **Rule Semantics**: Identical enforcement behavior +- **API Headers**: Same header format supported + +### Breaking Changes + +None expected. All existing profiles should work without modification. + +## API Specifications + +### Profile Management API + +#### Import Profile + +```http +POST /management/v1/profiles/import +Content-Type: multipart/form-data +Authorization: Bearer {token} + +{profile-xml-file} +``` + +**Response**: +```json +{ + "id": "12345", + "name": "Student-Read-Only", + "resourceName": "Student", + "status": "imported", + "ruleCount": { + "properties": 15, + "collections": 3, + "references": 2 + } +} +``` + +#### List Profiles + +```http +GET /management/v1/profiles?resource={resourceName}&status={active|inactive} +Authorization: Bearer {token} +``` + +**Response**: +```json +{ + "profiles": [ + { + "id": "12345", + "name": "Student-Read-Only", + "resourceName": "Student", + "isActive": true, + "createdAt": "2025-01-15T10:00:00Z" + } + ], + "total": 1 +} +``` + +#### Get Profile Details + +```http +GET /management/v1/profiles/{id} +Authorization: Bearer {token} +``` + +**Response**: +```json +{ + "id": "12345", + "name": "Student-Read-Only", + "description": "Restricts student data to read-only fields", + "resourceName": "Student", + "contentType": "application/vnd.ed-fi.student.read-only+json", + "isActive": true, + "rules": { + "properties": [ + {"path": "studentUniqueId", "type": "Include"}, + {"path": "firstName", "type": "Include"} + ], + "collections": [ + {"path": "addresses", "type": "Exclude"} + ], + "references": [ + {"path": "schoolReference", "type": "Include"} + ] + } +} +``` + +#### Export Profile + +```http +GET /management/v1/profiles/{id}/export +Authorization: Bearer {token} +``` + +**Response**: XML file download + +#### Update Profile + +```http +PUT /management/v1/profiles/{id} +Content-Type: application/json +Authorization: Bearer {token} + +{ + "description": "Updated description", + "isActive": false +} +``` + +#### Delete Profile + +```http +DELETE /management/v1/profiles/{id} +Authorization: Bearer {token} +``` + +### Data API with Profiles + +#### Read with Profile + +```http +GET /data/v5/ed-fi/students/{id} +Accept: application/json;profile=student-read-only +Authorization: Bearer {token} +``` + +#### Write with Profile + +```http +POST /data/v5/ed-fi/students +Content-Type: application/json;profile=student-write-limited +Authorization: Bearer {token} + +{student-data} +``` + +## Example Profiles + +### Example 1: Student Read-Only Profile + +```xml + + + + + + + + + + + + + + + + + + +``` + +**Use Case**: External reporting system that only needs basic student demographics and school associations. + +### Example 2: Student Write-Limited Profile + +```xml + + + + + + + + + + + + + + + + + + + + + + + + +``` + +**Use Case**: Student information system integration that can update basic demographics and addresses, but not assessment or program data. + +### Example 3: Assessment Limited Profile + +```xml + + + + + + + + + +``` + +**Use Case**: Assessment vendor that can read basic assessment results but not accommodations or detailed objectives. + +### Example 4: School Minimal Profile + +```xml + + + + + + + + + + + + + + +``` + +**Use Case**: Public-facing school directory that only shows basic information. + +### Example 5: Descriptor Full Access Profile + +```xml + + + + + + + +``` + +**Use Case**: Administrator with full access to manage descriptors. + +## Security Considerations + +### Authentication and Authorization + +- Profile management APIs require admin-level authorization +- Profile selection does not bypass authentication requirements +- Profiles enforce data filtering but do not replace authorization + +### Data Privacy + +- Profiles support FERPA/GDPR compliance by restricting PII +- Sensitive fields can be excluded from specific profiles +- Audit logging tracks which profiles are used for data access + +### Performance and DoS Protection + +- Profile cache prevents excessive database queries +- Rate limiting applied to profile management APIs +- Large profiles validated for complexity before import + +### Validation + +- XML schema validation prevents malformed profiles +- Rule conflict detection before activation +- Profile testing framework for pre-production validation + +## Performance Considerations + +### Expected Impact + +- **Profile Resolution**: <5ms overhead per request (cached) +- **Write Validation**: 5-15ms depending on document size +- **Read Filtering**: 10-20ms depending on document complexity +- **Cache Hit Rate**: Target >95% for production workloads + +### Optimization Strategies + +1. **Caching**: Aggressive caching of profile rules +2. **Compilation**: Compile filter expressions for reuse +3. **Lazy Loading**: Load rules only when needed +4. **Indexing**: Optimize database queries with proper indexes +5. **Parallel Processing**: Filter collections in parallel + +### Monitoring + +- Track profile cache hit/miss rates +- Measure enforcement overhead per profile +- Alert on slow profile operations (>100ms) +- Dashboard for profile usage statistics + +## Testing Strategy + +### Unit Tests + +- XML parsing and serialization +- Rule evaluation logic +- Header parsing +- Cache operations +- Individual middleware components + +### Integration Tests + +- End-to-end profile import/export +- Request pipeline with profiles +- Multi-resource profile scenarios +- Error handling paths + +### Performance Tests + +- Profile resolution throughput +- Enforcement overhead measurement +- Cache effectiveness validation +- Large document handling + +### Compatibility Tests + +- AdminAPI-2.x profile import +- XML format validation +- Identical behavior verification + +## Future Enhancements + +### Phase 2 Features (Post-1.0) + +1. **Profile Inheritance**: Parent/child profile relationships +2. **Dynamic Rules**: Database-driven rule modifications without XML +3. **Profile Versioning**: Track and rollback profile changes +4. **Profile Analytics**: Usage metrics and optimization recommendations +5. **GUI Profile Editor**: Visual profile builder in Admin UI +6. **Profile Testing Framework**: Automated validation before deployment +7. **Multi-Profile Support**: Apply multiple profiles simultaneously +8. **Conditional Rules**: Rules based on data values or user attributes + +## References + +- [AdminAPI-2.x Repository](https://github.com/Ed-Fi-Alliance-OSS/AdminAPI-2.x) +- [Data Management Service Repository](https://github.com/Ed-Fi-Alliance-OSS/Data-Management-Service) +- [Project Tanager Documentation](https://github.com/Ed-Fi-Alliance-OSS/Project-Tanager) +- [Ed-Fi Documentation](https://docs.ed-fi.org/) +- [Ed-Fi API Profiles Specification](https://edfi.atlassian.net/wiki/spaces/EDFICERT/pages/20874540/Profiles) + +## Glossary + +- **Profile**: Named set of rules that control resource visibility and modification +- **Content Type**: Media type identifier including profile designation +- **Rule**: Specific directive to include/exclude resource elements +- **Property**: Simple scalar field on a resource +- **Collection**: Array of child objects on a resource +- **Reference**: Link to another resource +- **Member Selection**: Strategy for including/excluding elements (IncludeOnly, ExcludeOnly, IncludeAll, ExcludeAll) + +## Appendix A: XML Schema Definition + +```xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +``` + +## Appendix B: Configuration Reference + +```json +{ + "Profiles": { + "Enabled": true, + "DefaultProfile": null, + "RequireProfileForWrites": false, + "CacheSettings": { + "Enabled": true, + "ExpirationMinutes": 15, + "MaxCacheSize": 1000, + "PreloadProfiles": [] + }, + "ValidationSettings": { + "StrictMode": true, + "AllowEmptyProfiles": false, + "ValidateOnImport": true + }, + "PerformanceSettings": { + "MaxDocumentSizeKB": 1024, + "MaxNestedDepth": 10, + "ParallelFilteringEnabled": true + } + } +} +``` + +--- + +**Document Version**: 1.0 +**Last Updated**: 2025-12-09 +**Status**: Draft for Review +**Authors**: DMS Architecture Team +**Reviewers**: Ed-Fi Technical Steering Committee diff --git a/docs/API-PROFILES-MIGRATION.md b/docs/API-PROFILES-MIGRATION.md new file mode 100644 index 000000000..4f91bc63f --- /dev/null +++ b/docs/API-PROFILES-MIGRATION.md @@ -0,0 +1,721 @@ +# API Profiles Migration Guide + +## Overview + +This guide provides detailed instructions for migrating API Profiles from AdminAPI-2.x to the Ed-Fi Data Management Service (DMS). The migration process is designed to be straightforward, with full backward compatibility for existing profile XML documents. + +## Table of Contents + +1. [Pre-Migration Assessment](#pre-migration-assessment) +2. [Compatibility Matrix](#compatibility-matrix) +3. [Migration Steps](#migration-steps) +4. [Validation and Testing](#validation-and-testing) +5. [Common Migration Scenarios](#common-migration-scenarios) +6. [Troubleshooting](#troubleshooting) +7. [Rollback Plan](#rollback-plan) + +## Pre-Migration Assessment + +### Step 1: Inventory Existing Profiles + +Create a complete inventory of all profiles currently in use: + +```bash +# List all profile XML files +find /path/to/adminapi/profiles -name "*.xml" -type f > profile-inventory.txt + +# Document profile usage +cat profile-inventory.txt +``` + +Create a spreadsheet documenting: + +| Profile Name | XML File | Resources | Used By | Last Modified | +|-------------|----------|-----------|---------|---------------| +| Student-ReadOnly | student-ro.xml | Student | Reporting System | 2024-06-15 | +| Assessment-Limited | assess-lim.xml | StudentAssessment | Vendor XYZ | 2024-03-20 | + +### Step 2: Identify Profile Dependencies + +Document which API consumers use which profiles: + +```sql +-- Example query for AdminAPI-2.x database +SELECT + c.ClientName, + p.ProfileName, + c.LastUsedDate +FROM ApiClients c +JOIN ClientProfiles cp ON c.ClientId = cp.ClientId +JOIN Profiles p ON cp.ProfileId = p.ProfileId +ORDER BY c.ClientName; +``` + +### Step 3: Review Profile Complexity + +Categorize profiles by complexity: + +- **Simple**: Single resource, <10 rules, no nested collections +- **Moderate**: Multiple resources, 10-50 rules, some nested collections +- **Complex**: Multiple resources, >50 rules, deeply nested collections + +This helps prioritize migration order (simple first). + +### Step 4: Backup Current Configuration + +```bash +# Backup profile XML files +tar -czf profiles-backup-$(date +%Y%m%d).tar.gz /path/to/adminapi/profiles/ + +# Backup AdminAPI database (if applicable) +pg_dump adminapi_db > adminapi_backup_$(date +%Y%m%d).sql + +# Store backups securely +aws s3 cp profiles-backup-*.tar.gz s3://your-backup-bucket/ +``` + +## Compatibility Matrix + +### Supported Features + +| Feature | AdminAPI-2.x | DMS | Migration Required | +|---------|--------------|-----|-------------------| +| XML Profile Format | ✅ | ✅ | No | +| IncludeOnly Member Selection | ✅ | ✅ | No | +| ExcludeOnly Member Selection | ✅ | ✅ | No | +| Property Rules | ✅ | ✅ | No | +| Collection Rules | ✅ | ✅ | No | +| Reference Rules | ✅ | ✅ | No | +| Nested Collections | ✅ | ✅ | No | +| Profile via HTTP Headers | ✅ | ✅ | No | +| Multiple Resources per Profile | ✅ | ✅ | No | + +### Key Differences + +| Aspect | AdminAPI-2.x | DMS | Impact | +|--------|--------------|-----|--------| +| Storage | File-based | Database-backed | Performance improvement | +| Management | Manual file editing | API + UI | Easier management | +| Caching | Application memory | Distributed cache | Better scalability | +| Validation | Runtime | Import-time + Runtime | Earlier error detection | +| Versioning | Git/file system | Database history | Audit trail | + +### Unsupported/Deprecated Features + +**None**. All AdminAPI-2.x profile features are supported in DMS. + +## Migration Steps + +### Phase 1: Setup DMS Environment (Days 1-2) + +#### 1. Install DMS + +Follow the standard DMS installation guide: + +```bash +# Clone DMS repository +git clone https://github.com/Ed-Fi-Alliance-OSS/Data-Management-Service.git + +# Setup using Docker Compose +cd Data-Management-Service/eng/docker-compose +docker compose up -d +``` + +#### 2. Configure Profile Feature + +Enable profiles in DMS configuration: + +```json +// appsettings.json or environment variables +{ + "Profiles": { + "Enabled": true, + "DefaultProfile": null, + "CacheSettings": { + "Enabled": true, + "ExpirationMinutes": 15 + } + } +} +``` + +#### 3. Verify DMS is Running + +```bash +# Health check +curl http://localhost:8080/health + +# Verify Management API access +curl -X GET \ + http://localhost:8080/management/v1/profiles \ + -H "Authorization: Bearer YOUR_TOKEN" +``` + +### Phase 2: Import Profiles (Days 3-5) + +#### 1. Validate XML Files + +Before importing, validate all XML files: + +```bash +# Install xmllint if not available +sudo apt-get install libxml2-utils # Ubuntu/Debian +brew install libxml2 # macOS + +# Validate each profile +for file in profiles/*.xml; do + echo "Validating $file..." + xmllint --noout "$file" + if [ $? -eq 0 ]; then + echo " ✓ Valid" + else + echo " ✗ Invalid - needs fixing" + fi +done +``` + +#### 2. Import Profiles via API + +Create an import script: + +```bash +#!/bin/bash +# import-profiles.sh + +DMS_URL="http://localhost:8080" +TOKEN="YOUR_ADMIN_TOKEN" +PROFILE_DIR="./profiles" + +for profile in "$PROFILE_DIR"/*.xml; do + filename=$(basename "$profile") + echo "Importing $filename..." + + response=$(curl -s -w "\n%{http_code}" -X POST \ + "$DMS_URL/management/v1/profiles/import" \ + -H "Authorization: Bearer $TOKEN" \ + -F "file=@$profile") + + http_code=$(echo "$response" | tail -n1) + body=$(echo "$response" | sed '$d') + + if [ "$http_code" -eq 200 ] || [ "$http_code" -eq 201 ]; then + echo " ✓ Successfully imported" + echo " Response: $body" + else + echo " ✗ Failed with status $http_code" + echo " Error: $body" + fi + echo "" +done +``` + +Run the import: + +```bash +chmod +x import-profiles.sh +./import-profiles.sh | tee import-log.txt +``` + +#### 3. Verify Import Success + +```bash +# List all imported profiles +curl -X GET \ + http://localhost:8080/management/v1/profiles \ + -H "Authorization: Bearer YOUR_TOKEN" \ + | jq '.profiles[] | {name: .name, resource: .resourceName, active: .isActive}' + +# Check specific profile details +curl -X GET \ + http://localhost:8080/management/v1/profiles/12345 \ + -H "Authorization: Bearer YOUR_TOKEN" \ + | jq . +``` + +### Phase 3: Configure API Clients (Days 6-8) + +#### 1. Update Client Configuration + +For each API client, update configuration to point to DMS: + +```json +{ + "edfi": { + "baseUrl": "http://your-dms-instance:8080", + "oauthUrl": "http://your-dms-instance:8080/oauth/token", + "clientId": "your_client_id", + "clientSecret": "your_client_secret", + "profile": "Student-ReadOnly" // Add profile name + } +} +``` + +#### 2. Update HTTP Client Code + +**Before (AdminAPI-2.x)**: +```csharp +var request = new HttpRequestMessage(HttpMethod.Get, "/data/v3/ed-fi/students"); +request.Headers.Accept.Add( + new MediaTypeWithQualityHeaderValue("application/vnd.ed-fi.student.student-readonly.readable+json")); +``` + +**After (DMS)**: +```csharp +var request = new HttpRequestMessage(HttpMethod.Get, "/data/v5/ed-fi/students"); +request.Headers.Accept.Add( + new MediaTypeWithQualityHeaderValue("application/json")); +request.Headers.Accept.First().Parameters.Add( + new NameValueHeaderValue("profile", "Student-ReadOnly")); +``` + +Or more simply: +```csharp +request.Headers.Accept.ParseAdd("application/json;profile=Student-ReadOnly"); +``` + +#### 3. Test Client Applications + +Test each client application in a test environment: + +1. Configure client to use DMS test instance +2. Execute typical read operations +3. Execute typical write operations +4. Verify data filtering works as expected +5. Check error handling for invalid requests + +### Phase 4: Parallel Testing (Days 9-14) + +#### Run Both Systems Side-by-Side + +```mermaid +graph LR + A[API Client] --> B{Load Balancer} + B -->|90% traffic| C[AdminAPI-2.x] + B -->|10% traffic| D[DMS] + C --> E[Legacy DB] + D --> F[DMS DB] + D --> G[Profile Cache] +``` + +#### 1. Route Test Traffic to DMS + +Use a load balancer or feature flag to route a percentage of traffic: + +```yaml +# Kong API Gateway example +- name: edfi-api-route + paths: + - /data + plugins: + - name: canary + config: + percentage: 10 # 10% to DMS + upstream_host: dms-instance +``` + +#### 2. Compare Results + +Create a comparison script: + +```python +import requests +import json + +ADMINAPI_URL = "http://adminapi/data/v3/ed-fi/students" +DMS_URL = "http://dms/data/v5/ed-fi/students" +PROFILE = "Student-ReadOnly" + +# Get data from both systems +admin_response = requests.get( + ADMINAPI_URL, + headers={ + "Accept": "application/vnd.ed-fi.student.student-readonly.readable+json", + "Authorization": f"Bearer {ADMIN_TOKEN}" + } +) + +dms_response = requests.get( + DMS_URL, + headers={ + "Accept": f"application/json;profile={PROFILE}", + "Authorization": f"Bearer {DMS_TOKEN}" + } +) + +# Compare responses +admin_data = admin_response.json() +dms_data = dms_response.json() + +# Check if same fields are present +admin_fields = set(admin_data[0].keys()) +dms_fields = set(dms_data[0].keys()) + +print(f"Fields in AdminAPI only: {admin_fields - dms_fields}") +print(f"Fields in DMS only: {dms_fields - admin_fields}") +print(f"Common fields: {admin_fields & dms_fields}") +``` + +#### 3. Monitor Metrics + +Compare key metrics: + +| Metric | AdminAPI-2.x | DMS | Acceptable? | +|--------|--------------|-----|-------------| +| Response Time (p50) | 50ms | 55ms | ✅ | +| Response Time (p95) | 200ms | 180ms | ✅ | +| Error Rate | 0.1% | 0.1% | ✅ | +| Throughput | 1000 req/s | 1200 req/s | ✅ | + +### Phase 5: Production Cutover (Day 15+) + +#### 1. Schedule Maintenance Window + +Announce maintenance window to all stakeholders: + +``` +MAINTENANCE NOTIFICATION + +Service: Ed-Fi API +Date: [DATE] +Time: [TIME] to [TIME] (2-hour window) +Impact: API will be briefly unavailable during switchover +Action Required: None (automatic switchover) +``` + +#### 2. Execute Cutover + +```bash +# 1. Stop AdminAPI-2.x (or put in read-only mode) +# 2. Verify DMS has all latest data +# 3. Update DNS/load balancer to point to DMS +# 4. Verify traffic is flowing to DMS +# 5. Monitor for errors + +# Verification script +#!/bin/bash +while true; do + response=$(curl -s -o /dev/null -w "%{http_code}" \ + http://api.edfi.org/health) + + if [ "$response" -eq 200 ]; then + echo "$(date): DMS healthy" + else + echo "$(date): DMS unhealthy (Status: $response)" + # Alert on-call team + fi + + sleep 10 +done +``` + +#### 3. Monitor Post-Cutover + +Monitor closely for 24-48 hours: + +- Response times +- Error rates +- Profile cache hit rates +- Database performance +- Client application logs + +#### 4. Gradual Client Migration + +If using both systems in parallel, gradually migrate clients: + +**Week 1**: Internal applications (10%) +**Week 2**: Low-risk external clients (30%) +**Week 3**: Medium-risk clients (60%) +**Week 4**: All remaining clients (100%) + +## Validation and Testing + +### Automated Test Suite + +Create comprehensive tests: + +```bash +# tests/profile-migration-tests.sh + +#!/bin/bash +set -e + +DMS_URL="http://localhost:8080" +TOKEN="YOUR_TOKEN" + +echo "Testing Profile Migration..." + +# Test 1: Profile exists +echo "Test 1: Verify profile imported" +profile_id=$(curl -s -X GET \ + "$DMS_URL/management/v1/profiles?name=Student-ReadOnly" \ + -H "Authorization: Bearer $TOKEN" \ + | jq -r '.profiles[0].id') + +if [ -n "$profile_id" ] && [ "$profile_id" != "null" ]; then + echo " ✓ Profile exists (ID: $profile_id)" +else + echo " ✗ Profile not found" + exit 1 +fi + +# Test 2: Profile filters response +echo "Test 2: Verify profile filtering" +full_response=$(curl -s -X GET \ + "$DMS_URL/data/v5/ed-fi/students/12345" \ + -H "Accept: application/json" \ + -H "Authorization: Bearer $TOKEN") + +filtered_response=$(curl -s -X GET \ + "$DMS_URL/data/v5/ed-fi/students/12345" \ + -H "Accept: application/json;profile=Student-ReadOnly" \ + -H "Authorization: Bearer $TOKEN") + +full_fields=$(echo "$full_response" | jq 'keys | length') +filtered_fields=$(echo "$filtered_response" | jq 'keys | length') + +if [ "$filtered_fields" -lt "$full_fields" ]; then + echo " ✓ Profile filtering works (Full: $full_fields, Filtered: $filtered_fields)" +else + echo " ✗ Profile filtering not working" + exit 1 +fi + +# Test 3: Write validation +echo "Test 3: Verify write validation" +invalid_write=$(curl -s -w "\n%{http_code}" -X POST \ + "$DMS_URL/data/v5/ed-fi/students" \ + -H "Content-Type: application/json;profile=Student-ReadOnly" \ + -H "Authorization: Bearer $TOKEN" \ + -d '{"studentUniqueId":"123","firstName":"Test","excludedField":"value"}') + +http_code=$(echo "$invalid_write" | tail -n1) + +if [ "$http_code" -eq 400 ]; then + echo " ✓ Write validation rejects excluded fields" +else + echo " ✗ Write validation not working (Expected 400, got $http_code)" + exit 1 +fi + +echo "" +echo "All tests passed! ✓" +``` + +### Performance Benchmarks + +```bash +# benchmark-profiles.sh + +#!/bin/bash + +DMS_URL="http://localhost:8080" +TOKEN="YOUR_TOKEN" +ITERATIONS=1000 + +echo "Benchmarking profile performance..." + +# Without profile +echo "Without profile:" +time for i in $(seq 1 $ITERATIONS); do + curl -s -X GET \ + "$DMS_URL/data/v5/ed-fi/students/12345" \ + -H "Accept: application/json" \ + -H "Authorization: Bearer $TOKEN" \ + > /dev/null +done + +# With profile +echo "With profile:" +time for i in $(seq 1 $ITERATIONS); do + curl -s -X GET \ + "$DMS_URL/data/v5/ed-fi/students/12345" \ + -H "Accept: application/json;profile=Student-ReadOnly" \ + -H "Authorization: Bearer $TOKEN" \ + > /dev/null +done +``` + +## Common Migration Scenarios + +### Scenario 1: Simple Read-Only Profile + +**AdminAPI-2.x XML** (no changes needed): +```xml + + + + + + + + + +``` + +**Migration**: Direct import, no modifications required. + +### Scenario 2: Complex Multi-Resource Profile + +**AdminAPI-2.x XML** (no changes needed): +```xml + + + + + + + + + + + + +``` + +**Migration**: Direct import, handles multiple resources automatically. + +### Scenario 3: Legacy Content-Type Headers + +**AdminAPI-2.x Client Code**: +``` +Accept: application/vnd.ed-fi.student.student-readonly.readable+json +``` + +**DMS Client Code** (update required): +``` +Accept: application/json;profile=Student-ReadOnly +``` + +**Migration**: Update client code to use new header format. + +## Troubleshooting + +### Issue 1: Profile Import Fails + +**Symptoms**: 400 Bad Request during import + +**Causes**: +- Malformed XML +- Invalid element names +- Missing required attributes + +**Solutions**: +```bash +# Validate XML syntax +xmllint --noout profile.xml + +# Check for common issues +grep -n "memberSelection" profile.xml # Verify attribute is present +grep -n "= '[cutover_time]'; +``` + +### Post-Rollback Actions + +1. Analyze root cause of failure +2. Create fixes or workarounds +3. Re-test in staging environment +4. Schedule new cutover date +5. Document lessons learned + +## Post-Migration Checklist + +- [ ] All profiles imported successfully +- [ ] All API clients updated and tested +- [ ] Performance metrics within acceptable range +- [ ] No increase in error rates +- [ ] Profile cache hit rate >90% +- [ ] Monitoring and alerting configured +- [ ] AdminAPI-2.x decommission plan documented +- [ ] Team trained on DMS profile management +- [ ] Documentation updated +- [ ] Stakeholders notified of successful migration + +## Support and Resources + +- **DMS Documentation**: [https://github.com/Ed-Fi-Alliance-OSS/Data-Management-Service/docs](https://github.com/Ed-Fi-Alliance-OSS/Data-Management-Service/docs) +- **API Profiles Design**: [API-PROFILES-DESIGN.md](./API-PROFILES-DESIGN.md) +- **Quick Start Guide**: [API-PROFILES-QUICKSTART.md](./API-PROFILES-QUICKSTART.md) +- **Ed-Fi Community**: [https://www.ed-fi.org/community/](https://www.ed-fi.org/community/) +- **Technical Support**: support@ed-fi.org + +--- + +**Last Updated**: 2025-12-09 +**Document Version**: 1.0 diff --git a/docs/API-PROFILES-QUICKSTART.md b/docs/API-PROFILES-QUICKSTART.md new file mode 100644 index 000000000..a5a20e9a0 --- /dev/null +++ b/docs/API-PROFILES-QUICKSTART.md @@ -0,0 +1,464 @@ +# API Profiles Quick Start Guide + +## Introduction + +This guide will help you quickly get started with API Profiles in the Ed-Fi Data Management Service (DMS). Profiles allow you to control which data fields are visible or modifiable for different API consumers. + +## What are API Profiles? + +API Profiles are XML-based policy documents that define: + +- Which properties of a resource can be read or written +- Which collections (arrays) are accessible +- Which references (links to other resources) are included +- Different access levels for different API consumers + +## Use Cases + +Common scenarios where profiles are valuable: + +1. **Vendor Integration**: Limit third-party vendors to specific data fields +2. **Reporting Systems**: Provide read-only access to aggregated data +3. **Parent Portals**: Show only relevant student information to parents +4. **Data Privacy**: Hide PII fields from unauthorized consumers +5. **Compliance**: Enforce FERPA/GDPR requirements at the API level + +## Prerequisites + +- DMS Platform installed and running +- Admin access to DMS Configuration Service +- Basic understanding of Ed-Fi resources and data model +- Familiarity with XML (helpful but not required) + +## Quick Start: 5 Minute Setup + +### Step 1: Download Example Profile + +Download one of the example profiles from the repository: + +```bash +curl -O https://raw.githubusercontent.com/Ed-Fi-Alliance-OSS/Data-Management-Service/main/docs/examples/profiles/student-read-only.xml +``` + +Or create your own `student-read-only.xml`: + +```xml + + + + + + + + + + + +``` + +### Step 2: Import Profile + +Import the profile using the Management API: + +```bash +curl -X POST \ + https://your-dms-instance/management/v1/profiles/import \ + -H "Authorization: Bearer YOUR_TOKEN" \ + -F "file=@student-read-only.xml" +``` + +Response: +```json +{ + "id": "12345", + "name": "Student-Read-Only", + "resourceName": "Student", + "status": "imported", + "ruleCount": { + "properties": 4, + "collections": 0, + "references": 0 + } +} +``` + +### Step 3: Use Profile in API Request + +Make a request with the profile specified in the `Accept` header: + +```bash +curl -X GET \ + https://your-dms-instance/data/v5/ed-fi/students/{id} \ + -H "Accept: application/json;profile=student-read-only" \ + -H "Authorization: Bearer YOUR_TOKEN" +``` + +Response will only include the fields specified in the profile: + +```json +{ + "id": "abc123", + "studentUniqueId": "604822", + "firstName": "John", + "lastSurname": "Doe", + "birthDate": "2010-05-15" +} +``` + +### Step 4: Verify Profile + +List all profiles to confirm import: + +```bash +curl -X GET \ + https://your-dms-instance/management/v1/profiles \ + -H "Authorization: Bearer YOUR_TOKEN" +``` + +## Understanding Profile Syntax + +### Basic Structure + +```xml + + + + + + + + + + +``` + +### Member Selection Options + +- **IncludeOnly**: Only listed elements are accessible (whitelist) +- **ExcludeOnly**: All elements except listed are accessible (blacklist) +- **IncludeAll**: All elements are accessible (default, no filtering) +- **ExcludeAll**: No elements are accessible (use with caution) + +### Element Types + +1. **Property**: Simple scalar field + ```xml + + ``` + +2. **Collection**: Array of child objects + ```xml + + + + + ``` + +3. **Reference**: Link to another resource + ```xml + + + + ``` + +## Common Patterns + +### Pattern 1: Read-Only Access + +Allow reading specific fields, prevent all writes: + +```xml + + + + + + + + + + +``` + +### Pattern 2: Limited Write Access + +Allow updates to specific fields only: + +```xml + + + + + + + + + + + + + + +``` + +### Pattern 3: Hide Sensitive Data + +Exclude sensitive fields from read operations: + +```xml + + + + + + + + +``` + +### Pattern 4: Multi-Resource Profile + +Control access to multiple resources: + +```xml + + + + + + + + + + + + + + + +``` + +## Testing Your Profile + +### 1. Test with Postman/Insomnia + +Import this collection to test profiles: + +**GET Request with Profile**: +- URL: `{{base_url}}/data/v5/ed-fi/students/{{student_id}}` +- Headers: + - `Accept: application/json;profile={{profile_name}}` + - `Authorization: Bearer {{token}}` + +**POST Request with Profile**: +- URL: `{{base_url}}/data/v5/ed-fi/students` +- Headers: + - `Content-Type: application/json;profile={{profile_name}}` + - `Authorization: Bearer {{token}}` +- Body: Student JSON + +### 2. Compare Responses + +Test the same request with and without profile: + +```bash +# Without profile (full response) +curl -X GET \ + https://your-dms-instance/data/v5/ed-fi/students/{id} \ + -H "Accept: application/json" \ + -H "Authorization: Bearer YOUR_TOKEN" \ + | jq . > full-response.json + +# With profile (filtered response) +curl -X GET \ + https://your-dms-instance/data/v5/ed-fi/students/{id} \ + -H "Accept: application/json;profile=student-read-only" \ + -H "Authorization: Bearer YOUR_TOKEN" \ + | jq . > filtered-response.json + +# Compare +diff full-response.json filtered-response.json +``` + +### 3. Validate Write Restrictions + +Test that excluded fields are rejected: + +```bash +# This should fail if Addresses are excluded +curl -X POST \ + https://your-dms-instance/data/v5/ed-fi/students \ + -H "Content-Type: application/json;profile=student-read-only" \ + -H "Authorization: Bearer YOUR_TOKEN" \ + -d '{ + "studentUniqueId": "604822", + "firstName": "John", + "lastSurname": "Doe", + "addresses": [{"city": "Austin"}] # Excluded field + }' +``` + +Expected response: `400 Bad Request` with error indicating profile violation. + +## Troubleshooting + +### Profile Not Found + +**Error**: `404 Profile 'xyz' not found` + +**Solution**: +1. Verify profile name matches exactly (case-sensitive) +2. Check profile is active: `GET /management/v1/profiles` +3. Ensure profile was imported successfully + +### Profile Not Applied + +**Symptoms**: Response includes all fields despite profile specification + +**Causes**: +1. Profile header not formatted correctly +2. Profile enforcement not enabled in configuration +3. Profile cache needs refresh + +**Solutions**: +```bash +# Check profile status +curl -X GET \ + https://your-dms-instance/management/v1/profiles/{id} \ + -H "Authorization: Bearer YOUR_TOKEN" + +# Verify header format +Accept: application/json;profile=profile-name +# NOT: Accept: application/json; profile=profile-name (no space before profile) +``` + +### Write Validation Fails + +**Error**: `400 Bad Request: Property 'xyz' is not allowed by profile` + +**Cause**: Attempting to write a field excluded by profile + +**Solution**: Remove the excluded field from request body or update profile to include it + +### Performance Issues + +**Symptoms**: Slow API responses with profiles enabled + +**Solutions**: +1. Check profile cache hit rate in monitoring +2. Simplify complex profiles (reduce nested collections) +3. Increase cache duration in configuration +4. Pre-load frequently used profiles on startup + +## Best Practices + +### 1. Naming Conventions + +Use descriptive names that indicate purpose: +- ✅ `Student-ReadOnly-Demographics` +- ✅ `Assessment-Vendor-Limited` +- ✅ `Parent-Portal-Access` +- ❌ `Profile1` +- ❌ `Temp` + +### 2. Start with IncludeOnly + +For security-critical scenarios, use `IncludeOnly` to whitelist allowed fields: + +```xml + + + +``` + +This is safer than `ExcludeOnly` which can accidentally expose new fields added in future versions. + +### 3. Test Before Production + +Always test profiles in non-production environment: +1. Import profile to test environment +2. Validate read/write operations +3. Verify error handling +4. Check performance impact +5. Review audit logs + +### 4. Document Profiles + +Add descriptions to profiles explaining their purpose: + +```xml + + + + +``` + +### 5. Version Control + +Store profile XMLs in version control: +``` +profiles/ +├── README.md +├── student-read-only-v1.xml +├── student-read-only-v2.xml +└── changelog.md +``` + +### 6. Monitor Usage + +Track which profiles are used most frequently: +- Profile selection counts +- Error rates per profile +- Performance metrics +- Cache hit rates + +## Next Steps + +### Learn More + +- [Full Design Documentation](./API-PROFILES-DESIGN.md) +- [Migration Guide](./API-PROFILES-MIGRATION.md) +- [API Reference](./API-PROFILES-REFERENCE.md) + +### Advanced Topics + +- Profile inheritance and hierarchies +- Conditional rules based on data values +- Multi-profile support (applying multiple profiles) +- Dynamic profile updates without restart +- Profile testing frameworks + +### Get Help + +- Review example profiles in `docs/examples/profiles/` +- Check troubleshooting section above +- Consult Ed-Fi community forums +- Contact Ed-Fi support team + +## Example Profiles Repository + +Find complete, tested example profiles at: +``` +docs/examples/profiles/ +├── student-read-only.xml +├── student-write-limited.xml +├── assessment-limited.xml +├── school-minimal.xml +└── descriptor-full-access.xml +``` + +Each example includes: +- Use case description +- Complete XML profile +- Test data +- Expected behavior + +--- + +**Need Help?** Join the Ed-Fi community at https://www.ed-fi.org/community/ diff --git a/docs/examples/profiles/README.md b/docs/examples/profiles/README.md new file mode 100644 index 000000000..d1f47d693 --- /dev/null +++ b/docs/examples/profiles/README.md @@ -0,0 +1,135 @@ +# Example API Profiles + +This directory contains example API Profile XML documents that demonstrate common use cases for the Ed-Fi Data Management Service (DMS). + +## Available Profiles + +### 1. student-read-only.xml +**Purpose**: Provides read-only access to basic student demographics +**Use Case**: External reporting systems that need basic student information +**Includes**: StudentUniqueId, names, birth date, school reference +**Excludes**: All collections, write access + +### 2. student-write-limited.xml +**Purpose**: Allows limited write access for demographic updates +**Use Case**: Student information system integration for basic data updates +**Includes**: Demographics, addresses, electronic mails +**Excludes**: Assessment data, program associations, identification codes + +### 3. assessment-limited.xml +**Purpose**: Restricts assessment data access to core fields only +**Use Case**: Assessment vendor with need-to-know access +**Includes**: Basic assessment results, student reference +**Excludes**: Accommodations, detailed objectives, performance levels + +### 4. school-minimal.xml +**Purpose**: Public-facing school directory information +**Use Case**: School finder applications, public directories +**Includes**: School ID, name, type, operational status, LEA reference +**Excludes**: Internal administrative data, staff assignments + +### 5. descriptor-full-access.xml +**Purpose**: Full access to descriptor resources +**Use Case**: Administrative users managing system descriptors +**Includes**: All descriptor fields (read and write) +**Excludes**: None + +## Testing Profiles + +Each profile can be tested using the provided test scripts and data: + +```bash +# Import a profile +curl -X POST \ + https://your-dms-instance/management/v1/profiles/import \ + -H "Authorization: Bearer YOUR_TOKEN" \ + -F "file=@student-read-only.xml" + +# Test the profile +curl -X GET \ + https://your-dms-instance/data/v5/ed-fi/students/{id} \ + -H "Accept: application/json;profile=student-read-only" \ + -H "Authorization: Bearer YOUR_TOKEN" +``` + +## Customizing Profiles + +To create your own profile: + +1. Copy one of the example files +2. Rename the profile in the `name` attribute +3. Modify the `Resource` name if targeting a different resource +4. Adjust `memberSelection` strategy (IncludeOnly, ExcludeOnly, etc.) +5. Add or remove `Property`, `Collection`, and `Reference` elements +6. Test in non-production environment +7. Import to production when validated + +## Profile Naming Guidelines + +- Use descriptive names that indicate purpose +- Include resource name if profile is resource-specific +- Add version suffix for breaking changes (e.g., `-v2`) +- Use kebab-case or PascalCase consistently + +Examples: +- ✅ `Student-ReadOnly-Demographics` +- ✅ `Assessment-Vendor-Limited-v2` +- ✅ `School-Public-Directory` +- ❌ `profile1` +- ❌ `temp` + +## Validation + +Before importing to production: + +1. **XML Validation**: Ensure XML is well-formed + ```bash + xmllint --noout student-read-only.xml + ``` + +2. **Schema Validation**: Verify against profile schema + ```bash + xmllint --schema profile-schema.xsd student-read-only.xml + ``` + +3. **Functional Testing**: Test actual API behavior + - Import to test environment + - Execute read/write operations + - Verify filtering works as expected + - Test error cases + +4. **Performance Testing**: Measure impact + - Compare response times with/without profile + - Check cache effectiveness + - Monitor resource usage + +## Documentation + +For complete documentation on API Profiles, see: + +- [API Profiles Design](../../API-PROFILES-DESIGN.md) - Comprehensive architecture and design +- [API Profiles Quick Start](../../API-PROFILES-QUICKSTART.md) - Getting started guide +- [Ed-Fi API Profiles Specification](https://edfi.atlassian.net/wiki/spaces/EDFICERT/pages/20874540/Profiles) - Official specification + +## Contributing + +To contribute new example profiles: + +1. Create profile XML following the guidelines above +2. Add comprehensive comments explaining purpose and use case +3. Provide test scenarios and expected behavior +4. Update this README with profile description +5. Submit pull request with changes + +## Support + +For questions or issues with these profiles: +- Review the Quick Start guide +- Check the troubleshooting section in the main documentation +- Post in Ed-Fi community forums +- Contact Ed-Fi support + +--- + +**Last Updated**: 2025-12-09 +**Compatible With**: DMS 1.0+ diff --git a/docs/examples/profiles/assessment-limited.xml b/docs/examples/profiles/assessment-limited.xml new file mode 100644 index 000000000..0ac93425c --- /dev/null +++ b/docs/examples/profiles/assessment-limited.xml @@ -0,0 +1,46 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/examples/profiles/descriptor-full-access.xml b/docs/examples/profiles/descriptor-full-access.xml new file mode 100644 index 000000000..727ae54dc --- /dev/null +++ b/docs/examples/profiles/descriptor-full-access.xml @@ -0,0 +1,47 @@ + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/examples/profiles/profile-schema.xsd b/docs/examples/profiles/profile-schema.xsd new file mode 100644 index 000000000..eca8a8c69 --- /dev/null +++ b/docs/examples/profiles/profile-schema.xsd @@ -0,0 +1,213 @@ + + + + + + + + + Root element for an API Profile definition. A Profile contains one or more + Resource definitions that specify access rules. + + + + + + + + Defines access rules for a specific Ed-Fi resource (e.g., Student, School). + + + + + + + + Specifies which elements are accessible for read operations (GET requests). + + + + + + + Specifies which elements are accessible for write operations (POST/PUT requests). + + + + + + + + The name of the Ed-Fi resource (e.g., "Student", "School", "Assessment"). + + + + + + + + + + Unique name for this profile (e.g., "Student-ReadOnly", "Assessment-Limited"). + + + + + + + + + + + Defines the filtering rules for read or write operations. + Contains properties, collections, and references that should be included or excluded. + + + + + + + + + + + Strategy for including/excluding elements: + - IncludeOnly: Only listed elements are accessible (whitelist) + - ExcludeOnly: All elements except listed are accessible (blacklist) + - IncludeAll: All elements are accessible (no filtering) + - ExcludeAll: No elements are accessible + + + + + + + + + + Represents a simple property/field on a resource (e.g., "FirstName", "BirthDate"). + + + + + + The name of the property as defined in the Ed-Fi resource schema. + + + + + + + + + + Represents an array/collection on a resource (e.g., "Addresses", "ElectronicMails"). + Can contain nested Property, Collection, and Reference elements. + + + + + + + + + + + The name of the collection as defined in the Ed-Fi resource schema. + + + + + + + Optional member selection strategy for items within the collection. + If not specified, inherits from parent ContentType. + + + + + + + + + + Represents a reference to another resource (e.g., "SchoolReference", "StudentReference"). + Can contain Property elements to filter which reference fields are accessible. + + + + + + + + + The name of the reference as defined in the Ed-Fi resource schema. + + + + + + + Optional member selection strategy for properties within the reference. + If not specified, inherits from parent ContentType. + + + + + + + + + + Enumeration of valid member selection strategies for filtering elements. + + + + + + + Only the explicitly listed elements are accessible (whitelist approach). + Most secure option for sensitive data. + + + + + + + All elements except the explicitly listed ones are accessible (blacklist approach). + Use when most fields should be available with few exceptions. + + + + + + + All elements are accessible (no filtering applied). + Use for full access scenarios. + + + + + + + No elements are accessible (complete restriction). + Use with caution, typically for denying access to entire collections. + + + + + + + diff --git a/docs/examples/profiles/school-minimal.xml b/docs/examples/profiles/school-minimal.xml new file mode 100644 index 000000000..88698372e --- /dev/null +++ b/docs/examples/profiles/school-minimal.xml @@ -0,0 +1,66 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/examples/profiles/student-read-only.xml b/docs/examples/profiles/student-read-only.xml new file mode 100644 index 000000000..9b924cfb9 --- /dev/null +++ b/docs/examples/profiles/student-read-only.xml @@ -0,0 +1,47 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/examples/profiles/student-write-limited.xml b/docs/examples/profiles/student-write-limited.xml new file mode 100644 index 000000000..4a882f086 --- /dev/null +++ b/docs/examples/profiles/student-write-limited.xml @@ -0,0 +1,90 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +