chore: add agg_qwen.yaml to multimodal deploy #2872
Conversation
Signed-off-by: GuanLuo <[email protected]>
GuanLuo
requested review from
hhzhang16,
indrajit96,
krishung5 and
whoisj
as code owners
WalkthroughAdds a new Kubernetes DynamoGraphDeployment manifest for a multimodal Qwen (Qwen2.5-VL-7B-Instruct) example using the vLLM backend. Defines four services—Frontend, EncodeWorker, VLMWorker, Processor—each in namespace agg-qwen with replicas=1, per-service images, GPU limits for workers, envFromSecret for Hugging Face token, and component-specific commands/args. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor User
participant Frontend as Frontend Service (Pod)
participant Encode as EncodeWorker (Pod, GPU=1)
participant VLM as VLMWorker (Pod, GPU=1, prefill)
participant Proc as Processor (Pod, GPU=1)
User->>Frontend: Submit multimodal request (text + image)
Frontend->>Encode: Run encode_worker.py\n--model Qwen2.5-VL-7B-Instruct
Encode-->>Frontend: Encoded embeddings
Frontend->>VLM: Run worker.py --worker-type prefill\n--model Qwen2.5-VL-7B-Instruct
VLM-->>Frontend: Prefill outputs
Frontend->>Proc: Run processor.py\n--prompt-template "USER: <image> <prompt> ASSISTANT:"
Proc-->>Frontend: Final response
Frontend-->>User: Return answer
note over Encode,VLM: Pods use envFromSecret=hf-token-secret
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Poem
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (8)
examples/multimodal/deploy/agg_qwen.yaml (8)
4-8: Confirm CRD scope and set metadata.namespaceAdd metadata.namespace: agg-qwen so the CR is namespaced and grouped with its Pods. If the CRD is cluster-scoped, ignore; otherwise this omission will block apply.
Apply:
apiVersion: nvidia.com/v1alpha1 kind: DynamoGraphDeployment metadata: name: agg-qwen + namespace: agg-qwen
11-18: Frontend image is placeholder; consider pull settings and portsIf this is meant to run as-is, add imagePullPolicy, imagePullSecrets (private registry), and expose containerPort if the Frontend serves traffic.
Apply:
extraPodSpec: mainContainer: - image: my-registry/vllm-runtime:my-tag + image: my-registry/vllm-runtime:my-tag + imagePullPolicy: IfNotPresent + ports: + - containerPort: 8080 + # imagePullSecrets: + # - name: my-registry-creds
26-35: Prefer exec-form command/args over /bin/sh -cExec form improves signal handling and avoids shell-quoting pitfalls.
Apply:
mainContainer: image: my-registry/vllm-runtime:my-tag workingDir: /workspace/examples/multimodal - command: - - /bin/sh - - -c - args: - - python3 components/encode_worker.py --model Qwen/Qwen2.5-VL-7B-Instruct + command: ["python3"] + args: + - components/encode_worker.py + - --model + - Qwen/Qwen2.5-VL-7B-Instruct
44-51: Use exec-form for VLMWorker tooAvoid /bin/sh -c.
Apply:
mainContainer: image: my-registry/vllm-runtime:my-tag workingDir: /workspace/examples/multimodal - command: - - /bin/sh - - -c - args: - - python3 components/worker.py --model Qwen/Qwen2.5-VL-7B-Instruct --worker-type prefill + command: ["python3"] + args: + - components/worker.py + - --model + - Qwen/Qwen2.5-VL-7B-Instruct + - --worker-type + - prefill
60-68: Processor quoting: make prompt-template unambiguousExec form avoids shell escaping; preserve literal "\n" by keeping YAML single quotes.
Apply:
mainContainer: image: my-registry/vllm-runtime:my-tag workingDir: /workspace/examples/multimodal - command: - - /bin/sh - - -c - args: - - 'python3 components/processor.py --model Qwen/Qwen2.5-VL-7B-Instruct --prompt-template "USER: <image>\n<prompt> ASSISTANT:"' + command: ["python3"] + args: + - components/processor.py + - --model + - Qwen/Qwen2.5-VL-7B-Instruct + - --prompt-template + - 'USER: <image>\n<prompt> ASSISTANT:'
19-20: Secret reference structure: verify CRD expects a stringIf the CRD mirrors Kubernetes PodSpec, it might require envFrom with secretRef.name. Confirm the schema; adjust if needed.
Example if standard:
envFrom: - secretRef: name: hf-token-secretAlso applies to: 36-37, 53-54
11-68: Add basic ops hardening (probes, securityContext, scheduling)
- Liveness/readiness probes for Python services.
- securityContext (runAsNonRoot, allowPrivilegeEscalation: false).
- nodeSelector/tolerations to land on GPU nodes.
Example:
mainContainer: image: my-registry/vllm-runtime:my-tag + securityContext: + runAsNonRoot: true + allowPrivilegeEscalation: false + livenessProbe: + tcpSocket: { port: 8080 } + initialDelaySeconds: 20 + periodSeconds: 10 + readinessProbe: + tcpSocket: { port: 8080 } + initialDelaySeconds: 5 + periodSeconds: 5 + nodeSelector: + nvidia.com/gpu.present: "true" + tolerations: + - key: "nvidia.com/gpu" + operator: "Exists" + effect: "NoSchedule"Tune ports/handlers per component.
8-13: Namespace existenceEnsure Namespace agg-qwen is created. Provide a companion manifest.
Example:
apiVersion: v1 kind: Namespace metadata: name: agg-qwen
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
examples/multimodal/deploy/agg_qwen.yaml(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Build and Test - dynamo
🔇 Additional comments (2)
examples/multimodal/deploy/agg_qwen.yaml (2)
8-68: Overall: solid example scaffoldStructure and component split look good; once the above tweaks are applied, this should be a reliable runnable example.
19-26: Add matching resource requests and retain the CRD’sgpukey
The DynamoGraphDeployment CRD schema definesresources.limits.gpuandresources.requests.gpu(it does not recognizenvidia.com/gpu), so don’t rename the key. To avoid BestEffort QoS, add matching requests for GPU, CPU, and memory:resources: limits: gpu: "1" + requests: + gpu: "1" + cpu: "2" + memory: "8Gi"Likely an incorrect or invalid review comment.
Signed-off-by: GuanLuo <[email protected]>
Signed-off-by: GuanLuo <[email protected]>
Signed-off-by: GuanLuo <[email protected]>
Signed-off-by: GuanLuo <[email protected]> Signed-off-by: nnshah1 <[email protected]>
Overview:
Details:
Where should the reviewer start?
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
New Features
Documentation