[TaskProcessing] Add audio-to-audio chat task type #53759
Conversation
| 'history' => new ShapeDescriptor( | ||
| $this->l->t('Chat history'), | ||
| $this->l->t('The history of chat messages before the current message, starting with a message by the user'), | ||
| EShapeType::ListOfTexts |
There was a problem hiding this comment.
Where do we get the texts from?
There was a problem hiding this comment.
From the chat conversation history in the chat UI.
Ideally we need a mixed list of text and audio but since we will get the transcription of audio responses (as optional output), my plan is to store that in a chat message so the history can be text-only for now. Wdyt?
There was a problem hiding this comment.
I see. But if the transcription is an optional output, we may not get it when other providers implement this task type. Do we schedule another transcription task then?
There was a problem hiding this comment.
I think so, yes.
I even think the text output could be mandatory and part of the task type output shape (because we need it for the history). Wdyt?
There was a problem hiding this comment.
Yeah, I was thinking that as well, let's make it mandatory
There was a problem hiding this comment.
The input transcription is optional though. Or is it? We need it too for the history.
julien-nc
force-pushed
the
enh/noid/taskpro-audio-chat
branch
from
a549444 to
b9c1fc0
Compare
julien-nc
force-pushed
the
enh/noid/taskpro-audio-chat
branch
from
b9c1fc0 to
cce12a9
Compare
Signed-off-by: Julien Veyssier <[email protected]>
julien-nc
force-pushed
the
enh/noid/taskpro-audio-chat
branch
from
cce12a9 to
af059cb
Compare
|
Rebased on master. |
This is going towards making it possible to have a voice interaction with the Assistant.
The Assistant chat interface will schedule such audio-to-audio chat tasks if Agency is not available.
This does not cover Agency for now.
Do we make a
ContextAgentVoiceInteractiontask type that takes voice as input?We just need to support voice input and voice output. In between, the interactions between the agent and the model are text-based. So the Context Agent app would most likely still schedule TextToTextChatWithTools tasks.
We can also go the easy road and do the 3 steps when Agency is involved: STT -> Context Agent -> TTS. So we would not need any change in Context Agent and no new task type.
To be discussed.