Add basic support for function calls in oai python server

[xaedes]

I wanted to try magentic with the oai python server.
To work correctly the oai api for function calls needed to be implemented.

Function calls allow the LLM to respond with function calls instead of only text messages.
A grammar is automatically generated which constrains the LLM response by the provided function json schemes.
When the client replies with function results they will be included in the chat prompt.

Example usage with magentic:

> server -m open-llama-7b-v2-q8_0.gguf # or some other model

> python api_like_OAI.py --chat-prompt "" --stop "USER:"

import openai
from magentic import prompt
from magentic import prompt_chain
from pydantic import BaseModel
from enum import Enum

# connect to api_like_OAI.py 
openai.api_base = "http://127.0.0.1:8081"
openai.api_key = ""

@prompt('Add more "dude"ness to: {phrase}')
def dudeify(phrase: str) -> str:
    ...  # No function body as this is never executed

print(dudeify("Hello, how are you?"))
# " Dude. How're ya doin', brotha? [insert random gibberish]..."

class Element(str, Enum):
    fire = 'fire'
    water = 'water'
    plant = 'plant'
    stone = 'stone'
    air = 'air'

class Superhero(BaseModel):
    hero_name: str
    age: int
    element: Element
    power: str
    enemies: list[str]

class SuperheroTeam(BaseModel):
    team_name: str
    members: list[Superhero]

@prompt("You create named superheros.\n\nA Superhero named {example_name}: {example}\nA Superhero named {name}:")
def create_superhero(name: str, example_name:str, example:str) -> Superhero:
    ...

@prompt("You create named superhero teams. Give each hero a name.\n\n" + 
         "Superhero team named {example_name}: {example}\nSuperhero team named {name}: ")
def create_superhero_team(name: str, example_name:str, example:str) -> SuperheroTeam:
    ...
    
superhero = Superhero(hero_name='Garden Man', age=30, power='Control over plants',
                      element=Element.plant, enemies=['Pollution Man', 'Concrete Woman'])

new_hero = create_superhero("Tree Mage", example_name=superhero.hero_name, example=superhero.model_dump_json())
print(new_hero)

team = SuperheroTeam(team_name="The Leafs", members=[superhero, new_hero])
new_team = create_superhero_team("Mountain Guard", example_name=team.team_name, example=team.model_dump_json())

print(team)
print(new_team)
# hero_name='Tree Mage' age=25 element=<Element.plant: 'plant'> power='control trees' enemies=['Garden Man', 'Humidity Girl']
# team_name='The Leafs' members=[
#  Superhero(hero_name='Garden Man', age=30, element=<Element.plant: 'plant'>, power='Control over plants', enemies=['Pollution Man', 'Concrete Woman']), 
#  Superhero(hero_name='Tree Mage', age=25, element=<Element.plant: 'plant'>, power='control trees', enemies=['Garden Man', 'Humidity Girl'])]
# team_name='Mountain Guard' members=[
#  Superhero(hero_name='Bear Biker', age=23, element=<Element.fire: 'fire'>, power='burning fire to turn into bike and attack', enemies=['Fireman', 'Flame Maiden']),
#  Superhero(hero_name='Cactus Climber', age=34, element=<Element.plant: 'plant'>, power='hanging from the top of a tree with cacti as climbing aids. Can use them for melee attacks or even to fire projectiles from her spines.', enemies=['Fireman'])]

[cebtenzzre]

Why does this PR include commented-out code?

[xaedes]

Good question, removed it now.

[ejones]

Love to see grammar getting implemented in the OAI layer! As for the JSON schema conversion, I might consider an approach like this:

1. start with examples/json-schema-to-grammar.py - I'm fairly confident in its handling of nesting, quoting, escaping, etc.
2. before processing, do a pass to inline any $refs from $defs, and then just process as usual with json-schema-to-grammar (or alternatively, add support for $refs/$defs to json-schema-to-grammar)

[ejones]

nit: I assume this is to match the other conditionals but the parentheses are unnecessary

[ejones]

nit: unnecessary outer parens in both statements

[ejones]

nit: idiomatic Python would be snake_case: post_data_decide etc

[ejones]

Looks like ws is actually unused (which could be fine)?

[ejones]

Should be boolean and also lowercase:

boolean ::= "true" | "false"

[ejones]

Running make_grammar in isolation, I'm noticing nested objects get rendered as {func}-object rather than the specific rule name. I believe it comes down to the use of schema_typename here?

>>> print(make_grammar([{'name': 'foo', 'parameters': {'type': 'object', 'properties': {'a': {'type': 'object', 'properties': {'b': {'type': 'object', 'properties': {'c': {'type': 'number'}}}}}}}}], {'name': 'foo'}))
foo-b ::= "{" "\"" "c" "\"" ":" number "}"
foo-a ::= "{" "\"" "b" "\"" ":" foo-object "}"
foo ::= "{" "\"" "a" "\"" ":" foo-object "}"

[ejones]

These still need to be quoted in the grammar, i.e., "1" | "2" instead of 1 | 2:

>>> print(make_grammar([{'name': 'foo', 'parameters': {'type': 'object', 'properties': {'a': {'type': 'number', 'enum': [1,2,3]}}}}], {'name': 'foo'}))
foo-a ::= ( 1 | 2 | 3 )
foo ::= "{" "\"" "a" "\"" ":" number "}"

[ejones]

Is there a reason arrays are treated specially, rather than a recursive case along with objects? Also, this approach seems to miss the element definition in the case of non-primitive elements:

>>> print(make_grammar([{'name': 'foo', 'parameters': {'type': 'object', 'properties': {'a': {'type': 'array', 'items': {'type': 'object', 'properties': {'a': {'type': 'number'}}}}}}}], {'name': 'foo'}))
foo ::= "{" "\"" "a" "\"" ":" foo-array-foo-object "}"
foo-array-foo-object ::= "[" ( foo-object ( "," foo-object )* )? "]"
root ::= foo
ws ::= [ \t\n]?
string ::=
  "\"" (
    [^"\\] |
    "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes
  )* "\""
bool ::= "True" | "False"
integer ::= ("-"? ([0-9] | [1-9] [0-9]*))
number ::= ("-"? ([0-9] | [1-9] [0-9]*)) ("." [0-9]+)? ([eE] [-+]? [0-9]+)?

[ejones]

AFAIK, JSON Schema does not require type in conjunction with enum. The types could just be inferred from the values themselves. This allows the server to be more permissive in its handling of schemas.

[ejones]

In general, would need to escape fn here as it might contain quotes or escapes.

[xaedes]

@ejones Ahh thanks for pointing out the json-schema-to-grammar.py, totally have overseen it! Will look to make use of this. I think I will use it without the property sorting, so that users can specify the order of properties just by the order in the json.

[ejones]

Yeah I mean, given that it's an OAI compatibility layer, I don't think there's a place for the property order as a parameter anyway.

FWIW, in json-schema-to-grammar.py, the prop order is there to solve the problem of what order object properties should be presented to the model. This could affect the generations as it determines which values of an object the model commits to first. Since JSON / JSON schema generally treat object properties as unordered, that might not be the same as the source order. But, again, since this is acting as a drop-in replacement for OAI, I'm not sure there's an alternative.

[LorenzoBoccaccia]

the prop order is there to solve the problem of what order object properties should be presented to the model

tho while the json schema say it's unordered, the way the json converter works is by rearranging the unspecified properties in alphabetical order. for json schema that are generated they are generated in the order that property appears, so if it could have a default 'iteration order' over the schema properties instead of choosing alphabetical, you'd get a easier to live with default.