Skip to content

grammars: x{min,max} repetition operator #6640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ochafik merged 36 commits into from
Jun 6, 2024
Merged

grammars: x{min,max} repetition operator #6640

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
0160469
grammars: x{min,max} repetition operator + tweak +/*/? to avoid dupli…
Apr 12, 2024
f2030e3
grammars: handle `x{n}` and fix `x{n,n}`
Apr 12, 2024
de0fd3f
grammars: document new repetition operators
Apr 12, 2024
9d9b5a3
grammars: nit
Apr 12, 2024
6b5518c
grammars: uniform use of int for min & max
Apr 12, 2024
0ceb69a
grammars: refactor parser test
Apr 12, 2024
8938a05
grammar: parsing tests w/ natural pretty print of updated expectations
Apr 12, 2024
0d7347f
grammars: much prettier print of expectations (+ TEST_GRAMMAR_PARSER_…
Apr 12, 2024
2e2df72
grammars: improve test pretty print again
Apr 12, 2024
ffe321d
grammars: pretty print rules and chars
Apr 12, 2024
a9351b8
grammars: fix copy rule skipping
Apr 12, 2024
9d8efa5
grammars: disallow `a{,}` (not allowed in regexps)
Apr 12, 2024
2d98ebf
Update common/grammar-parser.cpp
ochafik Apr 12, 2024
ec91342
grammars: fix copy rule skipping (again) & display of expectations
Apr 12, 2024
22faba6
grammars: more test cases
Apr 12, 2024
1fb7787
Merge remote-tracking branch 'origin/master' into grammar-reps
Apr 15, 2024
15585e0
grammars: update reps parsing to bring ? / * / + closer to before
Apr 19, 2024
93b754e
json: use new GBNF repetitions{m,n} syntax
Apr 19, 2024
2ecc2ae
grammars: update performance gotchas w/ repetition advice
Apr 20, 2024
a9a2983
Merge remote-tracking branch 'origin/master' into grammar-reps
Apr 21, 2024
d47f537
Update examples/json_schema_to_grammar.py
ochafik Apr 24, 2024
724f879
Update examples/server/public/json-schema-to-grammar.mjs
ochafik Apr 24, 2024
a61281f
grammars: comment on rule repetitions
Apr 24, 2024
d03c98e
grammars: ensure unambiguous number alternatives
Apr 24, 2024
21bac1e
grammar: nit typo switched error msgs
Apr 24, 2024
0c74ad3
grammar: nit numbering in comment
Apr 24, 2024
218f41f
json: update numeric rule to be unambiguous
Apr 24, 2024
2813835
Apply suggestions from code review
ochafik Apr 24, 2024
46fe648
Update examples/server/public/json-schema-to-grammar.mjs
ochafik Apr 24, 2024
eb7ccd8
json: fix integral-part
Apr 24, 2024
3c02508
Merge branch 'grammar-reps' of https://github.com/ochafik/llama.cpp i…
Apr 24, 2024
476c97d
Merge remote-tracking branch 'origin/master' into grammar-reps
Apr 30, 2024
990bf57
grammar: add repetition tests
Apr 30, 2024
d070aee
Merge remote-tracking branch 'origin/master' into grammar-reps
May 18, 2024
8266b7c
Merge remote-tracking branch 'origin/master' into grammar-reps
May 21, 2024
2b79d47
Merge remote-tracking branch 'origin/master' into grammar-reps
Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
grammars: ensure unambiguous number alternatives
  • Loading branch information
Olivier Chafik committed Apr 24, 2024
commit d03c98ed9ab4d5384a39cc0cd68cc05f620ed644
2 changes: 1 addition & 1 deletion common/json-schema-to-grammar.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ struct BuiltinRule {
std::unordered_map<std::string, BuiltinRule> PRIMITIVE_RULES = {
{"boolean", {"(\"true\" | \"false\") space", {}}},
{"decimal-part", {"[0-9]{1,16}", {}}},
{"integral-part", {"[0-9] | [1-9] [0-9]{0,15}", {}}},
{"integral-part", {"[0-9] | [1-9] [0-9]{1,15}", {}}},
Comment thread
HanClinto marked this conversation as resolved.
Outdated
{"number", {"(\"-\"? integral-part) (\".\" decimal-part)? ([eE] [-+]? integral-part)? space", {"integral-part", "decimal-part"}}},
{"integer", {"(\"-\"? integral-part) space", {"integral-part"}}},
{"value", {"object | array | string | number | boolean | null", {"object", "array", "string", "number", "boolean", "null"}}},
Expand Down
28 changes: 14 additions & 14 deletions tests/test-json-schema-to-grammar.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
boolean ::= ("true" | "false") space
char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F]{4})
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
null ::= "null" space
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
object ::= "{" space ( string ":" space value ("," space string ":" space value)* )? "}" space
Expand Down Expand Up @@ -233,7 +233,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
"type": "integer"
})""",
R"""(
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
root ::= ("-"? integral-part) space
space ::= " "?
)"""
Expand Down Expand Up @@ -298,7 +298,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
R"""(
char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F]{4})
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
root ::= "[" space string "," space number "]" space
space ::= " "?
Expand All @@ -314,7 +314,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
})""",
R"""(
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
root ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
space ::= " "?
)"""
Expand Down Expand Up @@ -381,7 +381,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
R"""(
decimal-part ::= [0-9]{1,16}
integer ::= ("-"? integral-part) space
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
item ::= number | integer
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
root ::= "[" space item ("," space item){2,4} "]" space
Expand Down Expand Up @@ -555,7 +555,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
additional-value ::= "[" space (number ("," space number)*)? "]" space
char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F]{4})
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
root ::= "{" space (additional-kvs )? "}" space
space ::= " "?
Expand All @@ -575,7 +575,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
boolean ::= ("true" | "false") space
char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F]{4})
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
null ::= "null" space
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
object ::= "{" space ( string ":" space value ("," space string ":" space value)* )? "}" space
Expand All @@ -597,7 +597,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
boolean ::= ("true" | "false") space
char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F]{4})
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
null ::= "null" space
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
object ::= "{" space ( string ":" space value ("," space string ":" space value)* )? "}" space
Expand Down Expand Up @@ -638,7 +638,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
additional-kvs ::= additional-kv ( "," space additional-kv )*
char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F]{4})
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
root ::= "{" space a-kv ( "," space ( additional-kvs ) )? "}" space
space ::= " "?
Expand All @@ -663,7 +663,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
additional-kvs ::= additional-kv ( "," space additional-kv )*
char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F]{4})
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
root ::= "{" space (a-kv a-rest | additional-kvs )? "}" space
space ::= " "?
Expand Down Expand Up @@ -691,7 +691,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
b-rest ::= additional-kvs
char ::= [^"\\] | "\\" (["\\/bfnrt] | "u" [0-9a-fA-F]{4})
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
root ::= "{" space a-kv ( "," space ( b-kv b-rest | additional-kvs ) )? "}" space
space ::= " "?
Expand Down Expand Up @@ -755,7 +755,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
decimal-part ::= [0-9]{1,16}
foo ::= "{" space (foo-a-kv )? "}" space
foo-a-kv ::= "\"a\"" space ":" space number
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
root ::= alternative-0 | alternative-1
space ::= " "?
Expand Down Expand Up @@ -799,7 +799,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
d-kv ::= "\"d\"" space ":" space number
d-rest ::= ( "," space c-kv )?
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
root ::= "{" space a-kv "," space b-kv ( "," space ( d-kv d-rest | c-kv ) )? "}" space
space ::= " "?
Expand Down Expand Up @@ -842,7 +842,7 @@ static void test_all(const std::string & lang, std::function<void(const TestCase
})""",
R"""(
decimal-part ::= [0-9]{1,16}
integral-part ::= [0-9] | [1-9] [0-9]{0,15}
integral-part ::= [0-9] | [1-9] [0-9]{1,15}
number ::= ("-"? integral-part) ("." decimal-part)? ([eE] [-+]? integral-part)? space
number- ::= "{" space number-number-kv "}" space
number-kv ::= "\"number\"" space ":" space number-
Expand Down