Skip to content

Quantization improvements for k_quants #2707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Aug 22, 2023
Merged

Quantization improvements for k_quants #2707

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f26f9ef
Improve LLaMA-2 2-, 3- and 4-bit quantization
Aug 13, 2023
77aea72
Minor 4-bit quantization improvement
Aug 13, 2023
ec9cb75
Some more fine tuning
Aug 13, 2023
4f8dcb1
Adding make_qkx2_quants
Aug 14, 2023
e9f1340
Another minor improvement
Aug 14, 2023
1c1f985
Q2_K improvement
Aug 14, 2023
404e43c
Iterating
Aug 16, 2023
9f78d4c
Revert Q5_K back to make_qkx1_quants
Aug 20, 2023
e2af308
Better Q6_K
Aug 21, 2023
b706339
make_qkx2_quants is better for Q5_K after all
Aug 22, 2023
35a0b97
Fix after rebasing on master
Aug 22, 2023
fdf73db
Fix for changed tensor names
Aug 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
make_qkx2_quants is better for Q5_K after all
  • Loading branch information
Iwan Kawrakow committed Aug 22, 2023
commit b7063393d87d90fd0d39132450f05bd9e5c58e1d
16 changes: 8 additions & 8 deletions k_quants.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -847,8 +847,8 @@ void quantize_row_q5_K_reference(const float * restrict x, block_q5_K * restrict
uint8_t L[QK_K];
float mins[QK_K/32];
float scales[QK_K/32];
//float weights[32];
//uint8_t Laux[32];
float weights[32];
uint8_t Laux[32];
#else
int8_t L[QK_K];
float scales[QK_K/16];
Expand All @@ -861,12 +861,12 @@ void quantize_row_q5_K_reference(const float * restrict x, block_q5_K * restrict
float max_scale = 0; // as we are deducting the min, scales are always positive
float max_min = 0;
for (int j = 0; j < QK_K/32; ++j) {
scales[j] = make_qkx1_quants(32, 31, x + 32*j, L + 32*j, &mins[j], 9, 0.5f);
//float sum_x2 = 0;
//for (int l = 0; l < 32; ++l) sum_x2 += x[32*j + l] * x[32*j + l];
//float av_x = sqrtf(sum_x2/32);
//for (int l = 0; l < 32; ++l) weights[l] = av_x + fabsf(x[32*j + l]);
//scales[j] = make_qkx2_quants(32, 31, x + 32*j, weights, L + 32*j, &mins[j], Laux, -0.5f, 0.1f, 15, false);
//scales[j] = make_qkx1_quants(32, 31, x + 32*j, L + 32*j, &mins[j], 9, 0.5f);
float sum_x2 = 0;
for (int l = 0; l < 32; ++l) sum_x2 += x[32*j + l] * x[32*j + l];
float av_x = sqrtf(sum_x2/32);
for (int l = 0; l < 32; ++l) weights[l] = av_x + fabsf(x[32*j + l]);
scales[j] = make_qkx2_quants(32, 31, x + 32*j, weights, L + 32*j, &mins[j], Laux, -0.5f, 0.1f, 15, false);
float scale = scales[j];
if (scale > max_scale) {
max_scale = scale;
Expand Down