Skip to content

Continuous layouts for quantization q4_0c #1073

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

Continuous layouts for quantization q4_0c #1073

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2219467
test-quantize: fix for q8_0 intermediates
unbounded Apr 15, 2023
a1e6fb9
q4_0c continous row layout
unbounded Apr 17, 2023
4bd781c
q4_0c: quantize support
unbounded Apr 17, 2023
ab543dc
q4_0c: AVX512 vec_dot and quantize impl
unbounded Apr 18, 2023
1b49d26
q4_0c: Arm Neon acceleration
unbounded Apr 20, 2023
2949725
q4_0c: prefetch on AVX-512 and ARM
unbounded Apr 24, 2023
d53f767
q4_0c: disable prefetching on M1
unbounded Apr 27, 2023
76692c9
q4_0c: avoid _mm512_loadu_epi64 instruction
unbounded May 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
test-quantize: fix for q8_0 intermediates
  • Loading branch information
@unbounded
unbounded committed May 4, 2023
commit 221946777c845c4665b1cb4d5055a39978f8c06f
10 changes: 5 additions & 5 deletions tests/test-quantize-perf.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ int main(int argc, char * argv[]) {
if (qfns.quantize_row_q && qfns.dequantize_row_q) {
printf("%s\n", ggml_type_name(type));

if (params.op_quantize_row_q_reference) {
if (params.op_quantize_row_q_reference && qfns.quantize_row_q_reference) {
printf(" quantize_row_q_reference\n");
for (size_t size : params.test_sizes) {
printf(" %zu values (%.2f MB)\n", size, 4*size/(float)(1024*1024));
Expand All @@ -242,7 +242,7 @@ int main(int argc, char * argv[]) {
printf("\n");
}

if (params.op_quantize_row_q) {
if (params.op_quantize_row_q && qfns.quantize_row_q) {
printf(" quantize_row_q\n");
for (size_t size : params.test_sizes) {
printf(" %zu values (%.2f MB)\n", size, 4*size/(float)(1024*1024));
Expand All @@ -256,7 +256,7 @@ int main(int argc, char * argv[]) {
printf("\n");
}

if (params.op_dequantize_row_q) {
if (params.op_dequantize_row_q && qfns.dequantize_row_q) {
printf(" dequantize_row_q\n");
qfns.quantize_row_q(test_data1, test_q1, largest);
for (size_t size : params.test_sizes) {
Expand All @@ -271,7 +271,7 @@ int main(int argc, char * argv[]) {
printf("\n");
}

if (params.op_quantize_row_q_dot) {
if (params.op_quantize_row_q_dot && qfns.quantize_row_q_dot) {
printf(" quantize_row_q_dot\n");
for (size_t size : params.test_sizes) {
printf(" %zu values (%.2f MB)\n", size, 4*size/(float)(1024*1024));
Expand All @@ -285,7 +285,7 @@ int main(int argc, char * argv[]) {
printf("\n");
}

if (params.op_vec_dot_q) {
if (params.op_vec_dot_q && qfns.vec_dot_q) {
printf(" vec_dot_q\n");
qfns.quantize_row_q(test_data1, test_q1, largest);
qfns.quantize_row_q(test_data2, test_q2, largest);
Expand Down