From 84da0f553ed884efecaf59e509838a2bb3f92358 Mon Sep 17 00:00:00 2001 From: jianyuzh Date: Tue, 19 Mar 2024 09:54:39 +0800 Subject: [PATCH 01/11] update readme sycl for new update --- README-sycl.md | 111 ++++++++++++++++++++++--------- examples/sycl/win-run-llama2.bat | 2 - 2 files changed, 80 insertions(+), 33 deletions(-) diff --git a/README-sycl.md b/README-sycl.md index 9359a94901677..ade09d30c2b99 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -29,6 +29,7 @@ For Intel CPU, recommend to use llama.cpp for X86 (Intel MKL building). ## News - 2024.3 + - New base line is ready: tag b2437. - Support multiple cards: **--split-mode**: [none|layer]; not support [row], it's on developing. - Support to assign main GPU by **--main-gpu**, replace $GGML_SYCL_DEVICE. - Support detecting all GPUs with level-zero and same top **Max compute units**. @@ -81,7 +82,7 @@ For dGPU, please make sure the device memory is enough. For llama-2-7b.Q4_0, rec |-|-|-| |Ampere Series| Support| A100| -### oneMKL +### oneMKL for CUDA The current oneMKL release does not contain the oneMKL cuBlas backend. As a result for Nvidia GPU's oneMKL must be built from source. @@ -254,16 +255,16 @@ Run without parameter: Check the ID in startup log, like: ``` -found 4 SYCL devices: - Device 0: Intel(R) Arc(TM) A770 Graphics, compute capability 1.3, - max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136 - Device 1: Intel(R) FPGA Emulation Device, compute capability 1.2, - max compute_units 24, max work group size 67108864, max sub group size 64, global mem size 67065057280 - Device 2: 13th Gen Intel(R) Core(TM) i7-13700K, compute capability 3.0, - max compute_units 24, max work group size 8192, max sub group size 64, global mem size 67065057280 - Device 3: Intel(R) Arc(TM) A770 Graphics, compute capability 3.0, - max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136 - +found 6 SYCL devices: +| | | |Compute |Max compute|Max work|Max sub| | +|ID| Device Type| Name|capability|units |group |group |Global mem size| +|--|------------------|---------------------------------------------|----------|-----------|--------|-------|---------------| +| 0|[level_zero:gpu:0]| Intel(R) Arc(TM) A770 Graphics| 1.3| 512| 1024| 32| 16225243136| +| 1|[level_zero:gpu:1]| Intel(R) UHD Graphics 770| 1.3| 32| 512| 32| 53651849216| +| 2| [opencl:gpu:0]| Intel(R) Arc(TM) A770 Graphics| 3.0| 512| 1024| 32| 16225243136| +| 3| [opencl:gpu:1]| Intel(R) UHD Graphics 770| 3.0| 32| 512| 32| 53651849216| +| 4| [opencl:cpu:0]| 13th Gen Intel(R) Core(TM) i7-13700K| 3.0| 24| 8192| 64| 67064815616| +| 5| [opencl:acc:0]| Intel(R) FPGA Emulation Device| 1.2| 24|67108864| 64| 67064815616| ``` |Attribute|Note| @@ -271,12 +272,35 @@ found 4 SYCL devices: |compute capability 1.3|Level-zero running time, recommended | |compute capability 3.0|OpenCL running time, slower than level-zero in most cases| -4. Set device ID and execute llama.cpp +4. Device selection and execute llama.cpp + +There are two modes to select device: + +- Single device: Use one device assigned by user. +- Multiple devices: Detect and choose all devices which have same top Max compute units automatically. + +|Device selection|Parameter| +|-|-| +|Single device|--split-mode none --main-gpu DEVICE_ID | +|Multiple devices|--split-mode layer (default)| + +Examples: + +- Use device 0: + +```sh +ZES_ENABLE_SYSMAN=1 ./build/bin/main -m models/llama-2-7b.Q4_0.gguf -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33 -sm none -mg 0 +``` +or run by script: + +```sh +./examples/sycl/run_llama2.sh 0 +``` -Set device ID = 0 by **GGML_SYCL_DEVICE=0** +- Use multiple devices: ```sh -GGML_SYCL_DEVICE=0 ./build/bin/main -m models/llama-2-7b.Q4_0.gguf -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33 +ZES_ENABLE_SYSMAN=1 ./build/bin/main -m models/llama-2-7b.Q4_0.gguf -p "Building a website can be done in 10 simple steps:" -n 400 -e -ngl 33 -sm layer ``` or run by script: @@ -293,7 +317,9 @@ Note: Like: ``` -Using device **0** (Intel(R) Arc(TM) A770 Graphics) as main device +detect 1 SYCL GPUs: [0] with top Max compute units:512 +or +use 1 SYCL GPUs: [0] with Max compute units:512 ``` ## Windows @@ -430,15 +456,16 @@ build\bin\main.exe Check the ID in startup log, like: ``` -found 4 SYCL devices: - Device 0: Intel(R) Arc(TM) A770 Graphics, compute capability 1.3, - max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136 - Device 1: Intel(R) FPGA Emulation Device, compute capability 1.2, - max compute_units 24, max work group size 67108864, max sub group size 64, global mem size 67065057280 - Device 2: 13th Gen Intel(R) Core(TM) i7-13700K, compute capability 3.0, - max compute_units 24, max work group size 8192, max sub group size 64, global mem size 67065057280 - Device 3: Intel(R) Arc(TM) A770 Graphics, compute capability 3.0, - max compute_units 512, max work group size 1024, max sub group size 32, global mem size 16225243136 +found 6 SYCL devices: +| | | |Compute |Max compute|Max work|Max sub| | +|ID| Device Type| Name|capability|units |group |group |Global mem size| +|--|------------------|---------------------------------------------|----------|-----------|--------|-------|---------------| +| 0|[level_zero:gpu:0]| Intel(R) Arc(TM) A770 Graphics| 1.3| 512| 1024| 32| 16225243136| +| 1|[level_zero:gpu:1]| Intel(R) UHD Graphics 770| 1.3| 32| 512| 32| 53651849216| +| 2| [opencl:gpu:0]| Intel(R) Arc(TM) A770 Graphics| 3.0| 512| 1024| 32| 16225243136| +| 3| [opencl:gpu:1]| Intel(R) UHD Graphics 770| 3.0| 32| 512| 32| 53651849216| +| 4| [opencl:cpu:0]| 13th Gen Intel(R) Core(TM) i7-13700K| 3.0| 24| 8192| 64| 67064815616| +| 5| [opencl:acc:0]| Intel(R) FPGA Emulation Device| 1.2| 24|67108864| 64| 67064815616| ``` @@ -447,13 +474,31 @@ found 4 SYCL devices: |compute capability 1.3|Level-zero running time, recommended | |compute capability 3.0|OpenCL running time, slower than level-zero in most cases| -4. Set device ID and execute llama.cpp -Set device ID = 0 by **set GGML_SYCL_DEVICE=0** +4. Device selection and execute llama.cpp + +There are two modes to select device: + +- Single device: Use one device assigned by user. +- Multiple devices: Detect and choose all devices which have same top Max compute units automatically. + +|Device selection|Parameter| +|-|-| +|Single device|--split-mode none --main-gpu DEVICE_ID | +|Multiple devices|--split-mode layer (default)| + +Examples: + +- Use device 0: ``` -set GGML_SYCL_DEVICE=0 -build\bin\main.exe -m models\llama-2-7b.Q4_0.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e -ngl 33 -s 0 +build\bin\main.exe -m models\llama-2-7b.Q4_0.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e -ngl 33 -s 0 -sm none -mg 0 +``` + +- Use multiple devices: + +``` +build\bin\main.exe -m models\llama-2-7b.Q4_0.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e -ngl 33 -s 0 -sm layer ``` or run by script: @@ -470,7 +515,9 @@ Note: Like: ``` -Using device **0** (Intel(R) Arc(TM) A770 Graphics) as main device +detect 1 SYCL GPUs: [0] with top Max compute units:512 +or +use 1 SYCL GPUs: [0] with Max compute units:512 ``` ## Environment Variable @@ -489,7 +536,6 @@ Using device **0** (Intel(R) Arc(TM) A770 Graphics) as main device |Name|Value|Function| |-|-|-| -|GGML_SYCL_DEVICE|0 (default) or 1|Set the device id used. Check the device ids by default running output| |GGML_SYCL_DEBUG|0 (default) or 1|Enable log function by macro: GGML_SYCL_DEBUG| |ZES_ENABLE_SYSMAN| 0 (default) or 1|Support to get free memory of GPU by sycl::aspect::ext_intel_free_memory.
Recommended to use when --split-mode = layer| @@ -507,6 +553,9 @@ Using device **0** (Intel(R) Arc(TM) A770 Graphics) as main device ## Q&A +Note: please add prefix **[SYCL]** in issue title, so that we will check it as soon. + + - Error: `error while loading shared libraries: libsycl.so.7: cannot open shared object file: No such file or directory`. Miss to enable oneAPI running environment. @@ -538,4 +587,4 @@ Using device **0** (Intel(R) Arc(TM) A770 Graphics) as main device ## Todo -- Support multiple cards. +- NA diff --git a/examples/sycl/win-run-llama2.bat b/examples/sycl/win-run-llama2.bat index cf621c6759314..1d4d7d2cdcb6f 100644 --- a/examples/sycl/win-run-llama2.bat +++ b/examples/sycl/win-run-llama2.bat @@ -6,8 +6,6 @@ set INPUT2="Building a website can be done in 10 simple steps:\nStep 1:" @call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" intel64 --force -set GGML_SYCL_DEVICE=0 -rem set GGML_SYCL_DEBUG=1 .\build\bin\main.exe -m models\llama-2-7b.Q4_0.gguf -p %INPUT2% -n 400 -e -ngl 33 -s 0 From 6299e8d4c370ede98dce0043a4aa722273f810b8 Mon Sep 17 00:00:00 2001 From: Neo Zhang Jianyu Date: Tue, 19 Mar 2024 16:56:03 +0800 Subject: [PATCH 02/11] Update README-sycl.md Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com> --- README-sycl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-sycl.md b/README-sycl.md index ade09d30c2b99..53507931d4168 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -553,7 +553,7 @@ use 1 SYCL GPUs: [0] with Max compute units:512 ## Q&A -Note: please add prefix **[SYCL]** in issue title, so that we will check it as soon. +Note: please add prefix **[SYCL]** in issue title, so that we will check it as soon as possible. - Error: `error while loading shared libraries: libsycl.so.7: cannot open shared object file: No such file or directory`. From 634e3fca418e02aa44b8919eee418a34b22d4e04 Mon Sep 17 00:00:00 2001 From: Neo Zhang Jianyu Date: Tue, 19 Mar 2024 16:58:12 +0800 Subject: [PATCH 03/11] Update README-sycl.md Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com> --- README-sycl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-sycl.md b/README-sycl.md index 53507931d4168..5044c1ac4d017 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -587,4 +587,4 @@ Note: please add prefix **[SYCL]** in issue title, so that we will check it as s ## Todo -- NA +- Support row layer split for multiple card runs. From 28b6f88ee0d6e295ae13f47c4a94b5e62f405fca Mon Sep 17 00:00:00 2001 From: Neo Zhang Jianyu Date: Tue, 19 Mar 2024 16:58:22 +0800 Subject: [PATCH 04/11] Update README-sycl.md Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com> --- README-sycl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-sycl.md b/README-sycl.md index 5044c1ac4d017..e39c4531779cb 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -475,7 +475,7 @@ found 6 SYCL devices: |compute capability 3.0|OpenCL running time, slower than level-zero in most cases| -4. Device selection and execute llama.cpp +4. Device selection and execution of llama.cpp There are two modes to select device: From 60392d78a0aa753968e3bb92b5151d7002b5e0e1 Mon Sep 17 00:00:00 2001 From: Neo Zhang Jianyu Date: Tue, 19 Mar 2024 16:58:33 +0800 Subject: [PATCH 05/11] Update README-sycl.md Co-authored-by: Abhilash Majumder <30946547+abhilash1910@users.noreply.github.com> --- README-sycl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-sycl.md b/README-sycl.md index e39c4531779cb..7e2212dc11db8 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -272,7 +272,7 @@ found 6 SYCL devices: |compute capability 1.3|Level-zero running time, recommended | |compute capability 3.0|OpenCL running time, slower than level-zero in most cases| -4. Device selection and execute llama.cpp +4. Device selection and execution of llama.cpp There are two modes to select device: From 89c37062a459b0b2c199adc4c6f48fb633b0346c Mon Sep 17 00:00:00 2001 From: Neo Zhang Jianyu Date: Wed, 20 Mar 2024 09:38:29 +0800 Subject: [PATCH 06/11] Update README-sycl.md Co-authored-by: AidanBeltonS <87009434+AidanBeltonS@users.noreply.github.com> --- README-sycl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-sycl.md b/README-sycl.md index 7e2212dc11db8..f5405e1298b54 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -274,7 +274,7 @@ found 6 SYCL devices: 4. Device selection and execution of llama.cpp -There are two modes to select device: +There are two device selection modes: - Single device: Use one device assigned by user. - Multiple devices: Detect and choose all devices which have same top Max compute units automatically. From 329ae27760387db51360cb8716d3ee3f4b462b6c Mon Sep 17 00:00:00 2001 From: Neo Zhang Jianyu Date: Wed, 20 Mar 2024 09:39:49 +0800 Subject: [PATCH 07/11] Update README-sycl.md Co-authored-by: AidanBeltonS <87009434+AidanBeltonS@users.noreply.github.com> --- README-sycl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-sycl.md b/README-sycl.md index f5405e1298b54..223e2bf0130d8 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -477,7 +477,7 @@ found 6 SYCL devices: 4. Device selection and execution of llama.cpp -There are two modes to select device: +There are two device selection modes: - Single device: Use one device assigned by user. - Multiple devices: Detect and choose all devices which have same top Max compute units automatically. From 9ac1008fd3dbe0499b556365c41731344589d6e4 Mon Sep 17 00:00:00 2001 From: jianyuzh Date: Wed, 20 Mar 2024 09:42:07 +0800 Subject: [PATCH 08/11] update by review comments --- README-sycl.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README-sycl.md b/README-sycl.md index 223e2bf0130d8..e560fe0ddf970 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -277,7 +277,7 @@ found 6 SYCL devices: There are two device selection modes: - Single device: Use one device assigned by user. -- Multiple devices: Detect and choose all devices which have same top Max compute units automatically. +- Multiple devices: Automatically choose the devices with the same biggest Max compute units. |Device selection|Parameter| |-|-| @@ -480,7 +480,7 @@ found 6 SYCL devices: There are two device selection modes: - Single device: Use one device assigned by user. -- Multiple devices: Detect and choose all devices which have same top Max compute units automatically. +- Multiple devices: Automatically choose the devices with the same biggest Max compute units. |Device selection|Parameter| |-|-| From 64d6765a09700ce5da50b6a70d1d62f8372854a3 Mon Sep 17 00:00:00 2001 From: jianyuzh Date: Wed, 20 Mar 2024 09:50:53 +0800 Subject: [PATCH 09/11] update w64devkit link --- README-sycl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-sycl.md b/README-sycl.md index e560fe0ddf970..794dfda42106c 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -381,7 +381,7 @@ a. Download & install cmake for Windows: https://cmake.org/download/ b. Download & install mingw-w64 make for Windows provided by w64devkit -- Download the latest fortran version of [w64devkit](https://github.com/skeeto/w64devkit/releases). +- Download the 1.19.0 version of [w64devkit](https://github.com/skeeto/w64devkit/releases/download/v1.19.0/w64devkit-1.19.0.zip). - Extract `w64devkit` on your pc. From 4fd4344216ac53ad84d70631abff33c796e90228 Mon Sep 17 00:00:00 2001 From: jianyuzh Date: Wed, 20 Mar 2024 09:56:46 +0800 Subject: [PATCH 10/11] update for verify device id part --- README-sycl.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/README-sycl.md b/README-sycl.md index 794dfda42106c..b207d389c7239 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -313,15 +313,19 @@ Note: - By default, mmap is used to read model file. In some cases, it leads to the hang issue. Recommend to use parameter **--no-mmap** to disable mmap() to skip this issue. -5. Check the device ID in output +5. Verify the device ID in output + +Verify to see if the selected GPU is shown in the output, like: -Like: ``` detect 1 SYCL GPUs: [0] with top Max compute units:512 -or +``` +Or +``` use 1 SYCL GPUs: [0] with Max compute units:512 ``` + ## Windows ### Setup Environment @@ -511,12 +515,16 @@ Note: - By default, mmap is used to read model file. In some cases, it leads to the hang issue. Recommend to use parameter **--no-mmap** to disable mmap() to skip this issue. -5. Check the device ID in output -Like: +5. Verify the device ID in output + +Verify to see if the selected GPU is shown in the output, like: + ``` detect 1 SYCL GPUs: [0] with top Max compute units:512 -or +``` +Or +``` use 1 SYCL GPUs: [0] with Max compute units:512 ``` From 37db7ef9d0b914cd06f16bd91d03d7e7005b093e Mon Sep 17 00:00:00 2001 From: Neo Zhang Jianyu Date: Wed, 20 Mar 2024 10:27:59 +0800 Subject: [PATCH 11/11] Update README-sycl.md Co-authored-by: Meng, Hengyu --- README-sycl.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README-sycl.md b/README-sycl.md index b207d389c7239..32adfda47769c 100644 --- a/README-sycl.md +++ b/README-sycl.md @@ -29,7 +29,7 @@ For Intel CPU, recommend to use llama.cpp for X86 (Intel MKL building). ## News - 2024.3 - - New base line is ready: tag b2437. + - New base line is ready: [tag b2437](https://github.com/ggerganov/llama.cpp/tree/b2437). - Support multiple cards: **--split-mode**: [none|layer]; not support [row], it's on developing. - Support to assign main GPU by **--main-gpu**, replace $GGML_SYCL_DEVICE. - Support detecting all GPUs with level-zero and same top **Max compute units**.