OpenCL kernel headers not found (macOS)

[cintainfinita]

MacOS BigSur, AMD Radeon Pro 5700 XT 16 GB

% ./john --list=build-info
Version: 1.9.0-jumbo-1+bleeding-b49c763c4 2021-06-24 21:32:53 +0200
Build: darwin20.5.0 64-bit x86_64 AVX2 AC
SIMD: AVX2, interleaving: MD4:4 MD5:5 SHA1:2 SHA256:1 SHA512:1
CPU tests: AVX2
$JOHN is ./
Format interface version: 14
Max. number of reported tunable costs: 4
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
SALT_HASH_SIZE: 1048576
SINGLE_IDX_MAX: 2147483648
SINGLE_BUF_MAX: 4294967295
Effective limit: Number of salts vs. SingleMaxBufferSize
Max. Markov mode level: 400
Max. Markov mode password length: 30
clang version: 12.0.5 (clang-1205.0.22.11) (gcc 4.2.1 compatibility)
OpenCL headers version: 1.2
Crypto library: OpenSSL
OpenSSL library version: 0101010bf
OpenSSL 1.1.1k 25 Mar 2021
File locking: fcntl()
fseek(): fseek
ftell(): ftell
fopen(): fopen
memmem(): System's
times(2) sysconf(_SC_CLK_TCK) is 100
Using times(2) for timers, resolution 10 ms
HR timer: mach_absolute_time(), latency 33 ns
Total physical host memory: 72 GiB
Available physical host memory: 56299 MiB
Terminal locale string: UTF-8
Parsed terminal locale: UTF-8

Path error:

% ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ./opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: :47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
^

Error building kernel ./opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1292 - clBuildProgram

I tried running john from a different path, no luck.

I specified the full path to the include instruction but still the file wasn't found.

Off the top of my head, to get around this you need to run john (at least once) from inside the opencl/ folder.

cd opencl
../john --test --format=opencl

Well, maybe you need to find the "right location" (if the 'opencl' folder does not fix it), as stated here https://www.openwall.com/lists/john-users/2014/08/03/2 (notice, this is the old headers location).

"OS X also has a problem with run-time compile of kernels that include
header files. A workaround is to cd to the src directory and run each
OpenCL format once. After that, the kernel binary is cached so you can
move away from the src directory."

[solardiz]

So the OpenCL compiler somehow ignored -I opencl that was passed to it. I wonder why this issue keeps occasionally reoccurring on various systems. Is there anything we're doing wrong (I don't think so)? Is there anything we can do better (e.g., maybe -I . along with opencl/ in paths inside the #include directives would have worked better across systems)?

@magnumripper Don't you use a similar Mac system now, yet aren't running into this issue?

[cintainfinita]

@claudioandre-br tried that (per that post, it mentions the "kernels" directory that is not there anymore, replaced by opencl, so I already did that) with no luck.

The odd part is that I modified the #include's to match my exact location of the .h files (full path) and it still says the file is not found.

This is a very old problem, I tried all the tricks from part workaround and none of them worked. Also, the kernels directory was moved to the opencl directory.

If I can help compiling and test any changes, count on me.

Thanks!

[magnumripper]

On 2021-06-25 14:39, Solar Designer wrote: So the OpenCL compiler somehow ignored `-I opencl` that was passed to it. I wonder why this issue keeps occasionally reoccurring on various systems. Is there anything we're doing wrong (I don't think so)? Is there anything we can do better (e.g., maybe `-I .` along with `opencl/` in paths inside the `#include` directives would have worked better across systems)? @magnumripper Don't you use a similar Mac system now, yet aren't running into this issue?

I haven't heard of that issue for years. If I ever did see it myself I'm sure I committed workarounds that should still be in place. magnum

[magnumripper]

What if you move all files in run/opencl to just run? That'd be ugly but it would be interesting to see if it matters.

[cintainfinita]

Moved everything to the run/opencl folder and made the file structure also:

% pwd
[REDACTED]/Tools/john/run/opencl
% ./john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Can't read source kernel: No such file or directory

That's the output.

What if you move all files in run/opencl to just run? That'd be ugly but it would be interesting to see if it matters.

Copying (not moving). And enable OpenCL verbose (the way it works in MacOS):

• To check if the Mac is compiling the kernel in some weird folder like /temp/cl88ddSdoui (it used to, right?)
• Well, hope we can see something new in the outputs.

[cintainfinita]

Hi @claudioandre-br and @magnumripper !

I don't really know how to enable verbosity for OpenCL in MacOS. I Googled it but didn't find anything useful. If you can help on that, I can try it.

I compiled a fresh copy and now instead of moving I copied the needed files to the run/opencl folder. Same error.

Running John binary from different locations:

moebius@kandinsky opencl % ../john --test --format=opencl
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Options used: -I opencl -cl-mad-enable -D__OS_X__ -D__GPU__ -DDEVICE_INFO=522 -D__SIZEOF_HOST_SIZE_T__=8 -DDEV_VER_MAJOR=1 -DDEV_VER_MINOR=2 -D_OPENCL_COMPILER -DHASH_LOOPS=1024 -DOUTLEN=20 -DPLAINTEXT_LENGTH=64 -DV_WIDTH=1 ../opencl/pbkdf1_hmac_sha1_kernel.cl
Build log: <program source>:47:10: fatal error: 'opencl_device_info.h' file not found
#include "opencl_device_info.h"
         ^

Error building kernel ../opencl/pbkdf1_hmac_sha1_kernel.cl. DEVICE_INFO=522
0: OpenCL CL_BUILD_PROGRAM_FAILURE (-11) error in opencl_common.c:1292 - clBuildProgram

moebius@kandinsky opencl % ./john --test --format=opencl 
Device 2: AMD Radeon Pro 5700 XT Compute Engine
Benchmarking: sha1crypt-opencl, (NetBSD) [PBKDF1-SHA1 OpenCL]... Can't read source kernel: No such file or directory

Thanks!

[magnumripper]

What if you move all files in run/opencl to just run? That'd be ugly but it would be interesting to see if it matters.

Copying (not moving). And enable OpenCL verbose (the way it works in MacOS):

Right, also move the headers to run, not the CPU binaries to opencl. From a normal tree it'd be like cd run; cp opencl/*.h .. However, it wont solve the below:

• To check if the Mac is compiling the kernel in some weird folder like /temp/cl88ddSdoui (it used to, right?)

Ah yes this sounds familiar. I fixed it at some point by starting to pass -I /full/path/to/john/opencl to the kernel runtime build instead of just eg. -I opencl but apparently that doesn't seem to kick in according to OP. Did we break it recently?

I wonder why I'm not seeing those problems at all? I run Big Sur on intel and M1 macbooks.

[magnumripper]

I also get just -I opencl but it does work for me.

[magnumripper]

Ah yes this sounds familiar. I fixed it at some point by starting to pass -I /full/path/to/john/opencl to the kernel runtime build instead of just eg. -I opencl but apparently that doesn't seem to kick in according to OP. Did we break it recently?

Looks like @solardiz broke it in 753dd9f.