Add proper "make install" and clean up "--prefix" / "--with-systemwide" handling

[solardiz]

We do have a make install, but it currently only (opportunistically) installs bash and zsh completions, and into directories unaffected by --prefix. The rest of its logic is somehow commented out:

install: strip shell-completion
#ifneq ($(prefix), ../run)
#       @echo Copying binaries to $(prefix)
#       $(MKDIR) $(prefix)
#       $(CP) -p -P -R ../run/*.{py,pl,rb} $(PROJ) $(prefix)
#       @echo Copying data and config files to $(prefix)
#       $(MKDIR) $(prefix)
#       $(CP) -p -P -R ../run/*.{chr,conf,lst} $(prefix)
#endif

Other than that, we already handle --prefix mostly similarly to how other typical autoconf'ed software does it:

dnl -DJOHN_SYSTEMWIDE
AC_ARG_WITH(systemwide, [AS_HELP_STRING([--with-systemwide],[install for all users])],,[with_systemwide=no])

AC_PREFIX_DEFAULT([../run])

dnl Handle --prefix -> JOHN_SYSTEMWIDE
dnl A prefix of /usr/local will mean execs in /usr/local/bin and the rest
dnl in /usr/local/share/john
AS_IF([test $prefix != "../run" && test $prefix != "NONE" ],
   [CFLAGS="$CFLAGS -DJOHN_SYSTEMWIDE -DJOHN_SYSTEMWIDE_EXEC='\"$prefix/bin\"' -DJOHN_SYSTEMWIDE_HOME='\"$prefix/share/john\"'"],
   [AS_IF([test $with_systemwide != no], [CFLAGS="$CFLAGS -DJOHN_SYSTEMWIDE"
   prefix=/usr/libexec/john])]
)

I think we should rework the logic around --with-systemwide, though. Do we know of existing users of the present logic?

[solardiz]

To answer my own question, --with-systemwide is used at least in @claudioandre-br's packaging/build.sh.

[solardiz]

I've just tried uncommenting all the lines in install: and running make install. Here's what parts of run it misses:

$ diff -ur ~/johninst/ run
Only in run: benchmark-unify
Only in run: bip-0039
Only in run: dictionary.rfc2865
Only in run: dns
Only in run: fuzz.dic
Only in run: john.bash_completion
Only in run: john.zsh_completion
Only in run: jtr_rulez.pm
Only in run: jtrconf.pm
Only in run: lib
Only in run: mailer
Only in run: makechr
Only in run: mongodb2john.js
Only in run: network2john.lua
Only in run: opencl
Only in run: oui.txt
Only in run: relbench
Only in run: rules
Only in run: stats
Only in run: ztex

I think most or all of these should actually be installed.

There's also no separation into bin and share that the comment in configure.ac announces and the build is in fact configured to expect:

$ ~/johninst/john -test
fopen: /home/solar/johninst/share/john/john.conf: No such file or directory

We'll also need to decide on what to install to where. In my own packaging of core for Owl, I have this:

$ rpm -ql john
/usr/bin/john
/usr/bin/unafs
/usr/bin/unique
/usr/bin/unshadow
/usr/libexec/john
/usr/libexec/john/john-avx
[...]
/usr/libexec/john/john-xop
/usr/share/doc/john-1.9.0
/usr/share/doc/john-1.9.0/CHANGES
[...]
/usr/share/doc/john-1.9.0/RULES
/usr/share/doc/john-1.9.0/mailer
/usr/share/doc/john-1.9.0/makechr
/usr/share/doc/john-1.9.0/relbench
/usr/share/john
/usr/share/john/ascii.chr
/usr/share/john/digits.chr
/usr/share/john/john.conf
/usr/share/john/lm_ascii.chr
/usr/share/john/password.lst

Notice how I treat mailer, makechr, and relbench as "documentation" - I didn't want to clutter /usr/bin with those. Jumbo adds many more files, for some of which it'd be tough to decide where they should go.

We also have so many *2john tools that they'd clutter whatever shared bin directory they might be installed into, but they're actually important for common uses of JtR, not just as "documentation". I don't know how to approach this best. We might want to check how existing distros' packages of JtR handle this.

Worse, some of our scripts depend on other files and directories found in run: bip-0039, dictionary.rfc2865, dns, lib. Would we be embedding paths to a share/john directory into them when we configure with a --prefix? We certainly don't want to be modifying git-managed files in-place. So would we be creating modified copies and using those during make install?

[solardiz]

FWIW, I've just checked how hashcat handles this.

They don't use argv[0] for this purpose and don't appear to hard-code any full pathnames at build time (the two approaches that JtR can currently use), but instead have system-specific code to infer the program binary's directory on Linux and Cygwin (surprisingly, the same code works!), macOS, and FreeBSD. On non-Cygwin Windows builds, they just assume invocation from the current directory (empty path). Other OSes are unsupported (will error out during build).

They do have make install and a few ways for packages to override things:

DESTDIR                 ?=
PREFIX                  ?= /usr/local

INSTALL_FOLDER          ?= $(PREFIX)/bin
SHARED_ROOT_FOLDER      ?= $(PREFIX)/share
SHARED_FOLDER           ?= $(SHARED_ROOT_FOLDER)/hashcat
DOCUMENT_FOLDER         ?= $(SHARED_ROOT_FOLDER)/doc/hashcat
LIBRARY_FOLDER          ?= $(PREFIX)/lib
LIBRARY_DEV_ROOT_FOLDER ?= $(PREFIX)/include
LIBRARY_DEV_FOLDER      ?= $(LIBRARY_DEV_ROOT_FOLDER)/hashcat

Here's what make install appears to do:

$ make install PREFIX=/home/solar/hashinst
install -m 755 -d                                                    /home/solar/hashinst/share
install -m 755 -d                                                    /home/solar/hashinst/bin
install -m 755 hashcat                                   /home/solar/hashinst/bin/
install -m 755 -d                                                    /home/solar/hashinst/share/doc/hashcat
install -m 755 -d                                                    /home/solar/hashinst/share/hashcat
install -m 755 -d                                                    /home/solar/hashinst/share/doc/hashcat/charsets
install -m 644 hashcat.hctune                                        /home/solar/hashinst/share/hashcat/
install -m 755 -d                                                    /home/solar/hashinst/share/doc/hashcat/docs
install -m 755 -d                                                    /home/solar/hashinst/share/doc/hashcat/layouts
install -m 755 -d                                                    /home/solar/hashinst/share/doc/hashcat/masks
install -m 755 -d                                                    /home/solar/hashinst/share/doc/hashcat/rules
install -m 755 -d                                                    /home/solar/hashinst/share/doc/hashcat/extra
install -m 755 -d                                                    /home/solar/hashinst/share/doc/hashcat/extra/tab_completion
install -m 644 example.dict                                          /home/solar/hashinst/share/doc/hashcat/
install -m 644 hashcat.hcstat2                                       /home/solar/hashinst/share/hashcat/
install -m 644 example0.hash                                         /home/solar/hashinst/share/doc/hashcat/
install -m 755 -d                                                    /home/solar/hashinst/share/hashcat/OpenCL
install -m 755 -d                                                    /home/solar/hashinst/share/hashcat/modules
find OpenCL/   -mindepth 1 -type d -execdir install -m 755 -d     /home/solar/hashinst/share/hashcat/OpenCL/{} \;
find modules/  -mindepth 1 -type d -execdir install -m 755 -d     /home/solar/hashinst/share/hashcat/modules/{} \;
find modules/  -mindepth 1 -type f -execdir install -m 644 {}     /home/solar/hashinst/share/hashcat/modules/{} \;
find OpenCL/   -mindepth 1 -type f -execdir install -m 644 {}     /home/solar/hashinst/share/hashcat/OpenCL/{} \;
install -m 644 example400.hash                                       /home/solar/hashinst/share/doc/hashcat/
install -m 644 example500.hash                                       /home/solar/hashinst/share/doc/hashcat/
install -m 755 example0.sh                                           /home/solar/hashinst/share/doc/hashcat/
install -m 755 example400.sh                                         /home/solar/hashinst/share/doc/hashcat/
install -m 755 example500.sh                                         /home/solar/hashinst/share/doc/hashcat/
install -m 644 extra/tab_completion/hashcat.sh                       /home/solar/hashinst/share/doc/hashcat/extra/tab_completion/
install -m 644 extra/tab_completion/howto.txt                        /home/solar/hashinst/share/doc/hashcat/extra/tab_completion/
install -m 755 extra/tab_completion/install                          /home/solar/hashinst/share/doc/hashcat/extra/tab_completion/
find charsets/ -type d -exec install -m 755 -d                    /home/solar/hashinst/share/doc/hashcat/{} \;
find charsets/ -type f -exec install -m 644 {}                    /home/solar/hashinst/share/doc/hashcat/{} \;
find docs/     -type d -exec install -m 755 -d                    /home/solar/hashinst/share/doc/hashcat/{} \;
find docs/     -type f -exec install -m 644 {}                    /home/solar/hashinst/share/doc/hashcat/{} \;
find layouts/  -type d -exec install -m 755 -d                    /home/solar/hashinst/share/doc/hashcat/{} \;
find layouts/  -type f -exec install -m 644 {}                    /home/solar/hashinst/share/doc/hashcat/{} \;
find masks/    -type d -exec install -m 755 -d                    /home/solar/hashinst/share/doc/hashcat/{} \;
find masks/    -type f -exec install -m 644 {}                    /home/solar/hashinst/share/doc/hashcat/{} \;
find rules/    -type d -exec install -m 755 -d                    /home/solar/hashinst/share/doc/hashcat/{} \;
find rules/    -type f -exec install -m 644 {}                    /home/solar/hashinst/share/doc/hashcat/{} \;
sed -i 's/\.\/hashcat/hashcat/'                         /home/solar/hashinst/share/doc/hashcat/example0.sh
sed -i 's/\.\/hashcat/hashcat/'                         /home/solar/hashinst/share/doc/hashcat/example400.sh
sed -i 's/\.\/hashcat/hashcat/'                         /home/solar/hashinst/share/doc/hashcat/example500.sh

This made me wonder how the installed hashcat binary would find its OpenCL kernels, etc. given that they're no longer in the same directory relative to the binary after such installation. And for me it does not:

$ ~/hashinst/bin/hashcat -b -d1 -m0 -O -w3
hashcat (v5.1.0-1804-g9b64a40+) starting in benchmark mode...

/home/solar/hashinst/bin/OpenCL/: No such file or directory

I wonder if there's some extra trick packagers are supposed to use here. Searching docs for the words/substrings "install", "package", "packaging" doesn't give any hint.

[solardiz]

I found that our README-DISTROS currently has this:

	make -s clean && make -sj4 strip &&
	rm -rf ../run/*.dSYM &&
	sudo mv ../run/{john,john-*} /usr/local/bin &&
	sudo mkdir -p /usr/local/share/john &&
	sudo cp -a ../run/* /usr/local/share/john &&
	sudo mv /usr/local/share/john/*.{pl,py,rb} /usr/local/share/john/{relbench,benchmark-unify,mailer,makechr} /usr/local/bin &&
	echo All Done

And also this:

PLEASE NOTE: You should definitely consider:

	sudo make shell-completion

Or something to that end - depending on what your tree looks like you might
simply want to symlink /usr/local/share/john/john.*_completion into
/etc/bash_completion.d instead.

So we already have almost a make install intended for distros in there, just not in that form, and not fully working right (it doesn't appear to install the un* and *2john tools that aren't scripts, but are symlinks to john or separate compiled programs).

I find it weird for those instructions to be for distros yet install under /usr/local, which is normally reserved for manual (not packaged) local installs. And overriding that to use /usr instead would result in a mess in system directories. For comparison, I just took a look at a package of ghostscript, which also includes many conversion tools - it does look somewhat messy, but not as much as installing all of jumbo's programs and scripts under /usr/bin would.

I currently think that we should start by having make install use /opt/john or /opt/openwall/john by default. Then we can keep a run directory in there, including its opencl and lib subdirectories, and also have doc nearby (and later also man). The /opt/openwall/john default would make most sense if we also have /opt/openwall/johnny, /opt/openwall/php_mt_seed, etc. for other related projects.

Trying to make everything install in a way suitable for /usr feels like fighting our own watermill - would be lots of differences between non-installed usage from run like we support now and this new kind of systemwide installs. I don't want to drop support for the current non-installed usage - in fact, that's how I intend to continue using JtR most of the time - so support for those differences would need to be maintained in the tree. I'd rather avoid that.

[solardiz]

README-DISTROS [...] not fully working right (it doesn't appear to install the un* and *2john tools that aren't scripts

I was wrong. There's this right after the SSE2 OpenMP build:

        sudo mv ../run/{john-*,*2john,unshadow,unique,undrop,unafs,base64conv,tgtsnarf,mkvcalcproba,genmkvpwd,calc_stat,raw2dyna,cprepair,SIPdump} /usr/local/bin &&

This line was easy to miss since it's between different builds, but having it in that place actually makes sense - to avoid having those binaries depend on a higher SIMD instruction set.

[anthraxx]

Hey, appreciate you thinking about this loudly by dropping everything here. However, as you seem to tend to wards it being not worth to have a distro install target on top of the regular "local checkout runtime" I would like to provide a tiny different point of view. While it's obviously true that work is involved maintaining that additional official installation way, it also brings several advantages:
One of them is that you can expect a more or less uniform way of distributing john across various distros. This includes additions to extra scripts or data that otherwise some distros may miss in a new release. Furthermore you would potentially also get less downstream packaging related issues to analyze and deal with on GitHub. If the additional installation way is implemented in a very flexible way including comprehensive make variables for the layout (like DATADIR MANDIR PREFIX LIBDIR etc etc) would give distros enough options to customize their local differences where files are placed. I think that downstream consumers will happily contribute patches with more flexible options themselves to suite their needs and you won't have to deal with every special case of distros yourself (besides reviewing patches of cause).

I believe in Arch Linux we have a very easy to understand package source format without lot of domain knowledge like macros and structure, so if you would like to take a peek and maybe grab some of this into the new installation routine, I'm happily inviting you: https://github.com/archlinux/svntogit-community/blob/packages/john/trunk/PKGBUILD

[xambroz]

Hello,
if I may add my 2 copper coins from the position of the Fedora packager of john - what would make my life easier as packager

1) Discussion on folders

For the purpose of "make install" I would recommend to have at least these definitions (taken from autoconf/rpm macros) on the Makefile level and respective #define clauses on the level of code. This would allow us packagers to move the stuff around easily :

DESTDIR                 ?= /
PREFIX                  ?= /usr/local

BINDIR                  ?= $(PREFIX)/bin
LIBEXECDIR              ?= $(PREFIX)/libexec
SYSCONFIGDIR          ?= $(PREFIX)/etc

DATADIR                     ?= $(PREFIX)/share
MANDIR                      ?= $(DATADIR)/man

# In future possibly also 
LIBDIR                         ?= $(PREFIX)/lib
INCLUDEDIR              ?= $(PREFIX)/include

# Maybe
LOCALSTATEDIR          ?= $(PREFIX)/var

# optional
DOCDIR                       

DESTDIR ... this is helper for distro packagers, imagine that like a disk image root or chroot environment to which stuff gets copied from the source folder after build during make install.
Common habit is to clean the folder before doing "make install" to be sure there is just the stuff from the package being build and nothing else. Please note that DESTDIR is conceptually different from the PREFIX, as you might want some stuff be outside of the PREFIX (typically configuration, in case of john /etc/john/john.conf ...)

PREFIX ... is directory which is common to all or majority of the directories in the project where the stuff gets installed within current root. It usually is / (system stuff) or /usr (normal applications - that is value usually used by packagers) or /usr/local (locally compiled applications - that is usually the value used by upstream).

BINDIR ... directory where all executables are to be installed, john itself, all the helpers like zip2john - usually /usr/bin for the system package, /usr/local/bin for the upstream local build

LIBEXEC ... directory for optional executed binaries (usually /usr/libexec). All john-CPU-FEATURE binaries should go to $(LIBEXECDIR)/john/john-CPU-FEATURE.

Personally I would vote to put all the john-CPU-FEATURE binaries to $(LIBEXECDIR)/john/ and have only a symbolic link in $(BINDIR)/john pointing to the best available feature binary.

SYSCONFIGDIR ... the system-wide configuration. Usually /etc/ , could be /usr/local/etc for locally compiled upstream package. All the configuration *.conf files should go to $(SYSCONFIGDIR)/john/.
In past there was /etc/john.conf, but as configs are now modular with the jumbo, I would recommend to create extra dedicated directory.

DATADIR ... the architecture independent data like charmaps, wordlists

SYSCONFIGDIR usually contains configuration, which might change for the specific machine, while DATADIR usually contains configuration or other architecture independent data, which should be considered read-only and should stay unmodified as comming from the distribution or upstream package.

MANDIR ... manpage for john should go to $(MANDIR)/man1/john.1

DOCDIR ... you might or might not bother with these. Here comes all the documentation , readmes, example codes for helpers, which are not necessarily in working productive quality, but still might be usefull to someone.
Most projects do not install documentation an keep it only in the local doc directory (possibly built with some document processing like sphinx).

So put together when package is copying stuff around during make install it looks something like this:

        $(INSTALL)  -d $(DESTDIR)$(LIBEXECDIR)/john
        $(INSTALL) -p -m 755 run/john-sse2  $(DESTDIR)$(LIBEXECDIR)/john/john-sse2

LOCALSTATE ... possibly you might thing of using $(LOCALSTATE)/log/john/ for logs and $(LOCALSTATE)/cache/john/ for pot files. Question is whether it is good idea to thing of storing the cracked passwords and log about the same in some generic /var/log/john/john.log - probably isn't. As these must be readable/writable only to root, personally I would say the current mode of creating those only under the current user's directory is OK.

LIBDIR ... in case you would be planning some plugin libraries for modules implementing particular password hashes
INCLUDE ... for distributing john header files ... for building modules.

**2) Current Gaps **

• GOOD - JOHN_SYSTEMWIDE_EXEC (executables) is already separated from JOHN_SYSTEMWIDE_HOME (architecture independent inputs)

• end-users - end users need possibility to change the configuration without necessarily going root and changing the systemwide configuration. Some of that could be done on the level of having user's "/.john/john.conf" or possibly "/.john/john-local.conf" (if added to the default configuration user level john-local.conf #5232)

• single directory - john is referencing majority of the stuff by $JOHN looking in the same directory for configuration, wordlists, charsets, opencl. I would recommend to separate from there the configuration.
  For example the systemwide john-local.conf is being searched in "$JOHN/john.local.conf" = "/usr/share/john/john-local.conf", while it should be /etc/john/john.local.conf, because it is file which is supposed to change locally, not delivered by upstream or distribution package.

There might be more places in the code like that.

Best regards
Michal Ambroz