Ensure only public API has default symbol visibility, hide private symbols

[rgommers]

This was attempted once in gh-3658 but abandoned. I think the right way of doing it is as pointed out in #3658 (comment), something like this in a header and then marking public functions with OPENBLAS_API:

#if defined (_WIN32) || defined (__CYGWIN__)
#  if defined (__GNUC__)
     /* GCC */
#    define OPENBLAS_EXPORT __attribute__ ((dllexport))
#    define OPENBLAS_IMPORT __attribute__ ((dllimport))
#  else
     /* MSVC */
#    define OPENBLAS_EXPORT __declspec(dllexport)
#    define OPENBLAS_IMPORT __declspec(dllimport)
#  endif
#else
   /* All other platforms. */
#  define OPENBLAS_EXPORT __attribute__ ((visibility ("default")))
#  define OPENBLAS_IMPORT
#endif

#if defined (OPENBLAS_SHLIB)
#  define OPENBLAS_API OPENBLAS_EXPORT 
#else
#  define OPENBLAS_API OPENBLAS_IMPORT
#endif

This resurfaced again in numpy/numpy#30299, specifically on Alpine (a Musl-based distro) because Musl libc has different behavior from glibc and doesn't support dlopen'ing a shared library with RTLD_LOCAL. This is what NumPy does for its vendored libscipy_openblas.so which hides the (non-suffixed) private symbols well enough on glibc-based distros.

The method above is the standard way of exporting a public API. I think we should try to implement that; it's a fair bit of plumbing but it should be doable and will prevent obscure issues (e.g., see MacPython/openblas-libs#79 for a previous one).

[mayeut]

If going this way, would it be worth to also include symbol prefix/suffix with such a macro ?
e.g. something like

#define OPENBLAS_API(return_type, base_name, args) OPENBLAS_EXPORT return_type prefix##base_name##suffix args

and a similar definition for internal symbols

#define OPENBLAS_IAPI(return_type, base_name, args) ...

[rgommers]

That's a great idea, thanks @mayeut. That may allow getting rid of the hacky objcopy business, and possibly make it easier to achieve other goals like switching from 64_ to the new Netlib convention (_64), or building a single shared library with both LP64 and ILP64 symbols.

I'm not sure internal symbols need the prefix/suffix mangling. Right now there are already unmangled internal symbols, and if their visibility is set to private it shouldn't matter anymore.