NOTES.UNIX: expand the description of RPATHs

Hopefully, this will make it more clear that it isn't only ELF
specific, even though there is a part that is (or even more
restrictively GNU ld.so specific).

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/6520)

NOTES.UNIX

@@ -5,38 +5,87 @@
  For Unix/POSIX runtime systems on Windows, please see NOTES.WIN.
 
 
- Shared libraries and installation in non-standard locations
- -----------------------------------------------------------
+ OpenSSL uses the compiler to link programs and shared libraries
+ ---------------------------------------------------------------
 
- Binaries on Unix variants expect to find shared libraries in standard
- locations, such as /usr/lib, /usr/local/lib and some other locations
- configured in the system (for example /etc/ld.so.conf on some systems).
- If the libraries are installed in non-standard locations, binaries
- will not find them and therefore fail to run unless they get a bit of
- help from a defined RPATH or RUNPATH.  This can be applied by adding
- the appropriate linker flags to the configuration command, such as
- this (/usr/local/ssl was the default location for OpenSSL installation
- in versions before 1.1.0):
+ OpenSSL's generated Makefile uses the C compiler command line to
+ link programs, shared libraries and dynamically loadable shared
+ objects.  Because of this, any linking option that's given to the
+ configuration scripts MUST be in a form that the compiler can accept.
+ This varies between systems, where some have compilers that accept
+ linker flags directly, while others take them in '-Wl,' form.  You need
+ to read your compiler documentation to figure out what is acceptable,
+ and ld(1) to figure out what linker options are available.
+
+
+ Shared libraries and installation in non-default locations
+ ----------------------------------------------------------
+
+ Every Unix system has its own set of default locations for shared
+ libraries, such as /lib, /usr/lib or possibly /usr/local/lib.  If
+ libraries are installed in non-default locations, dynamically linked
+ binaries will not find them and therefore fail to run unless they get a
+ bit of help from a defined runtime shared library search path.
+
+ For OpenSSL's application (the 'openssl' command), our configuration
+ scripts do NOT generally set the runtime shared library search path for
+ you.  It's therefore advisable to set it explicitly when configuring
+ unless the libraries are to be installed in directories that you know
+ to be in the default list.
+
+ Runtime shared library search paths are specified with different
+ linking options depending on operating system and versions thereof, and
+ are talked about differently in their respective documentation;
+ variations of RPATH are the most usual (note: ELF systems have two such
+ tags, more on that below).
+
+ Possible options to set the runtime shared library search path include
+ the following:
+
+    -Wl,-rpath,/whatever/path
+    -R /whatever/path
+    -rpath /whatever/path
+
+ OpenSSL's configuration scripts recognise all these options and pass
+ them to the Makefile that they build.  (In fact, it recognises anything
+ starting with '-Wl,' as a linker option, so for example, HP-UX'
+ '-Wl,+b,/whatever/path' would be used correctly)
+
+ Please do not use verbatim directories in your runtime shared library
+ search path!  Some OpenSSL config targets add an extra directory level
+ for multilib installations.  To help with that, the produced Makefile
+ includes the variable LIBRPATH, which is a convenience variable to be
+ used with the runtime shared library search path options, as shown in
+ this example:
 
     $ ./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl \
-        -Wl,-rpath,/usr/local/ssl/lib
+        '-Wl,-rpath,$(LIBRPATH)'
 
- Because the actual library location may vary further (for example on
- multilib installations), there is a convenience variable in Makefile
- that holds the exact installation directory and that can be used like
- this:
+ On modern ELF based systems, there are two runtime search paths tags to
+ consider, DT_RPATH and DT_RUNPATH.  Shared objects are searched for in
+ this order:
+
+    1. Using directories specified in DT_RPATH, unless DT_RUNPATH is
+       also set.
+    2. Using the environment variable LD_LIBRARY_PATH
+    3. Using directories specified in DT_RUNPATH.
+    4. Using system shared object caches and default directories.
+
+ This means that the values in the environment variable LD_LIBRARY_PATH
+ won't matter if the library is found in the paths given by DT_RPATH
+ (and DT_RUNPATH isn't set).
+
+ Exactly which of DT_RPATH or DT_RUNPATH is set by default appears to
+ depend on the system.  For example, according to documentation,
+ DT_RPATH appears to be deprecated on Solaris in favor of DT_RUNPATH,
+ while on Debian GNU/Linux, either can be set, and DT_RPATH is the
+ default at the time of writing.
+
+ How to choose which runtime search path tag is to be set depends on
+ your system, please refer to ld(1) for the exact information on your
+ system.  As an example, the way to ensure the DT_RUNPATH is set on
+ Debian GNU/Linux systems rather than DT_RPATH is to tell the linker to
+ set new dtags, like this:
 
     $ ./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl \
-        -Wl,-rpath,'$(LIBRPATH)'
-
- On modern systems using GNU ld.so, a better choice may be to use the
- new dtags, like this:
-
-    $ ./config --prefix=/usr/local/ssl --openssldir=/usr/local/ssl \
-        -Wl,--enable-new-dtags,-rpath,'$(LIBRPATH)'
-
- This sets DT_RUNPATH instead of DT_RPATH.  DT_RUNPATH is considered after
- the environment variable LD_LIBRARY_PATH, while DT_RPATH is considered
- before that environment variable (which means that the values in that
- environment variable won't matter if the library is found in the
- paths given by DT_RPATH).
+        '-Wl,--enable-new-dtags,-rpath,$(LIBRPATH)'