Non FIPS algorithms are not normally allowed in FIPS mode.
Any attempt to use them via high level functions will return an error.
The low level non-FIPS algorithm functions cannot return errors so they
produce assertion failures. HMAC also has to give an assertion error because
it (erroneously) can't return an error either.
There are exceptions (such as MD5 in TLS and non cryptographic use of
algorithms) and applications can override the blocking and use non FIPS
algorithms anyway.
For low level functions the override is perfomed by prefixing the algorithm
initalization function with "private_" for example private_MD5_Init().
For high level functions an override is performed by setting a flag in
the context.