This gist contains my notes about how the FIPS selftest signature check works in OpenSSL 1.x. Assumes basic awareness of the OpenSSL FOM (FIPS Object Module). My focus for this case is Solaris on SPARC.
The goal is to examine some of the inner workings of FIPS signature verification.
The FOM is built as fipscanister.o using designated code and linked into common OpenSSL build.
When the libcrypto.so library is loaded, the FOM will perform self test. If the self test fails,
the running program is abort()ed.