This was disabled earlier as part of diagnosing failing tests on ARM, which turned out to be due to aliasing, fixed by adding -fno-strict-aliasing. So, re-enabling it back.
This cleans up the CMake code and shows patterns more easily (to be refactored in the next commit).