Parallel, Multi-Axis Regression and Performance Testing with FreeBSD, OpenZFS, and bhyve

Michael Dexter <editor@callfortesting.org>

Contemporary Unix, defined as the sum of open source BSD Unix projects, Illumos distributions and GNU/Linux distributions, plus the OpenZFS cross-platform file system, can attribute their success to the collaborative work of like-minded academic, commercial, and volunteer developers around the world. Governed by a mix of licenses, best practices, community norms, and personal passion, open source projects like modern Unix operating systems and OpenZFS largely lack centralized Quality Engineering institutions, deferring Quality Engineering and Quality Control responsibilities to participating developers and the end user. This arrangement promises the widest-possible array of regression and performance testing tools, loads, and procedures, at the expense of providing any guarantees, true to the disclaimers of the licenses under which these projects are distributed. This paper will examine how “parallel, multi-axis” testing, defined as testing multiple software versions, operating systems, “options”, compilers, and architectures, or axes, in parallel, will improve the identification and isolation of reliability and performance regressions.