Effectivity of Selection Methods as Predictor of Job Performance

Most of the articles I've read that compare selection methods as predictor of job performance eventually (because some articles cite articles that then cite the study) cites this study. Looking it up in Google Scholar reveals that it has been cited by over 4500 studies (note not merely articles but studies published in scientific journals). This tells me that the study is credible.

The study itself is a meta-analysis of research studies conducted in the past 85 years (until 1998). The study and its findings is not without its critics. For instance, this study questions the validity of its conclusions given the uncertain quality of the original data of some studies included in the meta-analysis. However, I find that the criticisms (of that particular study) apply equally to the different selection methods. So, as far as the organizations' practical problem of choosing selection methods is concerned, I believe the conclusions in this (and a recent version) are still the best we've got (in terms of being backed by empirical data and scientific rigor.) 

A working paper that updates the study was made available in 2016. It expands the meta-analysis to cover 100 years (until 2016) of research findings. The main findings and their practical implications remain the same but some findings differ.

Both the original and the updated paper found tests of general mental ability (GMA) to be a better predictor of job performance than either structured and unstructured interviews. In fact, in the updated paper it was found to be the single most effective predictor of job performance (in the original it was a close second to work sample tests). Moreover, in both paper, GMA was also found to be the single most effective predictor of job-related learning (i.e. amount learned on-the-job and in trainings).

Both papers also found that when combined with GMA, integrity tests yielded the highest incremental improvement over GMA alone as predictor of job performance followed closely by structured interviews.

My take on these is pretty clear. Let's do GMA tests. Doing away with interviews altogether is not unjustified. If we must do interviews, let's do structured interviews. The rest of the selection methods are either too costly (in terms of effort and resources required like work samples) or just really poor predictors (reference checks, years of job experience, person-organization fit, personality tests).

Software Development as an Empirical Process

"If a process can be fully defined, with all things known about it so that it can be designed and run repeatably with predictable results, it is known as a defined process, and it can be subjected to automation. If all things about a process aren't fully known -- only what generally happens when you mix these inputs and what to measure and control to get the desired output -- these are called empirical processes."

"A defined process is predictable; it performs the same every time. An empirical process requires close watching and control, with frequent intervention. It is chaotic and unrepeatable, requiring constant measurement to ensure the desired result."

"Models of empirical processes are derived by categorizing observed inputs and outputs and defining the controls that cause them to occur within prescribed bounds. Empirical process modeling involves constructing a process model strictly from experimentally obtained input/output data, with no recourse to any laws concerning the fundamental nature and properties of the system. No a priori knowledge about the process is necessary; a process is treated like a black box."

"[S]cientists inspected the systems development process. They concluded that many of the processes, rather than being repeatable, defined, and predictable, were unpredictable and unrepeatable. With that, the scientists explained the difference between defined processes and empirical processes."

"The scientists stated, 'We are most amazed that your industry treats these illformed processes as defined, and performs them without controls despite their irregular nature. If chemical processes that we don't understand completely were handled in the same way, we would get very unpredictable results.'"

"The scientists recommended -- since our business is an empirical process -- that we use measurements and controls, as done elsewhere in the physical world. The scientists provided the concepts of defined and empirical processes .. and they told us that we needed controls to manage the empirical systems development process."

Controlled-Chaos Software Development. 
Retrieved on June 9, 2018 from
http://static1.1.sqspcdn.com/static/f/447037/6486035/1270926469320/Controlled-Chaos+Software+Development.pdf?token=NdnSJ23pP9yBLAfDSHQHZqIOc2U%3D