The first thing we did was to write 240 questions, developed to capture the same dimensions of personality as a gold-standard instrument widely used in academic research - NEO-PI-R. The resulting questionnaire consisted of 5 factors and 30 facets (6 facets per factor). In a pilot validity study, we could show that the 30 facets had high correlations with corresponding facets in NEO-PI-R, which is evidence of construct validity.
Once we collected a sufficient sample (N>300), analyses of Alva's data along with a literature search indicated that a structure with 30 facets was suboptimal. Most of the variance in the items could be explained using 10 - 15 factors, according to exploratory factor analysis done by our people science team. Leading researchers in Personality Psychology have come to similar conclusions (Saucier, 1998; Chapman, 2007; DeYoung, 2007; Soto & John 2017 to name a few).
Looking at the correlation structure, many of the 30 facets correlated highly with each other (i.e. they had conceptual and empirical overlap) and some facets correlated higher with a different factor than the one they belonged to; for example Compliance belonged to the factor Agreeableness, while the questions correlated more with the factor Conscientiousness. This finding made us decide that we needed to simplify the structure of our personality test.
We then tried out three different alternatives for the structure of the test.
First, a model with 5 broad factors was developed. This is in line with previous research, showing that five factors explains personality well. However, we found the factors to be too blunt, not capturing unique contributions by more detailed aspects of each factor. For example, this broad model doesn't give you the possibility to divide a score on the extraversion factor into on the one hand the level of assertiveness and on the other, the level of sociability (which is a common distinction within the Extraversion factor). Also, using IRT methodology would lead to over-emphasizing a few questions with high discrimination (according to the observed data) while disregarding others, which limits the content validity (domain coverage) of the factor scores.
Second, a model with 10 'aspects' of personality (2 per factor), inspired by DeYoung (2007) was developed. This forced us to look closer at each question of the test and come up with a methodology for assigning questions within a given factor to one of two aspects. By applying factor analysis with varimax rotation and retain the top 2 components, then reviewing each question conceptually, and analyzing the item-factor correlations (both with the original factor and the other 4), a proposed mapping for questions into the 10 facet model was found. Applying IRT methodology showed better content validity (domain coverage), but at the cost of administering double the number of questions to attain comparable precision compared to the simpler 5 factor model.
Third, after further literature search and investigating scree-plots from the factor analysis closely, a model with 15 facets was developed. The reason was that we found additional variance not explained by only 2 aspects as in the 10-aspect model. This model was inspired by Soto & John (2017), Saucier (1998) and Chapman (2007). The same methodology was used as when creating the 10-aspect model but with 3 components instead of 2 per factor. The final structure has a lot of overlap with Soto's 15 facets.
Once the structure of the personality test was finalized, a sample of 148 participants from the US was used to calibrate the model with IPIP-NEO-120. An open dataset with 545,836 responses to the english version of IPIP-NEO-120 was used as reference.
Chapman, B. P., 2007. Bandwidth and Fidelity on the NEO-Five Factor Inventory: Replicability and Reliability of Saucier’s (1998) Item Cluster Subcomponents. Journal of Personality Assessment, 88:2, 220-234.
DeYoung, C. G., Quilty, L. C. & Peterson, J. B., 2007. Between Facets and Domains: 10 Aspects of the Big Five. Journal of Personality and Social Psychology, 93:5, 880-896.
Saucier, G., 1998. Replicable Item-Cluster Subcomponents in the NEO Five-Factor Inventory. Journal of Personality Assessment, 70:2, 263-276.
Soto, C. J. & John, O. P., 2017. The Next Big Five Inventory (BFI-2): Developing and Assessing a Hierarchical Model With 15 Facets to Enhance Bandwidth, Fidelity, and Predictive Power. Journal of Personality and Social Psychology, 113:1, 117-143.