Treffer: A Survey of Challenges Experienced by New Learners Coding the Rorschach.

Learning to code the imagery, communication, and behavior associated with Rorschach responding is challenging. Although there is some survey research on graduate students' impressions of their Rorschach training, research has not identified which coding decisions students find to be the most problematic and time-consuming. We surveyed students to identify what they struggled with most when learning coding and to quantify how difficult it is to learn how to code. Participants (n = 191) from the United States, Brazil, Denmark, Israel, and Italy rated 57 aspects of coding using a 4-point scale that encompassed both the time required to code and the subjective difficulty of doing so. Mean ratings for coding in general indicated that students considered the overall task challenging. Ratings also revealed that students struggled most with Cognitive Special Scores, Determinants, and extrapolating from the tables to code Form Quality for objects that were not specifically listed. The findings offer suggestions about how to improve the guidelines for some of the more difficult variables and where it is most necessary to focus teaching time. Taking these steps might help new students in learning the Rorschach.

AN0122315829;4n301may.17;2019Feb28.13:25;v2.2.500

A Survey of Challenges Experienced by New Learners Coding the Rorschach. 

Learning to code the imagery, communication, and behavior associated with Rorschach responding is challenging. Although there is some survey research on graduate students' impressions of their Rorschach training, research has not identified which coding decisions students find to be the most problematic and time-consuming. We surveyed students to identify what they struggled with most when learning coding and to quantify how difficult it is to learn how to code. Participants (n = 191) from the United States, Brazil, Denmark, Israel, and Italy rated 57 aspects of coding using a 4-point scale that encompassed both the time required to code and the subjective difficulty of doing so. Mean ratings for coding in general indicated that students considered the overall task challenging. Ratings also revealed that students struggled most with Cognitive Special Scores, Determinants, and extrapolating from the tables to code Form Quality for objects that were not specifically listed. The findings offer suggestions about how to improve the guidelines for some of the more difficult variables and where it is most necessary to focus teaching time. Taking these steps might help new students in learning the Rorschach.

In 1974, John Exner published the first edition of the Comprehensive System (CS), with the goal of integrating the best features of the five previous systems that had been commonly used in the United States (Beck, Klopfer, Piotrowski, Hertz, and Rapaport). Based on the research available at the time and his own investigations, Exner ([4]) selected for the CS the most reliable and valid components of these systems. The CS provided a systematic approach to administration and coding, a format and procedure for calculating interpreted variables, and normative samples that grew to encompass both children and adults (Exner, [7]). Eventually, the CS became the dominant system taught in graduate training (Hilsenroth & Handler, 1995; Mihura & Weinle, [24]; Ritzler & Alter, 1986).

Although the CS is no longer evolving as a result of Exner's death in 2006, the Rorschach Performance Assessment System (R–PAS; Meyer, Viglione, Mihura, Erard, & Erdberg, [19]) was developed as a replacement for it. Four of the five R–PAS authors worked with Exner on his Rorschach Research Council, which met semiannually from 1997 through 2005 to review and complete research that would advance the CS. Although Exner planned that the Research Council would take over CS developments (Exner, [5]), no formal mechanism was in place to do so when he passed away. Nonetheless, R–PAS extends the work begun by the Research Council and aims to improve the applied use of the Rorschach by, among other things, reducing examiner variability (Meyer et al., [19]), optimizing the number of responses people give to the task (Pianowski, Meyer, & Villemor-Amaral, [25]; Viglione et al., [32]), reanchoring normative expectations to correct overpathologizing biases (Meyer, Erdberg, & Shaffer, [15]; Meyer, Shaffer, Erdberg, & Horn, [18]), ensuring interpretation is in line with each variable's validity evidence base (Meyer, Hsiao, Viglione, Mihura, & Abraham, [17]; Mihura, Meyer, Bombel, & Dumitrascu, [20]; Mihura, Meyer, Dumitrascu, & Bombel, [21], 2016), and making interpretation easier (Meyer & Eblin, [14]; Meyer et al., [19]). Although these changes are important, it is also the case that most of the variables coded in R–PAS are the same as variables that were coded in the CS.

According to recent survey data collected from accredited U.S. doctoral training programs in the fall of 2015 (Mihura, Roy, & Graceffo, [23]), the Rorschach is being taught in 63% of all programs, with the CS being taught in 53% and R–PAS being taught in 37%. Of the programs teaching the Rorschach, 85% cover the CS and 60% cover R–PAS. Thus, both systems are currently in active use in the United States. Although international data comparing CS to R–PAS instruction are not available, both systems are used internationally and have been translated into other languages.

Unlike self-report measures, the Rorschach requires extensive study and supervised practice to become proficient with its administration and scoring (Gacono, Evans, & Viglione, [8]; Meyer et al., [19]). Research has demonstrated that well-trained raters can code CS and R–PAS variables with good to excellent reliability (Kivisalu, Lewey, Shaffer, & Canfield, [12]; Meyer, [13]; Meyer et al., [16]; Meyer et al., [19]; Viglione, Blume-Marcovici, Miller, Giromini, & Meyer, [30]; Viglione & Meyer, [31]), and that coding reliability is very similar across different languages and cultures (Meyer et al., [15]). However, Viglione and Meyer ([31]) summarized some CS codes from multiple studies that revealed lower (but still acceptable) reliabilities, indicating that they are more difficult to code accurately. These codes concern vague Developmental Quality (DQv and DQv/+), the Form Dominance of color and shading variables (FC vs. CF vs. C and Form Shading vs. Shading Form vs. Shading), Form Quality (FQu and FQ+), certain Contents (Art, Ay, Sc, Bt vs. Na vs. Ls, Id), and Special Scores (DV vs. INC, ALOG, CONTAM vs. INC, PER vs. DR, Level 1 vs. Level 2). There are fewer data available concerning R–PAS codes, although similar instances of lower reliability have appeared for at least some of the same variables when coded in R–PAS (e.g., Vagueness, FQu%, Cognitive Codes; see Kivisalu et al., [12]; Viglione et al., [30]).

Several CS studies have investigated coding accuracy and the interrater reliability of coding categories[1] among students and new learners. Hilsenroth, Charnas, Zodan, and Streiner ([10]) examined coding accuracy among 29 graduate students enrolled in a clinical PhD program approved by the American Psychological Association. The authors found an agreement of 80% or more with most of the coding categories (i.e., Location, Developmental Quality, Form Quality, Pair, Content, Popular) but lower rates of agreement for Determinants (78%) and Special Scores (65%). Similarly, estimated kappa was less than.74 for Determinants, Form Quality, Z scores, and Special Scores. The latter coding category was the only category that showed an estimated kappa in the fair to good range of reliability (estimated κ =.56); all other estimated kappa values were higher.

Callahan ([1]) evaluated coding accuracy of CS protocols through a three-stage training experience, followed by an 8-week follow-up. The accuracy of coding all the Rorschach response segments improved over time, although the proportions of agreement with the expert scoring were generally lower for FQ (68.9%) and Special Scores (65.3%) at the 8-week follow-up protocol. Guarnaccia, Dill, Sabatino, and Southwick ([9]) investigated the association of training and experience with coding accuracy using a small sample of responses. Twenty-one second-level graduate students and 12 licensed psychologists coded 10 responses from clinical protocols and 10 responses from nonclinical protocols. The results showed significant but somewhat inconsistent differences in scoring accuracy. Students were more accurate for Contents (nonclinical responses) and DQ (clinical responses), whereas professionals were more accurate for FQ, Special Scores (nonclinical responses), and Contents (clinical responses). Although Popular and Pairs achieved scoring accuracy above 80% for both students and professionals, FQ and Special Scores were more difficult to score correctly for all participants.

With respect to R–PAS coding, the effects of training have not been studied extensively. However, Meyer et al. ([19]) examined interrater reliability for six codes that were new to R–PAS relative to the CS (Space Reversal, Space Integration, Aggressive Content, Oral Dependency Language, Mutuality of Autonomy Health, and Mutuality of Autonomy Pathology). Six coders each independently coded a set of 50 protocols from the R–PAS normative sample. The coders varied in their previous experience coding Rorschach protocols, ranging from being highly experienced to having coded just one protocol before the study began. However, all coders were applying the draft R–PAS coding guidelines for the first time (and the final guidelines were improved and clarified by the coding challenges they encountered). Across the six codes, the average of the pairwise reliability coefficients was interclass correlation coefficient (ICC) =.81. However, for the three most experienced coders the average ICC was notably higher at.87. The other studies systematically examining R–PAS coding reliability have relied on doctoral students as the coders (Kivisalu et al., [12]; Viglione et al., [30]) but have not compared students to more experienced coders.

Taken together, these studies suggest that students and new learners show lower reliability in general than more senior coders, as well as lower accuracy with FQ, Determinants, and Special Scores, but they do not provide definitive information regarding the minimum amount of training or experience required to code reliably. Moreover, these data suggest that there are complexities in the coding process that might require further investigation, that there are coding guidelines that would benefit from further specification, and that certain coding decisions might warrant more training time.

Difficulties in coding the Rorschach accurately might be due to unclear definitions of codes that are not fully specified in the standard CS training materials (Exner, [6], [7]). The brevity of these materials prompted Viglione ([28], [29]) to write Rorschach Coding Solutions, which is a detailed text for coding according to CS guidelines, and similar levels of detailed guidance were incorporated into the R–PAS manual (Meyer et al., [19]). Personal experience from the first author's more than 30 years in teaching CS coding reveals that many new students struggle with confusing abstract rules and exceptions to those rules when learning to code. Possible perceptions that the guidelines are arbitrary, insufficient, or less than helpful, and too difficult or too time-consuming all can undermine learning and prevent students from sustaining sufficient effort to learn how to code accurately. Thus, new learners who approach the Rorschach for the first time and graduate students who are still in training could offer valuable input on the specific scoring difficulties they encounter that might undermine their confidence of becoming a reliable coder.

This study is the first attempt to investigate student perspectives about difficulties they encounter in learning Rorschach coding. Data collection began in 2007 (Ptucha, Viglione, & Meyer, [31]), at a time long before R–PAS was introduced (Meyer et al., [19]) when a subgroup of the R–PAS authors was considering ways to make changes to the CS. Given this, CS variables were the sole focus of the investigation then, and they are reported in this study. Only later, well after this research was initiated, did it become clear that revisions to the CS would be impossible, which ultimately led to the R–PAS being created. Some of the findings from this study ultimately contributed to decisions that were made when creating R–PAS, most notably by dropping some coding categories and distinctions and by providing more elaborated coding instructions akin to those found in Rorschach Coding Solutions (Viglione, [29]). However, a systematic examination of the survey results has never been published. The findings to be reported have clear relevance to training programs that continue to teach the CS, of which there still are many. In addition, to the extent that survey results identify coding challenges that are intrinsically present when coding Rorschach-based perceptions, communications, and behavior despite the elaborated guidelines that are available for the CS (Viglione, [29]) and R–PAS (Meyer et al., [19]), the results will apply to the training of both CS and R–PAS students.

Nonetheless, the purpose of our survey was to discover what students struggle with the most when learning to code the Rorschach according to the CS. Answers to these concrete questions might inform more abstract concerns pertaining to the accessibility of the test for new learners and the practical barriers for coding reliably. Moreover, we also investigated whether more experience was associated with less coding difficulty. Findings might help to identify codes that require more instructional time, more detailed guidelines, and more practice calibrating to standards to achieve mastery. Ultimately, such information could conceivably increase the number of students who become proficient and use the task, as well as increase research on the Rorschach.

Method

New Learner Survey

The New Learner Survey is a 57-item measure that was developed for this study. The coding challenges selected for the survey were largely derived from Viglione's ([28], [29]) CS coding text. The topics addressed in that text were selected by tracking coding inconsistencies among multiple coders examining the same responses and by identifying common coding errors made by students in training. Most of the survey items are oriented toward common coding distinctions one must make (e.g., FT vs. TF vs. T), as opposed to rating the presence or absence of individual codes (e.g., T vs. No T). The surveys were self-administered, and items were listed in the same order as they are encountered when coding a response using the CS, starting with Location and Developmental Quality, then moving on to Determinants, Form Quality, Pairs, Contents, Popular, and Special Scores. However, the survey began with a single item asking about difficulty learning coding for the Rorschach as a whole. Given that our aim was to investigate what codes new learners struggle with the most, we asked raters to evaluate their experience subjectively through introspection. For example, one item asked about experienced difficulty coding Location in general, whereas other items asked about decisions between W, D, and Dd. Students were asked to rate each item on a 4-point scale of difficulty: 1 (simple, straightforward, and very easy to score; takes very little time to score), 2 (understandable and relatively easy to score but does take some work at times; takes a little time and effort to score), 3 (sometimes a challenge and sometimes confusing; often consumes a significant amount of scoring time), 4 (complicated, confusing, and difficult to score; consumes a great deal of scoring time). The New Learner Survey rating scale deliberately targeted the two intertwined components that make coding challenging: (a) how difficult and confusing the coding experience is, and (b) how much time it takes to execute a decision during the coding process. As a result, a score of 1 indicates that the coding category or decision is simple and takes very little time to code, whereas a score of 4 indicates that the coding category or decision is very difficult and requires a great deal of time to code. We created scale anchors that emphasized both relative difficulty and relative time because a simple decision typically can be made quickly and easily, whereas a difficult decision frequently requires additional time to sort through multiple facets of more complex coding criteria.

Participants

The New Learner Survey was administered to psychology graduate students who were in training under the supervision of or in classes with psychologists. Our aim was to investigate the opinion of beginning learners regarding the difficulties they encounter when coding Rorschach protocols. We were interested in the opinion of both new learners (e.g., graduate students who were attending their first Rorschach class) and students who had already completed their first Rorschach semester but who were still in training. Participant surveys were gathered internationally from multiple sites across the United States, as well as sites in Brazil, Denmark, Israel, and Italy. Students in Denmark, Israel, and Italy completed the surveys in English, and those in Brazil completed a Portuguese version of the survey. Overall, 207 psychology graduate students and trainees completed the survey. Approximately half of the contributors were from the United States (56%) and half from international locations. Because absolute beginners might have limited knowledge about Rorschach codes, we excluded participants who had coded fewer than two protocols (5 students) or who did not indicate how many protocols they had coded (5 students). Thus, no absolute beginners were included in the analyses. At the other end of the continuum of scoring experience, students with considerable experience were excluded. Operationally, this was defined as omitting the 6 students who had coded 45 or more protocols, which placed them above the 97th percentile of coding experience.

As a result, the final sample consisted of 191 participants. The majority of the student participants (69.1%) had already completed at least one semester of Rorschach instruction, and the other participants were attending their first Rorschach class. Most of the participants had coded more protocols than they had administered themselves. The median number of coded protocols was 8 with a mean value of 12.1 (SD = 9.3, range - = 2–40), whereas the median number of administered protocols was 6 with a mean value of 10.4 (SD = 9.0, range = 2–33). Absolute skew and kurtosis were lower than 1.1 (see later), so that these variables were normally distributed. The large standard deviations revealed that there was considerable variability in the number of records administered and coded. Because our aim was to investigate new learners' judgments, we also address the potential effect of experience on their judgments. In addition, because students who had not yet completed their first course of Rorschach instruction might not be able to provide informed responses about coding difficulty, we conducted control analyses that were limited to the 132 students who had already completed at least one semester of Rorschach training.

Statistical procedures

To address the normality of the distributions for the 57 survey variables with this relatively large sample, we considered the cutoff suggested by West, Finch, and Curran ([33]) of 2.0 for skew and 7.0 for kurtosis to identify a moderate departure from normality. Fifty-two variables had reasonably normal distributions with an average absolute value for skew of.536 (absolute value range =.006–1.817) and an average absolute value for kurtosis of.457 (absolute value range =.097–2.314). Five variables showed a nonnormal distribution (i.e., W vs. D, Pairs, H vs. (H), (H & A) vs. (Hd & Ad), and Popular).

To establish whether each mean rating for a "target" variable was higher or lower than the overall mean across all the ratings, we computed an overall mean using all the variables except the target variable being investigated. This is analogous to computing part–whole correlations after omitting the part from the whole. For example, we compared the mean rating of the target W versus D coding decision to the overall mean of the ratings for the other 56 variables excluding the rating for W versus D. We repeated this procedure for all the other 56 variables. Next, the target and overall means were compared by computing a paired-samples t test for all the items that were normally distributed and a sign test for those that violated the assumption of normality (Table 1). The overall mean that was the comparison point for each of the 57 rated items had an average across all 57 items of 1.92 and a range across each specific item from 1.90 to 1.93. Each of the standard deviations for these overall means was in the range between 0.36 and 0.37. Because these overall means and standard deviations are so similar, they are not separately reported in Table 1. However, they form the comparison point with the target means and standard deviations, and they are used to generate the t tests, p values, and Cohen's d values that also are reported in Table 1. Given the multiple comparisons, we decided to apply a correction for alpha using the conservative Bonferroni correction: Alpha of.05 was divided by 57, so that a difference is considered significant if p is less than.0009.

Table 1. Descriptive statistics, rating percentages, and paired-sample t test results comparing each of the 57 target coding categories and decisions to the overall mean of the other 56.

<p> <ephtml> <table><thead><tr><td /><td /><td /><td /><td /><td>Ratings (%)</td><td /><td /><td /><td>Rho</td></tr><tr><td /><td>Rank</td><td><italic>M</italic></td><td><italic>SD</italic></td><td><italic>Mdn</italic></td><td>1</td><td>2</td><td>3</td><td>4</td><td><italic>t</italic>(190)</td><td><italic>p</italic></td><td>Cohen's <italic>d</italic></td><td>No. scored prot</td><td>No. admin. prot</td><td>Completed semester</td></tr></thead><tbody><tr><td>Rorschach as a whole</td><td>3</td><td char=".">2.70</td><td char=".">.73</td><td>3</td><td>7</td><td>26</td><td>59</td><td>9</td><td char=".">15.62</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">1.40</td><td char=".">&#8722;.26<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.19<xref ref-type="fn" rid="t1fn0003" /></td><td char=".">&#8722;.18<xref ref-type="fn" rid="t1fn0003" /></td></tr><tr><td><bold>Location as a group</bold></td><td><bold>42</bold></td><td char="."><bold>1.63</bold></td><td char="."><bold>.62</bold></td><td><bold>2</bold></td><td><bold>44</bold></td><td><bold>50</bold></td><td><bold>6</bold></td><td><bold>1</bold></td><td char="."><bold>&#8722;7.38</bold></td><td char="."><bold>&#60;.001</bold><xref ref-type="fn" rid="t1fn0001" /></td><td char="."><bold>&#8722;0.60</bold></td><td char="."><bold>&#8722;.28</bold><xref ref-type="fn" rid="t1fn0004" /></td><td char="."><bold>&#8722;.27</bold><xref ref-type="fn" rid="t1fn0004" /></td><td char="."><bold>&#8722;.25</bold><xref ref-type="fn" rid="t1fn0004" /></td></tr><tr><td>&#8195;W vs. D</td><td>55</td><td char=".">1.20</td><td char=".">.45</td><td>1</td><td>82</td><td>16</td><td>2</td><td>0</td><td char=".">&#8722;11.83<xref ref-type="fn" rid="t1fn0002" /></td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;1.77</td><td char=".">&#8722;.29<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.23<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.20<xref ref-type="fn" rid="t1fn0004" /></td></tr><tr><td>&#8195;D vs. Dd for multiple objects responses</td><td>28</td><td char=".">1.86</td><td char=".">.72</td><td>2</td><td>32</td><td>51</td><td>15</td><td>2</td><td char=".">&#8722;1.14</td><td char=".">.255</td><td char=".">&#8722;0.10</td><td char=".">&#8722;.25<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.21<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.23<xref ref-type="fn" rid="t1fn0004" /></td></tr><tr><td>&#8195;D vs. Dd for "near Dd" responses"</td><td>18</td><td char=".">2.12</td><td char=".">.71</td><td>2</td><td>18</td><td>55</td><td>25</td><td>2</td><td char=".">4.28</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">0.37</td><td char=".">&#8722;.31<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.22<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.24<xref ref-type="fn" rid="t1fn0004" /></td></tr><tr><td>&#8195;DS vs. DdS</td><td>34</td><td char=".">1.82</td><td char=".">.72</td><td>2</td><td>36</td><td>48</td><td>16</td><td>1</td><td char=".">&#8722;2.08</td><td char=".">.039</td><td char=".">&#8722;0.18</td><td char=".">&#8722;.10</td><td char=".">&#8722;.08</td><td char=".">&#8722;.04</td></tr><tr><td><bold>DQ as a group</bold></td><td><bold>36</bold></td><td char="."><bold>1.82</bold></td><td char="."><bold>.74</bold></td><td><bold>2</bold></td><td><bold>36</bold></td><td><bold>49</bold></td><td><bold>13</bold></td><td><bold>2</bold></td><td char="."><bold>&#8722;2.29</bold></td><td char="."><bold>.023</bold></td><td char="."><bold>&#8722;0.19</bold></td><td char="."><bold>&#8722;.16</bold><xref ref-type="fn" rid="t1fn0003" /></td><td char="."><bold>&#8722;.15</bold><xref ref-type="fn" rid="t1fn0003" /></td><td char="."><bold>&#8722;.06</bold></td></tr><tr><td>&#8195;Evaluating synthesis: (DQ+ or v/+ vs. DQo or v)</td><td>30</td><td char=".">1.84</td><td char=".">.72</td><td>2</td><td>33</td><td>52</td><td>13</td><td>2</td><td char=".">&#8722;1.61</td><td char=".">.108</td><td char=".">&#8722;0.14</td><td char=".">&#8722;.19<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.19<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.13</td></tr><tr><td>&#8195;Evaluating form demand: (DQo or + vs. DQv or v/+)</td><td>27</td><td char=".">1.87</td><td char=".">.75</td><td>2</td><td>33</td><td>48</td><td>17</td><td>2</td><td char=".">&#8722;0.98</td><td char=".">.328</td><td char=".">&#8722;0.09</td><td char=".">&#8722;.13</td><td char=".">&#8722;.11</td><td char=".">.04</td></tr><tr><td><bold>Determinants as a group</bold></td><td><bold>7</bold></td><td char="."><bold>2.52</bold></td><td char="."><bold>.78</bold></td><td><bold>3</bold></td><td><bold>8</bold></td><td><bold>42</bold></td><td><bold>40</bold></td><td><bold>10</bold></td><td char="."><bold>11.62</bold></td><td char="."><bold>&#60;.001</bold><xref ref-type="fn" rid="t1fn0001" /></td><td char="."><bold>1.06</bold></td><td char="."><bold>&#8722;.19</bold><xref ref-type="fn" rid="t1fn0003" /></td><td char="."><bold>&#8722;.18</bold><xref ref-type="fn" rid="t1fn0003" /></td><td char="."><bold>&#8722;.25</bold><xref ref-type="fn" rid="t1fn0004" /></td></tr><tr><td>&#8195;M vs. FM vs. m</td><td>45</td><td char=".">1.53</td><td char=".">.65</td><td>1</td><td>55</td><td>38</td><td>7</td><td>1</td><td char=".">&#8722;9.55</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;0.78</td><td char=".">&#8722;.12</td><td char=".">&#8722;.04</td><td char=".">&#8722;.01</td></tr><tr><td>&#8195;Active vs. passive</td><td>21</td><td char=".">2.02</td><td char=".">.78</td><td>2</td><td>26</td><td>49</td><td>22</td><td>3</td><td char=".">1.98</td><td char=".">.049</td><td char=".">0.17</td><td char=".">&#8722;.22<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.18<xref ref-type="fn" rid="t1fn0003" /></td><td char=".">&#8722;.09</td></tr><tr><td>&#8195;Color vs. No Color</td><td>50</td><td char=".">1.36</td><td char=".">.57</td><td>1</td><td>69</td><td>26</td><td>5</td><td>0</td><td char=".">&#8722;13.81</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;1.22</td><td char=".">&#8722;.10</td><td char=".">&#8722;.16<xref ref-type="fn" rid="t1fn0003" /></td><td char=".">&#8722;.09</td></tr><tr><td>&#8195;FC vs. CF vs. Pure C</td><td>15</td><td char=".">2.32</td><td char=".">.75</td><td>2</td><td>13</td><td>46</td><td>37</td><td>4</td><td char=".">8.71</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">0.74</td><td char=".">&#8722;.10</td><td char=".">&#8722;.06</td><td char=".">&#8722;.04</td></tr><tr><td>&#8195;Shading subtypes: Y vs. T vs. V</td><td>5</td><td char=".">2.65</td><td char=".">.79</td><td>3</td><td>7</td><td>35</td><td>46</td><td>13</td><td char=".">14.65</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">1.28</td><td char=".">&#8722;.23<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.21<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.12</td></tr><tr><td>&#8195;Y vs. C'</td><td>14</td><td char=".">2.33</td><td char=".">.82</td><td>2</td><td>16</td><td>41</td><td>37</td><td>6</td><td char=".">8.32</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">0.70</td><td char=".">&#8722;.20<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.13</td><td char=".">&#8722;.10</td></tr><tr><td>&#8195;C' vs. No C'</td><td>32</td><td char=".">1.83</td><td char=".">.76</td><td>2</td><td>37</td><td>44</td><td>17</td><td>2</td><td char=".">&#8722;1.92</td><td char=".">.056</td><td char=".">&#8722;0.16</td><td char=".">&#8722;.10</td><td char=".">&#8722;.04</td><td char=".">.05</td></tr><tr><td>&#8195;FY vs. YF vs. Pure Y</td><td>11</td><td char=".">2.48</td><td char=".">.71</td><td>2</td><td>6</td><td>46</td><td>42</td><td>6</td><td char=".">13.51</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">1.07</td><td char=".">&#8722;.05</td><td char=".">.00</td><td char=".">&#8722;.05</td></tr><tr><td>&#8195;FT vs. TF vs. Pure T</td><td>12</td><td char=".">2.39</td><td char=".">.75</td><td>2</td><td>10</td><td>48</td><td>36</td><td>6</td><td char=".">11.35</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">0.88</td><td char=".">&#8722;.06</td><td char=".">&#8722;.01</td><td char=".">&#8722;.01</td></tr><tr><td>&#8195;FV vs. VF vs. Pure V</td><td>6</td><td char=".">2.54</td><td char=".">.68</td><td>3</td><td>4</td><td>44</td><td>46</td><td>6</td><td char=".">15.87</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">1.22</td><td char=".">&#8722;.14</td><td char=".">&#8722;.07</td><td char=".">&#8722;.06</td></tr><tr><td>&#8195;FC' vs. C'F vs. Pure C'</td><td>13</td><td char=".">2.33</td><td char=".">.76</td><td>2</td><td>13</td><td>46</td><td>36</td><td>5</td><td char=".">9.49</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">0.75</td><td char=".">.02</td><td char=".">.08</td><td char=".">.07</td></tr><tr><td>&#8195;Depth: FD vs. Vista</td><td>10</td><td char=".">2.51</td><td char=".">.79</td><td>3</td><td>10</td><td>38</td><td>43</td><td>9</td><td char=".">11.63</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">1.03</td><td char=".">&#8722;.12</td><td char=".">&#8722;.10</td><td char=".">&#8722;.12</td></tr><tr><td>&#8195;Reflections</td><td>49</td><td char=".">1.41</td><td char=".">.63</td><td>1</td><td>66</td><td>26</td><td>7</td><td>0</td><td char=".">&#8722;11.27</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;1.04</td><td char=".">.05</td><td char=".">.03</td><td char=".">.08</td></tr><tr><td><bold>FQ as a group</bold></td><td><bold>23</bold></td><td char="."><bold>1.98</bold></td><td char="."><bold>.70</bold></td><td><bold>2</bold></td><td><bold>24</bold></td><td><bold>57</bold></td><td><bold>17</bold></td><td><bold>2</bold></td><td char="."><bold>1.33</bold></td><td char="."><bold>.184</bold></td><td char="."><bold>0.11</bold></td><td char="."><bold>.00</bold></td><td char="."><bold>&#8722;.03</bold></td><td char="."><bold>.06</bold></td></tr><tr><td>&#8195;FQo vs. (FQu & FQ-)</td><td>35</td><td char=".">1.82</td><td char=".">.79</td><td>2</td><td>39</td><td>41</td><td>17</td><td>2</td><td char=".">&#8722;2.00</td><td char=".">.047</td><td char=".">&#8722;0.17</td><td char=".">&#8722;.02</td><td char=".">.01</td><td char=".">.07</td></tr><tr><td>&#8195;FQu vs. FQ-</td><td>19</td><td char=".">2.07</td><td char=".">.81</td><td>2</td><td>25</td><td>48</td><td>23</td><td>4</td><td char=".">2.83</td><td char=".">.005</td><td char=".">0.26</td><td char=".">.01</td><td char=".">&#8722;.03</td><td char=".">.11</td></tr><tr><td>&#8195;Extrapolation for objects not in FQ Table</td><td>9</td><td char=".">2.51</td><td char=".">.82</td><td>3</td><td>10</td><td>40</td><td>39</td><td>11</td><td char=".">11.56</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">1.01</td><td char=".">&#8722;.13</td><td char=".">&#8722;.10</td><td char=".">&#8722;.05</td></tr><tr><td><bold>Pairs</bold></td><td><bold>57</bold></td><td char="."><bold>1.15</bold></td><td char="."><bold>.42</bold></td><td><bold>1</bold></td><td><bold>87</bold></td><td><bold>11</bold></td><td><bold>2</bold></td><td><bold>0</bold></td><td char="."><bold>&#8722;12.41</bold><xref ref-type="fn" rid="t1fn0002" /></td><td char="."><bold>&#60;.001</bold><xref ref-type="fn" rid="t1fn0001" /></td><td char="."><bold>&#8722;1.99</bold></td><td char="."><bold>&#8722;.03</bold></td><td char="."><bold>&#8722;.05</bold></td><td char="."><bold>.16</bold><xref ref-type="fn" rid="t1fn0003" /></td></tr><tr><td><bold>Content as a group</bold></td><td><bold>38</bold></td><td char="."><bold>1.67</bold></td><td char="."><bold>.62</bold></td><td><bold>2</bold></td><td><bold>41</bold></td><td><bold>51</bold></td><td><bold>8</bold></td><td><bold>0</bold></td><td char="."><bold>&#8722;6.74</bold></td><td char="."><bold>&#60;.001</bold><xref ref-type="fn" rid="t1fn0001" /></td><td char="."><bold>&#8722;0.51</bold></td><td char="."><bold>&#8722;.21</bold><xref ref-type="fn" rid="t1fn0004" /></td><td char="."><bold>&#8722;.17</bold><xref ref-type="fn" rid="t1fn0003" /></td><td char="."><bold>&#8722;.08</bold></td></tr><tr><td>&#8195;H vs. (H)</td><td>53</td><td char=".">1.23</td><td char=".">.47</td><td>1</td><td>79</td><td>20</td><td>1</td><td>1</td><td char=".">&#8722;11.54<xref ref-type="fn" rid="t1fn0002" /></td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;1.69</td><td char=".">&#8722;.12</td><td char=".">&#8722;.07</td><td char=".">.10</td></tr><tr><td>&#8195;Whole vs. Detail: (H & A) vs. (Hd & Ad)</td><td>52</td><td char=".">1.27</td><td char=".">.53</td><td>1</td><td>77</td><td>20</td><td>3</td><td>1</td><td char=".">&#8722;10.87<xref ref-type="fn" rid="t1fn0002" /></td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;1.48</td><td char=".">&#8722;.15<xref ref-type="fn" rid="t1fn0003" /></td><td char=".">&#8722;.10</td><td char=".">.03</td></tr><tr><td>&#8195;Animal vs. Human: (A & Ad) vs. (H & Hd)</td><td>54</td><td char=".">1.21</td><td char=".">.43</td><td>1</td><td>80</td><td>19</td><td>1</td><td>0</td><td char=".">&#8722;22.65</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;1.80</td><td char=".">&#8722;.16<xref ref-type="fn" rid="t1fn0003" /></td><td char=".">&#8722;.14</td><td char=".">.03</td></tr><tr><td>&#8195;Ay vs. Art and other scores</td><td>26</td><td char=".">1.88</td><td char=".">.76</td><td>2</td><td>33</td><td>48</td><td>17</td><td>2</td><td char=".">&#8722;0.74</td><td char=".">.460</td><td char=".">&#8722;0.06</td><td char=".">&#8722;.07</td><td char=".">&#8722;.02</td><td char=".">.05</td></tr><tr><td>&#8195;Cl vs. Na and other scores</td><td>44</td><td char=".">1.57</td><td char=".">.70</td><td>1</td><td>55</td><td>33</td><td>12</td><td>0</td><td char=".">&#8722;8.34</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;0.66</td><td char=".">&#8722;.13</td><td char=".">&#8722;.07</td><td char=".">.01</td></tr><tr><td>&#8195;Isolation Contents: Ls vs. Bt vs. Na</td><td>31</td><td char=".">1.84</td><td char=".">.75</td><td>2</td><td>36</td><td>47</td><td>16</td><td>2</td><td char=".">&#8722;1.82</td><td char=".">.070</td><td char=".">&#8722;0.15</td><td char=".">&#8722;.15<xref ref-type="fn" rid="t1fn0003" /></td><td char=".">&#8722;.10</td><td char=".">.02</td></tr><tr><td>&#8195;Xy vs. An</td><td>48</td><td char=".">1.44</td><td char=".">.57</td><td>1</td><td>60</td><td>37</td><td>4</td><td>0</td><td char=".">&#8722;14.63</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;1.05</td><td char=".">&#8722;.08</td><td char=".">&#8722;.04</td><td char=".">.01</td></tr><tr><td>&#8195;An vs. Animal/Human Detail</td><td>37</td><td char=".">1.68</td><td char=".">.69</td><td>2</td><td>44</td><td>46</td><td>9</td><td>1</td><td char=".">&#8722;5.72</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;0.47</td><td char=".">&#8722;.06</td><td char=".">.02</td><td char=".">.06</td></tr><tr><td>&#8195;Fd vs. Bt/A/Ad</td><td>51</td><td char=".">1.29</td><td char=".">.51</td><td>1</td><td>74</td><td>24</td><td>3</td><td>0</td><td char=".">&#8722;20.00</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;1.47</td><td char=".">&#8722;.02</td><td char=".">&#8722;.00</td><td char=".">.04</td></tr><tr><td>&#8195;Hh</td><td>47</td><td char=".">1.44</td><td char=".">.63</td><td>1</td><td>62</td><td>33</td><td>4</td><td>1</td><td char=".">&#8722;13.27</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;0.98</td><td char=".">&#8722;.08</td><td char=".">&#8722;.06</td><td char=".">.00</td></tr><tr><td>&#8195;Hx vs. No Hx</td><td>33</td><td char=".">1.83</td><td char=".">.75</td><td>2</td><td>36</td><td>47</td><td>15</td><td>2</td><td char=".">&#8722;1.91</td><td char=".">.058</td><td char=".">&#8722;0.17</td><td char=".">.00</td><td char=".">.02</td><td char=".">.04</td></tr><tr><td>&#8195;Sc</td><td>46</td><td char=".">1.51</td><td char=".">.66</td><td>1</td><td>58</td><td>34</td><td>9</td><td>0</td><td char=".">&#8722;10.96</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;0.81</td><td char=".">&#8722;.19<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.11</td><td char=".">.00</td></tr><tr><td>&#8195;Sx vs. Hd/Ad vs. not</td><td>39</td><td char=".">1.66</td><td char=".">.69</td><td>2</td><td>46</td><td>42</td><td>11</td><td>1</td><td char=".">&#8722;6.29</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;0.49</td><td char=".">&#8722;.12</td><td char=".">&#8722;.07</td><td char=".">.02</td></tr><tr><td>&#8195;Id vs. Not</td><td>20</td><td char=".">2.03</td><td char=".">.85</td><td>2</td><td>28</td><td>47</td><td>19</td><td>6</td><td char=".">2.11</td><td char=".">.036</td><td char=".">0.18</td><td char=".">&#8722;.22<xref ref-type="fn" rid="t1fn0004" /></td><td char=".">&#8722;.14</td><td char=".">&#8722;.08</td></tr><tr><td><bold>Popular</bold></td><td><bold>56</bold></td><td char="."><bold>1.18</bold></td><td char="."><bold>.46</bold></td><td><bold>1</bold></td><td><bold>85</bold></td><td><bold>13</bold></td><td><bold>2</bold></td><td><bold>1</bold></td><td char="."><bold>&#8722;11.54</bold><xref ref-type="fn" rid="t1fn0002" /></td><td char="."><bold>&#60;.001</bold><xref ref-type="fn" rid="t1fn0001" /></td><td char="."><bold>&#8722;1.82</bold></td><td char="."><bold>&#8722;.11</bold></td><td char="."><bold>&#8722;.11</bold></td><td char="."><bold>&#8722;.06</bold></td></tr><tr><td><bold>Cognitive Special Scores as a group</bold></td><td><bold>1</bold></td><td char="."><bold>3.02</bold></td><td char="."><bold>.73</bold></td><td><bold>3</bold></td><td><bold>2</bold></td><td><bold>19</bold></td><td><bold>54</bold></td><td><bold>25</bold></td><td char="."><bold>23.74</bold></td><td char="."><bold>&#60;.001</bold><xref ref-type="fn" rid="t1fn0001" /></td><td char="."><bold>2.05</bold></td><td char="."><bold>&#8722;.10</bold></td><td char="."><bold>&#8722;.00</bold></td><td char="."><bold>&#8722;.08</bold></td></tr><tr><td>&#8195;Present vs. Absent</td><td>8</td><td char=".">2.52</td><td char=".">.85</td><td>3</td><td>12</td><td>37</td><td>40</td><td>12</td><td char=".">11.14</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">1.01</td><td char=".">&#8722;.15<xref ref-type="fn" rid="t1fn0003" /></td><td char=".">&#8722;.09</td><td char=".">&#8722;.10</td></tr><tr><td>&#8195;Deciding what Cognitive Special Scores applies: DV, INCOM, DR, FABCOM, ALOG, or CONTAM</td><td>2</td><td char=".">2.93</td><td char=".">.78</td><td>3</td><td>3</td><td>25</td><td>49</td><td>24</td><td char=".">20.27</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">1.79</td><td char=".">.01</td><td char=".">.08</td><td char=".">&#8722;.08</td></tr><tr><td>&#8195;Level 1&#160;vs. Level 2 Distinction</td><td>4</td><td char=".">2.66</td><td char=".">.88</td><td>3</td><td>7</td><td>39</td><td>34</td><td>20</td><td char=".">13.09</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">1.21</td><td char=".">.07</td><td char=".">.14</td><td char=".">.02</td></tr><tr><td><bold>Other Special Scores as a group</bold></td><td><bold>16</bold></td><td char="."><bold>2.32</bold></td><td char="."><bold>.81</bold></td><td><bold>2</bold></td><td><bold>13</bold></td><td><bold>50</bold></td><td><bold>29</bold></td><td><bold>8</bold></td><td char="."><bold>7.77</bold></td><td char="."><bold>&#60;.001</bold><xref ref-type="fn" rid="t1fn0001" /></td><td char="."><bold>0.68</bold></td><td char="."><bold>&#8722;.10</bold></td><td char="."><bold>.03</bold></td><td char="."><bold>&#8722;.09</bold></td></tr><tr><td>&#8195;AB</td><td>17</td><td char=".">2.15</td><td char=".">.85</td><td>2</td><td>23</td><td>45</td><td>25</td><td>6</td><td char=".">4.34</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">0.38</td><td char=".">.04</td><td char=".">.09</td><td char=".">.03</td></tr><tr><td>&#8195;AG</td><td>41</td><td char=".">1.63</td><td char=".">.69</td><td>2</td><td>48</td><td>42</td><td>9</td><td>1</td><td char=".">&#8722;7.64</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;0.55</td><td char=".">&#8722;.14</td><td char=".">&#8722;.10</td><td char=".">&#8722;.02</td></tr><tr><td>&#8195;COP</td><td>43</td><td char=".">1.59</td><td char=".">.72</td><td>1</td><td>53</td><td>38</td><td>7</td><td>2</td><td char=".">&#8722;8.36</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;0.63</td><td char=".">&#8722;.09</td><td char=".">&#8722;.03</td><td char=".">&#8722;.06</td></tr><tr><td>&#8195;GHR vs. PHR</td><td>22</td><td char=".">1.98</td><td char=".">.91</td><td>2</td><td>35</td><td>40</td><td>18</td><td>7</td><td char=".">1.17</td><td char=".">.245</td><td char=".">0.10</td><td char=".">&#8722;.03</td><td char=".">.02</td><td char=".">&#8722;.04</td></tr><tr><td>&#8195;MOR</td><td>40</td><td char=".">1.65</td><td char=".">.71</td><td>2</td><td>47</td><td>42</td><td>10</td><td>1</td><td char=".">&#8722;6.23</td><td char=".">&#60;.001<xref ref-type="fn" rid="t1fn0001" /></td><td char=".">&#8722;0.50</td><td char=".">&#8722;.09</td><td char=".">&#8722;.03</td><td char=".">.01</td></tr><tr><td>&#8195;PER</td><td>24</td><td char=".">1.97</td><td char=".">.82</td><td>2</td><td>32</td><td>44</td><td>21</td><td>4</td><td char=".">1.05</td><td char=".">.296</td><td char=".">0.09</td><td char=".">&#8722;.14</td><td char=".">&#8722;.03</td><td char=".">&#8722;.11</td></tr><tr><td>&#8195;PSV</td><td>25</td><td char=".">1.93</td><td char=".">.79</td><td>2</td><td>32</td><td>44</td><td>21</td><td>2</td><td char=".">0.27</td><td char=".">.784</td><td char=".">0.02</td><td char=".">&#8722;.10</td><td char=".">.02</td><td char=".">&#8722;.06</td></tr><tr><td><bold><italic>Z</italic> scores</bold></td><td><bold>29</bold></td><td char="."><bold>1.84</bold></td><td char="."><bold>.84</bold></td><td><bold>2</bold></td><td><bold>40</bold></td><td><bold>39</bold></td><td><bold>17</bold></td><td><bold>4</bold></td><td char="."><bold>&#8722;1.30</bold></td><td char="."><bold>.196</bold></td><td char="."><bold>&#8722;0.12</bold></td><td char="."><bold>&#8722;.17</bold><xref ref-type="fn" rid="t1fn0003" /></td><td char="."><bold>&#8722;.13</bold></td><td char="."><bold>&#8722;.16</bold><xref ref-type="fn" rid="t1fn0003" /></td></tr></tbody></table> </ephtml> </p> 2 Note. Bold is used for coding categories. The rating categories are as follows: 1 = simple, straightforward, and very easy to score; takes very little time to score; 2 = understandable and relatively easy to score but does take some work at times; takes a little time and effort to score; 3 = sometimes a challenge and sometimes confusing; often consumes a significant amount of scoring time; 4 = complicated, confusing, and difficult to score; consumes a great deal of scoring time. 10001 p <.0009. 10002 Sign test. 10003 p ≤.05. 10004 p ≤.01. <p>Cohen's d was computed to evaluate the magnitude of the difference between each target coding category and its comparison overall mean score. Even though these were paired samples data, we followed statistical recommendations by Dunlap, Cortina, Vaslow, and Burke ([3]) and used the standard d formula of computing mean differences and dividing by the pooled standard deviation to document how far apart the two sets of mean values were in standard deviation units. Consistent with characterizations in Cohen ([2]), values around |.20| would indicate a small effect size, |.50| a medium effect size, and |.80| a large effect size. However, it is important to keep in mind that the d values reported here are different than typical d values because we are comparing each target variable mean to an overall mean, which is a grand mean computed across all the other rated variables, rather than to other individual target means. For instance, the data to be presented comparing the item that was rated most difficult to code (Cognitive Special Scores) to the item rated least difficult to code (Pairs) would produce a d value of about |3.0|, although the d values we report in Table 1 comparing each of these items to their counterpart overall means is about |2.0|.</p> <p>Finally, Spearman correlations were used to evaluate the relationship between the coding difficulty judgments expressed by the participants and their level of experience. For indicators of experience, we used the number of protocols scored, the number of protocols administered, and whether the student had completed a semester of training.</p>

Results and discussion

<p>Coding a Rorschach record appears to be moderately difficult for new learners: About 59% of the students rated the overall difficulty in coding the Rorschach as 3 (sometimes a challenge and sometimes confusing; often consumes a significant amount of scoring time). The mean rating for coding the CS as a whole of 2.70 (SD =.73) is slightly lower than this benchmark score. However, averaging ratings across all 57 scores produced a mean difficulty rating of 1.92 (SD =.48). The large and significant difference between the mean rating of the Rorschach as whole and the overall mean across all the individual target ratings suggests that new learners experience the challenge of scoring all the codes for a full protocol as considerably more difficult than the average, single coding decision. From a different perspective, coders might experience an additive effect of difficulty across codes and responses.</p> <p>Reviewing Table 1, it can be seen that 21 coding decisions and categories have a mean rating over 2, and Cognitive Special Scores has a mean rating greater than 3. When separated into each coding category (i.e., Location, Determinants, etc.) and compared with the overall mean, the category with the highest (i.e., most difficult) rating is Cognitive Special Scores (M = 3.02, d = 2.05), followed by Determinants (M = 2.52, d = 1.06), and then Other Special Scores (M = 2.32, d = 0.68). Because these three coding categories were rated as the most difficult, we discuss their individual coding decisions more fully.</p> <p>As the most difficult and time-consuming coding category, 79% of participants rated Cognitive Special Scores as challenging (3) or complicated (4). All the individual coding decisions within this category have a mean rating significantly higher than the overall mean, with large to very large effect sizes relative to the overall mean of all remaining items (ds from 1.01–1.79). Thus, not only is the Cognitive Special Scores as a group the most complicated coding category that consumes a great deal of scoring time, but also within this group the primary coding decisions we asked about were all essentially equally difficult.</p> <p>Determinants are the second most difficult category overall (M = 2.52, d = 1.06) with a large effect size. Comparing each subcategory with its overall mean (Table 1), four areas have significantly higher mean scores with medium to large effect sizes: (a) deciding which shading subtypes to code (d = 1.28), (b) deciding between dimensionality based on form (FD) and dimensionality based on shading (V; d = 1.03), (c) deciding on the degree of Form Dominance (ds from 0.74–1.22), and (d) differentiating between Diffuse Shading (Y) and Achromatic Color (C'; d = 0.70). Among the remaining Determinants, Reflections, the presence or absence of Color, and the distinction between Human (M), Animal (FM), and Inanimate (m) Movements were reported as 1 (simple) or 2 (relatively easy) to code by more than 90% of the students. Moreover, their means are significant lower compared to the overall mean. Thus, as a whole, Determinants are challenging to code and consume significant time with decisions among shading subtypes, FD versus V, and Form Dominance for both color and shading responses taking the most time and effort. In contrast, coding Reflection, color versus not, and distinctions between M, FM, and m are considered easy to code.</p> <p>The category of Other Special Scores, which could be considered to be thematic codes rather than cognitive codes, was rated more than 0.5 SD above the overall mean of all the variables (M = 2.32, d = 0.68). In addition, the mean rating of the special score Abstract (AB; M = 2.15, d = 0.38) is slightly higher than the overall mean score, but Aggressive Movement (AG; M = 1.63, d = −0.55), Cooperative Movement (COP; M = 1.59, d = −0.63), and Morbid (MOR; M = 1.65, d = −0.50) have a mean rating significantly lower than the overall mean score. Furthermore, most of the students rated each individual coding decision as relatively easy to code. Given these results, it remains unclear which coding decisions contribute to the perceived relative high difficulty rating of this general coding category. To speculate, it might be due to the overall number and diversity of themes of the variables involved.</p> <p>Form Quality (FQ; M = 1.98, d = 0.11), is the next most difficult to code but not significantly more than the survey mean. FQ as a category was rated with a 3 (challenging) or 4 (complicated) by only 19% of the students. It is worth noting that the mean rating of extrapolation for objects not listed in the FQ tables (M = 2.51, d = 1.01) is significantly higher than the overall mean and among the highest of the individual coding decisions. Indeed, 50% of participants found this procedure challenging or difficult and time-consuming. This suggests that within the FQ coding category, extrapolation might account for a large part of the challenges for new learners. Simplifying and providing guidance for this process might improve utility and help new learners. One might also wonder whether extrapolation difficulties partially account for the finding that the interrater reliability of FQ coding is often low, particularly for FQu.</p> <p>New learners, however, perceived many scores as simple to code. The mean ratings of the coding decisions related to Location (M = 1.63, d = −0.60), Pairs (M = 1.15, d = −1.99), Content (M = 1.67, d = −0.651), and Popular (M = 1.18, d = −1.82) are significantly lower than the overall mean score. For Location, coding D versus Dd for "near Dd responses" (M = 2.12) is the only coding decision that has a mean rating greater than the overall mean, but the Cohen's d of −0.37 indicates a small to medium effect size. Interestingly, most of the Contents display medium to large differences in mean scores (Table 1), and all are lower than the overall mean score except for Idiographic contents (Id; M = 2.03, d =.18). Developmental Quality (DQ; M = 1.82, d = −0.19) and the Z scores (M = 1.84, d = −0.12) did not show any significant difference compared to the overall mean score (Table 1), suggesting that they are in the average range of difficulty.</p> <p>To address experience and training, we calculated Spearman rank-order correlations to analyze the relationship of coding difficulty ratings with three measures of experience and training: (a) the number of protocols scored, (b) the number of protocols administered, and (c) whether or not one had completed a first Rorschach course (see the final columns in Table 1). As expected, the correlations were predominantly negative with only one significantly positive correlation (completing a semester with Pairs, rho =.16). Thus, in general, more experience is associated with lower ratings of CS coding difficulty.</p> <p>The number of scored protocols produced 19 significant correlations with the coding categories and decisions (p <.05; rho >.14). Also, for 39 out of 57 items, correlation values for the number of scored protocols were greater than those obtained by the number of administered protocols (4 out of 57) and having completed the first Rorschach class (8 out of 57). It makes sense that practicing coding is the key component in the reduction of confusion encountered and time spent coding. The key findings for the number of scored protocols are the mean of the 57 correlations (rho = –.21) and the correlation with the Rorschach as a whole (rho = –.26) with a small to medium effect size. Thus, the biggest effect on ease of CS coding derives from the number of Rorschach protocols scored. Nevertheless, 38 of 57 correlations with the number of protocols scored were not significant, indicating that the effect was not detected for two thirds of the coding difficulty ratings. In part, this might be due to our decision to omit the most and least experienced student coders from the sample.</p> <p>Focusing on the associations with the number of protocols coded, it is difficult to identify patterns for those most or least affected by experiences. However, it does appear that Cognitive Special Scores, Form Quality, Form Dominance, Pairs, and Other Special Scores are least influenced by experience. Alternatively, for Location, other Determinant Codes, most Content decisions, and Z scores, practice appears to be most helpful in the experience of new learners.</p> <p>Location and Content were easier to code compared to the overall mean score, and it seems that practice makes them even easier to code. Pairs and Popular did not improve with practice, but the data suggest that they are easy to code at the beginning so that practice does not make them easier. However, FQ and Special Scores are more challenging categories, and training was not found to be particularly helpful. For these categories, better guidelines and instruction, or even a reformulation of the coding processes themselves, might be needed.</p> <p>Finally, we recomputed mean ratings of difficulty after excluding students who were still completing their first semester of course work. Limiting the results to the 132 students who had at least a semester of Rorschach experience produced no noticeable difference in the findings. The mean ratings showed essentially the same distribution as before across the 57 items (M = 1.90 vs. 1.92, SD = 0.46 vs. 0.47, minimum = 1.15 vs. 1.15, maximum = 2.98 vs. 3.02) and the correlation of the mean ratings in the refined sample with the mean ratings in the full sample was.9959. Similarly, the effect sizes obtained in the smaller but more experienced subsample were essentially unchanged relative to the effect sizes obtained in the full sample, with the correlation between these values being.9955.</p>

Conclusions and implications

<p>For the New Learner Survey, the students learning the Rorschach judged each coding category and decision to identify those that are difficult, challenging, and time-consuming. Overall, our results suggest that the students consider the CS as a whole to be more difficult and time-consuming to code than most of the single coding categories or decisions in the system, thus possibly experiencing an additive effect across codes in the challenge to learn coding. The students struggle most with Cognitive Special Scores, Determinants (particularly with shading subtypes, FD vs. V, and form dominance), and extrapolation for objects not listed in the FQ tables. Location, Pair, Content, and Popular seem to be easy to code, and DQ and Z scores had mean ratings in the average range. Finally, we found no evidence that the degree of experience as measured by the number of scored protocols, the number of administered protocols, or by having completed the first Rorschach class, influenced new learners' judgment about the perceived difficulty of Rorschach coding decisions. However, before accepting the relative independence of experience from perceived coding difficulty, further research with sufficient power would be necessary. In addition, we did not investigate the relationship between experience and accuracy (e.g., Callahan, [1]), so, despite our findings, it is highly likely that accuracy improves with experience.</p> <p>The results of this study raise interesting considerations about the interrater reliability of the CS and possibly extending to R–PAS. To address this, one can compare the New Learner Survey results with the results reported in the literature. Doing so, considerable similarity is found between the codes perceived as most difficult and time-consuming by the new learners and the codes that have lower interrater reliability. Based on findings in the literature, Viglione and Meyer ([31]) indicated that Form Dominance, shading subtypes, Cognitive Special Scores, and the distinction between Level 1 and Level 2 cognitive codes were the coding decisions with lower reliabilities. Lower interrater reliabilities for Determinants, FQ, and Special Scores were also reported by Hilsenroth et al. ([10]), Guarnaccia et al. ([9]), and Callahan ([1]) for the CS and by Kivisalu et al. ([12]) for R–PAS. These coding decisions also had higher mean ratings of difficulty in the New Learner Survey. Interrater reliability statistics measure how well two raters agree in coding the same Rorschach response. The fact that new learners find some coding decisions particularly confusing and time-consuming probably increases the probability of disagreement in coding the same Rorschach variable. That is, if a coding decision is not clear and obvious, the coders could use different strategies to resolve the dilemma and those strategies might not be shared among different sites or different coders.</p> <p>The importance of understanding which Rorschach variables are difficult to code for new learners can also provide guidance for improving teaching methods and identifying which codes need more detailed coding guidelines and practice materials. The findings from this study might, therefore, suggest how to reduce scoring time in practice and how to improve the training materials and the guidelines for some of the more difficult variables and, in doing so, improve the utility of the Rorschach from a new student's perspective. Also, the correlations suggest that coding records and responses rather than administering records is the key variable in making coding easier for students.</p> <p>DQ, as well as Pairs, Content, Popular, and Z scores were rated as easy or in the average range of difficulty, so that these categories might not need to be improved in training materials. Location as a category and its subcomponents were perceived as relatively easy to code and were generally correlated with experience. The correlation with experience suggests that training guidelines should focus on exercises in coding location with threshold examples to help students differentiate W versus D, D versus Dd in multiobject responses, and D versus Dd locations in "near Dd responses."</p> <p>Related to Determinants, students identified Form Dominance, shading subtypes, and distinctions between Y versus C' and FD versus Vista as the coding decisions that were more difficult to learn. Given the results of the correlations, it seems that more coding exercises might help students to become confident in distinguishing among shading subtypes, whereas more teaching time and threshold examples could lead to a better understanding of how to code the degree of Form Dominance and how to differentiate the two depth codes (i.e., FD vs. Vista). Although differentiating whether movement was active versus passive was not rated as difficult, practice coding made it easier.</p> <p>Students struggle with the process of extrapolation to determine FQ for objects not in the FQ tables, and experience does not help students in feeling confident about FQ codes. One specific suggestion would be to present a systematic method for FQ extrapolation, outlining via numbered steps or a flow chart all the possibilities that students might encounter during the procedure. The same recommendations apply to Cognitive Special Scores, for which additional experience might not be helpful in reducing difficulty. Among Other Special Scores, results from this study suggest that AB is a bit harder to learn than average and requires more teaching time. Additional guidelines and benchmark examples are likely to help with this code. In contrast, students considered AG, COP, and MOR relatively simple to code and to learn. Additional guidance for FQ extrapolation, differentiating cognitive codes, and classifying AB has been provided by Viglione ([29]) for the CS and by Meyer et al. ([19]) for R–PAS.</p> <p>This study focused on the CS, and some of its findings ultimately contributed to refinements in an advanced coding guide for the CS (Viglione, [29]). In addition, some of the findings contributed to decisions made when creating R–PAS (Meyer et al., [19]) once it became clear that the CS source materials (Exner, [7]) would not be revised (e.g., elaborated guidelines; no longer coding form dominance for Y, T, V, C,' and r; identifying ambiguous location boundaries to aid D vs. Dd Location decisions; practice coding responses accompanied by commentary to illustrate benchmark standards). On the other hand, R–PAS added a number of thematic codes, differentiated ways of using background white space, and added codes related to administration behaviors. These additions might offset some of the other improvements by introducing new coding challenges.</p> <p>Although refined coding guidelines, simplified coding requirements, and structured coding examples will help make it easier to learn how to code Rorschach responses, it should be recognized that some of the difficulties identified in this survey will remain difficulties for any multivariable system of coding. In part this is because there is an almost limitless degree of unique attributions, communications, and behaviors that can be exhibited by respondents when giving responses. In part this also is because there is an irreducible degree of uncertainty associated with assigning particular kinds of codes, including the task of assessing the visual fit of images to inkblot locations based on verbal and nonverbal communications (Form Quality) and the task of classifying dimensions of disordered thought and impossible or implausible imagery in inherently confusing thought-disordered communications (Cognitive Special Scores). Thus, coding the behaviors observed in the structured context of Rorschach responding will never be as simple as tallying responses to a self-report questionnaire or evaluating the correctness of verbal and nonverbal responses to a cognitive assessment measure. Alerting students to the genuinely complex demands of coding Rorschach task behaviors will not erase the challenges, but it might heighten the rewards that come from mastering such a skill.</p> <p>Although this study adds important new suggestions about CS training material and describes the new learners' point of view, it has some limitations. First, the only participant information we collected was country of origin, completion of a semester-long Rorschach course, and the number of protocols coded and administered. The study did not access other demographic information or potentially relevant training information; for example, age, hours of supervision, type of class attended, or whether records were administered and coded for training, clinical practice, or research. In addition, we did not determine to what extent students relied on the relatively brief standard training materials (Exner, [6], [7]) versus supplemental material designed to make CS coding easier and more reliable (Viglione, [28], [29]). These limitations might reduce the generalizability and validity of the findings, even though our study does provide practical implications for teaching and test utility. Second, when we developed the New Learner Survey, we used a subjective frame of reference in the form of asking students to report their experience of the difficulty and time involved in coding. It is not clear how the findings might be different if time and difficulty were separated in the survey. Future research might isolate the perceived difficulty of each coding decision and assess its relation to a more "objective" measure of time. Asking students about the amount of time they have actually spent learning the Rorschach codes during their training period or the amount of time actually spent on specific coding decisions are ways of addressing time. Finally, our study did not investigate how students perceive R–PAS coding decisions—on their own or compared to the CS. It would be interesting to replicate this study with R–PAS.</p>

Acknowledgments

<p>Part of these data were presented at the Society for Personality Assessment Annual Conference in New Orleans, LA, March 28, 2008, by Katrina Ptucha, Donald J. Viglione, and Gregory J. Meyer.</p> <p>We thank Edna Appel, Melissa Babb, Kathie Bates, Devon Berkheiser, Sakina Butt, Matt Del Giudice, Lisa Greenberg, Carla Hisatugo, Jan Ivanouw, Ramsey Khouri, Dixie Lee, Staphanie Lippman, Andre Felicio Lobato Sobral, Suzanna Lozada-Murray, Daniella Miranda, Ana Luiza Patriarca Mineo, Laura Parolin, Piero Porcelli, Aaron Upton, Jared Wilke, Latife Yazigi, and Tal Yelin for their help in administering the New Learner Survey.</p>

Disclosure

<p>Donald J. Viglione and Gregory J. Meyer are among the owners of the Rorschach Performance Assessment System and receive royalties on the sale of the R–PAS manual and associated products.</p> <p class="aug"> <p>By Donald J. Viglione; Gregory J. Meyer; Ana Cristina Resende and Claudia Pignolo</p> <p>Reported by Author; Author; Author; Author</p> </p>