“This is not criticizing, it is hurting”
In 2013 Andre van Delft and I wrote a a so-called ‘non-peer-review’ of an article published in the journal Psychology, Crime, and Law. This was the final reaction of the journal:
I went through your submission for Psychology, Crime, and Law and decided to reject it outright. The scope of the manuscript is not aimed at furthering any scientific point, but mostly bashing the authors of the original paper. Besides that, your tone is hardly less offensive than in the previous draft. I therefore decided to follow the reviewers of the first draft and not publish your manuscript.
You are of course now free to submit the paper elsewhere should you choose to do so.
The last sentence is quite remarkable, to use an euphemism: the man gives us permission to publish our work elsewhere…
Below the image – just a reminder of our attitude towards science – you find our non-peer review.
In January, we sent a request to the journal to officially withdraw the article that dealt with the difference in types of crimes committed by boys of indigenous origin on the one hand and of Moroccan descent on the other.
Our request was rejected because the magazine preferred an article by us that would go through their own process of peer reviewing.
In February, we offered our long, merciless non-peer review. In June we received the reaction. Our non-peer review was presented for reviewing to two scientists. One of them had simply refused to review it because he/she deemed our criticism to harsh.
The other reviewer was also of the opinion that our article was not suitable for publication, but he/she did discuss the content of our article even though he/she appeared not to have read it very carefully. We wrote it in our original text: a serious problem with the peer reviewing process – perhaps the greatest – is that it takes so much time.
The co-editor of the journal informed us that he partially agreed with the reviewers, but suggested that we would submit an improved article. Changes that would make our article more likely to be published: a) make it shorter, b) remove any wording specifically addressing one of the authors instead of the collective and c) remove any formulation wherein anger shines through.
In June we sent in this shorter version omitting our criticism of T.A.H. Doreleijers; less angry but not less critical, perhaps even slightly more critical.
Our article is an example of a non-peer review. Peer reviewing is onerous for various reasons. One of the pitfalls, ironically, lies in the two different meanings of the word ‘peer’. More precisely: in the different meanings of the concept ‘peer-pressure’.
Ideally, scientific articles are reviewed by honest fellow scientists who are not acquainted with the authors. Ideally the first priority of the reviewers is to protect the integrity of science. In real life these goals are hard to achieve. Apart from a reluctance to point out the errors of colleagues, there is always the pressure of conformity to prevailing paradigms. But most important: reviewing takes a lot of time . We had the opportunity to invest more time in this article than is available for the average reviewer.
While drafting this non-peer review we encountered ever more shortcomings in the reviewed article, ‘Moroccan adolescent suspect offenders in the Netherlands: Ethnic differences in offender profiles’ (Veen, Stevens, Doreleijers & Vollebergh, 2011, from here referred to as PCL11) and in the closely related article ‘Moroccan youth offenders: a class apart?’ (Veen, Stevens & Vollebergh, 2009, from here referred to as NL09).
Relation between both studies
NL09, in a footnote on page 17, mentions the existence of PCL11 thus:
“This chapter  is adapted from Veen, V.C., Stevens, G.W.J.M., Doreleijers, Th.A.H., & Vollebergh, W.A.M. [offered for publication]. Moroccan adolescent offenders in the Netherlands: Ethnic differences in offender profiles.” (emphasis ours. As it is in all of the quoted texts.)
Take note of the change in the title. The published version is titled: “Moroccan adolescent suspect offenders in the Netherlands: Ethnic differences in offender profiles”. There is more to be said about this aspect of the title. ‘Offender profiles’ associates easily with the verb ‘offender profiling’. The subject matter of ‘offender profiling’ or ‘criminal profiling’ (Turvey, 2002), however, is the almost perfect opposite of the gist of PCL11. In his own contribution, ‘Ethics and the criminal profiler’, of this standard go-to text, Turvey is very clear in distinguishing between investigative v. clinical goals:
(…) treatment issues are of little concern to the criminal profiler. (p 581)
This contrasts sharply with the angle of PCL11:
(…) understanding ethnic differences in offending behaviour is highly relevant, since this may have implications for the accessibility of ethnic minorities to intervention and treatment.
Throughout the article itself the authors also use the term ‘offender types’: a better description without the inapt association with ‘profiling’ that could bring with it an undertone of mere suspicion. We return to this subject when we deal with a related third and more serious transgression.
While NL09 writes that it is an adaptation of PCL11 in turn PCL11 alludes to NL09 only through a casual reference, while using a wrong year: 2008 instead of 2009.
Beside the introduction and discussion, NL09 included four chapters: the equivalent of PCL11, a chapter on ‘Emotional and behavioural problems’, one on ‘Mother-child relations’ and one on ‘Orientation on Dutch and Moroccans’.
The last three chapters focus on aspects of the upbringing and attitudes of the Moroccan adolescents that had participated in the study, compared with Dutch adolescents. Those chapters differ on a crucial aspect from chapter two of NL09 and PCL11: both groups of incarcerated adolescents were also compared with adolescents from the population of Moroccan and Dutch in general.
Nowhere in PCL11 was there any mention of the attitude of the parents of the adolescents being determined: that was only part of NL09. Therefore conscientious readers of PCL11 should have been surprised to find the following sentence on page 557, as part of its discussion section:
In line with this, we found that Moroccan adolescent suspect offenders and their parents are more oriented towards Dutch society than non-incarcerated Moroccan adolescents and their parents in the Netherlands (Stevens, Veen, & Vollebergh, 2008).
Some of the conclusions of PCL11 even are at odds with findings in NL09. In PCL11 the authors relate their interpretation of the results found to their worry about this:
(…) Since incarcerated youths of Moroccan origin primarily commit less serious types of offences, this may have consequences for their access to alternative intervention programmes.
The focus is on programmes like Functional Family Therapy: in other words approaching the criminal behaviour as a manifestation of emotional and behavioural problems. The NL09 chapter on ‘Emotional and behavioural problems’, however, establishes that the Moroccan participants show less of those problems than the Dutch, not more.
Misconstruing the words of others
On page 557 PCL11 refers to S. Harchaoui (2001):
Moroccan migrants are different in terms of religion, culture, education and language than Dutch natives and therefore face difficulties in participation in society’s institutions and the development of interpersonal contacts with members of the majority group (Harchaoui, 2001).
The referenced article was written by Harchaoui when he was judge-trainee. He extensively studied the literature on the subject and based his comments on his experience in court.
These are some of the phrases of the author Harchaoui himself (our translation):
Closer examination reveals, however, that what takes place [in the course of the judicial process] is a strategic or even manipulative use of such ‘cultural’ beliefs. (p 101) (…) Moroccan boys know exactly how far they can go. In addition, they choose an identity according to circumstances. (p 110)
So the authors of PCL11 substantiated their claim by referring to an article with a conclusion almost the exact opposite of their own.
Limitations or shortcomings?
The abstract of PCL11 starts with the claim that knowledge about the nature of crime committed by ethnic minority youths is lacking. The study is supposed to fill this assumed gap by comparing the types of crimes committed by incarcerated adolescents of native Dutch and of Moroccan origin.
Criminal record data were used to examine the offence history of 291 incarcerated adolescents. (…) A four-class model of offender types was found: property offenders, violent offenders, sexual offenders and arsonists. Property offenders were mainly Moroccan adolescents, the other offender types consisted predominantly of native Dutch adolescents.
The choice of words reveals a serious misunderstanding about the role of statistical models: statistical models are not found, they are constructed.
The Method section of PCL11 gives a rather extensive and straightforward description of the collected data and on the way the participants had been selected. It fails to emphasize, however, the inadmissibility of its comparisons to the world outside the institutions. A simple indication of the disturbing difference between the sections on this issue, can be found by simply tallying the occurrences of the word ‘participants': 45 times in the Method section and 3 times in the Discussion section. All three were on page 558. The first of these three reads:
The results of this study are subject to some limitations. Firstly, we only included adolescents who were placed in pre-trial arrest in a juvenile justice institution. This means that some incarcerated participants in this study only reached the status of ‘suspect’ and it is unknown whether these adolescents were guilty or may have proven to be innocent.
This is a remarkable way of starting this part on limitations of the study because it hints at serious shortcomings instead. In fact we have arrived at the announced third and worst transgression related to the use of the concept ‘just suspect’.
The statistics that are produced in PCL11 are not solely, not even mainly, based on the index offence (reason for incarceration this time), but on their criminal records as well. Although the data on these offences, coming from two sources, differ in quality, PCL11 does not discriminate between these; rather it mixes these and presents the limitation of part of the data as limitation of the results of the study.
Tables 2, 3, 4 and 5 present the results. Table 3 presents percentages per type of crime for offence history; unsurprisingly both rows (for Dutch and Moroccans) show percentages that add up to more than 100%. Table 2 and 4, however, are on the ‘index offence’ and first offence, and so should add up to 100% or better still: to 154 Dutch and 137 Moroccan adolescents. They do not. In table 2 four Dutch and one Moroccan adolescent are missing. In table 4 two Dutch and one Moroccan is missing. PCL11 mentions on page 548 that:
For eight participants (of 299 participants) data were found to be incomplete and were not further used in the analyses.
PCL11 does not mention that data were missing from the remaining 291 as well. The 291 participants were recruited in cooperation with 10 of the 11 juvenile justice institutions in the Netherlands. During almost two years all adolescents from native Dutch or Moroccan background that entered those institutions were approached. 13% of them did not want to participate. PCL11 gives no information about the distribution of these 45 refusals over Dutch and Moroccan adolescents.
There were other selection criteria that influenced the ‘randomness’ of the sample. Although NL09 is ‘adapted from’ PCL11 and is shorter, it writes about more limitations:
Only boys who stayed at least one week in preventive detention in a juvenile justice institution, were allowed to receive visitors and had no intellectual disability, were asked to participate in the study by an employee of the institution. (…) Of all the boys who were asked to participate in the study by an employee of the institution, 66% were interviewed. Most boys who were asked to participate in the study, but ultimately did not cooperate, were suspended before the interview could take place. Approximately 13% of the requested boys refused to participate.
Both emphasized limitations were not mentioned in PCL11.
From offences to perpetrators
The classification of types of crime, used to categorize the offences, was not the one used in official statistics (CBS, WODC ). Instead a classification, adapted from Van Kordelaar (2002), was used by the authors. Psychologist Van Kordelaar designed a tool with the explicit goal to support courts in deciding about the mental health of suspects in relation to their criminal behaviour. The complete title of the study was: ‘Decision Support Mental Health Examination: In the Criminal Law for Adults’, suggesting that this classification was not the obvious choice for a study involving only adolescents. In the lists of references, in NL09 as well as PCL11, by some mischance, the part after the colon has gone missing from the title. An unfortunate situation, but a more serious bother with usage of this classification is the fact that in PCL11 the mental health of the incarcerated adolescents was not a topic at all. The broader study NL09 does have a chapter about ‘Emotional and behavioural problems’, but that chapter contains no link to this classification. So this classification played a role only in the part of the study that focused on differences in the character and severity of the crimes committed by participants from the two groups. However, in the Van Kordelaar classification: ‘indecent exposure’ is in the same category with ‘rape’ and ‘sexual intercourse with a child’. ‘Setting fire to a waste container’ is in the same group with ‘arson causing mortal danger’. So the classification used, clearly was not based on the severity of the crimes committed, while the conclusions of PCL11 are in terms of severity.
The data on crimes, categorized into seven offence types, that were derived from the adapted Van Kordelaar classification, were converted to four classes of perpetrators. An unfortunate side effect of this conversion is that it removes the impact of the significant difference in the amount of offences per participant (PCL11 p 551). The conversion has been achieved by applying a ‘latent class analysis’ (LCA). The idea behind the latent class analysis is to identify latent classes: classes that are not manifest. That is not the way LCA is used here. Here, in the words of John Uebersax, “LCA simply serves as a convenient data-reduction tool” (2000-2010). Seven manifest classes are being regrouped into four of them. Ideally in a LCA the conditional probabilities will be close to 0 or 1, indicating that the items discriminate well across the classes and ensuring that the condition is met that the manifest variables have nothing in common after controlling for the latent variable. For the classes ‘sexual offenders’ and ‘arsonists’ the conditional probabilities are indeed close to 0 or 1.
The adolescents who are categorized as ‘property offenders’, however, had a probability of 0,9 of having committed a property offence but they also had a probability of 0,5 to have committed one or more crimes in the offence category ‘property with violence’. The highest conditional probabilities for the ‘violent offenders’ was a mere 0,7 and there were probabilities of 0,3 of 0,4 and 0,55 too.
The reason to perform this reclassification in the first place was: determining ‘offender types’. As part of the broader study NL09 this approach might have been appropriate. The introduction of PCL11 accounts for it thus:
(…) scientific knowledge on the differences in offender profiles between ethnic minority and majority youths is the first vital step in unravelling the mechanisms leading to delinquent behaviour in ethnic minority youths.
Knowing the broader context of PCL11, being part of NL09, one would expect that these offender profiles would play a role in the other chapters of the latter. Much to our surprise ‘offence profile’ (Dutch: ‘delictprofiel’) is not even mentioned in the other chapters: so this classification is exclusively used in the context of presenting differences in the types of crime the incarcerated adolescents had committed. We will elaborate on this subject at the end of the section on the presentations of the data.
The pragmatic approach of judges
A real and serious limitation was formed by the restriction to suspected adolescents that were incarcerated before trial. In PCL11 itself we can read that the decision to incarcerate these suspects, is based on a variety of considerations. This limitation is more important and more complex than it might seem at first sight. The prevalence of pre-trial detention has grown ‘explosively’ in the Netherlands, in some years making up almost 50% of the total prison population (Buruma, 2005). In fact there are serious concerns that this trend results in confronting courts with ‘faits accomplis’ where they possibly feel urged to adapt the sentence to the time the suspect has spent in detention before trial (Stevens, 2010). In fact some judges interviewed in this study suggest that pre-trial detention is an explicit policy from prosecutors to ensure that offenders get imprisoned at all. A selection based on being subject to pre-trial detention most certainly is not a random one.
It is highly probable that the different situations at home for adolescent suspect offenders from Moroccan and Dutch descent, plays a distinctive role in the decision to place the suspect in pre-trial detention or not. Prosecutors and judges decide about pre-trial detention in a pragmatic way (Wermink, De Keijser & Schuyt, 2012). This is a manifestation of an established tension between the theories of criminologists on the one hand and the practice of judges and others in the judicial system on the other.
Research of others
As mentioned before, the abstract of PCL11 starts with the claim that knowledge about the nature of crime committed by ethnic minority youths is lacking. One could argue that the authors merely display some boldness here by claiming that this research is lacking. The problem is that they actually refer to an article about that type of research and have the audacity to distort the results of that research to support their own main claim in PCL11. This is what the authors write about this study on page 547:
(…) showed that adult Moroccan offenders are often suspected of property offences (non-violent and involving violence), threat, vandalism and disorderly conduct, and to a lesser degree of more serious violent or sexual offences (Jennissen & Blom, 2007).
So a reference is made to a study by Jennissen and Blom. Contrary to the claim, that study was not just on adult Moroccans. In fact one of the most important and useful features of this study was that it differentiated on ages. With permission of the authors we reproduce three figures from that study as support of the main point of our argument, which can be found in the next section. (The translation of the captions is ours.)
Picture 2 zooms in on one type of crime: property with violence. The reason for this choice will be explained later. Here the figure for Moroccans is almost four times the one for Turks, more than 26 times the one for Dutch natives. Picture 3 is about the same type of crime. Here a differentiation is made over age groups. Note that only men were included. A selection was made from the majority/minority groups: the two groups with the highest crime rates, two other minority groups and the native Dutch were selected.
At the age of 18 the figure for Moroccans is almost 37 times as high as the one for the Dutch natives. For the age group 18-20 this ratio is a little over 37.
Abuse of statistics
It is amazing that the authors referred to Jennissen & Blom, 2007 the way they did. We are not going to elaborate on this, but instead focus on what could possibly explain the huge difference between the figures reported by Jennissen & Blom on the one hand and the claims in PCL11. In this section we describe some factors that could contribute to this difference.
Some of the factors relate to the ‘samples’ used. There was a huge difference on the size of the sample and there were smaller ones on the time base of the data, the type of selection of ‘participants’ and the categories used for different types of crime.
The categories of crimes used in both studies were similar but not equal. Jennissen and Blom combined data from the police recognition system with data from the municipalities. In this way they could include almost 200.000 offenders, almost 80.000 of them in the age range 12-24. PCL11 is based on a specific sample, consisting of 291 adolescents, incarcerated between May 2006 and February 2008.
With regards to when the offences were committed, one should remember that the crimes of the incarcerated adolescents that were subject of PCL11, were from their complete police records, not from the index offence. Therefore the time bases do not differ very much. A more recent study by Jennissen (2009) based on data from 2005 suggested that the over-representation of Moroccans and other minorities with unfavourable crime rates, was decreasing. The most recent data, however, shows that this trend unfortunately did not last (Van Rosmalen, Kalidien, & De Heer-de Lange; 2011)
The effect of the differences resulting from the other factors is more ambiguous. Together these factors could explain a part of the differences found, but it cannot explain all of it.
The substantial difference in the results is not found in the data as such, however, but in the way they are represented. The misrepresentation bears some similarity to the so-called ‘Simpson’s paradox’ (Pearl, J; 2009). Warnings about this nasty pitfall can be found in introductory statistics books. An example from social science of the great dangers of superficially interpreting statistical findings, goes by the name ‘Berkeley gender bias case’. The situation at hand bears similarity with this paradox, but there are also important differences. In the ‘Berkeley gender bias case’ the bias turned out to be in the other direction than was claimed, but was still small, although significant. The crime-rates of adolescents of Moroccans and Dutch descent however, differ markedly. Another distinction was that at Berkeley initially only cumulated data were taken into consideration. After more detailed study, zooming in on the figures, the misrepresentation could be explained and undone. In the case of PCL11 however, the focus of the study from the very beginning was on the differences between the types of crime the incarcerated Dutch and Moroccan adolescents had committed.
The logic is so totally flawed that it deserves to be mentioned in new editions of the aforementioned books on introductory statistics. This is a harsh judgement indeed, so we will elaborate on it in the next section.
Different presentations of the same data
We start with comparing table 5 from PCL11 with table 2.2 from NL09. Both tables are the concluding ones of the passages about the offender types. Rows and columns have the same captions and the figures are based on the same data. There is a remarkable difference however. In NL09 we find percentages per type of crime in rows, in PCL11 in columns. In table 1 we show the effect of this transformation.
The first two rows with ‘ratio M/D’, correspond with the presentations in NL09 and PCL11: based on percentages within the incarcerated groups, not of the populations. The third and last row with ‘ratio M/D’ is based on the same real numbers but here they are presented as percentage of the adolescents in the populations in general: taking into account that the Moroccans constitute less than 4% (22000 out of 576000, see table 1).
The presentation in the tables and in the discussion section of NL09 and PCL11 were based on ratios of ratios. In effect it were these ratios of ratios of incomparable ‘samples’, that made it to television shows and newspapers, where the findings were generalized to the world outside the institutions. The authors were aware of the political sensitivity of the subject of the over-representation of ethnic minority youth in crime and should have been aware of the danger of this generalizing. The last row of ratios, the more appropriate one, shows a disturbing over-representation for ‘arson’, a serious over-representation for the classes ‘sexual’ and ‘violence’ and a huge over-representation for ‘property’.
Table 2 presents the data for the real numbers of offences, not -constructed- types of offenders. The data are derived from table 3 in PCL11: ‘offences (based on offence history) by ethnic group’. Through the latent class analysis, the violence used in property crimes was swept under the carpet. In a way table 3 is the result of ‘rolling back’ the LCA.
The ratios for the four separate types of violent offences, i.e. ungrouping the middle column of table 2 to the constituent parts, would be 45, 15, 16 and 12.
We contacted the authors about the misrepresentation through NL09 and the appearance of G. Stevens on television. The authors did not respond. March 2010, in a letter to the commission on scientific integrity (CWI), Stevens admitted that the representation left the possibility of being misunderstood, but claimed that this risk was small . In spite of this acknowledgement in PCL11 this risk is inflated instead of deflated compared to NL09. Note that PCL11 was published a year after the acknowledgement.
We will not go into details about NL09 but will focus on the way PCL11 made the generalization from the incarcerated participants to society:
Moreover, the finding that Moroccan adolescents are far more often incarcerated for property offences than Dutch native adolescents, which could be seen as a relatively less serious type of crime, is of societal importance. (…) Firstly, for the public opinion on crime in ethnic minority groups, it is important to make known that although Moroccan adolescents are more often incarcerated than Dutch native adolescents, their criminal behaviour overall is less serious than the criminal behaviour of Dutch native youths.
This is a fallacy. The fallacy is somehow substantiated by the gist of the worry we mentioned before, about access to alternative intervention programs. The authors refer to two of those programs, Functional Family Therapy and Multisystemic Therapy, but then continue their argument with:
Participation to these intervention programs is hindered by longer pre-trial incarceration and imprisonment periods. Since incarcerated youths of Moroccan origin primarily commit less serious types of offences, this may have consequences for their access to alternative intervention programs.
So in effect the authors suggest not to incarcerate Moroccan offenders because they commit less serious crime. That suggestion is based on ratios of ratios. When the real numbers are taken into account it is clear that the huge over-representation of the Moroccan adolescents in ‘less serious’ types of crime ‘only’ overshadows the over-representation that can also be found in the more serious types.
Types of crime that do or do not pose a threat to community
The last paragraphs of the Discussion section of PCL11 are by far the most inappropriate, the most political ones. The following casual remark brings us back to the reason why, in picture 3, we chose to zoom in on ‘property with violence’.
(…) youth offenders who commit less serious forms of crime and do not directly form a threat to the community (…) (p 559)
Picture 5 is not directly from Jennissen & Blom 2007, but based on the same data.
The division of all ratios by the one for the Dutch natives provides the ‘baseline’, below left. This picture does not represent samples, so this ratios of ratios give rise to less distortion than the ones from NL09 and PCL11. The picture paints a sad situation indeed but one feature really stands out: by far the greatest difference between the groups was found for the crime type ‘theft with violence’. Note that this picture is based on the data for all ages; the feature is even more pronounced if one focuses on adolescents.
While since a couple of years the total number of crimes in the Netherlands is decreasing, ‘property with violence’ increases. An important portion of this type of crime involves robbery. The improvement of security measures of banks and shops have given rise to an increase of robbing people in their own houses. This contrasts sharply, in our opinion, with the promulgation “do not directly form a threat to the community”.
Meanwhile the picture also turns the spotlight on a remarkable omission of PCL11: the study included only one minority group.
The authors have dismissed valuable observations and recommendations of people like S. Harchaoui. This is not helpful for what is necessary: facing the problems the individual offenders cause and experience. It does not help in acknowledging the attitude and achievements of non-criminal Moroccans either. Neglecting the difference between minorities is insulting especially to other minorities who integrate less but behave better (Wing Che Wong, 2004):
So we cannot speak about an integration paradox, because the acceptation (sic) of the Chinese people by the Dutch population is not correlated to integration.
Abusing a branch of science, as pretense for a socio-political agenda is not only harmful for society; when uncovered, it has a detrimental effect on public trust in that branch of science. This may cause past good research to be disregarded, and future good research not to be executed at all.
We got useful suggestions on a draft version from Marco Bouman, policy advisor of the Rotterdam city council faction for the party ‘Leefbaar Rotterdam’, and from Roel Jennissen, researcher at the WODC, Wetenschappelijk Onderzoeks- en Documentatie Centrum (the research centre of the Justice Department of the Netherlands).
Buruma, Y. (2005) De explosieve groei van de gevangenispopulatie [The explosive growth of the prison poplation]. Buruma, Y, Vegter, P.C. (eds.), Terugkeer in de samenleving. Opstellen ter gelegenheid van het afscheid van Prof.dr. J.P.S. Fiselier van de Radboud Universiteit Nijmegen en de Rijksuniversiteit Groningen (pp. 49-65). Deventer: Kluwer.
Eggen, A.Th.J. (CBS), Kalidien, S.N. (WODC) (eds.) Criminaliteit en rechtshandhaving 2007 (2008) [Criminality and law enforcement] Boom Juridische Uitgevers.
Harchaoui, S. (2001). Hedendaags kwaad revisited: Kanttekeningen bij de Marokkaanse criminaliteit in Nederland [‘Evil nowadays’ revisited: Remarks on Moroccan criminality in the Netherlands]. Justitiële Verkenningen, 27, 99-114.
Jennissen, R.P.W. (2009) Criminaliteit, leeftijd en etniciteit. Over de afwijkende leeftijdspecifieke criminaliteitscijfers van in Nederland verblijvende Antillianen en Marokkanen. [Crime, age and ethnicity. About the deviant age-crime curves for Antilleans and Moroccans residing in the Netherlands]. The Hague: Wetenschappelijk Onderzoek- en Documentatiecentrum. Boom Juridische Uitgevers.
Jennissen, R.P.W., & Blom, M. (2007). Allochtone en autochtone verdachten van verschillende delicttypen nader bekeken [A closer look on suspects of native Dutch origin and other ethnic groups on various types of offences]. The Hague: Wetenschappelijk Onderzoek- en Documentatiecentrum. Boom Juridische Uitgevers.
Pearl, J. (2009) Causality: Models, Reasoning and Inference. New York: Cambridge University Press
Stevens, G.W.J.M., Veen, V.C., & Vollebergh, W.A.M. (2008). Marokkaanse jeugddelinquenten: Een klasse apart? [Moroccan juvenile offenders: a class of their own?]. The Hague: Nicis Institute.
Stevens, L. (2010). Voorlopige hechtenis en vrijheidsstraf. De strafrechter voor voldongen feiten? Nederlandsch Juristenblad, 85(24), 1520-1525. [Refereed version: Pre-Trial Detention: The Presumption of Innocence and Article 5 of the European Convention on Human Rights Cannot and Does Not Limit its Increasing Use] European Journal of Crime, Criminal Law and Criminal Justice, 17, 165-180
Suurmond, G., Van Velthoven, B.C.J., (2008) Werkt gevangenisstraf echt niet? [Is longer imprisonment really ineffective?] Justitiële Verkenningen, 34-2, 27-45.
Turvey, B.E., Ethics and the criminal profiler. Turvey, B.E., (eds.), (2002) Criminal Profiling: An Introduction to Behavioral Evidence Analysis. 573-587
Uebersax, J.S. (2000-2010) LCA Frequently Asked Questions. Weblog, not in print. URL: http://www.john-uebersax.com/stat/faq.htm
Van Kordelaar, W.F.J.M. (2002). Beslissingsondersteuning onderzoek geestvermogens [Support in decision making in forensic psychiatric assessment]. Deventer: Kluwer.
Van Rosmalen, M.M. (CBS), Kalidien, S.N. (WODC) & De Heer-de Lange, N.E. (CBS) (eds.) Criminaliteit en rechtshandhaving 2010 (2011) [Criminality and law enforcement] Boom Juridische Uitgevers.
Veen, V.C. & Stevens, G. & Doreleijers, T.A.H. & Volleberg, W. (2011)
Moroccan adolescent suspect offenders in the Netherlands: Ethnic differences in offender profiles. Psychology, Crime & Law, 17 (6), pp. 545-561.
Wermink, H, De Keijser, J, Schuyt, P, (2012). “Verschillen in straftoemeting in soortgelijke zaken” Nederlands Juristenblad (NJB) 2012-11:726-733
Wing Che Wong, (2004). Islamic and Chinese minorities as an integration paradox? Utrecht University.
 Groenendijk, F.A, (2012) Peer-reviewing en het aanzien van de wetenschap. Een pleidooi voor non-peer reviewing [Peer-reviewing and the reputation of science. A plea for non-peer reviewing]. Blogpost, not in print. URL: http://www.keizersenkleren.nl/?p=48
CBS is the abbreviation for Centraal Bureau voor de Statistiek: the official statistics bureau of the Netherlands on a lot of data. WODC is the abbreviation for Wetenschappelijk Onderzoeks en Documentatie Centrum: the scientific bureau of the Department of Justice.
URL to the letter in pdf-format: http://www.keizersenkleren.nl/wp-content/uploads/2012/11/100310-Stevens-reactie-op-CWI-UU-voorgenomen-conclusie.pdf