1.1 Understanding Writing Quality via Quantitative Linguistic Features

There has been a long-standing interest in understanding rater judgements of writing quality in first and second language research. This interest has adopted several theoretical and methodological lenses (e.g., see the overview in Reference Durrant, Brenchley and McCallumDurrant et al., 2021). One popular lens used to tap into these judgements and the inferences we can (and cannot) make from them has been the quantitative study of the relationship between linguistic features and writing quality grades. Under this lens, linguistic features are identified (normally by adhering to a specific theoretical framework that governs how to identify the features) and counted (manually or automatically with corpus software), and then relationships between these frequencies and writing quality grades are established numerically using statistical techniques such as correlation and regression analyses. Writing quality grades represent subjective ratings made by text evaluators who largely make their judgements from a predetermined set of criteria which to an extent presupposes what ‘good’ writing involves. These criteria therefore guide evaluators in their judgements (e.g., see the IELTS and TOEFL grade bandings mentioned in Reference Durrant, Brenchley and McCallumDurrant et al., 2021). This quantitative approach has enjoyed sustained popularity in the literature and is currently experiencing something of a ‘boom’, thanks to the increasing creation of corpus software tools which make the counting and analyses of such features increasingly user-friendly. This boom is well documented across overviews provided in Reference Durrant, Brenchley and McCallumDurrant et al. (2021) and Reference CrossleyCrossley (2020).

In their studies, researchers make two key assumptions. First, there is an assumption of, or perhaps appreciation for, the role that linguistic features themselves may play in our understandings of writing quality judgements as a measurable construct. This means there is an underlying belief that by counting linguistic features and looking at their relationships with writing quality via statistical methods, we can learn something about how these features may be being judged/perceived by raters. Second, and linking back to the first assumption, is the belief that the linguistic features chosen are (a) worth counting (because they have an established linguistic history/history in models of writing proficiency/quality), and (b) that they can indeed be reliably counted.

Findings of past feature-writing quality work have gone on to inform two often connected areas of research: the development of writing proficiency scales/rubrics by referring to differences in linguistic feature use across bandscales (e.g., Reference Hawkins and FilipovicHawkins & Filipovic, 2012), and/or the training of large-scale feedback and grading systems (e.g., Reference Chen, Zhang and BejarChen et al., 2017). Researchers have most persistently studied features of grammar (e.g., clauses (see Reference Bulté and HousenBulté & Housen, 2014)) and vocabulary (e.g., percentage of words appearing in the Academic Word List (Reference Daller, Turlik, Weir, Jarvis and DallerDaller et al., 2013)), with features of cohesion occupying an inconsistent position of interest for researchers (e.g., see the review in Reference Durrant, Brenchley and McCallumDurrant et al. (2021)). Features of phraseology, for example lexical bundles (e.g., see Reference Appel and WoodAppel & Wood, 2016), occupy an increasingly prominent position, especially in second language literature (e.g., see reference made to this emerging importance in Reference Durrant, Brenchley and McCallumDurrant et al. (2021) and Reference PaquotPaquot (2018, Reference Paquot2019)).

It is this latter linguistic area that this Element focusses on. The following sub-sections make an explicit case for the study of one specific area of phraseology: that of collocation. The sub-sections present the rationale for such a focus and explain how this Element contributes to understanding the role collocation may play alongside several non-linguistic writing assessment variables in shaping writing quality judgements.

1.2 The Rationale for Studying Collocations and Writing Quality

The use of appropriate language is viewed as a key component of success for meeting programme outcomes in the FYC programme our Element focusses on (CWPA, 2014; CWPA et al., 2011). In this sense, using appropriate academic language is therefore a requirement of fitting into students’ respective academic disciplines/communities. Reference Wray and BrownWray (2006, p. 593) notes on this matter that ‘when we speak, we select particular turns of phrase that we perceive to be associated with certain values, styles and groups’, with the learning of these phrases or word combinations acting as a badge of identity and this badge is linked to particular academic communities.

Later, Reference Wray, Siyanova-Chanturia and Pellicer-SánchezWray (2019, p. 267) emphasises that the status of a word combination as a formula lies in the decision-making or perceptions of the agent. She states that a formulaic sequence is ‘any multiword string that is perceived by the agent (i.e., learner, researcher, etc.) to have an identity or usefulness as a single lexical unit’. Reference Siyanova-Chanturia, Pellicer-Sánchez, Siyanova-Chanturia and Pellicer-SánchezSiyanova-Chanturia and Pellicer-Sánchez (2019, p. 6) highlight that there are several frameworks or reasons that guide the perceiver’s decision-making. It may entail high frequency of occurrence (since frequently produced strings, other than being useful by virtue of being frequent in language, may also benefit from being treated as a single unit), a teacher’s perceived value of a string (no matter how frequent), some sort of basic holistic storage and processing, a specific pragmatic function, or, indeed, something altogether different.

Although these definitions allow researchers flexibility in their theoretical and methodological approaches to capturing formulas, two particular approaches have dominated the literature: phraseological and frequency-based approaches (see Reference NesselhaufNesselhauf (2005) for an in-depth overview of the differences between these lenses). This Element grounds its theoretical and methodological conceptualisation of collocation in the frequency-based lens. Under a frequency-based lens, the tenets of collocation rest on understandings from Reference Firth and PalmerFirth (1968, p. 181), who believed that part of a word’s meaning is the ‘habitual collocations’ in which it appears. Meaning here is said to include both the concept with which the word is associated and the ways in which it is used. Reference Firth and PalmerFirth (1968) gives the example of ‘dark night’, resting his understanding on the belief that collocating words are part of each other’s meaning. Thus, because dark appears frequently alongside night, collocability with night is one of the meanings of dark (Reference FirthFirth, 1957, p. 196). Later, Reference Firth and PalmerFirth (1968) articulates these thoughts further to indicate that collocation is a type of mutual expectancy between words. Collocating words are said to predict each other, in the sense that the presence of one word makes the presence of the other more likely.

Corpus linguists have unpacked this mutual expectancy further by stating that collocation is ‘the relationship a lexical item has with items that appear with greater than random probability in its (textual) context’ (Reference HoeyHoey, 1991, p. 7), with Reference Jones and SinclairJones and Sinclair (1974) simply stating that words are collocates if they appear together more frequently than their individual word frequencies would predict. Under these views, there is also a psycholinguistic nature to collocation to consider, with Reference Sinclair, Steele and ThreadgoldSinclair’s (1987, p. 391) idiom principle setting out this mental association where ‘a language user has available to him or her a large number of semi-preconstructed phrases that constitute single choices, even though they might appear to be analysable into segments’. Further still, Reference HoeyHoey’s (2005, pp. 3–5) theory of lexical priming also draws attention to the psycholinguistic nature of collocation in that it is seen as the ‘psychological association between words … evidenced by their occurrence together in corpora more often than is explicable in terms of random distribution’. Under these guiding thoughts then, collocation is bound up in the idea that word combinations are more frequent than their individual word frequencies would explain and that there is a degree of non-random use to these pairings.

These thoughts have led researchers to develop multiple taxonomies and dictionaries of collocations (e.g., Reference Benson, Benson and IlsonBenson et al., 2009) with studies presenting word combinations such as (i) ‘heavy rain’, (ii) ‘rancid butter’, and (iii) ‘apologise profusely’, as collocations (e.g., Reference PaquotPaquot, 2018). Under the frequency-based school of thought, researchers have commonly captured collocations like these by focussing on the belief that collocations are pairs of words which regularly co-occur within a given span or window of text, for example two to four words either side of the node/search word. This approach has been criticised as capturing syntactically unrelated or uninteresting collocations (e.g., Reference Evert, Lüdeling and KytöEvert, 2009). More recently a smaller group of studies have identified collocations using syntactic parsers which capture collocations according to a particular syntactic dependency relationship. Dependency pairings are decided by parsing a text using an automated parser. Dependency grammar operates on the notion that in every sentence each word is dependent on another, apart from the root of the sentence which is independent (Reference DebusmannDebusmann, 2000). A word depends on another if it is a complement or a modifier of the latter. For example, dependency pairings which might be collocations include an adjective modifying a noun.

1.3 Emerging Questions from Current Studies

1.4 The Organisation of the Element

This Element proceeds by providing an overview of the FYC context in Section 2. Section 3 then guides readers through the current collocation-grade landscape and emphasises how the empirical work in the Element adds to this landscape. Section 4 sets out the methodological steps taken in the three individual studies. Then, Section 5 describes the results of the cluster analysis carried out to answer the first question, while Section 6 describes the results of the mixed effects modelling, carried out to partially answer the second question. Section 7 also helps answer the second question by qualitatively unpacking the possible reasons for the statistical relationships between the measures of collocation and writing quality by looking at text samples from the FYC corpus itself. Section 8 concludes the Element by summing up the key findings and limitations, and importantly how we reflect on our methodological approach and its promise in future work.

2.1 Overview of the Section

This section will explain the rationale for focussing on a FYC programme in the United States. The section will explain how these programmes may benefit from closer engagement with language instruction. We chose to base our study on the programme at the University of South Florida (USF) because of its focus on different writing tasks.

2.2 The Nature of the FYC Programme at USF

The University of South Florida is a large public university with a diverse student population. Of its 50,000 students, as many as 41 per cent identify as African American, Black, Asian American, Hispanic, Native American, or multiracial (USF, 2018). The university provides degrees in business, engineering, arts and social sciences, and interdisciplinary sciences ( USF, 2018).

As a state requirement, students who enter a Florida College or University State system have been required since Reference Aull2015–16 to complete thirty-six hours of general education coursework from a list of courses in communication, mathematics, social sciences, humanities, and natural science, among others. This requirement means students develop the academic and numeracy skills needed for the demands of university study.

In the FYC programme, students complete writing as a ‘process’. They develop strategies in pre-writing, co-authoring, revising, and editing, as well as learning to follow academic/disciplinary conventions for different genres. They must achieve a minimum C-grade to continue their studies.

The programme’s learning objectives are set out across two modules: ENC (English Composition) 1101 and ENC 1102. Some of these objectives include the following:

• Learning and applying strategies to facilitate a range of skills, including critical reading, the stages of process writing, and giving peer feedback.

• Composing academic genres and adhering to academic conventions (structure, citation, and linguistic features).

• Synthesising disparate or conflicting thoughts when evaluating questions/problems to form cohesive and collaborative solutions.

2.3 Teaching, Evaluation, and Feedback at USF

Modules are taught by permanent staff, adjunct instructors, and Graduate Teaching Assistants (GTAs). The ethos on the programme is that writing is constructed at a community level. This means peer review and teacher-led writing conferences feature heavily. Writing is commented on and evaluated using ‘My Reviewers’, a bespoke learning management system (LMS) which allows instructors and students to view programme material, draft and final projects, and to give peer and instructor feedback via PDF annotation tools.

Each of the six projects is worth between 20 and 30 per cent of students’ overall grade. Students are awarded the remaining percentage of their grade for homework tasks and class participation, equalling 100 per cent. Projects are evaluated using custom-made rubrics which instructors are trained to use. These rubrics evaluate projects according to analysis, use of evidence, organisation, focus, and style. An overall holistic grade out of fifteen points is awarded, expressed by the letters A–F. These bandings are shown in Table 1.

2.4 Language Instruction in FYC Programmes

3.1 Overview of Section

This section reviews relevant literature to set out the research landscape of collocation-grade studies. In doing so, we create a space for our work to contribute to filling identified gaps and addressing issues we raise in this area.

3.2 Definitions of Collocation and Methods of Identification

The frequency-based approach to defining and identifying collocations is influenced by two guiding principles: recurrence and co-occurrence. Recurrence is the repetition of the same word combination by a language user or a group of users (Reference Ellis, Meunier and GrangerEllis, 2008). It can be captured by looking at frequencies of word pairings in corpus data (Reference EvertEvert, 2004). Co-occurrence is an attraction between two words, captured by their appearance together more often than their individual frequencies would predict (Reference EvertEvert, 2004). As Reference Evert, Lüdeling and KytöEvert (2009) notes, the mere repetition of a word pair is not a sufficient indicator of a strong attraction between the words; a pair may be frequent without there being a strong attraction between the individual words. This is seen, for example, when a combination is frequent, but its component words are able to take many other partners. Reference Schmitt and SchmittSchmitt and Schmitt (2020) illustrate that the word ‘the’ co-occurs with almost every non-proper noun and thus does not have strong attraction to other words. Reference Schmitt and SchmittSchmitt and Schmitt (2020, p. 5) also explain that some words co-occur with only a small number of other words. The word ‘blonde’, for example, occurs almost exclusively with ‘hair’ and a few other nouns such as ‘woman’ or ‘lady’. We produce ‘blonde hair’, ‘blonde woman’, or ‘blonde lady’ but never ‘blonde wallpaper’ or ‘blonde paint’ (italics in authors’ original), although these latter combinations are syntactically and semantically possible.

Reference Firth and PalmerFirth (1968) discusses attraction as relating to the ‘mutual expectancy’ of words, while Reference SinclairSinclair (1991) draws on the ‘mutual choice’ that words seem to be subject to. These ideas of predictability and chance were also earlier captured in the work of Reference OsgoodOsgood (1952):

Reference SeretanSeretan (2011) ties these notions of frequency and predictability to key principles of statistics, namely: tendency and typicality. She highlights how scholars have defined a collocation as a typical, specific, and characteristic combination of words which are arbitrary, recurrent word combinations. Bringing these views together, collocations therefore comprise two or more words that appear near each other in a recurrent manner, and that co-occur more often than could be explained by random chance.

Traditional methods of capturing collocation have been termed positional as they operate under a span approach. This span is set by the researcher or by the software they are using and has been classically stated as four words to the left or right of the search word. However, this span of four words has been challenged (e.g., Reference SmadjaSmadja (1993), who uses five words). In many second language studies, it has been set at one or two words to only capture adjacent pairs (e.g., Reference BestgenBestgen, 2017; Reference Bestgen and GrangerBestgen & Granger, 2014; Reference Granger and BestgenGranger & Bestgen, 2014). Reference SeretanSeretan (2011) recognises the dangers of a span approach because it may capture syntactic noise – that is, word pairs which have no syntactic relationship. This is shown in example [1], whereby a span approach might capture ‘human rights’, and ‘human rights organisations’ as pairings but also the unrelated ‘human organisations’.

1. [1] Human rights organisations

Reference SeretanSeretan (2011) also notes that the span approach would fail to capture pairs if they fell outside the span boundary, as with ‘problem solved’ in example [2]:

1. [2] The problem is therefore clearly a deeply rooted one and cannot be solved without concerted action by all parties.

Recognising these limitations, others have recommended capturing syntactically related combinations by using automated tools such as the Stanford parser (Reference Manning, Surdeanu, Bauer, Bontcheva and ZhuManning et al., 2014). These parsers can capture multiple dependencies such as adjectives pre-modifying a common noun, adverbs modifying an adjective. These methods have the advantage of retrieving combinations efficiently. However, their accuracy with learner writing has only recently started to be documented, so researchers need to exercise caution in their use (e.g., see Reference Durrant, Brenchley, Szudarski and BarclayDurrant & Brenchley, 2021; Reference Huang, Murakami, Alexopoulou and KorhonenHuang et al., 2018; and the special issue on working with learner data edited by Reference KyleKyle (2021)).

The sub-sections that follow outline the vast array of association measures that have been used to capture the property of co-occurrence, but importantly have not been acknowledged much in first or second language writing studies. A full discussion of each measure’s respective formula is beyond the scope of this Element; however, full details of the formulas are provided in the supplementary material at: https://leemccallum.net/resources

3.3 Phraseological Complexity and Its Measurement

3.4 How Non-linguistic Variables May Shape Writing Quality

4.1 Overview of the Section

The literature review in the previous section highlighted two key issues. The first concerns capturing different properties of collocation and leads us to ask: how can we choose appropriate measures of collocation? The second concerns how we can understand the complexity of what shapes writing quality scores, and leads us to ask: how can we measure the potential role of different linguistic and non-linguistic factors in an appropriate way?

This section describes the methodologies underlying the three studies we carried out to answer these two questions. The first study addressed question one. It focussed on carrying out a cluster analysis of a set of association measures to determine the extent to which they measured different properties of collocation. The second study addressed the next question. It focussed on carrying out a type of complex logistic regression for ordinal response variables. In our case, the ordinal response variable was the set of FYC writing quality grade scores. This modelling included several fixed and random effects/predictors. Linguistic predictors included the collocation measures from the cluster analysis described below and a measure of collocation diversity. Non-linguistic predictors comprised the writing task types, the language status of the student writers, and random individual raters and writers. The latter set of predictors were used to capture the complexity of the grading context as set out in Sections 2 and 3. The third study adds an additional level of understanding around what shapes writing quality scores by carrying out a qualitative analysis of the linguistic predictors to shed light on particular patterns that may explain the statistical results uncovered in the second study.

The sub-sections that follow provide details of our study and reference corpora, the rationale for our selection of predictors, and the technical methodological steps we took in each of the two studies.

4.2 The FYC Corpus and Using MICUSP as a Reference Corpus

The study corpus comes from a sample of FYC argumentative essays produced by L1 and L2 writers from USF’s first-year writing programme. The study corpus comprises texts from Project 3 of ENC 1101 and Project 1 of ENC 1102 (see Section 2) because both focus on extensive essay writing, with Project 1 directly building on Project 3. Project 3focusses on setting out and describing key arguments of different stakeholders, while Project 1 considers these key arguments and outlines how stakeholders can reach a compromise (see Section 2 for comprehensive project details).

A breakdown of the corpus is shown in Table 2.

A breakdown of the corpus in terms of grades is also provided in Tables 3 and 4. Both tables show that across the modules, there are a high number of students achieving top A grade bands (scored as 13–15 points out of a total of 15) with fewer students receiving B grades (scored as 12–10 points out of a total of 15), and fewer still only just achieving passing C grade bands (scored 9–7 points out of a total of 15).

To calculate the association measures, MICUSP ( Michigan Corpus of Upper-Level Student Papers, 2009) was chosen as the reference corpus. A reference corpus is a larger corpus of language which is used as a comparison to the study corpus. In this case, a reference corpus is used to calculate the association measure values. Each dependency from the FYC ‘study’ corpus is looked up in MICUSP and from the frequency information in MICUSP, the association measure value is calculated. MICUSP was chosen because it contains similar tasks and topics to the tasks FYC writers would be expected to complete in their academic studies after completing the FYC programme. These tasks included argumentative writing like the FYC module tasks but also included literature reviews, lab reports, critical summaries of texts, and empirical research reports. Tasks were completed by L1 and L2 writers across multiple years of study and disciplines, including biology, education, economics, history, and engineering. Topics vary but include ‘fruit fly experiments’ and ‘standardised testing in education’. The use of the corpus complied with the MICUSP Fair Use Statement ( Michigan Corpus of Upper-Level Student Papers, 2009), and the formatted texts that were used to compute association measures were solely based on anonymised metadata.

Genre types were decided by two raters (Reference Römer and O’DonnellRömer & O’Donnell, 2011). Forty-four per cent of texts were reports, 22 per cent of texts were argumentative writing, 17 per cent were research papers, 7 per cent were critiques, 6 per cent were proposals, 3 per cent were response papers, and 1 per cent was creative writing. A second reason for choosing MICUSP related to its size, with the corpus itself representing one of the largest corpora of student writing across multiple levels of study (undergraduate and graduate), disciplinary subject, and language backgrounds (in alignment with the study corpus), MICUSP also includes non-native student writers (e.g., Arabic, Chinese, and Spanish L1 speakers). The access to 2.8 million words of student writing contained within MICUSP was also a draw in choosing a reference corpus as it is a valid representative of authentic student writing which could act as a target for the first-year writers in this Element.

However, no corpus construction follows a smooth path, and a caveat worth mentioning here relates to control of ensuring papers were upper-level grades. Summarising the creation of MICUSP, Reference Römer and O’DonnellRömer and O’Donnell (2011) note that they largely depended on the student’s assertion that their voluntary submission to the corpus had received an A– or A grade from their instructor. Although, Römer and O’Donnell request the name of the instructor, it is not documented how many texts were verified by asking the instructor for confirmation of grade, and it is also not elaborated in their summary or on the MICUSP repository, how writing grades were decided, according to writing rubric measures. This is a caveat worth bearing in mind when using MICUSP.

A breakdown of the MICUSP corpus in terms of academic subjects and topics is provided across Tables 5 and 6.

4.3 Identifying and Checking Dependencies

FYC and MICUSP texts were cleaned up by removing titles, reference lists, and references to figures and mathematical formulas to preserve student-produced language and avoid third-party tables, charts, graphs, and formulas. Both corpora were lemmatised and POS (part of speech) tagged and parsed using the Stanford Core NLP annotators (Reference Manning, Surdeanu, Bauer, Bontcheva and ZhuManning et al., 2014) (version 3.9.2) through the command line. Table 7 shows these procedures. Scripts One and Two were originally written by the second author as part of the work carried out in Reference Durrant, Moxley and McCallumDurrant et al. (2019), while Scripts Three and Four were written as part of the work carried out in Reference Durrant, Brenchley, Szudarski and BarclayDurrant and Brenchley (2021).

A sample of the parsed output is provided in Table 8. This table can be understood as showing the sentence ID (sentence number in the text), the word number, word in its original form, the lemma of that word, its part of speech (POS) as tagged under the Penn Treebank system (Reference Marcus, Marcinkiewicz and SantoriniMarcus et al., 1993), the word number being depended on (dep_on), and the dependency. In this Element, we based our work on the following five dependency types in their lemma forms:

• Adjectives with a modifying dependency (amod) on a common noun (e.g., ‘serious disease’)

• Adverbs with a modifying dependency on an adjective (advmod) (e.g., ‘almost impossible’)

• Adverbs with a modifying dependency on a verb (advmod) (e.g., ‘usually tend’)

• Common nouns with a subject dependency on a verb (nsubj) (e.g., ‘show charisma’)

• Common nouns with a direct object dependency on a verb (dobj) (e.g., ‘feel anxiety’).

Dependencies were extracted using ‘Script Three’. This script read through each text and extracted amod, advmod, nsubj, and dobj dependencies through a function labelled ‘dep.pairs’. Taking amod dependencies as an example, the function searched for rows that contained ‘JJ’ (adjective) in the POS column, a ‘amod’ tag in the dep column, and a ‘dep_on’ value that attaches or points to a word with ‘NN’ (common noun) in the POS column. The function then recorded the dependent adjective and the noun it depends on. The same procedure was used for extracting nsubj, advmod, and dobj dependencies.

Since the Stanford parser has traditionally been trained from the Penn Treebank (Reference Marcus, Marcinkiewicz and SantoriniMarcus et al., 1993), which contains sentences from the Wall Street Journal, the accuracy of parsing often non-standard learner English had to be determined to ensure that the output from the parser actually contained dependency combinations. For this check, the first author worked with a second annotator to check the POS tagging and dependency accuracy. Both annotators followed the original annotation guidelines from Reference Marcus, Marcinkiewicz and SantoriniMarcus et al. (1993) and Reference De Marneffe and ManningDe Marneffe and Manning (2008). Ten texts per grade level, which amounted to over 20 per cent of the total corpus (180 of 879 texts) were chosen for the check. The first author checked 10 per cent of the 180 texts (i.e., 18 texts) by recording the POS tags and then the dependency pairs. The second annotator coded the same eighteen texts and recordings between the two annotators were compared. Agreement was high for POS tags (92 per cent) and dependencies (86 per cent), and so the first author then coded the remaining 162 texts independently. This coding was then compared with the Stanford output.

Table 9 shows the parser performance in terms of precision and recall across grade level, rounded to the nearest percentage. The table also shows that the greatest discrepancies arise from adverb modifying syntactic types. The relatively low recall scores mean that the parser was unable to capture many true adverb dependencies, and so this dependency type was not considered further in the study. These low results may be due to the fact the parser is dealing with learner English, or it might be task or feature dependent. For example, in two recent studies by Reference Kyle, Crossley and VerspoorKyle et al. (2021) on verb-argument constructions and Reference Picoral, Staples and ReppenPicoral et al. (2021) on different types of phrasal and clausal structures, accuracy varied. In the former study, 80 per cent accuracy was reported and, in the latter, although higher accuracy was reported, task was seen as a potential source of variation/inaccuracy.

The other dependency findings corroborate similar checks on different structures from Reference Berzak, Kenney, Spadine, Erk and SmithBerzak et al. (2016), who reported 88.07 per cent accuracy with sentences from the Cambridge Learner Corpus. Reference Durrant, Brenchley, Szudarski and BarclayDurrant and Brenchley (2021) also reported that parser precision averaged 80 per cent and recall averaged 85 per cent for amod dependencies, while for dobj dependencies, precision averaged 76 per cent and recall averaged 24 per cent with texts written by English school children. Similarly, high rates of precision and recall were also found in Reference Kyle, Eguchi and GrangerKyle and Eguchi (2021), who reported an average accuracy rate for noun–adjective dependencies of 96.9 per cent, verb–adverb dependencies had an accuracy rate of 98.6 per cent, verb–direct object dependencies of 96 per cent, and verb–subject 95.4 per cent. Table 9 also shows that second language scores were consistently lower than first language scores. Although not studied further in this Element, there is evidence that learner errors have some relationship to parser errors (e.g., Reference Huang, Murakami, Alexopoulou and KorhonenHuang et al., 2018), and so the premise that second language writing is often more grammatically erroneous than first language writing may be explored to provide a justification for the findings here. Equally, it should be highlighted that these results and cautions are also not limited to the domain of learner corpus research or indeed the Stanford parser. Reference KyleKyle (2021) has also recently drawn attention to the fact that since parsing is dependent on prior annotation processes (tokenisation, POS tagging), there is a need to be aware that errors in these prior processes mean that the errors will also impact on the later parsing as the parser draws on these processes to carry out parsing the texts.

During this accuracy check, there were several tags and dependencies that we questioned. These often related to mistagging of individual words and therefore casting doubt on the dependencies. Examples of mistagging are shown in examples [3]–[5]. Examples [3]–[5] show inaccurate tagging with sentence [3] illustrating how the original ‘JJ’ tag for ‘individual’ should be ‘NN’; sentence [4] illustrating how the original ‘NNS’ tag for ‘states’ should be a verb (‘VBZ’: third person singular present); and sentence [5] illustrating how the original ‘NN’ tag for ‘addresses’ should be a verb (‘VBZ’):

1. [3] ‘Adolescents may look at an individual and wonder why it is they themselves cannot have the life the individuals on social media have’.

1. [4] ‘The 14th Amendment in the United States constitution states that anybody who is born within the borders of the United States, has the right to become an American citizen’.

1. [5] ‘An article called the Role of Communication Technology in Adolescents Relationships and Identity Development addresses that on Facebook triggers states of envy and resentment in many’.

The examples in Tables 10 and 11 show the relationship between mistagging and inaccurate dependencies; in Table 10, ‘committed’ was tagged as an adjective ‘JJ’ (instead of ‘VBD’) and parsed incorrectly as an amod dependency. In Table 11, we see a similar issue with tagging of ‘switching’, which has been tagged as ‘NN’ (singular noun or mass noun) instead of ‘VBG’ (verb gerund).

4.4 The Measures Used

The rationale for choosing association measures was based on examining measures across the groupings identified in key computational work by Reference EvertEvert (2004), Reference PecinaPecina (2005, Reference Pecina2010) and Reference SeretanSeretan (2011) as well as language learning studies showing differential patterns between different association measure types (e.g., Reference BestgenBestgen, 2017; Reference Bestgen and GrangerBestgen & Granger, 2014; Reference Durrant and SchmittDurrant & Schmitt, 2009; Reference Gablasova, Brezina and McEneryGablasova et al., 2017a; Reference PaquotPaquot, 2018, Reference Paquot2019). In choosing a set of association measures, we opted for measures which were already in use across different studies but also chose to include computationally simple and therefore more easily comparable measures which focussed on the basic idea of using contingency tables. This meant that we did not opt to use those measures labelled context measures in Reference PecinaPecina (2005, Reference Pecina2010) as these involved many complex mathematical calculations and were also measures which had no background in the literature we drew upon to guide our work. We wanted to use measures that would have some familiarity to as many potential readers as possible. Table 12 shows the final set of forty-seven association measures. The formulae for these forty-seven measures can be found in the supplementary materials on https://leemccallum.net/resources. The association measure values were calculated by extending the calculator developed by Reference DurrantDurrant (2020), where each column contained one association measure and each row contained a dependency, so across each row we could simply obtain an entry for each of the forty-seven association measures.

4.5 The Cluster Analysis Rationale and Procedures

To engage with the first question (‘How can we choose appropriate measures of collocation?’) we carried out a cluster analysis to illuminate measures which could tell us different types of information about collocation properties.

As a type of statistical analysis, cluster analysis is used to group similar cases (or variables in our study) together. In our study, what is being grouped together is the association measures. They are grouped together according to how similarly they assign an association score to the dependency pairs. This is a similar procedure to factor analysis (both exploratory and confirmatory) in the sense that all three group similar cases/variables together according to their similar properties (e.g., assigning similar values to items). Factor analysis was used in some of the studies mentioned in Section 3 in discussing how association measures may be tapping into different or similar properties of collocation (e.g., see Reference Durrant, Moxley and McCallumDurrant et al., 2019; Reference Kim, Crossley and KyleKim et al., 2018; Reference Kyle, Crossley and BergerKyle et al., 2018). However, the added benefit of choosing to tap into these potential similar and distinct groupings via a cluster analysis is that the technique visualises these groupings via a related heatmap of correlations and dendrogram. Both visualisations can be used alongside the numerical correlation data to inform our understandings of association measures.

The cluster analysis was based on dependencies that appeared five or more times in the reference corpus so as to ensure association measure calculations were reliable (e.g., see Reference PaquotPaquot, 2018, Reference Paquot2019). Table 13 shows those below threshold and absent dependencies which were not included in the cluster analysis and the total number of types that were included. Below threshold dependencies were dependencies appearing fewer than five times in MICUSP. Absent dependencies were those not found in MICUSP at all. The large percentage of absent and below threshold combinations are most likely to be because MICUSP is still a relatively small corpus for retrieving multi-word combinations, the latter being somewhat less frequent than individual words, even in large general corpora.

A snapshot of the most frequently occurring dependencies in the FYC corpus and their status in MICUSP is presented at https://leemccallum.net/resources

Hierarchical clustering was used to show the relations between the association measures as determined by the data rather than a predetermined theory from the literature (Reference Field, Miles and FieldField et al., 2012). Hierarchical cluster analysis was chosen because the analysis aimed to use the clustering as an exploratory method to show connections between measures, emerging from the data itself rather than any predetermining theory from the literature or the researcher (Reference Field, Miles and FieldField et al., 2012). This contrasts to non-hierarchical clustering where the researcher may predetermine the number of desired clusters (Reference LevshinaLevshina, 2015). Reference CrawleyCrawley (2013, p. 819) explains: ‘The idea behind hierarchical cluster analysis is to show which of a (potentially large) set of samples are most similar to one another, and to group these similar samples in the same limb of a tree. Groups of samples that are distinctly different are placed in other limbs’. This similarity is decided based on the distance between two samples. In this study of clustering variables, the distance is the distance between pairs of association measures.

To check the validity of the clustering solution, we checked the Average Silhouette Width (ASW), which represents how well-formed the clusters are and indicates how confident we can be in the clustering solution (Reference LevshinaLevshina, 2015). The statistic ranges from –1 to 1, with –1 indicating no clustering in the data and 1 indicating perfect separation of all clusters (Reference Brock, Pihur, Datta and DattaBrock et al., 2008). Reference LevshinaLevshina (2015) advises that an ASW value <0.2 signals a lack of substantial cluster structure in the data. This validity check is discussed in Section 5.

These decisions helped determine the different clusters in the cluster analysis solution and helped us determine which measures to retain and which to discard, thus developing an understanding of how we can distinguish between association measures which tap into similar/different collocation properties.

4.6 The Statistical Modelling Rationale and Procedures

To engage with the second question: ‘how can we measure and understand the potential role of different linguistic and non-linguistic variables in an appropriate way?’, we created an ordinal type of mixed effects model. In other words, the model was used to measure the role different variables played in shaping score variations in the FYC corpus. This aligns with comments from Reference Loerts, Lowie and SetonLoerts et al. (2020), who view essay grades as ordinal variables. The mixed model was a cumulative links model which treats the dependent variable (‘Final_Grade’) as ordinal and represents the effect of the independent variable on the dependent variable as logit odds, in a similar manner to Reference Haberman and SinharayHaberman and Sinharay (2010). These odds represent the chance of the independent variable having an effect on the dependent variable (Reference O’ConnellO’Connell, 2006). In line with established cumulative links modelling practices, these logit odds were transformed into more intuitive odds ratios (Reference WinterWinter, 2020).

To measure diversity, we followed Reference PaquotPaquot (2019) and more recent studies (e.g., Reference Jiang, Bi, Xie and LiuJiang et al., 2021) and calculated the RTTR for each dependency type: amod, nsubj, and dobj. The ratio was calculated by dividing the number of types by the square root of the number of tokens. We choose the RTTR due to its consistent use in the literature, for example see its continued use over time in vocabulary studies (e.g., Reference Bulté and HousenBulté & Housen, 2014; Reference Daller, Turlik, Weir, Jarvis and DallerDaller et al., 2013; Reference Hou, Verspoor and LoertsHou et al., 2016; Reference KimKim, 2014; Reference Llanes, Tragant and SerranoLlanes et al., 2018; Reference Lorenzo and RodríguezLorenzo & Rodríguez, 2014; Reference Treffers-Daller, Parslow and WilliamsTreffers-Daller et al., 2018; Reference Verspoor, Lowie, Chan and VahtrickVerspoor et al., 2017) as well as the aforementioned phraseological studies. The measure itself is also simple to calculate and easily interpreted: the higher the value, the more diverse the text is taken to be. However, it should be noted that like most other diversity measures, the RTTR is influenced by text length effects and this caveat should be remembered when using this measure of diversity.

First, the linguistic predictors of the RTTR and association measures were centred and standardised following procedures from Reference GriesGries (2013b). These procedures allow variables to operate on a scale of standard units, thus making variables comparable. Centring and standardisation procedures are also recommended to avoid possible model convergence issues (Reference WinterWinter, 2020).

The fixed effects of task and language status were coded as categorical variables with two levels (1 and 2), with the variable names ‘task 1’ and ‘task 2’ and ‘Language_status1’ and ‘Language_status2’ with Language_status1 = first language (native) speaker of English and ‘Language_status2’ = second language (non-native) speaker of English. In the model, the algorithm in R chooses one of these levels to act as a reference level and uses this reference level as a comparison with the other levels. The choice of reference level is often alphabetical or the first numerical level (Reference LevshinaLevshina, 2015). In the model output, the reference level will not be shown, but the level compared to it will be (e.g., see Reference LiuLiu, 2016; Reference WinterWinter, 2020, p. 184). This means that for the model output in Figure 4, ‘Language_status1’ (native speaker) is the reference level with the model output showing the significant result for ‘Language_status2’ (non-native speaker).

All models were created in R (Version 3.6.3) (R Core Development Team, 2014) using the ordinal package (Reference ChristensenChristensen, 2019). The mixed effects model was created by following the three-stage process set out by Reference GriesGries (2015). Stage one involved generating a maximal fixed effects model. Stage two involved generating a maximal random effects model. These two stages are described in Reference McCallumMcCallum (2021). Stage three joined these two models together to create the mixed model. The fixed effects model initially included all possible fixed effects predictors. These included the association measures from the cluster analysis for each dependency type, the Root TTR for each dependency type, and the effects of task and the writers’ language status. This ‘full’ model was then trimmed to remove non-significant predictors, with each new model compared to the previous one via the ‘anova’ function. The AIC (Akaike Information Criteria) values were inspected and the model with the lowest value was chosen as the most parsimonious model. The most parsimonious model was the one able to explain the most variance using the fewest number of predictors.

Stage two involved establishing the maximal random effects structure. An examination of the FYC data set indicated that individual writers contributed more than one essay to the corpus. We therefore modelled this random variation in line with other modelling works such as Reference PaquotPaquot (2019) and Reference Durrant, Brenchley, Szudarski and BarclayDurrant and Brenchley (2021).

To also answer the second question, we qualitatively inspected texts containing high- and low-scoring association measures and diversity scores in the FYC corpus. This analysis is presented in Section 7.

5.1 Overview of Section

The cluster analysis in this section answers the first question that we engage with: ‘how can we choose appropriate measures of collocation?’ The cluster analysis essentially groups together similar association measures in the sense that they assign similar values to the same dependency pair. In doing so, it highlights similar pairs and therefore larger clusters of measures while at the same time illuminating those association measures which are dissimilar and clearly tapping into different collocation properties.

This section presents the results of the cluster analysis. These results are interpreted using different sources of information. Key to this interpretation is the visual understanding of the measures we obtain from the cluster dendrogram itself (see Figure 1), in addition to the original heatmap correlations which were used to generate the clustering. These two visuals allow us to see the specific robust nature of the clustering and also put this clustering into perspective by considering how individual pairs of measures within and between clusters are correlated. The information here is therefore systematic and visual rather than making connections between measures through solely inspecting lists ranked association measure scores as Reference BrezinaBrezina (2018) and others have presented. A third source of information that we refer to is the theoretical criteria/guidance offered by Reference Gablasova, Brezina and McEneryGablasova et al. (2017a). Reference Gablasova, Brezina and McEneryGablasova et al. (2017a) present several theoretical criteria that researchers should consider for measure selection. These criteria relate to (i) the mathematical computation of the measure, (ii) the scale the measure operates on, and (iii) each measure’s practical effect in how its formula is able to illuminate or downgrade particular word combinations.

Figure 1 Dendrogram of association measures

5.2 The Clustering and Its Validation

The dendrogram in Figure 1 visually represents the (dis)similarity amongst association measures. It has been rotated 90 degrees to aid readability. The x-axis represents the linkage distance between the objects. This height symbolises the differences between the measures where the greater the height, the bigger the difference between variables in the cluster analysis. This figure also illuminates several ‘branches’ that connect the association measures that are being clustered. These branches vary in length, and the longer in length, the greater the difference between the measures (Reference GriesGries, 2013b). The y-axis shows the association measures that are clustered together. As Reference WiechmannWiechmann (2008) recognises, the interpretation of a dendrogram is often influenced by the researcher’s subjective judgement, and so the discussion has a degree of subjectivity from our own interpretation of the clustering.

5.2.1 Determining the Number of Clusters

The dendrogram visually supports three to six large clusters with several sub-clusters also apparent. To check the clustering’s internal validity, we calculated the ASW which represents how well-formed the clusters are and indicates how confident we can be in the clustering solution (Reference LevshinaLevshina, 2015). This was calculated with the silhouette function in the cluster package (Reference Maechler, Rousseeuw, Struyf, Hubert and HornikMaechler et al., 2015). The statistic ranges from –1 to +1, with –1 indicating no clustering in the data and +1 indicating perfect separation of all clusters (Reference Brock, Pihur, Datta and DattaBrock et al., 2008). Reference LevshinaLevshina (2015) advises that an ASW value <0.2 signals a lack of substantial cluster structure in the data. The ASW statistic for the three to six clusters is presented in Table 14.

Table 14 Average silhouette width

Cluster	3	4	5	6
ASW Values	0.323	0.328	0.341	0.333

The ASW values suggest greatest support for five clusters, with the lower ASW value for six clusters indicating less stability. Figure 2 highlights these five clusters. It has been rotated 90 degrees to aid readability.

Figure 2 Clusters in the dendrogram

5.3 Measure Retention

A valid consideration in making retention decisions is a measure’s history in language learning studies and the interpretability and performance of the measure with actual language data. Further to this, a key reason for the cluster analysis was to understand the place of the MI and the t-score in the literature and to uncover how different other measures are to those two measures which act as pillars representing exclusivity and frequency, as highlighted by Reference BrezinaBrezina (2018).

The dendrogram in Figure 2 presents one perspective on this. More detailed information can be gleaned from a heatmap representing the correlation matrix between all measures. This is shown in Figure 3. The strongest correlations are in red; shades of purple represent weak to approaching moderate correlations (r = 0.30–0.70) and white represents little correlation between measures. A downloadable visualisation of this map can be found at https://leemccallum.net/resources

Figure 3 Heatmap showing relationships across association measures

5.4 Between and within Cluster Observations

To determine levels of similarity between the association measures, we can make observations both between clusters and within individual clusters in the dendrogram.

In Cluster 1, the lack of height between the MI and Yulle’s ω and Yulle’s Q suggests these are more similar than any of the other measures. There appears to be a small degree of difference between the Dice measure and the other cluster members because of its greater branch height. Given this high collinearity and that we know the MI and Log Dice have been shown to tap into exclusivity, this leads to a conclusion that the measures in this cluster flag up exclusive pairings. Although when comparing the MI and Log Dice, Reference Gablasova, Brezina and McEneryGablasova et al. (2017a) support the scaling of the Log Dice because compared to the MI, it has a theoretical limit. Comparative work between the MI and the Log Dice has also found that the measures are highly correlated (Reference Öksuz, Brezina and RebuschatÖksuz et al., 2021) and when entered into a mixed effects model on collocation processing times with first and second language speakers, the Log Dice produced a better explanatory model than the MI. However, explanations around why this might be the case are in their infancy. In our study, we set out as taking both the MI and the t-score as representative association measures, seeking to learn about their relationships with other measures. Therefore, we take the MI as our starting marker and look at the relationships it has with other clusters’ members. However, this first cluster, like the other clusters, serves as a reminder that analysts are faced with making pairwise choices between association measures, whereby they should choose measures that are appropriate for their study’s aims.

From Cluster 2 to 5, there is a move away from exclusivity with measures sharing less correlations with the cluster of measures first represented in Cluster 1. Cluster 2 contains many measures that do not have an established connection to language learning, teaching, or assessment research. In weighing up measure retention, Figure 3’s heatmap indicates strong correlations between Jaccard and First Kulczynsky; and between Rogers-Tanimoto, Third Sokal Sneath, Hamann, and Second Sokal Sneath. The correlations between these measures exceed r = 0.80 and the measures show high collinearity with other measures in Cluster 1 (e.g., the correlation between the Third Sokal Sneath and Log Dice reaches r ≥ 0.73). Following general statistics literature (e.g., Reference Tabachnick and FidellTabachnick & Fidell, 2014) on what constitutes collinearity (r ≥ 0.70–0.90), the retention of all these measures would mean capturing very similar properties of collocation. The value of the cluster analysis appears clear here because it illuminates important relationships between coefficient measures that are sparsely elaborated on in the literature. This cluster also offers counterevidence to Reference SchneiderSchneider (2020), who comments that collinearity between association measures is rare because the clustering here shows high-measure collinearity. Given these observations, the measure which is most distinct from the MI in Cluster 1 is the geometric mean (r = 0.64) and so this measure is the measure retained for Cluster 2. The geometric mean is also a good measure for retention because although correlated with its cluster members, these correlations do not violate collinearity (r = 0.68).

Cluster 3’s starting height and branch length indicate that these measures share less collinearity than other clusters. Many measures originate from hypothesis testing, hence a degree of correlation between them. When the correlations in the heatmap are explored, the strongest correlation exists between mutual expectation and Russel Rao (r = 0.18). All other measures have low to no correlation. Reference Gries, Durrant, Gries and PaquotGries and Durrant (2021) previously noted that association measures are manipulations or regressions of each other, and while this is indeed true for much of the observations in other clusters, Cluster 3 shows measures with little correlation.

Considering the measures in Cluster 3, the most enduring measures in the language learning literature have been significance test measures. For example, the z-score, the odds ratio, and the Loglikelihood Ratio and Loglikelihood Ratio Squared (LLR²) have all been featured in several language studies (e.g., Reference Evert, Lüdeling and KytöEvert, 2009). Support for Loglikelihood measures is most consistent across the literature (e.g., Reference DunningDunning, 1993; Reference EvertEvert, 2004, Reference Evert, Lüdeling and Kytö2009; Reference PecinaPecina, 2005), while measures such as the odds ratio are criticised for being difficult to interpret (Reference Evert, Lüdeling and KytöEvert, 2009). There is a pragmatic argument to be made that either of the Loglikelihood measures are valid retention choices, given their support in the literature (e.g., Reference EvertEvert, 2004); however, there is more of a tendency to favour more mathematically robust and less-bias measures that do not overinflate language data (Reference EvertEvert, 2004), and so the squared loglikelihood ratio seems to be the most pragmatic retained measure in Cluster 3.

In Cluster 4, there is little difference between ‘Braun Blanquet’ and ‘Minimum Sensitivity’, while the ‘T Combined Cost’ is distinct from other measures and ‘Poisson Stirling’ and ‘Unigram Subtuples’ have some degree of difference. The other two measures – Braun Blanquet and Minimum Sensitivity – are highly correlated (r = 1.00). It is also worth noting here that while the Minimum Sensitivity has gained favour in some studies by performing well in identifying collocations (e.g., Reference WiechmannWeichmann, 2008), its use has been advised with caution as its values are difficult to interpret (Reference GriesGries, 2013b), its close relationship to the Braun Blanquet also means the same caveats should apply to Braun Blanquet. In contrast to the aforementioned measures, the T-combined cost and Delta P w2 | w1 appear distinct from other measures (shown by high height and lack of branch). Both measures also have lower correlations with other measures in the cluster, as indicated by the heatmap in Figure 3. Equally, in our quest to retain measures that are distinct enough from each other, referring back to the correlations between these measures in Cluster 4 and those in Cluster 1, illuminating exclusivity, the Delta P w2 | w1 is the least correlated with the MI (r = 0.31) when compared to the correlations between the MI and the T-combined cost (0.44). Therefore, in Cluster 4, we retain the Delta P w2 | w1 measure.

Cluster 5 contains the following association measures: U cost, Simpson, S cost, T score, Michael, Delta P w1 | w2, Second Kulczynsky, and the Fourth Sokal Sneath. The flat height between Simpson and S cost indicates high collinearity (r = 0.950). These measures are also highly correlated (r = 0.90, r = 0.90) with the Second Kulczynsky and the Fourth Sokal Sneath (r = 0.73, r = 0.73). Similar high correlations are found between the t-score and Michael (r = 0.95).

In deciding between the t-score and Michael, the t-score has an established history in language learning and is known to illuminate high-frequency pairings that are typical across genres/disciplines (e.g., Reference Durrant and SchmittDurrant & Schmitt, 2009; Reference Granger and BestgenGranger & Bestgen, 2014) and therefore is the measure most likely to be retained. However, both the dendrogram and the heatmap show evidence that the Delta P w1 | w2 measure is distinct enough from the t-score and all other measures as evidenced in its branch height and correlation value of r = 0.40 with the t-score. This means that in this last cluster we retain the Delta P and the t-score for their different properties.

To sum up across the five clusters, we have chosen measures which are distinct enough from each other to suggest that they tap into different information about collocation. The analysis of the dendrogram and heatmap lead us to retain six measures, which are mathematically distinct enough to highlight different properties of collocation: the MI, the geometric mean, the LLR², the Delta P w2| w1, Delta P w1|w2, and the t-score.

The cluster analysis helps reveal relationships between association measures because it visually represents various relationships between measures already mentioned in the literature and measures which have received sporadic attention. The clustering also facilitates understanding by placing commonly used measures into this visual picture to reinforce their position as being viable measure candidates that tap into different collocation properties. We can see, through the individual clusters and the within cluster relationships, that the MI, the geometric mean, Loglikelihood, t-score, and Delta P are all distinct enough to suggest they illuminate different types of word combinations. This aligns with previous findings from Reference Durrant, Moxley and McCallumDurrant et al. (2019), Reference Kim, Crossley and KyleKim et al. (2018), and Reference Kyle, Crossley and BergerKyle et al. (2018), who found similar distinctions, with Reference Kyle, Crossley and BergerKyle et al. (2018) also finding that the MI² was also sufficiently different from the MI (as is also shown in the cluster analysis here).

This section has explained how we decided which measures to retain and which to discard. It is also important to look holistically at the cluster analysis and reflect on the methodological clustering choices made at the start of Section 4. The choice of furthest neighbour clustering was made to produce tight clusters. We can see then that we have a spectrum of clusters, with Clusters 2, 3, 4, and 5 further away from the measures and underlying principle of ‘exclusivity’ that Cluster 1 is clearly based upon. With reference to Clusters 4 and 5, we can see a sparse picture of clustering with these measures furthest from those in Cluster 1, but not necessarily always strongly clustered together as pairs in their respective cluster. There is evidence, then, that this clustering corroborates the theoretical work of Reference BrezinaBrezina (2018) and Reference EvertEvert (2004, Reference Evert, Lüdeling and Kytö2009), who have referred to the properties that are flagged up by the association measures, operating along a cline or continuum. This continuum goes from exclusivity at one end to more of a focus on significance testing/raw frequency measures at the other end. The visual clustering supports this work and also highlights how the Delta P continues to be distinct from this potentially dichotomous picture, given their low correlation/cluster ties with other non-directional association measures.

6.1 Overview of Section

The statistical modelling presented in this section answers the second question that we engage with: ‘how can we measure and understand the potential role of different linguistic and non-linguistic variables in an appropriate way?’

This section focusses on explaining the inferences we can (and cannot) make from the mixed effects modelling results. The section works together with Section 7, which follows to illuminate important relationships between collocations and writing quality and several learner and contextual variables. These sections combined give an overall picture of collocation and writing quality relationships and remind us of the strengths and limitations that such modelling work affords us.

6.2 Mixed Model Results

This section describes how the key association measures selected in the previous section – the MI, geometric mean, LLR², Delta P w1|w2, Delta P w2|w1, and t-score – predicted the scores assigned to texts. An overview of the independent variable predictors in terms of effect type and also their descriptive statistics are shown in Table 15.

The mixed model, generated from the optimal fixed and random models, is shown in Figure 4.

Figure 4 Mixed effects model with individual student as random variation

The mixed effects model in Figure 4 and its summarised information in Tables 16 and 17 shows that nine fixed effects were significant predictors of the outcome variable of ‘Final_Grade’: eight of these predictors were collocation measures and the ninth predictor was ‘Language_status2’. The eight significant collocation measures included four positive coefficient collocation predictors. These were the rounded-up logit coefficients for:

• Mean MI for noun subject–verb combinations (‘Mean MI nsubj’) (β = 0.330, SE = 0.013, z = 2.616, p = 0.008);

• Mean MI for verb–direct object noun combinations (‘Mean MI dobj’) (β = 0.201, SE = 0.074, z = 2.718, p = 0.007);

• Adjective–noun diversity (‘Amod RTTR’) (β = 0.196, SE = 0.007, z = 2.552, p = 0.002);

• Noun–subject diversity (‘Nsubj RTTR’) (β = 0.135, SE = 0.075, z = 1.795, p = 0.007).

As the only non-linguistic predictor, language status (‘Language_status2’) was also significant (β = 0.376, SE = 0.155, z = 2.426, p = 0.015).

The remaining four negative coefficient predictors were the logit coefficients for:

• Mean LLR² adjective–noun combinations (‘Mean LLR² amod’) (β = –0.206, SE = 0.067, z = –3.101, p = 0.002);

• Mean Delta P for word 2| word 1 noun–subject verb combinations (‘Mean Delta P w2 | w1 nsubj’) (β = –0.201, SE = 0.009, z = –2.254, p = 0.024);

• Mean t-score noun–subject verb combinations (‘Mean t-score nsubj’) (β = –0.252, SE = –0.103, z = –2.433, p = 0.015);

• Mean Delta P w1|w2 verb–direct object–noun combinations (‘Mean Delta P w1| w2 dobj’) (β = –0.149, SE = 0.007, z = –2.118, p = 0.034).

Following advice in statistics literature (e.g., Reference LiuLiu, 2016; Reference WinterWinter, 2020), the logit odds were converted into odds ratios to aid interpretation. The key to interpreting these odds lies in whether the odds ratio is more than or less than 1. When less than 1, the odds of being beyond or at a particular grade level are expressed as being lower. When more than 1, the odds of being beyond or at a particular grade level are expressed as being higher. The model output in Figure 4 is therefore interpreted as follows.

For the positive coefficients, the odds of being at or beyond a particular final grade score were 1.387 times higher with a one unit increase in the mean MI noun–subject–verb combinations (‘Mean MI nsubj’); 1.223 times higher with a one-unit increase in the mean MI for verb–direct object noun combinations (‘Mean MI dobj’); 1.216 times higher with a one-unit increase in the adjective–noun diversity (‘Amod RTTR’); 1.144 times higher with a one-unit increase in the noun–verb diversity (‘Nsubj RTTR’). In other words, the use of higher ‘Mean MI nsubj’, ‘Mean MI dobj’, ‘Amod RTTR’, and ‘Nsubj RTTR’ means there are higher odds of texts being awarded a higher final grade score.

Those collocation predictors with negative logit coefficients can be interpreted as follows. In terms of the odds ratio (OR), the odds of being at or beyond a particular ‘Final_Grade’ score are lower by a factor of 0.861 for a one-unit increase in the mean Loglikelhood Ratio Squared adjective–noun combinations (‘mean LLR² amod’). The odds of being beyond a particular final grade score are lower by a factor of 0.817 for a one-unit increase in the mean Delta P w2| w1 for noun–direct object–verb combinations (‘Mean Delta P w2|w1 nsubj’). The odds of being beyond a particular final grade score are lower by a factor of 0.777 for a one-unit increase in the mean t-score noun–subject–verb combinations (‘Mean t-score nsubj’). The odds of being beyond a particular final grade score are lower by a factor of 0.862 for a one-unit increase in the mean Delta P w1|w2 noun–direct object–verb combinations (‘Mean Delta P w1|w2 dobj’). These results indicate that the odds of being at or beyond a particular grade level are lower with a one-unit increase in the predictors of ‘Mean LLR² amod’, ‘Mean Delta P w2|w1 nsubj’, ‘Mean t-score Nsubj’, and ‘Mean Delta P w1|w2 dobj’.

In the mixed effects model, ‘Language_status2’ indicates that the odds of non-native writers being beyond a particular grade level were 1.457 times greater than native writers. It is important to recognise the importance of this contextual finding and place it side by side with the findings of those that were positive linguistic coefficients. These results seem to indicate that in the case of the positive coefficients, there is a similar size of odds for grade increase with language status and both amod and nsubj diversity with the odds higher for language status, helping put into perspective two things. First, that there is a tendency for the odds of a grade increase to be shaped by multiple linguistic and non-linguistic variables, and second, in this case the non-linguistic predictor has slightly higher odds of grade increase than the linguistic predictors.

6.3 Model Validity

In terms of how confident we can be that these results reflect the ‘true’ odds of ‘Final_Grade’ increasing or decreasing, an examination of the 95 per cent confidence intervals (CI) in Table 15 indicates that all odds ratios for the measures fall within the CI range. This means that if the model was repeatedly estimated, the true odds would lie in the same range, 95 per cent of the time, thus giving confidence in the model.

We can also examine the threshold coefficients that make up the cumulative model. In Table 16, it is important to illuminate the different logit and odds ratios that appear across these comparisons. In Table 17, we see that the logit odds are negative for grade comparisons 7–13, the odds of the predictors, increasing grades decrease. There is a noticeable shift in direction at the two highest grade levels, that of comparisons between 13 and 14, and 14 and 15, whereby the logit odds and the odds ratio become positive. This appears to suggest that predictors have increased odds of increasing grade particularly at these higher levels.

Overall, the mixed effects model shows that the predictors of ‘Mean MI nsubj’, ‘Mean MI dobj’, and the amod (Amod RTTR) and nsubj (Nsubj RTTR) diversity dependencies have higher odds of increasing final grade scores.

In contrast, the predictors of the ‘Mean LLR² amod’, ‘Mean Delta P w2|w1 Nsubj’, ‘Mean t-score Nsubj’, and ‘Mean Delta P w1|w2 Dobj’ have lower odds of increasing final grade scores. To some extent this mirrors the findings from Reference PaquotPaquot (2019) who also found support for the MI as did other studies (e.g., Reference Bestgen and GrangerBestgen & Granger, 2014; Reference Durrant and SchmittDurrant & Schmitt, 2009; Reference Granger and BestgenGranger & Bestgen, 2014). However, in the case of this FYC context, support for the MI is for nsubj and dobj dependencies rather than amod dependencies. It appears that in the FYC context, rather than being more likely to award higher grades for more sophisticated amod dependencies like Reference PaquotPaquot (2019), the odds of FYC raters awarding higher grades to more diverse amod and nsubj use is greater. It is also worth bearing in mind that since these diversity measures contain both dependencies that would reach or not reach collocation status according to past MI thresholds, then it seems it is more likely that raters tend to be overall influenced by the variation of dependency types and to a lesser extent their average sophistication.

To evaluate the goodness-of-fit for the mixed effects models, there are several statistics available (e.g., see Reference LevshinaLevshina, 2015; Reference LiuLiu, 2016). Although traditional modelling literature has often cited the R² value to give an estimate of how much variance in grade is explained by a particular model, such an estimate is not recommended for this kind of logistic regression.

Reference LevshinaLevshina (2015) comments on the use of Pseudo R² as a measure of goodness-of-fit. Like many others (e.g., Reference Field, Miles and FieldField et al., 2012; Reference LiuLiu, 2016), she notes that in logistic regression models, the value of the R² is often lower than the R² in linear regression, even if the quality of models is comparable. Equally, Reference LevshinaLevshina (2015) notes that the Pseudo R² is less conceptually clear than the R² from linear regression. It is often misinterpreted as being the same concept as the linear R²; however, this is disputed by Reference Hosmer and LemeshowHosmer and Lemeshow (2000), who have made direct criticisms of the statistic. For these reasons, the ordinal package used in this study does not support the use of the Pseudo R².Footnote ¹

Considering this, the models created in this Element are simply evaluated by looking at the AIC values. The inclusion of the individual writer as a random variable results in a model which is more able to explain variation in grade, as indicated by its lower AIC value: 3598.11 compared with the fixed model only, which has an AIC value of 3599.58. This gives us confidence that the modelling type is a more appropriate way to model score variation than previously used linear regression techniques. Similar results have also been reported in Reference Haberman and SinharayHaberman and Sinharay (2010).

6.4 Conclusions

This section highlights the value of mixed effects modelling as a method of exploring relationships between collocation and writing quality, as well as a number of writing context and learner-specific variables. It highlights the contrasting relationships that exist between the individual association measures and writing quality grades, as well as the relationships that exist between measures of diversity and writing quality. It also furthers our understanding of how the language background of the writer has a relationship with writing quality grades.

Section 7 explores these results in more detail and seeks to uncover possible explanations for these statistical results by looking at the rhetorical functions that these word combinations perform.

7.1 Overview of Section

While study two presented a quantitative measurement of variables that may shape writing quality scores, the final study of this Element also engages with the second question through a qualitative lens. In looking qualitatively at the FYC essays, the analysis uncovers possible reasons for the statistical patterns uncovered in study two.

This section carries out a qualitative analysis of high- and low-scoring association measures and high- and low-scoring diversity across the FYC corpus. We focus on answering the following questions that arose from the role of association measures in the statistical modelling:

• Why might higher mean MI noun subject–verb (nsubj) combinations have higher odds of increasing grades?

• Why might higher mean MI direct object–verb (dobj) combinations have higher odds of increasing grades?

• Why might higher mean LLR² adjective–noun (amod) combinations have lower odds of increasing grades?

• Why might higher mean Delta P w2|w1 noun subject–verb (nsubj) combinations have lower odds of increasing grades?

• Why might higher mean Delta P w1|w2 direct object–verb (dobj) combinations have lower odds of increasing grades?

• Why might higher mean t-score noun subject–verb (nsubj) combinations have lower odds of increasing grades?

The section also addresses the following question that arose from the role of diversity measures in the statistical modelling:

• Why might higher amod RTTR and nsubj RTTR diversity have higher odds of increasing grades?

To help answer these questions, Tables 18–29 highlight high- and low-scoring combinations on each measure. Each table shows the top or bottom five ranked examples on a particular measure from the FYC corpus. In each case, only collocations occurring in more than three independent texts were included. This allowed us to explore how high- and low-scoring frequent combinations (i.e., those occurring in multiple texts) may be used and how this use may be an indicator of language use that FYC raters appear to value or devalue. A fuller set of dependencies is included as supplementary materials on https://leemccallum/net/resources.

7.2 Positive Relationships between Association Measures and Writing Quality

This section examines the use of high- and low-scoring MI combinations to offer explanation as to why high-scoring mean MI values may have greater odds of increasing grades.

7.3 Negative Relationships between Association Measures and Writing Quality

This section examines the use of association measures which have lower odds of increasing grade scores.

7.4 Positive Relationships between Diversity and Writing Quality Grades

7.4.1 Why Might Higher Amod RTTR and Nsubj RTTR Diversity Have Higher Odds of Increasing Grades?

A third set of important findings from the statistical modelling was the role played by amod and nsubj diversity measures. Both measures were found to increase the odds of higher grades in the FYC context. In attempting to understand why this diversity leads to higher odds of a grade increase, a qualitative inspection of texts with high- and low-amod and nsubj diversity was carried out. It should be noted that in many cases, higher amod and nsubj diversity were found irrespective of text length in higher graded texts. Example texts A and B in Figures 5 and 6 respectively, show this trend and the different uses of amod dependencies. Text A shows how a high-scoring diversity text uses many dependencies to perform different rhetorical functions and creates arguments; Text B, a lower scoring text, shows how the use of fewer dependencies is related to a narrower range of rhetorical functions and arguments. It is also worth highlighting here that these two examples at opposite ends of the diversity scale are equal in word count and therefore the issue of text length and diversity is accounted for in this case.

Figure 5 Text A high amod RTTR

Figure 6 Text B low amod RTTR

It should also be highlighted that Text A contained many instances of amod dependencies which could have been considered specific topic/discipline items (e.g., neural base). It is also worth commenting on the impression writer of Text A creates on the reader as they appear to have clear control of using these dependencies both for creating arguments and for particular rhetorical functions. Use of the dependencies is both varied and accurate to give a sense of clear ownership of the text and attempting to meet the task requirements by setting up the issue as debatable/controversial. In Text B, many of the amod dependencies used are repeated across adjacent sentences (e.g., great deal) and are simply descriptive. Equally, the overall tone of the text is subjective and bordering on emotive rather than building on a particular stance through academic argument. This lack of diversity and effective use may be factors that raters devalue in their judgements. It is also worth commenting that the use of amod dependencies is embedded around reference to academic literature in Text A, while in Text B, where amod dependencies are used and used with variation, they do not appear to be used in relation to use of source materials to support their claims. Overall, these issues of variation, rhetorical function, and intended meaning/tone are worth considering as factors which raters judge negatively.

For nsubj diversity, a similar pattern was present, as shown in Figures 7 and 8 for with Texts C and D. Text C shows the use of varied nsubj dependencies to promote the benefits of a vegetarian diet, while in Text D the dependencies used are not directly linked to the writer performing a particular rhetorical function or achieving a particular rhetorical effect with the issue of euthanasia. Like the contrast between Text A and B, nsubj appears being used in the higher scoring text to construct a literature-led argument. For example, note the use of nsubj dependencies to discuss the work of Dr Ngyuen, while in the lower scoring text, the discussion is not as critical and is instead more descriptive of the study being used rather than fully successfully using the study to support their arguments.

Figure 7 Text C high nsubj RTTR

Figure 8 Text D low nsubj RTTR

Several other rhetorical functions were facilitated through the use of different amod and nsubj dependencies in high-scoring amod and nsubj diversity texts. A summary of these functions with dependencies retrieved by the parser is shown in examples [22]–[30].

• Setting out the issues with a topic over time:

1. [22] The first recorded usage of marijuana dates all the way back to 10,000-Year World History of Hemp and 1) and continues to this very day. In recent history, the United States has had several states legalize the plant on different levels (recreational use, medicinal use, etc.).

• Generally critiquing the topic under study:

1. [23] The next few paragraphs will be describing what happened in Ukraine prior and during the invasion, the failures and importance of energy between the EU and Russia, and why this is more then just a coincidence as many critics might lead you to believe’.

• Making reference to source-based evidence:

1. [24] Nonetheless, the studies that have been conducted show empirical, credible evidence to support the medicinal benefits of marijuana for a variety of different patients and conditions.

• Showing support for the efforts of a particular stakeholder:

1. [25] Government space agencies have rapidly and, for the most part, reliably developed technology that improves everyday life.

• Interpreting the evidence presented to support a point:

1. [26] Dr. Davis results show that no matter the regulation provided the public will abuse of medical marijuana just as they had already abused of the drug in the 1920s’; Lovinger clarifies that, LSB, Cannabinoid receptors generally inhibits neuronal excitability and neurotransmitter (Lovinger 1156). What Dr. Lovinger is saying is that when the human body undergoes a treatment or use of marijuana, the drug does not allow for the nervous system to continue working in the ways that it should’).

• Indicating that there are different sometimes opposing views on the topic:

1. [27] There is another class of individuals that ignores the factual analysis and presents a different perspective.

1. [28] The researchers have done their research about this subject in afflication with the university of Bergen and the institute of global health and community medicine in Bergen, and come to the assumption that using surrogates in communities such as India, with lower economy and a different cultural view on the role of females and pregnancy, is something they would strondly advise against.

• Highlighting the benefits of taking action:

1. [29] In most aspects of life, a common goal can lead to an unexpected partnership.

1. [30] The process has the advantage of being in its infancy and the potential to become an even more accurate procedure.

7.5 Discussion

The functional uses explored in Section 7.4 go some way to helping students meet the fundamental primary learning objectives of their programme. In the ENC 1101 task, they are expected to synthesise multiple stakeholder perspectives in the form of a literature review. In the ENC 1102 task, they are expected to balance the competing views expressed in the ENC 1101 task by attempting to show how stakeholders can reach a compromise (see Section 2). The examples show how the dependency types go some way to perhaps facilitating greater task achievement as students navigate the task of setting out the key arguments between stakeholders and then looking at how these stakeholders can reach a viable compromise.

These dependencies and their inter-related functions also appear to contribute towards students showing critical thinking in that students ‘evaluate evidence, recognize and evaluate underlying assumptions, identify and evaluate chains of reasoning, and compose appropriately qualified and developed claims and generalizations’ (CWPA, 2014). The Postsecondary Framework for Success (CWPA et al., 2011) also describes how writers are asked to ‘write texts for various audiences and purposes that are informed by research (e.g., to support ideas or positions, to illustrate alternative perspectives, to provide additional contexts’).

These results also strengthen the points made by Reference AullAull (2015), who acknowledges that language patterns are not isolated structures that appear randomly but are instead linked to the macro-level writing processes/demands of the assignment whereby these patterns facilitate achieving the macro processes.

7.6 Conclusion

This study has helped highlight the importance of implementing a type of mixed effects modelling which accounts for the random effects structure of the sampling corpus and at the same time preserves the ordinal levels of the outcome/dependent variable. The analysis highlighted how a number of linguistic and non-linguistic fixed effects appeared to have the potential to increase or decrease the odds of essays being awarded higher or lower grades. However, an inspection of the threshold coefficients suggests that these effects are not uniform or operating in the same directions for all grade levels.

In relating findings from the statistical modelling to the literature, there is some support from the Element’s findings for the comments made by Reference Durrant, Brenchley, Szudarski and BarclayDurrant and Brenchley (2021). Their analysis of L1 children’s writing encouraged us to be cautious and appreciate that phraseological sophistication or indeed the phraseological complexity, when we include the sub-construct of diversity, is not a uniform construct that develops or is evaluated uniformly across educational contexts. The final section of this study illuminates the contributions to knowledge that the modelling has made and discusses what needs to be considered as a priority for future research.

8.1 Overview of Section

This Element focussed on answering one central question: which linguistic and non-linguistic variables may play a role in shaping the writing quality scores awarded to student essays in an FYC writing context?

In doing so, we also engaged with two subquestions which occupy space in the collocation–grade relationship literature: (i) how can we choose appropriate measures of collocation? and (ii) how can we measure and understand the potential role of different linguistic and non-linguistic variables in an appropriate way? The following sections summarise the key results obtained from our work, acknowledges its limitations, and discusses avenues for future research in FYC contexts.

8.2 Findings for Question 1: How Can We Choose Appropriate Measures of Collocation?

The cluster analysis found that measures in Cluster 1 tap into the property of exclusivity. Other clusters illuminated different properties, including highly frequent pairings, as illuminated by the t-score in Cluster 5 and aspects of directionality in Clusters 4 and 5. Overall, the cluster analysis was able to shed numerical and visual light onto the relationships between different association measures. In total, six measures were identified as distinct: the MI, the geometric mean, the LLR², the Delta P w2| w1, Delta P w1|w2, and the t-score. These six measures were used in study two to answer question 2.

8.3 Findings for Question 2: How Can We Measure and Understand the Potential Role of Different Linguistic and Non-linguistic Variables in an Appropriate Way?

A regression model known as a cumulative links or probability odds model was used to answer question 2. We chose this model because it could recognise grade scores as ordinal variables as well as incorporate interval and categorical predictor variables. This model has not featured much in learner writing research, and in drawing attention to such a model, we hope the learner writing community explores possible future avenues for working with this model.

The model highlighted that the Mean MI for nsubj dependencies and the Mean MI for dobj dependencies had higher odds of increasing grades.

The model also highlighted that the following linguistic predictors had lower odds of increasing grades:

• The mean LLR² for amod dependencies

• The mean Delta P for word 2 | word 1 nsubj dependencies

• The mean Delta P for word 1 | word 2 dobj dependencies

• The mean t-score for nsubj dependencies.

Overall, the model contributed to understanding the relationship between collocations and writing quality in different ways. It reiterated the important role that the MI, LLR², t-score, and Delta P may have in evaluating learner language. Although these association measures varied in their significance and direction, they all had higher or lower odds of a grade increase with their use.

The model also highlighted that the status of the writer as a second language writer had higher odds of increasing writing quality scores. When examining the odds ratios of the linguistic and non-linguistic predictors, it is important to interpret the linguistic predictors as having at least a similar effect on grade level as the non-linguistic predictor of language status. In fact, the odds ratio of language status was higher than the linguistic predictors. This finding offers an important glimpse into underlying FYC grading practices.

Several reasons may account for the language status of the writer being significant. Although it could be argued that the reason for this result is that raters could sub-consciously favour non-native writers, it is also possible that they are more likely to receive higher grades because they have had more exposure to writing through pre-university preparation courses and are therefore more in tune with writing for an audience and writing conventions (Reference LeeLee, 2019). It is also possible that non-native writers have higher odds of receiving a higher grade because of the make-up of the raters. It is a feature of the FYC environment that essays are graded by instructors with mixed experience levels. At USF there are GTAs with far less experience than other faculty members, and while it may be attested that rater experience is a factor in rater judgements, the result in this FYC context contrasts with much of the previous literature which has found that inexperienced raters are actually more severe in their rating until they receive training to balance out their subjective views (e.g., Reference WeigleWeigle, 1999). Equally, while previous research has found that non-native writing has been consistently graded lower than native writing (Reference Huang and FooteHuang & Foote, 2010), the FYC result obtained here seems to indicate that rating behaviour is context sensitive.

The final study in the Element also added to understandings of the relationships between linguistic predictors and writing quality scores by carrying out a qualitative analysis on high- and low-association measure scoring dependencies. This analysis found connections between how high- and low-association measure scoring dependencies were used to perform key FYC rhetorical goals. This included how high-scoring MI nsubj and dobj dependencies were used to introduce the focus of essays and emphasise the goal of compromising between stakeholders. For association measures with lower odds of a grade increase, dependencies comprised of more frequent generic words which were perhaps perceived by raters as being obvious ways of creating meaning.

High scoring amod and nsubj diversity texts also showed evidence of dependencies being used to perform particular rhetorical functions, including setting out issues with the topic over time and critiquing the topic, referring to sources as evidence, showing support for a particular stakeholder, capturing the range of different positions/views held about the topic, and emphasising the benefits of taking action/moving towards a compromise between stakeholders.

The study also adds weight to previous phraseology grade work in that it also finds that relationships between collocation and writing quality are not homogeneous but instead context specific, with other grading factors also playing a role in final grade allocation alongside the use of collocations (e.g., in line with Reference Durrant, Brenchley, Szudarski and BarclayDurrant & Brenchley, 2021).

Although the present study aligns with previous work in supporting the MI and t-score measures, the respective dependencies that have been illuminated are different. This Element also supports the diversity of amod and nsubj dependencies as having greater odds of increasing grade scores. However, there are several differences in the Element’s methods and context that need to be acknowledged as potentially contributing to these differences. One such difference is that the Element is based on dependency types as opposed to the use of adjacent pairs which have been automatically extracted in much of the previous literature (except for recent studies such as Reference Jiang, Bi, Xie and LiuJiang et al., 2021 and Reference PaquotPaquot, 2018, Reference Paquot2019, who also focus on dependencies).

Another reason for different results lies in the types of texts analysed. In previous studies, there has been a focus on second language proficiency where evaluations are concerned with whether or not students’ proficiency is adequate enough for university study. This was the case in both Reference Garner, Crossley and KyleGarner et al.’s (2019, Reference Garner, Crossley and Kyle2020) studies of the Korean placement test. The texts in this Element are extended essays which have been drafted and reviewed by instructors and peers. This type of writing differs from much of the literature which has involved short, timed writing (e.g., Reference Garner, Crossley and KyleGarner et al.’s (2019, Reference Garner, Crossley and Kyle2020) placement tests were timed as were the descriptive essays in Reference Bestgen and GrangerBestgen and Granger (2014). Finally, Reference PaquotPaquot’s (2018, Reference Paquot2019) work also differs from the present context by focussing on coursework texts written by postgraduate linguistics students who were evaluated under the CEFR scale.

The FYC context also evaluates a broad range of rhetorical and curriculum goals as opposed to the narrow focus on ESL courses that have been studied before. These types of different foci have been highlighted by Reference LeeLee (2019), who notes that ESL courses cover how to write and place emphasis on explicit language use, while FYC courses orient towards assessing multiple rhetorical goals and favour implicit language use.

8.4 Implications for the FYC Programme at USF

Overall, the methods and studies in this Element have contributed to a wider drive in the FYC literature to complement composition instruction with that of language support. The Element promotes the use of advanced statistical techniques to understand how aspects of collocation may contribute to grade allocation and ultimately facilitate students’ task completion and achievement of FYC programme goals. This Element supports the work of other scholars who have stated that language support should be embedded into FYC student learning, and more broadly FYC programme outcomes and frameworks (e.g., Reference AullAull, 2015, Reference Aull2017, Reference Aull2019; Reference Eckstein and FerrisEckstein & Ferris, 2018).

Our findings have implications for instruction and assessment on USF’s first-year writing programme. Section 7’s follow-up qualitative analysis highlighted how texts with high amod and nsubj diversity contained amod and nsubj dependencies that were used to perform a number of rhetorical functions. The connection between the association measure scores and their diverse use is important for future research. Future research needs to look at how the high- and low-scoring association measure combinations are being used to develop a sense of high or low achievement in the FYC tasks. This kind of sustained quantitative-qualitative examination would help identify how language is being used to facilitate meeting task and wider FYC programme goals. To this end, banks of authentic learner language may be invaluable for instructors as complementary aids that supplement the external ‘community comments’ that already exist within the My Reviewers platform.

These authentic language examples would allow instructors to clearly connect their process-oriented rhetorical instruction with examples of learner language that show students how to navigate the tasks and demonstrate ownership of their own texts. This dual approach means students receive both composition instruction and academic language support. Students should be supported to notice the wide array of language choices they have available to them in conveying their intended meaning. They should also be encouraged to make choices that are befitting of the tasks set.

8.5 Implications for Automated Scoring and Feedback Systems

The modelling work in this Element also has important implications for the possible creation of automated scoring and feedback systems on large-scale FYC programmes. There is clear opportunity to use the model equations our work is based on to inform how predictive automated systems may be trained to automatically score and give feedback on student texts (e.g., see Reference Haberman and SinharayHaberman & Sinharay, 2010). The model has important implications for the training of automated models of scoring and feedback which must balance of variables that underlie the grading process. We have modelled the influence of individual writer, task, and language status in this Element, and future training for models should be encouraged to include these variables in their work. This would be a valuable step towards modelling predictions with multiple variables in the future.

8.6 Critical Reflections and Ways Forward

It is important to interpret the study’s findings with a degree of caution and awareness of the caveats that apply to the research design. A first acknowledgement is that the measurement of sophistication in the form of association measures is limited by measuring sophistication using only association measures. In the future, studies might adopt a more multidimensional approach to studying sophistication with more than one measure type. This may include looking at lists of academic collocations (e.g., in a similar manner to Reference PaquotPaquot (2019)). The second study design decision that shapes the view of sophistication is the choice of reference corpus. While the choice of reference corpus was register appropriate for learner writing, this limited the view of sophistication by only looking at combinations present in another specialised corpus of writing, where size may have played a role in limiting the amount of language we could look at. A different picture of sophistication may be presented when the reference corpus is general in nature (e.g., use of the BNC or COCA) but it is still important to recognise that even in large general corpora multi-word units like dependencies and collocations more broadly are sometimes not found.

The study sample and our understanding of writers’ first language was restricted by relying on complete responses from learners. This information was gathered via a voluntary student survey, and this means that the inferences we can draw from these learner variables are limited to having enough complete information. Future modelling work with big data FYC repositories like this one will need to be carried out with this issue of patchy data in mind. Although mixed effects modelling has been promoted as a statistical technique that can handle missing data, in the case of the FYC repository, there are cases where absent data is the norm, and careful consideration in future work will be needed to determine exactly the amount of missing data such modelling is able to handle. As a field, this is an issue that learner corpus research is only starting to explore and as a community it is an endeavour that should be pursued for the foreseeable future in our work (e.g., in line with recommendations made by Reference GriesGries, 2015).

A further limitation relates to the exclusion of topic as a variable in the study. Students are permitted to choose their own topics, meaning that inferences and coding the variable for analysis depend on enough students choosing similar or the same topics. As a result of the idiosyncratic topic choices, grouping topics together was not possible. However, in the future the inclusion of topic as a random variable may be possible in a larger FYC sampling frame.

It is also pertinent to reflect on the limitations and caveats that apply to using cluster analysis. Although we took steps to show the validity of our working, there are several caveats that readers should be aware of. We choose a particular set of association measures, of which we accept are not an exhaustive list, but are instead one that covers multiple measures that have frequented linguistics research. Readers of other lines of word association research may uncover other measures or ponder the placement of the ‘contextual measures’ that we omitted for their costly calculation and infrequency in the literature we drew upon. Our clustering solution is also based on making important pre-analyses choices: the decision to scale and standardise the data to maximise the uniformity of the association measure values, the decision to employ farthest neighbour clustering, and the decision to choose spearman correlations to calculate the distance matrix. We made these decisions with reference to key statistical literature; however, other analysts who make different decisions at these junctions will no doubt obtain a slightly different clustering solution.

Similarly, there are caveats for our modelling. We opted to use cumulative links mixed effects modelling because of its ability to treat the dependent variable of ‘Final_Grade’ as ordinal. This aligns with other educational research (e.g., Reference O’ConnellO’Connell, 2006). We hope that our approach here plays a role in the modelling conversations that are currently taking place in the movement of examining learner writing via these kinds of models because there is still much to ponder about mixed effects modelling, particularly so when managing unbalanced data sets such as the FYC one we encountered (see also Reference McCallumMcCallum, 2019).

In this light, there are several other directions that future research should be encouraged to pursue. One direction is the relationships between context-specific measures of association. While the present study opted to focus on computationally simple measures of association, there is scope for carrying out further relationship-based work that looks at how mathematically complex context measures may tap into different aspects of context and collocation properties (e.g., see Reference Gries, Durrant, Gries and PaquotGries and Durrant’s (2021) support for KL divergence as a viable candidate of study).

With respect to the context, this study represents one of the first collocation-grade studies in FYC literature, and future replications of this work across other FYC contexts would strengthen claims that instruction and assessment on these programmes would benefit from being informed by (a) corpus linguistics techniques and (b) EAP pedagogic methodologies that home in on language as a central component of composing texts.

A penultimate direction is to acknowledge that these exploratory patterns are obtained indirectly via corpus data and their interpretation is limited, if only the corpus is used, to look at the rationale for such patterns. To this end, our study should be viewed as a starting point for further qualitative exploration of the construct of writing proficiency. The patterns and language examples may be further used in psycholinguistics and interview-based research to tap into why these language examples and patterns of positive or negative relationship with writing quality scores may occur. This kind of qualitative work would act as a natural complement to the statistical and qualitative analyses carried out in this Element.

A final direction relates more broadly to the grading process. The modelling process indicates that second language writers have higher odds of a grade increase, and to some extent this quantitative heavy modelling also supports the qualitative picture built up in previous grading studies (e.g., Reference Huang and FooteHuang & Foote, 2010). A qualitative follow-up study on this finding may help illuminate potential trajectories of FYC rater unconscious bias.

Overall, the modelling process has illuminated several statistical patterns that should be further investigated qualitatively. This would provide a fine-grained understanding of the relationship between collocations, dependencies, and learner and contextual FYC programme variables. This is a highly viable direction that future FYC work should be encouraged to take.