An investigation into online moderation

Citation metadata

Author: Lenore Adie
Date: Annual 2011
From: Assessment Matters(Vol. 3)
Publisher: New Zealand Council for Educational Research
Document Type: Article
Length: 7,065 words

Document controls

Main content

Abstract: 

The purpose of this paper is to understand how teachers' identities as assessors in a standards-referenced assessment system may be developed through their participation in online social moderation meetings. In these meetings, teachers negotiate and share their understandings of assessment standards and judgement decisions. In particular, the paper focuses on the relationship between the technology, the moderation processes and teachers' development in this assessment system. This paper draws on sociocultural theories of learning to analyse the qualitative data collected through observations of 11 online moderation meetings and interviews with the teachers involved in these meetings. The results provide insights into the mediating role of the technology with regard to teachers' development of shared meanings and common practices within a standards-referenced assessment system.

Full Text: 

Introduction

Internationally, many education systems have been moving towards systems of standards-referenced assessment as a response to the needs of learning and working in the 21st century, as well as increasing systemic requirements for accountability (Centre for Educational Research and Innovation, 2005; Zepke et al., 2005). Standards articulate the skills and knowledge that students should have attained at a particular juncture in their education, and the quality of their performance of the skill or knowledge. It is claimed that standards provide a defensible framework for informing, substantiating and making judgements (Wyatt-Smith & Bridges, 2006) and a mechanism for tracking student progress (Sadler, 1987). Yet questions are raised about the reliability of such assessment when it employs judgements of quality.

Within a standards-referenced system of assessment that relies on teacher judgement, test validity may be high but reliability can be compromised. Factors such as marker variability and bias, and differences between markers have been identified as affecting reliability (Black & Wiliam, 2006; Linn, 1993). A number of studies (Harlen, 2005; Maxwell, 2002; Wyatt-Smith, 1999) have revealed that teachers bring to an assessment task a diverse range of historical, cultural and social experiences that may work together to produce different interpretations of stated standards. This can occur even with the provision of marking guidelines and exemplars. For standards to have validity and reliability as an accountable assessment system there needs to be consistency amongst teachers' understandings of what counts as quality (Black & Wiliam, 2006; Stobart, 2006; Wiliam, 1998).

It has been suggested that social moderation is one way that the development of shared understandings of standards and quality assessment practices may be promoted (Harlen, 1994; Matters, 2006; Maxwell, 2006). Social moderation describes the process of teachers purposefully gathering to reach agreement on the standard of students' work. Maxwell (2002) defines this as "a process for developing consistency or comparability of assessment judgements across different assessors, programs and schools" (p. 1). The importance of social moderation in teacher professional development has been endorsed and continues to be endorsed by many education authorities and authors (Assessment Reform Group, 2005; Gipps, 1994; Harlen, 1994; Queensland Department of Education Training and the Arts, 2003; Roberts, Wilson, & Draney, 1997).

It is proposed that the development of teachers' judgement capabilities is facilitated by moderation processes as teachers become situated in a "community of judgement" (Wilson, 2004, p. 4) or an "assessment culture" (Wyatt-Smith & Bridges, 2006, p. 11). This process "functions as a motivation for teacher change as well as a catalyst for changing the assessment culture of the school or district" (Roberts & Wilson, 1998, p. 1). Involvement may support teacher assessment capabilities by developing professional dialogue, providing reassurance and offering opportunities for continuous professional growth (New Zealand Qualifications Authority, 2007).

Studies (Klenowski & Adie, 2009; Scarino, 2005; Wyatt-Smith & Bridges, 2006) have shown that when teachers meet face to face to moderate their judgement decisions at a local level, consistency of understanding of a standard may be achieved. However, problems occur when such calls are made for system-wide consistency and shared understanding of standards. A central question is: How can teachers develop a shared understanding of the qualities that denote different standards of performance across dispersed populations and remote regions?

The research discussed in this paper draws on the current Queensland assessment context as one example of a system-wide response to standards-referenced assessment. Queensland, Australia, provides an ideal location to study the potential of online moderation because the great distances between schools within the state prohibit the gathering of teachers face to face to discuss their understandings of curriculum, pedagogy and assessment. The area of Queensland is 1.72 million square kilometres, and some isolated schools can be at least 200 kilometres from their nearest neighbouring school (Queensland Department of Education Training, 2010).

Between 2006 and 2008, a new curriculum, assessment and reporting framework, which focused on the middle years of schooling and involved the establishment of essential learnings, defined standards and a common reporting system aiming to promote consistency of teacher judgement, was trialled in cross-sectoral schools in Queensland (Queensland Department of Education and the Arts, 2005). The research reported in this paper focused on a group of 50 teachers from 21 schools who were involved in moderating judgements in a synchronous online environment. Participating in social moderation within an online environment is a new practice for teachers and has not been adopted system-wide in any educational jurisdiction. This study attempted to understand how working in an online environment may contribute to, or inhibit, teachers' identity formation as assessors in a standards-referenced assessment system.

Literature review

The research formed part of an Australian Research Council (ARC) Linkage project (Project ID: LP0668910): Investigating Standards-Driven Reform in Assessment in the Middle Years of Schooling. Papers and presentations from this project have been concerned with the role and purpose of standards in teacher judgement and the development of consistency of teacher judgement through the process of social moderation (for example, Klenowski & Adie, 2009; Klenowski & Wyatt-Smith, 2010). This paper adds to this discussion by reviewing the moderation practice conducted in an online mode of moderation.

There have been limited studies conducted investigating the link between social moderation and teachers' professional development. Those conducted have returned positive results relating to the development of common understandings amongst small groups of teachers. For example, in a study of the consensus moderation process in Victoria, Australia, Ingvarson (1990) found that teachers' assessment knowledge increased with their participation in moderation processes over time. An evaluation of the local consensus events held in Queensland (Klenowski, 2007) reported benefits for both systemic requirements and teacher professional development, with a common understanding of the standards developing for teachers. Similar findings were found in much smaller case studies by Davidson (1999) and Malone, Long and De Lucchi (2004). Davidson (1999) reported on one Victorian school's implementation of the moderation process that involved teachers exposing their beliefs about learning and their teaching practices while working towards a consensus of assessment standards that also resulted in personal and professional development. Malone et al. (2004) discussed a trial of the moderation process conducted through the Lawrence Hall of Science, California, with a small group of teachers to build consensus of progress levels in an effort to improve student learning. These authors reported positive professional development for the teachers.

The Ontario Ministry of Education (2007, p. 2) describes the moderation process as involving the "collective wisdom of all participants" that supports the critical examination of assessment data and leads to targeted and improved instructional practices. The notion of critical examination is explored by Elmore and Jones (2007) who describe a practice of professional discussions regarding standards amongst school superintendents in Connecticut. These discussions were guided by a set of protocols that work to "separate the person from the practice" (p. 23). The authors noted that educators can feel challenged in these discussions when practice is embodied in the person. This finding was supported by Little, Gearhart, Curry and Kafka (2003) in their study of four school sites involving teacher examination of student work. While the benefits of teacher discussion of assessment judgements were apparent in this study, the authors also noted the difficulties that teachers faced in making their practice public amid the multiple dimensions of judgement making, and the time constraints of the meeting. However, the findings provide some insight into the practices that teachers employed in these discussions that may lead to teachers' learning with the goal of improving student work.

Other studies (Klenowski & Adie, 2009; Scarino, 2005; Wyatt-Smith &

Bridges, 2006) have also shown how, through participation in moderation discussions, teachers are encouraged and supported to situate assessment practices within the sociocultural context of their classrooms. In this process, teacher agency is developed and assessment is aligned with pedagogy for the benefit of student learning. Teachers need to see the benefits of involvement in the moderation process for student learning before commitment to such a rigorous exposure of their practices and understandings can be assured.

While providing evidence of the value of teachers purposefully meeting to discuss their judgement decisions, most of these studies have been concerned with local practices and not the development of common understandings of standards across the body of a state education system. Furthermore, when these practices are transferred to an online mode of moderation, there exists little empirical data investigating the impact of this context. The research reported in this paper will build on the findings from face-to-face meetings and contribute to the development of insights into the context of online social moderation as this contributes or inhibits teachers' developing identities as assessors within a standards-referenced assessment system.

Theoretical framework

The investigation of online moderation practices is approached from a sociocultural view of learning (Lave & Wenger, 1991; Vygotsky, 2003; Wenger, 1998), and a sociocultural view of technology (Slack & Wise, 2005). A sociocultural view of learning situates learning in the act of participation while acknowledging the different histories and understandings that converge in a gathering such as an online moderation meeting. The development of knowledge is understood as a social practice in which participants draw from a repertoire of resources to construct, share and reconstruct meanings in the process of developing common understandings. Positions taken by the teachers in the online moderation meeting reflect their understandings developed in the local context. The theoretical framework is grounded in sociocultural practices that focus on the relationship between meaning, practice and identity formation.

The technology in this context is viewed as agentive and contributing to the development of common understandings by affecting the interactions that take place (Slack & Wise, 2005), and therefore the learning that occurs. Slack and Wise (2005) use the concept of technological agency to understand the computer through the social and cultural context of which it is a part. This context has given a certain meaning to how the computer is perceived and used, the power attributed to it in this role and the contribution it makes to forming and shaping identity within a practice. This perception of technology focuses analysis on the many connections between the multiple factors that are at play within dynamic and changing contexts.

The definition of agency in this context differs from the commonly held definition of this term as usually involving humans, requiring acts of intention, and as something that is possessed. Slack and Wise (2005) define agency as a process or a relationship that involves participants, but they do not limit this participation to humans alone. Agency refers to "the ability to bend space, to make something happen" (Slack & Wise, 2005, p. 131). This redefinition of agency broadens the concept to include technologies as participants that can be involved in relations of agency. It is important to note that Slack and Wise (2005) do not attribute the computer with abilities to respond to engagement. What appears to be a radical definition of agency, on closer inspection, has been qualified to include technologies while not attributing to those technologies human qualities and ways of interacting, or any sense of intention. When technologies are viewed as agents in everyday life, investigations are opened up to consider the part played by the technology in transforming or contributing to an outcome. For example, teachers involved in online moderation may receive a weak connection and keep dropping out of the meeting, therefore reducing their contributions and providing a negative impression of meeting in such an environment; or the dynamics of turntaking in such an environment may cause frustration and inhibit the natural flow of the conversation. Such factors relating to technology play a part in shaping the conversation that will take place. The role of the technology cannot be negated in the dynamics of this context.

Considering Slack and Wise's (2005) notion of technological agency with Lave and Wenger's (1991) concept of communities of practice provides one way to investigate and analyse the connections between human and nonhuman participants in the act of identity formation within a practice. It provides an avenue to investigate how meaning is established, and how practice and identity may develop.

Research design

The research question for this study focused on the role technology plays in shaping teachers' interactions and their development of shared understandings of the stated standards. The question was framed as:

How does teachers' involvement in online moderation meetings support their participation in the practice of standards-referenced assessment?

The aim of this question was to identify how teachers' involvement in an online moderation meeting acted to support both teachers' practice of, and identity formation in, a standards-referenced assessment system.

This study is concerned with understanding the relationships between the various elements (human or nonhuman, concrete or abstract) that connect with the practices of online moderation and standards-referenced assessment. The links between the concrete and abstract elements being studied are understood as contingent and dependent on the various other elements within the sociocultural context (Slack & Wise, 2005). To understand teachers' interpretations of the standards and their co-constructed meanings arrived at through discussion and negotiation, the research design of this study needed to support methods that place the researcher "close to the action". A research orientation that focuses on learning as a social process is concerned with investigating social practice and the evolving place of the participant within this practice. The language and actions of the participants are indicative of their learning trajectory as knowing how to talk and act within a practice are essential to inclusion as a participant of a practice (Wenger, 1998). Conceptualising the technology as an agent (Slack & Wise, 2005) in the process of online moderation broadened the investigation to include exploration of how the technology interacts and affects in some manner the negotiation of judgement decisions and the development of an assessment identity.

The research involved Queensland middle school teachers at different year levels (Years 4, 5, 6 and 9), in different curriculum areas (English, science and mathematics), in diverse geographic locations, and in a range of sociocultural contexts.

Teachers in this study met online to moderate student work using the WebEx[C] online meeting centre. WebEx[C] allows for audio, video and text to be incorporated in meetings through the sharing of documents, applications and desktops. Participants were invited to "attend" the online moderation meeting through email and communicated during the meeting via telephone while interacting with the materials online. Software features such as the hands-up icon allowed participants the opportunity to contribute to the discussion. Participants met in real time to view or annotate student work samples using highlighters, text or pointers. While WebEx[C] allows for audio, video and text to be incorporated into meetings, only the audio and text were available for this project. Consequently, participants could view and annotate samples of student work, but could not see each other. To participate in the online moderation session, teachers needed to have access to a phone, preferably one that was handsfree and located close to their computer. Figure 1 shows a screen shot from one of the online moderation meetings.

For the purposes of this paper, there is a focus on the qualitative data collected through observations of 11 online moderation meetings conducted during 2007 and 2008 and interviews with the middle school teachers involved in these meetings conducted pre- and post-moderation. The researcher acted as facilitator of these meetings--setting up the meetings and supporting teachers with use of the technology but not being involved with the moderation of student work samples.

Each online meeting involved two to eight teachers from two to four schools discussing common assessment tasks. At times a number of teachers from one school were clustered around one computer. The conversations of 10 of the 11 meetings were transcribed.

The teachers involved in this study were moderating their judgements on common assessment tasks. The moderation practice was new for the majority of middle school teachers involved in the study.

Table 1 is a summary of the data collected and lists the number of meetings that were run and those that did not run, and the number of pre- and postmoderation interviews that were conducted.

Table 2 details information about the teachers involved in the meetings in terms of their gender, school sector, school level (primary or secondary), year level (Year 4, 5, 6 or 9), discipline discussed in the meeting (English, mathematics or science) and their locality (remote, regional, city). Of this group, one teacher participated in an online meeting in both 2007 and 2008.

Methods of grounded theory were used to analyse the data (Charmaz, 2006). This version of grounded theory views the researcher as involved in and influencing the research context and understands the development of theory as an "interpretive portrayal of the studied world, not an exact picture of it" (Charmaz, 2006, p. 10). As such, the theoretical framework used in this study was useful for directing the gaze of the analysis, but it did not provide predetermined codes. Coding of the data commenced from the beginning of data collection, so that constant comparison of the data was an ongoing process. Data were coded and labelled according to the analytic sense that became apparent through immersion in the field and with the data.

In grounded theory, the focus is on the analysis and understanding what is happening in an event. The review of Lave and Wenger's (1991) work and Wenger's (1998) work, in particular, has provided a possible framework from which to view the collected data and to respond to the research questions. When answering the grounded theory question of "What is happening here?" the focus is on the development of practice and identity within systems of standards-referenced assessment as this is supported through participation in online moderation.

Findings and discussion: The mediating role of the technology

The study focused on teachers' practices in online moderation and sought to understand how involvement in this process supported the development of an assessment identity. Sociocultural theories of learning emphasise the multiple dimensions impacting on each participant and thus the instability of the connections between participation and learning. In the following section, three different factors that weakened these connections are discussed with reference to teachers' developing identities in a standardsreferenced assessment system. The factors considered are the lack of visual cues, the issue of electronically reproduced items and teachers' declared anxiety with the technology.

First, in many of the interviews teachers expressed the discomfort they experienced when they were required to expose their practice, question others' decisions and defend their own, particularly in a context which lacked the visual cues of face-to-face interactions. The following response is representative of the opinions expressed by the majority of the teachers involved in the online meetings:

The fact that you don't know the people that you're moderating with has its advantages, and it also has its disadvantages in that you can't see how they're reacting to the sorts of things that you're saying about the piece of work ... So many of us are so good at reading body language and facial signs and things like that, that it's very difficult to operate without those, and I sort of know that from just even moderation within [sic] people that you know really well, sometimes something will be said, or some suggestion made, or a point made, and you know straightaway that they either agree or disagree, even though they might say something different, by their facial expressions. So I guess that, that was all lost. (Teacher 1, Post-moderation interview, November 2008)

The teacher's comment illustrates the displacement in this online environment that does not meet a socially and culturally constructed expectation of effective communication. As another teacher stated, "There's nothing quite like having the round-table discussion" (Teacher 2, Post-moderation interview, November 2008). In face-to-face interactions, facial cues provide information that participants can use to clarify the meaning of the spoken words. The inability to use facial cues to discern and convey meaning of utterances in the online meetings placed teachers, in their own reckoning, at a disadvantage in this context, both as a receiver and provider of messages. The strategies teachers reported that they used to determine the true intent of a statement were not available in the online meetings. As a result, teachers asserted that this restricted their participation in the negotiation process of the moderation.

In addition to the visual limitations of the meetings, teachers needed to consider how to produce legible documents that could be read online. In one of the meetings, a teacher had typed the student scripts verbatim because of the poor quality of the scanned samples. While this task was time consuming for the teacher, it also produced uncertainty for the other teachers moderating the student samples from this school. The teachers were unsure of what work had been generated by the students themselves, especially with regard to spelling and punctuation. While the issue was resolved for the teachers through their conversation, it raised important ethical issues for the practice of online moderation that includes the need for scripts that can produce a legible image when scanned. For example, is there a need for teachers to declare that they have typed students' texts and include their reasons for doing so; and, further, should the original work samples be sent with

these typed texts for verification? It is not the intention to answer these types of queries, but rather to signpost the new set of expectations that may emerge if this practice is accepted as a way of doing moderation.

Finally, fear of the technology presented as a significant inhibitor of the teachers' successful participation in the moderation practice. For example, in the following extract, the teacher used the words fear and threatened in relation to technology when discussing her teachers' involvement in online moderation. She acknowledged that as familiarity with technology develops, the fear that her teachers experienced would reduce, but that the teachers will require support to develop the confidence needed to work online:

See, all of my teachers are very much aware of the value of moderation itself. I think their fear of online moderation would be the technology, because we don't have much, in early years particularly we don't have much technology in the classroom, so they would feel a little threatened by having to use that technology ... That's the factor I think that we need to deal with, and encourage people, but again, it's one of those things that the more you use it, the less the fear becomes, you know, a problem. (Teacher 3, Post-moderation interview, February 2009)

The level of anxiety associated with using the technology can inhibit successful functioning in the meeting. In such a situation the online meeting has a weakened chance of contributing to the teacher's developing identity as an assessor in a standards-referenced assessment system. Teachers may experience such a negative reaction to the technology that they fail to fully participate in the online moderation practice.

The three factors discussed have illustrated how the technology can act to inhibit teachers' developing assessment identities within the standardsreferenced context. Yet there were also elements of the technology, that when incorporated into the moderation discussion, acted to strengthen assessment identities by involving teachers in focused and deeper discussions of the assessment evidence in relation to the stated standards.

The capability to overcome some of the difficulties of working with the technology was built into the WebEx[C] program through the annotation tools. When these tools were utilised to support the communication act--in particular, the negotiation process and the development of a shared meaning of a standard--they proved to be a valuable aid. The teachers used the tools to highlight and annotate the evidence in a piece of student work that they believed illustrated a particular standard. The annotation tools provided participants with a space to position themselves within the practice. For example, in the following extract highlighting the text is used to lead the other participants to exactly where they should be looking for the evidence, as it is discussed by the teacher. (See Figure 2 for the screen shot of this section of the meeting.)

So this was originally a B, we originally gave this one a B because of this question. And then we changed it because we saw this [marking with highlighter] . and this bit of working out at the bottom here [highlighted], as well ... which shows the correct answer. So, we originally had a B, but then we changed it to an A because we found the 1110 metres and the correct amount of steps . even though if you look on the next page, it's got the wrong answer. (Online moderation meeting, Year 6 mathematics, November 2008)

The tools directly contributed to the teachers focusing on the evidence provided in the samples. All participants were able to access and use these tools. The annotation tools acted to attribute an equal opportunity of power within the online meeting. While teachers found it difficult to "know when to speak", by accessing the annotation tools and highlighting a part of the work that was being discussed, teachers could contribute to any discussion. Yet these tools existed within the social relations of the meeting. By engaging with the tools participants were declaring and exposing their thinking, and perhaps their intention to talk (and further expose their understandings). It is the interaction between the technology, other artefacts, the language used and the participants that positioned participants within the meeting and attributed them power to contribute.

Teachers could choose to exercise that power in meetings, but many did not take up that opportunity. This is illustrated in a post-moderation interview where the teacher acknowledged her discomfort with the technology and then reflected on how, by the end of the meeting, engagement with the annotation tools focused attention on the evidence in the student's work:

I guess I was a little bit nervous about using the technology and I felt that it wasn't sort of until the end that we were actually doing that, you know, highlighting and writing things and sort of sending them off to each other. So, yeah, I did feel that at some points we weren't looking at, you know, the pointatable [sic] sort of evidence that put the student in the different standards for the different products, but we were sort of talking generally, but as the--as it [the meeting] progressed, then we got better at that. (Teacher 4, Postmoderation interview, November 2008)

In this extract, the teacher recognised that as the meeting progressed, the teachers started to use the tools more effectively and that, once this occurred, they then started to focus on matching the evidence to the standards. The simple act of using the annotation tools to highlight evidence was instrumental in the teachers' closer examination of the evidence, and matching this specifically to a standard.

While the technical environment supported other ways to communicate, participants needed to accept and use these tools as part of the moderation practice. The computer in such a practice is viewed as a new mediational means (Vygotsky, 1997). The use of this new mediational means for conducting moderation resulted in a change for the individuals involved in the practice. In holding on to their historic identities as face-to-face communicators, the teachers passed blame to the computer for their lack of communication skills in this new context. It is apparent that new skills will need to be practised and learned to master the communication demands of online moderation.

Sociocultural theories of learning highlight the influences of historic practices in learning a new practice. Historic relations with technology positioned the teachers to be receptive in differing degrees to possible learning opportunities from their involvement in the online moderation meetings. Yet this relationship is ambiguous. Close attention was paid to teachers' interactions in the meeting in terms of their working with (or their frustration with) the technology, and how this behaviour resonated with their pre-and post-moderation comments.

It is proposed that to participate in an online context, teachers need to be willing to work with the technology. This entails willingness on the part of the teachers to have a go, and to problem solve if difficulties arise. Being computer literate assisted in this context but, as the following examples will show, was not a requisite factor. For example, in one online meeting, one teacher was experiencing problems with his speakerphone while trying to negotiate with another whose phone connection was very poor. Nevertheless, the teacher had followed the conversation so far and continued to ask clarifying questions and to paraphrase, to ensure they had a correct understanding of the other's reasoning:

Teacher 5: It's probably just easier holding the phone to the ear, I think. Okay, we're looking at the same sample. And we're looking at the product number 2 and 3. Is that correct?

Teacher 6: [unclear]

Teacher 5: What was the comment?

Teacher 6: That the things between the advertisement and the specific school community wasn't clear enough. It was an environmental message [unclear] you know, go out, dig a hole, plant a tree but I was expecting my students to have much closer [unclear] something that [unclear]. (Online moderation meeting, Year 9 English, November 2008)

This conversation continued with Teacher 5 restating Teacher 6's message to clarify her intended meaning. Teacher 5 is experienced in the practice of social moderation and he persisted with the protocols of the practice despite the hindrances of the technology. His historic identity within the practice of social moderation supported his continued persistence. However, this was not the response in all meetings. In a counter example, another participant who was also experienced in social moderation practices withdrew from a meeting. The conflict of identities as a moderator and as technologically competent was evident as teachers apologised for their early exit from the moderation meeting, yet cited the technology as inhibiting the functioning of the meeting. Wenger's (1998) notion of power as "the interplay between identification and negotiability" (p. 207) is useful in understanding the conflict of identities as these inhibited the effective negotiation of the judgement practice.

The question that arose from these observations and analysis focused on the factors that may influence teachers' successful participation in the online moderation. One recurrent theme that appeared unrelated to teachers' experience with technology or with moderation practices was a problem-solving attitude coupled with a willingness to "have a go". For example, in her post-moderation interview, one teacher stated, "I liked using the computer, it was fun." This teacher's narrated identity connected with the technology. The experience was a good one, in fact, it was "fun". Indeed, all of the teachers involved in this particular online moderation meeting commented positively about their experience and their learning as a result of this meeting, even though this meeting was riddled with technological mishaps. Yet the teachers laughed their way through these mishaps and were still laughing about them in the followup interviews. The following extract from this online moderation meeting illustrates such a happening. In this example, the teachers have marked on the continua where they believe the standard of a criterion should be graded but after further discussion agree that this judgement should be changed. One teacher asks how to erase her first annotation, and then another's highlighter colour changes unexpectedly:

Teacher 7: How do I get rid of mine [her annotation]?

Researcher: You go to your eraser in your tools ...

Teacher 7: Yeah.

Researcher: ... and you click on the down arrow and it says "Clear my pointer".

Teacher 7: There we go. I got it.

Researcher: Yep. Okay. So you're happy with that then? ?: [undecipherable].

Researcher: Oh, you changed colours [name].

Teacher 8: [Laughing] I have, I don't know how it's happened. [Laughing]. (Online moderation meeting, Year 6 mathematics, November 2008)

This extract demonstrates the teacher's willingness to learn how to work within this technological context by asking questions, and the evident enjoyment at being successful in this context: "I got it." This teacher had self-identified in her pre-moderation interview as a technological novice, evidenced further by her problems with electronically sending her work samples without the support of her school technician. It also shows the other teacher's carefree disposition and lack of anxiety when unexpected happenings occurred with the annotation tools. Successfully working within this technological context did not necessarily correlate with being an old-timer in the practice of online communication but rather with being one who is willing to problem solve when working with the technology.

Another interesting anomaly was the observations of those teachers who were familiar with working in an online teaching environment. For example, three of the online moderation meetings that were conducted involved teachers from Schools of Distance Education. These teachers work with students in mostly remote locations through increasing amounts of online collaboration. Observations of these meetings showed that these teachers had already adopted a number of protocols and practices which other teachers new to the practice were still developing. For example, in one meeting, instead of persisting with a bad connection, as many of the teachers did, the teachers from the distance education school logged off and then reconnected to another computer with a different phone:

We're going to have to dial off for a moment and then we'll come back in with a different phone so hopefully we can communicate better. (Online moderation meeting, Year 6 science, November 2007)

Spoken this way, "logging off" appears such a simple act. Yet this act of "logging into" the meeting was daunting for the majority of teachers, and the thought of disconnecting and reconnecting to the meeting was not considered by many participants who instead persisted with bad telephone connections. The observed practices of the distance education teachers illustrates how, as new experiences become common practice, transformation in terms of identity and practice may occur.

Identity and practice are built through interactions with other participants in the online moderation meeting, and further constructed and reconstructed through the other interconnected networks that constitute the sociocultural environment. Technology is not a neutral object in this context (Slack & Wise, 2005). The teachers' identities shape, and are shaped by, the technological culture in which they must operate. The success of teachers to negotiate judgement decisions with reference to standards of performance in the online moderation meeting is related to their relationship with the technology. Teachers do not enter these meetings equally. Differing levels of technological competency can be both a supportive and inhibiting factor to successful participation in the negotiation practice of the online moderation meeting and the development of a shared understanding of performance standards. Yet, as one's identity as proficient in the use of the technology for the purpose of moderation develops from novice to expert, the ability to focus on the job at hand--that is, matching evidence to the standard and interpreting the evidence--should become less onerous.

Conclusion

For these participants, the lack of preparation for--and thus, knowledge of--this new practice was problematic for the introduction of online moderation meetings. The teachers attempted to access and use their historical knowledge of how to communicate, how to moderate and how to assess. Online moderation brings together multiple historic and new practices, to form ultimately a new way of conducting moderation that may address systemic concerns for consistency in a standards-referenced assessment system.

It is evident that online modes of moderation will need to incorporate new protocols for interacting that teachers will need to adopt. Currently at issue is whether teachers will be deterred by their first experiences to the extent that they are not interested in learning more about this mode of conducting moderation. Evidence from the study reported in this paper has shown that the technology can support the negotiated practice of moderation when teachers use the tools to focus on the evidence as this aligns with the standard. Online modes of moderation offer an opportunity to connect teachers from different sociocultural contexts, across diverse areas and dispersed populations to share their understandings of standards, and in so doing develop their assessment identity.

References

Assessment Reform Group. (2005). Assessment systems for the future: Working paper 1: Aims and outcomes of the first year's work of the project. Retrieved 23 June 2011, from http://www.educ.cam.ac.uk/research/projects/arg/ASF%20 Working%20Paper%201%20Draft%2010.pdf

Black, P., & Wiliam, D. (2006). The reliability of assessments. In J. Gardner (Ed.), Assessment and learning (pp. 119-131). London: Sage.

Centre for Educational Research and Innovation. (2005). Formative assessment: Improving learning in secondary classrooms. Paris: Organisation for Economic Co-operation and Development.

Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. London: Sage.

Davidson, J. (1999). All things in moderation: A whole school approach to authentic assessment. In H. Fehring (Ed.), Literacy assessment: A collection of articles from the Australian Literacy Educators' Association (pp. 81-84). Newark, NJ: International Reading Association.

Elmore, R. F., & Jones, J. F. (2007). Professional networks and school improvement. Arlington, 6(4), 20-24.

Gipps, C. V. (1994). Beyond testing: Towards a theory of educational assessment. London: Falmer Press.

Harlen, W. (1994, April). Concepts of quality in student assessment. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.

Harlen, W. (2005). Trusting teachers' judgement: Research evidence of the reliability and validity of teachers' assessment used for summative purposes. Research Papers in Education, 20(3), 245-270.

Ingvarson, L. (1990, April). Enhancing professional skill and accountability in the assessment of student learning. Paper presented at the annual meeting of the American Educational Research Association, Boston, MA.

Klenowski, V. (2007). Evaluation of the effectiveness of the consensus-based standards validation process. Retrieved 14 January 2008, from http://education. qld.gov.au/corporate/newbasics/html/lce_eval.html

Klenowski, V., & Adie, L. (2009). Moderation as judgement practice: Reconciling system level accountability and local level practice. Curriculum Perspectives, 29(1), 10-28.

Klenowski, V., & Wyatt-Smith, C. (2010). Standards, teacher judgement and moderation in contexts of national curriculum and assessment reform. Assessment Matters, 2, 107-131.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge: Cambridge University Press.

Linn, R. L. (1993). Linking results of distinct assessments. Applied Measurement in Education, 6(1), 83-102.

Little, J. W., Gearhart, M., Curry, M., & Kafka, J. (2003). Looking at student work for teacher learning, teacher community, and school reform. Phi Delta Kappan, 85(3), 185.

Malone, L., Long, K., & De Lucchi, L. (2004). All things in moderation. Science and Children, 41(5), 30-34.

Matters, G. (2006, May). Statistical moderation and social moderation around Australia. Paper presented at the International Association for Educational Assessment conference, Singapore.

Maxwell, G. (2002). Moderation of teacher judgements in student assessment: Discussion paper on assessment and reporting. Brisbane, QLD: Queensland School Curriculum Council.

Maxwell, G. (2006, May). Quality management of school-based assessments: Moderation of teacher judgements. Paper presented at the 32nd International Association for Educational Assessment conference, Singapore.

New Zealand Qualifications Authority. (2007). Consultation on the reviewed Good Practice Guidelines for the Moderation of Standards Based Assessment document. Retrieved 23 March 2008, from http://www.nzqa.govt.nz/providers-partners/ assessment-and-moderation/assessment-of-standards/

Ontario Ministry of Education. (2007). Teacher moderation: Collaborative assessment of student work. Retrieved 2 October 2010, from http://www.edu.gov.on.ca/eng/ literacynumeracy/inspire/research/capacityBuilding.html

Queensland Department of Education and the Arts. (2005). Queensland curriculum, assessment and reporting framework. Brisbane, QLD: Strategic Policy and Education Futures.

Queensland Department of Education Training. (2010). Region maps: North Queensland region. Retrieved 27 July 2011, from http://education.qld.gov.au/ schools/maps/

Queensland Department of Education Training and the Arts. (2003). Growing an assessment culture: Overview of the assessment and reporting agenda 2002-2006. Brisbane, QLD: The State of Queensland.

Roberts, L., & Wilson, M. (1998). Evaluating the effects of an integrated assessment system on science teachers' assessment perceptions and practice. BEAR Report Series, University of California, Berkeley, SA-98-1, 1-17. Retrieved 14 January 2008, from http://bearcenter.berkeley.edu/publications/teacher98.pdf

Roberts, L., Wilson, M., & Draney, K. (1997). The SEPUP assessment system: An overview. Berkeley, CA: University of California.

Sadler, D. R. (1987). Specifying and promulgating achievement standards. Oxford Review of Education, 13(2), 191-209.

Scarino, A. (2005). Going beyond criteria for judging performance. Babel, 39(3), 8-16.

Slack, J. D., & Wise, J. M. (2005). Culture + technology: A primer. New York: Peter Lang.

Stobart, G. (2006). The validity of formative assessment. In J. Gardner (Ed.), Assessment and learning (pp.133-146). London: Sage.

Vygotsky, L. S. (1997). The instrumental method in psychology. In R. Rieber & J.

Wollock (Eds.), The collected works of L. S. Vygotsky: Vol. 3: Problems of the theory and history of psychology (pp. 85-90). New York: Plenum.

Vygotsky, L. S. (2003). Mind in society. In M. P. Munger (Ed.), The history of psychology: Fundamental questions (pp. 387-398). New York: Oxford University Press.

Wenger, E. (1998). Communities of practice: Learning, meaning and identity. Cambridge: Cambridge University Press.

Wiliam, D. (1998, July). The validity of teachers' assessments. Paper presented at the 22nd annual conference of the International Group for the Psychology of Mathematics Education, Stellenbosch, South Africa.

Wilson, M. (2004). Assessment, accountability and the classroom: A community of judgement. In M. Wilson (Ed.), Towards coherence between classroom assessment and accountability (pp. 1-19). Chicago: University of Chicago Press.

Wyatt-Smith, C. M. (1999). Reading for assessment: How teachers ascribe meaning and value to student writing. Assessment in Education, 6(2), 195-223.

Wyatt-Smith, C. M., & Bridges, S. (2006, May). Assessment for learning: An Australian study in middle schooling. Paper presented at the International Association for Educational Assessment conference, Singapore.

Zepke, N., Leach, L., Brandon, J., Chapman, J., Neutze, G., Rawlins, P. et al. (2005). Standards-based assessment in the senior secondary school: A research synthesis. Palmerston North: Massey University College of Education.

The author

Lenore Adie is a lecturer with the Queensland University of Technology. Her research focuses on assessment and moderation processes as these contribute to supporting teachers' pedagogical practices and student learning. Lenore has worked as a primary school teacher and in administration positions in state and private schools within Queensland, Australia. Email: lenore.adie@qut.edu.au


Table 1 Summary of data collected

                                                2007   2008   Total

Number of meetings run                           4      7      11

Number of meetings organised that did not run    1      4       5

Number of pre-moderation interviews              9      11     20

Number of post-moderation interviews             5      17     22

Table 2 Summary of teacher demographics (of transcribed meetings)

Variable                    n    Variable                       n

School sector   State       15   Curriculum   English           31

                Private     29                Mathematics       12

                Catholic    6                 Science           7

                Distance    7    Location     Remote            6
                education

School level    Primary     36                Regional centre   38

                Secondary   14                City              6

Year level      Year 4      3    Gender       Male              9

                Year 5      26                Female            41

                Year 6      7

                Year 9      14

Source Citation

Source Citation   

Gale Document Number: GALE|A307413784