If the product works, but what about the people?

This is probably going to be an even more incoherent ramble than normal but I have been trying to write posts around a number of things for the last couple of weeks I’m going to try and merge them.

A couple of weeks ago, I read this post by David Wiley. At the time I tweeted:

I confess to a more than a bit of this sentiment, and not just in relation to OER,   “Much of the OER movement has a bad attitude about platforms.” I am always wary when the focus is on developing platforms and not developing the people who will use these platforms.

I was once in a meeting where I put forward the “people and process not platforms and products” case. I was told that what was being discussed was platform “in the Californian sense of platform”.  . .  I’m sure a classic WTF look must have passed over my face, but it was explained that this meant people as well as technology.  Geography aside, three years later this sense of platform doesn’t seem to be that wide spread or acknowledged. Maybe I need to go to California. But I digress.

Not long before the Wiley post I was reading the Pearson White Paper on learning design.  It caused me a bit of unease too.  Part of me was delighted to see learning design being recognised by, whatever might happen to them, a significant player in the education technology provider field.   Using learning design to help product design is a bit of a no brainer. Technology should be driven by educational need or as Pearson put it :

“Products and systems that effectively leverage learning design can deliver superior learning outcomes.”

One example in the paper referred to work they had done in social science classes

“we quickly recognized that students were easily distracted by conventional textbooks. This told us we needed to eliminate distractions: any extraneous cognitive load that doesn’t promote learning. Fortunately, our learning design work reveals many proven techniques for accomplishing this. REVEL segments all content into manageable pieces and presents it via a consistent structure. It provides strong signaling cues to highlight key material and places all relevant content on screen simultaneously to offer a continuous, uninterrupted experience”

Which kind of related to this point from the Wiley post:

“Our fixation on discovery and assembly also distracts us from other serious platform needs – like platforms for the collaborative development of OER and open assessments (assessments are the lifeblood of this new generation of platforms), where faculty and students can work together to create and update the core materials that support learning in our institutions. Our work in OER will never be truly sustainable until faculty and students jointly own this process, and that can’t happen until a new category of tools emerges that enables and supports this critical work. (Grant money for OER creation won’t last forever.)

And don’t even start trying to explain how the LMS is the answer. Just don’t. “

Well of course Pearson do try to explain that:

“As testing progresses, we can overcome problems that compromise outcomes and build a strong case that our design will support learning. The very same work also helps us tightly define assessments to find out if the product works in real classrooms”

Of course they don’t really touch on the OER aspect (all their learning design stuff has been made available with CC goodness) but I’ll come back to that.

That phrase “if the product works”, I keep coming back to that.  So on the one hand I have to be pleased that Pearson are recognising learning design. I have no argument with their core principles .  I agree with them all.  But I am still left with the niggle around the  assumption that the platform will “do” all the learning design  for both staff and students. That underlying  assumption that if only we had the right platform all would be well, everything could be personalised, through data and analytics and we’d have no retention issues.  That niggles me.

I was part of a plenary panel at the HESPA conference last week called “the future of learner analytics” where a number of these issues came up again.   The questions asked by this group of educational planners really stimulated a lot of debate. On reflection I was maybe a bit of a broken record.  I kept coming back not to platforms but people and more importantly time.  We really need to give our staff and students (but particularly our staff) time to engage with learning analytics.   Alongside the technical infrastructure for learning analytics we need to asking where’s the CPD planning for analytics?  They need to go hand in hand. Cathy Gunn, Jenny McDonald and John Milne’s excellent paper “the missing link for learning from analytics” sums this up perfectly:

there is a pressing need to add professional development and strategies to engage teachers to growing range of learning analytics initiatives If these areas are not addressed, adoption of the quality systems and tools that are currently available or underdevelopment may remain in the domain of the researchers and data analysis experts” 

There seems to be an assumption that personalisation of learning is a “good thing” but is it?  Going back to learning design, designing engaging learning activities is probably more worthwhile and ultimately more useful to students and society than trying to create homogenised, personalised chunked up content and assessments.  Designing to create more effective engagement with assessment and feedback is, imho, always going to be more effective than trying to design the perfect assessment platform.

In terms of assessment, early last week I was also at a Scotbug (our regional Blackboard user group) meeting, where I was in a group where we had to design an assessment system. This is what we came up with – the flipped assessment – aka student generated assessments.

img_0107

Not new, but based on pedagogy and technology that is already in use ( NB there’s been a really great discussion around some of this in the ALT list this weekend).   I don’t think we need any new platforms for this type of approach to assessment and feedback – but we do need to think about learning design (which encapsulates assessment design) more, and give more time for CPD for staff to engage more with the design process and the technologies they either have to,  use or want to use.  This of course all relates to digital capability and capacity building.

So whilst  we’re thinking about next gen platforms, learning environments, please let’s not forget people. Let’s keep pressing for time for staff CPD to allow the culture shifts to happen around understand the value of OER, of sharing, of taking time to engage with learning design and not just having to tweak modules when there’s a bit of down time.

People are the most important part of any  learning environment – next gen, this gen, past gen. But people need time to evolve too, we can’t forget them or try to design out the need for them for successful learning and teaching to take place. Ultimately it’s people that will make the product work.

Badges? Certificates? What counts as succeeding in MOOCs?

Oops, I did it again. I’ve now managed to complete another MOOC. Bringing my completion rate of to a grand total of 3 (the non completion number is quite a bit higher but more on that later). And I now have 6 badges from #oldsmooc and a certificate (or “statement of accomplishment”) from Coursera.

My #oldsmooc badges

My #oldsmooc badges

Screenshot of Coursera record of achievement

Screenshot of Coursera record of achievement


But what do they actually mean? How, if ever, will/can I use these newly gained “achievements”?

Success and how it is measured continues to be one of the “known unknowns” for MOOCs. Debate (hype) on success is heightened by the now recognised and recorded high drop out rates. If “only” 3,000 registered users complete a MOOC then it must be failing, mustn’t it? If you don’t get the certificate/badge/whatever then you have failed. Well in one sense that might be true – if you take completion to equate with success. For a movement that is supposed to be revolutionising the (HE) system, the initial metrics some of the big xMOOCs are measuring and being measured by are pretty traditional. Some of the best known success of recent years have been college “drop outs’, so why not embrace that difference and the flexibility that MOOCs offer learners?

Well possibly because doing really new things and introducing new educational metrics is hard and even harder to sell to venture capitalists, who don’t really understand what is “broken” with education. Even for those who supposedly do understand education e.g. governments find any change to educational metrics (and in particular assessments) really hard to implement. In the UK we have recent examples of this with Michael Gove’s proposed changes to GSCEs and in Scotland the introduction of the Curriculum for Excellence has been a pretty fraught affair over the last five years.

At the recent #unitemooc seminar at Newcastle, Suzanne Hardy told us how “empowered” she felt by not submitting a final digital artefact for assessment. I suspect she was not alone. Suzanne is confident enough in her own ability not to need a certificate to validate her experience of participating in the course. Again I suspect she is not alone. From my own experience I have found it incredibly liberating to be able to sign up for courses at no risk (cost) and then equally have no guilt about dropping out. It would mark a significant sea change if there was widespread recognition that not completing a course didn’t automatically equate with failure.

I’ve spoken to a number of people in recent weeks about their experiences of #oldsmooc and #edcmooc and many of them have in their own words “given up”. But as discussion has gone on it is apparent that they have all gained something from even cursory participation either in terms of their own thinking about possible involvement in running a MOOC like course, or about realising that although MOOCs are free there is still the same time commitment required as with a paid course.

Of course I am very fortunate that I work and mix with a pretty well educated bunch of people, who are in the main part really interested in education, and are all well educated with all the recognised achievements of a traditional education. They are also digital literate and confident enough to navigate through the massive online social element of MOOCs, and they probably don’t need any more validation of their educational worth.

But what about everyone else? How do you start to make sense of the badges, certificates you may or may not collect? How can you control the way that you show these to potential employers/Universities as part of any application? Will they mean anything to those not familiar with MOOCs – which is actually the vast majority of the population. I know there are some developments in California in terms of trying to get some MOOCs accredited into the formal education system – but it’s very early stages.

Again based on my own experience, I was quite strategic in terms of the #edcmooc, I wrote a reflective blog post for each week which I was then able to incorporate into my final artefact. But actually the blog posts were of much more value to me than the final submission or indeed the certificate (tho I do like the spacemen). I have seem an upward trend in my readership, and more importantly I have had lots of comments, and ping backs. I’ve been able to combine the experience with my own practice.

Again I’m very fortunate in being able to do this. In so many ways my blog is my portfolio. Which brings me a very convoluted way to my point in this post. All this MOOC-ery has really started me thinking about e-portfolios. I don’t want to use the default Coursera profile page (partly because it does show the course I have taken and “not received a certificate” for) but more importantly it doesn’t allow me to incorporate other non Coursera courses, or my newly acquired badges. I want to control how I present myself. This relates quite a lot to some of the thoughts I’ve had about using Cloudworks and my own educational data. Ultimately I think what I’ve been alluding to there is also the development of a user controlled e-portfolio.

So I’m off to think a bit more about that for the #lak13 MOOC. Then Lorna Campbell is going to start my MOOC de-programming schedule. I hope to be MOOC free by Christmas.

eAssessment Scotland – focus on feedback

Professor David Boud got this year’s eAssessment Scotland Conference off to a great start with his “new conceptions of feedback and how they might be put into practice” keynote presentation by asking the fundamental question ‘”what is feedback?”

David’s talk centred on what he referred to as the “three generations of feedback”, and was a persuasive call to arms to educators to move from the “single loop ” or “control system” industrial model of feedback to a more open adaptive system where learners play a central and active role.

In this model, the role of feedback changes from being passive to one which helps to develop students allowing them to develop their own judgement, standards and criteria. Capabilities which are key to success outside formal education too. The next stage from this is to create feedback loops which are pedagogically driven and considered from the start of any course design process. Feedback becomes part of the whole learning experience and not just something vaguely related to assessment.

In terms of technology, David did give a familiar warning that we shouldn’t enable digital systems to allow us to do more “bad feedback more efficiently”. There is a growing body of research around developing the types of feedback loops David was referring to. Indeed the current JISC Assessment and Feedback Programme is looking at exactly the issues brought up in the keynote, and is based on the outcomes of previously funded projects such as REAP and PEER. And the presentation from the interACT project I went to immediately after the keynote, gave an excellent overview of how JISC funding is allowing the Centre for Medical Education in Dundee to re-engineering its assessment and feedback systems to “improve self, peer and tutor dialogic feedback”.

During the presentation the team illustrated the changes to their assessment /curriculum design using an assessment time line model developed as part of another JISC funded project, ESCAPE, by Mark Russell and colleagues at the University of Hertfordshire.

Lisa Gray, programme manager for the Assessment and Feedback programme, then gave an overview of the programme including a summary of the baseline synthesis report which gives a really useful summary of the issues the projects (and the rest of the sector ) are facing in terms of changing attitudes, policy and practice in relation to assessment and feedback. These include:
*formal strategy/policy documents lagging behind current development
*educational principles are rarely enshrined in strategy/policylearners are not often actively enaged in developing practice
*assessment and feedback practice doesn’t reflect the reality of working life
*admin staff are often left out of the dialogue
*traditional forms of assessment still dominate
*timeliness of feedback are still an issue.

More information on the programme and JISCs work in the assessment domain is available here.

During the lunch break I was press-ganged/invited to take part in the live edutalk radio show being broadcast during the conference. I was fortunate to be part of a conversation with Colin Maxwell (@camaxwell), lecturer at Carnegie College, where we discussed MOOCs (see Colin’s conference presentation) and feedback. As the discussion progressed we talked about the different levels of feedback in MOOCs. Given the “massive” element of MOOCs how and where does effective feedback and engagement take place? What are the afordances of formal and informal feedback? As I found during my recent experience with the #moocmooc course, social networks (and in particular twitter) can be equally heartening and disheartening.

I’ve also been thinking more about the subsequent twitter analysis Martin has done of the #moocmooc twitter archive. On the one hand, I think these network maps of twitter conversations are fascinating and allow the surfacing of conversations, potential feedback opportunities etc. But, on the other, they only surface the loudest participants – who are probably the most engaged, self directed etc. What about the quiet participants, the lost souls, the ones most likely to drop out? In a massive course, does anyone really care?

Recent reports of plagiarism, and failed attempts at peer assessment in some MOOCs have added to the debate about the effectiveness of MOOCs. But going back to David Boud’s keynote, isn’t this because some courses are taking his feedback mark 1, industrial model, and trying to pass it off as feedback mark 2 without actually explaining and engaging with students from the start of the course, and really thinking through the actual implications of thousands of globally distributed students marking each others work?

All in all it was a very though provoking day, with two other excellent keynotes from Russell Stannard sharing his experiences of using screen capture to provide feedback, and Cristina Costa on her experiences of network feedback and feeding forward. You can catch up on all the presentations and join in the online conference which is running for the rest of this week at the conference website.

Enhancing engagement, feedback and performance webinar

The latest webinar from the JISC Assessment and Feedback programme will take place on 23 July (1-2pm) and will feature the SGC4L (Student Generated Content for Learning) project. Showcasing the Peerwise online environment the project team will illustrate to participants how it can be used by students to generate their own original assessment content in the form of multiple choice questions. The team will discuss their recent experiences using the system to support teaching on courses at the University of Edinburgh and the findings of the project. The webinar will include an interactive session offering participants the opportunity get first hand experience of interacting with others via a PeerWise course set up for the session.

For further details and links to register for this free webinar are available by following this link.

Binding explained . . . in a little over 140 characters

Finding common understandings is a perennial issue for those of us working in educational technology and lack of understanding between techies and non techies is something we all struggle with. My telling some of the developers I used to work with the difference between formative and summative assessments became something of an almost daily running joke. Of course it works the other way round too and yesterday I was taken back to the days when I first came into contact with the standards world and its terminology, and in particular ‘bindings’.

I admit that for a while I really didn’t have a scoobie about bindings, what they were, what the did etc. Best practice documentation I could get my head around, and I would generally “get” an information model – but bindings, well that’s serious techie stuff and I will admit to nodding a lot whilst conversations took place around me about these mysterious “bindings”. However I did eventually get my head around them and what their purpose is.

Yesterday I took part in a catch up call with the Traffic project at MMU (part of the current JISC Assessment and Feedback programme). Part of the call involved the team giving an update on the system integrations they are developing, particularly around passing marks between their student record system and their VLE, and the development of bindings between systems came up. After the call I noticed this exchange on twitter between team members Rachel Forsyth and Mark Stubbs.

I just felt this was worth sharing as it might help others get a better understanding of another piece of technical jargon in context.

Couple of updates from the JISC Assessment and Feedback Programme

As conference season is upon us projects from the current JISC Assessment and Feedback programme are busy presenting their work up and down the country. Ros Smith has written an excellent summary post “Assessment and Feedback: Where are we now?” from the recent International Blended Learning Conference where five projects presented.

Next week sees the CCA Conference and there is a pre-conference workshop where some of the assessment related standards work being funded by JISC will be shared. The workshop will include introductions to:

• A user-friendly editor called Uniqurate, which produces questions conforming to the Question and Test Interoperability specification, QTIv2.1,

• A way of connecting popular VLEs to assessment delivery applications which display QTIv2.1 questions and tests – this connector itself conforms to the Learning Tools Interoperability specification, LTI,

• A simple renderer, which can deliver basic QTIv2.1 questions and tests,

• An updated version of the comprehensive renderer, which can deliver QTIv2.1 questions and tests and also has the capability to handle mathematical expressions.

There will also be demonstrations of the features of the QTI Support site, to help users to get started with QTI.

The workshop will also provide an opportunity to discuss participants’ assessment needs and to look at the ways these might be addressed using the applications we have available and potential developments which could be part of future projects.

If you are interested in attending the conference, email Sue Milne sue.milne@e-learning-services.org.uk with your details as soon as possible.

Design Studio update: Transforming Assessment and Feedback

Those of you who regularly read this blog, will (hopefully) have noticed lots of mentions and links to the Design Studio. Originally built as a place to share outputs from the JISC Curriculum Design and Delivery Programmes, it is now being extended to include ouputs from a number of other JISC funded programmes.

The Transforming Assessment and Feedback area of the Design Studio now has a series of pages which form a hub for existing and emergent work on assessment and feedback of significant interest. Under a series of themes, you can explore what this community currently know about enhancing assessment and feedback practice with technology, find links to resources and keep up to date with outputs from the Assessment and Feedback and other current JISC programmes.

Assessment and Feedback themes/issues wordle

This is a dynamic set of resources that will be updated as the programme progresses. Follow this link to explore more.