Summary of #GCUGamesOn Evalution Findings

As promised this post shares the summary findings from our recent online event, GCU Games On. As I’ve written about before we developed this very quickly (in a month from idea to online) so we were very aware of some of the pedagogic shortcomings of our overall design. However given the rapid development time during the start of summer holidays when most of our subject experts were on holiday we had to make some very pragmatic design decisions.

Overall the feedback was pretty positive and the whole experience is helping to shape our developing strategy to open, online courses. (Nb the text below has been adapted from an internal report).

Background

GCU Games On was an open online event designed to celebrate, explore and share experiences during the Glasgow 2014 Commonwealth Games. It ran between 16 July and 8 August 2014.

Instigated by the PVC Learning and Student Experience, it was developed in little over a month. Due to the time constraints (one month from idea to being available openly online) a simple design was developed which included: background and contextual information with relevant links, making a wish on our digital wishing trees, at least one twitter based activity and a medal quiz challenge each week. Sharing experiences of Glasgow 2014 via twitter was encouraged each week. Daily email updates were sent to all registered participants.

The event was delivered via the new Blackboard Open Education platform.

Participation

  • Registrations: 211
  • Countries: 12 excluding the UK
  • Digital Badges issued: 174
  • Tweets: 424
  • Digital wishes: 107

Evaluation

Of the 211 registrations, 22 completed the survey giving a 10.4% response rate. In addition, due to the use of social media (and in particular, twitter) a number of informal responses to the event were shared.

Summary Findings

The majority of respondents to the survey were female, aged between 25 to 65, based in the UK with no connection to GCU. The majority of participants were based in the UK, with 36% based in both Glasgow and Scotland respectively. 18% of respondents were from the rest of the UK, and there were equal numbers (4.5%) of respondents from other Commonwealth countries and non Commonwealth countries. From registration information we know we had registrations from Australia, India, Trinidad & Tobago, Ireland, Israel, Denmark, Canada Italy, Israel, New Zealand, Spain and South Korea.

59% of respondents had no connection with GCU and 45% of respondents cited wanting to experience online learning at GCU as their main reason for participating. The vast majority of respondents had some form of formal educational qualification, 45% up to Masters level.  This correlates to general trends in open online courses, but may also reflect a network effect from the Blended Learning Team’s network and promotion of the event. 95% of respondents found the site easy or partially easy to use.  54% of respondents completed all of the activities.

Open feedback was generally positive about the experience.

“I really enjoyed this as a bit of fun.  What I got out of it most was seeing new blackboard system in operation and it looks and feels very impressive.”

“I think looking at the Twitter feed this was spot on for what it was trying to achieve. Much fun was had by all it seems and the course gave a great scaffold to talk about their experiences at the games.”

“I do know it is hard to pull together a learning experience around an event like this and I guess that was weakness of this approach.  At times I think really perhaps due to lack of substance or clear learning outcomes – the learning design was a bit hit or miss – but I think you did achieve outcome of getting folks to engage with learning platform which was I think what it was about rather than the content”

 

GCU Games On Gold Medal

GCU Games On Gold Medal

A peak underneath the Swan like MOOC #moocscoted

Picture of a swan

(image: http://www.desktop-nature-wallpaper.com/birds/swan.html)

About a year ago I wrote a post called Preparing for the Second Wave after attending and presenting at an internal staff development event at Newcastle University.  At that time Newcastle hadn’t committed to MOOCs and was grappling with issues of being part of the second wave of MOOC activity. After the event I commented:

“I suspect that for a number of the UK institutions in the first wave of MOOC activity, the reputational benefits are the key driver. Many of them can afford to underwrite the costs of developing and running the courses in the short term without having to think too much about the longer term benefits/costs . . .Maybe it wouldn’t be a bad thing for those institutions not involved with MOOCS just now, to take a step back to consider the most beneficial aspect of MOOCs for their aims and objectives before trying to become part of the second wave.”

A year later and Newcastle is firmly part of that second wave along with a number of other UK institutions as part of FutureLearn. Now I now all the ed tech hipsters are “so-o over MOOCs” but the questions around the long term costs and benefits MOOCs have still to be answered. For an institution like mine who hasn’t been part of the first, second or third wave of MOOC activity, we are still very interested to see what we can learn from others to help us develop our own strategies which may or may not involve an element of MOOC-yness.

Yesterday the Jisc RSC Scotland and the University of Strathclyde hosted an event on MOOCs in Scottish Education. Teams from the Universities of Glasgow and Strathclyde shared their experiences to-date with FutureLearn. And the team who just keep giving from the University of Edinburgh shared their experiences from their ever increasing experience and research of MOOCs on a variety of platforms including FutureLearn.

If you want anyone to convince you of the positive benefits of MOOCs, then look no further than Professor Niamh NicDaeid, University of Strathclyde. Hearing her speak about the the murder mystery themed introduction to forensic science course almost made even my MOOC weary self consider signing up if they run it again.  What a joy to hear someone continually emphasise the importance of fun in learning.

Niamh also talked about the experience of actually running the MOOC and the amount of work behind the scenes to keep its swan like appearance for the learners.  As anyone who has done any kind of online delivery will know once something is live and running there is a huge amount of world that needs to be done behind the scenes. With discussion boards getting around over 6000 posts a week the effect is multiplied beyond most peoples experience.

Staff time for both development of courses and running MOOCs is crucial.  We heard yesterday that Glasgow is committing £2.5 million to developing online learning, we know Edinburgh has a pot double that size, and although Strathclyde didn’t quote any figures it has obviously made a substantial commitment. Again for institutions like mine who maybe haven’t got such deep pockets, there are some fundamental investment questions that need to be addressed about where, what and how to invest and future developments. We aren’t in the MOOC club, unlike Glasgow we weren’t invited to the FutureLearn party, unlike Strathclyde we weren’t gallus enough to “chap on the door” and ask to be let in. And now do we even want to be in the club?  Maybe we are better off doing something in a different way.

FutureLearn (like many MOOCs, and courses) is pretty content driven, and there was lots in the presentations from both the Glasgow and Strathclyde teams about the development process. My colleague Linda Creanor and I did notice get a bit of a “them and us” division creeping in between academic staff and the learning technologists who seemed to be just doing the ‘techie’ stuff. I hope that this is just an impression and not the truth. Certainly the strengths of the skills of all members of teams was emphasised but there was just a bit of a niggle of LTs being put to the bottom of the pile.   I raise this issue in the hope that I will be shot down with evidence to the contrary.

In terms of institutional drivers and evaluation it still seems to be reputation,  staff development and wider engagement with online learning for campus based activities that are key.

Presentations from the day will be made available via the RSC website, and I just want to thank everyone involved for providing a very informative session.

Collaborative auto-ethnography – an antidote to big data in MOOCs?

Firstly, thanks to @helencrump for the title of this post. The alternative title was ‘#oldsmooc – the MOOC that keeps giving”. But I think Helen’s one sounds much more impressive 🙂

I’m not quite sure where I’m going with this post but there was a flurry of activity on my twitter stream last night/this morning and I just wanted to try and capture some of the ideas that have been floated.

Regular readers of this blog will know that a group of #oldsmooc-ers recently presented a paper at the EMOOCs Conference, called “Signals of success and self directed learningwhere we took a collaborative auto-ethnographical approach to create a range of narratives which described our different measures of success in participating in the mooc.  We felt that this personal reflection would help to address some of the gaps in understanding what actually motivates learners in MOOCs and probably more importantly for us as a group to explore the extent “connection, self-efficacy and self-directed strategies facilitate learning in a MOOC”.   As a self selecting group we were very conscious that we could not ensure our narratives were typical of other leaners on the MOOC, but we did try and collect some more data from other participants.  I have tried to document the story of our collaboration, which has been a key  signal of success for us all.

Getting the paper accepted was a great moment for us all, as was the conference presentation and tweets it generated.  We were still unsure if our approach and methodology really had any traction.  But last night Paige altered us all to this bit of activity on the current #rhizo14 MOOC.  Again through connections and network (aka some of our group taking part in another mooc) it seems our ideas and approach are being explored by other learners.

I think this is really important.  We know that existing accepted educational metrics don’t really apply in the MOOC context, particularly for retention. Despite the promise of MOOCs and big data being able to give us insights into how people learn, I, like others, am still not so sure about some of the methods being used and in turn the patterns that are emerging. As well as the quantitative data, we need to get much more qualitative data exploring as many different narratives as possible from learners. It’s only by doing that that can we really start to help develop our understanding of how people define success in MOOCs. And in turn, we can ask more challenging questions of/from the quantitative data.

Being a bear of very little brain, I like seeing the pretty patterns and swirly diagrams, but find it confusing when they don’t seem to relate to my own experiences.  Mind you,  if this article in the Sunday Observer is to be believed we won’t need to grapple with big data -v – little data -v- educational theory for much longer as soon the google robots will have worked it out all out for us and will have “fixed” education.

My experience of learning on MOOCs has been very different from my traditional educational experiences. I know I didn’t (and still don’t really) enjoy formal education, and I am much happier (and hopefully more creative) in connected, loosely structured learning experiences than read a bit, do the test, read a bit more ones.

Anyway below is a collation of the tweets from last night this morning, which range from us being all “check us out with starting an auto-ethnographic revolution” to more serious questions about the nature of open collaborative spaces, self disclosure and the importance of failure. On the last point, Pat Parslow referenced the “confessional” booth at the PELeCon Conference, which again got me thinking about the use of the language of guilt around what are perceived to be non traditional ways of doing things. But that’s probably a post for another day.

Deconstructing my (dis)engagement with MOOCs part 2

Following from my early post, I’ve attempted to use the classifiers outlined in the #lak13 paper on disengagement in MOOCs, in the context of my experiences. Obviously I’ve modified things a bit as what I’m doing is more of a self reflection of my personal context -so I’ve made the labels past tense. I’m also doing a presentation next week at the University of Southampton on the learner perspective of MOOCs and thought that these classifications would be a good way to talk about my experiences.

Firstly here are the MOOCs I’ve signed up for over ( the ? years are when I was aware but not active in MOOCs)

MOOCs I've took!

MOOCs I've took!

Now with the course engagement labels

My MOOC engagement with labels

My MOOC engagement with labels

And finally aligned to trajectory labels

My MOOC participation using trajectory labels

My MOOC participation using trajectory labels

A big caveat, not completing, disengaging and dropping out does not mean I didn’t learn from each he experience and context of each course.

More to come next week including the full presentation.

Deconstructing my own (dis)engagement with MOOCs

No educational technology conference at the moment is complete without a bit of MOOC-ery and #lak13 was no exception. However the “Deconstructing disengagement: analyzing learner sub-populations in massive open online courses” paper was a move on from the familiar territory of broad, brush stroke big numbers towards a more nuanced view of some of the emerging patterns of learners across three Stanford based Coursera courses.

The authors have created:

” a simple, scalable, and informative classification method that identifies a small number of longitudinal engagement trajectories in MOOCs. Learners are classified based on their patterns of interaction with video lectures and assessments, the primary features of most MOOCs to date . . .”

” . . .the classifier consistently identifies four prototypical trajectories of engagement.”

As I listened to the authors present the paper I couldn’t help but reflect on my own recent MOOC experience. Their classifier labels (auditing, completing, sampling, disengaging) made a lot of sense to me. At times I have been in all four “states” of auditing, completing, disengaging and sampling.

The study investigated typical Coursera courses which mainly take the talking head video, quiz, discussion forum, final assignment format and suggested that use of the framework to identify sub-populations of learners would allow more customisation of courses and (hopefully) more engagement and I guess ultimately completion.

I did find it interesting that they identified that completing learners were most active on forums, something that contradicts my (limited) experience. I’ve signed up for a number of the science-y type Coursera courses and have sampled and disengaged. Compare that to the recent #edcmooc which again was run through Coursera but didn’t use the talking head-quiz-forum design. Although I didn’t really engage with the discussion forums (I tried but they just “don’t do it for me”) I did feel very engaged with the content, the activities, my peers and I completed the course.

I’ve spoken to a number of fellow MOOC-ers recently and they’re not that keen on the discussion forums either. Of course, it’s highly likely that people I speak to are like me and probably interact more on their blogs and twitter than in discussion forums. Maybe its an arts/science thing ? Shorter discussions? I don’t really know, but at scale I find any discussion forum challenging, time consuming and to be completely honest a bit of a waste of time.

The other finding to emerge from the study was that completing and auditing (those that just watch the videos and don’t necessarily contribute to forums or submit assignments) sub-populations have the best experiences of the courses. Again drawing on my own experiences, I can see why this could be the case. Despite dropping out of courses, the videos I’ve watched have all been “good” in the sense that they were of a high technical quality, and the content was very clear. So I’ve watched and thought “oh, I didn’t know that/ oh, so that’s what that means? oh that’s what I need to do”. The latter being the point that I usual disengage as there is something far more pressing I need to do 🙂 But I have to say that the experience of actually completing (I’m now at 3 for that) MOOCs was far richer. Partly that was down to the interaction with my peers on each occasion, and the cMOOC ethos of each course design.

That said, I do think the auditing, completing, disengaging, sampling labels are a very useful addition to the discourse and understanding of what is actually going on within the differing populations of learners in MOOCs.

A more detailed article on the research is available here.

Badges? Certificates? What counts as succeeding in MOOCs?

Oops, I did it again. I’ve now managed to complete another MOOC. Bringing my completion rate of to a grand total of 3 (the non completion number is quite a bit higher but more on that later). And I now have 6 badges from #oldsmooc and a certificate (or “statement of accomplishment”) from Coursera.

My #oldsmooc badges

My #oldsmooc badges

Screenshot of Coursera record of achievement

Screenshot of Coursera record of achievement


But what do they actually mean? How, if ever, will/can I use these newly gained “achievements”?

Success and how it is measured continues to be one of the “known unknowns” for MOOCs. Debate (hype) on success is heightened by the now recognised and recorded high drop out rates. If “only” 3,000 registered users complete a MOOC then it must be failing, mustn’t it? If you don’t get the certificate/badge/whatever then you have failed. Well in one sense that might be true – if you take completion to equate with success. For a movement that is supposed to be revolutionising the (HE) system, the initial metrics some of the big xMOOCs are measuring and being measured by are pretty traditional. Some of the best known success of recent years have been college “drop outs’, so why not embrace that difference and the flexibility that MOOCs offer learners?

Well possibly because doing really new things and introducing new educational metrics is hard and even harder to sell to venture capitalists, who don’t really understand what is “broken” with education. Even for those who supposedly do understand education e.g. governments find any change to educational metrics (and in particular assessments) really hard to implement. In the UK we have recent examples of this with Michael Gove’s proposed changes to GSCEs and in Scotland the introduction of the Curriculum for Excellence has been a pretty fraught affair over the last five years.

At the recent #unitemooc seminar at Newcastle, Suzanne Hardy told us how “empowered” she felt by not submitting a final digital artefact for assessment. I suspect she was not alone. Suzanne is confident enough in her own ability not to need a certificate to validate her experience of participating in the course. Again I suspect she is not alone. From my own experience I have found it incredibly liberating to be able to sign up for courses at no risk (cost) and then equally have no guilt about dropping out. It would mark a significant sea change if there was widespread recognition that not completing a course didn’t automatically equate with failure.

I’ve spoken to a number of people in recent weeks about their experiences of #oldsmooc and #edcmooc and many of them have in their own words “given up”. But as discussion has gone on it is apparent that they have all gained something from even cursory participation either in terms of their own thinking about possible involvement in running a MOOC like course, or about realising that although MOOCs are free there is still the same time commitment required as with a paid course.

Of course I am very fortunate that I work and mix with a pretty well educated bunch of people, who are in the main part really interested in education, and are all well educated with all the recognised achievements of a traditional education. They are also digital literate and confident enough to navigate through the massive online social element of MOOCs, and they probably don’t need any more validation of their educational worth.

But what about everyone else? How do you start to make sense of the badges, certificates you may or may not collect? How can you control the way that you show these to potential employers/Universities as part of any application? Will they mean anything to those not familiar with MOOCs – which is actually the vast majority of the population. I know there are some developments in California in terms of trying to get some MOOCs accredited into the formal education system – but it’s very early stages.

Again based on my own experience, I was quite strategic in terms of the #edcmooc, I wrote a reflective blog post for each week which I was then able to incorporate into my final artefact. But actually the blog posts were of much more value to me than the final submission or indeed the certificate (tho I do like the spacemen). I have seem an upward trend in my readership, and more importantly I have had lots of comments, and ping backs. I’ve been able to combine the experience with my own practice.

Again I’m very fortunate in being able to do this. In so many ways my blog is my portfolio. Which brings me a very convoluted way to my point in this post. All this MOOC-ery has really started me thinking about e-portfolios. I don’t want to use the default Coursera profile page (partly because it does show the course I have taken and “not received a certificate” for) but more importantly it doesn’t allow me to incorporate other non Coursera courses, or my newly acquired badges. I want to control how I present myself. This relates quite a lot to some of the thoughts I’ve had about using Cloudworks and my own educational data. Ultimately I think what I’ve been alluding to there is also the development of a user controlled e-portfolio.

So I’m off to think a bit more about that for the #lak13 MOOC. Then Lorna Campbell is going to start my MOOC de-programming schedule. I hope to be MOOC free by Christmas.

Preparing for the second wave

Last Friday I was delighted have been invited to the “what are MOOCs?” staff development seminar at Newcastle University

I started the day with a presentation around the the history, pedagogy, myths and media of MOOCs, followed by Sian Bayne who gave a very open presentation about the experiences at Edinburgh and in particular of the #edcmooc. Suzanne Hardy (based at Newcastle) then reflected on her experience as a student on the #edcmooc, and also raised some very pertinent points for fellow staff members on the potential opportunities and pitfalls of developing MOOCs as part of institutional provision.

Suzanne’s storify provides an excellent summary of the day which I won’t try to replicate, and there will be an links to all the presentations as well as more commentary on the UNITE blog very soon.

It was, as ever, really useful to hear the thoughts of “normal” staff members. By that I mean your average lecturer/support person who doesn’t know much about MOOCs, hasn’t been a student on one and who has only heard bits and pieces about the whole phenomenon and isn’t part of the edtech twitterati. Newcastle, unlike Edinburgh, but like many Universities not just in the UK but around the world, hasn’t been part of the “first wave” of activity. So what are the institutional benefits to becoming involved now that the initial splash is over? Is it a case of just having to be seen to do “something” to keep up with your peer institutions? Or can you afford to take some more time to see how things play out? As Sian emphasised throughout the day, there is an awful lot of research that needs to be done to show the actually effectiveness (or not) of MOOCs. (This recently published survey of teachers experiences although mainly US based is a step in that direction) .

As Patrick McAndrew pointed out during his keynote at #cetis13 perhaps what we really need to think about is less of the “m” and more of the “o”. In other words concentrate on developing and sharing open practice and resources and in turn open courses/content which meet specific institutional aims. As we all know there are many variations of open. And again Patrick as pointed out, by using one of the big MOOC providers you could be putting at least one more barrier in front of your “open” course.

I suspect that for a number of the UK institutions in the first wave of MOOC activity, the reputational benefits are the key driver. Many of them can afford to underwrite the costs of developing and running the courses in the short term without having to think too much about the longer term benefits/costs or indeed any potential lock downs/change of service agreements from platform providers.

Maybe it wouldn’t be a bad thing for those institutions not involved with MOOCS just now, to take a step back to consider the most beneficial aspect of MOOCs for their aims and objectives before trying to become part of the second wave. And in the meantime, like this well know VC, encourage more insight and reflection for both staff and students with a try before you buy (or sell!) attitude.

My presentation (with thanks to #ds106 participants for many of the images)