Gareth Jones

After upgrading to Flash player 11.1 this week (11.1.102.55 to be exact) I noticed that a number of my games and applications were no longer displaying correctly. The screen would either be blank on startup or after a couple of button presses, and after that point right-clicking on the stage would present me with a “movie not loaded” message and not much else.

At first I thought that Adobe had simply dropped the ball again. It wouldn’t be the first time, let’s be honest.

I considered reverting back to a previous player version but of course that wouldn’t solve the problem for everyone else who is using my applications so I decided instead to investigate.

One of the applications that was affected was a personalised video messenger I’ve just finished for a major UK greetings card company. The strange thing was that the application would work on some screens when it was in normal mode, but not at all when it was in debug mode. The only difference between debug mode and normal mode was that for the benefit of the non-technical staff at the card company the application would output its traces to a text field at the bottom of the screen. No other logic was different so naturally my first thought was that it was related to dynamic text fields or maybe an embedded font issue.

After some messing around it turned out that it wasn’t the text field or the embedded font at all. In fact in turns out that Flash player 11.1 is happy with all of that stuff: until you run the SWF through SWF Protector 3. At that point the SWF again either refuses to work at all or refuses to work on screens that use dynamic text fields. For every Flash player version prior to 11.1 the SWF works fine so I can only assume that either there is a bug in 11.1 that SWF Protector 3 exposes or Adobe has deliberately changed something that happens to trip up this particular obfuscator.

So, I’ve had to consign SWF Protector 3 to the bin and am now using Kindi SecureSWF which, although technically a more secure product isn’t as user-friendly as there is no batch import of SWFs and no option to have the protected file over-write the original. I’m actually hoping the next Flash player resumes its compatibility with SWF Protector 3, but with SWF Protector 4 due out for PC any minute now there is also hope that DCOMsoft can resolve the problem themselves.

Today marks two weeks since Learnalot‘s launch and during that time almost 40 UK schools have signed up for trials. While 40 schools only represents a fraction of the total number of schools in the country, it is already interesting to read the wide range of feedback that we’re getting from teachers who are using our software.

Most of the feedback we’re receiving is incredibly positive. The quality of the software, the range of scenarios, the graphics, the sound, the supporting documents – everything is going down well. There is a discrepancy in one area of feedback however, and that is the difficulty of the questions.

While most schools seem to feel that we’ve pitched the questions at just the right level, some have asked us to make the resources harder and one has even asked us to make it possible to disable the hints function and force the students to work it all out for themselves without any help whatsoever. Brutal!

On the other hand, we’ve also received feedback that the questions in what is actually our easiest resource – Britain’s Got Power – are too difficult and that their students are struggling with it.

While I have no intention of naming individual schools, teachers or students, the pattern in the feedback is certainly worth discussing and exploring in more detail so that we can better understand exactly why this discrepancy exists and whether there is anything that we, as an educational software provider, should do about it.

Although the data and questions within the activities are randomised, they’re always within sensible parameters and as such the range of difficulty for each student can, I think, be considered negligible and therefore be rounded up to a constant. So if the difficulty of the software is a constant, why are we being asked by some teachers to make the software easier and to make it harder by others?

There could be a variance in the abilities of students, of course. While some students excel in maths, others excel in other areas and can find maths problematic. But such variance would be extraordinary on a school level and is much more likely to exist on an individual level with every school in the land being home to students on either end of the spectrum.

Could it be student age – and by loose extension, ability and/or experience? Taking extreme cases, one school that thinks we have it just right has students a full year younger than another school that thinks we need to make the resources easier, and although this is an extreme in one direction, there are no extremes in the opposite direction as would be expected so we can probably discount student age as a plausible factor as well.

What about student sex? Whilst the school that most strongly believes that our resources are too difficult is a girls’ school, other girls’ schools have not said anything at all about the resources being too difficult. Similarly, the schools who want us to make the resources more difficult are made up of both boys and girls in fairly equal measure, and still more schools that are made up of mostly boys have said that they feel the difficulty is spot-on as it is.

Just to remove all doubt that student sex is not a factor, in a recent study published in Science, Professor Janet Hyde used data from around 7 million US children in 10 US states from grade 2 through to grade 11 and showed that the difference in performance between the two sexes was negligible and that while boys were marginally ahead in some states, girls were marginally ahead in others.

Our own results, though obviously from a much smaller pool, would seem to corroborate this finding as several of the top spots on our leaderboards are occupied by girls. The gender ratio of the schools must therefore be irrelevant when trying to work out the reason for the discrepancy in requests to alter the difficulty.

So again, why are some schools asking us to make the resources easier while others are asking us to make them harder if we can assume that each school on average has students of approximately equal ability?

From what I’ve seen so far, the answer is the teachers themselves.

One school told me that although their students are now doing very well with the resources, this wasn’t always the case. At the beginning the students were struggling and thought the resources were too difficult, but she said this was because they were used to getting their answers on a plate. They weren’t used to having to think about the best way to solve a problem using first principles. They weren’t used to not being given the answers if they didn’t get the question right the first time round. She stayed the course however and since then the students have accepted that they need to do the work themselves, they have embraced the challenge and have excelled in every resource. Those students are now dominating several of the leaderboards and are eagerly awaiting the next resource so that they can continue to flex their mathematical muscles.

On the other side of the scale, another teacher whose students initially struggled with the resources said that they were using another service instead which provided simpler activities, and he reported some of the more difficult questions in our resources as “bugs” that needed fixing.

Intrigued by the difference in approach between these two teachers, I recalled back to when I was at school and what my teacher used to do with us in order to have something to compare against. Indeed, my teacher didn’t use any educational software at all because back then our school had only just migrated from blackboard and chalk to whiteboard and marker pen. The “ICT room” consisted of around 5 computers and was only ever used by students on the “computing” course. In any case, a persistent theme of my maths lessons were that we were told what we had to learn and we practised it until we knew it. There was no such thing as choosing an alternative task that we were happier with because it was less challenging.

Looking again at the contrasting approaches on display here, after discounting every other variable it seems that the teachers themselves are the key factor in why we’re receiving such mixed feedback on the difficulty of the resources. Some teachers are pressing ahead and weaning their students off resources that almost complete themselves and are achieving incredible results in the process, whereas others are seemingly happy to let their students continue with a false sense of achievement with software that doesn’t offer any challenge at all.

While it’s clear that the teaching of maths in the UK is not consistent even from a sample as small as ours (it will be interesting to see what happens when we start looking at hundreds of schools), I think it’s clear which method leads to better performance in students.

So where does this leave Learnalot?

Well, we aren’t going to start developing software that almost completes itself because that’s a pointless exercise.

What we will do instead is continue to challenge and engage our learners and push them to the limits of their abilities because this is how students learn new skills and improve existing ones. We will broaden the range of difficulty exhibited in the resources ever so slightly to make them more inclusive to those who aren’t as strong in maths and more challenging to those who are gifted, and we will continue to do so with scenarios that students find interesting and engaging.

This, I think, is the way to bring out the best in young learners and by building a suite of resources that covers the entire syllabus in this way, I don’t see why we won’t have a significant effect on these learners’ maths skills.

I got a phone call this morning from Joanne Bailey from Public Service Review. She told me about their publications and how the latest edition would be focussed on Michael Gove, the current UK education secretary, and how he wants to improve attainment levels in schools for maths and science.

Apparently, Learnalot had been flagged up in one of their meetings as a company that might be able to help Michael achieve this goal. How flattering. She explained how they would like to feature an article on Learnalot that would follow Michael’s piece and asked whether I’d be interested in that – to which I naturally replied yes. She then told me that “of course” this would come at a cost – £2,396 plus VAT in fact – and that the article would be written by us and not them.

So, I was basically being asked to pay for advertising space in their magazine, which made it expensive irrespective of their subscriber list, but as the magazine seemed like a high-quality publication and the claims she made of their subscriber list were impressive, I still wanted to know more about them before turning them down. I said I was interested but that I would need to check a few things first, and she said that they would send over the agreement. Slightly concerning was that she then asked that I sign the agreement immediately and return it to them before she went for another meeting – which was in 5 minutes. I wasn’t about to sign over nearly 3 thousand pounds in 5 minutes, so I said I’d return the agreement after checking that everything was alright first and not before.

Joanne took from this suspicion that I wanted to see some examples of previous editions first to reassure me of the quality of the magazine. This seemed a little pointless really since by her own account large chunks of it had been written by third parties, and in any case it wasn’t the quality of the magazine that had my alarm bells ringing anyway.

Still concerned by the request to agree within 5 minutes, I got in touch with Laura at Hamilton House, our PR company, and asked them what she knew about PSR. In the meantime, I received a copy of the proposal from them with a note along the top asking me to email the signed copy “straight away”.

In the meantime I Googled the company and found this page, a blog from a guy named Andrew Jaffe and a load of comments from people who had all experienced the same thing. It turns out that this stories about short deadlines and meetings is something that PSR like to use a lot! I forwarded the link to Laura who then called me and kindly suggested getting in touch with PSR on our behalf to find out more about them and their subscriber list. She too was concerned about this “straight away” business and told me that she had never heard of a publication asking for such immediate commitment before. She also thought the piece was expensive – even for a glossy magazine.

Laura got back to me within minutes and said that Joanne had given her some information, but not the specific information that she had requested: namely the all-important breakdown of readership. She suspected therefore that the readership was not at all made up of the maths teachers that I was hoping to reach, and without a definitive printed statement of the readership it was difficult to consider otherwise. Laura also said that she had found Joanne somewhat difficult to deal with.

As soon as I got off the phone with Laura it rang again. I answered and it was Joanne. Joanne claimed that Laura had been very rude to her, which I really didn’t believe. She also tried to nullify Hamilton House’s concerns over her worrying need for immediate agreement and their expensive pricing by suggesting that she didn’t like their website.

Attacking the website and approach of a company one has chosen to work with is a rather odd approach, and overall I was unable to work out how that was relevant. The remarks about Hamilton House’s website were also made despite Joanne’s claim at the beginning of the conversation that her internet access was down because of a virus. In all, this did nothing but confirm beyond doubt that I needed to give Public Service Review a wide berth.

Update: Here’s another link that contains some useful information on Public Service / PSCA.

Read more…