SCE was one of a group of foundations that supported a new National Research Council report called: “Education for Life and Work: Developing Transferable Knowledge and Skills in the 21^st Century.” The report was a rigorous, research-guided attempt to clear up what we mean by terms like “21^st century skills” and “deeper learning”—with an eye toward actionable ideas that educators and others can employ to help children learn.

You can read the full report here and a brief summary here.

We believe this work is an important step forward toward giving children an education that matters. But we want to hear what you think. So we invite you to take a look, read up, and let us know. What’s missing? What did the researchers miss? What else would you want to know? How can we make it more useful to people in the field? Comment here, or reach out to us at info@scefdn.comfdn.org.

Welcome to the fourth in a series on why the digital learning media market is broken. (For an introduction, and links to the rest of the posts, click here.) Today, we’re talking about testing and evaluation—what, in theory, people and organizations would use to find out whether children are actually learning something (or more things) by using digital tools.

Resource gaps: The folks with the money, from foundations to venture capitalists to the federal government, fail to invest enough to properly ascertain whether projects and products actually teach and engage children. The buyers lack the resources to test the quality of digital learning products—reflecting an underutilization of the billions of dollars[i] spent by school districts on computers and technology infrastructure. There’s a lot to be culled from the results of the data systems investments spurred by Race to the Top. But in most states, that’s just not yet happening in a way that’s useful.

Information gaps: There is a paucity of rigorous research evaluating the efficacy of digital learning products.[ii] There is no consensus on which aspects of these tools are worth testing. Several magazines, websites, and industry organizations rate or review digital media, but the rubrics and methods vary drastically, and are seldom based on the learning value of the products. Government agencies with the power to be arbiters of quality are just beginning to think about these problems in a serious way.

Infrastructure gaps: With so many players, the market for ratings/certification of high-quality products is fragmented, with no universally accepted system. Although several organizations offer valuable services to test the learning value of specific products, the costs are often too high for start-up companies (or even established firms), and most are geared toward the traditional K-12 “ed tech” market. The establishment of Common Core Standards will be useful here—once you have a common standard of “what is to be learned,” it’s much easier to ask “does this product help children learn it?” But the Standards leave much to be desired on complex skills like systems thinking, and are extremely weak on important social-emotional skills like resilience, persistence, and empathy.

Misaligned incentives: Because the sector lacks a common quality standard, much less a trustmark like the Good Housekeeping Seal, many purchasing decisions are made based on marketing muscle and sales relationships, not on what works. So businesses have little incentive to pay to rigorously test whether their products are effective for learning.

It’s hard to write about this without mentioning that SCE and our partners are about to launch a major project focused on addressing these issues, but this series is focused on what’s wrong. We’ll get to how to capitalize on what’s right soon enough.

Disagree? Find us on Twitter or reach out via our site. We’re a learning foundation and we believe in iterative knowledge and action. Everything we know (or think we know) we know because people like you have told us.

Welcome to the third in a series on why the digital learning media market is broken. (For an introduction, and links to the rest of the posts, click here.) Today, we’re looking at design and development—the process by which ideas and research about digital learning become actual digital learning objects.

Resource gaps: If you’re reading this, you already know that investment in learning technologies is escalating, reaching well into the hundreds of millions of dollars annually. Unfortunately, investors are relatively risk-averse, and most capital is deployed to conservative, sustaining innovations, such as skill-and-drill software that mimics traditional school activities. Few investors seem interested in technologies that capitalize on the unique qualities of interactive media: personalized software that adapts to the learner; participatory products that encourage children to collaborate and create; highly engaging, immersive games and simulations that encourage and reward persistence and exploration (including entertainment media products not originally intended for educational use). Even fewer spend money on technologies that offer opportunities to engage in and practice Deeper Learning and 21st Century Skills such as problem solving, creativity, persistence, and collaboration. Essentially, most of the folks with money are using 21st century tools to make 19th century education faster and cheaper. A similar pattern of risk-averse behavior exists in both the foundation and public sectors. [i]

Information gaps: As we noted in a previous post, there’s little knowledge transfer between academia and industry. Many new investors, especially venture capital firms, entering the sector lack the experience to know what they don’t know—especially about how children learn. Many are experts in consumer entertainment technology who know only the education system that they (and perhaps their children) lived through. While this isn’t a terrible thing, we’re already seeing the creation of successful, profitable “education” companies whose products do not actually help children learn anything useful. For those who value rigor, there is a dearth of publicly available knowledge about design practices for digital learning products. The few places where this knowledge does exist are too nascent, academic, or insular to provide accessible, actionable information.

Infrastructure gaps: Digital learning start-ups, for-profit or nonprofit, can turn to only a handful of incubators and service providers as they build their products. Most of the conferences where they might network and learn are either too conservative, or not market-oriented enough to be relevant to creating actual products.

Misaligned incentives: America’s assessment system and our education system are both years behind the skills that children must learn to succeed. Consequently, investors and entrepreneurs have an easier time validating the profit potential of (and making money from) less-than-innovative tools. Education companies lack incentives to invest much in high-risk product development, or to seek out and share knowledge about new learning approaches and goals. The same is true of entertainment companies, which bet on massive entertainment profits, not the uncertain potential of learning media.[ii] None of these companies has much of an incentive to design software or services with children from low-income families as the target audience, because there are more lucrative consumers and families with the money and the interest in products that sound like they will help their children gain an educational edge. The most talented entertainment media developers and education curriculum designers face considerable financial risk, and limited chance of upside, if they opt to leave stable jobs to join digital learning start-ups. Amongst digital learning creators, there exists a vague and minimal merit system: While there are several award programs for digital learning media projects, compensation is insufficient to spur consistent and extraordinary innovation.

Disagree? Find us on Twitter or reach out via our site. We’re a learning foundation and we believe in iterative knowledge and action. Everything we know (or think we know), we know because people like you have told us.

Welcome to the second in a series on why the digital learning media market is broken. (For an introduction, and the rest of the posts, click here.)

Resource gaps: Research is actually one of the few somewhat bright spots in the digital learning field. The MacArthur Foundation—along with Hewlett, Gates, and a handful of others—have invested tens of millions of dollars on academic research here during the past few years. Partly because of their advocacy and field-building, the DL hype has reached Washington, leading to hundreds of millions in government funding in the past few years, and major corporations from Disney to Dell are spending millions more. Researchers can also draw on significant progress during the past decade in relevant disciplines like learning sciences, cognitive science, computer science, education, child development, and neuroscience.

However, the field remains nascent, and so does funding and other resources. More than 40 years after The Oregon Trail was developed at Carleton College, there are still barely a dozen full-fledged university departments or centers dedicated to digital learning research,[i] and few tenure-track positions devoted to researching digital media and learning. Spending in education research in general lags that in others sectors, and education technology is no exception. Even tenured professors tell us they’re finding it difficult to attract grants from the government, foundations, or companies.

Information gaps: The relatively young field and small funding pool—along with fast, continuous changes in technology use—create a situation where even experts do not yet fully grasp how, when, whether, and why digital learning works. This is no surprise given the funding deficiencies, but it’s a massive problem that hasn’t been solved, and given the speed of innovation it seems like a tough challenge.

Infrastructure gaps: Some of the most valuable (and most often overlooked) work by MacArthur in this area has been its grantmaking in field-building, working to create the academic infrastructure for digital learning—conferences, online forums and offline communities, and dedicated laboratories. But all of these structures, no matter who has funded them, remain immature.[ii] This means there are too few connections or technology transfer points between academia and industry, with some notable exceptions. Researchers find it difficult to persuade existing businesses[iii] to use discoveries in building products, and seldom have the skills (or support) to realize research findings in either commercial or nonprofit products. And, of course, not every researcher wants to collaborate with industry. The knowledge transfer and partnership difficulties run the other way as well—digital media companies (from Zynga to Disney) collect vast troves of data and knowledge that are inaccessible to the academic community.

Misaligned Incentives: Commercial efforts are competitive and proprietary; hence, important results derived from corporate research are closely held. Tenure-track academic researchers are rewarded more for publishing papers than for transforming pioneering research into real-world application, or sharing theories about research failures. Despite the growth of interdisciplinary research, many academics are still incented to compete with other disciplines rather than collaborate. What’s more, the natural speed of academia is slower than that of industry—the pressures push professors toward accuracy and deliberation. The same is true of the incentives for young researchers to specialize in very niche areas. None of these is necessarily bad, but it does mean there are many, many researchers examining the effects of broadcast television on children, and comparatively few examining, for example, mobile apps.

That’s it for now…up next: Design & Development.

Disagree? Find us on Twitter or reach out via our site. We’re a learning foundation and we believe in iterative knowledge and action. Everything we know (or think we know) we know because people like you have told us.

To celebrate Digital Learning Day, SCE is going to give our sector a little tough love, and kick off a series of blog posts we’re calling “Digital Learning: A Broken Market“…

In a perfect world, kids and adults would have easy access to super-fun, super-effective digital learning media—games, apps, videos, creative suites. Technology would be an important complement to traditional schooling, empowering educators, parents, and kids to personalize and enhance learning. Children would be living in an anytime, anywhere, any-device, always-on blended learning ecosystem. All kids would have the chance to acquire the knowledge and skills they need to fulfill their potential as citizens, workers, and human beings.

We do not live in that world. In fact, it’s almost laughable how far we are from that world. But go to a boosterish conference and watch a digital learning luminary wax on about how technology is going to save the education system, and you could be forgiven for believing that we’re 12 months away from an edtech-powered learning revolution.

And you’d be wrong.

Digital learning is powerful, but it is far from perfect. Unfortunately, it’s in danger of becoming the most overhyped education fad since the small schools movement. So what to do to make sure digital learning fulfills its promise? The first step toward answering that question is to identify what the challenge actually is.

At SCE, we often view digital learning as a marketplace, and we think the root problem is that the marketplace is broken. With the help of a few hundred partners, advisors, friends, grantees, and education gadflies, we have spent the last year and a half examining some of the underlying causes of market dysfunction. We are going to spend the next several weeks blogging about these problems, and SCE will publish the collective as a free digital white paper in the spring. (Future posts will describe the work of SCE and others to repair and cultivate the market.)

As a foundation, our focus is on the public interest—in this case, as many children learning as much as possible. But we believe strongly that markets can serve the public good, and that this marketplace has the potential to serve the interests of children and society without sacrificing long-term profits. So the framework we are using (see graphic below) looks very much like a typical business innovation pipeline. It begins on the left, with ideas for new digital learning technologies, processes, or objects; and it moves toward the right, with learning moments[i] taking place with the support of digital media.

 

 

The specific path from idea to learning by children depends on the type of technology, the creator, and the business or sustainability model. For example, a publisher of mobile games must find distribution and sales channels, as well as revenue streams. A university-based enterprise may require government and foundation subsidies, and would need a platform to extend its reach. In most cases, however, successful technology development and usage appears to follow a general sequential path, with successes and failures at each step sending useful information back and forth along the pipeline.[ii]

In our view, the most challenging deficiencies at each stage of the pipeline fall into four categories:

• Resource gaps: a lack of financial and human capital.
• Information gaps: a lack of data, analysis, or general knowledge.
• Infrastructure gaps: a lack of technology or common marketplace platforms to build on, or field infrastructure to support the sector.
• Misaligned incentives: dynamics that create financial rewards, prestige, power, or accomplishment for behavior that runs counter to the public interest

So that’s how we’re framing the problems and the solutions. The remainder of this series will examine how these deficiencies play out at each stage of the pipeline—starting at the beginning, with digital learning’s research and academic community.

Until then, we welcome your thoughts, criticisms, and questions.

We’ll be collecting all the posts in the Broken Market series here, so feel free to bookmark this page and return in the coming weeks.

• Why Research is Broken
• Why Design & Development are Broken
• Why Testing & Evaluation are Broken

SCE Executive Director Ryan Blitstein participated in a panel on educational standards at the Cooney Leadership Forum. Here are some raw notes of things Ryan was planning to talk about before the panel ran out of time. (Questions in italics from moderator Kevin Clark.)

• What should stand for evidence in media companies’ assertions that products have educational value?

There is a hierarchy of evidence, just like in academic research or a court of law. The basic evidence is anecdote: The average kid can tell in five minutes whether a product is engaging or not; and if it’s not engaging, the educational value is zero. The average adult can tell fairly quickly whether something is terrible or, at least, possibly useful. The levels of rigor go up from there: experts or quasi-experts (like learning sciences graduate students or chemistry teachers) can spend an hour or two with a product and review it; or they can analyze the inputs—who built this, did they use evidence-based design, or is it similar to an existing product that works well? And then there are assessments and efficacy trials, which are strongest when there are no ethical conflicts, large numbers of child users, trustworthy tests, and randomization.

• Who is responsible for helping not only students, but also teachers, afterschool providers, and parents, with media literacy skill development?

We are all responsible for helping students. The digital divide is not just an infrastructure issue; it is a knowledge and skills issue. Children who don’t become digital natives at an early age are at a huge disadvantage. If you are working with children today, you need to find a way to acquire digital media literacy skills. If you manage or spend time helping adults that work with children—from principals to pediatricians— you have a responsibility to acquire these skills, and work to find a budget to help train others in them. If you’re funding these organizations, either fund this skill development or help your grantees find another entity that can.

• Might educational (teacher/caregiver/parent) guides that could accompany apps be feasible, possible, and/or useful?

This is an idea that has been around for some time, and it does have potential. We see successful analogues in parent or teacher guides that come with children’s books, in annotated versions of classic books, or even book club questions in the back of novels. You can even do this in real time—PBS Digital, for instance, has some user interfaces for parents that actually surround the child’s content on the screen. As digital natives become adults, it will be more realistic to have digital dashboards that help parents or educators track and guide kids’ use of apps.

·         How can we make evaluation of impact and a research component more feasible?

The sector needs to work to increase the demand for high-quality, research-tested products. This will incentivize companies to pay to test their products.

Until we can change the market dynamics, the organizations that can afford to pay for research need to step up. So school systems evaluate the products they use, and share the results with others. Foundations need to include more evaluation dollars in their development grants. And publishers that take corporate social responsibility seriously must incorporate more research into their product pipelines.

• What types of incentives can be offered industry to produce both entertaining and educational media?

• Are there new ways to incent “kids first content” in the apps marketplace?

• I don’t know that there are new ways, but there are plenty of methods that have worked in other markets that no one has really pushed in the digital learning media sector yet—and they can come from the public, private, or citizen sector. It’s no secret that the marketplace responds to profit potential, and the public-interest community can provide incentives all along the innovation pipeline.
• Large, ambitious prizes have been shown to drive pathbreaking innovation and surface new ideas.
• Advance market commitments—basically, aggregating demand for products with too little supply—have worked in the pharmaceuticals world.
• Foundations could use program-related investments or government could use matching grants to bring more private capital in at the commercialization stage.
• Ratings, certifications, and trustmarks (like the Good Housekeeping seal) can shift consumer buying behavior toward higher-quality products.
• And, of course, there is regulation, which when done wrong can stifle innovation, but at its best can protect children from the low-quality products that the free market sometimes produces.

Is there a role for government?

Yes, there are plenty of useful roles for government:

·         Regulation, of course. If policymakers can cut outdated regulations—for instance, seat-time requirements, or curriculum regulations that favor textbooks over digital media—they can make a huge impact
·         The public sector has huge amounts of money to scale up the most effective, innovative programs and practices
·         Government can also invest in research and early-stage innovation that the private sector sees as too risky, and provide common platforms to build off of (such as the Internet and broadband systems)