Onderzoek hergebruik van leermaterialen / Study into the reuse of learning materials

(English version below the Dutch text)

Er is weinig bekend over de mate waarin open leermaterialen worden hergebruikt en over hoe dat hergebruik dan plaatsvindt. Dit heeft onder meer tot gevolg dat niet duidelijk is hoe effectieve ondersteuning bij hergebruik kan plaatsvinden en welk type leermaterialen het meest hergebruikt wordt, zodat daar voorrang aan kan worden gegeven bij ontwikkelen van open leermaterialen.

Om meer inzicht te krijgen in in welke mate en op welke wijze hergebruik van open leermaterialen plaatsvindt is door Robert Schuwer (lector Open Educational Resources Fontys Hogescholen) en Marjon Baas (ICT&O Adviseur hogeschool Saxion en PhD student bij Universiteit Leiden) een survey opgesteld. We vragen docenten in het hoger onderwijs in Nederland (hbo en wo) die gebruik maken van digitale leermaterialen 10-15 minuten van hun tijd om deze survey in te vullen De survey staat open tot 15 oktober en is te vinden op http://bit.ly/reuse2019. Er is zowel een Nederlandstalige als Engelstalige versie beschikbaar.

De resultaten zullen mede input zijn voor activiteiten die in de zone “Naar digitale (open) leermaterialen” van het Versnellingsplan zullen worden uitgevoerd.

We hopen op een hoge response!

Study into the reuse of learning materials

Little is known about the extent to which Open Educational Resources are reused and how they are reused. One of the consequences of this is that it is not clear how effective support can be provided for reuse and what type of educational resources are most frequently reused, so that these can be given priority when Open Educational Resources are developed.

In order to gain more insight into the extent and manner in which Open Educational Resources are reused, Robert Schuwer (professor in OER at Fontys University of Applied Sciences) and Marjon Baas (ICT&O Advisor at Saxion University of Applied Sciences and PhD student at Leiden University) have drawn up a survey. We ask instructors from Dutch research universities and universities of applied sciences who are using digital learning resources 10-15 minutes of their time to complete this survey. The survey is available in an English version and is open until 15 October. It can be found at http://bit.ly/reuse2019

The results will feed into activities to be carried out in the area “Towards digital (open) learning materials” of the Dutch Acceleration Plan

We hope for a high response!

Towards a definition of learning materials



In the zone “Towards digital (open) educational resources” from the Dutch Acceleration Plan for Educational Innovation using ICT, 7 universities and 2 Universities of Applied Sciences are collaborating to realize the ambition that in 2023, higher education institutions in the Netherlands are able to offer teachers and learners the opportunity to put together and use their optimal mix of learning resources.

Until now, a precise definition of what is meant by “educational resources”, “learning resources” or the more commonly used phrase “learning materials” is lacking. In a background document the zone has produced for presentation purposes, the following description of “learning material” is provided:

It is difficult to give a definition of learning material. The primary role of learning material is to provide the content (learning content) in a certain form (textual, auditory, visual or a mix of these forms). Examples of learning material are digital textbooks, slide decks or MOOCs. This approach excludes educational resources such as digital whiteboards and VR glasses.

Learning materials can make use of sources that, viewed in isolation, are not primarily intended to serve as learning materials, but which, placed in a (learning) context, may have that function. Think, for example, of a Youtube video with information about the Eiffel Tower that is used by a student to answer a question about technical constructions. For that student, this video is part of his optimal mix. It is possible for a teacher to refer to the video as early as on his or her assignment, as well as for a student to look for sources that will help him or her to make the assignment and come across the video in the process. Other examples of such sources are the Wikipedia, newspaper articles and games.

This description is currently sufficiently accurate for the purpose of presentations, but it raises the question if a more precise definition can be found. Such a definition can help in future decisions to include or exclude certain artefacts in the activities of the zone.

It seems that “learning materials” is a fuzzy concept. An individual has an intuition of what it means, but it is difficult to define it more precisely. Some examples to illustrate this.

Example 1

The Draft version of the UNESCO OER Recommendation, discussed on 27 and 28 May in Paris (not publicly available yet) gives the following definition of Open Educational Resources:

Open Educational Resources (OER) are learning, teaching and research material in any format and medium that resides in the Public Domain or are under copyright that have been released under an open license that permits no-cost access, reuse, repurpose, adaptation and redistribution by others.

This definition does not define in a precise manner what learning materials are, nor what carriers could be considered, but focus on the accessibility and usability of it. It seems to be based on the intuitive meaning every individual has. In the debates in Paris, there were no remarks made about the exact definition of “learning, teaching and research material”. Most alternative definitions of OER are similar, or it provides examples of OER in the definition, formulated as e.g. “(…) educational resources (lesson plans, quizzes, syllabi, instructional modules, simulations, etc.)” (see here for some examples).

Example 2

In a survey about practices of reuse that was taken in Fall 2018 in two contexts (Teachers in a Bachelor ICT program at a UoAS in the Netherlands and a Dutch community of practice for teachers in a Bachelor Nursing program), one question was about the types of learning materials used within their courses. The following options, for a great deal assembled from previous surveys on OER ((Schuwer & Janssen, 2016), (De los Arcos et al, 2015)), were presented:

  • Textbook
  • (Powerpoint) presentations
  • Videos (e.g. knowledge clips, tutorials)
  • Assignments
  • Tests
  • Pictures (e.g. photos)
  • (Part of a) course of colleagues
  • (Part of a) third party course
  • Articles
  • Interactive games
  • Digital tool (e.g. an online coding environment)
  • Otherwise, namely…

The option “Digital tool” was added at the version of the survey in the Bachelor ICT program, because of feedback from teachers on a test version of the survey where this option was not present. Many of the around 200 respondents provided input on the “Otherwise” option. Summarized and without input that was a redraft of another option, this leads to:

  • Online tutorials
  • Quizzes
  • Live coding demos
  • Online conference
  • Official developers documentation
  • Websites (e.g. Skills online)
  • Blogs (could be considered a special type of articles)
  • References to websites of publishers (possibly important because of copyrights)
  • Discussion forum
  • VR
  • Search engines
  • Practical scenarios for simulation education
  • Kahoot, menti, answergarden, Youtube, Pathlet

This shows that teachers have a broad perspective on learning materials, including many examples of digital tools (see the last bullet point). Because the zone starts from the perspective of the teacher and student, this broad perspective should be taken in consideration in the definition of learning material.

Example 3

During a discussion in a workshop to come to a broad accepted view of (digital) learning materials, participants connected quality issues to differentiate learning materials from other kinds of materials (e.g. materials having undergone peer review).


Hertzell (n.d.) distinguishes between five types of definitions:

  • Lexical definitions attempt to report usage.
  • Stipulative definitions are those which specify or stipulate the meaning of a word or phrase. Sometimes these involve the introduction of new terms, or the stipulation of new meaning for old terms.
  • Extensional definitions are simply a list of all the things to which the term applies.
  • Intensional definitions list a set of properties such that the term applies to all things having that set of properties, and to nothing else.
  • Ostensive definitions indicate the meaning of a term by providing a sample of the things denoted.

According to Wikipedia, a lexical definition is the sort of definition one is likely to find in the dictionary. Searching Wikipedia, no lexical definitions could be found for “educational resource”, “learning resource” or “learning material”. In the examples above, both intensional definitions (e.g. in the Draft version of the UNESCO OER Recommendation) as ostensive definitions (e.g. in the survey) are found. What types of definition could provide the most accurate description of learning materials is yet unknown.


To find a more accurate definition of learning material, a small and somewhat superficial literature review has been conducted. In Google Scholar, the following queries were formulated:

  • definition “learning materials”
  • definition “educational resources”
  • definition “educational materials”

The results were sorted by relevance and for each query the first 20 results were taken into account.

Results of the small and superficial literature review

In most papers found, a definition of another concept than learning materials was given (e.g. the concept “Learning Analytics”). In some cases these concepts were tightly connected to learning materials, but without a definition of the latter. This is similar as in the examples provided earlier, thereby relying on the intuitive picture of learning materials of an individual.

The concepts “learning materials/resources”, “teaching materials/resources” and “educational materials/resources” are sometimes distinguished, but without a precise definition. The distinction is formulated like “resources used for teaching” and “resources used for learning”.

Downes (2007) notices “It  seems clear (…), that there ought not to be an a priori stipulation that something may or may not be an educational resource. Such stipulation may only serve to limit discussion unproductively.” (p. 31). This may be comparable with the intuition-based approach for describing learning materials. For the subset of OER, he uses the characteristics “type of resource” (e.g. software, papers, courses) and “resource media” (e.g. Web pages, CD-ROM, paper-based).

Tuomi (2013), in the context of OER, recognizes the intuition-based definition of “educational resource” and tries to overcome that with a lexical definition of resource: “a stock or supply of materials or assets that can be drawn in order to function effectively” (p. 61). This approach seem to exclude resources like human beings (unless e.g. teachers are considered as forming a pool of experts available), but could still include chalkboards and presentation screens which, based on intuition, is not what individuals typically have in mind when talking about educational resources.

Some of the papers found provide a definition of the term “Learning object”. In (IEEE, 2002), the following definition is provided:

A learning object is defined as any entity, digital or non-digital, that may be used for learning, education or training.

Wiley (2000) attempts to define the term “Learning object”. He comes to a working definition (p. 6):

Any digital resource that can be reused to support learning.

He considers five types of learning objects (p. 18):

  1. Fundamental: an individual digital resource uncombined with any other.
  2. Combined-closed: a small number of digital resources combined at design time by the learning object’s creator, whose constituent learning objects are not individually accessible for reuse (recoverable) from the combined-closed learning object itself.
  3. Combined-open: a larger number of digital resources combined by a computer in real-time when a request for the object is made, whose constituent learning objects are directly accessible for reuse (recoverable) from the combined-open object.
  4. Generative-presentation: logic and structure for combining or generating and combining lower-level learning objects (fundamental and combined-closed types).
  5. Generative-instructional: logic and structure for combining learning objects (fundamental, combined-closed types, and generative-presentation) and evaluating student interactions with those combinations, created to support the instantiation of abstract instructional strategies (such as “remember and perform a series of steps”).

These and alternative definitions of Learning Object focus on the reusability of learning materials and the opportunities to create learning materials by combining other learning materials. Or as Friesen (2010) formulates: “Each definition highlights (either directly or indirectly) modularity as a technological and design attribute for the object and its content, emphasizing the ‘self-contained,’ ‘building block’ or ‘object-oriented’ nature of the technology” (p. 2).

Although in later years the learning object approach of developing learning materials has been considered as disappointing (see e.g. (Sinclair et al, 2013)), the definitions provided can be used to come to a definition of learning material.

Mishan (2005), in the context of language learning, considers material as a combination of two elements:

  • Text: Paper-based or electronic (audio or visual) data which can be in graphic, audio or print form and includes video, DVD, television, computer-generated or recorded data.
  • Language learning task/s based on it. Task is described as: Learner undertaking in which the target language is comprehended and used for a communicative purpose in order to achieve a particular outcome (goal). (p. xiii)

In this definition, the content, technical type of content, carrier (all in the description of “text”) and learning aspect are distinguishable elements of learning materials. This is comparable with the approach of (Downes, 2007) for OER.

Bundsgaard and Hansen (2011) provide an ostensive definition of learning materials (p. 32):

We understand learning materials as artifacts, e.g. textbooks, blackboards, computers (…)

They lack a more precise definition for their aim: evaluating learning materials in the context of a design for learning (with the latter more precise defined). They distinguish three characteristics of learning materials for investigation in the context of a learning design (p. 33):

  • the potential learning potential, that is, the affordances and challenges of the learning material, and the competences supposedly supported when working with the material;
  • the actualised learning potential, that is, the potential for learning when the design for learning is enacted by integrating the learning material in a situation in a given context; and,
  • the actual learning, that is, how the participants actually develop their competences through working with the learning material or enacting a design for learning.

With this perspective, anything can evaluate as suitable learning material, as long as the evaluation on these three characteristics is positive.

A comparable perspective on what constitutes learning materials is given in (Barker & Campbell, 2010). They write (emphasis added by me) (p. 225):

Defining what we mean by learning materials is more difficult. However, we think that “anything used for teaching and learning” captures the essence of what we are interested in. This approach makes the defining characteristic of learning materials their function and context, as opposed to characteristics that are inherent to the resource; this contrasts them with many other resource such as images, simulations, audio, etc which are more readily defined by resource specific characteristics.

This definition is very broad and includes also laptops, VR glasses and even chalkboards. In the remainder of their paper they describe metadata models for learning materials.


The small and superficial literature review did not come up with a more precise definition of learning materials. In many cases the definition is intensional or ostensive, only focusing on certain aspects learning materials should possess (e.g. accessability, reusability), the role it fulfills (e.g. used for teaching and learning) or is implicitly provided by formulating models for learning materials.

The latter approach gives handles for a practical viewpoint on learning materials without the need to define learning materials in a precise manner. Two of such approaches can be extracted from the articles.

Approach 1: use of a quality model

In this approach, a quality model to distinguish worthwhile learning materials from other artefacts is available. The approach in (Bundsgaard & Hansen, 2011) is an example of this approach. There, a three-step procedure is defined to distinguish learning materials from other artefacts in a specific context. A generic description of learning materials then becomes

Learning material is anything that fulfills the requirements set by the quality model

In the definition of the quality model, the community in which the quality model will be used can add the requirements they consider important for learning materials to the model. This makes this description of learning materials context-dependant.

Approach 2: use of a metadata model

This approach is an extension of the description used in (Barker & Campbell, 2010). It requires the availability of a metadata model to describe learning materials. A generic description for learning material then becomes

Learning material is anything potentially useful for teaching and learning that can be meaningful described by the metadata model

This description requires some detailing of “meaningful”: when is a description considered to be meaningful? Furthermore, in most cases members of a community decide when an artefact is potentially useful for teaching and learning, e.g. based on own experiences. Therefore, this approach also is context-dependant.


Both approaches can be combined to decide whether or not an artefact qualifies as learning material. Context can be added in the requirements of the quality model or the metadata model. E.g. in contexts where a vocabulary is available to describe the content for a certain field, one could add the requirement to a metadata model that for that field it is mandatory to add one or more items of that vocabulary to the description of learning materials. The concepts from (Wiley, 2000), (Mishan, 2005) give examples of characteristics for learning materials for which requirements can be formulated.

For the zone of the Dutch Acceleration plan we could decide to use the definition:

A learning material can be used for teaching and learning and can be meaningful described by the Dutch standard for metadata NL-LOM

Our first task now is to form an opinion of a meaningful description. Experiences should then make clear if this approach is sufficiently accurate for our purposes or that some form of a quality model is also needed to define learning materials in a more accurate manner.


de los Arcos, B., Farrow, R., Pitt, R., Perryman, L-A., Weller, M. & McAndrew, P. (2015). OER Research Hub Data 2013-2015: Educators. OER Research Hub. https://oerknowledgecloud.org/sites/oerknowledgecloud.org/files/20151117-OER-Hub-Data-Report.pdf

Barker, P. A., & Campbell, L. M. (2010). Metadata for learning materials: an overview of existing standards and current developments. Technology, Instruction, Cognition and Learning, 7(3-4), 225-243. https://pureapps2.hw.ac.uk/ws/portalfiles/portal/10607125/TICLpaper.MetadataForEducation_postref.pdf

Bundsgaard, J. & Hansen, T. (2011). Evaluation of Learning Materials: a Holistic Framework. Journal of Learning Design, 4(4). 31-44. https://files.eric.ed.gov/fulltext/EJ963326.pdf

Downes, S. (2007). Models for Sustainable Open Educational Resources. Interdisciplinary Journal of Knowledge and Learning Objects, Vol 3. 29-44.

Friesen, N. (2009). Open Educational Resources: New Possibilities for Change and Sustainability. The International Review of Research in Open and Distributed Learning, 10(5). https://doi.org/10.19173/irrodl.v10i5.664

Hertzel, R. (n.d.). Five Types of Definitions. https://www.taiwanbasic.com/wash/defin.htm (viewed on 1-7-2019)

IEEE (2002). 1484.12.1-2002 – IEEE Standard for Learning Object Metadata.

Mishan, F. (2005). Designing Authenticity Into Language Learning Materials. Intellect, Bristol.

Schuwer, R. & Janssen, B. (2016). OER and MOOCs in the Netherlands: current state of affairs. Open Education Global Conference, Krakow. https://www.robertschuwer.nl/download/Paper-OEC-Krakow-2016-DEF.pdf

Sinclair, J., Joy, M., Yin-Kim Yau, J. & Hagan, S. (2013). A Practice-Oriented Review of Learning Objects. IEEE Transactions on Learning Technologies, 6(2). 177-192.  https://doi.org/10.1109/TLT.2013.6

Tuomi, I. (2013). Open Educational Resources and the Transformation of Education. European Journal of Education, 48(1). 58-78. https://doi.org/10.1111/ejed.12019

Wiley, D. (2000). Connecting Learning Objects to Instructional Design Theory: A Definition, a Metaphor, and a Taxonomy. In Wiley D. (ed), The Instructional Use of Learning Objects. Agency for Instructional Technology and Association for Educational Communications & Technology, Bloomington. http://members.aect.org/publications/InstructionalUseofLearningObjects.pdf

Naar digitale (open) leermiddelen

Ruim een jaar nadat het idee van de Versnellingsagenda Onderwijsinnovatie werd gepresenteerd is op 1 januari jongstleden het vierjarig programma Versnellingsplan Onderwijsinnovatie met ICT gestart. In het verstreken jaar is door veel mensen gewerkt aan het opzetten van een structuur voor dat programma, het vertalen van de agenda naar een meerjarenplan en het aantrekken van de mensen die het programma gaan uitvoeren. Er zijn acht thema’s gedefinieerd waarmee en waarvoor de versnelling in innovatie bereikt moet worden. In het jargon van dit plan worden die thema’s aangeduid met zones. Voor iedere zone is een aanvoerder benoemd middels een procedure waarbij instellingen voor hoger onderwijs een aanvoerder uit hun instelling konden voordragen. Ieder van die kandidaten heeft een sollicitatiegesprek gevoerd met leden van de stuurgroep van het plan. De stuurgroep heeft uiteindelijk de benoemingen bepaald. Instellingen konden ook hun belangstelling tonen om aan één of meer van de zones deel te nemen. Dat heeft uiteindelijk geleid tot de samenstelling van de zones die nu gestart zijn.

   >> Overzicht van alle zones
   >> Versnellingsplan Onderwijsinnovatie met ICT

Ik heb het voorrecht aanvoerder te zijn van zone 4: naar digitale (open) leermiddelen. De idee is dat de deelnemende instellingen gedeelde ambities op dit thema gezamenlijk uitwerken en implementeren. De versnelling ontstaat dan door het principe “samen kom je verder dan alleen”. Het uiteindelijke doel van zone 4 is te komen tot een situatie waarbij instellingen in staat zijn een optimale mix van open en gesloten leermaterialen te kunnen aanbieden.

De eerste bijeenkomsten hebben we gebruikt om de gezamenlijke ambities te inventariseren. Na clustering van die ambities in categorieën heeft dat geleid tot de volgende schematisch weergegeven categorieën:


Beschrijving en samenhang categorieën

Een korte beschrijving van deze categorieën en hun samenhang:

Categorie: didactiek

Digitale leermiddelen zijn niet los te zien van de context waarin ze gebruikt worden. Bij ontwerpen van onderwijs- en leeractiviteiten geeft het principe van constructive alignment houvast bij het bepalen en ontwerpen van die context. Onder constructive alignment wordt verstaan het afstemmen van activiteiten, didactiek, assessment en middelen op de beoogd te behalen leerdoelen of leeruitkomsten. In de ambities wordt in deze context met name benoemd het doel om met digitale leermiddelen beter in staat te zijn onderwijs op maat te aan te kunnen bieden. Anderzijds kan de mix van leermaterialen, en met name de open leermaterialen leiden tot didactische werkvormen die met alleen gesloten leermaterialen niet of heel lastig te realiseren zijn. Ontwikkelingen op dat laatste gebied staan ook wel bekend als Open Pedagogy en Open Educational Practices.

Categorie: creatie en hergebruik open leermaterialen

Om delen en hergebruiken van open leermaterialen te bevorderen zijn naast een technische infrastructuur ook andere elementen nodig. Onderzoek naar adoptie van open delen en hergebruiken van leermaterialen heeft aangetoond dat awareness en activering van docenten en organiseren van ondersteuning nodige voorwaarden zijn om docenten hun leermaterialen open te delen. Vakcommunity’s hebben een rol om gedeelde leermaterialen actueel te houden, om co-creatie van leermaterialen over instellingsgrenzen heen te organiseren en in duurzaam maken van initiatieven voor open delen en hergebruiken van leermaterialen. Daarnaast vergen open leermaterialen investeringen van een instelling. Hoe de business case er voor een instelling uitziet en hoe die optimaal kan worden gemaakt (bijvoorbeeld door meer samenwerking met andere instellingen) wordt in deze categorie verder verkend en geoperationaliseerd.

Categorie: gesloten leermateriaal

Te verwachten is dat in veel contexten een mix van open en gesloten leermaterialen leidt tot een optimale situatie voor het leren van een student. Hoe een dergelijke mix eruit ziet en hoe dit op een wijze te realiseren is waarbij hoge kosten voor student en instelling worden vermeden, geen lock-in op een uitgeversplatform plaatsvindt en toegang tot de leermaterialen gegarandeerd blijft zijn voorbeelden van vraagstukken in deze categorie. Het project eStudybooks is een expliciete casus binnen deze categorie. Het kan onder meer dienen als input voor het opstellen van een business case voor gesloten leermateriaal voor een instelling.

Categorie: infra

Met name voor open leermaterialen is ondersteuning voor delen en hergebruiken nodig om de mogelijkheden van dergelijke leermaterialen te kunnen gebruiken. Essentieel daarbij is een technische infrastructuur met de juiste functionaliteiten om de gewenste ondersteuning effectief en efficiënt te kunnen bieden. Ambities zijn genoemd op implementatie van een repository voor de opslag van leermaterialen, een zoekfunctie waarmee docenten gewenste leermaterialen effectief kunnen localiseren en (exploreren van) gebruiken van Artificial Intelligence technieken om een dergelijke infrastructuur te optimaliseren.

Niet expliciet genoemd, maar waarschijnlijk wel nodig voor een optimale infrastructuur is een transparante aansluiting tussen de infrastructuur voor open en die voor gesloten digitale leermaterialen.

Categorie: voorwaardelijk

Om adoptie van open leermateriaal binnen een instelling te bevorderen zijn enkele randvoorwaarden nodig. Binnen deze zone worden met name open beleid en een waardering voor openheid genoemd als te realiseren ambities.

Wat gaan we doen in 2019?

Na inventarisatie van de ambities en afstemming met de deelnemende  instellingen is een globaal jaarplan voor 2019 vastgesteld. Onder andere de volgende onderwerpen zullen worden opgepakt.

  • Onderzoek naar hoe docenten tot hun keuze voor leermaterialen komen
  • Ervaringen / good practices delen over gebruik van social tools als Perusall en FeedBackFruits
  • Ontwikkeling van een (prototype) zoekschil naar (open) digitale leermaterialen, gedeeld binnen een instelling of gedeeld met de wereld
  • Ervaringen / good practices opdoen, verzamelen en delen met gebruik van open didactieken
  • Ervaringen / good practices opdoen, verzamelen en delen met effectief organiseren van ontwikkeling van digitaal leermateriaal in de driehoek docent – ondersteuning – onderwijskunde
  • Meer inzicht krijgen in gezamenlijke beelden die stakeholders hebben van gebruik van digitaal leermateriaal

Ieder onderwerp wordt opgepakt door een subgroepje met leden van de zone. Ieder subgroepje bepaalt welke activiteiten zullen worden uitgevoerd en hoe de resultaten worden gedissemineerd.

Meer weten?

Er zijn verschillende manieren om op de hoogte te blijven van de ontwikkelingen in deze zone of van het programma.


Zoekend naar een metafoor voor deze zone kwam ik uit bij de 6e symfonie, de Pastorale, van mijn favoriete componist Ludwig van Beethoven. We gaan vol goede moed beginnen aan dit programma (Erwachen heiterer Empfindungen bei der Ankunft auf dem Lande), realiseren resultaten, maar zullen daarbij ook geduld moeten hebben (Szene am Bach), we zullen zeker genoegen hebben bij het uitvoeren van de activiteiten en zoeken daarbij samenwerking met andere partijen (Lustiges Zusammensein der Landleute), het gaat ongetwijfeld ook knetteren (Gewitter und Sturm), maar we gaan er alles aan doen om tot een mooi en succesvol resultaat te komen (Hirtengesang. Frohe und dankbare Gefühle nach dem Sturm).