Skip to content

3. Evaluation Methods

In this chapter, you will learn about the options available for conducting evaluations. You will be familiarised with different evaluation methods and receive guidance on how to select one or more suitable evaluation methods.

3.1 Introduction

To assess whether learning/teaching objectives have been achieved and to identify the beneficial and detrimental aspects of a VE project, we can take informal steps or conduct a formal evaluation. In doing so, we should always take students’ perspective into account. On an informal level, for example, we can determine through our own (unsystematic) observations that something is not working or that a learning objective has not been achieved. However, this is often not possible in VE projects because students interact with one another asynchronously via communication channels to which we as teachers do not have access (e.g. via messenger apps) or because several groups work virtually in parallel (from home) and we have no insight into the exchange. This means that we need to find ways to gain insights into students’ projects and the associated learning processes.

8. Activity:

Think about how you could gain insights into the teaching and learning processes involved in VE projects. Then compare your ideas with the solution below.

✔
Saved Your answer has been saved
✘

A project or course can be evaluated in countless ways. The most familiar and frequently used method is an (online) questionnaire, where students can give lecturers feedback on their teaching. However, a teaching analysis poll (TAP) – a group discussion led by an external party – can also provide valuable information about your own teaching and the VE project. It is often also possible to use more informal methods, such as feedback discussions, which can be supported by visual forms of feedback (e.g. target method). The choice of evaluation method(s) will always depend on the objective and function of your evaluation. In principle, all qualitative and quantitative methods of empirical research can also be used for evaluations.

As mentioned above, it often does not make sense to assess the achievement of learning/teaching objectives on a purely quantitative level, as you risk overlooking individual differences and learning processes (cf. Arnold et al. 2018: 395ff.). It is better to conduct a holistic evaluation using qualitative methods – or a combination of qualitative and quantitative methods (if necessary at various points in time) – to capture the views, judgements and suggestions of all participants and to evaluate different learning/teaching objectives, materials, tasks, etc. This makes both objective and subjective learning outcomes visible.

3.2 Evaluation Methods

As we cannot provide a comprehensive introduction to (empirical) evaluation methods here, you will find an overview of different evaluation methods – with their advantages and disadvantages – below. We will then focus our attention on questionnaires, as this is one of the most frequently used evaluation methods. This is followed by a brief discussion of the aspects that need to be taken into account when selecting an evaluation method.

informal evaluation methods

formal evaluation methods

Would you like more information?

Further reading:

  • Die eigene (Online)Lehre formativ evaluieren. Beispiele für Ebenen, Fragen und eTools von Nathalie Pöpel (2021).
    You can download the PDF for free here: Handreichung Pöpel (2021)
  • Feedback-Instrumente zur Steuerung und Bilanzierung von Prozessen der Hessischen Lehrkräfteakademie (2018).
    You can download the PDF for free here: Feedback-Instrumente HLA (2018)

3.3 A detailed look at questionnaires as an evaluation method

9. Activity:

a) On the following slides, you will find lots of helpful hints and tips for creating evaluation questionnaires. Please read the information carefully.
b) After working through the slides, you can put your knowledge to the test in a short quiz.

The following didactic dimensions can help you evaluate the methodology and didactics of your VE project. Many standardised and tested evaluation questionnaires from educational institutions contain questions on these dimensions. Of course, these dimensions can also be taken into account when using other forms of evaluation.

Didactic dimensions for teaching evaluations (based on the TUM evaluation form):

  • Course design and structure: Teaching content is structured clearly and transparently, and the course is structured logically and designed well to provide a basis for the desired teaching/learning outcomes.
  • Teaching: Complicated matters are explained understandably in a variety of ways, different methods are used, and students are handled flexibly.
  • Use of media: Teaching and learning materials have a clear and engaging design, suitable and focused visualisations are used, and teaching is supported by the use of materials and tools.
  • Teacher engagement and learning atmosphere: A positive and fear-free atmosphere within the study group is a basic necessity, especially in challenging VE projects. This is mainly driven by teachers, who should foster an open and understanding relationship with students.
  • Student requirements: This regards the complexity/difficulty and scope of content, speed/progress, and workload.
  • Acquisition of knowledge and skills: This relates to the development and testing of 21st-century skills, technical input, active participation of students.

10. Activity:

Assign the following evaluation items to the corresponding dimensions. Most statements should be rated on a 5-point scale from “not at all true” to “completely true”.

(The items are based on evaluation questionnaires from various universities.)

3.4 How do I choose one or more suitable evaluation methods?

As there are advantages and disadvantages to each method, a combination of methods is usually the best option. If you want to gauge the long-term effects of your VE project or obtain strong evidence for its impact, the survey methods to be used will be different to those you would use to examine short-term or medium-term effects. Many of the formal methods listed above can also be used in standardised comparative studies (with control groups) and/or in longitudinal studies.

Your choice of method or combination of methods will mainly depend on what you want to achieve through your evaluation, which is why we have discussed the functions and objectives of evaluations in detail above. So, you should first think about the conclusions that should or must be drawn from your planned evaluation. Once you have identified the objective of your evaluation, this will determine whether you should conduct a formal or informal evaluation. For example, if you want to obtain reliable, detailed results for a fundamental overhaul of your VE project, formal methods are certainly more suitable than informal ones. However, if you just want to get a basic idea as to whether the project is working and students are satisfied, informal methods will probably suffice.

After making this decision, always draw up one or more questions before choosing a survey method. These questions will often reveal appropriate survey methods. For example, if you ask about the learning processes that are taking place, it doesn’t make sense to use a summative method. However, if you are only interested in measurable skills or in the knowledge gained by your students, methods for measuring learning progress (possibly with pre- and post-tests) are a good idea. Qualitative empirical methods can be used if you want to understand learning or communication processes, individual or group-related learning pathways or the relationships between complex action situations. These questions will also help to determine the timing of your evaluation (formative or summative).

Once you have drawn up your questions, you should think about how much time and other resources you have at your disposal. This will also help you decide which type of evaluation you can conduct.

11. Activity:

In the following exercise, consider which method would be most suitable to answer each evaluation question.

Please note: Of course, these questions could also be answered using different/additional methods. However, only one of the proposed methods is a good fit for the questions asked here in each case.

All illustrations on this site by @storyset (https://www.freepik.com/author/stories)

translated by: KERN AG, Sprachendienste Leipzig

This work © 2024 by Prof Dr Almut Ketzer-Nöltge is licensed under CC BY-NC-ND 4.0