Free download from www.hsrc p ress.ac.za Compiled by the Research Programme on Human Resources Development, Human Sciences Research Council Commissioned by JET Education Services and funded by the Business Trust Published by HSRC Press Private Bag X9182, Cape Town, 8000, South Africa www.hsrcpress.ac.za © 2005 Human Sciences Research Council First published 2005 All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. ISBN 0-7969-2041-9 Cover by Flame Design Produced by comPress Distributed in Africa by Blue Weaver Marketing and Distribution, P.O. Box 30370, Tokai, Cape Town, South Africa, 7966, South Africa. Tel +27 +21 701-4477 Fax: +27 +21 701-7302 email: booksales@hsrc.ac.za Distributed worldwide, except Africa, by Independent Publishers Group, 814 North Franklin Street, Chicago, IL 60610, USA. www.ipgbook.com To order, call toll-free: 1-800-888-4741 All other enquiries, Tel: +1 +312-337-0747 Fax: +1 +312-337-5985 email: Frontdesk@ipgbook.com Free download from www.hsrc p ress.ac.za Contents List of figures and tables iv Executive summary v Acknowledgements ix Abbreviations x 1 Introduction 1 1.1 Structure of this report 2 2 Features of the Quality Learning Project 5 3 Data preparation, statistical procedures and methodology 7 3.1 Introduction to the statistical analysis 7 3.2 Data preparation 7 3.3 Data reliability problems 9 3.4 Analytical strategy 13 3.5 School effects 15 3.6 Household effect variables 19 3.7 Removing the household effects from the school environment and language and mathematics interest and experience variables 21 3.8 The set of explanatory variables 24 3.9 The language and mathematics test scores 24 3.10 Do richer communities have better schools? 27 3.11 Regression of language and mathematics scores on the explanatory variables 28 4 Commentary on the findings 33 Appendices Appendix 1 Districts participating in the QLP 35 Appendix 2 The QLP Model 36 Appendix 3 The programmes of the QLP 39 Appendix 4 Sampling methodology 41 Appendix 5 Developing and administering the instruments 46 Appendix 6 Conversion of data from the learner questionnaire into the list of variables used in this study 52 Appendix 7 Learner background questionnaire 55 Appendix 8 List of schools ordered by general factor, Grade 11 factor and Senior Certificate 73 References 76 Free download from www.hsrc p ress.ac.za Figures Figure 1: Survey sample 5 Figure 2: Education of mothers 11 Figure 3: Education of fathers 11 Figure 4: Household wealth vs household income 12 Figure 5: General school factor on household wealth 27 Figure 6: School test factor on household wealth 28 Figure A3.1: QLP programmes 40 Tables Table 1: Total sample obtained for the baseline fieldwork in 2000 6 Table 2: Language most often spoken at home, by population group 9 Table 3: Education of mother and father, by grade 10 Table 4: Expected level of the household wealth score at different levels of income 13 Table 5: Analysis of variance of school environment, interest and learning experience scores 16 Table 6: Correlation matrix and factor analysis of the regression coefficients of school- level variables 17 Table 7: Regression coefficients for Senior Certificate pass rates on general and Grade 11 factors, 1999 and 2000 18 Table 8: Distribution of the use of the language of instruction at home 19 Table 9: Distribution of household scores 20 Table 10: Regression of study aids, meals, parental support, time use and home reading scores on whether an African language is spoken at home, logarithm of household wealth, parental support score (time use and home reading only) and a Grade 11 dummy variable 21 Table 11: Regression of interest and experience residuals on home effects variables 22 Table 12: Correlation matrix and factor analysis of the regression coefficients of school- level variables after the school and household effects have been removed 23 Table 13: Distribution of language and mathematics marks 25 Table 14: Factor analysis of test scores and Senior Certificate results 26 Table 15: Percentage test scores regressed on explanatory variables 29 Table 16: Language scores by quintile 31 Table 17: Mathematics scores by quintile 32 Table A 2.1: Outcomes and covariates at district level 37 Table A 4.1: Number of schools sampled per district 42 Table A 4.2: Number of learners assessed by school district 44 Table A 4.3: Total sample obtained for the baseline fieldwork in 2000 45 Table A 5.1: List of instruments 46 Table A 5.2: Test topics – mathematics, Grades 9 and 11 48 Table A 5.3: Reading and writing skills tested, Grades 9 and 11 49 Table A 5.4: Testing duration of the assessment instruments in minutes 50 iv ©HSRC 2005 List of figures and tables v Free download from www.hsrc p ress.ac.za Introduction Despite all the difficulties associated with the expansion of its educational system during the twentieth century, South Africa has done well in systematically lengthening the average education of each successive age cohort. By the 1990s, more than 90 per cent of the 7–16 age group was enrolled in school, although not every learner was putting in a full day’s attendance. But the quality of the output from the school system has left much to be desired. For many years, close to half of the Senior Certificate candidates have failed the examination outright. And international comparisons, such as the Third International Mathematics and Science Study of 1995, have given no comfort. South Africa was bottom of both the mathematics and science tables, even though a number of similar, middle-income countries were included in the study. In seeking to identify the reasons for this situation, it is important to relate educational outputs (competencies, as measured for instance by Senior Certificate examinations or standardised tests) to inputs. The most obvious input is that of the school itself – the quality of teachers, facilities and management. Another input is that of the household – the education of the parents, household income and wealth, and support for learners. Education is a joint product of school and home, and learners who are backed by strong household resources have an advantage. The abilities and proclivities of individual learners is a third input; learners from the same household and the same school can end up with very different profiles of achievement. Determining the relative contributions of these inputs to educational outputs is not straightforward. Partly this is a data problem. The information necessary to carry out a comprehensive analysis is extensive and usually not fully available. Partly it is a statistical problem: many of the explanatory variables are themselves related in complex ways, so identifying the true drivers of the situation under analysis is difficult. Moreover, in South Africa very little educational production function analysis has been undertaken, so there are few landmark results from which one can take one’s bearings. It is no exaggeration to say that educational production function analysis in South Africa is in a preliminary exploration phase. The results of this study must be interpreted in that light. Up until the Quality Learning Project’s (QLP) baseline study in 2000, no South African data set had ever included test results, school characteristics and information on the household circumstances of individual learners. Before that one could, as in the analysis of Senior Certificate results, relate results to schools and schools to the communities within which they were located. But one could not relate individual learners to the households from which they came. The QLP data set therefore offers a new analytical opportunity. The research question posed for this study was: what are the effects of socio-economic variables on educational outcomes in the QLP schools? Limitations of the study Before describing the methods and the findings of the analysis, it is important to note two limitations of this study. The first is that, within the time and resources available, it was not possible to use the full QLP data set. This study, therefore, concentrates on the learner background questionnaire (which elicited information on households) and the learner achievement questionnaire. It did not use (with one small exception) the data v ©HSRC 2005 Executive summary Free download from www.hsrc p ress.ac.za vi ©HSRC 2005 from the school or educator questionnaires. Of course, the schools attended by learners were identified in the background and achievement questionnaires, so that school effects could be identified. But they were not correlated with a wider set of characteristics of schools. Thus schools appear in this study through the eyes of learners, rather than through those of teachers or principals. The second limitation of this study is the reliability of the information about households. The information collected covered: • Education of parents; • An indicator of household wealth, as evidenced by items within the home; • Language mostly spoken at home; • How often the language of instruction is spoken at home; • Frequency of meals; • Reading at home; • Parental practices supporting education; • Time use. Unfortunately, the data on the education of parents turned out to be unusable. Preliminary regression results produced incoherent results and checking of average educational levels in the immediate environment of the school from the 1996 Census suggested that educational levels had been exaggerated, particularly by Grade 9 pupils. The exaggeration was probably worse at the bottom than at the top. It was also not systematic, so that there was no way of working back from reported to actual education. The variable had to be discarded – a serious blow, since it is a very important one. Its absence makes the findings of this study less certain than they would otherwise be. There may also have been some (but less fatal) exaggeration of items contained within the home. The conclusion from this experience is that household variables really need to be collected from adults within the households rather than learners at schools. The reporting of educational levels in population censuses since 1960 has been reasonably accurate and is certainly much more accurate than the QLP data. The explanatory variables A great many questions in the learner background questionnaire dealt with the school environment, and the interest and experience of learners in the language of instruction and mathematics in both Grade 9 and Grade 11. These data were combined into nine indices (one for school environment and eight covering interest and experience by grade and subject). Each index measured positive or negative orientation of the learner towards the topic in question. The first question becomes: how much of the variance in indices could be explained by the schools themselves? The answer turned out to be between 10 and 30 per cent. The correlations between indices were such that one could identify the most generally favoured schools and relate them to performance in the 1999 and 2000 Senior Certificate examinations. Schools approved by pupils generally did better in the Senior Certificate examinations, though the correlation was not perfect. So one can conclude that the indices, in part, measured something about the school. But the 70 per cent or more of unexplained variance meant that they measured other things as well. Some of the remaining variance was explained by differences between households. The rest can be Learner Performance in South Africa Free download from www.hsrc p ress.ac.za ascribed to individual learner motivation and morale. The statistical analysis enables one to separate out which portion of the indices should be ascribed to which levels. Turning now to the household – the main entity of interest in this study – it quite rapidly emerged that language use affected test results. If the language of instruction was used a lot in the home, test results rose. This really was an issue for households in which an African language was mostly spoken at home. For others, the language mostly used at home and the language of instruction was likely to be the same. The other fundamental household variable is household wealth. One could then start to relate other household variables to these two. Thus: • The study-aid score rises with household wealth as does the meal score (though at a given level of household wealth, this is lower for African-language households); • Interestingly, parental support for study is stronger among African-language households and rises with household wealth. It drops from Grade 9 to Grade 11; • Time-use patterns conducive to study are likely to be better if an African language is spoken at home, better as household wealth and parental support rises, but worse in Grade 11 than in Grade 9. Reading at home rises with household wealth and parental support, but is less extensive in African-language households. By using the household explanatory variables one could remove the effects of households as well as schools on the school environment and interest/experience indices. The correlation among the purged indices threw up three patterns: • A general orientation to school; • An interest in mathematics; • A positive interest in subjects, but negative experience of them. The analytical approach enabled one to separate out the data into individual-level, household-level and school-level information, in such a way as to reduce the correlations between the explanatory variables. Other variables were added, namely: gender and age for grade in the case of learners and urban or rural location in the case of schools. The test results The test results had the following characteristics: • They were higher for the language of instruction than mathematics; • There was a considerable deterioration in language scores between Grade 9 and Grade 11. This finding is significant because experts, at grade appropriate levels, set the tests. The achieved improvement (off a low base: the median mark in Grade 9 was 38 per cent) is not equal to the expected improvement; • The level of mathematics performance in Grades 9 and 11 was approximately the same and extremely poor (median in Grade 9 was 7 per cent and in Grade 11 was 8 per cent). Eighty per cent of pupils scored less than 15 per cent in Grade 9 and 16 per cent in Grade 11. Even at the 95th percentile, pupils achieved a mark of less than 30 per cent in both grades. There is a significant but not perfect correlation between school achievement in the tests and its achievement in the Senior Certificate examinations of 1999 and 2000. Tests and Senior Certificate results are measuring approximately the same thing. There are also significant correlations between: vii ©HSRC 2005 Executive summary Free download from www.hsrc p ress.ac.za • The school general quality factor perceived by pupils and the mean of the logarithm of household wealth derived from pupils at the school; • The school achievement quality factor (derived from test results and Senior Certificate results) and the mean of the logarithm of household wealth. Not surprisingly, richer communities tend to have better schools. The relation between test results and the explanatory variables Language of instruction Individual level variables: the ones which matter are general positive orientation (which has a positive effect) and the over-age for grade variable (which has a negative effect). The effects are stronger at Grade 9 than Grade 11. The gender coefficient is not significantly different from zero at the 5 per cent level. Household level variables: African home language is a marked disadvantage at the Grade 11 level. Whether the language of instruction is spoken at home often is a marked advantage. Household wealth exerts a positive effect. Study aids and meals exert a positive influence over and above the expected level. School level variables: The school general factor exerts a positive influence. Other things being equal, rural schools do better than urban schools. Mathematics Because the average result is so poor, school and household variables explain considerably less of the mathematics score; individual ability and proclivity is the main cause of such variation in marks. This is confirmed by the result when one adds in the language mark residual (after school and home effects have been removed). Conclusions The findings of this study can be generalised only for QLP schools and not the national school system. Suburban (as opposed to township) schools are entirely unrepresented in the QLP universe. QLP schools are drawn from the bottom 70–80 per cent of schools, but they have not been randomly drawn. Inferences to the wider school system would be risky. Nonetheless, the general finding in this study – that social and economic variables do not play an enormous role in determining performance at the individual level, with the exception of language variables – deserves further investigation. Moreover, included in the set of social and economic variables are behavioural variables that can change at existing levels of household incomes and wealth. The rules to be followed are simple: • Feed learners as well as possible; • Equip them with a full range of inexpensive study aids; • Talk to them often in the language of instruction. By contrast, household wealth does not give much of an edge in school performance. And competence in the language of instruction is valuable not only in itself, but as a means to improved mathematics performance. viii ©HSRC 2005 Learner Performance in South Africa Free download from www.hsrc p ress.ac.za Acknowledgements Nick Taylor and Mark Orkin played a pivotal role in conceiving this research project in discussion at a meeting between the HSRC and Joint Education Trust (now JET Education Services) staff in early 2001. We thank the following people for their various contributions to the development of this report, in alphabetical order: • Nicolaas Claassen and Godwin Khoza for their inputs; • Helen Perry for her useful comments and suggestions on the draft and for providing an introduction to production functions; • Jacques Pieterse for his advice on statistical aspects; • Phumudzo Singo for administrative assistance; • Elsie Venter for data cleaning, preparation and for running the procedures; • Penny Vingevold for her support and critical input; • Mariette Visser for preparing the Senior Certificate data; • All those attending the presentation of the first full draft of this paper, including Carol Deliwe. ix ©HSRC 2005 Free download from www.hsrc p ress.ac.za Abbreviations FRC Free reponse question JET Joint Education Trust MCQ Multiple choice question NBI National Business Initiative NGO Non-governmental organisation QLP Quality Learning Project SCE Senior Certificate Examination x ©HSRC 2005 Free download from www.hsrc p ress.ac.za . the interest and experience of learners in the language of instruction and mathematics in both Grade 9 and Grade 11. These data were combined into nine indices. study must be interpreted in that light. Up until the Quality Learning Project’s (QLP) baseline study in 2000, no South African data set had ever included test