Firmino, J. Guilherme, A., Leme, A. C., Nunes L. C. (2023). The Termination of Publicly Funded Private School Contracts: Supply and Demand Side Effects. Journal of School Choice, 17(1), 82-102. 

Abstract: We evaluate the demand and supply side effects of a policy which cuts funding to a significant portion of publicly funded private classes in Portugal, Contratos de Associação (AC), in which students do not pay tuition fees and are under the same admission criteria as in Public Schools – i.e., these private schools cannot select students based on their socioeconomic status or prior achievement. This policy established that from 2016/17, Private school classes would no longer receive Contratos de Associação funding if the government deemed that there were enough nearby Public Schools with capacity to absorb the new student cohorts. Compared to the pre-reform student cohort, we find that affected students changed the demand patterns for different types of schools: both the number of Public school classes and regular Full-Fee Private School classes increased, but the rise in the former was larger in absolute terms. Nevertheless, this pattern of movements was heterogenous across different sub-groups of the student population. While non-low-income students switched both to Public and Private school classes, we verify practically a one-to-one movement of the low-income student cohort from AC schools to Public schools only. On the supply-side, we find that the decrease in AC schools was not fully compensated by the increase of schools that are now under different Private School systems and that some of these schools later shut down.

https://doi.org/10.1080/15582159.2023.2169810

Almeida, S., Firmino J., Mesquita, J. Hortas, M. J., Nunes, L. C. (2022). Academic performance and territorial patterns of students with an immigrant background in the Lisbon Metropolitan Area. Evidence at the municipal, school and class levels. Cidades, Comunidades e Territórios.

Abstract: This paper focuses on the territorial distribution of students with an immigrant background enrolled in the 3^rd cycle of basic education in Portugal and on the differences in the academic performance of students enrolled in the last year of this cycle based on their birthplace and immigrant background when compared to their native peers in the Lisbon Metropolitan Area. These differences are examined by estimating several linear regression models using as dependent variable three performance indicators – student’s results in the 9^th grade national exams in the Maths and Portuguese Language subjects, as well as a binary indicator of a successful academic record during the 3^rd cycle. The observed results confirm the hypothesis that there are significant differences in the students’ academic performance depending on their immigrant background and birthplace: (i) 2^nd-generation and 1^st-generation students perform worse than Native students; (ii) students from Brazil and PALOP countries have the most significant differences compared to students from Portugal. We also identify that a substantial part of these differences is already present in the end of the 2^nd cycle of basic education. Furthermore, our results indicate that a considerable part of the differences is explained by factors inherent to the school and the class of the student, and not so much to the municipality, which might indicate the existence of some type of segregation experienced by these students, either at intra-municipality level (by the different schools) or intra-school level (by the different classes).

https://revistas.rcaap.pt/cct/article/view/25864

Reis, A. B. (2022). Education Policies. In J. M. Fernandes, P. C. Magalhães, A. C. Pinto (Ed.) The Oxford Handbook of Portuguese Politics. Oxford University Press.

Abstract: The last 45 years in Portugal witnessed a huge expansion of enrolment rates in secondary and higher education. Simultaneously, the participation in international assessments like PISA since 2000 shows an improvement in Portuguese results. Until 2015 there was a trend to extend external evaluation from secondary to lower levels of education, which contributed to stronger school accountability namely through the publication of school rankings. In 2015 some national exams were eliminated, and the future evolution is uncertain. Teachers’ training, evaluation and remuneration have probably been the main factor of dispute in the political arena regarding the educational system. The renewal of the teaching personal that will be needed in the next decade is one of the major challenges facing the educational system and will depend a lot on the redefinition of these policies.

https://global.oup.com/academic/product/the-oxford-handbook-of-portuguese-politics-9780192855404?cc=pt&lang=en&

Nunes, L. C., Reis, A. B., Freitas P., Conceição, D. (2022). Rotatividade dos docentes nas escolas públicas portuguesas. (Policy Brief). Centro de Economia da Educação da Nova School of Business and Economics. 

Resumo: Neste Policy Brief analisamos a rotatividade dos docentes entre agrupamentos de escolas do ensino público em Portugal Continental entre os anos letivos de 2008/09 e 2017/18. Averiguamos ainda se estes movimentos estão associados às características dos agrupamentos e à composição do seu corpo docente, por exemplo, ao nível do tipo de contrato e do nível de ensino. No contexto atual de escassez de professores em Portugal, em que a estabilidade é um fator determinante na atração de docentes para a carreira, é particularmente importante conhecer o grau de rotatividade dos atuais docentes em diferentes tipos de escolas. Este trabalho pretende contribuir para um conhecimento mais detalhado da rotatividade dos docentes e desta forma sustentar políticas públicas que conduzam a uma maior estabilidade dos docentes nas escolas.

https://www.novasbe.unl.pt/Portals/0/KnowledgeCenters/Economics%20of%20Education/Projects/Policy%20Brief%20Rotatividade%20Docentes/Policy%20Briefing_2_Final_PT.pdf

Nunes, L. C., Reis A. B., Freitas P., Conceição D. (2022). Medidas Educativas no Contexto Atual de Falta de Professores. (Policy Brief). Centro de Economia da Educação da Nova School of Business and Economics.

Resumo: Neste Policy Brief fazemos um enquadramento das necessidades de recrutamento futuras de novos docentes, assim como de algumas características do atual corpo docente em comparação com outros sistemas de ensino. Apresentamos também alguns dados sumários sobre a evolução recente do desempenho dos alunos portugueses e que devem estar presentes num momento de definir políticas educativas com vista ao recrutamento de novos docentes. Finalmente, elencamos um conjunto de medidas educativas que têm sido apresentadas e consideradas para mitigar o problema da escassez de professores. Identificamos ainda na literatura científica os estudos realizados no passado sobre políticas similares implementadas em diversos sistemas de ensino, assinalando a evidência apresentada sobre os seus impactos no recrutamento de docentes e no desempenho dos alunos.

https://www.novasbe.unl.pt/Portals/0/KnowledgeCenters/Economics%20of%20Education/Projects/Policy%20Brief%20Necessidades%20Docentes/Policy%20Briefing_Midterm_Final.pdf

https://www.novasbe.unl.pt/Portals/0/KnowledgeCenters/Economics%20of%20Education/Projects/Policy%20Brief%20Rotatividade%20Docentes/Policy%20Briefing_2_Final_PT.pdf

Luís Catela Nunes, Ana Balcão Reis, Carmo Seabra, Pedro Freitas and Rodrigo Ferreira, Pedro Carneiro. 2021. “O Impacto do Professor nas Aprendizagens do Aluno: Estimativas para Portugal.” EDULOG - Fundação Belmiro de Azevedo.

Este projeto tem como objetivo determinar o impacto que um professor pode ter no processo de aprendizagem dos alunos, dependendo da qualidade dos seus métodos de ensino. Para quantificar a qualidade do docente, esta pesquisa recorrerá ao valor adicionado de cada um, em particular à aprendizagem média dos seus alunos – algo que normalmente é calculado através da evolução das notas dos alunos em dois períodos adjacentes (antes e depois do aluno ser exposto aos métodos de ensino desse professor). Para isso, a pesquisa conduzida requer o acesso a microdados administrativos de alunos e professores, disponíveis apenas nos EUA e em alguns países em vias de desenvolvimento. Foram também criadas novas bases de dados em Portugal que vinculam os professores a cada aluno em cada ano escolar e aos resultados do 4.ª, 6.ª, 9.ª e 12.ª ano de escolaridade. Estes permitiram, deste modo, a realização deste tipo de estudos em Portugal.

https://www.edulog.pt/storage/app/uploads/public/60d/5a3/31d/60d5a331d8bf4706882738.pdf

Angelo, Catarina, and Ana Balcão Reis. 2021. “Gender Gaps in Different Grading Systems.” Education Economics 29 (1).

This paper analyzes the impact of grading practices on the gender gap in student achievement. We examine the gender difference in the difference between teacher grading and scores on national exams to test whether there are gender differences associated with different grading systems. We use data on 21 subjects across humanities and sciences for students taking exams between the 4th and 12th grades from 2007 to 2018 in Portugal. Our results indicate that a grading system based on exams favors boys, while one based on teacher assessment favors girls.

https://doi.org/10.1080/09645292.2020.1853681

Silva, Pedro Luís, Luís Catela Nunes, Maria do Carmo Seabra, Ana Balcão Reis, and Miguel Alves. 2020. “Student Selection and Performance in Higher Education: Admission Exams vs. High School Scores.” Education Economics 28 (5).

Versão mais recente apresentada no Fourth Lisbon Research Workshop: Economics, Statistics and Econometrics of Education (http://cemapre.iseg.ulisboa.pt/educonf/4e3/Parallell_Sessions.php)

In many countries, students are selected into higher education institutions based on their achievement at high school, measured by scores given by their teachers and obtained on final national exams. This paper compares these two measures in terms of their ability to predict students' success in higher education. Accounting for sample selection problems, we find that the high school score is a stronger predictor of students' performance at university. However, the score obtained in the mathematics national exam still adds information. We conclude that both measures are complementary, a result that is relevant for university recruitment and selection policies.

https://doi.org/10.1080/09645292.2020.1782846

Câmara Leme, Afonso, Josep-Oriol Escardíbul, Luís Catela Nunes, Ana Balcão Reis, and Maria do Carmo Seabra. 2020. “The Effect of a Specialized versus a General Upper-Secondary School Curriculum on Students’ Performance and Inequality. A Difference-in-Differences Cross-Country Analysis.” Applied Economics 52 (39).

Countries differ in their academic upper-secondary school systems whereby some require their students to choose a specialization track from a set of areas – typically natural sciences, economic sciences, or humanities – and follow that specialization for the course of their upper secondary education years whereas, by contrast, others follow a general curriculum in which students are not required to follow a single specialization and thus, may receive a more general education. Because countries follow only one system or the other, a cross-country analysis is required to estimate the possible effects on students’ achievement of these institutional differences. We measure achievement with the scores in reading and mathematics in international assessment programmes. An international difference-in-differences approach is chosen to account for country heterogeneity and unobserved factors influencing student outcomes, by using data from three different large-scale international student assessments. Results suggest that there is a negative effect from specialization tracking and that this may be greater for specific sub-groups of the student population – in particular, for students with a low socioeconomic status.

https://doi.org/10.1080/00036846.2020.1734183

Reis, Ana Balcão. 2020. “Public Funding of Higher Education: Who Gains, Who Loses?” International Journal of Economic Theory 16 (2).

This paper analyses the effects of public funding of higher education on individuals’ welfare, taking into account the hierarchical nature of the education system and the fact that parents may complement basic public education with private tutoring. Although public funding financed by a proportional income tax increases access to higher education, some of the agents who gain access lose in welfare terms. Moreover, it is shown that the marginal agent regarding access to higher education would always prefer a pure private funding system in which he would not send his offspring to higher education but would also not pay tuition fees or pay taxes.

https://doi.org/10.1111/ijet.12176

Cerdeira, José Miguel, Luís Catela Nunes, Ana Balcão Reis, and Maria do Carmo Seabra. 2018. “Predictors of Student Success in Higher Education: Secondary School Internal Scores versus National Exams.” Higher Education Quarterly 72 (4).

In many countries entrance to Higher Education is determined by the performance of students in secondary school and/or the scores obtained in national exams. The relative weight of these two scores on the admission decision is a relevant policy topic, given its implication on who is admitted to university. The purpose of this paper is to investigate the relative predictive power of these two measures on the academic performance of students in Higher Education. It makes use of a dataset of Bachelor students from Portuguese Higher Education institutions with detailed information about their characteristics and past achievement results. The measure of academic achievement considered is the bachelor’s final average score. The main finding is that the scores given by teachers in secondary school are better predictors of subsequent performance than the access exam scores. The relevance of factors like working status, social support and gender vary with the reputation of the degree and the type of Higher Education institution, university versus polytechnic. A noteworthy result is that the added predictive contribution of parents' education is always negligible when past success measures are already taken into account.

http://doi.org/10.1111/hequ.12158

Nunes, Luís Catela, Ana Balcão Reis, and Maria do Carmo Seabra. 2018. “Is Retention Beneficial to Low-Achieving Students? Evidence from Portugal.” Applied Economics 50 (40).

The role of retention as an educational tool to overcome under-achievement is a hotly debated issue, especially given that the results in the literature are not consensual. The Portuguese case is particularly well suited to study this issue: all students must take standardized national exams at specific grades. Moreover, the available dataset tracks the performance of students over time. Therefore, we are able to measure the impact of students’ retention on their subsequent academic performance since we can control for each student’s initial level of ability at the moment of retention. We use a propensity score matching approach, in which retained and promoted 4th grade students are matched according to their socioeconomic characteristics and the scores obtained in national exams. To address potentially remaining endogeneity biases, we also use the culture of retention at school level as an instrumental variable. The results suggest that in some situations retentions may have on average a positive impact on future achievement. However, in the cases where statistically significant impacts are found, the estimated magnitudes are relatively small. Our results are relevant for countries with high retention rates that are considering alternative educational policies to promote students’ achievement.

http://dx.doi.org/10.1080/00036846.2018.1444261

Freitas, Pedro, Luís Catela Nunes, Ana Balcão Reis, Maria do Carmo Seabra, and Adriana Ferro. 2016. “Correcting for Sample Problems in PISA and the Improvement in Portuguese Students’ Performance.” Assessment in Education: Principles, Policy & Practice 23 (4).

The results of large-scale international assessments such as Programme for International Student Assessment (PISA) have attracted a considerable attention worldwide and are often used by policymakers to support educational policies. To ensure that the published results represent the actual population, these surveys go through a thorough scrutiny to ensure their validity. One important issue that may affect the results is the occurrence of different participation rates across groups with heterogeneous average student scores. In this study, we illustrate how problems of representativeness of the samples may be corrected by the use of post-stratified weights. We use the case of Portugal, a country where we uncover a meaningful difference between the actual population and the PISA samples in terms of the distribution of students per grade and track of studies. We show that after post-stratification, the temporal evolution of the PISA scores is quite different from the one reported.

https://doi.org/10.1080/0969594X.2015.1105784

Morgado, António, Tiago Neves Sequeira, Marcelo Santos, Alexandra Ferreira-Lopes, and Ana Balcão Reis. 2016. “Measuring Labour Mismatch in Europe.” Social Indicators Research 129 (1).

We calculate aggregate and comparable measures of mismatch in the labour market for 30 European countries. These indicators measure vertical mismatch (related to the level of education, e.g. overeducation, and undereducation) and horizontal mismatch (related to the field of education) and are comparable across countries and through time. In European countries, between 15 % to nearly 35 % of workers have a job for which they have more (or less) qualifications than the usual level. Approximately 20 % to nearly 50 % work in a job for which they do not have the usual field qualification. There is a great variability on mismatch across European labour markets. Undereducation affects more workers than overeducation in most European countries. Low correlations between mismatch and unemployment indicate that mismatch should be regarded as an additional informative variable, thus useful to characterize labour markets.

https://doi.org/10.1007/s11205-015-1097-0

Nunes, Luís Catela, Ana Balcão Reis, and Maria do Carmo Seabra. 2015. “The Publication of School Rankings: A Step toward Increased Accountability?” Economics of Education Review 49 (December).

This paper contributes to the discussion of the effects of the publication of school rankings based on students’ scores on national exams. We study the effectiveness of this (low stakes) accountability mechanism. Our results suggest that the publication of rankings has clear effects upon families and schools in Portugal. After the rankings publication, fewer students enrol in schools that are rated poorly and the probability of closure of these schools increases. These effects are stronger for private schools.

https://doi.org/10.1016/j.econedurev.2015.07.008

Ana Balcão Reis, Luís Catela Nunes, and Maria do Carmo Seabra. 2015. “School Rankings: The Impact on Public and Private Schools.” In The School and the Students’ Performance, edited by Fundação Francisco Manuel dos Santos, 1st ed., 71–86.

As a result of the 1st Conference of the Education Month, the book School and students’ performance was developed, highlighting the main topics approached during the Conference. The chapter “The school Rankings: the impact on public and private schools” was written by Ana Balcão Reis, Luís Catela Nunes and Carmo Seabra. It focuses on the impact of the first release of school’s rankings on the Portuguese schools, given its importance for public policy. The authors conducted research with the goal of understanding how did the schools’ performance change after the release of the rankings and if there was a significant difference between public and private schools. The results suggest different effects on public and private schools. On average, private schools present better positions in the rankings, and there is an increasing divergence between the two types of schools. While public schools remained still, a great part of private schools with bad performances closed.

Capítulo e livro publicados (em português): https://www.ffms.pt/FileDownload/535cb4cc-4cc8-4de5-9a40-a9bfcaceaef4/livro-a-escola-e-o-desempenho-dos-alunos

Gender gaps in different assessment systems: The role of teacher gender

Autores: Catarina Angelo, Ana Balcão Reis

Previous research has identified a gender gap in the difference between teacher grading and scores on national exams at the end of secondary school. We go a step further and look at how teacher characteristics may influence this gender gap. We find that exams are relatively more favorable for boys, regardless of the teacher gender or the gender matching. Results suggest that having a male teacher tends to increase the assessment gap for all students through a greater decrease from teacher grades to exam scores, the impact being less for boys.

Nova SBE Working Paper Series No. 640

http://hdl.handle.net/10362/133797

No country for young kids? The effects of school starting age throughout childhood and beyond

Autores: Gonçalo Lima, Luís Catela Nunes, Ana Balcão Reis, Maria do Carmo Seabra

Being the youngest in a cohort entails many penalties. Using administrative data of every public-school student in Portugal, we show that although performance gains from being 1-year older fade quickly from primary education to high school, age-related penalties persist through a combination of grade retention, educational tracking and testing policies. Those that start school younger are more likely to repeat grades and ultimately drop out from school. Older entrants are more likely to enroll in scientific curricula in high school, are more successful at accessing public higher education and enroll in more selective undergraduate courses.

Nova SBE Working Paper Series No. 639

http://hdl.handle.net/10362/133796

Academic performance and territorial patterns of students with an immigrant background in the Lisbon Metropolitan Area

Autores: Sílvia de Almeida, João Firmino, José Mesquita, Maria João Hortas, Luís Catela Nunes

This paper focuses on the territorial distribution of students with an immigrant background enrolled in the 3^rd cycle of basic education in Portugal and on the differences in the academic performance of students enrolled in the last year of this cycle based on their birthplace and immigrant background when compared to their native peers in the Lisbon Metropolitan Area. These differences are examined by estimating several linear regression models using as dependent variable three performance indicators – student’s results in the 9^th grade national exams in the Maths and Portuguese Language subjects, as well as a binary indicator of a successful academic record during the 3^rd cycle. The observed results confirm the hypothesis that there are significant differences in the students’ academic performance depending on their immigrant background and birthplace: (i) 2^nd-generation and 1^st-generation students perform worse than Native students; (ii) students from Brazil and PALOP countries have the most significant differences compared to students from Portugal. We also identify that a substantial part of these differences is already present in the end of the 2^nd cycle of basic education. Furthermore, our results indicate that a considerable part of the differences is explained by factors inherent to the school and the class of the student, and not so much to the municipality, which might indicate the existence of some type of segregation experienced by these students, either at intra-municipality level (by the different schools) or intra-school level (by the different classes).

Nova SBE Working Paper Series No. 638

http://hdl.handle.net/10362/117663

Student segregation across and within schools. The case of the Portuguese public school system

Autores: João Firmino, Luís Catela Nunes, Sílvia de Almeida, Susana Batista

We provide the most comprehensive description of student segregation in the Portuguese public school system to date, a system that exhibits interesting institutional features potentially linked with the student segregation issue (e.g. school catchment areas, course tracking, and almost no central regulations regarding class composition). The analysis uses the entire regular student population enrolled in all public schools of continental Portugal (grades 1 to 12, from 2006/07 to 2016/17). Looking at three segregation dimensions – economic, academic, and immigrant – at both between and within-school levels and using a novel dissimilarity index recently proposed in the literature aimed at better capturing systematic segregation, we find that segregation, on median, is mild, across time, grades, and regions. The most important exception is the case of within-school academic segregation. During upper-secondary schooling, in particular, when students are divided across classes according to own course-tracking decisions, it doubles. Moreover, within-school academic segregation estimates have the largest interquartile ranges, within a given year, grade, or region, pointing to heterogeneity in the way different schools set up classes internally in terms of students’ academic characteristics. Academic and economic segregation are positively associated, at both between and within school levels. The Portuguese segregation insights are also compared to those from other geographies.

Nova SBE Working Paper Series No. 633

http://hdl.handle.net/10362/94327

Teacher quality, alternative hiring policy and student outcomes: Evidence from India

Autores: Ana Balcão Reis, Madhuri Agarwal

Versão mais recente apresentada no XXVII Meeting of the Economics of Education Association (http://2018.economicsofeducation.com/2018/06/26/final-program-available/)

Good students or unexpectedly good students? The impact of teacher expectations on internal score

Autores: Pedro Freitas, Rodrigo Ferreira

Versão mais recente apresentada no XXVII Meeting of the Economics of Education Association (http://2018.economicsofeducation.com/2018/06/26/final-program-available/)

Class composition effects. Evidence from Portugal using panel data

Autor: João Firmino

Versão mais recente apresentada no XXVII Meeting of the Economics of Education Association (http://2018.economicsofeducation.com/2018/06/26/final-program-available/)

Class Composition and student achievement in Portugal

Autores: João Firmino, Luís Catela Nunes, Ana Balcão Reis, Maria do Carmo Seabra

Versão mais recente apresentada no Fórum Estatístico of Direção-Geral de Estatística da Educação e Ciência

In this paper we estimate class composition effects impacting on achievement levels of Portuguese students. Endogeneity between student achievement and student non-random sorting across schools and classes may prevent the correct identification of class composition effects. Using student level cross sectional data of 6th and 9th graders (2011/12 academic year) provided by MISI dataset we contrast a relatively recent estimation procedure in the literature – involving a proper instrument (IV) coupled with School Fixed Effects (SFE) – with usual OLS as means to properly identify the composition effects free of endogeneity bias. Several dimensions of class composition were identified as consistently impacting national exam scores on Portuguese and Mathematics. Namely, the proportion, in a given class, of pupils: 1) under the relevant grade reference age; 2) of low-income households (negative impact) and 3) with home access to internet (positive impact), to mention a few. Many of the effects are statistically significantly asymmetric (e.g. an increasing proportion of students aged at or below the relevant grade reference age in a class seems to affect positively this type of classmate while hurting those aged above it). Non-linear effects are also analysed. In turn, class size yields no significant effect on achievement, while class gender composition uniquely affects boys’ achievement in Portuguese. Given that in the past recent years Portugal has been put under tight public budgetary management it is even more important to identify class compositional effects. Their identification, which this paper contributes to, can provide policy orientations capable of delivering positive increments to student achievement while, at the same time, be budget neutral. Taking the results obtained it seems that optimally allocating students across classes seems more attractive than to increase teacher spending to cut class size.

Nova SBE Working Paper Series No. 624

https://economicsofeducation.com/wp-content/uploads/2020/02/091.pdf

​Value-Added Measurements under High-Teacher Turnover

Autores: Pedro Freitas, Rodrigo Ferreira

Versão mais recente apresentada no International Workshop on Applied Economics of Education

On the Sources and Mechanisms of Human Capital Externalities

Autor: Pedro Freitas

Versão mais recente apresentada no XXVI Meeting of the Economics of Education Association (http://2017.economicsofeducation.com/user/programa2.php?lang=en)

Gender gaps in different grading systems

Autoras: Catarina Angelo, Ana Balcão Reis

This paper analyses the impact of grading practices on the gender gap in student achievement We examine the gender difference in the difference between teacher grading and scores on national exams to test whether there are gender differences associated with different grading systems. We use Portuguese data on 21 subjects across humanities and sciences for the whole population of students taking exams at the end of the 6th, 9th, 11th and 12th grades from 2007 to 2016. Results show that the difference in scores between teacher grading and exams is on average positive for boys and girls, but higher for the latter. This is verified across the whole distribution of exam scores. Thus, our results indicate that a grading system based on exams favors boys while one based on classroom evaluation favors girls.

Nova SBE Working Paper Series No. 625

https://run.unl.pt/bitstream/10362/82728/1/WP625.pdf

(How) do non-cognitive skills programs improve adolescent school achievement? Experimental evidence

Autor: Pedro Martins

Non-cognitive skills programs may be an important policy option to improve the academic outcomes of adolescents. In this paper, we evaluate experimentally the EPIS program, which is based on bi-weekly individual or small-group non-cognitive mediation short meetings with low-performing students. Our RCT estimates, covering nearly 3,000 7th- and 8th-grade students across over 50 schools and a period of two years, indicate that the program increases the probability of progression by 11% to 22%. The e ects are stronger amongst older students, girls, and in language subjects (compared to maths).

Nova SBE Working Paper Series No. 614

https://run.unl.pt/bitstream/10362/82731/1/WP614.pd