สมาคมครูหนุ่มหล่อแห่งประเทศไทย
Group Blog
 
All Blogs
 
When Evaluations Do Not Function as Learning Exercises:


When Evaluations Do Not Function as Learning Exercises:
The 1989-1999 Objective 1 Evaluations in Italy

By

Robert Leonardi, London School of Economics
R.Leonardi@lse.ac.uk

1. Introduction: The low socio-economic performance of Italian southern regions
In 2003 the European Union’s cohesion policy entered its fourteenth year of existence. Since its inception in 1989, the policy has covered a significant part of the EU member states’ territory and population. Starting in 1989, Italy had eight regions in Objective 1 (Abruzzo, Basilicata, Calabria, Campagnia, Molise, Puglia, Sardinia and Sicily). During the second CSF cycle, Abruzzo became Italy’s and the EU’s first region to exist from Objective 1. During the current CSF cycle, 2000-2006, a second Italian region, Molise, has exited from Objective 1, and it is expected that for the fourth CSF cycle, 2007-2013, two more Italian regions, Basilicata and Sardinia, will join their previous two Italian counterparts already mentioned in going into transition out of Objective 1, thereby leaving only half of the previous eight regions still benefiting from Objective 1 programmes.
At first glance the constant and steady stream of Italian regions existing from Objective 1 seems to suggest that the country has had an enviable level of success in conceiving and implementing its Objective 1 programmes. However, if we look at the data in Table 1 comparing the relative performance of the Italian Objective 1 regions vis-à-vis other Mediterranean regions in the European Union, we see that the level of GDP per capita as expressed in PPS has hardly budged during the last two decades and half are relatively worse off than they were in 1980. In absolute terms the Italian Objective 1 regions have improved their levels of well-being from the levels that existed in 1980 but in relative terms the other Northern Mediterranean Objective 1 and non-Objective 1 regions are quickly catching up with their southern Italian counterparts. Why is this the case, and why haven’t the southern Italian regions been able to use the Structural Funds to maintain the gap between Italian and other Mediterranean regions which existed in 1980 and why have Italy’s Objective 1 regions—despite their access to conspicuous Structural Funds financing vis-à-vis their northern counterparts—not even been able to grow at the same rate as other Italian regions.
The thesis of this paper is that part of the reason for this less than brilliant performance levels of southern Italian regions is, in part, due to the inability to make full use of Structural Funds financing to begin a new phase in socio-economic development. Graphs 1 to 8 illustrate the low socio-economic performance of southern Italian regions between 1981 and 2000 vis-à-vis their northern and central Italian counterparts in terms of GDP per capita, investments, and unemployment. The same trend is registered for other socio-economic variables, such as activity rates and employment levels.
The careful review of the Graphs also point out the differentiated level of socio-economic performance between two groups of southern regions: the four regions with the smaller populations (Abruzzo, Basilicata, Molise and Sardinia) and the four larger regions (Calabria, Campania, Puglia and Sicily). The investment data in Graphs 5 and 6 are particularly eloquent in highlighting the significant differences between the smaller vis-à-vis the larger regions.
Another significant difference between small and large southern Italian regions is noticeable in the capacity to spend by the regions (see Table 2) over the three programming cycles of the Structural Funds. During the first period, 1989-93, Basilicata and the other smaller regions had already demonstrated higher levels expenditure capacities that the other southern regions, and this difference through the 19994-99 period and into the new programming cycle.
The question that can be asked relative to the entirety of Italy’s southern regions is “why have they performed so poorly vis-à-vis their Mediterranean counterparts?” and within Italy itself “why have the larger southern regions performed so poorly in comparison to their smaller neighbours?” The thesis of this paper is that the southern Italian regions are at a disadvantage visi-a-vis other Mediterranean and EU Objective 1 regions because prior to the reform of the Structural Funds in 1988 and up to 1992 (i.e., three years into the first CSF cycle) they were the beneficiaries of a national regional policy that basically was a sectoral development policy that did not contain many of the features common today in the EU’s cohesion policy approach. We will argue that this previous experience with a fully developed national regional development policy slowed down the mechanism for adapting to and learning the new approaches implemented by the instituted by the EU’s cohesion policy.

2. Italy’s regional development policy prior to 1988.
Before the initiation of an EU regional development policy, the Italian South was the recipient of an extensive national one that immediately began in the postwar period. In 1950 Italy inaugurated an ambitious regional development policy for its less-developed regions in the South by creating with law 646 the Cassa per opere straordinarie di pubblico interesse nell’Italia meridionale (Casmez) or “Fund for extraordinary projects of public interest in southern Italy”. The primary goal of the Casmez was to reduce the gap in levels of development present in southern Italy vis-à-vis the rest of the country. However, the Casmez was not a fully developed regional development policy. Instead, it was sectorially oriented—that is, it focussed on specific economic sectors as a means of spurring overall socio-economic development. The Casmez began with investments in land reclamation, reforestation, and water supply projects to help modernise agricultural production in the South. These interventions were not grouped by region. The Casmez operated throughout the South as a whole and did not distinguish between one or groups of regions vis-à-vis the whole.
In 1959 industrialization was added as a goal for the operation of the Casmez through the designation of the first industrial growth poles around vertically organised capital intensive facilities in steel manufacturing (Bagnoli and Taranto), petrochemicals (Brindisi and Gela), oil refining (Gela and Milazzo) and automobile manufacturing (Termini Imirese and Termoli). Six years later tourism was defined as an important economic sector in spurring growth. 1965 also saw the initiation of programmes to aid particularly impoverished areas. Twenty-one years later in 1986 the Casmez was substituted by the Agency for the South (Agensud) through law 64 in recognition of the regularity of the national government’s intervention in the South. Unexpectedly, the Agensud was terminated in 1992 when the first Amato government put an end to the extraordinary intervention of the Italian state in the southern regions and integrated all national development programmes for the Mezzogiorno into the Structural Funds programmes—i.e., rather than intervening with national programmes in pursuing a completely separate objectives the Amato government restricted national intervention to the cofinancing of Structural Funds interventions .
With an initial allocation of one thousand two hundred billion lire (600 million euros) for the Casmez (decade of the 1950s) and then 87 thousand billion lire (43 billion euros) for the decade in which the Agensud was supposed to operate from1986 onwards (both allocations were worth an annual GDP shock of 3% for the South), the gap between North and South surprisingly did not change. The gap between the southern regions and their centre-north counterparts remained substantially the same. In 1950 GDP per capita in the North was almost four to 1 (3.81/1) times higher than in the South. Thirty-five years later that ratio was still 3.78/1. the time-span was short term (single year financial allocations that were stretched out over a ten-year period), no instruments of verification were instituted other than those normally used by the national administration to report its expenditures, no regional participation war foreseen, nor was there a role for social partners. The evaluation of programmatic outputs and outcomes was not a normal part of the policy making or implementation cycle.
As illustrated in Figures 1 and 2 the differences between the Cassa per il Mezzogiorno approach to policy making and implementation was strikingly different from the one that was implemented with the initiation of the EU’s cohesion policy for Objective 1 regions. Thus, Italy’s southern regions experienced a three-year overlap between the two policies from the 1989 to 1992 when the Italian national regional development policy for the South was suddenly terminated and all national support for regional development was subsequently subsumed into the country’s participation in the co-financing of Structural Funds expenditures. What is of concern here is: how was the shift between one approach and the other managed? Were administrators given access to training courses on the new approach and were stages in Brussels or experimental programmes undertaken to transfer the knowledge of how to use the new principles, rules and procedures on behalf of the administrative personnel involved in the implementation of the programmes?
The answer to these questions is simply that no systematic training programme was undertaken to help administrative personnel to make the immediate transition during the first CSF cycle, 1989-93, from the existing Fund for the South approach to the formulation and implementation of development programmes and that instituted by the EU’s cohesion policy and that during the second CSF cycle, 1994-99, Italy did begin a whole series of training programmes at the national, regional and local levels on how to participate in Structural Funds programmes, but these training programmes (entitled PASS or Sub-Programme for the Formation of Personnel of the Public Administration) were purely voluntary in nature (administrators could participate in the training programmes if they wanted to) but no incentives (e.g., increase in pay or administrative status) were provided by the funding authority (Department for Public Administration of the Presidency of the Council of Ministers) for those who participated in the programmes. Financial incentives were strong for those who offered the courses (i.e., consulting firms that had won the public bids to set-up the training courses) rather than for those participating in the courses that came from the public administration. Thus, the lack of a coherent programme to prepare the national, regional and local administrators in the new rules and procedures led to an effective paralysis as observed by the ex-post evaluation of the Objective 1 programmes in Italy. Given the haphazard transition from the old national regional policy and the new cohesion policy, “these problems have paralysed a local bureaucracy which is still tied to the outdated organization and procedural rules”. Thus, the winding down of the national regional policy and the implementation of the new cohesion policy required not only a shift in rules and procedures but also a shift of administrative responsibilities--from the national to the regional-- for the operational programmes.
The implications of these two significant changes can be conceptualised by the contents of Figure 2 which describes the conceptual loop associated with the decision making and implementation processes associated with the previous national regional policy and the Community Support Framework (Figure 3). The major differences between the two conceptual frameworks are: 1. the cohesion policy is multi-level in its inputs and management while the national regional policy involved only one actor—the nation-state; 2. the decisions on what to invest in were never part of a comprehensive programme but were rather the result of piecemeal decisions by the national authorities; 3. regions did not have a role in conceiving or carryout out development policies. The responsibility resided exclusively with the national authorities; 4. Evaluations were not part of the decision-making or implementation process. All controls were internal to the national government; and 5. The interventions were conceived as one-off or extraordinary actions rather than as part of a normal programme of interventions.
The method used to implement cohesion policy within the EU represents an approach that not only values but requires the participation of different socio-economic forces as well as different levels of government in both the policy making and policy implementation phases of the process. As indicated in Figure 3 the formulation and implementation of the EU’s cohesion policy is based not only on a system of a vertical multi-level governance but also on a horizontal form of governance capable of involving a multiplicity of political/administrative as well as socio-economic actors in the process.
On the left-hand side of Figure 3 we find the multiple institutional actors involved in the multi-level aspects of governance used to formulate and carry out the cohesion policies. In Italy where sub-national regional governmental actors have official responsibilities in economic development, vocational education, transport, agriculture, tourism, and other socio-economic sectors, the sub-national level actively participates in the formulation of its own operational programs for Objective 1. The negotiations are carried out through formal meetings, seminars, roundtables and other types of encounters designed to achieve agreement between the national and regional levels on the planning and financial commitments that provide the legal basis in carrying out the cohesion policy in Objective 1 areas.
Another aspect of the planning system created by the cohesion policy that is reflected in Figure 3 is the conceptual separation of the phase of decision-making from implementation. The decision-making phase begins with the formulation of the development programmes for the Objective 1 region, but before policies can be formulated an analysis needs to take place of the socio-economic context based on consultations with representatives of civil society and organized interests to identify the strengths and weakness of the territory and the priorities that need to underpin the future development programme. Such an evaluation was never carried out in an explicit manner to feed into the policy formulation process before 1989. Studies were instead carried out to document the level of need and formulate possible solutions to the South’s socio-economic problems, but the actual policies that were implemented were not explicitly tied to the analysis of socio-economic conditions as is the case now.
In Italy the national government continues to be responsible for the formulation of the initial development plan that is then transformed into the Community Support Framework in negotiations with DG Regio. Regional governments are subsequently responsible for the formulation of their individual operational programmes (ROPs). The ROPs are subsequently finalized into complementary planning documents that have attempted during the third CSF cycle to define into stricter terms the actual nature of the “measures” and “actions” that will be used to operationalise the programme objectives.
What comes out of the SWOT analysis is the selection of the development priorities that will be technically translated into specific “axes” or sectors that can range from aid to industry, investment in infrastructure and support for local development schemes. The choice of axes provides the basis for preparing the planning document for an integrated approach to the development of the regions under consideration. The planning document also provides for a multi-annual financial package whose overall socio-economic impact is estimated through the conduct of an empirical ex-ante evaluation.
In Italy the operational programmes are divided between those formulated and implemented by the regions and those formulated and implemented by the national government. During the first two cycles of the cohesion policy (1989-93 and 1994-99) the division between national and regional operating programmes was approximately 50%-50%. During the current cycle, 2000-2006, the majority of funds (70%) have been shifted down to the regional level.
The system of governance of the cohesion policy is based on a number of management principles that are quite different from those that operated during the national regional development policy. These include the explicit interaction between measures and actions contained in the programme in order to increase the synergy and the co-financing of measures by different institutional actors (e.g., the EU, national and regional governments, and the private sector) to provide additional funding and increase the impact of the investments.
With regard to implementation, each operational programme incorporates rules and procedures on the management of the funds and the choice of beneficiaries; the control and oversight on the implementation of the programmes, measures and projects; the monitoring and reporting on the progress of programmes over time; on the evaluation of the impacts of the projects on local and regional socio-economic conditions; and, finally, on the certification of the completion of the projects and the provision of final payment.
Within this list of procedures, evaluation has become one of the fundamental activities associated with the implementation of EU cohesion policies. In addition to the ex-ante evaluation of the expected socio-economic impacts to be produced by the programme priorities and expenditures, the current (2000-2006) cohesion policy cycle foresees an obligatory evaluation of the impact of the programme half-way through the planning period (i.e., mid-term evaluation) in order to ascertain whether the initial goals of the programmes are being met and implementation is taking place according to schedule and the expected results are being achieved. If not, modifications can be undertaken to make the programmes and their specific measures more effective and easier to implement. Those programmes which have met their spending and impact targets will be able to benefit from a 4% increase in financial allocations from the Commission.
Finally, once the programmes have been brought to completion within the timeframe provided, an ex-post evaluation is conducted to measure the overall impact in both quantitative and qualitative terms. This ex-post evaluation is important not only for the learning process concerning what was and was not successful in the programme for the benefit of the authorities at the regional and national levels responsible for the implementation of the programmes, but it also is important for the European level institutions, such as the Commission, the Court of Accounts and Parliament. The Commission is responsible to the member states and to the European Parliament to provide period reports on the outputs and outcomes of the programmes while the Court of Account has the obligation to report to the European Parliament that the programme expenditures have taken place according to the strict procedures contained in the regulations governing Community expenditures. The ex-post evaluation are also important to everyone concerned in providing data and insights into the changes that are necessary in preparing the regulations and rules for subsequent cohesion policy cycles.
Accordingly, in addition to an ex-post evaluation of the efficient and effectiveness of the programmes in achieving their objectives, a full reporting of the expenditures is undertaken by the regions and national governments to document all that has been spent on the programmes and provide for the restitution of all money not spent within the allotted provided of time. A detailed independent evaluation of the expenditures is also conducted by the national and EU Court of Auditors on a sample representing 5-10% of the overall budget to make certain that what was spent was carried out according to the rules and regulations set out by the European Commission and the national authorities. All of this elaborate system of oversight and cross-checks was completely absent in the follow-up to the Italian national approach to regional policy in the South prior to 1988.
Given that the EU’s cohesion policy was not intended from the beginning as a one-off intervention but, rather, as part of a continuous series of socio-economic “shocks” designed to stimulate the adaptation and growth of the local and regional economy in relation to the challenges of the Single Market, the phases of the decision making and implementation process described in Figure 3 can be conceived as a series of interactive learning processes. Learning or the modification of existing procedures allows for changes to be made if initially there was a less than perfect understanding of the local socio-economic context, a short-sighted choice of projects or priorities was made, or if a faulty implementation process was chosen. The repetition of the planning process over a number of cycles and the formalization of evaluation procedures permit corrections to be made to all phases of decision-making and implementation. Evaluation of programme implementation has been introduced into the procedure through the formal request of annual evaluations as well as the intermediate and ex-post evaluations that have become so crucial in correcting priorities, projects, and expenditures at mid-term in the process and once the programme has been completed in order to inform the subsequent planning cycle.
In Italy six out of the eight original Objective 1 regions have continued to receive full funding over all three planning periods. Thus, those that have remained in the programme should have had the opportunity to learn from their previous mistakes and profit from their successes in making subsequent formulations of their programmes more responsive to local socio-economic conditions and more effective in mobilizing local and regional socio-economic forces in achieving sustainable endogenous development. The repetition of the process also permits “institutional learning” to take place at all three levels. In the first place, in evaluating the results of the programmes the Commission can achieve a better understanding of how the rules and regulations in the use of the Funds can be improved to increase their impact, transparency and efficiency. The national governments, on their part, are in a position to improve their level of coordination in guaranteeing a smoother implementation of the programmes. Finally, the regions can learn what works and what does not work in spurring socio-economic development and thereby improve their overall selection of projects to be financed. In the final analysis, it is the projects selected that are in a position to spur development and encourage a multiplier effect from the private sector.
The question remains: how can we be certain that effective learning takes place and what are the important mechanisms that are necessary to ensure that the maximum amount of learning is carried out by the administrative structures responsible for decision-making and implementation of the Structural Funds? According to our perspective, the maximization of learning takes place when the returns or “internal benefits” to the responsible administrative structure outweigh the costs incurred. We will illustrate what we mean by costs and benefits in the discussion of the constituent parts of Figure 4.
As we have already mention above, the innovative nature of the reform of the Structural Funds and the need to wind down the old national regional policy for the South presented significant problems of learning and acceptance on the part of the national and regional institutions. For the national institutions the new rules and procedures represented a significant loss of power and centrality in the management of the development process while for the regions the new policy transferred to them a responsibility that previously they had not assumed. Both set of changes generated a certain amount of bureaucratic resistance.
The formulation of the 1989-93 programme followed a top-down approach that was controlled by the national government. In formulating the first CSF in 1989 no consultations with the regions or the private sector took place. The Italian CSF was not accompanied by an ex-ante evaluation of the expected outcomes, nor did it have a list of the empirical indicators that could be used during the 1989-93 period to measure progress in the implementation of the programmes. No mid-term evaluation was foreseen, and it was difficult to close the books on programme expenditures that were first postponed until 1996 and then extended until the end of 1997. The holes in the control and reporting procedures made it difficult to verify expenditures at the regional or national levels. The provisions requiring independent evaluations or outside technical assistance in implementing the programmes were not applied during the first period, and most member states including Italy did not take advantage of this possibility.
An additional problem associated with the first programming cycle was the exclusive role allocated to the public sector in the financing of the development programmes. In the 1989-1993 period all national co-financing of the programmes came from the public sector. The funding sources identified by the CSF as coming from the private sector were in reality government agencies or corporations such as the railroads, the national telephone company, and other public sector corporations.
The implementation of the 1994-1999 programmes continued to suffer recurring problems. In the Italian case the government was faced with the additional need to reduce government spending in order to satisfy the Maastricht convergence criteria in relation to its annual budget deficit. But there were also examples of significant learning taking place. In Italy the national government instituted a more formal reporting procedure so that the Ministry of the Treasury, responsible for the reporting of Structural Fund expenditures, could provide semesterly reports on what was being spent at the national and regional levels on the cohesion policy.
A basic shortcoming of the second CSF cycle continued to be the lack of a systematic ex-ante evaluation of the programme capable of providing unambiguous empirical targets that then could be verified in the intermediate and ex-post evaluations. Without an explicitly empirical ex-ante evaluation it was difficult to determine whether, in the final analysis, the programme objectives were in the process of being achieved as verified by the intermediate evaluation or had been achieved once the ex-post evaluation was carried out. This problem was repeated at the individual project level where the empirical objectives of the interventions were not always stipulated beforehand and projects were in a number of occasions selected to respond to political commitments rather than for their added value and impact on the local socio-economic structure.
All in all, during the second cohesion policy cycle significant improvements were made in accounting for the expenditure of funds but less progress was achieved in making sure that the implementation was conducted in an efficient and effective manner. Such a guarantee could only be provided if the rules required a systematic and thorough monitoring of the investment projects chosen from the very beginning of the programmes and by making sure that the final evaluation of the programmes provides a detailed analysis of how a sample of projects were conceived, managed and completed.
Further changes in procedures were undertaken in preparing for the third cycle in the EU’s cohesion policy. The formulation of Regulation 1260 was completed in June 1999 and introduced some significant changes in the administration of Structural Fund spending. In addition to the new Regulation, the member states and regions had accumulated eleven years of experience in managing cohesion policies, and therefore, had solidified a series of good practices and had made a clear commitment to transparency. The combination of the new Regulation and the accumulated experience served to create an acquis or model of good governance for the use of the Structural Funds.
The 1999 regulations introduced two innovation that would prove to have significant and widespread effects on the implementation of the Structural Funds by the national government and regions. The first was the introduction of the n+2 principle for transfers. According to this new rule the Commission transferred 7% of the total budget once the operational programme was adopted. Within two years the region or member state had to demonstrate that the amount was spent or otherwise there would be an automatic clawing back of the sum by the Commission. The n+2 rule provided an incentive on the part of the managing authorities of the operational programmes to speed up the initiation of the programmes and not to procrastinate as had been the case with the two previous cycles.
The second major innovation was the introduction of the 4% reserve fund for the Objective 1 programmes and linking the disbursement of this Community reserve to the performance of the regions and national government in using the previous allocations of funds as documented by the “intermediate” or mid-term evaluation. Accordingly, the requirements linked to the disbursement of the performance reserve served to focus attention on the quality of the programme evaluation. In the 2000-2006 period evaluation stopped being only a formal requirement on the part of the Commission to satisfy the formal provisions of the regulations. Instead, the intermediate evaluation became the central operation in increasing programme budgets and providing a rational basis for reconfiguring some of the interventions during the second half of the programming cycle.







Create Date : 06 กุมภาพันธ์ 2554
Last Update : 6 กุมภาพันธ์ 2554 9:00:13 น. 0 comments
Counter : 1066 Pageviews.
Share

ชื่อ : * blog นี้ comment ได้เฉพาะสมาชิก
Comment :
  *ส่วน comment ไม่สามารถใช้ javascript และ style sheet
 

hangclub
Location :


[Profile ทั้งหมด]

ฝากข้อความหลังไมค์
Rss Feed
Smember
ผู้ติดตามบล็อก : 29 คน [?]

Friends' blogs
[Add hangclub's blog to your web]
Links
 
MY VIP Friend

 Pantip.com | PantipMarket.com | Pantown.com | © 2004 BlogGang.com allrights reserved.