THE THIRD SUSTAINABLE LIVELIHOODS PROJECT (P125232):
CONSULTANCY SERVICE FOR CONDUCTING THE PROJECT’S MID-TERM REVIEW SURVEY
TERMS OF REFERENCE
I. SUMMARY
As part of monitoring and evaluation of the Third Sustainable Livelihood Project, the Ministry of Finance of Mongolia requires a consultancy to conduct the project’s Mid-term Review (MTR) survey. The firm contracted to implement the MTR survey will assess whether the project is on track in terms of achieving the set development objectives, identify areas of improvement, and provide recommendations for the remaining two years of the project. The MTR should lay the ground for the final evaluation of the SLP3 in 2022.
The MTR must ensure consistency and comparability with the methodology and process followed for the collection and analysis of the baseline study for the project conduced in 2017 to allow appropriate analysis and comparison. Under the MTR, the contractor will conduct two types of surveys, namely (i) a Public Opinion Survey of soum citizens and (ii) an LDF-related Human Resources and Institutional Capacity Survey. Data and information should be obtained using a multi-stage stratified random sample and cover all or some of the 21 aimags and sampled soums from those aimags.
The contractor is expected to be on board by early September 2020, so the field data collection to take place in September and October 2020 and the data analysis to last for another month after the data collection. The contractor’s work should be completed in approximately four (4) months.
The contractor should possess extensive experience in conducting quantitative and qualitative surveys, data collection and management, as well as project evaluation and organizational development. The firm should also have experience in partnering with local government officials and local communities and solid understanding of the budget law, local budgeting and budget execution processes in Mongolia.
II. BACKROUND
2.1 The Third Sustainable Livelihoods Project
The Government of Mongolia, represented by the Ministry of Finance (MOF), is implementing the Third Sustainable Livelihoods Project (SLP3), which is funded by the World Bank and the Swiss Agency for Development and Cooperation. The development objective of the project is “to improve governance and community participation for the planning and delivery of priority investment in rural areas of Mongolia.”
Within the broader development objective of the project, the SLP3 aims to build capacity and improve information sharing to the citizens on the Local Development Fund (LDF) and to help enhance capabilities at the local aimag and soum levels of managing the LDF. The LDF provides funds to soums to support capital investment in local public infrastructure and services. The Integrated Budget Law specifies eligible areas for investment such as pasture management-related investments, which aim to enhance risk management of rural soums. Also, the Law stipulates that local governments must utilize LDF allocations in accordance with priorities identified by the citizens through a robust of community participation process. Inclusion of community participation in the local budget preparation and execution process is a major step forward empowerment of citizens and a major reform of the citizen-government relationship.
Under the capacity building component, the revised LDF regulation has been approved by the Minister of Finance on Sep 12, 2018 and registered by the Ministry of Justice and Internal Affairs on Oct 12, 2018 with the following major changes in substance:
- LDF subprojects to be approved shall always be supported and prioritized by local citizens;
- Subprojects financed by LDF shall be consistent with the long-term, medium-term and short-term development policy documents specified in Article 6 of the Law on Development Policy and Planning;
- General definition of eligible subprojects was clarified; and
- The roles and responsibilities of each stakeholder in the LDF processes were better defined in the Regulation.
The SLP3 comprises the following components:
(i) Capacity building for Local Governance and Livelihoods (Component 1);
(ii) Good Governance Performance Based Support Program (Component 2); and
(iii) Project Management, Monitoring and Evaluation (Component 3).
2.2. Project Management, Monitoring and Evaluation
The objective of the Component 3 is to measure progress, results and achievements of the SLP3 and the LDF in general, as well as significant outcomes of the SLP3 and the LDF for the intended beneficiaries. An important purpose of M&E is to provide valuable information and feedback on the project, upon which the management builds its decisions and corrective measures on further activities. It also aims to showcase how well the project has turned out for beneficiaries and all others involved. The Project M&E is guided by the Results Framework, which specifies the results indicators, their baseline and target values, frequency of monitoring, sources of data, and instruments through which data is collected.
The Result Framework for SLP3 was revised in September 2017 to take into account lessons learned from the initial years of implementation. The 2017 revisions aimed at strengthening the project's focus on LDF management. The current list of project development objective (PDO) indicators include:
1) Percentage of soums' citizens surveyed reporting that LDF financed investments reflect their priority needs, disaggregated by gender.
This will show citizens’ perceptions of improvements in their access to public services, the responsiveness and transparency of local investment financing supported by the LDF.
2) Average soums' Annual Performance Assessment score:
This will measure the percentage increase in average Annual Performance Assessment (APA) scores of soums: measuring how far soums progress from year to year in their compliance with the same sets of APA performance measures.
3) Participation: Percentage of bagh households participating in public bagh meetings:
Reflecting the rate of citizen participation in the discussion of budget priorities shows how the project helped to meet the IBL provision of community participation in budget preparation and execution, a major reform of the citizen-government relationship.
The baseline survey to collect baseline data and other initial information for the SLP3 was conducted by an independent research firm in 2018. Two main objectives were focused during the baseline survey processes, including: (1) establishing baseline data against the Project Development Objective and Intermediate Result Indicators; and (2) identifying and recommend appropriate results of key indicators that serve as a baseline to compare the progress and success of the project in relation to its relevance, effectiveness and efficiency. The major methodology and methods used and applied in the baseline survey included the Public Opinion Survey and Review on LDF operation.
Since the project reached its mid-term, the MOF intends to undertake a review of the progress towards the intended results of the project, with a view to informing further areas for improvement and acceleration in the remaining years of the Project implementation. Therefore, the MTR report will be an important document to serve as a reference and guide the further actions of MoF/SLP3 team and the project stakeholders.
2.3 Purpose and objectives of the MTR
The main purpose of the Mid Term Review (MTR) is to evaluate the performance of the SLP3 against the desired results as articulated in its Results Framework.
Specific objectives of the MTR include:
1. Assess SLP3 progress to date against workplan targets for the realization of intermediate and long term results;
2. Highlight the most significant achievements that the SLP3 has made so far, and identify programme components and/or mechanisms that have led to the observed results;
3. Identify key success stories, emerging good practices and lessons learnt through the implementation of SLP3 to date and develop recommendations for the further refinement of programme.
III. METHODOLOGY AND APPROACH
The MTR must follow the baseline methodology and process and ensure comparability to the baseline survey, although a reduced scope might be applied for the mid-term review, depending on the methodological approach that will be defined in the mid-term inception report. Two main surveys to be carried out under the MTR are as follows:
1. The Public Opinion Survey
2. The LDF related Human Resources and Institutional Capacity Survey
The detailed methodology will be designed by the contractor during the inception phase.
3.1 Sampling and data collection
The survey will have multiple target populations – soum citizens for the “Public Opinion Survey” (POS), and staff who are relevant to LDF implementation, in particular to the sub-projects that are financed by LDF, for the “LDF related Human Resources and Institutional Capacity Survey”.
The Public Opinion Survey (POS):
The below is a list of the minimum survey interest areas, based on which the contractor shall develop a survey instrument (a questionnaire):
Citizen awareness on LDF: familiarity with LDF and overall transparency and disclosure to the public, whether the citizens are informed about the soum development plan before it is translated into project proposals, and reports of budget execution and performances, and whether the citizens acknowledge their rights or obligations regarding local budget planning and execution.
Citizen Participation: Participation in planning and preparation of sound investment proposals, identification of budget priorities, budget preparation and adoption, budget execution and procurement, reporting, monitoring and oversight, and sustainable asset management. More specifically whether the LDF financed projects meet the local needs, whether the citizens are available and interested to conduct M&E on project activities, whether the citizens’ participation in M&E would accepted by the officials in local government, etc.
Sampling unit: The POS will obtain representative information on target population as best as possible using a nationwide random, multi-stage stratified sample. The target population of the POS would be soum citizens/adults, so that the survey will be taken from randomly selected individuals who live in the sampled soums. However, sampling individuals is not practically feasible. Thus, we recommend a household as a sampling unit and an individual from a household to be selected with special selection procedure such as the last birthday technique, Kish method, etc.
Sample size: Total sample size for the POS is expected to be 1,100 considering potential drops due to nonresponse, data collection quality, etc.
The LDF related Human Resources and Institutional Capacity Survey:
The assessment of performance and abilities of the institutions related to the LDF and of capacity of the personnel running the LDF implementation in selected soums will be undertaken using specially developed instruments.
Sampling unit: The sampling unit for this survey is soum but the actual interviewees for the review are to be civil servants responsible for the LDF working in soums and baghs. The qualified survey respondents may include Head of the Citizen Representatives’ Khural or the Secretary of the Khural, Soum Governor, Head of the Soum Administration, Head of Finance of the Soum or Government staff from the Finance Department, Soum administration staff who is responsible for the LDF, Bagh Governor, etc.)
Sample size:. The sample size and sampling methodology shall be proposed by the contractor. Sampling methodology and overall recruitment rate will be one of criteria for contractor selection.
The Contractor will develop a survey implementation plan for collecting, managing, and cleaning survey data. This plan, including documentation describing how it plans to conduct the pre-testing, fieldwork, and data cleaning in a way that minimizes costs and maximizes accuracy must be submitted to the PIU for comments and acceptance before survey implementation begins.
3.2 Validation mechanisms
The team should use a variety of methods to ensure the validity of the data collected and the proposal should provide clear description of these methods.
IV. MANAGEMENT AND CONDUCT OF MTR
4.1 The SLP3 Technical Team
The Ministry of Finance (MOF) will be a contracting party with the Contractor, whereas day-to-day management and supervision will be provided by the Project Implementation Unit (PIU) of the SLP3. The Contractor will be reporting to the PIU. The technical team at the SLP3 PIU will have the following responsibilities:
- In general, manage the contract with the Contractor;
- Review and provide comments on the operational work plan and timetable and review all the documentation provided by the Contractor;
- Consult with the Contractor on the list of key concepts, definitions such as the sampling plan, survey content, coverage, and sustainability;
- Review with the Contractor the pertinence of the themes and variables used in the study for the program. Similarly, review the precision and scope of the questions used to gather information and data;
- Review instruments, tables, formats, questionnaires, etc. to be used to collect data in communities;
- Conduct random spot checks by its staff or by hiring a Data Quality Reviewer to ensure the reliability and the accuracy of data collected by the Contractor, including verification of the quality of the data collected by the Contractor; and
- Provide the Contractor with information on project activities and other inputs as needed and including the following:
o Introductory session on the LDF by the MOF and the PIU;
o The Project Results Framework and materials worth sharing for the purposes of the MTR (Vol 2, Vol.3, Vol 4 and SLP3 Result Reports)
o Previous survey instruments, methodology and results conducted by consultants of the SLP3; and
o APA criteria and manual, and preliminary/final results.
4.2 The Ministry of Finance (MOF)
As a contracting body and an implementing agency of the Project, the MOF will be reviewing and accepting main deliverables including the work plan, survey instruments and reports. The duration of report review by the MOF will be at least 2 weeks after submission.
4.3 The World Bank
As the financier and oversight body of the Project, the World Bank will provide comments on the draft deliverables and ensure that the contracting process and contract management are consistent with the relevant procurement guidelines of the World Bank.
4.4 Resources Available to the MTR team
The Contractor will have available the following documents:
At the project level:
- The Financing Agreement, the Project Implementation Manual (PIM), the Results Framework, etc.;
- Annual work plan/procurement plan and semi-annual and annual reports for 2016-2019;
- Baseline report;
- Reports on Annual Performance Assessment (APA) of the LDF conducted by the SLP3 (2017-2019);
- Special survey reports (technical audit reports, social study reports, etc); and
- Relevant project staff as well as local project officers (LPOs) will be available for interviews remotely or in person depending on the methodology employed by the contractor.
In addition, the PIU/MOF will brief the contractor on the LDF at the beginning of the assignment. Specific contents and research questions for each survey are to be developed in detail. All the survey instruments and strategy shall be accepted by the PIU before the implementation/fieldwork.
V. DELIVERABLES AND OUTPUTS
The contractor will be required to deliver the following key deliverables in Mongolian and English.
1) Inception report: containing the MTR framework, detailed sampling methodology, work plan and logistical arrangements, such as training of the interviewers, etc.
2) Debriefing of preliminary findings: The full team will debrief SLP3 PIU team, Implementing Partners, MoF, WB and SDC team on their findings, conclusions and recommendations using a PowerPoint presentation and any other briefing materials as required. Project partners will provide feedback during the briefing meeting.
3) MTR report (draft and final) to be structured as follows:
o Executive Summary
o Introduction
o MTR Objectives and Methodology
o Data analysis and findings of MTR
The Public Opinion Survey
The LDF related Human Resources and Institutional Capacity Survey
Data table showing progress against the project development objective and intermediate indicators
o Conclusions and Recommendations
o Reference
o Annexes (detailed data to supplement the overall summary evaluation report, including interviewee list, details of the data collection instruments, key documents reviewed, etc.)
4) Soft copy of the final and full report with all attachments accompanied by the complete and cleaned data and syntax with variables description and analysis.
Each deliverable should be first submitted in a draft form to the PIU. Based on the results of discussion, the Contractor will make the necessary changes and then will submit a final deliverable. Unless otherwise noted, the PIU will provide comments/feedback within 10 (ten) business days of receipt of deliverables, and the Contractor shall provide a revised deliverable within 7 (seven) business days after receiving feedback. This process will be repeated as required by the PIU or otherwise the deliverable shall be considered accepted.
VI. CONSULTING FIRM QUALIFICATION/SELECTION CRITERIA
This section details the key traits that the PIU has identified as important for the successful completion of activities outlined in this TOR. The contracting firm should ideally possess the following characteristics:
- An understanding of the good governance and public financial management, especially budget law, local budgeting and budget execution processes in Mongolia.
- Extensive experience with survey data collection and management. Specifically, the firm must have conducted at least 5 nationwide surveys independently and successfully for programs and projects of international donors and government agencies. Past survey reports and reference contacts should be submitted as evidence;
- Demonstrated experience in producing high quality, credible evaluations or reviews and projects on M&E (examples required);
- The firm shall be operating as a legally and financially autonomous firm in the consulting business for at least 5 years. Audited financial statements for the last three years should be attached;
- The availability of human resources and the ability to rapidly mobilize and coordinate a number of qualified staff in multiple locations at the same time;
- The firm should have enough equipment and office space for data entering and processing and capability to enter and process large amounts of data in a limited time period;
- The firm should comply with international standards and professional ethic requirements of public survey;
- Experience in communication and cooperation with local stakeholders including local government officials; and
- Demonstrated experience in dealing with international development agencies.
Following characteristics will bring advantages if they are possessed:
- If the firm has experience in conducting survey in areas of public perception, community participation, local governance and local administration operations will have advantage; and
- If CSPro , SPSS, or STATA is proposed as a data collection, entry and analyzing technology, the contracting firm shall have the same experience on larger surveys. Such advanced technology that increases survey quality will be preferred.
Composition of the MTR Team
The MTR team is expected to consist of a team leader, a sociologist – researcher, a database manager, field supervisors and enumerators, and a local service delivery expert. The contractor may propose an alternative team structure, which would then need to be justified.
Team Leader
The Team Leader will be wholly responsible for managing all deliverables and the MTR process. The responsibility for training on survey instruments and management of the enumerators will solely lie with the Team Leader. It will also be the Team Leader’s role to effectively organize the enumerators into teams, with supervisors, and ensure that each enumerator collects data uniformly. SLP3 PIU team may provide support to the team leader as needed, but the overall logistics management of office space, training venues, transportation, etc, will be the task of the team leader.
Duties and responsibilities:
1) Guide the data collection effort and oversee its effective implementation.
2) Design and schedule all surveys.
3) Oversee technical aspects including sampling, interviewer training, data collection, data entry, and variable aggregation and descriptive reports.
4) Participate directly in the oversight of fieldwork, through site visits, review of progress and review of primary data.
5) Maintain relationships with the firm, the PIU and other relevant data-collection entities.
6) Manage budgets and expenses.
7) Prepare and submit reports according to the agreed-upon timeline.
8) Ensure that appropriate resources are made available and managed in order to achieve the objectives of the contract.
9) Ensure that implementation of Contractor activities are in accordance with WB and GoM policies and procedures.
Qualifications:
1) A minimum of a masters’ degree in economics and/or statistics, sociology or equivalent experience.
2) Experience in applying qualitative and quantitative evaluation methods, with a minimum of 5 years of professional experience in programme and project’s evaluation of relevance to policy making demonstrating a strong record in designing and conducting/leading evaluations.
3) Demonstrated ability to lead and manage a multi-disciplinary team, and to achieve specific and measurable results.
4) Ability to deal effectively with stakeholders at all levels, including government staff at soum, aimag, and national level.
5) Proven experience in developing reports to international organizations and project funded by them.
6) Fluency in Mongolian and English, both written and oral.
Sociologist – Researcher
Duties and responsibilities:
1) Assist the team leader in guiding the data collection efforts.
2) Develop survey instruments.
3) Conduct pre-testing and manage focus groups.
4) Direct the interviewers to gather information and carry out the fieldwork.
5) Provide technical assistance in sampling, interviewer training, data collection, data entry and data cleaning, and variable aggregation and descriptive reports.
6) Analyze, evaluate and interpret data.
7) Prepare and report findings and disseminate results.
8) Represent the team leader in his/her absence.
Qualifications:
1) A masters’ degree in sociology and/or statistics, or equivalent experience.
2) At least 5 years of experience in public opinion poll surveys.
3) Demonstrated ability to lead and communicate effectively.
4) Fluency in Mongolian and English, both written and oral.
Database Manager
Duties and responsibilities:
1) Develop data entry systems.
2) Supervise the data entry process.
3) Ensure data quality through entry checks and logic checks, etc.
4) Prepare documentation of survey database including metadata.
5) Support the Team leader and the Researcher in terms data processing.
Qualifications:
1) A university degree in a relevant subject or equivalent experience.
2) Demonstrated experience in database design, entry systems, data consolidation and quality control, preferably involving survey data.
3) Excellent written and verbal communication skills in Mongolian.
Field Supervisor(s)
Duties and responsibilities:
1) Visit staff in the field to oversee and monitor their actions and to assure that they are proceeding with questionnaire collection according to the guidelines established by the PIU.
2) Coordinate lodging and travel arrangement for all staff.
3) Arrange for all completed tests and questionnaires to be filed and for copies to be delivered to the data entry team.
4) Coordinate between field staff and data entry staff to assure timely completion of deliverables.
Qualifications:
1) A masters’ degree in sociology and/or statistics, or equivalent experience.
2) Demonstrated ability to lead and manage a field team, and to achieve specific and measurable results.
3) Demonstrated experience managing similar projects in Mongolia with a focus on data collection.
4) Understanding of qualitative and quantitative research methodology.
Local Service Delivery Expert
Duties and responsibilities:
1) Consult the survey team in terms of local service delivery, specially in the context of the LDF related Human Resources and Institutional Capacity Survey (through engaging in a desk review, survey instrument design, data analysis and reporting).
2) Provide technical support to the review of the practical implementation of the revised LDF regulation.
Qualifications:
1) Advanced degree in a relevant field: Public Administration, Public finance, Economics, Legal and other relevant fields.
2) At least 5 years of professional experience in Public Financial Management, especially budget, public finance or procurement-related fields.
3) A strong background in local government procedures.
4) A good understanding of the Government of Mongolia’s local service delivery and LDF policies and procedures.
VII. DURATION OF CONTRACT AND SCHEDULE OF PAYMENTS
The MTR is expected to commence in September 2020 with an inception phase followed by intensive data collection (desk review, interviews, and surveys), analysis and report writing. A debriefing/data validation meeting for presentation and discussion of preliminary findings should be conducted in mid November 2020, and the final revised MTR report should be delivered by end December 2020. Proposed total workdays for the contract is 70 workdays.
An estimated timetable for submission of deliverables is provided in Annex-1. The schedule outlined in this timeline is negotiable depending on the Contractor’s ability to mobilize staff and develop the needed survey tools. Weekly progress reports shall be provided by the Contractor to the PIU, which is not illustrated in the table in Annex 1. Adjustments to the schedule could be made after the date when the contract is signed, when mutually agreed upon amendments to the comprehensive work plan and delivery schedule are deemed necessary.
Payment of fees will be based on the delivery of outputs, as follows:
• Upon satisfactory contribution to the inception report: 20%
• Upon satisfactory contribution to the draft final report: 50%
• Upon satisfactory contribution to the final evaluation report: 30%
VIII. TECHNICAL APPLICATION/PROPOSAL
The technical proposal must include:
• A signed cover letter responding to the call clearly identifying the applicant (by name, title, organization), primary contact (by mailing address, telephone number and email address).
• Proposal of no more than 10 pages written in Mongolian. The proposal should clearly demonstrate a solid understanding of the TOR and ability to deliver the requirements covering, among others, the following:
o MTR Approach, Methodology & Tools, and proposed Activities and indicative timelines.
o Applicant’s suitability for the assignment including institutional capacity and expertise (including at least three examples of past relevant experience), team composition if applicable with clear indication of the role of each team member and CVs and short biographies of key team members highlighting their previous similar work.
The financial proposal must include:
The financial proposal/budget must detail the applicant’s anticipated costs in MNT broken down by individual line items. It must include budget notes or justifications to explain in detail the basis for how individual line item costs were derived and rates used for the calculations. While there is no page limit for the financial proposal, but applicants are encouraged to be as concise as possible while still providing the necessary details.
Note: Shortlisted applicants will be contacted for a face-to-face interview and presentation of their proposal.
