THE THIRD SUSTAINABLE LIVELIHOODS PROJECT (P125232):
CONSULTANCY SERVICE FOR CONDUCTING THE PROJECT END-LINE SURVEY
TERMS OF REFERENCE
I. SUMMARY
As part of the monitoring and evaluation of the Third Sustainable Livelihood (SLP-3) Project, the Ministry of Finance of Mongolia requires a consultancy service to conduct the project End-line survey. The firm contracted to conduct the End-line survey will focus on the project impact in the line with the Project Development Objective (PDO) Indicators and overall achievement of the project and will assess whether the project is on track in terms of achieving the set development objectives, identify areas of improvement, and provide recommendations for the remaining period of the project.
The End-line survey must ensure consistency and comparability with the methodology and process followed for the collection and analysis of the baseline study for the project conducted in 2017 and the midterm review survey conducted in 2021 to allow appropriate analysis and comparison. Under the end-line survey, the contractor will conduct following surveys, namely (i) a Public Opinion Survey of soum citizens, (ii) an LDF-related Human Resources and Institutional Capacity Survey, and (iii) LDF operation effectiveness surveyData and information should be obtained using a multi-stage stratified random sample and cover all or some of the 21 aimags and sampled soums from those aimags. Where possible, the sample should return to the same aimags and households that were sampled during the baseline and midline review.
The contractor is expected to be on board by early April 2023, so the field data collection to take place in May and June 2023, and the data analysis to last for another month after the data collection. The contractor’s work should be completed in approximately four (4) months.
The contractor should possess extensive experience in conducting quantitative and qualitative surveys, data collection and management, as well as project evaluation and organizational development. The contractor should also have experience in partnering with local government officials and local communities and solid understanding of the budget law, local budgeting and budget execution processes in Mongolia.
II. BACKROUND
2.1 The Third Sustainable Livelihoods Project
The Government of Mongolia, represented by the Ministry of Finance (MOF), is implementing the Third Sustainable Livelihoods Project (SLP-3), which is funded by the International Development Association (IDA), World Bank Group, and the Swiss Development agency, Switzerland.
The development objective of the project is “to improve governance and community participation for the planning and delivery of priority investment in rural areas of Mongolia.”
Within the broader development objective of the project, the SLP-3 aims to build capacity and improve information sharing to the citizens on the Local Development Fund (LDF) and to help enhance capabilities at the local aimag and soum levels of managing the LDF. The LDF provides funds to soums to support capital investment in local public infrastructure and services. The Integrated Budget Law specifies eligible areas for investment such as pasture management-related investments, which aim to enhance risk management of rural soums. Also, the Law stipulates that local governments must utilize LDF allocations in accordance with priorities identified by the citizens through a robust of community participation process. The inclusion of community participation in the local budget preparation and execution process is a major step forward empowerment of citizens and a major reform of the citizen-government relationship.
Under the capacity building component, the revised LDF regulation has been approved by the Minister of Finance on Sep 12, 2018, and registered by the Ministry of Justice and Internal Affairs on Oct 12, 2018 with the following major changes in a substance:
- LDF subprojects to be approved shall always be supported and prioritized by local citizens;
- Subprojects financed by LDF shall be consistent with the long-term, medium-term, and short-term development policy documents specified in Article 6 of the Law on Development Policy and Planning;
- General definition of eligible subprojects was clarified; and
- The roles and responsibilities of each stakeholder in the LDF processes were better defined in the Regulation.
The SLP-3 comprises the following components:
(i) Capacity building for Local Governance and Livelihoods (Component 1);
(ii) Good Governance Performance Based Support Program (Component 2); and
(iii) Project Management, Monitoring and Evaluation (Component 3).
2.2. Project Monitoring and Evaluation
As part of component 3, the objective of the Monitoring and Evaluation (M&E) of the project is to measure the progress, results, and achievements of the SLP-3 and the LDF in general, as well as significant outcomes of the SLP-3 and the LDF for the intended beneficiaries. An important purpose of M&E is to provide valuable information and feedback on the project, upon which the management builds its decisions and corrective measures for further activities. It also aims to showcase how well the project has turned out for beneficiaries and all others involved.
The Project M&E is guided by the Results Framework, which specifies the results indicators, their baseline and target values, frequency of monitoring, sources of data, and instruments through which data is collected.
The Result Framework for SLP-3 was revised in October 2022 to take into account lessons learned from the initial years of implementation. The current list of project development objective (PDO) indicators are described below, and the detailed Result Framework and other indicators will be provided to the contractor upon contract commencement.
1) Average soums' Annual Performance Assessment score:
This will measure the percentage increase in average Annual Performance Assessment (APA) scores of soums: measuring how far soums progress from year to year in their compliance with the same sets of APA performance measures.
2) Percentage of soums' citizens surveyed reporting that LDF financed investments reflect their priority needs, disaggregated by gender.
This will show citizens’ perceptions of improvements in their access to public services, and the responsiveness and transparency of local investment financing supported by the LDF.
3) Participation: Percentage of bagh households participating in public bagh meetings:
Reflecting on the rate of citizen participation in the discussion of budget priorities shows how the project helped to meet the IBL provision of community participation in budget preparation and execution, a major reform of the citizen-government relationship.
The baseline survey to collect baseline data and other initial information for the SLP-3 was conducted by an independent research firm in 2018. Two main objectives were focused during the baseline survey processes, including: (1) establishing baseline data against the Project Development Objective and Intermediate Result Indicators; and (2) identifying and recommending appropriate results of key indicators that serve as a baseline to compare the progress and success of the project in relation to its relevance, effectiveness and efficiency. The major methodology and methods used and applied in the baseline survey included the Public Opinion Survey and Review on LDF operation.
The Midterm review (MTR) survey to assess whether the project is on track in terms of achieving the set development objectives, identifing areas of improvement, and providing recommendations for the remaining years of the project was conducted by an independent research firm in 2021. Three main objectives were focused during the MTR processes, including: (1) assess SLP-3 progress to date against workplan targets for the realization of intermediate and long-term results ; (2) highlight the most significant achievements that the SLP-3 has made so far, and identify program components and/or mechanisms that have led to the observed results; and (3) identify key success stories, emerging good practices, and lessons learned through the implementation of SLP-3 to date and develop recommendations for the further refinement of the program. The major methodology and methods used and applied in the MTR survey included the Public Opinion Survey, LDF human resource, institutional capacity survey, and Focus group discussion.
Since the project reached its final years, the MOF intends to undertake a review of the progress towards the intended results of the project by conducting the end-line survey, which will focus on the project impact in the line with the PDO Indicators and overall achievement of the project. The End-line Survey report will be an important document to serve as a reference and guide the further actions of MOF, SLP-3 team, and the project stakeholders.
III. OBJECTIVE OF THE ASSIGNMENT
The objective of the End-line Survey is to evaluate the performance of the SLP-3 against the desired results and achievement as articulated in its Results Framework.
Specific activities of the End-line Survey include:
1. The Public Opinion Survey: to identify citizen participation in Bagh meetings, the level of citizen participation at the various LDF stages, degree of knowledge of the LDF (including the annual soum budget), and their activities against project indicators as an end-line measure of performance. This activity will be carried out through household surveys.
2. The LDF related Human Resources and Institutional Capacity Survey: to assess of performance and abilities of the institutions related to the LDF and of the capacity of the personnel running the LDF implementation in selected soums. This activity will be carried out through surveys administered to the Head of the Citizen Representatives’ Khural or the Secretary of the Khural, Soum Governor, the Head of the Soum Administration, Head of Finance of the Soum or Government staff from the Finance Department, Soum administration staff who is responsible for the LDF, Bagh Governor, etc
3. Review on LDF operation: Desk review of MIS data to determine the current situation of the LDF operation in connection with good governance, the roll-out of the Budget Law, capacity to deliver by local governments, and the institutionalization of the LDF M&E system in a participatory manner. In addition, the status and quality of a limited number of sub-projects financed by LDF will be assessed through site observations.
4. Determine the effectiveness of the APA: to determine whether APA has contributed to the improvement of the Soum LDF implementation, has result of the APA been made available to the public and how, and what kind of activities were contributed to strengthen the capacity. The APA effectiveness survey will be conducted through surveys administered to the Head of the Citizen Representatives’ Khural or the Secretary of the Khural, Soum Governor, the Head of the Soum Administration, Head of Finance of the Soum or Government staff from the Finance Department, Soum administration staff who is responsible for the LDF, Bagh Governor, etc.
5. Survey Analysis: Using collected data and information, the Contractor will conduct analyses and write reports on the following subjects:
a. The Public Opinion Survey Result
b. The LDF related Human Resources and Institutional Capacity Survey Result
c. The Result of Review on LDF operation
d. Good practices and lessons learnt: Identify key success stories, emerging good practices and lessons learnt through the implementation of SLP-3 to date and develop recommendations for the further sustainability of programme.
IV. SCOPE OF WORK
The End-line Survey must follow the baseline and midterm review survey methodology and process and ensure comparability to the baseline survey, although a reduced scope might be applied for the End-line Survey review, depending on the methodological approach that will be defined in the End-line survey inception report. Following surveys to be carried out under the End-line Survey are as follows:
1. The Public Opinion Survey
2. The LDF related Human Resources and Institutional Capacity Survey
3. Review on LDF operation and determine the effectiveness of the APA on LDF
The detailed methodology will be designed by the contractor during the inception phase. Specific contents and research questions for each survey are detailed below, however, the contractor can add more as needed.
4.1 Survey Instruments Design and Testing
The survey will have multiple target populations – soum citizens for the “Public Opinion Survey” (POS), and staff who are relevant to LDF implementation, in particular to the sub-projects that are financed by LDF, for the “LDF-related Human Resources and Institutional Capacity Survey”. The Project Implementation Unit of the SLP-3 will share the instruments used for baseline and midterm surveys and the consultant is expected to revise and adapt the instrument to meet the goals of the endline survey.
The instrument and strategy shall be accepted by the PIU before the implementation. Before developing the survey instruments, the Contractor needs to develop surveying strategies and variables to set benchmarks that will be compared throughout program implementation and completion. Before developing strategies/variables, we recommend the Contractor to get familiar with project documents that will be provided upon contract commencement and other relevant laws and regulations that regulate relations in the project implementation area. In addition, the contractor should organize one/multiple workshops with the client where the developed research instruments are reviewed, new indicators are added/dropped, question/response rate phrasing is assessed, etc.
The Public Opinion Survey (POS):
1. Quantitative survey-POS
The survey instruments will be a questionnaire. Key interest areas and indicators of the survey should be to represent project results within the project goals and the project result framework. Since the contractor will make recommendations on the project result framework, this survey shall cover as many aspects as possible around project implementation area. The below is a list of the minimum survey interest areas, based on which the contractor shall develop a survey instrument (a questionnaire):
Citizen awareness on LDF: Familiarity with LDF and overall transparency and disclosure to the public, whether the citizens are informed about the soum development plan before it is translated into project proposals, and reports of budget execution and performances, and whether the citizens acknowledge their rights or obligations regarding local budget planning and execution.
Citizen Participation: Participation in planning and preparation of sound investment proposals, identification of budget priorities, budget preparation and adoption, budget execution and procurement, reporting, monitoring and oversight, and sustainable asset management. More specifically whether the LDF- financed projects meet the local needs, whether the citizens are available and interested to conduct M&E on project activities, whether the citizens’ participation in M&E would be accepted by the officials in local government, etc.
Citizen satisfaction on LDF subproject outcomes: Whether the subprojects meet the local priorities including pasture risk management etc., and quality and impacts of those outcomes.
The questionnaire can be organized in separate blocks of questions by contents and logic sequence of questions. The main principles used in the development of the questionnaire should be the following:
- Maximize the extent to ensure obtaining of wide variety of opinions that can be measured over time
- Maximize the extent to which indicators use appropriate quantitative measures to improve reliable comparability over time
- Be concise and should not be overloaded with generic questions. The wording and content of questions must be clear to respondents and appropriate to avoid possible defects such as response bias.
- Accommodation within a reasonable scope to make the fieldwork feasible in terms of timing and budget.
Sampling unit: The POS will obtain representative information on target population as best as possible using a nationwide random, multi-stage stratified sample. The target population of the POS would be soum citizens/adults, so that the survey will be taken from randomly selected individuals who live in the sampled soums. However, sampling individuals is not practically feasible. Thus, we recommend a household as a sampling unit and an individual from a household to be selected with special selection procedures such as the last birthday technique, Kish method, etc.
The contractor shall keep the proportion of individuals in terms of their gender, occupation and age group based on national or local demographic statistics and ensure representation from soum center and non-soum center baghs in order to prevent from selection bias through random sampling
Sample size:. In order to absorb nationwide opinion to the greatest extent possible, we require a maximum margin of error of plus or minus 3 percent at the 95 percent confidence level for meaningful and reasonably accurate estimates, and total sample size is expected to be 1,552 households (one respondents per household) considering potential drops due to nonresponse and data collection quality.
2. Qualitative survey –Citizen focus group discussion (FGD)
Sampling unit- The sampling unit for this survey is soum citizens.
Sample size: The sample size and sampling methodology shall be proposed by the contractor.
The LDF-related Human Resources and Institutional Capacity Survey:
The assessment of performance and abilities of the institutions related to the LDF and of capacity of the personnel running the LDF implementation in selected soums will be undertaken using specially developed instruments.
Sampling unit: The sampling unit for this survey is soum but the actual interviewees for the review are to be civil servants responsible for the LDF working in soums and baghs. The qualified survey respondents include the Head of the Citizen Representatives’ Khural or the Secretary of the Khural, Soum Governor, the Head of the Soum Administration, Head of Finance of the Soum or Government staff from the Finance Department, Soum administration staff who is responsible for the LDF, Bagh Governor, etc.
Sample size:. The sample size and sampling methodology shall be proposed by the contractor. Sampling methodology and overall recruitment rate will be one of the criteria for contractor selection.
The Contractor will develop a survey implementation plan for collecting, managing, and cleaning survey data. This plan, including documentation describing how it plans to conduct the pre-testing, fieldwork, and data cleaning in a way that minimizes costs and maximizes accuracy must be submitted to the PIU for comments and acceptance before survey implementation begins.
Review on LDF operation: The Review covers LDF operation at the local administration level including Bagh and Soum focusing citizen participation, budget preparation and planning, budget execution and procurement, reporting, monitoring and evaluation and operation and maintenance of subprojects financed and disclosure and transparency. Desk review of MIS data to determine the current situation of the LDF operation in connection with good governance, the roll-out of the Budget Law, capacity to deliver by local governments, and the institutionalization of the LDF M&E system in a participatory manner. In addition, a limited number of sub-projects financed by LDF will be assessed through site observations with a scope in terms of outcomes, such as targeting, efficiency, and effectiveness.
Determine the effectiveness of the APA: to determine whether APA has contributed to the improvement of the Soum LDF implementation, has result of the APA been made transparent, and what kind of activities were contributed to strengthen the capacity. The APA effectiveness survey will be conducted through surveys administered to the Head of the Citizen Representatives’ Khural or the Secretary of the Khural, Soum Governor, the Head of the Soum Administration, Head of Finance of the Soum or Government staff from the Finance Department, Soum administration staff who is responsible for the LDF, Bagh Governor, etc.
Pre-testing
After developing both survey instruments, the contractor shall organize pre-testing and piloting them, implement necessary changes and/or additions. The pre-testing and piloting will be administered to a small number of soums and citizens with different features (no less than 5 soums and 40 citizens) that will not participate in the actual data collection. The PIU will be present during the pretesting, if necessary.
All these processes shall be documented and reported to the PIU. The Contractor will also be responsible for checking the internal consistency of the instruments being used by conducting field pretests before implementation. The Contractor will translate the survey into English during the instrument development and once it is finalized send it to the PIU as requested.
4.2 Interviewer Training and Data Collection
The Contractor shall prepare a plan for recruiting, hiring, and training data collection staff as part of the training plan. The Contractor shall develop the training materials and manuals for interviewers and data operators and submit them to the PIU for approval. All training materials and manuals must be finalized before the training takes place. The plan shall address the number of interviewers to be hired to complete the survey on schedule; the development of training guides for interviewers; data collection manuals; survey pretests; and oversight to ensure interviewers can successfully complete their duties. Interviewers must be adequately trained prior to administer the survey in order to ensure maximum quality The contractor shall administer an enumerator test after the training and before the pilot with a minimum passing score. No interviewers will be sent to the field until they have demonstrated a sufficient understanding of the protocol. The survey process should include survey management that involves inspecting each completed survey while still in the field to ensure that less than 2 percent of the responses to individual questions are recorded incorrectly or in a manner that is inconsistent with the responses to other questions. The PIU will do random audits to ensure the reliability and accuracy of data collected by the Contractor. The inception report shall include details on how the contractor will monitor and supervise data collection and ensure data quality.
The Contractor will also develop a Survey Implementation plan for collecting, managing, and cleaning survey data. This plan, including documentation describing how it plans to conduct the pre-testing, fieldwork, and data cleaning in a way that minimizes costs and maximizes accuracy must be submitted to the PIU for comments and acceptance before survey implementation begins.
The Survey Implementation plan shall include a deliverable submission timeline, description of developing the sample frame, data storage arrangements, data entry arrangements, geographic scope and logistical arrangements to reach sampled communities/respondents, enumerator team composition and deployment to field sites, fieldwork and data cleaning in a way that minimizes costs and maximizes accuracy must be submitted to the PIU for comments and acceptance before survey implementation begins.
The contractor shall deploy CAPI tools instead of the conventional paper-based questionnaires and manual data entry. The implementation plan should specifically incorporate the above issues with the CAPI- based data collection instruments.
The Contractor is expected to provide satisfactory oversight of surveyors in the field. Specifically, the Contractor must provide on-site management of surveyors that is sufficient to observe the activities of the surveyors, identify problems in their administration of the surveys, and correct those problems to ensure that administered surveys are complete and devoid of obvious inconsistencies. Incomplete, inaccurate, or inconsistent surveys must be corrected by replacing the respondent with another randomly selected respondent of the same target group. The Contractor shall develop reserve sample of respondents in cases of non-response/absence of the respondent. Contractor will work to ensure that questions asked to respondents are complete and consistent with other responses. The PIU will also be accompanying the Contractor and its data-collection staff periodically during their time in the field.
During the entire data collection period, the Contractor should submit weekly reports . Observing the present progress and problems from weekly reports and field participation, the PIU will provide comments and other guidance on the data collection process of the Contractor. Furthermore, the Contractor could propose appropriate adjustments in the implementation plan based on findings from the progress of the data collection as part of the weekly report. Each field team shall document their presence in assigned soums and areas with photo and reference letters with signatures by officials etc. The contractor shall gather GPS coordinates of enumerators/respondents that can be obtained at the offset of conducting the interview, if electronic data capture is deployed. The contractor will attach them to a weekly reports and the data collection completion report. The firm should comply with international standards and professional ethic requirements of public s including voluntary participation, informed consent, anonymity, confidentiality, potential for harm, and results communication (When collecting personal data from data subjects for the purpose of research, researchers must be transparent about the information they plan to collect, the purpose for which it will be collected, with whom it might be shared and in what form. Researchers must ensure that personal data used in research is thoroughly protected from unauthorized access and not disclosed without the consent of the data subject. Researchers must always behave ethically and not do anything that might harm a data subject).
4.3 Documentation of Data Set
The Contractor will determine the software, staffing, and process to be used for data entry and include them in the implementation plan. The PIU will approve the software, staffing, and process used for data entry. The data entry software shall have features, for instance, its interface should mirror the survey instruments, include adequate consistency checks, allow sufficient space to adequately capture all open-ended responses, constrain numerical responses to the appropriate range of values, etc.
Aside from software, the Contractor is strongly recommended to implement double entry and reconciliation from the paper copy to ensure data quality. (If the Contractor proposes using an alternative approach to double entry, the Contractor must justify its use based on previous successful experience.) The Contractor will be required to format datasets and store and transfer data according to standards agreed upon with the PIU. The Contractor will provide basic data documentation, known as metadata. Metadata standards and requirements include the following:
- A data dictionary including descriptions, labels, and coding schemes for all variables;
- Information on data collection methods, number of visits, substitutions, final dispositions, etc.
- Information on data cleaning processes;
- External resources including training manuals, earlier drafts etc. should be collected and will be appended.
The Contractor will be responsible for processing the data to look for problems in a way that can be used by others as well as being responsible for analyzing the data itself. The keystroke error rate should be below 1 percent or the Contractor may be required to reenter data to reach that rate.
Alternatively, the contractor may wish to deploy CAPI -based instruments instead of the conventional paper- based questionnaires and manual data entry. If that is the case implementation plan should contain specific plan ensuring high data quality through programmed tools for example consistency, range and constraint checks and other automatic error detection features including some form of data encryption.. Additional data collection such as attaching GPS coordination, photos, videos, etc. would be recommended if CAPI technology is utilized. The contractor shall provide the client with access to the central database at the beginning of data collection, to ensure the possibility to monitor survey implementation progress.
The Contractor should make itself available to PIU in the event that requires additional information about the data. The Contractor will be responsible for resolving any ambiguities that arise as a result of these discussions. This may require that the Contractor refer back to the paper copies of the questionnaires if ambiguities in data entry cannot be resolved in another way. The Contractor is responsible for tabulating the data and organizing it according to regional or socioeconomic information stipulated as requested by the PIU.
Data entry progresses and correction activities shall be reported to the PIU through Weekly reports.
The Contractor shall be aware that all reports, data, data entry programs and other documents prepared by the Contractor shall become and remain the property of the PIU, and the Contractor is not authorized to use the data or derivatives of the data for its own purposes in any form without the express written consent of the PIU. The Contractor shall be responsible for keeping survey instruments for the duration of the SLP-3 for internal control.
4.4 Validation mechanisms
The team should use various of methods to ensure the validity of the data collected and the proposal should provide a clear description of these methods.
V. END-LINE SURVEY IMPLEMENTATION RESPONSIBILITIES
5.1 The SLP-3 Technical Team
The Ministry of Finance (MOF) will be a contracting party with the Contractor, whereas day-to-day management and supervision will be provided by the Project Implementation Unit (PIU) of the SLP-3. The Contractor will be reporting to the PIU. The technical joint team including MOF and SLP-3 PIU will have the following responsibilities:
- In general, manage the contract with the Contractor;
- Review and provide comments on the operational work plan and timetable and review all the documentation provided by the Contractor;
- Consult with the Contractor on the list of key concepts, and definitions such as the sampling plan, survey content, coverage, and sustainability;
- Review with the Contractor the pertinence of the themes and variables used in the study for the program. Similarly, review the precision and scope of the questions used to gather information and data;
- Review instruments, tables, formats, questionnaires, etc. to be used to collect data in communities;
- Conduct random spot checks by its staff or by hiring a Data Quality Reviewer to ensure the reliability and the accuracy of data collected by the Contractor, including verification of the quality of the data collected by the Contractor; and
- Provide the Contractor with information on project activities and other inputs as needed and including the following:
o Introductory session on the LDF by the MOF and the PIU;
o The Project Results Framework and materials worth sharing for the purposes of the End-line Survey;
o Previous survey instruments, methodology, and results conducted by consultants of the SLP-3; and
o APA criteria and manual, and preliminary/final results.
5.2 The Ministry of Finance (MOF)
As a contracting body and an implementing agency of the Project, the MOF will be reviewing and accepting main deliverables including the work plan, survey instruments, and reports. The duration of the report review by the MOF will be at least 2 weeks after submission.
5.3 The World Bank
As the financier and oversight body of the Project, the World Bank will provide comments on the draft deliverables and ensure that the contracting process and contract management are consistent with the relevant procurement guidelines of the World Bank. In addition, the WB will be reviewing and accepting main deliverables including the work plan, survey instruments, and reports.
5.4 Resources Available to the End-line Survey team
The Contractor will have the following documents:
At the project level:
- The Financing Agreement, the Project Implementation Manual (PIM), the Results Framework, etc.;
- Annual work plan/procurement plan and semi-annual and annual reports for 2016-2022;
- Baseline survey report;
- Mid-term review survey report;
- Reports on Annual Performance Assessment (APA) of the LDF conducted by the SLP3 (2017-2022);
- Special survey reports (technical audit reports, social study reports, etc); and
- Relevant project staff as well as local project officers (LPOs) will be available for interviews remotely or in person depending on the methodology employed by the contractor.
In addition, the PIU/MOF will brief the contractor on the LDF at the beginning of the assignment. Specific contents and research questions for each survey are to be developed in detail. All the survey instruments and strategies shall be accepted by the PIU before the implementation/fieldwork.
VI. DELIVERABLES
The contractor will be required to deliver the following key deliverables in Mongolian and English.
1) Inception report: containing the End-line Survey framework, detailed sampling methodology, work plan, and logistical arrangements, such as training of the interviewers,inplemenation plan etc.
2) Draft report/Debriefing of preliminary findings: The full team will debrief SLP-3 PIU team, Implementing Partners, MOF, and WB team on their findings, conclusions, and recommendations using a PowerPoint presentation and any other briefing materials as required. Project partners will provide feedback during the briefing meeting.
3) Final End-line survey report to be structured as follows:
o Executive Summary
o Introduction
o End-line Survey Objectives and Methodology
o Data analysis and findings of End-line Survey
The Public Opinion Survey
The LDF- related Human Resources and Institutional Capacity Survey
Review on LDF operation
Data table showing progress against the project development objective and intermediate indicators
o Conclusions and Recommendations
o Reference
o Annexes (detailed data to supplement the overall summary evaluation report, including interviewee list, details of the data collection instruments, key documents reviewed, etc.)
4) Completion Report: containing an overview of the End-line Survey process including write up of any challenges and how they were solved.
5) Soft copy of the final and full report with all attachments accompanied by the complete and cleaned data and syntax with variables description and analysis.
Inception report,draft final report, final report, and completion report should be first submitted in a draft form to the PIU. Based on the results of the discussion, the Contractor will make the necessary changes and then will submit a final deliverable. Unless otherwise noted, the PIU will provide comments/feedback within 10 (ten) business days of receipt of deliverables, and the Contractor shall provide a revised deliverable within 7 (seven) business days after receiving feedback. This process will be repeated as required by the PIU or otherwise the deliverable shall be considered accepted.
6.1 Work Plan
The work plan must include all the items listed previously, including in part the following:
1. An overview of the plans for staffing the project
2. A timeline for the execution and completion of tasks
3. An operational scheme and the organization of human resources needed for the Contractor to undertake data collection, processing of data collected, producing reports, and documentation of the data sets.
The initial work plan shall be submitted one week after the commencement of the Services under the contract. The PIU will provide comments within one calendar week of receipt, and the Contractor shall provide a revised work plan within one week.
6.2 Interviewer Training Plan
The Interviewer Training Plan shall address the structure of the training and detail a timeline and budget for training. Interviewers must be adequately trained prior to implementing each data collection instrument. This plan shall be submitted one week after the commencement of the Services under the contract.
6.3 Survey Instrument Development and Implementation
The Contractor must present the following: a) survey instruments before pre-testing, b) pre-testing methodology c) results of the pre-testing, and d) final instruments. The PIU shall be allowed to accompany the Contractor while field testing the instrument, as desired. These shall all be completed and submitted within 2 weeks of the commencement of the Services under the contract, or as negotiated with the PIU, but any problems identified in this phase must be brought to the attention of the PIU immediately so that they can be resolved before the full survey is fielded.
6.4 Implementation Plan
The Contractor must present an Implementation Plan for the survey detailing the methodology for: a) data collection, b) data entry, c) quality control, and d) regular reports as described in the next section to provide information on the status of the project. In addition, the Contractor should submit a Compensation Plan as part of the Implementation Plan. This document must be based on the Contractor’s initial proposal, but it should be revised to reflect comments and recommendations of the PIU and findings by the Contractor. A draft of this document should be submitted one week after the contract is signed. Survey pre-testing will not begin until the PIU approves the work plan, preliminary survey instrument, and plans for pre-testing the instrument. Survey fieldwork and interviews will not begin until the PIU approves the final survey instrument and pre-testing results, as well as work plans for the data entry and cleaning process, field quality control measures, and data processing tabulation. The PIU and other related officers shall be allowed to accompany the Contractor while survey taking place, as desired. The implementation plan shall be submitted one week after the commencement of the Services under the contract or as negotiated with the PIU.
6.5 Weekly Updates
The Contractor will provide the PIU with weekly updates on the status of the project via e-mail and will be available for conference calls or in-person meetings as requested to discuss the project’s status directly. The Contractor could propose appropriate adjustments in the survey and implementation plan based on findings from pre-testing, pilot testing, and the review of field conditions as part of the weekly updates in addition to formal documentation.
6.6 Copies of Completed Surveys
While the survey is being implemented, the Contractor will be asked to provide the PIU with some copies of completed surveys as an attachment to the weekly report The contractor shall develop a research archive, where all research outputs (surveys, field notes, research protocols, audio recordings, etc.) are stored. Ex-post survey completion, the client can request full access to the archive. These copies can either be on a CD in electronic, scanned format, or paper photocopies of the original documents. The Contractor shall be responsible for keeping all other original copies of the files for the duration of the SLP-3 for internal control.
6.7 Data collection completion report
After the dataset is submitted, the Contractor shall submit a Data Collection Completion Report. The Data Collection Completion Report includes information beyond the dataset itself and should include:
a. Overview of the survey
b. a Summary of the data collection methodology and any deviations from the proposed methodology that occurred during data collection along with a tabulation of all interviews conducted and the number of interviews with incomplete or inconsistent information including the responses in question.
c. Documentation of survey implementation (sampling protocols and documents that prove their presence in assigned soums and areas etc.)
d. Survey instruments
e. Metadata (including documentation of survey implementation, descriptions of all variables, and coding scheme)
The Data Collection Completion Report should be submitted within a timeframe determined by the PIU. The PIU will review the Data Collection Completion Report and provide comments and recommendations to the Contractor. The Contractor will have five (5) business days to respond to comments and recommendations unless an alternative timeline is negotiated between the Contractor and the PIU.
6.8 Final Reports and Presentation
The End-line final Survey Report, which is detailed in the Scope of Work section should be submitted within a timeframe determined by the PIU. But before the submission of the End-line final survey, the contractor shall first prepare and deliver an outline of the final report to SLP-3 PIU. PIU will review the report and provide comments and recommendations to Contractor. The Contractor will have ten (10) business days to respond to comments and recommendations unless an alternative timeline is negotiated between the Contractor and the PIU.
Once survey analysis reports have been completed, the consultant will hold a presentation for the PIU, the MOF and WB to introduce the survey results.
VII. CONSULTING FIRM QUALIFICATION
This section details the key traits that the PIU has identified as important for the successful completion of activities outlined in this TOR. The contracting firm should ideally possess the following characteristics:
- A prior working experience of the good governance and public financial management, especially budget law, local budgeting, and budget execution processes in Mongolia with an extensive experience with survey data collection and management. Specifically, the firm must have conducted at least 5 nationwide surveys independently and successfully for programs and projects of international donors and government agencies. Reference documents including reference letter, past survey performance report and/or Acceptance certificate issued by previous employers, and contacts of previous employers should be submitted as evidence;
- The firm shall be operating as a legally and financially autonomous firm in the consulting business for at least 5 years. Audited financial statements for the last three years (2020, 2021 and 2022) should be attached;
- The firm should have permanent staff that can serve in key team roles (as outlined below) and experience recruiting high quality consultants, and the ability to rapidly mobilize and coordinate a number of qualified staff in multiple locations at the same time;
- The firm should have enough equipment and office space for data entering and processing and capability to enter and process large amounts of data in a limited time period;
- The firm should comply with code of professional ethics to conduct the public survey including voluntary participation, informed consent, anonymity, confidentiality, potential for harm, and results communication (When collecting personal data from data subjects for the purpose of research, researchers must be transparent about the information they plan to collect, the purpose for which it will be collected, with whom it might be shared and in what form. Researchers must ensure that personal data used in research is thoroughly protected from unauthorized access and not disclosed without the consent of the data subject. Researchers must always behave ethically and not do anything that might harm a data subject).
- Experience in communication and cooperation with local stakeholders including local government officials; and
- Demonstrated experience in dealing with international development agencies.
Following characteristics are required:
- The firm should have an experience in conducting survey in areas of public perception, community participation, local governance and local administration operations;
- A proven track-record in developing and deploying CAPI systems (ODK, Kobo, Survey Solutions or other);
- An advanced command of qualitative data processing software;
- An experience in using GIS software packages, etc.
Exclusion grounds
If the firm have taken part in the APA under the SLP-3 project should be excluded from the procurement process.
Composition of the End-line Survey Team
The End-line survey team is expected to consist of a team leader, a sociologist – researcher, a database manager, field supervisors and enumerators, and a local service delivery expert. The contractor may propose an alternative team structure, which would then need to be justified.
Team Leader
The Team Leader will be wholly responsible for managing all deliverables and the End-line survey process. The responsibility for training on survey instruments and management of the enumerators will solely lie with the Team Leader. It will also be the Team Leader’s role to effectively organize the enumerators into teams, with supervisors, and ensure that each enumerator collects data uniformly. SLP-3 PIU team may provide support to the team leader as needed, but the overall logistics management of office space, training venues, transportation, etc, will be the task of the team leader.
Duties and responsibilities:
1) Guide the data collection effort and oversee its effective implementation.
2) Design and schedule all surveys.
3) Oversee technical aspects including sampling, interviewer training, data collection, data entry, variable aggregation, and descriptive reports.
4) Participate directly in the oversight of fieldwork, through site visits, review of progress and review of primary data.
5) Maintain relationships with the firm, the PIU and other relevant data-collection entities.
6) Manage budgets and expenses.
7) Prepare and submit reports according to the agreed-upon timeline.
8) Ensure that appropriate resources are made available and managed in order to achieve the objectives of the contract.
9) Ensure that implementation of Contractor activities are in accordance with WB and GoM policies and procedures.
Sociologist – Researcher
Duties and responsibilities:
1) Assist the team leader in guiding the data collection efforts.
2) Develop survey instruments.
3) Conduct pre-testing and manage focus groups.
4) Direct the interviewers to gather information and carry out the fieldwork.
5) Provide technical assistance in sampling, interviewer training, data collection, data entry and data cleaning, and variable aggregation and descriptive reports.
6) Analyze, evaluate and interpret data.
7) Prepare and report findings and disseminate results.
8) Represent the team leader in his/her absence.
Database Manager
Duties and responsibilities:
1) Develop data entry systems.
2) Supervise the data entry process.
3) Ensure data quality through entry checks and logic checks, etc.
4) Prepare documentation of survey database including metadata.
5) Support the Team leader and the Researcher in terms of data processing.
Field Supervisor(s)
Duties and responsibilities:
1) Visit staff in the field to oversee and monitor their actions and to assure that they are proceeding with questionnaire collection according to the guidelines established by the PIU.
2) Coordinate lodging and travel arrangement for all staff.
3) Arrange for all completed tests and questionnaires to be filed and for copies to be delivered to the data entry team.
4) Coordinate between field staff and data entry staff to assure timely completion of deliverables.
1) Demonstrated experience managing similar projects in Mongolia with a focus on data collection.
2) Understanding of qualitative and quantitative research methodology.
Local Service Delivery Expert
Duties and responsibilities:
1) Consult the survey team in terms of local service delivery, specially in the context of the LDF related Human Resources and Institutional Capacity Survey (through engaging in a desk review, survey instrument design, data analysis and reporting).
2) Provide technical support to the review of the practical implementation of the revised LDF regulation.
VIII. TIMETABLE
In order to collect the data and produce required reports, the Contractor must propose a timetable to the PIU for the submission of all deliverables to be completed. The table below shows an estimated timetable for the submission of deliverables. The schedule outline in this timeline is negotiable depending on the Contractor’s ability to mobilize staff and develop needed survey tools. Please note that weekly progress reports shall be provided by the Contractor to the PIU in weekly base, which is not illustrated in the table below.
Estimated Approximate Timetable for Submission of Deliverables
Activity (Deliverables) Week after the commencement of the contract
Commencement of the contract after signature
Introductory session on LDF by the Ministry and the PIU Week 0
Implementation plan, work plan, and training plan and receive comments Week 1
Submission of the revised implementation plan, work plan and training plan and a preliminary draft of the survey instrument Week 2
Final survey instrument drafts for pre-testing in English and Mongolian Week 3
Pre-testing and necessary changes on instruments Week 4
Final survey instrument Week 5
Staff Training Week 5
Data collection begins Week 6
Data collection analysis Week 8
Debriefing/data validation meeting and presentation to SLP-3 project team (this team includes PIU staff, MOF, and WB ) Week 10
Contractor resolves any problem with the preliminary dataset Week 11
Data collection finishes Week 12
Contractor to submit draft report along with the complete dataset Week 12
SLP-3 project team provide feedback to contractor on End-line Survey draft report Week 13
Contractor to submit Final End-line Survey to SLP-3 project team Week 15
Present End-line Survey Analysis reports and presentations to PIU, the Ministry of Finance, the representative from the Country office of the World Bank Week 15
The Ministry of Finance verifies that deliverables comply with the TOR Week 16-17
The Country office of the World Bank will provide final approvals on main deliverables
Notes: All dates listed are calendar week
IX. CONTRACT DURATION AND SCHEDULE OF PAYMENT
The End-line survey is expected to commence in April 2023 with an inception phase followed by intensive data collection (desk review, interviews, and surveys), analysis and report writing. A debriefing/data validation meeting for presentation and discussion of preliminary findings should be conducted in mid-May 2023, and the final revised MTR report should be delivered by end of June 2023. The proposed total workdays for the contract is 80 workdays. Consultant will be selected in accordance with the World Bank Guidelines: Selection and Employment of Consultants by World Bank Borrowers, October 2016. If your firm/organization is interested in the above-mentioned assignment please submit the Expression of Interest no later than 12:00 PM, April 27, 2023. Online submission is NOT available for this selection. If you need any further details, please feel free to contact us.
Contact address: Third Sustainable Livelihoods Project, 8th Floor, Tod Tower, Ankara street-23, 4th khoroo, Chingeltei district, Ulaanbaatar, Mongolia. Tel: 7000-9828.
Payment of fees will be based on the delivery of outputs, as follows:
• Upon contract signing: 10%
• Upon satisfactory submission of the inception report: 40%
• Upon satisfactory submission of to the draft final report: 30%
• Upon satisfactory submission of the final evaluation report: 10%
• Upon satisfactory submission of the completion report 10%
X. LANGUAGE AND COMMUNICATION OF REPORTS AND PRESENTATIONS
All reports including inception and final reports, progress reports and other intermediaries must be submitted in both English and Mongolian. Presentations are to be made in both Mongolian and English, as specified by the PIU
2023 оны 04 сарын 18