Japanese Journal of Evaluation Studies
Online ISSN : 1884-7161
Print ISSN : 1346-6151
ISSN-L : 1346-6151
Volume 22, Issue 2
Displaying 1-12 of 12 articles from this issue
Special Issue: Formative evaluation for “innovation programs”
  • Iwao Oshima
    2022Volume 22Issue 2 Pages 1-2
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS
    Download PDF (555K)
  • Evolvability of the Practitioner and peer Based Empowerment Evaluation using CD-TEP:the PBEE/CD-TEP approach
    Iwao Oshima, Kenta Shindo, Yuriko Minamotom
    2022Volume 22Issue 2 Pages 3-14
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      Various social issues around human services that are not able to take appropriate action have been expanded these days in Japan, and new and effective innovative approaches for the solutions have been explored. For this situation, a new method of formative evaluation that can foster human service innovative “effective models” introduced with new perspectives into more effective Evidence-Based Practices has been needed.

      In response to this, authors have developed a formative evaluation named “Practitioner and peer Based Empowerment Evaluation using CD-TEP:the PBEE/CD-TEP approach, which use bottom-up and collaborative evaluation methods based on practices in the fields, and where both practitioners and consumers actively and proactively participate in the evaluation activities.

      In this article, we introduced the philosophy and values of the development of the method, and presented the developmental process. In the result, we described outlines and fundamental framework of the PBEE/CD-TEP, and discussed significance, range of application, issues to be solved and evolvability of the method compared with other evaluation methods.

    Download PDF (1457K)
  • Aki Yonehara
    2022Volume 22Issue 2 Pages 15-26
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      The history of educational evaluation is rooted in “measurement” for a summative evaluation such as achievement examination. However, after the educational measurement movement in the early twenties, the emphasis was shifted to more formative aspect of evaluation. Reflecting on the current global and domestic trend, some significant changes can be observed in common among the SDGs' educational evaluation, the OECD's definition and framework of Social Emotional Skills, and the National Course of Study in Japan. These changes include the tendencies: first, the purpose of education has changed into more qualitative direction; second, the concept of education has been captured in the wider social context; and lastly, more diverse stakeholders need to participate and coproduct in the evaluation process. Such new tendencies indicate the increasing needs for participatory formative evaluation and require us to reconsider the meanings of evaluation indicator and evidence and the function of evaluation activities.

    Download PDF (835K)
  • Evolution of the Concept and Practice towards Evaluation for Management Support
    Katsuji Imata
    2022Volume 22Issue 2 Pages 27-37
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      Since around 2015, “social impact measurement” has become a topic of discussion among many in Japan who are working to address contemporary social issues, especially by those in the private sector. At the same time, the concept is far from being firmly established in the evaluation discipline. This paper attempts to organize the concept of social impact measurement and argues that discussions of its theory and practice in Japan and abroad have led to the growth and evolution of the concept. Specifically, the paper will focus on how it is evolving since the mid-2010s by paying attention to the field of impact investing and impact measurement and management(IMM)in North America(especially in the U.S.), as well as how it has been treated in Japan by examining the work of the Working Group on Social Impact Measurement of the Cabinet Office. The result is a conceptual reframing of its measurement focus to a learning, improvement and management focus. Lastly, the paper discusses the prospects for this approach to evolve into a new form of management and decision-making support with an evaluative framing.

    Download PDF (872K)
  • − Focus on the What Works Centres in the child welfare −
    Naoyuki Ieko
    2022Volume 22Issue 2 Pages 39-51
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      In the UK, the central government has promoted a policy of emphasising evidence in social policy since around 2000, and evidence-based practice has been required for human services. Evidence-based initiatives in the field of child welfare have been promoted in conjunction with reforms to improve the professionalism of human service workers, and there are particularly high expectations for preventive policies such as early help and early intervention. As austerity has shifted the policy focus from large-scale trial to the use of evidence. Central government-led What Works Centre has been established, primarily as a knowledge brokering organisation to support the use of evidence on effectiveness in policy and practice decision-making. The function is to generate, translate and apply evidence. In the child welfare sector, the approach is linked to the funding of formative evaluation to develop innovations in human services into Evidence-Based programmes.

    Download PDF (926K)
  • − A Case Study Using the CD-TEP Approach −
    Junko Shimizu, Kenta Shindo
    2022Volume 22Issue 2 Pages 53-67
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      This article examines a case study of the development and implementation of an evaluation guide for the nationwide intermediate welfare and art support centers for the disabled, which was developed using an approach of the Circular Dialogue between Program, Theory, Evidence, and Practices(CD-TEP). By reviewing that process, this study analyzes how the CD-TEP-based evaluation guide promoted formative evaluation in the field and to what level the approach is implementable. The result shows that the guide utilization could have promoted the formative evaluation practices. However, it was observed that the role of the model effectiveness and fidelity scale in the evaluation guide has diminished. Also, the study on the implementability of the CD-TEP method partially using the framework of implementation science(CFIR)showed that for the philosophy as an evaluation approach to penetrate practice, appropriate governance and systems are needed, and the necessity of technical implementation support was affirmed.

    Download PDF (1096K)
  • ―Focusing on the Evaluation Capacity Building―
    Kenta Shindo, Iwao Oshima
    2022Volume 22Issue 2 Pages 69-82
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      Practitioner-Based Empowerment Evaluation(CD-TEP)approach is a useful method of program development and evaluation, aiming to obtain, implement, and disseminate effective program models. As part of promoting this approach, it is essential to build the evaluation capacity of the practitioners and organizations engaged in these programs. In this context, we examined the significance and role of the EBP(Evidence-Based Practices)Technical Assistance Center, which supports the building of evaluation capacity.

      The results indicate that evaluation capacity building leads to a change within practitioners and organizations, and even to an improvement of service outcomes or client outcomes. Accordingly, it appears that an effective EBP(Evidence-Based Practices)Technical Assistance Center performs(1)development of effect model through collaboration, (2)planning for assessment and evaluation capacity building, (3)provision of opportunities of training, consultation, observation, etc., and(4)networking.

    Download PDF (1016K)
Article
  • Focusing on the classification of expense by COFOG
    Yuichi Ubukata, Toru Satou, Hiroichi Kawashima
    2022Volume 22Issue 2 Pages 83-95
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      We would like to point out that local governments in Japan focus so much on the development of accounting information that they lack the perspective of information users, especially residents. In Europe and the United States, efforts are being made to visualize local government budgets from the perspective of residents in an easy-to-understand manner. In this case, a classification standard called COFOG is used. The purpose of this paper is to consider the possibility that COFOG can contribute to the creation of accounting information from the perspective of residents. In addition, we discuss the possibility of the improving accountability and transparency of local governments’accounting information by using COFOG. Local governments will be required to develop public accounting information that shows their willingness to explain to residents in the future. Therefore, by using COFOG, it will be possible to prepare detailed accounting information from the perspective of residents. As a result, it will be easier for residents to understand the overall picture of local government management, and there is a possibility of promoting resident participation.

    Download PDF (839K)
  • - Focusing on Regulatory Oversight Body and Experiment Evaluations -
    Yuichi Murakami
    2022Volume 22Issue 2 Pages 97-109
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      To prevent “regulatory capture,” this article discusses a systematic evaluation to ensure “better regulation” and a way to institutionalize it, focusing on(1)the Regulatory Oversight Body(ROB), its dilemma of political accountability and expertise, its comparison with the US, EU, and Japan, and(2)regulatory experiment evaluations. ROB should be placed above and out of the related policy community to ensure meaningful oversight and objective evaluation, and to eventually prevent the regulation from being captured by partial interests. Then, (3)a desirable regulatory evaluation system is envisioned, according to a review of the Special Zone System of Japan. The “stimulus-response” modeled regulatory evaluation can be properly conducted in well-organized regulatory experiments. The main points consist in overcoming the difficulties in causal inference and selection bias, rendering implementation and evaluation activities more objective, and reducing the final decision maker’s discretion after obtaining the evaluation result.

    Download PDF (894K)
Research Note
  • Hazuki Ikeda
    2022Volume 22Issue 2 Pages 111-122
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      Performance measurement is a relatively simple evaluation method, and evaluation systems that use performance measurement as their central method are widespread in the public sector. Since performance measurement is a method of evaluation by comparing actual values measured using predetermined indicators with target values, the quality of the indicators is important. Therefore, it is necessary to set appropriate indicators and review inappropriate ones. There are various ways to improve the quality of indicators, one of which is to organize and learn from the information on the indicators before the review, the reasons for the review, and the indicators after the review. However, this paper finds that there is little information on the status of implementation of the review of indicators and the reasons for the review. It also reveals that there is a lack of inductive information, such as information on the review of indicators, and describes how this information can help improve indicators.

    Download PDF (788K)
  • Batkhuyag Undrakh
    2022Volume 22Issue 2 Pages 123-135
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      In Ulaanbaatar, around 80% of air pollution is caused by the use of coal for heating and cooking. In response to this problem, in 2018, the Mongolian government banned the use of coal and allocated substantial resources to the development of refined coal briquettes, as stipulated in Resolution No. 62, “Prohibition of The Use of Raw Coal”.

      This study examines the implications of the air pollution control by applying the logic model theory. The application of the theory takes four steps. First, I conducted a need assessment of Resolution No. 62. Second, I implemented a process assessment of the resolution clarifying the causal relationship between each activity and the KPI(Key Performance Index)to examine the outcome of each activity. Third, I compared the KPI between different outcomes. Fourth, I reached the air quality indexes before and after Resolution No. 62 came into effect. The last section is the conclusion and policy recommendations.

    Download PDF (1702K)
  • Yurie Kishimoto
    2022Volume 22Issue 2 Pages 137-147
    Published: September 30, 2022
    Released on J-STAGE: September 28, 2023
    JOURNAL FREE ACCESS

      In recent years, university contributions to local communities have spread nationwide with various government support measures. It is important to link these efforts to education and research to achieve synergistic effects, and the number of community contribution programs conducted as part of classes has been increasing every year and is expected to continue to increase in the future.

      This paper focuses on programs that link university education with community contributions and are offered as part of the coursework. Although these programs are characterized by aspects of education and public policy, we present as an issue that evaluations aimed at improving the quality of the programs from a public policy perspective are few in number and insufficiently verified compared to studies on student learning effectiveness. We will also discuss the concrete effects of quality improvement evaluation, and the significance of evaluation for the purpose of quality improvement, using three typical evaluations conducted at universities as examples: accreditation evaluation, class evaluation, and learning evaluation (grade evaluation).

    Download PDF (1063K)
feedback
Top