N - Program Evaluation
Evaluate programs and services using measurable criteria.
Introduction and Explication
The successful performance of any information organization is measured by its value to constituent communities (Gilman, 2018). As responsible stewards of knowledge and information, user-centered library and other information professionals recognize that our organizations must iteratively evaluate stakeholder needs and experiences with the aim of improving the quality, value, and sustainability of existing services and programs; identifying areas in need of improvement; and informing the design and implementation of new innovative services/programs (Gilman, 2018). Evaluation projects may be initiated in response to an identified problem or area of concern, or as part of a library or other information organization's strategic plan. Regardless, the overarching goal when conducting a services/programs evaluation using measurable criteria is to meet or exceed user needs and expectations and, in turn, ensure the long-term viability of the organization.
Evaluation of Services and Programs
Evaluation is essential for information organizations seeking to improve current services/programs and create new innovative, inclusive, and technologically robust offerings that appeal to Digital Information Age society (Clarke, 2018, p. 284). Meaningful and actionable evaluation should be conducted from within the context of user satisfaction and is dependent upon an evaluator's dynamic understanding of the quality of relationships between the organization, its users, and collaborating institutions (Matthews & Hinchliffe, 2017, p.4). Evaluation of any information center's services and programs should use measurable criteria drawn from user feedback; strategies and tools that represent professional principles, best practices, and standards; and the organization’s key guiding documents such as the mission statement, values statement, budget, and strategic plan/s. The use of measurable criteria to evaluate the quality of services and programs produces evidence-based results which guide current and future decision-making and enable libraries or other information organizations to plan for and manage change.
Evaluation projects and resulting reports are themselves evidence of an organization's integrity, social responsibility, and stakeholder accountability (Matthews & Hinchliffe, 2017,p. 4; National Archives and Records Administration, 2016). Beyond improving specific services/programs, evidence-based evaluation projects "may be designed to enhance the visibility of library services, describe their impact, and strengthen the library’s political position among stakeholders" (Matthews & Hinchliffe, 2017, p.4). Therefore, it is critical that the assessing organization communicates the findings of an evaluation project broadly and inclusively (Matthews & Hinchliffe, 2017, p.5). Frequently, this is achieved via the publishing and promotion of public-facing annual reports which also serve as justification for program and service budget approval and funding requests.
Measurable Criteria
In the Digital Information Age, the quality of an information organization’s design, provisioning, and performance of user-centric programs and services is increasingly determined and directed by data (Free, 2017). Therefore, evaluation of these services and programs requires evaluators to invite, collect, and interpret feedback (i.e., data) from constituent community members (Matthews & Hinchliffe, 2017, p.3). A summative approach to evaluation is best when evaluating users' perceptions of the quality of the effect and/or result of a service or program (Matthews & Hinchliffe, 2017, p.4). All user data must be documented and managed throughout their lifecycles by the evaluating organization in a way that complies with relevant data privacy principles, standards, laws, and regulations (Free, 2017).
Both quantitative and qualitative methodologies effectively "extract and derive meaning from collected data" (Cervone, 2018, p. 321). Quantitative data collection and analysis is measured numerically and analyzed and presented statistically, whereas qualitative methods collect data that is not numerically measurable but may be categorized and interpreted (Cervone, 2018, p. 325). Examples of qualitative assessment tools include observation, open-ended surveys, suggestion boxes, focus groups, and interviews. Generally, the qualitative methodological approach is most useful when evaluators want to gain contextual insight into user-organization relationships, user experiences with current programs and services; and strategic, user-centered ideas for new service/program design, implementation, and evaluation (Cervone, 2018, p. 325; Free, 2017; Matthews & Hinchliffe, 2017, p. 33).
Evidence
Evidence 1: Oregon Museum of Science and Industry: Environmental Scan and SWOT Analysis Links to an external site.
The Oregon Museum of Science and Industry (OMSI): Environment Scan and SWOT Analysis is a group-generated qualitative evaluation of the strengths, weakness, opportunities, and threats related to an existing information organization's programs and services. Crafted for Information Professions (INFO 204, S22), the assignment directed the team to conduct an environmental scan of the organization and evaluate its program and service-specific information and communications strategies, channels, and resources within the context of the principles and best practices of the information profession. To accomplish this, the group gathered information and context about OMSI's history; target audience and other internal and external stakeholders and information communities; and the sociocultural and political trends influencing the museum's services and programs. Critical documents were reviewed including the organization's mission, values, goals, numerous strategic plans, and annual reports. Inventories of OMSI's services and programs, technology resources, organizational collaborators and competitors, and professional associations were generated, and operational and project budgets reviewed.
Evaluative criteria were drawn from the American Alliance of Museum's (AMA, 2020) TrendsWatch 2020: Museums as Community Infrastructure; the AMA's (2018) Museums' as Economic Engines report (2019); the Institute of Museum and Library Services' (IMLS, 2021) report entitled, New Research Underscores Role Museums, Libraries Play to Create Healthier, More Equitable America. The group assessed how well the organization's community engagement initiatives align with both its mission and current trends; the roles and accessibility of technology in programs and services; the organization's compliance with laws, regulations, and professional standards, principles, and best practices; the impact of its budget strategy on program and service design, delivery, and sustainability; and analysis of the effectiveness of its collaborative relationships in program/service design, delivery, and sustainability.
This evaluation report represents my knowledge and skills with Competency N by reflecting my ability to qualitatively evaluate an information organization's user-centered services and programs within the contexts of its mission, the information profession's best practices, and larger environmental trends. The overarching goal of the project was to provide the information organization with an innovative, evidence-based assessment report containing a detailed, evidence-based list of actionable recommendations that enables it to re-center its priorities for program and service design, delivery, and evaluation onto its existing and potential information communities' information needs; increase equitable access and use of technologies and other and educational programs and services; and update policies and procedures to adequately reflect the strategic priorities of the information profession so it may truly embrace its role as a Digital Information Age community information hub.
Evidence 2: ChatGPT: An Ethics Impact Statement & Data Privacy Impact Assessment Links to an external site.
Written as a practicum for AI, Data, & Ethics (INFO 287, F23), ChatGPT: An Ethics Impact Statement & Data Privacy Impact Assessment includes a fundamental rights impact assessment (FRIA) and data privacy impact assessment (DPIA) conducted on behalf of a hypothetical organization deploying ChatGPT as a marketing strategy to enhance customer engagement, answer customer queries, and find products to meet their individual needs. Measurable evaluative criteria were drawn from AI HLEG's (2021) Ethics Guidelines for Trustworthy Artificial Intelligence (AI), AI HLEG's (2020) Assessment list for Trustworthy Artificial Intelligence (ALTAI), and the Personal Data Protection Commission Singapore's (PDPC) Guide to Data Protection Impact Assessments (DPIA) (2021). Each of these inter-related assessment strategies are founded on the high-level ethical principles/fundamental human rights of respect for human autonomy; prevention of harm; fairness; and explicability with the aim of protecting stakeholder data privacy in compliance with relevant laws, regulations, standards, and professional best practices (AI HLEG, 2021).
Specific assessment criteria outlined in AI HLEG's (2021) ALTAI include human agency and oversight; technical robustness and safety; privacy and data governance; transparency; diversity, non-discrimination, and fairness; societal and environmental well-being; and accountability. The PDPC's (2021) DPIA assessment criteria include the specific identification of what personal data is handled by the system or process and the reasons it is collected; identification of the ways in which personal data flows through the system or process; identification of risks to data privacy through comparing the way data is handled to the required best practices outlined by a relevant personal data protection authority; amending systems/processes to address data privacy risks; creation of new organizational policies; ensuring identified risks are mitigated before deploying he system/process, and; iteratively auditing the system/process to ensure all risks are sufficiently addressed.
This document demonstrates my ability as an information professional to evaluate programs and services using measurable criteria based on the principles, strategies, and tools required to accountably and transparently mitigate risks posed by ChatGPT to stakeholder data privacy. To guide an organization through the process, I outline in detail the specific ethical risks to personal data posed by ChatGPT, and articulate the steps, key planning activities, strategies for identifying the types and purposes of data collected by the system/process. The aim of this data protection assessment and implementation plan is to develop an organizational action plan and related policies and procedures for ensuring trustworthy AI practices that facilitate societal well-being and compliance with ethical, legal, and professional requirements. As information organizations increasingly deploy complex, socio-technical artificial intelligence (AI) tools such as ChatGPT to support reference services and other customer service functions, FRIAs and DPIAs will enable them to maintain compliance with the ethical principles, values, and best practices mandated by library and information science (LIS) profession; facilitate revision and further development of relevant standards and policies related to the use of chatbots and other AI systems in information environments; and continue to act in transparent, explicable, and trustworthy ways that mitigate harms caused to their stakeholders' data privacy (AI HLEG, 2021; SAA, 2020).
This risk assessment, entitled Assessment of the Archdiocese of Seattle’s Archives Disaster Response and Preparedness Plan, was crafted as an assignment in Preservation Management (INFO 259, S23).The assessment's purpose was to test the effectiveness and usability of an archives' preservation program's disaster preparedness and response plan and its compliance with the Society of American Archivist's (SAA, 1994) Guidelines for Evaluation of Archival Institution and the Northeast Document Conservation Center's (NEDCC, 2015) Emergency Preparedness Risk Assessment (2015).
An effective disaster preparedness plan is a core archival preservation priority and responsibility (Ritzenhaler, p. 8, 1993). In the LIS context, a disaster is defined as a "sudden, unexpected event that results in the loss of records and information essential to an organization's continued operation" (Franks, 2018, pp. 219-245). As a result of a disaster and resulting interruptions to economic activities, an information organization, and particularly archives and records and information management (RIM) programs, may be hindered from, or even incapable of, providing access and use to stakeholders seeking evidence and other information held in their collections. To prevent or mitigate delays in user access services during and immediately following a disaster (e.g., earthquake), effective archival institutions proactively prepare, test, and evaluate the outcomes of a disaster preparedness and recovery plan (Franks, 2018, pp. 219-245).
The SAA's (1994) Guidelines for Evaluation of Archival Institutions (Guidelines) direct archives to "establish systematic programs of preservation management that are integrated with every other archival function through a coordinated set of activities designed to maintain records for use, either in their original form or in some other usable manner" by giving "priority to activities that mitigate the deterioration of materials or information and that encompass groups of material (environmental controls, storage management, disaster preparedness, staff and user education, holdings maintenance, security, and reformatting) over activities that redress damage such as item level conservation treatment" (SAA, 1994). The SAA (1994) Guidelines act as "an objective and consistent framework" with which archives may measure the compliance of their user-centered programs and services with the principles and best practices required by the profession (SAA, 2020).
For a list of measurable criteria specific to archival disaster preparation and response planning, the SAA (1994) recommends the use of the Northeast Document Conservation Center's (NEDCC, 2015) Emergency Preparedness Risk Assessment. Measurable evaluation criteria include identifying a disaster planning and response manager; developing an inter-departmental disaster planning and response team including facilities staff and archival staff trained in collections care; crafting a written document outlining the plan and team member responsibilities; a disaster team training program; a communications strategy and workflow; an initial response and assessment strategy/workflow; an identified recovery site; security procedures; a salvage strategy; a documented, comprehensive, and accessible supply of those items necessary to salvage and/or transport collection materials to an environmentally stable recovery site; and iterative review, testing, and updating of the plan (NEDCC, 2015). Further, when designing a disaster plan it is critical to prioritize human safety first by ensuring that the safety equipment, tools, supplies, and decision-making workflows are in place and regularly reviewed (Franks, 2018, p. 232).
This archives-specific disaster planning a preparation plan assessment and report acts as evidence of my mastery of the knowledge and skills necessary to meet the requirements of Competency N by elucidating my capacity to qualitatively test and evaluate an information center's plan using measurable criteria identified by the SAA (n.d.) and the NEDCC (2015). As an emerging archivist, the ability to develop an effective and viable disaster plan, and the capacity to test and evaluate its effectiveness and usability using measurable criteria identified by professional associations is one of my core responsibilities. Further, my ability to proactively identify areas which may be lack sufficient specificity toward successful response and recovery outcomes will enable my information organization to meet its professional mandate of ensuring optimal long-term preservation of, access to, and use of, digital and physical archival collections (SAA. 1994; Ritzenhaler, p. 8, 1993).
Conclusion
The effective information professional recognizes the importance of iterative, user-centric, evidence-based evaluation of services and programs. The use of measurable criteria to design and implement assessment strategies and tools, information organizations may identify and evaluate the strengths, weaknesses, opportunities, and threats to their ability to meet and exceed their organization missions and strategic goals, as well as their professional mandate to provide reliable and trustworthy information and evidence to their stakeholders. Further, such forward-thinking evaluations facilitate data-informed improvements to existing programs and services and support the creation of new, innovative initiatives which strengthen the organization's relationship with its users.
References
AI HLEG. (2020, July 17). Assessment list for trustworthy Artificial Intelligence (ALTAI) for self-assessment | Shaping Europe’s digital future. Digital-Strategy.ec.europa.eu. https://digital-strategy.ec.europa.eu/en/library/assessment-list-trustworthy-artificial-intelligence-altai-self-assessment Links to an external site.
AI HLEG. (2021, March 8). Ethics guidelines for trustworthy AI | Shaping Europe’s digital future. European Commission. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai Links to an external site.
American Alliance of Museums. (2020). TrendsWatch2020. https://www.aam-us.org/programs/center-for-the-future-of-museums/trendswatch-2020/ Links to an external site.
American Alliance of Museums & Oxford Economics. (2018). Museums as economic engines. American Alliance of Museums. https://www.aam-us.org/2018/01/19/museums-as-economic-engines/ Links to an external site.
Cervone, H. F. (2018). Managing data and data analysis in information organizations. In S. Hirsh (Ed.), Information services today: an introduction (2nd ed., pp. 314–330). Rowman & Littlefield.
Clarke, R. I. (2018). Innovative library and information services. In S. Hirsh (Ed.), Information services today: an introduction (2nd ed., pp. 278–287). Rowman & Littlefield.
Franks, P. C. (2018). Records and information management (2nd ed.). American Library Association.
Free, D. (2017, January 23). ACRL proficiencies for assessment in academic libraries. Association of College & Research Libraries (ACRL). https://www.ala.org/acrl/standards/assessment_proficiencies Links to an external site.
Gilman, T. (2018). Learning and research institutions. In S. Hirsh (Ed.), Information services today: an introduction (2nd ed., pp. 81–93). Rowman & Littlefield.
IMLS. (2021). New research underscores role museums, libraries play to create healthier, more equitable America. Institute of Museum and Library Services. https://www.imls.gov/news/new-research-underscores-role-museums-libraries-play-creat Links to an external site.e-healthier-more-equitable-america Links to an external site.
Matthews, J. R., & Hinchliffe, L. (2017). Evaluation and measurement of library services. Bloomsbury Publishing USA.
National Archives and Records Administration. (2016, August 15). Annual performance plan and report. National Archives. https://www.archives.gov/about/plans-reports/performance-accountability Links to an external site.
Northeast Document Conservation Center. (2015). Session 8: Emergency Preparedness. Northeast Document Conservation Center. https://www.nedcc.org/preservation101/session-8/8preparing-for-disaster Links to an external site.
PDPC. (2021, December 14). Guide to data protection impact assessments. Www.pdpc.gov.sg; Personal Data Protection Commission Singapore. https://www.pdpc.gov.sg/Help-and-Resources/2017/11/Guide-to-Data-Protection-Impact-Assessments Links to an external site.
Ritzenthaler, M. L. (1993). Preserving archives and manuscripts. Society of American Archivists.
Society of American Archivists. (n.d.). Northeast Document Conservation Center. Society of American Archivists. Retrieved April 9, 2024, from https://www2.archivists.org/consultants/northeast-document-conservation-center Links to an external site.
Society of American Archivists. (n.d.). Northeast Document Conservation Center. Society of American Archivists. Retrieved April 9, 2024, from https://www2.archivists.org/consultants/northeast-document-conservation-center Links to an external site.
Society of American Archivists. (1994, January). Guidelines for evaluation of archival institutions. Society of American Archivists. https://www2.archivists.org/groups/standards-committee/guidelines-for-evaluation-of-archival-institutions Links to an external site.
Society of American Archivists. (2020, August). Core values statement and code of ethics. Society of American Archivists. https://www2.archivists.org/statements/saa-core-values-statement-and-code-of-ethics Links to an external site.