Community development financial institution (CDFI) leaders and stakeholders know that understanding the effectiveness of their programs and services is critical to enhancing the social and economic impact in the communities they serve. As the industry grows, partners, funders, and investors are increasingly pressing CDFIs to engage in impact measurement and management (IMM)[1] and conduct program evaluations.[2]
Yet, the industry is diverse: All CDFIs are at different stages of developing IMM and program evaluation practices. CDFIs need additional information and resources to determine leading practices for measuring the impact of their activities. One resource is the Federal Reserve’s CDFI Survey, fielded nationally since 2019.
The 2023 CDFI Survey gathered information from 453 respondents. Three types of CDFIs made up 95 percent of respondents: loan funds (48 percent), credit unions and cooperatives (35 percent), and banks – including bank holding companies (12 percent). The survey asked CDFIs’ perspectives on tracking output and outcome metrics. Importantly, these questions help demonstrate where CDFIs are and where they want to go with IMM and program evaluation.
This brief presents findings from the survey to understand:
- what output and outcome metrics CDFIs measure and want to measure,
- why CDFIs track metrics, and
- challenges that CDFIs face in tracking metrics.
Explore the Findings
The Federal Reserve’s Survey of Community Development Financial Institutions (CDFIs) is administered biennially to CDFIs across the country. The 2023 survey gathered information from 453 CDFI respondents from April 24 to June 2.
The brief also examines differences in output and outcome tracking across different types of CDFIs. We look at variations in approaches and challenges in tracking metrics across business lines (e.g., small business finance, consumer finance, residential real estate finance), institution types, and capacity as measured by staff and asset sizes.
Examining IMM and program evaluation variations across CDFIs allows us to better understand the industry’s challenges and opportunities. Ultimately, this will help us enhance the industry’s learning culture, IMM practices and processes, and evaluative frameworks and goals to maximize CDFI effectiveness and impact.
Key Findings and Implications
Among the brief’s key findings:
- CDFIs are interested in measuring the mid- and long-term outcomes of their activities (e.g., financial performance of clients), but outputs remain easier to collect. They want to track metrics that align with their primary business line, but some outcome metrics are of universal interest (figure 1 and figure 2).
- CDFIs track metrics to inform organizational learning and decision making and to attract and retain grants and debt capital. The reasons why CDFIs track metrics differ by institution type and organizational capacity (figure 3, figure 4, figure 5, and figure 6).
- CDFI challenges in tracking metrics include resource and capacity constraints and data collection and measurement issues (e.g., no standardization in which metrics to collect, difficulty tracking metrics after products close). Challenges in tracking metrics vary according to institution types and organizational capacities (figure 7, figure 8, figure 9, and figure 10).
Among the implications of the findings:
- CDFIs and industry stakeholders must address tensions between outputs-focused reporting requirements and each CDFI’s ability to track progress and understand mission impact.
- Resources and capacity building are required to transition from outputs to outcomes.
- Funders, investors, and industry stakeholders should invest in capacity building to encourage growth and increase impact—particularly among smaller, capacity-constrained CDFIs.
What CDFIs measure and want to measure
The Federal Reserve’s 2023 CDFI Survey asked respondents to select which output and outcome metrics they tracked, and which they wanted to track but could not at the time of the survey.
The survey offered a range of response options (see examples in the callout box). However, due to survey limitations, the options were not exhaustive of all possible evaluative and impact measures. Although the available answer options may have influenced how respondents answered, they could specify additional metrics they tracked or wanted to track by selecting “other.”
| Metric | Description | Examples from Survey |
|---|---|---|
| Activities | CDFI products and services | CDFI loans, grants, and development services provided to clients and customers |
| Outputs | The direct products of CDFI lending, grant making, and development services | Number of clients or customers, dollar value of deployed products, client demographics |
| Outcomes | The short- and long-term changes to clients, customers, and communities that occurred because of CDFI activities | Client, customer, or community stories, number of jobs projected to be created, actual number of jobs created, job quality, financial performance of clients or customers |
Outputs are easier to measure; outcome data are more desirable.
Figure 1 presents actual and desired CDFI outputs and outcomes at the time of the survey. Respondents most frequently cited tracking output metrics, which measure the direct products of CDFI lending, grant making, and development services. For example, 96 percent of respondents tracked the number of clients served, and 86 percent tracked the dollar value of deployed products.
Financial institutions often routinely collect output metrics, so CDFIs are effectively their own data sources for this information. These data help quantify how CDFIs serve their communities through their financial products.
In addition to outputs, some respondent CDFIs were already tracking the outcomes of their activities. Outcome metrics demonstrate how CDFI loans and development services have affected clients, customers, or communities. For example, 38 percent of respondents tracked actual jobs created or retained, and 45 percent tracked projected jobs created or retained.
Although outputs of CDFI activity were widely collected, some gaps remained between what respondents wanted to track and what they could not. Gaps were wider for outcome metrics compared to outputs. For example, 61 percent to 82 percent of respondent CDFIs indicated an interest in tracking mid- or long-term outcomes such as clients’ financial performances, the number of jobs projected to be created, or the actual number of jobs created. Only 38 percent to 52 percent were doing so at the time of the survey.
Fewer CDFIs can track longer-term outcomes because they are difficult to measure. Some long-term outcomes are broad. Additionally, they are not easily quantifiable and take years to come to fruition. Even if the metrics are quantifiable, like income or business revenue changes, the data can be hard to access, necessitating rigorous and costly collection of clients’ or customers’ private data. Nevertheless, such information is vital and provides a more nuanced understanding of how CDFI investment and activities change communities.
CDFIs want to track metrics that align with their primary business line, but some outcome metrics are of universal interest.
Survey responses affirm that CDFIs are interested in tracking different metrics based on their business focus and institution type because the two are often interrelated. For example, many credit unions offer consumer finance, and a large proportion of loan funds provide small business finance. In this section, to streamline discussion, we focus on business lines rather than institution types. Top reporting business lines in the survey were consumer finance, small business finance, residential real estate finance, commercial real estate finance, and home purchase and improvement finance.
Output metrics were already widely collected across all institution types:
- More than 93 percent of respondents from the top reporting business lines were already tracking or interested in tracking the number of clients they serve.
- Most respondents from top reporting business lines were also already tracking or interested in tracking the value of deployed loans and products (90 percent to 100 percent), although slightly fewer CDFIs in consumer finance (86 percent) were interested in this metric. This may be due to smaller dollar value products for each loan in consumer finance.
Regulatory law influences the variation in collecting demographic data. Regulated CDFIs in consumer finance—including banks, bank holding companies, and credit unions—must comply with the Equal Credit Opportunity Act (ECOA). However, to ensure that CDFI target markets are being served, certified CDFIs that receive financial assistance from the CDFI Fund (including banks and credit unions) are permitted to collect client demographic information otherwise prohibited by ECOA.
Figure 2 presents the share of CDFIs in top reporting business lines that collect and want to collect various outcome metrics. More variation was apparent in outcome than in output metrics. Across the top business lines, most CDFIs want to collect information on customer financial performance (74 percent to 90 percent) and impact stories (75 percent to 89 percent).
Of all business lines, small business lenders are the most interested in measuring the number of projected or actual jobs created (90 percent and 89 percent, respectively). Commercial and residential real estate-focused CDFIs are moderately interested in tracking these metrics, but they are much less of a priority for consumer finance and home purchase business lines.
Though more difficult to measure, CDFIs want to track how clients advanced economically over time and to collect clients’ stories.
The outcome metrics that most respondents want to collect are client financial performance and client stories. Similarly, of the 77 respondents who wrote what “other” metrics they track or want to track, many are interested in longer-term changes in client financial well-being and place-based measures of economic improvement. For example, one respondent said they wish to track clients’ financial well-being through “improvement in credit score, increased savings, and reduced debt.” Another mentioned they wanted to track lending outcomes in low- to moderate-income census tracts.
Several respondents want to document how they have moved their communities away from predatory lenders. One said they are interested in accessing “longitudinal data on access to mainstream banks and credit unions” and measuring the resulting “decrease in the use of predatory or nonbank products and services.” Examining CDFI impact in these areas would underscore the industry’s mission of reaching underbanked individuals and communities.
Why CDFIs track metrics
CDFIs track metrics for various reasons, including to attract and retain funding and capitalization and to inform organizational learning and decision making. To be sure, the two reasons are not mutually exclusive. A CDFI may track a particular metric because a funder requires the organization to report on it and because the metric will help inform organizational learning and decision making.
Ideally, CDFIs should prioritize metrics that reflect changes they aim for among the communities they serve and those that promote organizational learning.[3] However, in their 2017 brief, “From Compliance to Learning: Helping Community Development Financial Institutions Better Determine and Demonstrate Their Results,” Theodos and Seidman document the CDFI tendency to collect and track metrics because of funder pressures and requirements rather than to use “the information to learn and grow.”[4]
Theodos and Seidman call for CDFIs and their stakeholders to prioritize learning and focus on collecting and tracking data for their own purposes and needs, embedding learning into organizational routines. Fielded six years after Theodos and Seidman’s brief, the 2023 CDFI Survey provides data to determine where CDFIs and the industry stand on the journey “from compliance to learning.”
CDFIs track metrics to inform organizational learning and decision making and to attract and retain grants and debt capital.
In the CDFI Survey, respondents were asked to select all reasons why they track metrics and subsequently identify the top reason. Figure 3a presents all reasons selected, whereas Figure 3b focuses on the top reason. Most figures in this section focus on the top reasons why CDFIs track metrics, which is most reflective of their highest priorities.
The figures illustrate that CDFIs use metric collecting and tracking to inform program design and organizational decision making. At the same time, priorities may be driven by the need to satisfy external requirements to retain existing funding or attract new investment dollars.
Figure 3a illustrates that CDFIs use the metrics they track for organizational learning purposes:
- Larger shares of respondents track metrics to inform their staff, management, and board and track their products’ success (88 percent and 82 percent, respectively).
- Slightly smaller shares track metrics to satisfy funder requirements and attract new funding (78 percent and 74 percent, respectively).
Satisfying funder requirements and attracting new funding and investment rise in relative importance when looking at the top reason for tracking metrics (Figure 3b):
- Similar shares of respondents said that tracking the success of their products (26 percent) and satisfying funder requirements (25 percent) are their top priorities.
- Slightly smaller shares said that attracting new funding is the third most important reason (18 percent).
The responses reinforce how revenue- and mission-driven metric tracking are not mutually exclusive. For example, 14 percent of respondents indicated that they track metrics to attract new clients and customers. Because they are depositories, credit unions and banks rely upon customers to increase revenue. Both institutions rely on metrics to attract new clients and build revenue streams, and both want to grow their customer bases to further their missions of expanding capital access to individuals and communities that mainstream financial institutions often overlook.
CDFIs track metrics for different reasons based on institution type and business model.
Figure 4 illustrates how business models of CDFI institution types and, particularly, the revenue streams thereof drive the top reasons for metric tracking. For example, 30 percent of loan funds chose “satisfy funder requirements” as their top reason for tracking metrics, whereas 25 percent of banks and 17 percent of credit unions selected that reason. This aligns with the fact that just 5 percent of credit unions’ capital comes from borrowed funds, whereas 46 percent of loan funds’ capitalization is from borrowed funds and equity equivalent investments.
Thirty-one percent of credit unions chose “attract new clients or customers” as their top reason for tracking metrics, compared to just 2 percent of loan funds. As depositories, credit unions’ capital for lending comes from customer deposits, whereas funders and investors such as banks, philanthropies, corporations, and governments tend to provide capitalization for loan funds.
CDFIs with less capacity must track metrics to attract new funders and retain existing ones.
Researchers have theorized that, in an ideal set of circumstances, there would be a virtuous circle where CDFIs collect and evaluate metrics to inform organizational learning and decision making, ultimately enhancing CDFI capacity.[5] Presently, capacity-constrained CDFIs often feel beholden to funder and investor requirements for survival and growth. This tendency is particularly acute among the smallest and most resource-constrained organizations.
Figure 5 shows the latter tendency in the 2023 CDFI Survey data. In this figure, capacity is measured as the number of full-time equivalent (FTE) employees, with small CDFIs having under 10 employees, mid-sized having 11 to 50 employees, and large having more than 50 employees.
Smaller percentages of large CDFIs reported funder-related concerns as their top reason for tracking metrics. Eleven percent of large CDFIs reported that attracting new funding or investment is their top reason for tracking metrics, whereas 18 percent of mid-sized CDFIs and 23 percent of small CDFIs selected this as their top reason.
Likewise, larger percentages of small and mid-sized CDFIs reported the need to satisfy funder requirements as their top reason (27 percent and 26 percent, respectively). In contrast, 20 percent of large CDFIs reported funder requirements are the key driver of metric tracking.
Interestingly, the findings concerning learning culture motivations and number of FTEs are more varied. For example, between 23 percent and 27 percent of CDFIs of all sizes said that tracking product success was their key reason for collecting metrics. For a larger percentage of mid-sized and large CDFIs, informing staff, management, and board was their key reason.
Regardless of asset size, roughly three out of 10 loan funds tracked metrics to satisfy funder requirements.
Figure 6 considers a different measure of capacity, namely assets. This figure presents data for CDFI loan funds to streamline the discussion. Number of FTEs and asset size are highly but not perfectly correlated. For the full dataset, the correlation coefficient between these two variables is 0.74 with a p-value of < 0.01.
About 30 percent of loan fund respondents said that satisfying funder requirements is the top reason they track metrics, regardless of asset size. The loan fund business model depends highly on contributed revenue and debt capital from funders and investors. Thus, CDFI loan funds of all asset sizes feel that they must collect metrics to sustain that model.
A growth mindset drives the logic of metric tracking for small and emerging loan funds:
- Thirty percent of loan funds with $10 million or less in assets cited attracting new funding or investment as their top reason for tracking metrics.
- Only 19 percent of loan funds with large asset sizes and 21 percent of CDFIs with a medium amount of assets selected attracting new funding or investment as their top reason.
In contrast:
- A larger percentage of large loan funds—as measured by asset size—said that informing staff, management, and the board was their top reason for tracking metrics.
- Only 7 percent of loan funds with $10 million or less in assets cited the same key reason.
This finding indicates that larger loan funds are more accustomed to using metrics to support organizational decision making.
Challenges CDFIs face in tracking metrics
The CDFI Survey asked respondents what factors impede their ability to track output and outcome metrics. Figures 7a and 7b present all challenges and the top challenges CDFIs face in tracking metrics.
CDFIs face challenges in tracking metrics due to resource and capacity constraints as well as data collection and measurement issues.
The most selected responses to this question fit into two categories: resource and capacity constraints and data collection and measurement challenges.
Regarding resource and capacity constraints, the top challenges were:
- Staff lack time to collect data (22 percent).
- It is too costly to collect data (13 percent).
With regard to measurement and data collection, the top challenges were:
- CDFIs find it difficult to collect client data after products close (21 percent).
- Clients and customers are reluctant to share data (17 percent).
Resource and capacity constraints are not mutually exclusive from data collection and measurement challenges. For example, locating customers to respond to a survey can be difficult for months or years after a loan closes. This challenge could be eased somewhat if CDFIs had more time and resources to maintain customer contact information and stay in touch with customers over time.
The top challenges CDFIs face in metric tracking vary by CDFI institution type.
Figure 8 presents the top challenges by CDFI institution type. The greatest share of loan funds (29 percent) said their top challenge is difficulty collecting borrower data after products close. Although loan funds often provide intensive technical assistance to borrowers during and after the lending process, the loan fund business model, unlike with credit unions and banks, does not lend itself to regular interactions with borrowers. As a result, smaller percentages of credit unions and banks selected this as a top concern.
Of loan funds and banks, 25 percent and 27 percent, respectively, said that staff time is a top concern, compared to 16 percent of credit unions. Similarly, compared to credit unions, a larger percentage of loan funds and banks reported client reluctance to share data as a top challenge. Conversely, compared to loan funds, larger proportions of credit unions and banks cited cost and a lack of standardization in metrics as top challenges.
As regulated entities, credit unions and banks are accustomed to reporting standardized financial data to regulatory agencies. Unlike financial data, mission-focused metrics typically vary from one type of funding agency to the next (e.g., banks, foundations, government agencies, or others). Loan funds are not regulated and are potentially more accustomed to variations in reporting requirements in the financial realm or mission and impact realm.
CDFIs with fewer FTEs cited lack of staff time as a top tracking challenge. In contrast, CDFIs with more FTEs reported difficulty collecting data after products close, and lack of standardization is their top challenge.
The most capacity-constrained CDFIs feel such constraints in data collection and tracking. Figure 9 presents the top challenge to metric tracking by number of FTEs. Thirty-five percent of small CDFIs identified lack of staff time as their top challenge compared to 19 percent of mid-sized CDFIs and 10 percent of large CDFIs.
Larger percentages of mid-sized and large CDFIs selected cost or a lack of standardization in metric tracking as their top choice compared to small CDFIs. Banks and credit unions have more FTEs on average than loan funds. According to the 2023 CDFI Survey, banks and holding companies had an average of 129 FTEs whereas credit unions had 62 and loan funds had 21. As discussed above, this finding aligns with the bank and credit union perspective on the lack of standardization as a challenge.
Loan funds with larger assets are most challenged by the difficulty of collecting data after products close, whereas those with smaller assets are most concerned by the lack of staff time for reporting.
Figure 10 considers another measure of capacity, namely asset size. The figure depicts the top challenge loan funds of various asset sizes faced. Capacity constraints are most pronounced among smaller and emerging loan funds:
- Thirty-four percent of loan funds with assets under $10 million cited a lack of staff time as the chief challenge to tracking metrics.
- Comparatively, 42 percent of the largest loan funds, by asset size, said that difficulty tracking data after products close is their top concern.
Conclusions and implications
This brief examines where CDFIs and the industry currently stand with collecting and tracking metrics and what they hope to achieve in these areas.
Most CDFIs track their activities and near-term outputs, such as the number of clients served and the dollar value of loans deployed. However, many CDFIs are interested in moving toward measuring the mid- and long-term outcomes of their programs and activities.
CDFIs track metrics for various reasons, most of which align with the dual goals of attracting and retaining funding and capitalization and informing organizational learning and decision making. The top CDFI goals in metric tracking vary by institution type and revenue models.
CDFIs with less capacity cited funding and capitalization as the top reasons for tracking metrics, whereas CDFIs with more capacity cited organizational strategy and decision making as top reasons. Resource and capacity constraints, as well as data collection and measurement challenges, present the most significant barriers to metric tracking for CDFIs.
Loan funds are most challenged by collecting data after products close. In contrast, credit unions’ top challenges are a lack of standardization in industry metrics and the high cost of data collection. The lack of staff time to collect metrics is most challenging for CDFIs with limited capacity.
The findings have several implications for CDFIs and industry stakeholders:
- Transitioning from outputs to outcomes requires resources and capacity building. Outcome tracking is resource-, capacity-, and time-intensive. Funders, investors, and industry stakeholders should provide financial support and help build capacity to assist CDFIs in transitioning from outputs to outcomes tracking. This work is not easy, but it will pay dividends in understanding the effectiveness of CDFI activities and increasing public awareness of the industry’s value and mission impact.
- CDFIs and industry stakeholders must address tensions between outputs-focused reporting requirements and each CDFI’s ability to track progress and understand mission impact. For some time, the CDFI industry has wrestled with the tension between developing industry-level standards and definitions versus providing space for the unique mission, strategy, and local context of each CDFI. Ideally, the industry should have the intellectual space and resources to do both: aggregate standard metrics that speak to industry-level impact while at the same time allow CDFIs the creative space, resources, capacity building tools, and support to develop and track metrics that align with their unique organizational goals.
- Funders, investors, and industry stakeholders should invest in capacity building to encourage growth and increase impact—particularly among smaller, capacity-constrained CDFIs. Data reporting requirements should not overburden smaller and emerging CDFIs. Instead, they should be encouraged and supported in developing frameworks that measure how their work effects change in communities and in aligning metric tracking with these frameworks. Industry stakeholders can support smaller CDFIs through capacity building tools, training, funding, and other resources.
Moving forward, Opportunity Finance Network (OFN), regional Federal Reserve Banks, and other CDFI researchers will continue to study how CDFIs demonstrate impact, their motivations for measurement, and challenges in IMM and program evaluation. Ultimately, the research aims to support industry growth that allows CDFIs to more fully achieve their mission of reaching financially underserved individuals and communities.
Acknowledgements
The authors would like to thank the following colleagues who reviewed and provided helpful feedback on this brief:
Opportunity Finance Network
Sacha Adorno
Alexander Carther
Brent Howell
Seth Julyan
Self-Help
Allison Freeman
Federal Reserve Bank of Richmond
Emily Corcoran
Sierra Latham
Deloitte
Jamie Mcall
Fahe
Katy Stigers
Authors
Endnotes
[1] In this brief, we follow McCall, Vivanco, and Smith’s (2023) terminology whereby, “Impact measurement and management (IMM) refers to a continual process involving methods and leading practices to collect quantitative and qualitative data. The reason for these data collection efforts is to track progress against targeted outputs and outcomes.” Jamie McCall, Eugenia Vivanco, and Adrienne Smith, “Five research priorities for community development financial institutions: Advancing financial inclusion through evidence-based practice,” Deloitte, Raza, and OFN (October 2023). See also Jo Barraket and Nina Yousefpour, “Evaluation and social impact measurement amongst small to medium social enterprises: Process, purpose and value,” Australian Journal of Public Administration 72, no. 4 (2013): pp. 447–58; Jenifer Mudd, “Impact measurement for CDFI small business lenders,” Technical Assistance Memo (Philadelphia, PA: Opportunity Finance Network, 2013); Hannah Jones, Vaughan Jones, and Juan Camilo Cock, “Impact measurement or agenda-setting?,” in Community Research for Community Development, ed. Marjorie Mayo, Zoraida Mendiwelso-Bendek, and Carol Packham (London, UK: Palgrave Macmillan, 2013), pp. 43–64.
[2] Program evaluation uses impact measurement data “to assess whether a product, program, or activity performs as intended and is consistent with the organization’s mission” (McCall, Vivanco, and Smith 2023). See also Jennifer Grafton, Anne M. Lillis, and Sally K. Widener, “The role of performance measurement and evaluation in building organizational capabilities and performance,” Accounting, Organizations and Society 35, no. 7 (October 1, 2010): pp. 689–706; Fred Mayhew, “Aligning for impact: The influence of the funder–fundee relationship on evaluation utilization,” Nonprofit Management and Leadership 23, no. 2 (2012): pp. 193–217; Martin Ravallion, “Evaluating anti-poverty programs,” in Handbook of Development Economics, eds. T. Paul Schultz and John Strauss, 4 vols. (New York, NY: Elsevier, 2008), pp. 3787–3846; Howard White, “A contribution to current debates in impact evaluation,” Evaluation 16, no. 2 (April 1, 2010): pp. 153–64.
[3] Malcolm Macpherson, “Performance measurement in not‐for‐profit and public‐sector organizations,” Measuring Business Excellence 5, no. 2 (January 1, 2001): pp. 13–17; Carolina Small Business Development Fund, “The economic impact of assisting small firms: entrepreneurship in uncertain times,” FY2022 Economic Impact Evaluation (Raleigh, NC, 2022); Jamie McCall, Eugenia Vivanco, and Adrienne Smith, “Five research priorities for community development financial institutions: Advancing financial inclusion through evidence-based practice,” Deloitte, Raza, and OFN (October 2023).
[4] Brett Theodos and Ellen Seidman, “From Compliance to Learning: Helping Community Development Financial Institutions Better Determine and Demonstrate Their Results,” Urban Institute (May 2017), p. 2.
[5] Williams, Teshanee, Jamie McCall, Natalie Prochaska, and Tamra Thetford. “How Community Development Financial Institutions (CDFIs) Are Shaped by Funders through Data Collection, Impact Measurement, and Evaluation.” Paper presented at the 51st Annual Meeting of the Association for Research on Nonprofit Organizations and Voluntary Action (ARNOVA), Raleigh, NC, November 18, 2022.