Refinancings were popular in 2020, but not every household caught the wave.

From January to October of last year, only 6% of Black borrowers refinanced their mortgages, versus 12% of white borrowers. The findings appear in a new report by economists at the Federal Reserve Banks of Atlanta, Philadelphia and Boston.

Researchers matched borrower data from Fannie Mae, Freddie Mac and FHA loans with data from firms that track mortgage performance, including Equifax and mortgage-data firm Black Knight Inc. The report found that 14% of Asian borrowers refinanced, while borrowers who identified as Latino clocked in at 9%.

Of an estimated $5.3 billion of savings for all households that refinanced during the 10-month period examined in the Fed report, only $198 million, or 3.7%, went to Black households. Borrowers who refinance generally pay a fee in order to lock in lower interest rates and reduce the total cost of their loan.

Historically low rates combined with people spending more time at home led millions to renegotiate the terms of their mortgages, resulting in eight million refinances over the course of 2020, according to data from Freddie Mac.

The savings from refinancing can be significant. Freddie Mac estimates doing so could save nearly half of Black and Latino households around $1,200 a year.

While borrowers with variable mortgages may benefit when central banks cut interest rates, America’s most common home loan is the 30-year fixed mortgage, which requires going through the refinancing process to take advantage of lower rates.

The U.S. mortgage market involves some key players that play important roles in the process. Here’s what investors should understand and what risks they take when investing in the industry. WSJ’s Telis Demos explains. Photo: Getty Images/Martin Barraud The Wall Street Journal Interactive Edition

“It brings up an issue: When rates go down, who does it help?” said Sam Khater, vice president and chief economist of the economic housing and research division at Freddie Mac. “The Fed is in this era of monetary policy accommodation, but I would argue it benefits people who are already more well-off.”

Both practical and historical factors are likely holding Black and Latino households back from taking advantage of refinancing, according to Fed researchers.

For one, the closing costs associated with refinancing average about $5,000, according to Freddie Mac estimates. And because homeowners with less cash or wealth on hand roll additional costs—such as title insurance and origination fees—into their new mortgages, that debt can erode the benefit of the refinancing as it compounds, according to Lauren Lambie-Hanson, an adviser and research fellow at the Philadelphia Federal Reserve who worked on the report.

The pandemic’s economic fallout, which had a disproportionate effect on the employment and finances of Black and Latino households, has further complicated decision-making around making changes to their loans.

SHARE YOUR THOUGHTS

What should be done to address the racial homeownership gap? Join the conversation below.

“We know that in the pandemic, unemployment both spiked most for minority borrowers and it recovered more slowly,” making those groups more likely to go into mortgage forbearance, said Ms. Lambie-Hanson. She and her research team found that among borrowers who were current on their mortgages as of February 2020, 15.6% of Black borrowers missed at least one payment by February 2021, compared with 6.5% of white borrowers. Borrowers who enter forbearance are ineligible for mortgage refinancing.

“So there are these frictions that are a lot easier to navigate if you have a lot of cash,” said Ms. Lambie-Hanson.

The legacy of discrimination against minorities in the financial system, including a history of practices including redlining—denying or limiting financial services based on race—can discourage them from calling their lenders to ask about refinancing options or push for better rates.

Many minority families have developed a wariness of the country’s housing and banking infrastructure. That often results in a hesitancy to apply for programs that might benefit homeowners whose mortgages were hard-fought, said Benjamin Keys, associate professor of real estate and finance at the Wharton School of the University of Pennsylvania.

“For a lot of people, they may have struggled to qualify for their first mortgage and they may be able to make those payments, and so they think well, I don’t want to go through the ordeal of qualifying for a mortgage again,” said Mr. Keys. “I don’t want to have to have my credit record pulled again and scrutinized. I don’t want to have to have someone come by the house again and have it appraised.”

Earlier this month, to commemorate the 100-year anniversary of the 1921 Tulsa Race Massacre, the Biden administration announced new programs to address housing inequities, including an interagency effort to combat discrimination in the home appraisal process. The administration cited research from the Brookings Institution that found that Black-owned homes were undervalued by $48,000 on average, resulting in a cumulative loss of $156 billion.

“When you’re talking about refinancing, let’s be clear that home values, particularly in Black neighborhoods, don’t increase in the manner in which they should,” said Andre Perry, senior fellow at the Brookings Institution and author of “Know Your Price: Valuing Black Lives and Property in America’s Black Cities,” who co-wrote the report.

“It’s really discouraging when people try to refinance and they learn that their home values are lower than what they should be,” Mr. Perry said.

Write to Julia Carpenter at Julia.Carpenter@wsj.com