Many Americans’ biggest worry is that they haven’t saved enough for retirement. A survey from the American Institute of CPAs confirms that: 41 percent of CPA financial planners reported that coming up short in retirement topped the list of their clients’ financial concerns. But if building a large enough nest egg is an important step, another is protecting those funds from being derailed. One big potential drain and a growing problem for seniors, according to the Consumer Financial Protection Bureau (CFPB), is student loan debt. The number of consumers age 60 and older with outstanding student debt quadrupled between 2005 and 2015. An estimated 2.8 million 60-somethings owe on student loans, with an average balance of $23,500.

Student loan debt can jeopardize your retirement security in more ways than one. If you think you may still be paying off education debt when you retire, here’s what you need to know.

A piece of paper with student loan debt written on it.

The Price You Pay for Defaulting

Social Security is an important part of the retirement picture for many seniors. Forty-five million retired workers and their dependents receive Social Security retirement benefits as of June 2017, with an average monthly benefit of $1,369. Fifty percent of married seniors and 71 percent of single retirees rely on Social Security for 50 percent or more of their monthly income.

The problem with having student loan debt in retirement is that your Social Security benefits can take a hit if you default on what you owe. The federal government can garnish your benefits, as well as your income and tax refund, to recoup defaulted student loan debt. According to the CFPB, the number of borrowers age 65 or older who had their Social Security benefits seized—or “offset,” as it’s called—because of defaulted student loans increased from 8,700 to 40,000 between 2005 and 2015.

Limited Remedies

Defaulting on your loans could spell financial disaster if Social Security is the centerpiece of your overall retirement strategy. Certain retirees may be able to have their student debt discharged if they’re totally and permanently disabled, but that’s not an option for everyone.

If you’re worried about compromising your Social Security because of defaulted student loans, you could look into income-based repayment once you retire. If your income goes down, having your student loan payments adjusted accordingly could make them easier to keep up with from month to month. The catch is that income-based repayment may have you paying on your loans longer, meaning you’ll pay more overall when the interest is added in. That may, however, be the lesser of two evils if the alternative is seeing your Social Security check shrink.

Shrinking Savings for Retirement

Of course, student loan debt can also cause headaches during your working years. The money you’re using to pay your loans is money that isn’t being added to your 401(k) or Traditional IRA, both of which can help you grow your savings while enjoying some tax advantages. According to the CFPB, the median IRA balance for savers ages 50 to 59 was $25,000 higher for those who didn’t have student loan debt. And retirees with student debt were more likely to skip out on necessary healthcare needs because they couldn’t afford them—which is hardly ideal.

If this is something you’re worried about, you should be taking every opportunity to save while you’re still working. A Traditional IRA, for instance, allows you to save up to $5,500 annually, or $6,500 if you’re 50 or older, as of 2017. These accounts allow for tax-deferred growth and contributions are tax deductible. Saving $5,500 a year from age 35 to 65 in a Traditional IRA that earns a 7 percent annual return would give you nearly $556,000 for retirement. And your savings could grow even more if you get an earlier start.

One thing to remember about Traditional IRAs is that you can’t make contributions forever. At age 70½, you’re required to begin taking minimum distributions from your account, based on your life expectancy. If you think you’ll be able to hold off on tapping your retirement savings longer than that, you may want to consider saving in a Roth IRA instead, which doesn’t require minimum distributions (though there’s an income limitation to have one and no tax deduction on contributions). A financial advisor can help you determine which IRA is better for your situation and assist you in developing strategies for managing your student loans so debt doesn’t ruin your retirement.

Compare Popular IRA Providers

Provider Fidelity Investments Merrill Edge
Name Fidelity Roth IRA Merrill Edge IRA E*Trade IRA
Description Get a range of investment choices, tax advantages and 1:1 help with a Fidelity Roth IRA Learn More Get up to $600 when you invest in a new Merrill Edge IRA. Plus one-on-one guidance, actionable insights and easy-to-use tools. Learn More Invest for retirement at E*TRADE. Learn More