A $320,000 house with 3.5% down is $11,200 at the closing table, plus maybe $7,000 to $9,000 in closing costs. That math, not the interest rate, is what most first-time buyers are actually staring at. Can I get in without borrowing from my cousin?
Lots of lenders will tell you yes. What they don’t tell you is that they’re saying yes to very different loans, and the gap between a good one and a mediocre one can run past $20,000 over seven years. Here’s how the low down payment world is actually put together, and how to find a lender who lives in it comfortably.
“Low Down Payment” Is Four Different Products
People hear the phrase and picture one thing. In practice, four programs do almost all the work, and each comes with its own credit floor, its own mortgage insurance, and its own fine print.
- Conventional 97 (3% down). Offered through Fannie Mae’s HomeReady and Freddie Mac’s Home Possible, or as a standard 97% LTV loan. Typically wants a 620 credit score. The income-capped versions come with cheaper PMI, which matters more than most buyers realize.
- FHA (3.5% down). A 580 score gets you the full 3.5%. Between 500 and 579, you’re looking at 10% down instead. There’s a 1.75% upfront mortgage insurance premium and an annual premium of roughly 0.55% that usually stays for the life of the loan unless you refinance out.
- VA (0% down). No monthly mortgage insurance at all, just a one-time funding fee of 2.15% for most first-time users.
- USDA (0% down). Restricted to eligible rural areas and capped by household income, with a 1% upfront guarantee fee and 0.35% annual.
The difference between 3% and 3.5% on that $320,000 house is $1,600. Real money, but not the deciding factor. The deciding factor is what happens after closing.
Conventional 3% vs. FHA 3.5%: Run Your Own Numbers
On a $300,000 loan, FHA’s upfront premium is $5,250 and typically gets rolled into the balance. Annual premium runs about $1,650. Conventional PMI might land anywhere from 0.4% to 1.1% of the loan annually depending on your credit score, and here’s the part that matters: it comes off automatically at 78% loan-to-value and you can request removal at 80%.
With a 760 score, conventional usually wins by a wide margin. At 640, FHA’s pricing is often better and more forgiving about debt-to-income. If you plan to stay 12 years and put 3.5% down, FHA’s permanent insurance can quietly cost you more than the higher rate you’d pay elsewhere. Ask any lender you talk to for both scenarios on paper. The one who can’t produce them is telling you something.
Not Every Lender Offers Every Program
This is where shopping goes sideways. Some banks only originate conventional loans. Some credit unions keep everything in-house and never touch FHA. Big call-center operations will push whatever their compensation plan rewards that quarter, and low down payment borrowers are easy to steer because they don’t know what’s available.
A lender with a big ad budget in your podcast feed isn’t automatically a bad choice, but budget and value are unrelated, and the ones spending the most on marketing rarely have the sharpest pricing. It helps to know how to sort the mortgage companies that actually deliver from the ones running a sales pitch before you hand over your Social Security number five times in one afternoon.
Mortgage brokers are worth a conversation here. A broker with 30 wholesale lenders can place you with the program that fits your file instead of fitting your file into the one program they have.
Where the Down Payment Money Comes From
Most first-time buyers don’t have 3% sitting in savings. That’s fine, because the rules anticipate it.
Gifts. Fannie and Freddie allow the entire down payment to be a gift on a one-unit primary residence, even at 97% LTV. You’ll need a signed gift letter and a paper trail showing where the money came from.
Down payment assistance. State housing finance agencies, cities, and nonprofits run thousands of programs nationally, many structured as a second mortgage at 0% that’s forgiven after a set number of years. These stack with FHA and conventional loans, but not every lender participates in them. Ask directly: “Do you originate with [your state HFA] DPA second liens?”
Retirement accounts. A 401(k) loan or a one-time IRA withdrawal (up to $10,000 penalty-free for a first home) can cover the gap. Lenders want to see the terms, and they’ll count a 401(k) loan repayment against your debt-to-income.
Seller concessions. Capped at 3% of the price when your LTV is above 90% on a conventional loan, and 6% on FHA. In a soft market, asking for 3% toward closing costs is normal, not rude.
One detail that trips people up: any large deposit gets sourced. Moving $6,000 from your savings account is fine. Moving $6,000 in cash from a shoebox is a problem. Season your funds 60 days before applying if you can.
The Numbers That Separate One Lender From Another
Your quoted rate is the least useful number on the page. What actually decides the deal is the combination of rate, APR, origination fees, mortgage insurance pricing, and how fast the shop can close. Two lenders quoting 6.5% can differ by $4,000 in total cost on the same house.
Request a Loan Estimate from at least three lenders on the same day, for the same loan amount and program, and compare them side by side. The five numbers that actually decide a lender comparison are worth memorizing before those estimates land in your inbox, because they’re buried in sections most people skim.
Underwriting Gets Stricter as the Down Payment Shrinks
Counterintuitive but true. Less equity means more risk to the lender, so expect closer scrutiny. Some programs want one to two months of reserves after closing. Debt-to-income ceilings often sit at 45% to 50%, and FHA’s automated approval can push higher if there are compensating factors like a long employment history or utility payments you can document.
Knowing what underwriters actually check takes the mystery out of a conditional approval, and it stops you from scrambling for documents at 9 p.m. the night before your rate lock expires. If your credit file is thin, meaning fewer than three active tradelines or a short history, look specifically for lenders who genuinely work with a thin credit file rather than assuming a 640 score disqualifies you everywhere.
What to Ask Before You Commit to Anyone
Four questions, asked on the first call, will sort the field fast.
- Which low down payment programs do you originate directly, and do you have access to down payment assistance in my county?
- Can you send me a Loan Estimate for both the conventional 97 and FHA options at my credit score?
- Who will service this loan after closing, and can I reach a human during underwriting?
- What’s your average days-to-close, and what happens if my rate lock expires because of an appraisal delay?
Vague answers on the first and last questions are a warning sign. Get the preapproval stage right and everything downstream gets easier, which is why choosing the right lender for preapproval deserves more of your time than browsing listings does.
Start With Your Own Numbers, Not a Lender’s Pitch
Before you call anyone, pull your credit reports, add up three months of bank statements, and write down your gross monthly income, all minimum debt payments, and the amount you could lose from savings without touching your emergency fund. Then calculate the payment at 3.5% down including taxes, insurance, PMI, and HOA. If that number clears your comfort zone by $300 or more, you’re in a strong position to negotiate, and you can walk away from a bad Loan Estimate with a clear conscience.
If it doesn’t clear, the answer isn’t always a bigger down payment. Sometimes it’s a cheaper ZIP code, a smaller house, or six months of paying down a car loan to drop your debt-to-income from 47% to 41%. Those moves cost nothing and often save more than shopping ten lenders for a quarter-point rate cut ever will.