Middle-aged[1] and older[2] men see their total testosterone levels decrease by 0.4% to 1.6% per year, many of whom had lower-than-average levels even in their 30s.[3] Bioavailable testosterone decreases by about 2–3% a year.[1]
Those numbers can be pretty misleading, though. Men approaching middle age tend to exercise a lot less and eat a lot worse. So it’s hard to say what a “natural” decline in testosterone looks like on a population-wide basis.
It’s kind of like saying muscle mass decreases 1–2% a year once you hit middle age. That decrease can have a lot to do with more time spent on work and family and less time spent trying to get ripped.