Your testosterone can decline starting your 20s, depending on how much stress your matrix is under. So, for women, that can start as early as 28. Usually your testosterone declines by about 1% per year.