Germline stem cells (GSCs) produce gametes throughout the reproductive life of many animals, and intensive studies have revealed critical roles of BMP signaling to maintain GSC self-renewal in Drospophila adult gonads. Here, we show that BMP signaling is downregulated as testes develop and this regulation controls testis growth, stem cell number, and the number of spermatogonia divisions. Phosphorylated Mad (pMad), the activated Drosophila Smad in germ cells, was restricted from anterior germ cells to GSCs and hub-proximal cells during early larval development. pMad levels in GSCs were then dramatically downregulated from early third larval instar (L3) to late L3, and maintained at low levels in pupal and adult GSCs. The spatial restriction and temporal down-regulation of pMad, reflecting the germ cell response to BMP signaling activity, required action in germ cells of E3 ligase activity of HECT domain protein Smurf. Analyses of Smurf mutant testes and dosage-dependent genetic interaction between Smurf and mad indicated that pMad down-regulation was required for both the normal decrease in stem cell number during testis maturation in the pupal stage, and for normal limit of four rounds of spermatogonia cell division for control of germ cell numbers and testis size. Smurf protein was expressed at a constant low level in GSCs and spermatogonia during development. Rescue experiments showed that expression of exogenous Smurf protein in early germ cells promoted pMad downregulation in GSCs in a stage-dependent but concentration-independent manner, suggesting that the competence of Smurf to attenuate response to BMP signaling may be regulated during development. Taken together, our work reveals a critical role for differential attenuation of the response to BMP signaling in GSCs and early germ cells for control of germ cell number and gonad growth during development.