Basketball at the 2004 Summer Olympics took place at the indoor arena in the Helliniko Olympic Complex for the preliminary rounds, with the latter stages being held in the Olympic Indoor Hall at the Athens Olympic Sports Complex.

In each competition, 12 teams took part in the Olympics. The host nation (Greece) automatically received a berth in both tournaments. Winning in the 2002 World Championships also earned a berth for Serbia and Montenegro (as Yugoslavia at the time) in the men's tournament and the United States in the women's.

The remaining 10 spots in each tournament were allocated to the five Olympic continents. Each continent held a tournament to determine its entries. Africa received 1 spot in each sex, Oceania received 2, and Europe received 3. America received 3 places in the men's tournament but only 1 in the women's, while Asia received 1 in the men's and 3 in the women's.

Reigning gold medallists from the Sydney 2000 Summer Olympics in both the men's and women's event, the United States of America failed to retain its men's title, but successfully defended its women's title. Both the men's and women's gold medal games were held on August 28, 2004.