Three days is not a lot of time on the south coast. If you arrive already knowing what you want and do not spend time deciding, you can see two or three places properly. The mistake most people make with a short trip is trying to see five or six places, which means you see all of them badly. Pick a tighter route and commit to it.
Here is the version that works best for most three-day visitors.
Day 1: Arrive and go straight to Galle. Check into accommodation inside or near the fort. Spend the afternoon walking the ramparts and the back lanes. The fort takes about two hours to cover properly on foot. Eat dinner inside the fort. Sleep in Galle.
Day 2: Morning in Galle, then move to Weligama. Walk the fort early before the tour groups arrive. By 10am or 11am, take a bus or tuk-tuk to Weligama, about 30 minutes east. Check in, have lunch, spend the afternoon on the bay. If you want to surf, this is the afternoon to try it. Most surf schools on Weligama beach offer walk-in sessions. Eat dinner in Weligama. Sleep in Weligama.
Day 3: Day trip to Mirissa and Ahangama, then head home. Take a morning tuk-tuk to Mirissa, about 15 minutes east. Walk to Coconut Tree Hill early and swim near Parrot Rock if the sea is calm. Back to Weligama for a late breakfast or early lunch. Then take a tuk-tuk to Ahangama, about 10 minutes west toward Galle, for an afternoon coffee at one of the cafes near Kabalana beach before heading back toward Colombo or the airport in the evening.
If you only have three days and whale watching is the reason you came, adjust this route: fly in, go straight to Mirissa, take the whale watching boat the following morning, then spend your final day in Weligama or Galle. The key is not trying to do both Galle and whale watching well in three days. Pick the one that matters more.
What this misses. Hiriketiya, Tangalle, Tissamaharama, Udawalawe, Polhena, the non-touristy inland spots. All of them. Three days gets you the surface of the coast, not the substance of it. If this is a scouting trip before a longer visit, that is a perfectly good use of three days. If this is your only time here, come back.