San Andres Island is, as Kathy stated, a required stop to satisfy the requirements of the Passenger Services Act. That Act required non-US flagged vessels departing from one US port but returning to another US port (e.g., your Panama Canal cruise) to stop at a "distant foreign" port. The only distant foreign ports in the vicinity of your cruise are Aruba, Bonaire, and Curacao (ABC Islands) and South America. Since San Andres is actually Colombian, it satisfies the requirements of the Act.
When we stopped in San Andres, the Captain put a tender in the water so that we could go ashore. San Andres is designated as a separate "country" by the Traveler's Century Club. Therefore, many of us wanted to make the trip ashore to get one more country. If given, the opportunity, I would go ashore if only for a short while. We stayed in the dock area but others did take cabs to the other side of the island.
PS The PSA is sometimes called the Jones Act. The Jones Act, much more recent, applies to the shipping of merchandise and not people.