Wearable fitness devices have demonstrated the capacity to improve overall physical activity, which can lead to physical and mental health improvements as well as quality of life gains. Although wheelchair athletes who participate in adaptive sports are interested in using wearable fitness trackers to capture their activity, we have observed low adoption of wearable fitness trackers among wheelchair athletes. We interviewed five wheelchair athletes and three physical and occupational therapists to explore fitness activities, experience with wearable technology, and potential uses for wearable fitness devices. None of the wheelchair athletes we interviewed had previously used any wearable fitness devices, however four out of five were interested in tracking their physical activity. We present five thematic areas helpful for thinking about wearable computing systems and accessibility challenges that arise based on incorrect assumptions about the athletic community. We highlight opportunities for improving the impact and accessibility of fitness tracking technologies for wheelchair athletes. These opportunities include improving the analysis of data from existing sensors, instrumenting the custom equipment used by adaptive sport athletes, and revising the language used in the presentation of fitness data to create a more inclusive community of users.