Cooperative communication is a technique that can be employed to meet the increased throughput needs of next-generation WiMAX systems. In a cooperative scenario, multiple stations can jointly emulate the antenna elements of a multi-input multi-output (MIMO) system in a distributed fashion. Although distributed space-time coding (DSTC) is being considered by the IEEE 802.16j/16m standards for spatial diversity gain, it has several inherent drawbacks. These are addressed in the recently invented randomized distributed space-time coding, called R-DSTC. In this paper, we present the framework for the RDSTC technique in the emerging relay-assisted WiMAX network, and develop a cooperative medium access control (MAC) layer protocol, called CoopMAX, for R-DSTC deployment in an IEEE 802.16 system. Our scheme couples the MAC layer with the physical (PHY) layer for performance optimization. The PHY layer yields significant diversity gain, while the MAC layer achieves a substantial end-to-end throughput gain. Through extensive simulations, we evaluate the performance of CoopMAX and show that it can generate capacity gains of up to about 77% for an IEEE 802.16 network.