From what I have seen, it's a matter of access mostly. I did not learn to swim till I was nine. I was able to learn because I had access at a new middle school who had a pool(PS 145). I had a hard time at first for I could not float to save myself, even when I filled my lungs fully. but I could swim under water the full length of the pool. I do think our bone density is a factor for some, as is fat ratio. I was a wire for most of my life.
I did keep swimming when ever I got a chance, I swam for one year in high school, but even the coach did not like me. I became a scuba diver at 15, and have gone diving to many parts of the world. At eighteen I was even able to free-dive to 90 feet. So my take is that we can do it, and even excell at it.
http://faculty.ucmerced.edu/sites/default/files/kdawson4/files/swimming_surfing_in_africa.pdf
Please read the above, you will be very surprised indeed.
AXE
West Africans by the sea swam better then the Brits, read above.