Probably didn't hurt them. You might want to put a good coat of wax on them. The rough bits you feel are likely road sand that hit the wheels under normal driving conditions. If you don't see anything, I wouldn't really worry about it. The wheels are clear coated from the factory, but that clear coat can take a beating in normal driving.

 

I normally put a nice coat of carnauba paste wax on my rims, not only keep them shiny, but also to protect the clear coat.