Keep doing what your doing... carefully and it should pop off. Or you could take the wheel off and slip the entire unit out the back. I removed mine and they sit in the garage collecting dust and other stuff. I like being able to touch the hub when traveling. They are strictly cosmetic, they do look good.