Refugees in Durban say they have nowhere safe to return to after being forced from a makeshift camp.