"A privilege may not be a right, but, under the constitution of the country, I do not gather that any broad distinction is drawn between the rights and the privileges that were enjoyed and that were taken away."