We do not call anything wrong, unless we mean to imply that a person ought to be punished in some way or other for doing it; if not by law, by the opinion of his fellow creatures; if not by opinion, by the reproaches of his own conscience. This seems the real turning point of the distinction between morality and simple expediency. It is a part of the notion of Duty in every one of its forms, that a person may rightfully be compelled to fulfil it. Duty is a thing which may be exacted from a person, as one exacts a debt. Unless we think that it might be exacted from him, we do not call it his duty.
John Stuart Mill, Utilitarianism (London: Longmans, Green, and Co., 1879), Chapter V, “On the Connexion Between Justice and Utility.”