Refusing an unlawful order

The Nuremberg principles settled that acting on the order of a superior does not relieve a person of responsibility, provided a moral choice was in fact open to them. That clause is the interesting part, because it makes responsibility follow the possibility of refusing. An automated system executes or it does not, according to its specification, and a specification that provides for refusal is still the specification. Nothing is risked in the refusing. A person who refuses may lose a career, a rank or their liberty, and that cost is what makes the refusal worth anything, in an army, a hospital or a company. Human oversight written into a procedure is not the same capacity. Oversight becomes real only where the one overseeing can say no and pay for it.