The impartial spectator is a type of high-level consciousness that transcends the ego (self-love). You can equate it to a'moral sense'[prajna-paramita in Buddhism], just like ‘vision’ is a visual sense and ‘hearing’ is a auditory sense. ‘Conscience’ is then derived from this moral sense:
These must be left to the decision of the man within the breast, the supposed impartial spectator, the great judge and arbiter of our conduct.
It is strange that this sentiment, which Providence undoubtedly intended to be the governing principle of human nature, should have been so unnoticed that it has not gotten a name in any language. The word ‘moral sense’ is very recently formed. It cannot yet be considered as part of the English tongue. The word ‘conscience’ does not immediately denote any moral faculty by which we approve or disapprove. Conscience supposes the existence of some such faculty.
Though this ‘moral sense’ is present in all humans..:
..it is not readily or so easily activated, because it is very often clouded by ego or self love (ahamkara in Sanskrit):
He soon identifies himself with the ideal man within the breast. He soon becomes himself the impartial spectator of his own situation... If he wants the impartial spectator to enter into the principles of his conduct he must humble the arrogance of his self-love..
The concept of impartial spectator is derived from David Hume..
..though the Hindus defined it first, to be later developed by Buddhists, creating guidelines and practices to develop the moral sense.
An Arab man experiences injustice. By watching nationalist propaganda, his mind connects Western imperialism to the injustice. His strong feelings increase his ego enough to cloud his moral sense to become a terrorist and successfully attacks a Western target. The Western soldier sees this and his moral sense guides him to strike back at the terror camp. However, his ego forces him to rush the job and hit innocent people (collateral damage). Those victims suffer injustice and sees the moral duty to strike back. In all cases and without exception, the ego (self love) is the cause of the corruption of moral sentiments as it prevents the mind from putting itself in the situation of others (fellow feeling):
However, it is also the great and most universal cause of the corruption of our moral sentiments. Moralists in all ages have complained that wealth and greatness are often regarded with the respect and admiration due only to wisdom and virtue, and that poverty and weakness is unjustly given the contempt due only to vice and folly. We desire to be respectable and to be respected. We dread to be contemptible and to be condemned.Theory of Moral Sentiments, Part 4