If you think the latter, that may be a problem in the future, when humanity merges with non-biological — or at least, questionably biological — technology. This won’t happen all be at once. One might add some artificial blood cells first, then upgrade the liver, enhance the nervous system… with each step, improving over nature. Is each step also a step toward exclusion from the ethical community? It would seem unlikely.

However, suppose we figure out how to completely virtualize our consciousness into a different substrate? The “uploading your mind to a computer” scenario. Identity questions aside, this would seem put us in the same realm as the self-conscious AI. So would your questions have more pertinence if the self-aware entity was once a human biological being?

I keep wondering how we could justify shutting down a self-aware entity?

What of its rights?

Are we obligated to help it survive? Indefinitely, since it would have no natural life-span?

Could we compel it to work for us without entailing slavery?

I think these questions are being overlooked in the hunger for more powerful digital tools.

We should be wary of the ethical ground we tread when creating an intelligence. My fear is that we will reach a crucial breakthrough before any real ethical guidelines are reached and before any resulting policies are enacted.

I fear the danger to ourselves, should we become happy slavers…