WaitButWhy’s Tim Urban says we must be careful with AGI because “you don’t get a second chance to build god” – if God v1 is buggy, we can’t iterate like normal software because it won’t let us unplug it. There might be 1000 AGIs and it could only take one going rogue to wipe us out.
