I just am not sure that there are random elements on these panels.
I think that it's not random at all - but that the only people jumping are Angel and Illyria. The house (Beverly Hills mansion)/place is the same. So we see it back in colonial times, and - in the Smile Time flashback - as it was before LA went to Hell. The people we see might be the original owners.