Collateral ancestor is a legal term referring to a person not in the direct line of ascent, but is of an ancestral family.  This is generally taken to mean a brother or sister of an ancestor (hence a "collateral ancestor" is never an ancestor of the subject).

Collateral descendant is used to refer to a descendant of the brother or sister of an ancestor.

The general way of referring to a blood relative who is not an ancestor or descendant is as a  collateral relative.

Cf. Allied Family