The industry seeks a toehold in a lucrative market, which could tie bitcoin and other digital assets more closely to traditional finance — and bring some risk for retirees.