[input-output-hk/cardano-sl] [CBR-389] Don’t construct WalletAddress IxSet (#3439)

I have no idea what these «cached failures» are. I don’t think they are related to this branch (indeed, CI didn’t even seem to have run..?).