Royal Bank of Scotland will pay $5.5 billion to settle one of the two major U.S. investigations into allegations it mis-sold mortgage-backed bonds.