Why are the banks leaving Britain?

What's happening to British banks are they going to all leave the UK and go to Europe so they can operate more easily across borders?  Or is this a silly scary story trying to promote the soft Brexit option for Britain.

Banks in Britain are used to "passporting", that is to say, they are used to being able to have branches in other countries and selling services across borders in the European Economic Area of the EU.  Fears that this could be at risk when Britain starts the negotiations on the terms of leaving the EU early 2017.  Is this a realistic fear, if so is it really something to be scared of. This video examines the motives behind Anthony Browne's (chair of the British Banker's Association, BBA) statement that many banks will relocate to the UK because this possibly faulty reasoning.