Unauthorised advertising banners which have been hanging from the facade of a Grade II listed building for about four years must be removed following a planning inspector's ruling.