My parents told me this every time we went past it when I was growing up.

https://londonist.com/2016/06/why-does-apsley-house-have-the...

It looks from Wikipedia that that's a popular name rather than official one.

The house was given the popular nickname of Number One, London, since it was the first house passed by visitors who travelled from the countryside after the toll gates at Knightsbridge.[3] It was originally part of a contiguous line of great houses on Piccadilly, demolished to widen Park Lane: its official address remains 149 Piccadilly, W1J 7NT.[4]

However, if you posted a letter it would probably get there if the Post Office were determined enough.