As soon as votes were counted in the race to be Tory candidate for London mayor, Zac Goldsmith’s problem became clear. He had won comfortably, but just 9,200 party members bothered to vote — compared with the 80,000 who took part in Labour’s contest. Goldsmith praised his party for a ‘civilised and constructive’ debate, unlike the ‘divisive and vicious’ battle won by Sadiq Khan. But if Labour can call on a machine whose activists outnumber the Tories by nine to one, the Conservative candidate faces a real disadvantage.

The size of Khan’s vote, Goldsmith thinks, is deceptive and swollen by trade union members. But in May, he concedes, ‘They will be out en masse, combined with the Corbyn-istas. So yes, they know that this matters. They know that they have to throw the kitchen sink at it. And they will.’ He doesn’t sound like he’s looking forward to it.

Goldsmith’s first political battle came in 2010, when he won Richmond Park from the Lib Dems. He was tipped for greatness but, on arrival at Westminster, showed no interest in climbing the greasy pole. Instead, he tried to make it more slippery — campaigning for voters to have the power to sack their MPs and threatening to resign his seat if David Cameron granted a third runway to Heathrow. If he wins in May, he says, he’ll resign his seat anyway. ‘I don’t think that you can do both jobs.’

We meet in an upcycling shop in a West London council estate, where he’s talking to former soldiers who have started a business saving furniture from the skip. He remains a keen environmentalist and is standing on a pledge to make London the ‘greenest city on Earth’. But his priority, he says, is housing. ‘If you are disadvantaged enough to qualify for social housing, or wealthy enough to buy or inherit a home, you’re OK. But if you are anywhere in the middle you are more or less stuffed. That is a real problem for London and it will dominate the life of the next mayor — and the mayor after that.’

He has much praise for the outgoing mayor, but says that the prosperity Boris Johnson oversaw has spread unevenly. ‘The problem is that, over the last seven years, that prosperity wasn’t felt across the whole of London. So we are, undoubtedly, at a point now where Londoners have been priced out of their own city. It is like a giant social experiment and we don’t know where it’s going to end — but it’s not good news.’ So he may well position himself as a social avenger, despite being the son of a billionaire. Not that his background as a disadvantage. As he once put it: ‘I was born into a position of privilege and am, therefore, not corruptible.’

He describes his relationship with David Cameron as ‘very good’ — but it isn’t too good. ‘He never fails, when introducing me at events, to say that I have been a pain in the arse. But I don’t think that’s a bad thing. I’m not a pain in the arse for the sake of it. I only cause trouble where I think the government has got it wrong.’

He says that it is essential for the mayor of the capital to have a working relationship with the government. Londoners, he says, are ‘hopelessly infantilised’ because the city ‘only holds on to about 7 per cent of the wealth we generate. You have to beg for some of that back to pay for transport infrastructure, housing grants, police — all the things that London needs. I accept that it’s not a wildly sexy message, but the first line of any mayor’s job description is getting a good deal from government.’

He concedes that he has much work to do in selling himself. ‘Part of the problem I have is that I’m reasonably well known in my constituency and very well known in the Westminster bubble. But there’s a difference between someone knowing your name and knowing who you are. So my job is to meet as many people as possible in the boroughs that I have spent less time in, to build bridges. And to mobilise Conservative-minded people — because without them, I don’t have a campaign.’

When Lynton Crosby was called in to help Boris, he was aghast at how few Conservative-minded people were available. The activists, Crosby grumbled, were having their afternoon nap at the times they were needed to push leaflets through doors. Crosby then built up a machine that twice took Boris to victory in what is now a Labour city. Goldsmith is using CTF, Crosby’s firm, but the soon-to-be-Sir Lynton won’t play a hands-on role. Goldsmith wants to make the campaign his own, rather than try to recreate the Boris formula. ‘I’m certainly not setting out to become Boris Mark II; anyone who does that now is deluded. He is unique.’

Goldsmith speaks of his victory as being possible, but not probable — after all, he says, Labour won 400,000 more votes than the Tories in London in the last election so there’s quite a gap to close. ‘It’s going to be very, very tough,’ he says. ‘For me, and for the Conservative party.’