In the past few years Warren Buffett has faced some criticism for urging investors to follow his lead without acknowledging that his reputation and pocketbook allow him to cut far better deals than the average investor. From big-money investments in General Electric and Goldman Sachs Group in 2008 to Bank of America last summer, Buffett put his bankroll behind financial institutions under siege, but was able to wring out lucrative terms to do so.

While Buffett’s position is a benefit for him in such deals, there are other trades the Oracle of Omaha would love to put on but can’t do to the size of his investment portfolio. Buffett detailed one recently: housing.

On the heels of his annual shareholder letter – in which Buffett admitted to being “dead wrong” about the timing of a housing recovery – the billionaire told CNBC that distressed single-family homes were one of the best investment opportunities out there. Unfortunately for Buffett, the cost and logistics of making such an investment in large enough size to move the needle for Berkshire Hathaway is prohibitive. (His housing bets come in companies Berkshire owns like Clayton Homes, Acme Brick and insulation maker Johns Manville, among others.)

The challenges of scaling a bet on distressed single-family homes is a theme that has been echoed by other high-profile investors, from Buffett's fellow Forbes 400 member Sam Zell to hedge fund manager Bill Ackman, both of whom have voiced similar perspectives over the past year.

The problem for the folks managing billions of dollars is scaling the opportunity. Buying up the distressed properties is no small feat, but manageable. It is however, a real challenge to manage and service the homes as rental units, unlike say multi-family units in which oversight of a centralized property is far easier.

While the Buffetts and Zells of the world have been unable to find a way to execute such a strategy, investors like David Goldenberg are doing just fine with it. Goldenberg, a former trader at firms like Citigroup, Credit Suisse and Bank of Tokyo, owns a handful of properties on Long Island, which he manages himself, and which he says are generating a handsome return.

The four investment homes he currently owns were all bought out of foreclosure, and Goldenberg made considerable investments in repairs, from new roofing to rewiring. The tricky part is that it takes him several months from buying a home -- he has been focusing on two-family properties – until it is up to his standards and ready to be rented out. The time and effort is worth it though, as keeping tenants has not been a problem.

Goldenberg, who tries to cater to the $100,000-income household in Long Island – hardly upper crust in one of the wealthiest parts  of the country – says things have changed since he first started buying properties in 2009. At the time, “foreclosures were a dirty market,” he explains, and “some realtors wanted money under the table.”

While Goldenberg has found success with his investment properties, he also manages them himself, which basically amounts to a full-time job. For the Warren Buffett’s of the world, management is one of the biggest challenges to scaling up a housing investment that could impact a portfolio in the billions, and why an idea akin to a single-family home real estate investment trust is such a difficult proposition. (Multi-family REITs can enlist management companies to deal with dozens, if not hundreds, of units, nearly impossible on individual homes unless they are in very close proximity.)

Once the big players do get into the space – and you can bet they are considering it given the possibility of bulk sales of properties owned by Fannie Mae and Freddie Mac – the real lucrative opportunity may be for the managers who can handle residential properties.