Category Archives: Articles

What’s it like to be a thousand years old?

by

2

Ever notice how as we get older, time passes more quickly? Lately, I feel like a week passes in what used to be a day.

I found an interesting page from the late 90s (its formatting ironically stuck in time) that tries to explain our perception of time as a function of age, mainly our memory of the past and expectation for the future.

Well before that, I’d based a whole science fiction story on a similar idea, extrapolating what it might be like to be perfectly healthy at a thousand years-old. At that scale, we might be able to see and communicate with things that are just too slow for us today, like genetically enhanced grape vines for example…

What kinds of relationships could we have between old and young, once the gap in ages gets into hundreds or thousands of years? AI Singularity or not, might we find it just too hard to communicate and relate among mere humans across such a wide gap in perceptual speed?

Logtime: the hypothesis that our age is our basis for estimating time intervals, resulting in a perceived logarithmic shrinking of our years as we age

Source: Logtime: Logarithmic Time Perception With Aging

 

Bonus: see the movie “The Man From Earth” for an even more interesting slant.

Palmer Luckey Wants to Build the Matrix

by

2

It’s worth remembering that virtual reality has never always been about gaming. Any real virtual reality enthusiast can look back at VR science fiction. It’s not about playing games … “The Matrix,” “Snow Crash,” all this fiction was not about sitting in a room playing video games. It’s about being in a parallel digital world that exists alongside our own, communicating with other people, playing with other people.

Source: Oculus Rift Inventor Palmer Luckey

Palmer Luckey wants to build The Matrix. I can totally relate. I wanted to do something similar back in my 20s, when it was called “Cyberspace,” the “Metaverse,” “the Other Plane,” or, for me, “Reality Prime.”

He’s a bit off base with a few things though. It’s not so much that VR died in the 90s, around the time he was born. The hype certainly died down, so there weren’t many media artifacts to review later. But VR thrived in many forms, including making billions in MMO gaming, sans HMDs. Immersive VR also survived at the very high end (e.g., CAVEs, etc..) for oil exploration, simulation, and more.

What really happened, apart from the H/W remaining too expensive for mass adoption, until cell phone demand drove component prices down, is that a lot of people working in VR realized that there were better ways to serve the world. In other words, we moved on to bigger and better things.

It’s nice, btw, that he gives a shout-out to his time at ICT. World-class inventor, Mark Bolas’ open-sourced HMD design was apparently instrumental for defining the first Oculus Rifts. Palmer may be aware of more design differences between his and Mark’s inventions than I am. Rift has certainly come a long way since then. It’s quite nice, if not quite done.

But what about this “Matrix” thing?

In the film, people are plugged into the global AI network, their realities (and bodies) controlled by mysterious AI entities with varying motives, all centered on control. We’re a long way from that in real life. But still, the analogy may hold for what Facebook, Oculus’ new benefactor, is already doing.

In the movie, there was a weak (IMO) plot device where the AIs were secretly exploiting humans as batteries. It’s weak because: thermodynamics. People are relatively poor transformers of food into energy. What about alternatives in geothermal, nuclear or fusion power, you ask? You have to just accept this bit of superficial fiction on faith. Fair enough.

However, if you replaced the idea of “people as batteries” with “people as wallets” connected to the grid, now you’re onto something, allegorically speaking.

It’s not energy that people collectively produce to benefit the AIs, but rather new/monetizable value, which can be dollars, attention or even new ideas and intellectual property, all fungible.

To many people working in big internet technologies, customers are already fairly abstract entities, never seen directly, but more like “wallets” and “personal data” plugged in at arbitrary endpoints. These customers somehow make things (real or virtual, doesn’t matter), they make money and charge-up their bank accounts, almost like batteries.

That’s not Facebook’s concern, for the most part, as it’s beyond their corporate capability to create so much original content and value at this scale. They can collect and connect it very well though.

But affecting how people spend their money and time is Facebook’s core business model. That is: influencing your “brand thinking,” consumption and spending habits with targeted and personalized content, especially ads, even selling your data to third parties who will. That’s their bread and butter.

To get these “internet attached wallets” to open up for advertisers for the maximum return on investment, Facebook needs your “personal data” to get to know you better. For that, they provide a socially compelling service that gets you to share your life freely without worrying your pretty little head about who owns that data you created or where it goes next.

So yes, in a strong sense, Facebook is a lot like those AIs who provide an immersive world for the humans to blithely live their lives while unwittingly producing a commodity the AIs need. The main difference is that unlike the Matrix, we don’t spend all of our time in Facebook — yet. But Facebook would very much like to improve that metric, using VR and companion mobile devices (chat, text, voice) as the medium.

In the near future, Facebook will know what you like (and want) simply by how you look at things or how you react emotionally, with no manual “like” button needed. They could continue to experiment on you, as they’ve done before, to mine your personality, and potentially even control you, most subtly, by conveniently filtering and mediating your interactions, social and otherwise.

If that doesn’t seem plausible, read my previous post about how it works. This isn’t science fiction. And it doesn’t require anyone with “bad intentions” either, just bad business models, and it will happen. The result is inevitable without adequate constraints, given the push to always make more money with a bounded set of people, roughly 7 billion. It will take much longer until Facebook gets into the business of making more customers. (I’m kidding, I hope).

To be clear, I think very highly of Facebook and Oculus’ engineering talent, product designers, and leadership. I have a healthy respect for their achievements and capabilities, which only adds to these concerns — if they succeed. They don’t seem to want (or believe) this outcome as conscientious individuals, and yet they’re already building it collectively, brick by brick.

So when people openly throw around that they’re inspired by and building towards “The Matrix,” then I think we need to ask even more emphatically about social impact and ethics and demand to know who will ultimately control this new power we’re unleashing.

Palmer is right. This is not about games. The stakes are so much higher than that.

What do you think?

 

 

 

 

How to Design a Product for Billions: 7 Lessons

by

3

It’s not often one gets to work on a product with a billion anything. I’m extremely fortunate to have had the opportunity to design/invent/iterate in the earliest days of at least three products in that league, and I’ve learned from others who have had more.

Here, I’m going to focus on Google Earth, which has well over a billion users, thanks to Google’s scale. I’ll pick examples from my experience to help illustrate the lessons.

Continue reading →

Verizon is either very dumb or they think you are

by

0

[as always, my personal technical opinions here]

Verizon Wireless sells out customers with creepy new tactic – latimes.com.

The LA Times author laments helplessly for most of the linked article. First, he claims Verizon is more evil than Google or Yahoo (who also effectively sell your data) because Verizon charges more up front. Well, I do agree companies that make money off our personal data should give us a cut or a discount, assuming we opt-in at all. It’s our data, right?

But the issue isn’t how badly Verizon is ripping us off relative to other companies. That’s a whole other box of lame.

Here the author gets closer:

Customers may be hard-pressed to understand fully what’s going on with the “enhanced” program. The Verizon Wireless notice is decidedly short on details.

Again, none of the companies listed in this post do a good job of “informed consent” IMO. If what Verizon is doing is not so bad, then people should be able to come to that conclusion given the honest facts, explained in terms they can understand, and then agree.

Were you aware of this? Probably not. Did you agree? Probably not.

But it gets better:

Debra Lewis, a Verizon Wireless spokeswoman, explained to me that when a customer registers on the company’s “My Verizon” website to see a bill or watch TV online, a “cookie,” or tracking software, is downloaded onto the customer’s home computer.

Most cookies are benign, allowing websites to provide better service to frequent visitors.

Verizon Wireless’ cookie allows a data-collection company working on Verizon’s behalf — Lewis declined to name which one — to gather information on which sites you visit after you leave “My Verizon.”

Hint: it could be any of these trackers, found by visiting the offending site with Ghostery installed.

  • Acxiom
  • Adobe Test & Target
  • Bazaarvoice
  • BlueKai
  • DoubleClick Floodlight
  • ForeSee
  • Gomez
  • Monetate
  • Omniture (Adobe Analytics)

You may want to install Ghostery and start blocking almost everything like this. These trackers do you no good, and you can put the few exceptions you need in the settings. You will be surprised to see how many tracking turds are quietly slipped into your pocket. The worst part is, if I go to Verizon’s site to find out how to opt out of this bullshit, I apparently get tracked for it…

Here’s the actual opt-out information, btw. FWIW, I collected the above list of trackers after ensuring I was opted out on the site. I guess they promise, on behalf of these random third party trackers, the data won’t be shared…

Sure.

But it gets even crazier:

That information is “anonymized,” Lewis said, to mask the Verizon customer’s identity and is then shared with marketers, which can use the info to provide ads on the customer’s Verizon Wireless device that match his or her home-computer interests.

What makes this all potentially class-action-worthy IMO (IANAL) is that Verizon claims that by simply replacing your name with a unique ID, the service is anonymous and therefore safe.

“In addition to the customer information that’s currently part of the program, we will soon use an anonymous, unique identifier we create when you register on our websites.”

Does anyone else remember when AOL put their carefully anonymized search logs out on the internet…

It took almost no time for someone to figure out a lot of who was who despite the obfuscation, because the same “anonymous” IDs were used for each search by the same person, over and over, and could therefore be accumulated and cross-referenced. Simple deduction, Watson.

How hard will it be for a 3rd party marketer, given a list of your most frequent locations and your “double secret anonymous” ID, to look up the address of said frequent locations, and a few ownership records of your most frequent location (where you spend roughly half your life) and forever associate your real name and secret ID?

When combined with other websites that you may sign into, how hard would it be to discover your other usernames, some already tied to your real name, and tie them all together into a single linked identity and activity file you never get to see?

Not at all. It’s already common.

There is an actual body of research behind anonymizing people and GPS coordinates that Verizon seems to be [negligently, IMO] ignoring. There are viable techniques to present targeted ads (based only on what Verizon knows about you) without leaking that data to any third party.

Any claims of effective anonymity after sharing this kind of data with third parties are IMO false and misleading. They’re counting on everyone being dumb. Don’t be.

 

The New Rift

by

2

This post by Peter Berkman gets closest to the meat of our collective concerns over the Oculus sale to Facebook. John Carmack even responded in the comments, while Palmer took to reddit. Raph Koster and Blair McIntyre do pretty good analyses in their own rights, putting the pieces together.

Disclaimer: I know many of the people mentioned here and consider them friends. I’m sticking to only what is publicly revealed information. And I am of course happy for these individuals who are getting to work on dream projects and making lots of money as a side benefit.

Here are some collected facts that are out in the open:

1. Industry Legends John Carmack and Michael Abrash have each talked publicly about wanting to build the Metaverse (of Snow Crash fame). I totally get what’s in this deal for them: build it with the right tech, the right people and at Facebook scale. Got to love the opportunity to realize that dream.

2. Cory Ondrejka (kick-ass VP of Mobile at Facebook) was formerly CTO of LindenLab/Second Life. He’s built a slightly less scalable version of the Metaverse already. Notice a “Mark, Chris and Cory” named in Palmer’s blog post. I have no doubt they’re completely sincere about wanting to see VR succeed in its own right.

3. Peter is also right in assuming that Oculus will add real gaze tracking. Palmer hints (skip to just before 19:00) about it here. Oculus is not alone, I expect. While there are many patents (of various quality) on this, there are still many viable ways to determine a user’s gaze, and many reasons to do so. Facebook has even more on its wishlist, I bet, beyond the basics of providing better visuals or more Natural UI.

4. Companies like Tobii make their living today by (effectively) enabling mind-reading via a user’s gaze. Your supermarket shelves were probably arranged with this kind of tech strapped to willing users. Your favorite website’s layout was probably tested using this tech in a lab. Just imagine the power of knowing what everyone is looking at and why, of being able to read subtle emotion in the face, including those micro-expressions that always reveal the truth.

5. Blair is also right that Zuckerberg’s AR long-term vision isn’t a typo. For all the arguments about VR vs. AR vs. camera-based vs. see-through being better for this or that, the bottom line is that VR sucks for mobile. That’s based purely on the old “wall meets face” principle, if not the geek factor of walking around talking to people with a brick over your eyes.

Facebook needs to win on mobile. Yes, it’ll take a few years just to get consumer-friendly seated VR right. But when Oculus eventually gets their glasses down to sunglass size and adds forward-facing cameras that mix real and virtual, then watch out[side]. Here’s the key part of Zuckerberg’s quote again:

But this is just the start. After games, we’re going to make Oculus a platform for many other experiences. Imagine enjoying a courtside seat at a game, studying in a classroom of students and teachers all over the world or consulting with a doctor face-to-face — just by putting on goggles in your home.

This is really a new communication platform. By feeling truly present, you can share unbounded spaces and experiences with the people in your life. Imagine sharing not just moments with your friends online, but entire experiences and adventures.These are just some of the potential uses. By working with developers and partners across the industry, together we can build many more.

One day, we believe this kind of immersive, augmented reality will become a part of daily life for billions of people. [emphasis mine]

6. I’ve heard Oculus folks also talk about telepresence as one of the killer apps for VR too, esp. when one can solve the “eye gaze” problem. You’ve probably experienced that in crappy modern video chat, which is generally how I’d imagine women feel when you stare at their chests. Interestingly, Cory’s old boss from Linden, Philip Rosedale, has announced he’s working on the eye gaze problem in his new startup too, and using some cool software called FaceShift to get a jump on the harder problems there.

[For full disclosure, I did some work on avatar-based telepresence as well, but I wasn’t entirely satisfied with the result. FWIW, I found a better approach, which works with no hardware sitting on your face.]

7. Finally, Facebook makes its money by selling its users. Let’s not be coy. It’s a lot of money and a lot of users. To be fair, is it really any worse than how NBC offers up its users to advertisers (since before I was born)? Well, no and yes. It’s the same “ad, ad” world as before. We learn by age four to distrust the loud man on the TV. But the key difference is that NBC can’t see you, can’t really know your thoughts, except via gross statistics (think Neilsen ratings). Facebook really wants to know you, individually, and forever.

Now, companies like Facebook and Google provide immensely desirable services for free too, more so than NBC IMO. The problem lies in the concept of “informed consent.” Both tech companies still suck at the human stuff IMO, esp. in terms of giving users clear information up front to make that decision wisely, and then giving full control of their data after. It’s still mostly one-sided today, and it has to change.

Ideally, they’d start to emulate companies who’ve adopted an obsessive customer focus. It’s not about what people will tolerate, but what they really need. Earning our trust can’t be an afterthought or a win by attrition.

[for a positive comparison, TV’s “Neilsen families” give truly informed consent and the company promises not to market to them as well.]

So we come back to Peter’s insightful post. The problem with the Oculus sale is not that we all supported Palmer’s fortuitous kickstarter and got no share of the spoils. I got the product I ordered, still sitting in its box, alas. It’s not that he and Brendan Iribe “sold out” the indie gaming dream for big bucks. There are plenty of reasons to seek this level of protection and financial support in the light of Sony (and others) gearing up for a big fight. Gamers should be happy to have more viable options to choose from at the lowest prices.

Hell, I joined the so-called “evil empire” in 2008 to try to use their massive scale to do some good in the world, so who am I to judge? (jury is still out on my contributions, fwiw)

No. The heart of the problem is that VR is the most powerful means ever invented to pipe external ideas into your internal world. It’s pushing remote-controlled information almost directly into your brain via every sense possible. Consider that people believe a lot of crap they can’t even see. You’ll believe this crap on so many levels. That’s why it’s called virtual reality, ok? It’s as close to real as it gets, while still being entirely artificial.

The Facebook purchase highlights the reverse of that flow as well, reminding us that VR may become one of the best ways to pipe your internal world out — via data mining, classification and onto untold dissemination. That’s actually one of the reasons I got into the field, 20 years ago: to more easily tell stories that were rattling around my brain that would have taken millions of dollars and hundreds of people to produce as movies or games. But it has a dark side too, like when the information is sucked out of us without our informed consent or control.

Just think what the headlines would have said last week if Facebook had instead bought a brain-sensing startup like Emotiv, or invested in fMRI brain scanning tech to extract your thoughts. See what I’m talking about? VR doesn’t work quite the same way as those, but it gets to the same place in the end. The protective walls between you and the world come down in favor of more bandwidth in and out.

So closing that loop, even crafting individually designated virtual worlds using all of your private information, Facebook will own the most potent means available, short of mind-control drugs, to read and write to your private inner world, your thoughts, your actions, your dreams. It can free you, or it can enslave you.

What they do with it is entirely up to them (plus certain market and “other” forces). And if that doesn’t scare you, at least a little, then you may already be sold.

 

Head Gears

by

0

Most head mounted VR gear brings me back to my teenage years, where my orthodontist tried to make me wear a night brace to straighten my teeth.

It works while you sleep, he said.

You try sleeping with your head in a vice, I said.

He didn’t care. He got paid $40/month regardless of how long it took. Needless to say, I soon got a new orthodontist. And I’ve kept trying on various VR gear too.

I originally favored the CAVE projection kind of display and built a few variations of my own, including a six-sided one at Disney. The main benefit is zero latency (ignoring stereo parallax changes) — in other words, the image is already there when you turn your head. No blurriness. The main downside, of course, is who has room or money for an 8′ cube in their living room. Not practical until we get digital wallpaper or big flexible roll-up screens.

But even still, I happily bought into the Oculus Rift’s kickstarter, eager to try again. I love seeing people so enthusiastic about this stuff, especially new blood.

Though I’ve personally used the Rift for many minutes at a time, my own purchased dev kit is still sitting in its box, alas, waiting for me to find time to build something useful. The head-tracking latency was actually very good, but the original display felt much like the world was made of LiteBrite. If you’ve never tried it, here’s a good oculus rift simulator to try.

I just pre-ordered the 2nd gen dev kit too, which fixes much of the resolution issue, and I’m sure comes in an even nicer box.

I’m hopeful that Carmack can solve some of the rendering latency issues that fast OLED displays alone can’t. I did some research in this area too, fwiw. There’s a lot that can still be done to wring the delays out of various pipelines.

Some friends and I also got to try out the new Sony Morpheus HMD at GDC this week. We had to get in line the moment the expo doors opened, just to get a ticket to stand in line to wait to try. But it was worth it, I keep telling myself.

The resolution was impressive. The persistence of their LCD displays was not as good as the Rift’s. Now, I can’t be sure what they’re using inside, but I would have thought they’d throw some 4k SXRD panels in there, just like they use in their nicest projectors.

I thought those were akin to DLP in terms of super-fast switching time, but I’m not so sure anymore. Maybe there isn’t room for front-reflection in the optical path. I can say that the LCD-like images we saw seemed to be over-driven and washed out a bit, mostly suffering from slow switching times. Brightness was great, but black blacks were in short supply.

In any event, no one got nauseated, which is a small victory for those of us who can’t watch the Blair Witch Project without dramamine. The Rift also does well on that front. But in both cases, using a simple laptop trackpad or arrow keys to navigate puts me back on the vomit comet.

A nice omni-directional treadmill might do the trick. Someone had one of those on display too, but I didn’t get to try it — seems to need special slippery shoes. But if we’re going through the trouble of a 4′ treadmill, why not just go back to using CAVEs? If it’s just a matter of integrating your furniture, my friends in MSR solved that nicely.

For what it’s worth, I still have my money on real see-through AR displays as the ultimate winner. Let me walk in the real world, augmented with new content. Yes, indeed.

 

Whats Amazing?

by

1

It’s not the $19B price tag of Facebook’s WhatsApp acquisition that’s most amazing to me. That’s just math (lots of math). More users * more engagement * a wealthy/desperate suitor = bigger deals.

It’s not most amazing to me that the deal works out to over $250M per employee, if evenly split. For comparison, Google seems to be worth about $8.5M per employee, last I checked. Apple reportedly pulls in a whopping $2.3M yearly per employee in revenues alone. (average Apple employees, how much of that do you get paid again?).

WhatsApp just did more and faster with fewer people. That’s what makes a great team. I’m sure everyone was amazing. Dysfunctional teams or people barely earn back their salaries, if they survive at all.

The motivation for the deal is not amazing at all, since Facebook desperately needs to connect the very people who don’t need Facebook to connect, i.e., the people you see every day. It seemed almost inevitable, given the trends.

No. What is most amazing to me is the powerful principle behind this simple note at right — one of the core principles that apparently made WhatsApp so popular with hundreds of millions of users. I found this via the Sequoia blog post about the sale, and I honestly had no idea.

Jan keeps a note from Brian taped to his desk that reads “No Ads! No Games! No Gimmicks!” It serves as a daily reminder of their commitment to stay focused on building a pure messaging experience.

It’s most amazing to me, and most inspiring, because there are so many professional CEOs and advisors out there who try to convince their startups that they need to collect and hoard as much user data as possible, then sell it surreptitiously, while also pushing ads and wringing every last monetizable cent with games and gimmics to keep people addicted and virally engaged.

Those same fine folks somehow duck out (or get fired) when the users finally complain, defect and disappear. And they apparently never learn from their mistakes. But they do come back, with the same tired story again and again.

WhatsApp proved them wrong and proved it 19 billion times over.

Build value for users. Give them what they want and need, every day. That’s the recipe for success. This kind of success is something I can truly appreciate and admire.

 

P.S. I’d like to think that this is the real reason Facebook bought them, given FB’s reliance on those same tired ads, games, etc… Maybe it wasn’t for more European and emerging market penetration. Maybe it wasn’t merely to disarm a growing competitor. Maybe it’s for Facebook to become more like them? Maybe that’s why they put Jan on the board. If so, good for them.

When You Fall in Love, This Is What Facebook Sees

by

0

When You Fall in Love, This Is What Facebook Sees – Robinson Meyer – The Atlantic.

I’ve heard that credit card companies know when you’re cheating or about to get divorced. I’d imagine things like flowers, gifts and hotels would be a tip off, but general expenses probably approach double once a couple is separated.

This data about facebook posts makes perfect sense too. People using FB will post more flirtatiously until the relationship starts. But after, why post so much? Even in real life, it’s rare for couples to make strong public displays of affection once they’re together. If they do, it may come more from insecurity about the relationship, or insensitivity to others’ feelings, than from some unparalleled eternal flame.

This highlights one of Facebook’s core challenges — how to capture sentiments shared between people who spend a lot of time together.

Holographic Phone: Stupidly Brilliant

by

0

holographic phone

This Is The Coolest Augmented Reality Idea Yet For Using Your Smartphone | Gizmodo Australia.

This idea reportedly comes from a competition that Meta (Space Glasses) is holding. The idea is to project a holographic image of your phone in space (using said glasses) and let you virtually interact with it, instead of taking the phone out of your pocket to do exactly the same.

Why is it a brilliant idea?

It’s simple. People get accustomed to their phone’s UIs. Projecting the phone holographically requires not a single new thought and changes nothing about the core experience. Well, it does lose out on touching that sleek and sexy touch-screen, feeling the nicely balanced weight of the phone in your hand, and of course key sensors like accelerometers (to a degree) and cameras (at all) to certain phone experiences.

So I guess that means you couldn’t run old augmented reality apps on your holographic phone for a recursive experience. Oh well. There goes a nice photo op.

Why is this a stupid idea?

Your head mounted device can [eventually] paint pixels anywhere you look. It can detect touch anywhere it can see your hands. Why would we limit ourselves to drawing a 4″ screen when we have an infinitely large screen on our head?

It’s a lot like saying, “Hey, we got used to small CRT TVs so let’s draw a small TV inside our brand new 60″ flat screen TV so people don’t have to learn something new.”

Interfaces for AR will run the gamut from holographic virtual actors who become your daily assistant, to making every physical surface in the world potentially interactive by touch, sight and sound. Why would we limit ourselves to UI mechanisms that were designed around the limits of small screens and touch?

Just for the experience of not having to take our phone out of our pocket? Are we really that lazy? If so, ask yourself how much you’d be willing to pay to use your phone without taking it out of your pocket. I’d pay maybe $1.

This really comes down to a core question about AR. Is it about being the ultimate hands-free device, principally meant to deliver us from holding our phone in our hands or up to our faces? Or is it about re-imagining the analog world with new digital layer(s) of content on top?

I can see an app like this being very popular, at least in the way the fart app is popular. That’s only because people’s imaginations are presently too limited. They just haven’t seen the best ideas yet.

On the other hand, it’s turning out that the most popular interface for your new 60″ flat-screen TV with billions of streaming video options is not some new fancy XBox-like natural UI, but rather just your phone.

So what do I know? People may ultimately find ‘stupid’ brilliant.

 

More Magic

by

0

The idea that most man-made objects can be represented with sweep surfaces (cylinders, tubes, squares, etc..) isn’t that new. Second Life primitives used exactly the same principle, with some interesting extensions for cuts, twists, tapers and so on.

But selecting photographic imagery based on implicit primitives, in-painting (hallucinating) the background and unseen object views, and (occasional) relighting of the object is all extremely clever and very useful. Combine this and a system that can relight virtual objects based on scene shadows and you have a paint program that can revise reality, at least virtually, but in a way that would fool almost anyone.

The end-goal of all this work is something I used to call “parametric 3D video” — which roughly means we take one or more 2D video streams, split out the objects, backgrounds, people into separate and fully adjustable pieces, send them as 3D content vs. pixels, and then re-synthesize the result from any angle at the receiving end, along with any changes you want to make.

3D (color + depth) video capture makes the problem much easier. Techniques like this paper are still needed to finish the job, but they can be much more automatic in terms of finding and cutting objects.

 

▶ 3-Sweep: Extracting Editable Objects from a Single Photo, SIGGRAPH ASIA 2013 – YouTube.