I just went to a conference and attended an augmented and virtual reality session. This is not my topical area, but I thought it was an interesting area. Ran across a few things that I thought might be of interest to some of you.
Virtual reality (VR) is where you are immersed in an entire digital world. Occulus is an example of this, with those glasses that look like welders glasses. I tried out a pair from Facebook Reality. They had this game where your left hand had a red sword, your right had a blue sword, and cubes with slash direction marks on them came at you in a guitar hero fashion. Walls would come at you and you would have to side step them. It was fun, but the weight on the face was high. THey had a whole head harness. Also you sweat and steam up with this system.
Augmented reality (AR) is where you see the real world, and it is augmented in some fashion. People can see your eyes. For instance those of you who have watched the BBC Sherlock Holmes, where there are text labels showing up, they could do that to label things you can see in the scene. Or you could have a virtual assistant who shows you which way to walk on your maps route to destination.
There is a big push right now to identify virtual assistants (as opposed to voice assistants like Siri or Alexa) are going to be the next big thing, so a lot of players are getting into AR/VR. I have also listened to tech podcasts in which virtual personalities are being generated, that interact as digital humans and do not do mundane tasks for you. Facebook Reality was a big sponsor at this conference (their name was on the bag that they give at registration), and they have grabbed the lead guy from the Occulus project to head up their AR/VR team.
The top 3 big problems this research community faces are (1) resolution, (2) field of view, and (3) vergence accommodation.
Resolution is perhaps the easiest to understand. They are looking for 120 pixels per degree, which should let you resolve the New York Times held at arms length. They are not there yet, but this seems to be a pretty straightforward problem.
Field of view is the angle you can see from within the device. Right now they are at 140 degrees. The problem with wide field of view is while you can see a lot of things out there, if you try to actually look at anything in that field of view, you have to deal with pupil accommodation else you see nothing.
The last one is the hardest to understand, vergence accommodation. When you as a human look at something, you focus in and get fine detail in the area you are looking at, and that resolution fades as you go outside the region of interest. This is pretty important because it reduces the processing requirement of your optical system and brain to just restrict it to the region of interest. The brain will frequently ignore things outside of the region of interest, as seen in this famous selective attention experiment. One of the speakers had done a calculation that if you tried to have everything in focus, with the field of view and resolution requirements, you would pass so much data that the current would melt the wires. So they absolutely must limit the bandwidth with region of interest resolution.
The problem with a VR system is that if you try to look at something distant, everything is in focus, and if you look at something close, everything is out of focus. This is because they have a fixed focal length lens in the system set to image infinity. This is the vergence accommodation problem then- they have to create an optical system that not only can know what you are looking at, but also vary the vocal length to resolve the region of interest.
They have developed variable focus lenses, as well as multifocal lenses (which are in focus for some points at all distances), and they have tried to use both technologies at the same time with deep learning to resolve the problem, and it is very difficult.
Also the whole eye tracking problem is not very easy. Pupils vary across humans, and natural eye movements are hard to track, and people have occlusions in their pupils which throw everything off, and to do all of this in real time is pretty tricky for a large variety of users.
We humans are on a path to global myopia (near sightedness). By 2050, almost the entire population will have myopia. They attribute this to staring at hand held devices like smart phones, and there is also damage due to blue light.
But there is hope because if they actually solve this AR/VR problem, they can create active prescription glasses. Here these active glasses would track your eye to see what you are looking at, project an eyesafe laser to your retina to measure what corrective lens you need, then change an variable focus lens to resolve what you are looking at. This would beat the crap out of my progressive lenses since I need to tilt my head to bring what I want into focus, and these would let you look through any part of the lens and see with focus, and if your vision changed, it would adjust on the fly. This will likely be a byproduct of research aimed at AR.
The whole virtual assistant goal also needs to have scene reconstruction. The example they give is that say you are looking at a broken 3D printer. If you are say looking at the writing table, and that is your region of interest, and you ask the virtual assistant to help you solve the problem, it would not work. The reason could be that you used a wrong printing material, and you are not looking at it. So to give meaningful help the virtual assistant must reconstruct your scene, and this means a lot of cameras on the glasses. They demonstrated 6 camera glasses that could partially construct the scene and have a virtual assistant lead you to a desired office. The problem is that even with 6 cameras, they do not reconstruct the entire scene, and you have this sort of clipped reconstruction.
Therefore they are likely going to push for a 360 degree camera system as part of these glasses. The more points of view they have the more realistic the scene will be. And this byproduct could really have a lot of spinoff applications.
The AR glasses have gotten much smaller, looking like prescription glasses with a little dongle thing near the right temple. They are going to probably keep pushing packaging on this and make it much smaller. What is great about the AR is you have air flow and the weight is low, so it avoids the weight and sweat problem of VR systems like the Facebook Reality one I tried.
On the VR front, they have this idea of an embedded assistant. The example they gave is if you watch a movie in VR, you can dramatically enhance the experience by having a movie theater surrounding the screen in the virtual reality (think Mystery Science Theater 3000). So that surrounding theater is where you could put the assistant, without degrading the user experience.
Now there are very serious privacy concerns. First of all, if they can track your eye movements, they can see what you are interested in, and serve you up advertisements. This is data mining on a whole new level. Also if they have 360deg cameras sending data back, we have issues with that (consider the flap over police with body cameras). And worse, if they decide they can get an even better scene by sharing data across multiple users, big brother will indeed be watching you with no privacy possible. I think President Obama nixed the Google glasses owing to concerns over the cameras and privacy.
This explains why Facebook and Google are so interested in this stuff…there is a whole new world of data mining and advertising to exploit. I noticed that the speaker from Facebook Reality was very careful not to discuss this, and it was only a university professor bringing up this concern. We as a society certainly have a lot of ethical issues to resolve with data mining and privacy.
