Bo Ji and Brendan David-John were looking forward to getting a pre-ordered Apple Vision Pro headset. Playing with the new gizmo isn’t top of mind for the two Virginia Tech professors, though. Privacy concerns are their business.
Ji and David-John agreed it could be a “game changer” in the computing platform world. Apple says its new device, set for wide release on Friday, “seamlessly blends your digital content with your physical surroundings.”
The Virginia Tech professors and colleagues in the Commonwealth Cyber Initiative have already created a technology that would address what they say is a key issue with the Vision Pro and other virtual reality units: bystander privacy protection. Their development, called BystandAR, has proven nearly 100% effective in filtering out all but the most essential outside data from the immersive augmented reality world, they said.
Augmented reality, mixed reality and spatial computing are all terms for what David-John and Ji said is a way to see the real world in front of you, while the Apple set also offers simultaneous 3D games, multimedia and other computing options. It’s a step beyond “putting this headset on and just disappearing from the world around you,” David-John said in a video conference last week.
“This is something that you can use for productivity,” said David-John, an eye-tracking expert. “You can use it for work, but you can also use it at home. And [Apple wants] to kind of have you wearing that for a longer period of time and see the world around you.
“So they really market it — they don’t even like to use the words ‘mixed reality,’ they actually like the [term] ‘spatial computing’ … [with] the idea that you’re integrating apps and computing with the space around you.”
It’s not a new concept. Google Glass, for instance, was a pioneer in the field, but the $1,500 device got neither the sales nor the customer reviews to survive, and privacy concerns further hampered its marketability. Google quit selling and supporting its Enterprise edition in late 2023.
Apple, however, has an established customer base for its $3,500 release that Google didn’t necessarily have for take-home products, not to mention its own credit card to help consumers make just such a purchase.
“So I think part of this play is to be very strategic, and to kind of tap into their consumer base and get people to really integrate this with the real world and … just seeing where this fits in their life, as opposed to just very niche specialty cases,” David-John said.
Videos posted at Apple.com show a developer guiding a neophyte through processes that include pushing his thumb and finger together, pushing and pulling to call up screens from an array, then expanding and contracting them. Videos and panoramic photos seen on the glass can mimic the experience of being there, according to the videos. Users can adjust how much of the real world they want to see and can pull their laptop into the situation to get work done.
It all sounds very future-is-now, but long-standing issues that have plagued many in the digital realm might exist here, too. Privacy is among them.
Ji, a computer science and augmented reality expert whose recent research centers on AR and user privacy; David-John, who has been doing a lot of research on virtual reality headsets; and Virginia Tech graduate student Matthew Corbett spearheaded BystandAR to address what exists outside the headset. The Apple unit requires cameras, depth sensors, eye trackers and microphones to mix natural and virtual worlds. But those sensors can potentially pick up visual and other data from bystanders who don’t necessarily know it’s happening.
BystandAR, which the Commonwealth Cyber Initiative demonstrated last summer at a tech conference in Finland, deals with bystander privacy by distorting images and voices before integrating the information into the AR device — ensuring outsiders’ privacy while maintaining the user experience. BystandAR identified and protected more than 98% of people within the data stream, and didn’t offload unprotected bystander data to another device for analysis, the group found.
Google Glass and the more recent Facebook-adjacent Metaverse inspired the technology, which could also apply to newer developments such as Apple Vision Pro.
“The modern mixed-reality devices that make the Metaverse viable require vast information about the physical world and can also violate the privacy of unsuspecting or unwilling bystanders,” according to an article they co-wrote for the Institute of Electrical and Electronics Engineers’ publication, IEEE Security & Privacy.
The writers saw two key issues: “The first [are] the technical vulnerabilities present in modern [mixed-reality] devices. These vulnerabilities stem from coarse-grained permissions for third-party applications and the ability of these applications to offload raw sensor data and violate the privacy of bystanders.”
Ji discussed a theoretical situation in which an emergency room nurse using augmented reality to pull up patient data accidentally accesses information from family members or other patients nearby.
“So we really want to make sure that the visual data privacy of the bystanders can be protected in these kinds of scenarios,” Ji said. “So that’s what we did. We tried to use some key insights from psychological studies, saying that the eye gaze and also the voice are highly effective indicators of who is the subject, who is the bystander of the interaction.”
The group used an AR headset’s built-in sensors to determine and block out unwitting bystanders in real time, Ji said.
The new Apple device looks like the sort of thing that could be recording, David-John added, but Meta and Google Glass before it had sleeker designs that might not look like a recording unit.
“So how do we really consent to being recorded in that context?” David-John said. “And one of the things we tap into here is like, is there some sort of implicit consent? We’re having a conversation or making eye contact; we’re talking to each other. That’s the point where we kind of flip the switch and let the sensor see your face.
“By default [with BystandAR], we’re not going to see anything, we’re going to kind of essentially black out all those pixels. No way an app or even your first-party developer, if they do it the right way, can even access your face, being some random person in the background. But once you start to have that conversation, you’re kind of tapping into that social contract implicit consent that, hey, I’m interacting with this person; if they do have a headset of some sort on, they might be able to access some of my data and use that for whatever the application is.”
In the augmented reality world, email account names and passwords are just the beginning of valuable information, Ji said. Other security and privacy concerns include emotions, sexual orientation and product preferences, all of which a wearable smart device’s sophisticated sensors could infer as they analyze you in a mixed reality, Ji said.
All of that information could be stolen via hacking or phishing. Apple devices are typically loaded with security features, David-John said, but those wouldn’t be the only virtual-reality wearable units on the market now or on the horizon. In a world where social media accounts are routinely compromised, a spatial computing device could leave one vulnerable to various tricks.
“If users are struggling to protect their accounts in a Facebook environment, think about a social environment in VR,” David-John said. “I’m putting on a headset. This person looks real and they’re talking to me, but they have convinced me to go to some chat room somewhere. And they are going to find some way to steal my information or just phish out that other information. … It’s spatial computing, right? I can easily believe that someone is talking to me. And that’s different than just a chatbot. So maybe I will give up that personal information much easier if this person is in a 3D environment.
“Maybe they look similar to me. It knows my race and ethnicity and knows the sports teams that I like, and it shows up in a Buffalo Bills jersey, things of that nature. So I think those are the risks. And those are really not too understood at this point. There are some initial studies and initial thoughts, but that’s really the emerging ecosystem that we haven’t seen. Like you mentioned the scale of this thing, right? Maybe Apple opens us up to the scale of this and in a way that Meta hasn’t. That’s the developing piece right now.”
The goal, they said, is to have protective systems in place across the market by the time such devices are in widespread use. They have filed a patent based on the technology and are looking to license opportunities to such companies as Apple, Meta and Microsoft, Ji said. They expect that the companies would integrate BystandAR into their systems as application programming interfaces, as opposed to add-on software that consumers purchase, he said.
“These things are coming fast,” David-John said. “And we’re trying to get ahead of them from that side of it as well as developing the technical support for those systems.”
For now, the Apple headsets’ use will be limited to where Wi-Fi is available. Ji said the technology exists for headsets to operate on cellular data, but there is no demand yet, so most of the mixed-, virtual- and augmented-reality sets on the market are shipped Wi-Fi-only and don’t support cellular.
On a related front, Ji leads a group — including colleagues from Purdue University, Wayne State University and Ohio State University — that received a $1.2 million National Science Foundation award to improve the efficiency and performance of next-generation wireless networks. If their work goes well, they’ll create an environment in which people can take advantage of even heavier technology as it becomes available.
Headsets’ use in remote health care, virtual training, tourism and more are on the horizon. For such deeply immersive applications, a higher-speed data rate and ultra low latency (in which a network communicates with the least lag time) are requirements.
“So far I think in most cases [today], we are talking about maybe just one user or just a couple of users,” Ji said. “What I have in mind is that in the future, there could be applications where a large number of users will be involved. … Wi-Fi may not be sufficient to transmit this immersive content at a high data rate and low latency because you have a lot larger number of users. So in these kinds of scenarios, I think 5G or 6G or even beyond can play an important role here.”