Say you’ve got problems at work. Boss putting too much work on you, but diplomacy isn’t your strong suit? Shy about asking for a raise? Or maybe you’re a boss who’s not great at communicating needs without sounding heavy-handed?
Maybe your employees need some motivation, but you’re not beyond the “van down by the river” oration stage?
As the kids used to say: There’s an app for that.
Or at least there might be, depending on tests that a couple of Virginia Tech researchers are conducting.
Eugenia Rho, an assistant professor in the computer science department, and Lance Wilhelm, who is pursuing his computer science Ph.D. there, have teamed up for an AI-based conversational role-play site called CommCoach.org. Its chatbot-style interface offers customizable scenarios and coaching that Rho and Wilhelm hope can help people talk better in workplaces.
The two have gathered a group of volunteers who are testing CommCoach in twice-weekly sessions.
“One of the primary interests of our lab is looking at how people communicate, where communications break down and what is the role of technology in potentially assisting where … breakdowns happen,” said Rho, whose pre-academic life included work on Wall Street after growing up in four different countries.
Wilhelm, an active duty Air Force major with a bachelor’s degree in physics from the U.S. Air Force Academy and a master’s in analytics from Georgia Tech, said his experience with multiple levels of the service’s “leadership laboratory” gave him the idea of sharing such opportunities with everyone.
“Sometimes technical experts are thrown into leadership roles because of their performance because they’re stellar, but they don’t have that background and experience in leadership,” Wilhelm said. “And now they’re asked to lead teams. That’s just one example.”
He and Rho wondered if they could build a model that gave people the space to “fail … in a safer environment,” he said. “But more importantly, we wanted to know what are the upsides, what are the downsides, what are the blindsides that we may be missing, or what we need to pay attention to and be cognizant of when we go to deploy these technologies to real people in real workplaces.”
Their team started their work with an interview study, looking at leaders in professions including military, academia, finance and government, Rho said. They asked how those bosses used AI and how they would envision a tool to help improve their communication. After they published that study last year, they began working with organizational psychologists and executive coaches to build CommCoach and learn how people would engage with it.
“We found in our first study that users really wanted to bring their own situations, their own contexts,” Wilhelm said. “They maybe even wanted to, especially in role play, customize the interlocutor, the training partner, to give it some personality traits to be able to do that.”
The researchers built out their model to ground it within leadership qualities while providing areas for the system to adapt to user experiences. Large language models are already trained on massive amounts of information about leadership, communication theory and the like.
Beyond that, the team is relying on the study participants and the scenarios that they want to practice, which the AI can then adapt and use in the future, including levels of difficulty and intensity of interactions over a 10-week, twice-weekly course.
“So the AI’s adaptation, in terms of how it was developing subsequent training content and objectives, those became the object of the research study itself,” Rho said, adding that “there was never a definitively correct answer other than monitoring for safety and making sure that it’s not spewing incorrect advice.
Wilhelm and Rho have access to the back-end data and said they can monitor and flag, if necessary, any troubling large language model responses to users. As with any new user interface, there is wonkiness, as well. A text screen providing a theoretical boss’s interaction with an employee, for example, was layered behind the user screen in such a way that one couldn’t see all of the words.
Wilhelm said he responds to emails about problems with the system, while keeping his own eye on the system to make sure it’s running smoothly.
Users didn’t have to share details about their jobs, and could instead create personas to practice with.
In a test run, the digital coach came across as friendly and encouraging, making solid points about how to handle such matters as a boss dropping an extra assignment with a tight deadline while you already are approaching another deadline. (Note: Such a scenario is not part of the daily experience at Cardinal News. 🙂 ) Coach does not refrain from offering improvements.
In one response, it says: “The conversation *feels* resolved, but you may have traded one context-switch problem for another. Next time, before you commit to a tight timeline, pause and say something like: ‘I can do it today if we’re clear that this means X gets pushed to Friday — is that acceptable?’ That keeps you in the driver’s seat rather than just absorbing the squeeze.”
While the work was focused on top-down communication, the researchers have learned that people are interested in upward conversations and peer-to-peer talks.
They have yet to collect all the data from the ongoing study and aren’t sure whether this AI HR tool will ultimately work. Wilhelm said his outlook is a mix of skepticism and techno-optimism, but as he conducted interviews for the first paper, he heard multiple perspectives that included apprehensions and blind spots — such as whether a bot could provide emotional realism and adapt to different user work situations.
Data privacy was a concern as well, Wilhelm said.
“And so I think I now have come to appreciate that we need to understand this technology better before it’s then deployed,” he said. “And that’s the goal.”
With anything AI-related, users are often bound to encounter bad information, poor advice, even so-called “hallucinations” that don’t help at all. The researchers are keeping eyes open for those, as well.
“People are probably going to use these systems for that. In fact, there are already applications out there that are using AI as coaches. So I think then it may come onto us as researchers to provide some transparency as to the pros and cons of using those systems, and then, hopefully, maybe influence those actual deployed systems to be safer, more effective, things like that.
“This is very much a work in progress, as you can see.”
