The AI driving how we experience the US Open

The US Open is underway in New York and on the surface that doesn't seem like a Jetpacks kind of topic to cover. But I've been working with IBM recently on a few projects and the idea came up to do a sponsored episode digging into their work on keeping the digital wheels turning behind the scenes at major sporting events all around the world.

So today I'm speaking with Dane Gambrill, Tech, Media and Entertainment Integrated Accounts Manager for Global Markets at IBM, to learn more about everything from the raw engineering to the Watson artificial intelligence that is now being put to work to deliver the US Open to the world this year. With no live crowds there are a lot more reasons than usual to ensure the event is not just broadcast, but that it offers new kinds of interactivity for fans to feel as close to the action as possible while we can't be there together.

We look at the new interactive options being explored, how Watson AI is helping to refine the crowd atmosphere to keep the hype high without making it feel too weird to do that, and also how some work from past years helped to prepare for this year without realising that would be an added benefit.

We kick off when I ask Dane to start with the basics - when we see the IBM logo on sporting events, how much deeper does it go than just being some kind of brand partnership?

Dane Gambrill: Yeah, it’s a question we get asked all the time when our logo pops up on the broadcast. But IBM's been very involved in sports for decades in fact. Really, an the early part of our involvement was helping sports embrace the internet, really digitize and improve the global reach of their events over time.

It's something you will see with other organisations – “OK, an event is on, what technology can we showcase” - but for things like the US Open, it's a 29-year history of partnership, it's always been about customer-led innovation that has been enabled by technology.

With the US Open as a partner, as a customer, it’s really about; what are the big things that they need to achieve for the tournament year on year? What are the problems that they need to overcome? With that lens of innovation, we look to see what it is we can do with them, for them.

Seamus Byrne: How much data actually needs to fly around an event like this? Ultimately, it's tennis. What we're watching is an incredibly analog traditional experience, but I guess it's partly the way we watch it. I'm sure there's also– when we think about it - I guess everything from scorekeeping, to the communication behind the scenes between their teams. What are all the different, or some of the key aspects of that tech behind the scenes that IBM helps to run?

Gambrill: Yes, I think your first point around data. Data comes in many forms, in many different volumes. With a tournament such as tennis, there's the video that is broadcast, that is converted into live and on-demand digital video experiences, but I guess when it comes to the US Open, around 2011 we really started to re-imagine, "Well, with all this data that we have, what if we were able to apply data and AI technologies over the top of that to give new insights, not just to the players, and the talent, and the commentators, but actually put those insights in the hands of sports fans, in the hands of tennis fans?"

Some of the technologies that were built on over the years, things such as ‘Key to the Match’, which looking at what is the three KPIs or key performance indicators that player A needs to execute on in order to be successful with that particular match? It's building on data and now with artificial intelligence, it brings all sorts of new possibilities. For this year's tournament, we're very, very excited that a lot of natural language understanding technology is now embedded in fan experiences.

Byrne: Yes, cool. It's funny. While you were explaining that, it started to hit me that, again, when we're watching a broadcast that now, there's so many visualizations that they'll have for things like where the shots are landing on the court. There's all these monitoring systems taking place. It means there's a lot more going on than just who scored the last point, but it's every other aspect of exactly how they're placing their shots, and the success rate of different placement of shots. All that kind of stuff is able to be tracked now! You just can't compare old-school tennis players to the modern players on that statistical level, because there's so many new ways that they've come up with to track exactly how different people perform.

Gambrill: Yes and, hopefully these new technologies and the digitization of data and visuals actually improve the game itself by way of calls being made and those sorts of things. I guess with the USTA and many of the tennis tournaments that IBM's involved with over the years it's really about also, how can we put this data, these visualizations in the hands of customers, so that customers, being fans, being sponsors, can actually choose their own adventure of how they want to consume the tennis event.

It's been really interesting, some of the customer research we've done over the years around what are the demographics of a sports fan? There are four pillars to it. There are those that are only in it for the social engagement, so their community, and being able to interact with other fans is important to them. There's those that are happy for that linear experience, that experience you get from a broadcast point of view, but want it on-demand, want it to be able to be consumed on any device.

I guess it's the next two tiers that are very interesting to us, and where a lot of innovation has really come, it is the fans that want that commentary-based insight, they want to be an expert themselves. They want to immerse themselves in the data, not just before a match takes place, but as a match is progressing. Then we're seeing this new demographic emerging of those that actually want to interact, they want to participate as the game is happening. That's creating all sorts of new experiences with fans, as we've been doing with the US Open this year.

Byrne: Yes, cool. Let's talk a bit more about that, because I guess, we all know that broadcast is where most people will see an event. Particularly this year in that feeling that, I guess, Arthur Ashe Stadium won't be full in the same way that it has been in the past, so it really puts more of that emphasis on how do we keep people excited and engaged beyond what they might be experiencing if they're lucky enough to have tickets to the event.

Gambrill: Yes, absolutely. June 17 was a very big day for the USTA and IBM and a lot of other partners in the ecosystem. It was the day in which it was decided that, for the first time, the US Open would be spectator-less. How do you now pivot? How do you now re-imagine and recreate a tournament that does not have fans in it? A lot of the thinking behind what it is we would do in the 75 days or the 12-weeks leading up to right now is how can we still give fans that experience where they can interact with one another on their love and passion for tennis? How is it that we can still give them real deep insights, but taking that to new levels and how is it that we can improve the engagement of the sport as they're consuming, I guess the broadcasts, and the on-demand, and the video highlights?

One of the first technologies we created for this year was Open Questions, with IBM Watson Discovery. That is all about really, at the heart of it, facilitating debate amongst tennis fans. What we've done is used a lot of the Watson technology to go and explore unstructured data. Of blogs of tennis websites, news outlets, even the US Open's own digital archives to formulate a pros and cons argument around key questions such as, is Roger Federer the best ever men's tennis player? These are the conversations that fans would actually naturally have as they participate in the game.

It's not just pre-canned, with these news outlets and archives coming up with a pros and cons analysis, we actually give fans the opportunity to, not just vote yes or no, but give context as to why they're voting yes or no. Out of that, together with all the articles that we've analysed, provide a structured debate, provide a pros and cons on whether Roger is or isn't the best men's tennis player. That's a really exciting technology that's just been put in the hands of fans in the last 48 hours since the tournament's been on then. I don't know, Seamus, have you had an opportunity to have a go of that yet? Are you a tennis fan?

Byrne: I am one of those fly-by-night tennis fans, that turns up for the finals, and I totally get into it for the finals, but then I'm like, yes. I'll just monitor things as it progresses. I haven't had a play with that yet, but what is it trying to pull out of those conversations? Is it picking up on keywords that people are writing in? Because I love that idea that it's not just the classic, "Well, here are some bars sliding backwards and forwards on the screen that are just percentages," but actually giving people more of a chance to express something about that idea.

Gambrill: Yes, absolutely. It's all about these open questions, how do they give fans a way to engage remotely in this iconic event? This event's the biggest I think that has happened since March and the world had changed, but really, some of the key technologies behind it is a lot of the Watson natural language processing technology. There's technology from IBM research, so IBM showcased Project Debater in the latter part of last year, and really being able to analyse millions and millions of documents and news feeds, and other sports sources for insight, to then come up with this hypothesis and these pros and cons.

It's not necessarily about a right or wrong, but just facilitating debate. We, as humans, love debate. At the dinner table, during the broadcast, or even at the actual stadium itself. It's very good technology but early days and sees how it evolves.

Byrne: Look I see that amongst some of the other ideas I'm seeing is a match insights type system from Watson Discovery. What kinds of ideas might be behind that match insights system?

Gambrill: It's again, using some of the technology that we just spoke about. I think technology is great when it can be repurposed, and repeated, is when you really get scale. Match insights with Watson Discovery, again is really about empowering fans to become experts about players, about the tournament, about the matchups and coming, ahead of each match in fact.

Again, it uses natural language processing technology to search and understands millions of articles, blogs, even what a thought leader is saying leading into a match, to then gather the most relevant information. Then out of those statistics, actually creating a narrative out of it as well, so it's relatable data that we can consume to get fact-based insights on players ahead of match play.

Byrne: Yes, nice. Look, there's probably an interesting nerdy bit to touch on here which is that whole idea of dealing with unstructured data, because I hear it come up more and more in a lot of IT discussions in general. I guess you're trying to just get an AI system like Watson to be able to just go, "All right, there's a big pile of data floating around out there. How do we find something useful that's lurking inside that?" Is this one of those key areas that it feels like that's the strength of AI in the modern environment, or there are other parts of dealing with unstructured data that apply in different ways? What are your thoughts around that to the big picture on unstructured data?

Gambrill: Yes, 80% of the world's data is unstructured, and it's data that, typically, isn't in a database. A lot of it is videos, it's images, it's the way in which we're naturally talking right now. It's how do you– and the whole purpose behind Watson and IBM AI is about augmentation, so actually helping clients make better decisions, helping clients get faster access to insights, more accurate access to insights throughout the organisation.

Certainly, unstructured data is in the way we've modeled it here for tennis, it's all around, making sure you're pointing it to the right data sources. It's all about making sure you have the right architecture to support that. There's a saying that, "There's no AI without IA.", i.e, there's no artificial intelligence without a correct information architecture. That's all about sources, it's all about the process, the workflows. Certainly, natural language is one of the greatest pulls of the market when it comes to Watson at the moment.

Another pretty simple application of it is the questions that you can ask on the US Open website right now in a virtual agent that has been created. This morning I wasn't watching the broadcast, but I was able to type a question in my language to this virtual agent asking, "When is Alex D playing?" Now I said D, because I didn't quite remember how to spell de Minaur, but it was to me, quite amazing that the virtual agent trained by Watson and humans was able to, "Yes, this is Alex de Minaur, and he's playing at this time, at this place."

There's so many applications from a Q&A and virtual agent perspective as well.

Byrne: Actually, that's a really great point because again, I know the number of times where particularly, for sporting events, I wanti to just dig one piece of information out of the schedule, and needing to scroll around a big scheduling board or something when I'm not quite sure what I'm looking for. Being able to just type in something that is actually a really handy way to find out just that one thing really quickly. That's really great.

Gambrill: Absolutely. It comes back to the point, you need an information architecture with the right data sources integrated in, so that you can give fans for simple questions like that, schedules or scores, being able to give that answer. The expectation we would typically have if we were talking to someone, whether it be a call centre agent, we expect– as fans, we are, I would say, the hardest customers to deal with. We're fanatical. We want information in the moment. It's all about that real-time engagement. It's been really great to help power those experiences for the US Open

Byrne: Fans, deeply parochial. They will let you know if they could not find the thing.

Gambrill: [laughs] Yes, that's right.

Byrne: Look, I think that there's a third part of this new experience list I'm looking at that I'm interested in. AI Sounds, is that helping out with some of the whole, how do we keep the atmosphere feeling right? Is that the idea there?

Gambrill: Yes, it is. Again, it builds on some innovation that we put in place for the US Open last year. Also has been applied to Wimbledon, which was around AI highlights. Being able to enable the digital and editorial teams of the USTA to actually compile highlight clips that would drive the most engagement. Last year's highlight clipping solution and technology, we listen to the noise of the crowd to determine the excitement score of that particular clip or that particular play.

It was through doing acoustic analytics, another area of our technology, that we actually had a digital impression of the noise of the crowd and were able to understand how that noise is used in the context of a certain player or a certain shot. It's that sort of concept that has led to the AI Sounds this year. We've got the highlights, we've got the noise crowds, how is it that we can apply, sort of a stadium sound, even though there's no fans in the stadium, so it really draws upon that technology.

Again, I think, what does this mean to your organizations, what does this mean to business? I think of that kind of infrastructure-intense organisations that have a lot of manufacturing and a lot of equipment. There's a lot of IoT use cases where we apply that similar technology of analytics and drawing out acoustics. What does a normal sound actually sound like for this machine? What does it sound like for this equipment? Are the decibels higher, are they 5% higher? Well, if it's 5% higher against the normal, that can be a problem, for example. A lot of applications around acoustics. This was something that is now in the hands of the digital team of the US Open and the broadcast partners to use as they wish.

Byrne: That's really cool. It's funny, isn't it? It's like the return of the foley artists from the old days, or the laugh track back in the old days of sitcoms. Now, with sport, there's such a subtlety there where– I know we all felt a but jarred by it when it first started coming in the football and things here in Australia. Then you realised it would seem weird if there was nothing there. It does feel like everyone's kept tweaking that sense of what's a little bit to help with that feeling of excitement without over-egging it so that people don't notice that it feels wrong because nobody is there. It's a fascinating thing.

Again, I really quite like that idea that you always learn from last year's highlight system. You would never think, "What if in future, we now need to actually use what we learned because of the old audio," to now, "How do we replace the audio, now that it's missing."

Gambrill: Yes, absolutely. You raise a key point around learning. We're still learning in this space, the art and science of it all. I've consumed sporting experiences where perhaps it's gone a little over-the-top. We did some research for the US Open, the USTA leading up to what are we doing in a 12-week window. It's got to be guided by customers, and 48% actually said that they wanted a more interactive digital experience. I think something like AI from a sound and acoustic point of view, we're certainly measuring, is it going to be effective? Is this part of what customers want, but it's an ongoing pursuit of learning. Yeah, it will be really interesting to see how it plays out.

Byrne: Look are there any other– because I do love random stat dumps. Are there any other cool numbers floating around in terms of how much data has to be managed over the course of an event like this, or any other clever numbers that you have floating around? Because I do, I'm a sucker for numbers?

Gambrill: Now, look. There'll be petabytes of video generated. Obviously, every match, and the higher definitions, and the higher volumes that come through from a data and analytics standpoint. I believe we're now sifting through 40 million different data points to determine the keys to the match as part of the IBM SlamTracker capability for the US Open.

When it comes to, I think, one of the first enablers, we spoke about open questions, being able to facilitate debate. The first part of that process is actually scanning the unstructured data of 14 million articles and blogs. We're not just talking about 100, 200 blogs, it’s 14 million. I think that's a pretty fair enough sample size to be able to generate good pros and cons today. I'm sure more data will come over time as to fans weighing into those sort of questions and those debate processes. It's really all a lot of data, I guess I would conclude.

Byrne: Yes. Look, clearly, IBM doesn't just do the tennis, can you flag some of the other major events that I guess you apply this other tech to around typical year or other big things out on the horizon that some of this goes into? Because then I want to finish by talking about the awesome humans who hide behind the technology out there, but what are some of those other big things that all of this work is applied to?

Gambrill: The great thing about sports, I guess, and I often look at it as the birthplace of innovation, because of that real-time, in the moment consumption model, because the customers are typically some of the most demanding customers of any demographic. What we do notice is that with every tournament, whether it's the tennis or not, there's always a building process. We're always looking to stand on the shoulders of the tournament that happened previously, it's a really healthy co-opetition.

The US Masters, the golf, is another fantastic event that we continue to partner in, highlight clipping was certainly something that we do for them. Not just taking it in the noise of the crowd, but also the body language of the golf player to again, derive an excitement score. Really, a lot of the other innovation has been around really giving clients the ability to choose their own adventure as they consume these digital properties, whether it be web, mobile, or any other form of digital application.

Certainly, my involvement, I started with IBM in the Sydney 2000 Olympic Games as an IT and data quality control analysts, and to see it evolve since even 2000. The experiences we get as fans now has been quite remarkable. I think other applications I've seen with data and AI technologies has been injury prevention and injury analytics to help sporting teams really understand how likely their talent or their players to be injured based on 90 to 100 different data feeds that are coming in.

Byrne: Wow, that's valuable.

Gambrill: Yes, and little things like being able to pick up, "Player A why we've noticed that your treading 5% less on your right heel, compared to yesterday." Sometimes, the talent don't even know that something might be up, but that's what data is telling us, so it's about going in exploring that and then conditioning training accordingly. Then potentially even guiding match play sort of duration as well.

So many things we're involved with still in the sports ecosystem, and of course, a lot of what we do for sports, from core infrastructure, from a cloud capability, being able to give clients the ability to reach audiences globally, and use these new advanced technologies, emerging technologies, to enhance fan experiences.

Byrne: That's cool. Look, I have just realised that we might have been very close to working together on the Sydney 2000 games because I was a final year in my digital media course, and the IBM was looking for people at the universities to come and help out, and a bunch of my friends did. I managed to get a job working with NBC in their media team. I remember at the time being like, "Ah." Because I think I'd missed the application window for the IBM gig, so I was like, "Ah, damn," but managed to get another cool, final year uni gig working at the games, anyway, but I do really remember that IBM did a whole bunch of stuff right there in Sydney.

Gambrill: Yes. It's a fantastic time, but the best is yet to come, I hope.

Byrne: All right. It's funny, what we thought was high tech then is very old hat in so many ways.

Gambrill: That's right. I don't think we can underestimate this. There's still a lot that goes on into making sure that the scoreboarding systems work accurately, and how the scores and stats are distributed across the ecosystem from the digital websites, the broadcasters, to affiliates and partners. This is still amazing to me even now, how that happens in real-time, but yes, this last decade, I think there's been a massive step-change in terms of how data, in terms of how AI are really now enriching the fan and the player experience for everyone's benefit.

Byrne: I guess, yes. I wanted to finish up with a bit of a shout out to the engineers who sit behind the scenes and all this stuff, because as much as I guess we give so much credit to the AI systems themselves, I know there's probably dozens or maybe even hundreds of engineers lurking behind the scenes, making sure everything works properly. I always love that feeling that they know they've done their job well if nobody notices that we're ever there.

Gambrill: Look, it's heroic efforts. In terms of what I've noticed, with the USTA and the IBM partnership, and the ecosystem surrounding, that they face challenges like every organization is facing right now. Everyone's had to learn how to pivot to do their job working from home, to be remote. When it comes to the production, the digital teams powering the US Open, even the statisticians. The statisticians doing their job from their home offices in places like Jacksonville right now, today.

It is incredible to see the massive shift and pivot that they can, not just pull this off, but really step changing enriching the fan experience, but being able to do that from home. I think the other heroes in this whole equation, the USTA and IBM teams that took us through a 12-week process, pivoting from a spectator-full to a spectator-less environment, and all in 12 weeks. Using design thinking and using Garage Methods to reimagine what it is we could do virtually for the tournament that's on right now. Big shout out to all the engineers for sure, for making it happen.

Byrne: Yes, that's amazing because I know– when there were a lot of the questions around which different kinds of sports would be more capable of pushing through than others. On one level, you think, well, tennis is a one-on-one or a two-on-two type of sport. Again, there's so many of those layers of the people around the edges of the court. Even just I guess ball boys and girls, and men and women. I haven't watched it yet, I'm not sure if there's reduced numbers of those people, even on the court just to try to manage this stuff better, but there's a lot more than just the players.

Gambrill: Yes, definitely only a few in the stadium. I was just watching Alex de Minaur's match just before this interview, and he won his first set, but you can definitely hear one or two hand claps, that's about it, and certainly less within the stadium. It's just been amazing to see. It's truly the biggest sporting events since March that has decided to go ahead and you just see how they've been able to do that with remote production, using cloud technologies to enable them to continue to do what it is they're doing, it's been remarkable.

I would maybe finish, I remember a story in March I had one of my IBM colleagues down from the US. He had just finished working with Fox Sports on the 2019 FIFA world cup. It was quite amazing that the whole remit of that project was to enable remote post-production. The cost of sending a whole thing to Russia, I think the operational issues in doing that, so how is it that they could use commodity internet to move 2,700 live feeds between Moscow and their post-production facilities in Los-Angeles using commodity internet networks?

It was just fantastic just last week to hear and understand. Then the award had been won, Technical Emmy award had been won for that project which was really an engineering effort to make it all possible.

Byrne: Yes, little did they know how important that technology would be right around the corner.

Gambrill: Yes, and when my colleague was here in March, terms such as lockdown weren't really in our vocabulary then, but we were starting as we were active in the market that week starting to realize, well, remote production isn't just a cost thing anymore, what if companies were forced to do it? Not just sporting events, but post-production for movie creation and all those sorts of things. It was just amazing to us that within a matter of weeks, this all became a reality and it was just tremendous that the technology was there to be able to facilitate that and make that happen.

Byrne: Yes, that's incredible. Thank you so much for your time. Are there any other cool things that people should keep in mind or any last thoughts?

Gambrill: Yes, I would just suggest even if you're not a tennis fan– I always think about all these innovations with sport, what does this mean to me, what does this mean to organisations in Australia? I think just consuming, immersing yourself in the app experience and the digital experience, it might even give you some ideas as to what you can apply to your business in relation to some of the AI technology and the other technologies that we've spoken about. There are four or five Australian players left. Go team Australia, we're all cheering for you remotely and really appreciate your time, Seamus.

Byrne: No, you're very welcome. I think it's a great point about the way you can learn lessons from this stuff. I know that idea of– chatbots is one of those areas that I think there's a lot of bad ones out there, and when you can see a good experience, the fact that it's not actually, it's not only the domain of big companies to actually be able to access some of this tech now. Being able to apply that to even relatively small businesses is actually possible instead of needing to work with just what are now becoming very dated old-school chatbots as well.

Gambrill: Yes. Just last point, I think the benefit is through the cloud, and through microservices and APIs, this technology has never been more accessible. A lot of the technology we spoke about some of it is applications, but a lot of it is APIs and microservices coming together to really solve the customer problem. Engineers, customers can get started right now with this technology and maybe– because you love stats, Seamus, one of the greatest pulls of the market we've seen since March is companies consuming Watson API that relate to chatbots.

In fact, it's gone up 52% since March to be able to power things like developing chatbots and virtual agents. It's really something that is accessible now. I think everyone should think about not just to solve the problems of customer service capacity that might have diminished but well, how can you actually use that to automate customer interactions, to be able to give them answers to questions they have, not just from a service point of view, but also sales and those sort of things. A lot of the Watson APIs are available on the IBM Cloud and they encourage people to go and check them out.

Thanks again to Dane Gambrill, from IBM for that fun and insightful chat, and thanks to IBM for sponsoring this episode of the show.

Right now all sponsorships are going toward an initiative to create opportunities for more Australian writers to gain paid experience working with Byteside to cover tech, games and digital culture.

I'm excited to try to add a rung or two to that ladder of opportunity that has been missing in the journalism industry lately so I am very thankful to the companies who sponsor the Byteside network and help me to bring that effort to life.