How AI Can Help Improve Live Broadcasts

Sports Media has always been a ripe playing field for new technologies, perceived and otherwise. As someone who is constantly pitched new technologies, from the nascent to the preposterous, I was always quick to accuse those who peppered their sales pitches imbuing Artificial Intelligence of exaggeration at best and overt expropriation at worst. Frankly, the fact that AI is mentioned anywhere in a sales pitch could not have been more annoying!

Recently, many technologies I have been a part of have had me walk some of that back and insofar as a utility, it makes an excellent case for AI's contribution to the production world as we continue to learn how to use it. One of the most visible ways that AI can help Sports Media is simply by improving and creating new video frames out of existing images and doing it rapidly.

Part of the issue with venturing onto a project that utilizes AI in the sports media industry is where to start. We need to ascertain that AI is a part of the process— or a time saver— to create a product that already exists as a goal, not simply using AI to do so organically. The tool cannot dictate the solution. Instead, production creatives often find themselves looking for an answer, or a product, versus seeking to ask a question as to how it should come about.

Some brief examples of how Fox Sports is using AI in live broadcasts - and the simple questions we asked to produce them - are instructional in what the technology is presently doing and what it may be capable of in the future.

We Need Clearer Replays

EVS, a Belgian company, and Fox Sports have been working on and improving a technology called Xtramotion. For background, an increasing number of cameras in sports broadcasts are Super Slow Motion (SSMO) cameras - those that shoot over 60 frames per second for smooth, definitive replays without motion blur. For instance, many of the cameras in this year's Daytona 500 were SSMOs. However, many essential angles were not— mainly for size reasons (predominantly, the in-car cameras). To help with this, Xrtramotion uses AI to interpolate existing frames to create new ones between them in producing the Super Motion effect without the need for higher frame rate capture.

The novel thing about sports production technologies is that they must be quick. Previous iterations of this product relied on cloud processes that would churn out the new clips. However, the time it took to upload and download slowed the process significantly, devaluing the outcome. This new process, which utilizes an on-prem machine attached to the server network, allows for a rapid turnaround— a new Super Motion clip is ready as quickly as five seconds for air. The more the algorithm learns what makes a good Super Motion clip, the better and clearer it will become.

AI processes like this will continue to help quickly enhance original capture. Other concepts include automatic background defocusing and predictive tracking to composite graphics within the live video, perhaps without conventional chroma-keying.

We Want to Create a Special Moment in the Field of Dreams

Live processes are only one area where Fox Sports and our colleagues have had success. This year's Field of Dreams event allowed us to take on a project using new Nvidia technology and Silver Spoon Animation's rendering. Probably no place other than the Field of Dreams would have allowed us to bring back an extraordinary rendition of Harry Caray's iconic "Take Me Out to the Ballgame" during the 7th-inning stretch of the game.

Fox selected a picture of Harry to composite over a live stand-in, who did several takes of the song at a filming session at the Field before the event. We then used this element as a cutout which was overlaid on the live video and cut by the director so that the performance could occur live. 

Like Xtramotion, sports timelines dictated an accelerated schedule for compositing the face and delivery of the element. In that same production, we used also used standins to recreate greats like Hank Aaron, Babe Ruth, Sparky Anderson, Bob Gibson, and others for post-produced clips using the same process.

Unlike Xtramotion, this technology is harder to determine how this can proliferate meaningfully, but specific solutions are already being considered. Perhaps using another technology stack, something similar in the NBA is being used to achieve almost the opposite effect, allowing fans to put themselves virtually in place of athletes using their smartphones and scans utilizing the phone's cameras.

We Want to Know What the Drivers are Saying

NASCAR is the only sport where we can hear athletes say during competition. Their radios are open - free and clear for television to use and for fans to hear during the broadcasts. The problem is that the audio quality from in-car radios often makes what they say hard for the TV audience to understand.

What is being worked on by Fox Sports and Vertitone is a method of simultaneous transcription. While speech-to-text is nothing new, being able to do this using degraded audio requires some AI intervention in terms of the transcription engine - explicitly being able to pick up words and phrases which specific drivers or teams may use to make intelligent guesses as to what is said. The goal is to provide lower-third graphics which faithfully transcribe the conversation between spotters, drivers, and crew chiefs. Additionally, this engine can be used to better transcribe these conversations for archival purposes and posted programs.

The trick is speed. These things need to happen rapidly to be useful for production. The longer it takes to turn around, the less valuable it becomes.

“Recently, many technologies I have been a part of have had me walk some of that back and insofar as a utility, it makes an excellent case for AI's contribution to the production world as we continue to learn how to use it”

Putting the End First

Putting the end product at the forefront of the project allows the process to dictate whether AI is required and has been the current key to determining whether the investment, which can be significant, is needed. Many methods are perfectly okay without AI and the costly development and training around it. However, as models are trained on different products and learn the sometimes esoteric nature of sports— after all, a NASCAR race is fundamentally different than a soccer match - the runway for various applications and be shortened and economized.

Our live broadcast industry is the rugged proving ground of technology as it is so unforgiving in terms of quality, speed, and accuracy. We are defining the specific requirements in advance and ensuring that elements like turnaround time, resolution, and framerate are critical to ensuring the correct outcome.

Like all industries faced with the utility of AI assistance, the fact is that there are a great many applications that are coming about that will create significant opportunities and also fundamental questions in our industry. AI purports to cut cameras, select replays, and display graphics. As formulaic as sports broadcasts can be, specific production processes are ripe for automation at the hands of a machine that knows the rules, both of the game and broadcasts in general.

While this is true, it is also true the art of telling sports stories is challenging to automate. Many times, those crucial moments in sports broadcasts are at the "edges" and navigating those edges - those unusual occurrences— is traditionally what AI processes struggle with. For now, what we find fascinating and what we are concentrating on is how the technology can enhance our creativity.