When I first heard that text-to-video AI tools were being released, I immediately thought about the potential to create a visual version of an audiobook, or put more simply, a movie for any book I wanted. A considerable amount of important knowledge is inaccessible because of the format in which it is presented. I wanted to see if I could expand the ways in which knowledge from books is acquired.
To pilot this idea, I decided to use the text-to-video software, Runway, and the first paragraph of a story created by ChatGPT:
“In the heart of the dense Amazon rainforest, where the canopy casts dappled shadows on the forest floor below, a small team of intrepid explorers embarked on a journey unlike any other. Led by the seasoned adventurer, Dr. Amelia Rivers, the group ventured deep into uncharted territory, their senses heightened by the cacophony of wildlife echoing through the emerald expanse. Armed with determination and curiosity as their compass, they pressed forward, unaware of the ancient secrets and thrilling encounters awaiting them amidst the lush greenery of the wild jungle.”
My goal was to try to get a video without having to write a prompt myself. I wanted to be able to take the direct text from the book and create the video. On my first attempt, I tried to use the entire first paragraph and I quickly realized I was going to have a few problems:
When multiple sentences were provided, the software struggled to determine the main subject. In the opening paragraph there were multiple options: the rainforest, a team of intrepid explorers, and Dr. Amelia Rivers. Trying to include all three in one video clip would be difficult.
The second problem I noticed was that individual sentences didn’t give any indication of the desired style for the video. Should the setting be dark and gloomy, light and airy, or even animated?! There were various creative decisions that stand-alone sentences couldn’t convey.
At this point I decided to enlist ChatGPT to help me get a better prompt without having to make creative decisions myself.
I was impressed with this response, and so I decided to go further and see if the software would be able to help me imagine the text how a professional cinematographer might:
By using the first shot directly in Runway as a prompt, I got the video above. This video turned out well, but I wanted to see if ChatGPT could help me create a more detailed guide for the software, so I posed an intriguing question: could it help me write a prompt tailored specifically for Runway?
The prompt format worked smoothly, so I proceeded to create a prompt for each shot ChatGPT provided. Ideally, the video book would be accompanied by captions and a voice over, but below are the five shots edited together sequentially.
In the end, the experiment resulted in an incoherent hodgepodge of clips rather than a scene from a film, but I still see potential in the software reaching a point where it can create a cohesive visual interpretation of a book. In the future, I believe the opportunity to operate Runway more as a chat rather than a single video creator would allow me to specify basic information like characters, setting, and style of front, leading to more continuity and a better final product. Although I didn’t achieve the desired output, I’m excited to continue using text-to-video platforms as they improve.
Maxwell McIntosh
