I played around with RunwayML and a myriad of other programs utilizing machine learning algorithms to create two different projects. I couldn't pick just one project, mostly because they were vastly different from each other from an output perspective and I wanted to explore more than one avenue and model. I was able to process the "Oil Birds of California" video while I was training the models for the North American Cryptid Generator and ended up with two different results.
Oil Birds of California
For this project I used a neural style transfer technique with the aid of EbSynth to create a video collage of birds located in the Bolsa Chica Ecological Reserve. Just North of the reserve is Huntington Beach - the site off a recent oil spill caused by a leak in an offshore pipeline, which is maintained by a local energy company. The leak created a toxic, 13-square-mile oil slick, which has since spread into coastal wetlands home to several endangered avian species. Not only do oil slicks damage avian life, it also kill marine algae called phytoplankton, for example, which can disrupt entire food chains for all coastal animals.
I wanted to coat the entire environment of this video in oil, so I used the neural style transfer method to transfer an image of the oil spill near Huntington Beach onto videos of these local birds. I wanted to add some audio as well, so I used an app called Melobytes to generate an atmospheric audio track based on the image of the oil spill that I used to generate the video. The results were varied and I ended up generating several sounds which I combined/looped with the video.
After some editing in Adobe Premiere, here is the end result:
North American Cryptid Generator
For my second experiment I wanted to create a creature generator. For this I wanted to train a model using StyleGAN2 that will generate images of creatures using a dataset of images of popular cryptids in America.
A cryptid is an animal-like creature that has been claimed to exist, but whose existence has never been proven. They are closely tied to the geological regions in which they are seen or believed to exist. The legends that are perpetuated about these creatures tend to precede the physical evidence behind their existence and have become an endearing part of our cultural zeitgeist. Cryptozoology is its own field of study born out of the desire to understand the myths and legends behind these creatures.
A somewhat blurry visual map of some popular cryptids in the United States
In order to get this data, I created a list of 32 of the most popular cryptids in the United States taken from several maps and databases found on various websites, since I was interested in tracking visual similarities between the creatures that exist specifically in my part of the world.
Some of the cryptids I included were the Mothman, Bigfoot, El Chupacabra, Ozark Howler, Wampus Cat, Goat Man, Frog Man, Jersey Devil, Champ, etc.
Once I created my list, I batch downloaded the top 200 results from Google images into a folder. I then manually vetted the images, removing images that were obviously not supposed to be in the dataset. I ended up with around 1700 images for training my ml model on RunwayML.
I trained two models using the StyleGAN2 architecture, with two different pre-trained models to produce varying results. Many of the images I downloaded were illustrations or black and white photos of the cryptids, so I thought using a model trained on botanical illustrations would be a good start - with the idea that it would generate illustrations of cryptids.
I was met with mixed results after training the model with 9000 training steps to generate illustrations based on my cryptid images, so I decided to train another model pre-trained on images of cats, with the same number of training steps. This model would try to produce photorealistic images based on the cryptid dataset. I found that this model mode varied results that better resembled what you might expect an image of a cryptid to look like (both illustrative and photographic).
I found images I really liked from each model, you can see samples of the model that was pre-trained on images of cats below
I ran some images through the "Image Super Resolution" model on RunwayML to up-res them for posting on this blog. It did a pretty good job!
You can see samples of the model that was pre-trained on images of botanical illustrations below
With these renders, I decided on one particular image to use to create some sort of cryptid as a 3D model that I could then rig and animate, or 3D print it and paint it. Below is the image that I used, with the renders of the 3D model to follow. I unfortunately didn't have a ton of time to design it, but would definitely like to return to it in the future to continue working on it.
All good cryptids have a name and a story, but I didn't want to be the one to give it a name and a backstory. I decided to generate one using a film script editor in RunwayML based on GPT-2 architecture. I only gave the first two sentences as a prompt and then kept running the script editor with the last few sentences that it generated until I had a semi-comprehensive story.
As for the name, I decided for the first-half to be named after the mountain range where I grew up in Utah. As for the name of the creature, I decided to use a legendary creature name generator in order to come up with the second part. I clicked on it a few times when "Oumen" popped up. It's kind of a silly name, since it's very close to "omen" but in in a weird way it suits the computer-generated story about this creature - so for now it stays. Anyway below is the full generated story about the Wasatch Oumen.
The Wasatch Oumen
One time I was walking my dog through the woods, on a day much like any other day. As I was walking I noticed strange tracks in the mud to the left of the main trail. They were huge and unrecognizable... too large to be a bear. I stopped and stared. There were countless of them running along the base of the trees and into the underbrush. I stood there, the smell of earth filling my nose. I decided to follow the tracks, feeling the desire for adventure. After a long while, I reached a clearing and in the corner of my eye I saw the strange beast that had made these tracks. It looked like a huge golden bear, standing there watching me. I could see the intense pain in his eyes, as though he were being punished for his sins, just as the bear in the old man’s story had been. But upon further inspection, it didn't actually have any fur. It appeared to have no face, just skin. It was gray in color, and it had nothing of life in it, apart from its perfect teeth.
As I approached him, the grey face and black eyes almost filled the frame. But that didn't stop me from walking towards the giant beast. However as i came closer it suddenly split into two, and it turned into a wolf. It sniffed my shoe, and then got up, appearing to be angry. It then howled as though trying to shake me off. But I struck it with a blow to the head and it ran back into the woods. And as the seconds passed, I could hear it howling in pain and anger. I continued on, now much more certain of what I had seen.
As I came to a small creek I could hear a faint hiss growing slowly louder, heading towards my direction. I sprinted out of the path, back towards the main road when I saw a pair of large golden eyes high up in the tree. The main bush was breathing. I heard the sound for the first time as I was falling. The eye was the last thing I saw before I hit the ground. The sound echoed through the woods as I lay there helpless. Then silence. Then the beginning of a song. It's a tune we know as happy. a father and his two sons. The song continues to play, but I can no longer remember the words. My eyes open. Then fade up from the bottom, the sound of engine roar surrounding me. Soft, and distant like a dream. But now I can hear two voices. They call out my name. When I came to, I heard muffled voices coming from my jacket pocket. I must have accidentally called my mother. My mother and sister were yelling at me through the phone. As I got up to recall the beast I had seen transform in front of me, I began to scream in terror and pain. As soon as I pulled back the pocket the screams stopped. Fading in and out, hard to follow the direction of the voices. My mother hung up.
That's all I remember of that day. I managed to get back to the car and found that my dog was happily awaiting my return. As I started the engine, a glimmer of light reflected on two golden eyes in the treetops above. With a shiver I drove away. And as I did so many times before. I hoped the same would happen now. But as I did so many times before, something was missing.
For good measure, I also ran the text through a free TTS app. Here's the computer generated text above read by a computer.
Comments