Tech is awesome, but let’s be honest: Sometimes it gets a bit boring. Computers and smartphones change incrementally from year to year, software switches to subscriptions, and innovation, in general, isn’t so common. Every once in a while, though, a new tech development is actually surprising and exciting. That’s how I feel about this new AI, which will draw almost anything you throw at it.
The software is called DALL·E 2, developed by OpenAI, which uses machine learning to generate new images based on a text description provider by the user. You can see examples of how this works on the website: The developers let you mix and match different subjects, actions, and settings to see how DALL·E 2 produces different images from those descriptions.
For example, “an astronaut riding a horse in a photorealistic style” generates images of exactly that, resulting in different “photos” of an astronaut on a horse in what look like MTV promo pics. Swap “photorealistic style” for “pencil drawing,” and suddenly you’re looking at what appear to be multiple sketches of that spaceman on their steed, only nobody actually drew anything — it’s entirely produced by AI.
It goes way beyond these impressive tech demos, though. From tests I’ve seen, the AI can accept the most detailed of requests, and produce images you might to see hanging in a modern art museum. Director Karen X Chang showed this off, first asking DALL·E 2 to draw a girl walking up an infinity staircase, which it did with ease, then asking DALL·E 2 to draw a girl walking up an infinity staircase in the clouds made of cookies, which it also did, with ease.
How DALL·E 2 works
The project really started in January 2021 with DALL·E. The goal was the same as DALL·E 2, to create a program that could generate new images from text. Since then, OpenAI has figured out how to greatly improve the image quality, reintroducing the software as DALL·E 2.
DALL·E 2’s developers train the system using labelled images. DALL·E 2 likely sees thousands of images of cars labelled “car,” houses labelled “house,” coffees labelled “coffee,” etc. It uses that database of knowledge to generate its best guess as to what you’re asking for. If it has enough training to know each piece of your description, you’ll get remarkably accurate images of things you’ve never seen before, such as a “monkey doing its taxes while wearing a funny hat.”
A lot of machine learning programs work this way, but one of the differences with DALL·E 2 is it’s able to “fill in the blanks,” as OpenAI puts it. Essentially, DALL·E 2 looks at your text description and thinks about the context you didn’t provide. If you’re asking for a “dog playing basketball outside,” it’s going to assume you meant a basketball court, which might be found around a park, which then means you should see trees, grass, sky, etc.
That said, DALL·E 2’s team openly admit gaps in the system’s knowledge exist, and can result in some interesting and insightful discoveries. If the system hasn’t been trained on “howler monkeys,” asking it to draw a “howler monkey” could result in images of screaming monkeys, rather than the particular type of monkey you’re looking for.
DALL·E 2’s application goes beyond generating new images, too. You can use the tool as an AI-powered photo editor, removing elements of images you don’t like, or adding new components that were never there in the first place. It can also generate new images based on existing images: If you need to paint the Mona Lisa, but you can’t paint, DALL·E 2 makes it look like you can.
DALL·E 2’s developers are keen to keep DALL·E 2 away from misuse, including generating images based on hate or violence. As such, the team has opened the software up to a limited pool of “trusted testers,” but will eventually invite more people to test DALL·E 2 for themselves.
That said, there’s no clear timeline on that front, nor is there much of an explanation to what makes for a “trusted tester.” If you’re interested, it seems worth it to sign up for the DALL·E 2 waitlist now (I signed up when writing this piece). Provided you meet OpenAI’s criteria, perhaps you’ll be asked to test out DALL·E 2 for yourself.