Automatic Comic Book Generation

Creative AI in 2022

Artificial intelligence is increasingly automating many different tasks in all aspects of our lives. People have long imagined that creativity is uniquely human. Today however, many creative capabilities are also being exhibited in machines. Every year new architectures of generative networks redefine the limits of what is possible. AI has the potential to produce art, music, poetry, plays, and even video games. One largely untapped domain is comic book generation and developing AI tools to increase the efficiency of comic book creation and co-creation, which is what my research focuses on.

Comics generation using AI

Comics challenge current creative AI technologies in several ways. Firstly, they combine sequential images with text in a unique and complex way. The creation process supposes a thorough understanding of script writing, lay-out, rhythm, image composition and dialogue writing, to name just a few of the involved steps. Secondly, many AI models are built for photorealistic purposes (both for image understanding as well as image generation), making it non-trivial how to translate or retrain them for line-art.

While fully automatic comic book generation is the long-term moonshot, we believe there are also plenty of opportunities for creating AI tools that can be used with comic artists in co-creative ways to empower their artistic process.

Up until today we haven't seen any proper autonomously generated comics yet. The attempts done so far needed human assistance. However, recent AI progress made comic book generation within reach. Models like GPT-3 have produced incredible text, often indistinguishable from human-written text, and the mind blowingly artistic images in all kinds of different styles generated by the DALL-E 2 and Stable Diffusion text-to-image generator stun even the best artists. While fully automatic comic book generation is the long-term moonshot, we believe there are also plenty of opportunities for creating AI tools that can be used with comic artists in co-creative ways to empower their artistic process. Some examples of tasks where AI could offer automated support to comic artists are:

  • text-to-image sequence: given a script, make a sequence of panels controlling the narrative time and the telling time.
  • text-to-page lay-out: given a script, create proposals for positioning of the image panels.
  • xtext-to-image composition: given a script, draw scenery, characters and text balloons in positions to inspire the artist.
  • inking assistant: Turning pencil drawings into inked pages maintaining a specific inking style.
  • color assistant: coloring black & white drawings.
Typically the main research goal is tackled in work packages of increasing difficulty, combining more modalities like text, image, characters, rhythm, speech, etc...

Some samples of StyleGAN2-ADA trained on drawings from the Kinky & Cosy animated films.
A so-called style transfer, between a real photo of Nix and a drawing of Cosy.

A PhD research at KU Leuven

Marnix Verduyn a.k.a. NIX graduated from KU Leuven as a Master of Electrical Engineering. After working briefly as an engineer, he quickly shifted to a career in the arts. After twenty years of experience as a comic artist, scriptwriter, animator and creator of interactive exhibitions, he went back to university to deepen his interest in Artificial Intelligence at Erasmus Hogeschool Brussels. The following year at KU Leuven, he started an Advanced Master degree in AI. His master thesis investigates the generation of new drawings with generative adversarial networks after training them on his own comic book artwork, thus finally combining both his passions, comics and mathematics. Marnix is deeply enthusiastic to continue his research in this fascinating and unexplored area in the form of a doctoral study under supervision of Professor Luc De Raedt at the unit DTAI (Declarative Languages and Artificial Intelligence) of the KU Leuven department Computer Science, which is the largest AI lab in Flanders. With that goal in mind, he now works part-time as a scientific assistant and is preparing a dossier to raise the necessary funds, thus focusing on both useful and interesting research, resulting in cutting-edge comic book generation technologies.