Stroke-based Rendering

Repository for the survey on stroke-based painterly rendering algorithms: Stroke-based Rendering: From Heuristics to Deep Learning

Note: Not all GitHub links are the official implementations!

Greedy heuristics

Edge alignment: Align stroke direction with edges and contours of image.

• Low-level: e.g. edges from Sobel filter or Canny edge detection

• High-level: e.g. semantic / salient edges and contours of objects

Region approximation: Replace image segments with strokes.

• Image moments: Soft / fuzzy image segmentation algorithms

• imgage segmentation "Normal" image segmentation algorithms (hard edges)

Name	Algorithm	Notes / Description
Paint by numbers: Abstract image representations	Low-level edge alignment	The first painterly rendering paper!
Processing images and video for an impressionist effect	Low-level edge alignment
Painterly rendering with curved brush strokes of multiple sizes	Low-level edge alignment	First algorithm with curved strokes. [Java] [Python (Unofficial)]
Image and video based painterly animation	Low-level edge alignment
Painterly rendering controlled by multiscale image features	Low-level edge alignment
Interactive painterly stylization of images, videos and 3D animations	Low-level edge alignment
Contour-driven Sumi-e rendering of real photos	Low-level edge alignment	Connected to Artist Agent
Painterly rendering with content-dependent natural paint strokes	Low-level edge alignment
Painterly rendering using image salience	High-level edge alignment
Abstracted painterly renderings using eye-tracking data	High-level edge alignment
From image parsing to painterly rendering	High-level edge alignment
Video painting with space-time-varying style parameters	High-level edge alignment
Artistic composition for painterly rendering	High-level edge alignment
An algorithm for automatic painterly rendering based on local source image approximation	Image moments
Multiscale moment-based painterly rendering	Image moments
Artistic vision: painterly rendering using computer vision techniques	Image segmentation
Empathic painting: interactive stylization through observed emotional state	Image segmentation
Emotionally aware automated portrait painting	Image segmentation
Abstract art by shape classification	Image segmentation
A generic framework for the structured abstraction of images	Image segmentation	[Code]

Iterative error minimization

Brute force: Systematically try out many different paramters until a good result is achieved

• Stroke database: Choose best stroke from database

Randomized search: Randomly perturb stroke parameters to find good painting. Different stochastic optimization algorithms are possible.

• Hill climbing
• Simulated annealing
• Genetic / evolutionary optimization

Gradient descent: Change parameters in negative gradient direction. Good and fast results, but usually requires differentiable rendering.

Name	Algorithm	Notes / Description
Paint by Numbers	Hill climbing	The first painterly rendering paper!
Paint by relaxation	Brute force
Anipaint: Interactive painterly animation from video	Brute force
Interactive painterly rendering with artistic error correction	Stroke database
A painterly rendering based on stroke profile and database	Stroke database
Stroke based painterly rendering with mass data through auto warping generation	Stroke database
Portrait painting using active templates	Stroke database
Random paintbrush transformation	Hill climbing
Optimization of paintbrush rendering of images by dynamic mcmc methods	Hill climbing
Genetic paint: A search for salient paintings	Genetic / evolutionary optimization
A unified scheme for adaptive stroke-based rendering	Simulated annealing
Image-based painterly rendering by evolutionary algorithm	Genetic / evolutionary optimization
Evolved art with transparent, overlapping, and geometric shapes	Genetic / evolutionary optimization	[Code]
Paintings, polygons and plant propagation	Genetic / evolutionary optimization, Simulated annealing, Hill climbing	[Code]
Automatic and interactive evolution of vector graphics images with genetic algorithms	Genetic / evolutionary optimization
Incremental Evolution of Stylized Images	Genetic / evolutionary optimization	[Code]
Generative art using neural visual grammars and dual encoders	Genetic / evolutionary optimization	[Code]
Modern evolution strategies for creativity: Fitting concrete images and abstract concepts	Genetic / evolutionary optimization	[Code]
Customizing painterly rendering styles using stroke processes	Reaction-diffusion process	[Code]
Neural painters: A learned differentiable constraint for generating brushstroke paintings	Gradient descent	[Code]
Differentiable vector graphics rasterization for editing and learning	Gradient descent	[Code]
Stylized neural painting	Gradient descent	[Code]
From Objects to a Whole Painting	Gradient descent
Rethinking style transfer: From pixels to parameterized brushstrokes	Gradient descent	[Code]
Differentiable drawing and sketching	Gradient descent	[Code]
CLIPDraw: Exploring text-to-drawing synthesis through language-image encoders	Gradient descent	[Code]
StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation	Gradient descent	[Code]
CLIP-CLOP: CLIP-Guided Collage and Photomontage	Gradient descent	[Code]

Deep learning

Supervised Learning: Predicts sequence of stroke parameters from pixel canvas by using differentiable rendering

Deep Reinforcement Learning (DRL): Agent predicts next stroke to paint, given canvas and target.

• Model-free DRL: Does not use differentiable rendering
• Model-based DRL: Uses differentiable rendering

Name	Architecture	Notes / Description
Neural painters: A learned differentiable constraint for generating brushstroke paintings	Supervised Learning	[Code 1] [Code 2]
Unsupervised image to sequence translation with canvas-drawer networks	Supervised learning	[Unofficial code]
Strokenet: A neural painting environment	Supervised learning	[Code]
Paint transformer: Feed forward neural painting with stroke prediction	Supervised learning	[Code]
Synthesizing programs for images using reinforced adversarial learning ("SPIRAL")	Model-free DRL	[Code]
Unsupervised doodling and painting with improved spiral ("SPIRAL++")	Model-free DRL	[Unofficial code]
Paintbot: A reinforcement learning approach for natural media painting	Model-free DRL
LpaintB: Learning to paint from self-supervision	Model-free DRL
Learning to paint with model-based deep reinforcement learning	Model-based DRL	[Code]
Demystify Painting with RL	Model-based DRL
Content masked loss: Human-like brush stroke planning in a reinforcement learning painting agent	Model-based DRL	[Code]
Combining semantic guidance and deep reinforcement learning for generating human level paintings	Model-based DRL	[Code]
Intelli-Paint: Towards Developing Human-like Painting Agents	Model-based DRL

Video painting

Different techniques for extending painterly rendering algorithms to handle video.

Name	GitHub	Notes / Description
Primitive Pictures	[Code]	Ready to use application / GitHub for video painting
AniPainter	[Code]	Digitally painted and robot painted videos
Processing images and video for an impressionist effect
Painterly rendering for video and interaction
Image and video based painterly animation
Video painting with space-time-varying style parameters
Motion map generation for maintaining the temporalcoherence of brush strokes
Interactive painterly stylization of images, videos and 3D animations
Painterly animation using video semantics and feature correspondence
Anipaint: Interactive painterly animation from video