How to use embeddings in Stable Diffusion

Hyeat

In the ever-evolving world of AI-driven art generation, Stable Diffusion has emerged as a remarkable tool for crafting visually stunning images. Among its many capabilities, one intriguing feature that stands out is the use of embeddings, often referred to as textual inversion. In this post, we will delve into the fascinating realm of embeddings, exploring what they are, how they work, where to find them, and how to make the most of their creative potential.

What is Embedding?

Embedding, in the context of Stable Diffusion, is the outcome of textual inversion—a technique that allows you to imbue your images with unique styles or objects without altering the underlying model. What sets embedding apart is its ability to breathe new life into your creations using as few as 3-5 sample images.

How Does Textual Inversion Work?

Textual inversion achieves its magic not by introducing new styles or objects, as other fine-tuning methods can do, but by finding a way to describe these concepts within the language model itself, all without modifying the model. This ingenious process involves tokenizing new keywords and converting them into distinctive embedding vectors used by the model for image generation.

New embedding is found for the new token S* through textual inversion.

Examples of Embeddings

Embeddings can be used to introduce fresh objects or styles into your images. Imagine seamlessly injecting a toy cat into a scene or infusing a unique artistic style into your artwork. The possibilities are as limitless as your imagination.

Below is an example of injecting a toy cat. Note that the new concept (toy cat) can be used with other existing concepts (boat, backpack, etc) in the model.

Example of embedding an object.

Embeddings can also be a new style. The example below shows embedding a new style and transferring the style to different context.

Where to Find Embeddings

To embark on your embedding journey, you can explore repositories like the Stable Diffusion Concept Library hosted by Hugging Face or Civtai, which offers a wealth of custom embeddings. These resources open the door to a world of creative possibilities.

Stable Diffusion concepts library.

How to Use Embeddings

The process of using embeddings can vary depending on your preferred interface:

Web Interface

Stable Diffusion Conceptualizer offers a user-friendly way to experiment with embeddings without the need for downloads. By identifying the desired embedding and its triggering token, you can quickly incorporate new styles or objects into your images.

First identify the embedding you want to test in the Concept Library. Let’s say you want to use this Marc Allante style. Next, identify the token needed to trigger this style. You can find it in the file token_identifier.txt, which is <Marc_Allante>.

Putting in the prompt: <Marc_Allante> a dog

Gives you the unique Marc Allante style.

AUTOMATIC1111

For a more hands-on approach, you can download an embedding file from the Concept Library (usually named “learned_embedds.bin”). Rename it to a unique keyword, place it in the appropriate folder, and restart your GUI. By including the filename in your prompt, you can trigger the desired style or object.

Next, rename the file as the keyword you wanted to use this embedding with. It has to be something not exist in the model. marc_allante.bin is a good choice.

Put it in the embeddings folder in the GUI’s working directory:

stable-diffusion-webui/embeddings

Restart the GUI. In startup terminal, you should see a message like:

Loaded a total of 1 textual inversion embeddings.
Embeddings: marc_allante

Use the filename as part of the prompt to

For example, the following prompt would work on AUTOMATIC1111.

(marc_allante:1.2) a dog

We get the image with the expected style.

A Shortcut to Embeddings in AUTOMATIC1111

“Within AUTOMATIC1111, it’s essential to be aware that embeddings may not always function as expected, and even a slight variation can lead to failure. Please remember that v1 embeddings cannot be used interchangeably with v2 embeddings, and vice versa, due to their compatibility with different language models.

To ensure you’re utilizing embeddings effectively, AUTOMATIC1111 has a clever trick up its sleeve. Look for the button situated between the trash and copy buttons, which resembles a miniature iPod (apologies if this reference is before your time).

Click this button, and you’ll gain access to a comprehensive list of available embeddings, conveniently located under the ‘Textual Inversion’ tab. Selecting any of these embeddings will automatically insert them into your prompt. This feature is especially valuable as it streamlines the process, eliminating the need to manually input the embedding magic word and ensuring a smoother user experience.”

Note on using embeddings in AUTOMATIC1111

If you pay attention to the prompt, you would notice I have increased the strength of the triggering keyword marc_allante. I found that it is necessary to adjust the keyword strength. This may have something to do with the way AUTOMATIC1111 loads the embedding.

You may have to play with the keyword strength to get the effect you want. Below is an example of varying the strength while keeping the seed and everything else the same.

Adjust keyword strength to get the effect you want.

Exploring Some Fascinating Embeddings

To get your creative juices flowing, here are a couple of embeddings to try out:

wlop_style

Unlock the unique artistic style of the renowned artist wlop with this embedding, best used with SirVeggie’s wlop-any custom model.

Kuvshinov:

Dive into the captivating world of Russian illustrator Kuvshinov, perfectly complemented by Stable Diffusion v1.4.

Difference between embedding, dreambooth and hypernetwork

There are three popular methods to fine-tune Stable Diffusion models: textual inversion (embedding), dreambooth and hypernetwork.

  • Embedding:
    • Purpose: Embedding allows the introduction of a new concept without modifying the entire model.
    • Storage: The embedding vectors are stored in files with extensions like .bin or .pt.
    • File Size: These files are very compact, usually occupying less than 100 kilobytes.
  • Dreambooth:
    • Purpose: Dreambooth involves fine-tuning the entire model to inject a new concept.
    • Storage: The resulting model is typically saved with the same file extension as other Stable Diffusion models, which is often .ckpt.
    • File Size: The file size for a dreambooth-fine-tuned model is relatively large, typically ranging from 2 to 4 gigabytes (GB).
  • Hypernetwork:
    • Purpose: Hypernetwork is an additional network component attached to the denoising UNet of a Stable Diffusion model. It is used for fine-tuning the model without fundamentally altering it.
    • Storage: The hypernetwork model is typically saved in files.
    • File Size: These files are of moderate size, typically around 100 megabytes (MB).

Pros and Cons of Using Embeddings

Embeddings come with their own set of advantages and challenges. On the positive side, their small file size allows you to store multiple embeddings easily and use them together. However, determining which model to use and adjusting keyword strengths can sometimes pose challenges.

In conclusion, embeddings are a captivating tool in the Stable Diffusion arsenal, enabling artists and creators to infuse their artwork with new styles and objects effortlessly. While they may present some nuances to master, the creative possibilities they unlock make the journey well worth it. So, dive into the world of embeddings, experiment, and let your imagination run wild in the realm of AI-driven artistry.

Leave a Comment