Embeddings in Machine Learning (A Beginner's Guide)

When we talk about how machine learning models understand language, one of the biggest questions that comes to mind is how the model can recognize that words like Kubernetes, Pods, and Clusters are related, even though they are completely different words.

The answer lies in a concept called Embeddings. These Embeddings are everywhere in modern deep learning, such as recommendation engines, deep neural networks, and encoder and decoder models.

In this blog, I explained the following.

• What are embeddings in Machine learning, and why are embeddings important?
• How do they understand context and meaning?
• The most common types of embeddings.
• Real-world examples

Let's get a deep dive into it.

What are Embeddings?

Embeddings are a way of converting data like words, images, or even pieces of code into numerical representations called Vectors. After converting, they get stored in a Vector database.

For example,

What is Kubernetes? becomes something like [0.12, -0.45, 0.88, …]

So now you might ask, how can embeddings be created?

It can be done via an embedding model.

For example, AWS Titan Text embedding v2 is an embedding model. Also, Word2Vec, which is a two-layer neural network embedding model developed by Google.

The Neural network converts any input word, sentence, document, or image into a fixed-length array of numbers.

The following image illustrates how embeddings are created. All the data is sent to an embedding model, and it converts the data into vectors.

If you notice the image above, there is something called n-dimensional space.

Let's understand this with an example.

After the embedding model converts your input into a vector like

[0.6, 0.3, 1.5, 0.2, ...]

Each number in the above vector represents one dimension. So if your vector has 1024 numbers, it exists in a 1024-dimensional space.

Think of it this way,

If you want to locate a city, you have 2 dimensions. (Latitude and Longitude)

Now, say you want to locate the word pod in embedding space. You need 1024 numbers, each one capturing a tiny aspect of its meaning.

pod = (0.12, -0.45, 0.88, 0.03, ... 1020 more numbers)
          |       |      |     |
      k8s-related  small  fragile  isolated ...

Here you can just keep in mind,

More dimensions = more detail = more meaning captured.

And here is the part that makes embeddings useful. Each dimension captures a meaning, so inputs that mean similar things end up close together in this space.

Consider the following three sentences.

1. My Kubernetes pod keeps crashing.
2. What is a CrashLoopBackOff?
3. My application restarts continuously in the cluster.

These sentences use different words, but they describe a similar problem.

An embedding model understands the semantic relationships among concepts such as pod failures, container crashes, and application restarts.

So when it converts these sentences into vectors, the related converted vectors are grouped as shown in the image below.

Now you might ask, what is a Vector space?

You might have got some basic idea of what embeddings are. Now let's see how an embedding model understands context or the meanings of the words.

How Does an Embedding Model Understand Context?

You might be wondering, how does the model know which words are related in the first place?

An embedding model is a model pretrained on billions of words from books, websites, articles, code, etc. So it has context on most of the data you provide to the model. So when it converts these to vectors, it will group related words close to each other.

These models learn by reading sentences and noticing patterns, such as which words often appear together and how their meanings change depending on their surrounding words.

It does this by making a prediction during training. Given the surrounding words, it predicts the missing word. It gets very good at understanding which words belong together.

For example,

• Python is a programming language.
• Python is seen in the jungle.

Even though the word Python is the same, the surrounding words (programming language and seen in the jungle) help the model determine the correct meaning.

Word2Vec gives each word one fixed vector, no matter how it's used, so Python gets the same vector whether you are talking about the language or the snake.

BERT and other Transformer-based models go a step further. They adjust a word's vector based on the surrounding sentence.

So Python gets a different vector depending on whether it appears next to a programming language or a jungle.

Embedding Model Workflow

Now that you understand what embedding is and how it works, let's look at the technical aspects of an embedding model.

When we use an embedding model to generate embeddings, there are 4 major steps inside the model.

Step 1 - Tokenization

Before the model starts to process your text, it needs to break it down into smaller pieces it can work with. These pieces are called tokens.

For example,

My Kubernetes pod keeps crashing.
        |
        v
[My, Kube, ##rnet, ##es, pod, keeps, crash, ##ing]

The model then converts each token into a token ID. Why? Because computers can only understand numbers, not words, to find similarities and study the patterns, the text must be converted to numbers.

Step 2 - Model Processing

Now, the model only has a list of token IDs. But those IDs carry no meaning. The numbers don't tell you anything about the pod or mean how it connects to crashing.

Model Processing is the step where the model figures out the meaning and relationships between all the tokens.

Each token passes through multiple Transformer layers. In each layer, every token looks at every other token in the sentence.

For example,

pod -> crashing   = 0.91(very high — strong connection)
pod -> keeps      = 0.61(medium — keeps describes pod behaviour)
pod -> Kubernetes = 0.45(medium — Kubernetes gives pod context)
pod -> My         = 0.08(very low — My doesn't affect pod's meaning)

This is called attention, where the model pays more attention to words that matter to each other.

By the end of the final transformer layer, the model understands what each word means based on the entire sentence, not just the word alone.

Step 3 - Pooling

After Model Processing, you have one vector per token. But you need one vector for the entire sentence.

For example,

My Kubernetes pod keeps crashing

My       -> [0.2,  0.1,  0.4]
Kube     -> [0.8,  0.6,  0.9]
##rnet   -> [0.7,  0.5,  0.8]
##es     -> [0.6,  0.4,  0.7]
pod      -> [0.9,  0.3,  0.7]
keeps    -> [0.3,  0.6,  0.5]
crash    -> [0.8,  0.9,  0.6]
##ing    -> [0.7,  0.8,  0.5]

These are 8 vectors for one sentence. We need to make it a single vector to represent the full sentence.

You may ask why?

When you need to compare two sentences, you need to compare them as one vector. You cannot compare 8 vectors against 3 other vectors.

For example,

1. My Kubernetes pod keeps crashing - 8 token vectors
2. Pod crashed - 3 token vectors

To compare these two sentences mathematically, both must have the same shape. If one sentence has 8 token vectors and the other has only 3, they cannot be compared directly because there is no clear way to match each token from one sentence with the other.

The process of combining them into a single vector by averaging them is called Pooling.

"My Kubernetes pod keeps crashing"
        |  after pooling
[3.2,  -8.1,  5.6,  0.7,  -4.3,  ...]

Step 4 - Normalization

Imagine you have two sentences,

Sentence A → [3.2,  -8.1,  5.6  ...]   <- large numbers
Sentence B → [0.4,  -0.9,  0.6  ...]   <- small numbers

Without normalization, Sentence A might appear more important to the embedding model, just because it has larger numbers, even if Sentence B actually is similar to Sentence A.

What does Normalization do?

Normalization scales all vectors to the same length without changing their meaning. This makes sure that similarity comparisons focus on the meaning of the vectors, not the size of their values.

Before normalisation
[3.2,  -8.1,  5.6,  ...]   ← raw, unscaled, length = 10.4

         |  scale down to length 1.0

After normalisation
[0.12, -0.45, 0.88, ...]   ← scaled, length = 1.0

Now lets understand where embeddings are used in AI/ML today.

Key Use Cases Of Embeddings

Embeddings help models to understand the meaning and relationships behind words, sentences, and other types of data. This makes machine learning algorithms extract useful patterns and make smarter decisions.

Here are some of the key factors that describe why embeddings are important.

Reduce data dimensionality

Data scientists use embeddings to reduce the data dimensionality from high-dimensional to low-dimensional space.

For example,

An image can be considered as high-dimensional data because each pixel has a colour value, which is a separate dimension.

When we present high-dimensional data for training the model, it requires more computational power, and it takes more time to learn.

So, embeddings reduce the number of dimensions by identifying patterns between various features. This consequently reduces the computing resources and time required to process raw data.

Train LLMs

Embeddings improve data quality when training LLMs.

For example, data scientists use embeddings to clean the training data of irregularities that affect model learning.

Without embeddings, you search for an issue pod not starting, and the search engine looks for those exact words.

But there is a document that says the container failed to initialize, or the application is not starting, never comes up, even though all three have the same meaning. Because the search engine has no idea whether they are related or not.

This can be called a keyword search or word matching.

With embeddings, instead of matching words, the search engine compares meanings. Pod not starting, container failed to initialize, and application not starting all get converted into vectors that get closer to each other in a vector space.

So when you search for one issue, the results for all three come up because the model understands that they describe the same problem, not just the same words.

This is called semantic search ( understands the meaning ).

RAG pipeline

One of the major use cases of embeddings is that they are the core concept of building an RAG (Retrieval-Augmented Generation) pipeline.

When the documents are added to the knowledge base, they are first split into smaller chunks. Each chunk is then converted into a numerical vector using an embedding model (such as AWS Titan Text Embeddings V2).

These vectors are stored in a vector database like Amazon S3 Vectors. When a user asks a question, the same embedding model converts the query into a vector.

The vector database then compares this query vector with the stored vectors and finds the most relevant chunks based on meaning, not just exact keywords.

Semantic search is the core of every RAG system, and embeddings are what make it work.

This is why a question like "Why does my pod keep restarting?" can retrieve a document titled CrashLoopBackOff Troubleshooting Guide even though the words are different.

The embedding model understands that both are talking about the same concept.

Common Types of Embeddings

Once the data is converted into vectors, the main thing here is the kind of embedding you use, which decides what the machine can actually understand from it.

Embeddings mainly differ depending on what they convert, whether it is a single word, a complete sentence, a graph, or an image.

Let's explore the types of embeddings and how they are used.

Word Embeddings

These types of embeddings represent individual words as vectors that capture their meanings and relationships to other words.

For example,

An embedding model, which was trained or fine-tuned on technical texts, learns the terms Kubernetes and Docker, not only the words, but also the meaning, like orchestration and containerization.

So it finds out that both of the terms are related and groups them up close together inside a vector space.

Word2Vec is a model developed by Google in 2013, used to create word embeddings by a two-layer neural network.

Sentence Embeddings

Sentence embeddings represent the meaning of an entire sentence or short paragraph instead of individual words.

This makes sentence embeddings especially useful for tasks such as semantic search, question answering, document retrieval, and Retrieval-Augmented Generation (RAG).

For example,

The pod crashed due to an OOM error, and the container was killed because of a memory shortage.

Both have completely different words but describe the same problem.

Sentence embeddings place them close together in vector space, which is exactly what makes RAG retrieval work.

Sentence embeddings are used for semantic search, document retrieval, summarisation, and grouping.

Some of the popular models, like Sentence-BERT (SBERT), generate high-quality embeddings by fine-tuning BERT for semantic similarity tasks.

Universal Sentence Encoder (Google), and Doc2Vec, are also some other models used for sentence and document embeddings.

Contextual Embeddings

These types of embeddings help AI understand words based on their context. This means that the same word can have different meanings, but it depends on how it is used in a sentence, which leads to more accurate language understanding.

For example,

The word pipeline from the sentence, the RAG pipeline retrieves the chunks and gets a vector close to the retrieval model and data.

But the pipeline in the sentence, The pipeline burst during inspection gets a vector close to engineering.

Same word, completely different meaning, and the model knows it. This type of embedding, which differs based on concepts, is called Contextual Embeddings.

Major use cases of contextual embeddings are Named-entity Recognition (identifying specific services, error types, and timestamps inside raw log text) and question-answering.

Some of the models used for this type of embedding are ELMo (Embeddings from Language Models) and BERT.

Image Embeddings

Image embeddings can be easily called numerical representations of images. It converts an image into a list of numbers that represent its important features, like shapes, colours, patterns, and objects.

Image embeddings help AI systems to understand and compare images based on their content. As a result, these types of embeddings are widely used in tasks such as image search, image recognition, image classification, and finding similar images.

Let's consider an example,

A screenshot of the pod eviction due to resource pressure can be placed near a Grafana dashboard that captures the same incident in the same vector space, making it possible to search infrastructure diagrams or match error screenshots to documentation automatically.

For Image embedding, we can use models like ResNet, CLIP (OpenAI), and Google’s Vision AI.

Graph embeddings

Graph embeddings help AI understand graph data by converting each node (a node is just a single point in a graph) into a numerical vector. These vectors get important information about how nodes are connected and their role.

By converting graph data into vectors, AI models can easily analyze relationships, find similar nodes, and make predictions, such as in social networks, recommendation systems, and knowledge graphs.

Interconnected nodes that are closely related in the graph are usually placed close together in the embedding space.

For example,

Users (nodes) who are connected or have similar friends will have similar embeddings. These embeddings help in link prediction, recommendation, and community detection.

For specific use cases like Recommendation systems and fraud detection, graph embeddings can be used. Popular models like Node2Vec and GraphSAGE are used for this type of embedding.

Multimodal Embeddings

Multimodal embeddings combine text, image, audio, and other data into one shared embedding space.

Search engines that understand text + image queries or AI assistants that process multiple data types use Multimodal Embeddings.

Popular models like CLIP (OpenAI), Flamingo (DeepMind), and Gemini (Google DeepMind) can be used for Multimodal embeddings.

Real World Example: Google BERT

In October 2019, Google integrated BERT directly into its search engine (one of the most significant improvements to search understanding in its history)

Google said that it was its most significant step forward for its search algorithm in the past five years, which impacts nearly 560 million queries a day.

BERT takes a search query and outputs a vector that encodes the meaning of the query, not just its keywords.

This vector captures the meaning of the query and is what Google actually uses to find and rank relevant results.

BERT is a bidirectional model (meaning it looks at the words that come before and after a term to determine context). This is especially important for words that change meaning depending on how they are used. Small words like "to", "for", or "with" can completely change the intent of a search.

For example,

A query like 2019 Brazil traveler to USA needs a visa was previously misunderstood by Google because it could not parse the word to in context.

After BERT, Google correctly understood the query was about a Brazilian traveling to the US, not the other way around.

Some of the other example may be considered recommender systems such as a food-recommendation application, where when users input their food or meals, the app suggest them the food they like according to their favourite ones.

Conclusion

Embeddings are like the secret language that helps AI understand the world. They turn words and images into numbers that carry meaning.

When building an LLM-based application, we can use an Agent Framework (LangGraph, CrewAI) to integrate embedding models and vector stores, enabling us to quickly pull in text data and generate and store embeddings.

In upcoming blog posts, we will explore vector databases and RAG (Retrieval-Augmented Generation) and build a real-time chatbot application.