Embedding – The Core of LLM

Whether you're working with a simple open-source AI model or the most advanced systems for processing and generating text, they all share one fundamental principle: converting text into a vector matrix and searching for similarities within their database. The result is then further processed into "human language."

How does it work? Let's demonstrate with a simple Python script:

1. from openai import OpenAI
2. import re
3. def normalize_text(text):
4.      text = text.lower()
5.      text = re.sub(r"\s+", "", text)
6.      return text
7. client = OpenAI(api_key='sk-proj-....')
8. def get_embedding2(text, model="text-embedding-3-small"):
9.      normalized_text = normalize_text(text)
10.      return client.embeddings.create(input=[normalized_text], model=model).data[0].embedding
11. emb = get_embedding2("Some text")
12. print(emb)

In this script, we first normalize the text by converting it to lowercase and removing extra spaces. Then, using the OpenAI client with your API key, we create an embedding (which is 1536th dimensional vector of float numbers) for the normalized text with a specified model. The resulting vector captures the semantic features of the text, ready to be used for similarity searches or further processing.

This simple example highlights the underlying process that powers many large language models today. Enjoy exploring and experimenting with embeddings!