---

Unpacking Retrievers with LangChain

Data Mastery Series — Episode 37: LangChain Website (Part 12 )

Connect with me and follow our journey: Linkedin, Facebook

---

Welcome to Episode 37 of the Data Mastery Series! Today, we continue our deep dive into LangChain by exploring retrievers, a critical component of intelligent AI workflows. If you’ve been following along, you know how LangChain is revolutionizing the way we build and scale AI applications.

Before we dive in, let’s recap our journey so far:

• Part 1: LangChain Model I/O Basics
• Part 2–3: Prompt Templates and Few-Shot Prompts
• Part 4–6: Deep Dive into Chat Models (Part 1, Part 2, Part 3)
• Part 7: LLM Fundamentals
• Part 8: Output Parsers
• Part 9: Document Loaders
• Part 10: Text Splitter
• Part 11: Embeddings and Vector Stores

Note:This post is inspired by insights from the official LangChain documentation and represents my practical learning journey.

What Are Retrievers?

Retrievers are interfaces in LangChain designed to fetch relevant documents based on an input query. They do not store the data themselves but act as a bridge to access information from various sources. They process a string query and return a list of relevant documents.

For demonstration purposes, we’ll use two text files: “Tortoise and the Hare.txt” and “ลูกหมูสามตัว.txt”.

Tortoise and the Hare.txt

Deep within a sun-dappled clearing of the DC Forest, lived a hare named Piti, famed for his lightning speed. He would streak past the other animals, a blur of brown fur that left them breathless in his wake. One crisp morning, Piti was bragging about his agility, puffing out his chest and flicking his tail with unconcealed pride.
"There's no creature in this entire forest faster than me!" he declared, his voice echoing through the trees.
A slow, rumbling voice came from behind a nearby thicket. It was Donato, a tortoise known for his steady pace and unwavering determination.
"Speed isn't everything, Piti," Donato rumbled. "Even the slowest can achieve victory, if they set their mind to it."
Piti burst into laughter. "You? Win a race against me? Donato, that's the most ludicrous notion I've ever heard!"
To everyone's surprise, Donato challenged Piti to a race. The other animals gathered around, buzzing with excitement at the prospect of such an unequal competition. Even the wise old owl hooted in amusement, his amber eyes twinkling with anticipation.
The race began. Piti shot off like a furry bullet, leaving Donato in a cloud of dust. The animals cheered for the hare, certain of his victory. But Piti, brimming with overconfidence, spotted a patch of wildflowers bursting with color. He darted off the track, unable to resist the temptation of a tasty treat.
Meanwhile, Donato plodded on steadily, never stopping, never wavering. He may have been slow, but his determination burned bright.
Back on the track, Piti, feeling sluggish from his snack, decided to take a nap under the shade of a towering oak. "Old Donato won't catch up to me anyway," he thought arrogantly.
He drifted off to sleep, picturing himself crossing the finish line first to a chorus of cheers. But time crawled by for the sleeping hare, while for Donato, it marched on relentlessly.
Donato, inch by inch, made his way towards the finish line. The animals, who had initially mocked him, now cheered him on, their voices echoing through the forest. They were impressed by his unwavering perseverance.
Finally, Donato, with a triumphant plod, crossed the finish line. Piti woke up with a start, his ears twitching in disbelief. He saw, to his utter humiliation, the crowd celebrating Donato's victory.
The hare had lost the race, not to speed, but to slow and steady determination. The cheers of the animals resonated through the DC Forest, a testament to the fact that slow and steady truly does win the race.

ลูกหมูสามตัว.txt

กาลครั้งหนึ่งนานมาแล้ว มีแม่หมูอยู่กับลูกหมูสามตัว ลูกหมูทั้งสามโตหมดแล้ว ถึงเวลาที่พวกเขาต้องสร้างบ้านของตัวเองและออกไปผจญภัยในโลกกว้าง
ลูกหมูโต ชื่อ ข้าวสวย- พี่ชายคนโต เป็นหมูที่ค่อนข้างขี้เกียจ เขาตัดสินใจสร้างบ้านอย่างรวดเร็ว เจอกองฟางข้าวใกล้ๆ นึกในใจว่า "แค่นี้ก็พอแล้ว!" จากนั้นเขาก็รีบๆ สร้างบ้านฟางฟางที่กางออกง่ายๆ
ลูกหมูรอง ชื่อ ข้าวหอม- น้องชายคนที่สอง ไม่ได้ขี้เกียจเท่าพี่ชาย เขาตัดสินใจสร้างบ้านที่แข็งแรงกว่า เขาเก็บกิ่งไม้และก้านไม้จากป่ามาสร้างบ้านไม้ มันไม่ใช่บ้านที่แข็งแรงที่สุด แต่เขาคิดว่ามันก็น่าจะเพียงพอ
ลูกหมูเล็กชื่อ ข้าวสุก - น้องชายคนสุดท้อง ขึ้นชื่อเรื่องการทำงานหนักและวางแผน เขาต้องการบ้านที่จะทำให้เขามีความปลอดภัยอย่างแท้จริง เขาใช้เวลาหลายวันในการเก็บอิฐที่แข็งแรงและสร้างบ้านอิฐที่แข็งแรงอย่างระมัดระวัง เขายังสร้างปล่องไฟและประตูที่เหมาะสมพร้อมกับกุญแจที่แข็งแรง
วันแดดสดใสวันหนึ่ง มีหมาป่าตัวใหญ่ใจร้ายมีนามว่า แกงส้ม เดินเลาะเลียบไปตามทาง มันได้กลิ่นหอมกรุ่นของหมูย่างและเดินตามกลิ่นนั้นไปจนถึงบ้านฟางของลูกหมูตัวแรก
"ลูกหมู ๆ เปิดประตูให้ข้าเข้าไปสิ!" หมาป่าคำราม
"ไม่! ข้าจะไม่ให้เจ้าเข้ามา!" ลูกหมูตัวเล็กตกใจร้องเสียงแหลม
หมาป่าสูดลมพ่นออกแรง ๆ เป่าบ้านฟางจนพังทลาย! ลูกหมูตัวเล็กกรี๊ดร้องและวิ่งหนีเร็วที่สุดเท่าที่จะวิ่ง
เขาวิ่งไปที่บ้านไม้ของพี่ชาย หมาป่าวิ่งตามมาติดๆ และเมื่อมาถึงบ้านไม้ มันก็ร้องขอ "ลูกหมู ๆ เปิดประตูให้ข้าเข้าไปสิ!"
"ไม่! พวกเราจะไม่ให้เจ้าเข้ามา!" ลูกหมูทั้งสองตัวร้องออกมาพร้อมกัน
หมาป่าสูดลมพ่นออกแรง ๆ เป่าบ้านไม้จนพังทลาย! ลูกหมูทั้งสองตัวกรี๊ดร้องและวิ่งหนีเร็วที่สุดเท่าที่จะวิ่ง ไปที่บ้านอิฐของน้องชายคนสุดท้อง
หมาป่าตอนนี้หิวโซกว่าเดิม วิ่งตามพวกมันไป เมื่อมาถึงบ้านอิฐ มันตะโกนว่า "ลูกหมู ๆ เปิดประตูให้ข้าเข้าไปสิ!"
"ไม่! พวกเราจะไม่ให้เจ้าเข้ามา!" ลูกหมูทั้งสามตัวตะโกนออกมาพร้อมกัน ตอนนี้พวกเขารู้สึกกล้าหาญมากขึ้นเพราะพวกเขาปลอดภัยอยู่ข้างใน
หมาป่าสูดลมพ่นออกแรง ๆ เป่าบ้านอิฐจนสุดแรงเกิด มันเป่าแรงจนหน้าแดง แต่บ้านอิฐก็ไม่ขยับเขยื้อน หมาป่าหงุดหงิดและพ่ายแพ้ ยอมแพ้และเดินโซซัดเซไปในป่า
ลูกหมูทั้งสามตัว ปลอดภัยอยู่ในบ้านอิฐ พวกเขาได้เรียนบทเรียนอันมีค่าในวันนั้น ความขยันหมั่นเพียรและการวางแผนนั้นสำคัญ และรากฐานที่แข็งแกร่งสามารถช่วยให้คุณปลอดภัยจากอันตราย พวกเขาอยู่ด้วยกันอย่างมีความสุขตลอดไปในบ้านอิฐที่แข็งแรง ไม่เคยลืมความสำคัญของการเตรียมตัว

1) Vectorstore Retrievers: Simple and Powerful

The vectorstore retriever is the simplest type, leveraging vector embeddings to find documents based on semantic similarity.

• Index Type: Vectorstore
• Uses an LLM: No
• Use Cases: Quick setup for basic similarity-based search.

Example - Vectorstore Retrievers

Load and split documents

loader = TextLoader("Tortoise and the Hare.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=250, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

Create embeddings and vector store

embeddings = OpenAIEmbeddings(api_key=OPENAI_API_KEY)
db = FAISS.from_documents(texts, embeddings)

Create retriever and query

retriever = db.as_retriever()
docs = retriever.invoke("Who win? Why type of animal?, What name? and Why win?")

Example of output of docs

"""
Chunk 1:
To everyone's surprise, Donato challenged Piti to a race. The other animals gathered around, buzzing with excitement at the prospect of such an unequal competition. Even the wise old owl hooted in amusement, his amber eyes twinkling with anticipation.

Chunk 2:
The hare had lost the race, not to speed, but to slow and steady determination. The cheers of the animals resonated through the DC Forest, a testament to the fact that slow and steady truly does win the race.

Chunk 3:
Tortoise and the Hare

Chunk 4:
The race began. Piti shot off like a furry bullet, leaving Donato in a cloud of dust. The animals cheered for the hare, certain of his victory. But Piti, brimming with overconfidence, spotted a patch of wildflowers bursting with color. He darted off the track, unable to resist the temptation of a tasty treat.

"""

Customization Options for Vectorstore Retrievers

• Maximum Marginal Relevance (MMR):
  Balances relevance and diversity in results.

Example - Vectorstore Retrievers - MMR

retriever = db.as_retriever(search_type="mmr")
docs = retriever.invoke("Who win? Why type of animal?, What name? and Why win?")

Example of output of docs

"""
Chunk 1:
To everyone's surprise, Donato challenged Piti to a race. The other animals gathered around, buzzing with excitement at the prospect of such an unequal competition. Even the wise old owl hooted in amusement, his amber eyes twinkling with anticipation.

Chunk 2:
Tortoise and the Hare

Chunk 3:
The hare had lost the race, not to speed, but to slow and steady determination. The cheers of the animals resonated through the DC Forest, a testament to the fact that slow and steady truly does win the race.

Chunk 4:
Deep within a sun-dappled clearing of the DC Forest, lived a hare named Piti, famed for his lightning speed. He would streak past the other animals, a blur of brown fur that left them breathless in his wake. One crisp morning, Piti was bragging about his agility, puffing out his chest and flicking his tail with unconcealed pride.

"""

• Similarity Score Threshold:
  Filter results based on a similarity score:

Example - Vectorstore Retrievers - Similarity Score

retriever = db.as_retriever(
search_type="similarity_score_threshold",
search_kwargs={"score_threshold": 0.5}
)
docs = retriever.invoke("Who win? Why type of animal?, What name? and Why win?")

Example of output of docs

"""
Chunk 1:
To everyone's surprise, Donato challenged Piti to a race. The other animals gathered around, buzzing with excitement at the prospect of such an unequal competition. Even the wise old owl hooted in amusement, his amber eyes twinkling with anticipation.

Chunk 2:
The hare had lost the race, not to speed, but to slow and steady determination. The cheers of the animals resonated through the DC Forest, a testament to the fact that slow and steady truly does win the race.

Chunk 3:
Tortoise and the Hare

Chunk 4:
The race began. Piti shot off like a furry bullet, leaving Donato in a cloud of dust. The animals cheered for the hare, certain of his victory. But Piti, brimming with overconfidence, spotted a patch of wildflowers bursting with color. He darted off the track, unable to resist the temptation of a tasty treat.

"""

• Top-K Results:
  Limit the number of returned results:

Example - Vectorstore Retrievers - Top-K

retriever = db.as_retriever(search_kwargs={"k": 1})
docs = retriever.invoke("Who win? Why type of animal?, What name? and Why win?")

Example of output of docs

"""
Chunk 1:
To everyone's surprise, Donato challenged Piti to a race. The other animals gathered around, buzzing with excitement at the prospect of such an unequal competition. Even the wise old owl hooted in amusement, his amber eyes twinkling with anticipation.

"""

2) ParentDocument Retrievers: Context Matters

When documents are split into smaller chunks for embedding, you may lose the broader context. ParentDocument retrievers address this by:

1. Splitting documents into small chunks for accurate indexing.
2. Returning the larger parent document or combined chunks during retrieval.

• Index Type: Vectorstore + Document Store
• Uses an LLM: No
• Use Cases: Granular indexing with a broader context during retrieval.

Example - Parent Document Retriever - full documents

Load and split documents

loaders = [
TextLoader("../../Tortoise and the Hare.txt"),
TextLoader("../../ลูกหมูสามตัว.txt"),
]
docs = []
for loader in loaders:
docs.extend(loader.load())

Split and store documents

child_splitter = RecursiveCharacterTextSplitter(chunk_size=250)
vectorstore = Chroma(collection_name="full_documents", embedding_function=OpenAIEmbeddings())
store = InMemoryStore()

Initialize retriever

retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=store,
child_splitter=child_splitter,
)
retriever.add_documents(docs, ids=None)

Retrieve parent documents

sub_docs = vectorstore.similarity_search("Who win? Why type of animal?, What name? and Why win?")
retrieved_docs = retriever.invoke("Who win? Why type of animal?, What name? and Why win?")

Example of output of retrieved_docs

"""
Chunk 1:
Tortoise and the Hare
Deep within a sun-dappled clearing of the DC Forest, lived a hare named Piti, famed for his lightning speed. He would streak past the other animals, a blur of brown fur that left them breathless in his wake. One crisp morning, Piti was bragging about his agility, puffing out his chest and flicking his tail with unconcealed pride.
"There's no creature in this entire forest faster than me!" he declared, his voice echoing through the trees.
A slow, rumbling voice came from behind a nearby thicket. It was Donato, a tortoise known for his steady pace and unwavering determination.
"Speed isn't everything, Piti," Donato rumbled. "Even the slowest can achieve victory, if they set their mind to it."
Piti burst into laughter. "You? Win a race against me? Donato, that's the most ludicrous notion I've ever heard!"
To everyone's surprise, Donato challenged Piti to a race. The other animals gathered around, buzzing with excitement at the prospect of such an unequal competition. Even the wise old owl hooted in amusement, his amber eyes twinkling with anticipation.
The race began. Piti shot off like a furry bullet, leaving Donato in a cloud of dust. The animals cheered for the hare, certain of his victory. But Piti, brimming with overconfidence, spotted a patch of wildflowers bursting with color. He darted off the track, unable to resist the temptation of a tasty treat.
Meanwhile, Donato plodded on steadily, never stopping, never wavering. He may have been slow, but his determination burned bright.
Back on the track, Piti, feeling sluggish from his snack, decided to take a nap under the shade of a towering oak. "Old Donato won't catch up to me anyway," he thought arrogantly.
He drifted off to sleep, picturing himself crossing the finish line first to a chorus of cheers. But time crawled by for the sleeping hare, while for Donato, it marched on relentlessly.
Donato, inch by inch, made his way towards the finish line. The animals, who had initially mocked him, now cheered him on, their voices echoing through the forest. They were impressed by his unwavering perseverance.
Finally, Donato, with a triumphant plod, crossed the finish line. Piti woke up with a start, his ears twitching in disbelief. He saw, to his utter humiliation, the crowd celebrating Donato's victory.
The hare had lost the race, not to speed, but to slow and steady determination. The cheers of the animals resonated through the DC Forest, a testament to the fact that slow and steady truly does win the race.

"""

Example - Parent Document Retriever - split_parents

Load and split documents

loaders = [
TextLoader("../../Tortoise and the Hare.txt"),
TextLoader("../../ลูกหมูสามตัว.txt"),
]
docs = []
for loader in loaders:
docs.extend(loader.load())

Split and store documents

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)
vectorstore = Chroma(collection_name="split_parents", embedding_function=OpenAIEmbeddings(api_key=OPENAI_API_KEY))
store = InMemoryStore()

Initialize retriever

retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=store,
child_splitter=child_splitter,
)
retriever.add_documents(docs)

Retrieve parent documents

sub_docs = vectorstore.similarity_search("Who win? Why type of animal?, What name? and Why win?")
retrieved_docs = retriever.invoke("Who win? Why type of animal?, What name? and Why win?")

Example of output of retrieved_docs

"""
Chunk 1:
He drifted off to sleep, picturing himself crossing the finish line first to a chorus of cheers. But time crawled by for the sleeping hare, while for Donato, it marched on relentlessly.
Donato, inch by inch, made his way towards the finish line. The animals, who had initially mocked him, now cheered him on, their voices echoing through the forest. They were impressed by his unwavering perseverance.
Finally, Donato, with a triumphant plod, crossed the finish line. Piti woke up with a start, his ears twitching in disbelief. He saw, to his utter humiliation, the crowd celebrating Donato's victory.
The hare had lost the race, not to speed, but to slow and steady determination. The cheers of the animals resonated through the DC Forest, a testament to the fact that slow and steady truly does win the race.

"""

3) Multi-Vector Retrievers

Multi-vector retrievers allow the creation of multiple vector embeddings for each document, facilitating more precise searches.

• Index Type: Vectorstore + Document Store
• Uses an LLM: Sometimes during indexing
• Use Cases: When relevant information can be extracted and indexed differently from the source text.

Example - Multi-Vector Retriever - Smaller chunks

Load and split documents

loaders = [
TextLoader("../../Tortoise and the Hare.txt"),
TextLoader("../../ลูกหมูสามตัว.txt"),
]
docs = []
for loader in loaders:
docs.extend(loader.load())

Load and split documents

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000)
docs = text_splitter.split_documents(docs)

Create vectorstore and storage

vectorstore = Chroma(collection_name="Smaller_chunks", embedding_function=OpenAIEmbeddings())
store = InMemoryByteStore()

Initialize retriever

retriever = MultiVectorRetriever(
vectorstore=vectorstore,
byte_store=store,
id_key="doc_id",
)

Setting document id and split to smaller chunks

doc_ids = [str(uuid.uuid4()) for _ in docs]
child_text_splitter = RecursiveCharacterTextSplitter(chunk_size=400)
sub_docs = []
for i, doc in enumerate(docs):
_id = doc_ids[i]
_sub_docs = child_text_splitter.split_documents([doc])
for _doc in _sub_docs:
_doc.metadata[id_key] = _id
sub_docs.extend(_sub_docs)

retriever.vectorstore.add_documents(sub_docs)
retriever.docstore.mset(list(zip(doc_ids, docs)))
docs_smaller_chunks = retriever.vectorstore.similarity_search("Who win? Why type of animal?, What name? and Why win?")[0]

Example of output of docs_Smaller_chunks.page_content

"""
The hare had lost the race, not to speed, but to slow and steady determination. The cheers of the animals resonated through the DC Forest, a testament to the fact that slow and steady truly does win the race.
"""

Example - Multi-Vector Retriever - summary

Load and split documents

loaders = [
TextLoader("../../Tortoise and the Hare.txt"),
TextLoader("../../ลูกหมูสามตัว.txt"),
]
docs = []
for loader in loaders:
docs.extend(loader.load())

Load and split documents

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000)
docs = text_splitter.split_documents(docs)
chain = (
{"doc": lambda x: x.page_content}
| ChatPromptTemplate.from_template("Summarize the following document:\n\n{doc}")
| ChatOpenAI(max_retries=0, api_key=OPENAI_API_KEY)
| StrOutputParser()
)
summaries = chain.batch(docs, {"max_concurrency": 5})

Create vectorstore and storage

vectorstore = Chroma(collection_name="summaries", embedding_function=OpenAIEmbeddings(api_key=OPENAI_API_KEY))
store = InMemoryByteStore()

Initialize retriever

id_key = "doc_id"
retriever = MultiVectorRetriever(
vectorstore=vectorstore,
byte_store=store,
id_key=id_key,
)
doc_ids = [str(uuid.uuid4()) for _ in docs]

Create summary documents

summary_docs = [
Document(page_content=s, metadata={id_key: doc_ids[i]})
for i, s in enumerate(summaries)
]

#add documents and data
retriever.vectorstore.add_documents(summary_docs)
retriever.docstore.mset(list(zip(doc_ids, docs)))
sub_docs = vectorstore.similarity_search("Who win? Why type of animal?, What name? and Why win?")
retrieved_docs = retriever.invoke("Who win? Why type of animal?, What name? and Why win?")

Example of output of retrieved_docs

"""
Chunk 1:
He drifted off to sleep, picturing himself crossing the finish line first to a chorus of cheers. But time crawled by for the sleeping hare, while for Donato, it marched on relentlessly.
Donato, inch by inch, made his way towards the finish line. The animals, who had initially mocked him, now cheered him on, their voices echoing through the forest. They were impressed by his unwavering perseverance.

Chunk 2:
Finally, Donato, with a triumphant plod, crossed the finish line. Piti woke up with a start, his ears twitching in disbelief. He saw, to his utter humiliation, the crowd celebrating Donato's victory.
The hare had lost the race, not to speed, but to slow and steady determination. The cheers of the animals resonated through the DC Forest, a testament to the fact that slow and steady truly does win the race.

Chunk 3:
หมาป่าตอนนี้หิวโซกว่าเดิม วิ่งตามพวกมันไป เมื่อมาถึงบ้านอิฐ มันตะโกนว่า "ลูกหมู ๆ เปิดประตูให้ข้าเข้าไปสิ!"
"ไม่! พวกเราจะไม่ให้เจ้าเข้ามา!" ลูกหมูทั้งสามตัวตะโกนออกมาพร้อมกัน ตอนนี้พวกเขารู้สึกกล้าหาญมากขึ้นเพราะพวกเขาปลอดภัยอยู่ข้างใน
หมาป่าสูดลมพ่นออกแรง ๆ เป่าบ้านอิฐจนสุดแรงเกิด มันเป่าแรงจนหน้าแดง แต่บ้านอิฐก็ไม่ขยับเขยื้อน หมาป่าหงุดหงิดและพ่ายแพ้ ยอมแพ้และเดินโซซัดเซไปในป่า

Chunk 4:
Piti burst into laughter. "You? Win a race against me? Donato, that's the most ludicrous notion I've ever heard!"
To everyone's surprise, Donato challenged Piti to a race. The other animals gathered around, buzzing with excitement at the prospect of such an unequal competition. Even the wise old owl hooted in amusement, his amber eyes twinkling with anticipation.

"""

Example - Multi-Vector Retriever - Hypothetical Queries

Load and split documents

loaders = [
TextLoader("../../Tortoise and the Hare.txt"),
TextLoader("../../ลูกหมูสามตัว.txt"),
]
docs = []
for loader in loaders:
docs.extend(loader.load())

Load and split documents

text_splitter = RecursiveCharacterTextSplitter(chunk_size=500)
docs = text_splitter.split_documents(docs)

Define function schema

functions = [
{
"name": "hypothetical_questions",
"description": "Generate hypothetical questions",
"parameters": {
"type": "object",
"properties": {
"questions": {
"type": "array",
"items": {"type": "string"},
},
},
"required": ["questions"],
},
}
]

Create question generation chain

chain = (
{"doc": lambda x: x.page_content}
| ChatPromptTemplate.from_template(
"Generate a list of exactly 6 hypothetical questions that the below document could be used to answer:\n\n{doc}"
)
| ChatOpenAI(max_retries=0, model="gpt-4o", api_key=OPENAI_API_KEY).bind(
functions=functions, function_call={"name": "hypothetical_questions"}
)
| JsonKeyOutputFunctionsParser(key_name="questions")
)

Example of output of chain.invoke(docs[0])

"""
['What could happen if the hare continues to boast about his speed in the forest?',
"How might the other animals react to the hare's constant bragging?",
"What lessons could be learned from the hare's attitude towards speed and competition?",
'What are the potential consequences for the hare if he challenges another animal to a race?',
'How might the dynamics in the forest change if the hare loses a race to a slower animal?',
'What strategies might the hare employ to maintain his reputation if challenged by another animal?']
"""

Create hypothetical questions

hypothetical_questions = chain.batch(docs, {"max_concurrency": 6})

Create vector store and multi vector retriever

vectorstore = Chroma(
collection_name="hypo-questions", embedding_function=OpenAIEmbeddings(api_key=OPENAI_API_KEY)
)
store = InMemoryByteStore()
id_key = "doc_id"
retriever = MultiVectorRetriever(
vectorstore=vectorstore,
byte_store=store,
id_key=id_key,
)

#setting ids for each document and create document for each question
doc_ids = [str(uuid.uuid4()) for _ in docs]
question_docs = []
for i, question_list in enumerate(hypothetical_questions):
question_docs.extend(
[Document(page_content=s, metadata={id_key: doc_ids[i]}) for s in question_list]
)

#Add document to vector store and doc store
retriever.vectorstore.add_documents(question_docs)
retriever.docstore.mset(list(zip(doc_ids, docs)))
sub_docs = vectorstore.similarity_search("Who win? Why type of animal?, What name? and Why win?")
retrieved_docs = retriever.invoke("Who win? Why type of animal?, What name? and Why win?")

Example of output of retrieved_docs

"""
Chunk 1:
The race began. Piti shot off like a furry bullet, leaving Donato in a cloud of dust. The animals cheered for the hare, certain of his victory. But Piti, brimming with overconfidence, spotted a patch of wildflowers bursting with color. He darted off the track, unable to resist the temptation of a tasty treat.
Meanwhile, Donato plodded on steadily, never stopping, never wavering. He may have been slow, but his determination burned bright.

Chunk 2:
Piti burst into laughter. "You? Win a race against me? Donato, that's the most ludicrous notion I've ever heard!"
To everyone's surprise, Donato challenged Piti to a race. The other animals gathered around, buzzing with excitement at the prospect of such an unequal competition. Even the wise old owl hooted in amusement, his amber eyes twinkling with anticipation.

"""

Key Takeaways

Retrievers are a cornerstone of intelligent data retrieval in LangChain. In this episode, we explored:

1. Vectorstore Retrievers: Simple and efficient for semantic similarity searches.
2. ParentDocument Retrievers: Perfect for maintaining context while indexing granular data.
3. Multi-Vector Retrievers: Ideal for complex or multi-dimensional data retrieval.

In Episode 38, we’ll explore other retrieval techniques such as Self-Query Retrievers, Contextual Compression, Time-Weighted Retrievers, and more. Stay tuned to unlock the next level of intelligent data retrieval! 🚀

---

Data Science Explore the world of data science with Donato_Story

Dashboard Discover the power of data visualization with Donato_Story

Donato_Journey Join me on my journey (Thai version)

Course_Review Discover the training courses with Donato_Story (Thai version)

Let’s Connect!

Your thoughts and feedback are invaluable. Feel free to share them in the comments or connect with me on

• Medium: medium.com/donato-story
• Facebook: web.facebook.com/DonatoStory
• Linkedin: linkedin.com/in/nattapong-thanngam