AI & Generative AI

Unpacking Document loaders with LangChain

Data Mastery Series — Episode 34: LangChain Website (Part 9)

29 Nov 202425 min readLangChainRAGDashboard

Unpacking Document Loaders with LangChain

Data Mastery Series — Episode 34: LangChain Website (Part 9)

Connect with me and follow our journey: Linkedin, Facebook

Welcome to Episode 34 of the Data Mastery Series, where we continue our exploration of LangChain, a transformative tool for integrating AI into real-world applications. If you’ve been following along, you’ve already seen how LangChain elevates data workflows. Here’s a recap of the journey so far:

Part 1: LangChain Model I/O Basics
Part 2–3: Prompt Templates and Few-Shot Prompts
Part 4–6: Deep Dive into Chat Models (Part 1, Part 2, Part 3)
Part 7: LLM Fundamentals
Part 8: Output Parsers

Note:This post is a reflection of my learning journey with LangChain, inspired by insights from the official documentation and related resources. The content is based on resources found link. I hope it proves valuable to you!

In today’s episode, we’re diving into Retrieval, an essential step in building AI systems that interact with external data. Specifically, we’ll explore how LangChain handles document loading — a critical first step in the Retrieval Augmented Generation (RAG) process.

What Is Retrieval?

Retrieval refers to the ability of AI models to fetch and utilize external data that is not part of their original training set. This process enhances the relevance and accuracy of AI-generated outputs by providing context from up-to-date or domain-specific data.

Key Steps in Retrieval Augmented Generation (RAG):

Retrieve: Fetch data from external sources.
Generate: Use the retrieved data as context to generate text with LLMs.

LangChain supports this entire workflow with tools for document loading, text splitting, embedding creation, and more.

Key Components of LangChain’s Retrieval Workflow

Document Loaders: Document Loaders are the entry points for bringing external data into LangChain. They handle data ingestion from diverse sources such as websites, PDFs, databases, and more. LangChain offers extensive support for various document loaders, making it easy to connect to almost any data source.
Text Splitters: Large documents are segmented into smaller chunks to improve processing efficiency. Text splitters ensure that the data is divided into manageable pieces while maintaining coherence.
Text Embedding Models: These models convert text into numerical vectors that represent semantic meaning. Similar ideas are located close together in vector space, enabling effective similarity-based search and retrieval.
Vector Stores: Vector stores are specialized databases for storing and retrieving text embeddings. LangChain integrates with several vector storage options, including both in-memory and cloud-based solutions.
Retrievers: Retrievers fetch the most relevant information from vector stores based on a given query. LangChain supports multiple retriever types, including:
- Simple Semantic Search: A straightforward approach for quick results.
- Parent Document Retriever: Provides full document context even if only part of it matches the query.
- Self-Query Retriever: Reformulates complex queries for better accuracy.
- Ensemble Retriever: Combines multiple retrieval methods for robust results.
Indexing: Efficient indexing improves retrieval speed and accuracy. LangChain’s tools help optimize indexing, avoiding redundant calculations and ensuring that stored data is always up-to-date.

LangChain simplifies RAG by providing tools for document loading, text splitting, embedding creation, and more. Let’s start with the foundation: Document Loaders.

Highlighting Document Loaders:

1. CSV: Structuring Tabular Data for AI

CSV (Comma-Separated Values) is one of the most common formats for structured data storage. LangChain’s CSVLoader efficiently converts CSV files into Document objects, making the data ready for processing by LLMs.

The mlb_teams_2012.csv file can be accessed here.

Basic CSV Loading: Quickly parse and transform CSV files into usable documents.

Example - Basic CSV Loading

Initialize the loader with the CSV file path

loader = CSVLoader(file_path='/path/to/mlb_teams_2012.csv')

Load data

data = loader.load()

Print loaded data

data

Example Output (only 2 row):

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/mlb_teams_2012.csv', 'row': 0}, page_content='Team: Nationals\n"Payroll (millions)": 81.34\n"Wins": 98'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/mlb_teams_2012.csv', 'row': 1}, page_content='Team: Reds\n"Payroll (millions)": 82.20\n"Wins": 97'),
'''

Customizing CSV Loading: You can customize how the CSV file is parsed using the csv_args parameter. For example, specify delimiters, quote characters, or custom field names.

Example - Customizing CSV Loading

loader = CSVLoader(
file_path='/path/to/mlb_teams_2012.csv',
csv_args={
'delimiter': ',',
'quotechar': '"',
'fieldnames': ['MLB Team', 'Payroll in millions', 'Wins']
}
)

data = loader.load()
data

Example Output (only 2 row):

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/mlb_teams_2012.csv', 'row': 0}, page_content='MLB Team: Team\nPayroll in millions: "Payroll (millions)"\nWins: "Wins"'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/mlb_teams_2012.csv', 'row': 1}, page_content='MLB Team: Nationals\nPayroll in millions: 81.34\nWins: 98'),
'''

Using a Column as Source: Specify a column (e.g., “Team”) to act as the source in the metadata for each Document.

Example - Customizing CSV Loading

loader = CSVLoader(file_path='/path/to/mlb_teams_2012.csv', source_column="Team")
data = loader.load()
print(data)

Example Top 2 row of Output:

'''
[Document(metadata={'source': 'Nationals', 'row': 0}, page_content='Team: Nationals\n"Payroll (millions)": 81.34\n"Wins": 98'),
Document(metadata={'source': 'Reds', 'row': 1}, page_content='Team: Reds\n"Payroll (millions)": 82.20\n"Wins": 97'),
'''

2. File Directories: Handling Multiple Files Efficiently

LangChain’s DirectoryLoader simplifies the process of loading multiple files from a directory, making it ideal for large-scale projects.

I have created a folder in Google Drive containing two files:
- Test 1.docx: Contains the text “Test 1.”
- Test 2.docx: Contains the text “Test 2.”

Loading Files from a Directory with Adding Progress Indicators: Track the loading progress for better visibility, especially with large directories.

Example - File Directories Adding Progress Indicators

Load Markdown files from a directory

loader = DirectoryLoader('/path/to/Test', glob="**/*.docx", show_progress=True)
docs = loader.load()

Output:

'''
100%|██████████| 2/2 [00:00<00:00, 25.07it/s]
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/Test/Test 1.docx'}, page_content='Test 1'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/Test/Test 2.docx'}, page_content='Test 2')]
'''

Using Multithreading for Faster Loading: Speed up file loading by processing multiple files simultaneously.

Example - File Directories Using Multithreading for Faster Loading

Load Markdown files from a directory

loader = DirectoryLoader('/path/to/Test', glob="**/*.docx", use_multithreading=True)
docs = loader.load()
docs

Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/Test/Test 1.docx'}, page_content='Test 1'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/Test/Test 2.docx'}, page_content='Test 2')]
'''

Customizing the Loader Class: Handle specific file types, such as Word documents, by defining custom loader classes.

Example - Customizing the Loader Class

loader = DirectoryLoader(
'/path/to/Test',
glob="**/*.docx",
loader_cls=UnstructuredWordDocumentLoader
)

docs = loader.load()
docs

Output:

Handling Encoding and Errors: Auto-detect encoding and handle errors gracefully, ensuring smooth processing across diverse file formats.

Example - Handling Encoding and Errors (Auto-detect file encoding)

loader = DirectoryLoader(
'/path/to/Test',
glob="**/*.docx",
loader_cls=UnstructuredWordDocumentLoader,
loader_kwargs=text_loader_kwargs,
)
documents = loader.load()
documents

Output:

Example - Handle loading errors silently (avoiding exceptions)

loader = DirectoryLoader(
"/path/to/Test",
glob="**/*.docx",
loader_cls=UnstructuredWordDocumentLoader,
silent_errors=True, # Files with errors will be skipped
)
documents = loader.load()

doc_sources = [doc.metadata["source"] for doc in documents]
doc_sources

Output:

'''
['/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/Test/Test 1.docx',
'/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/Test/Test 2.docx']
'''

3. HTML: Extracting Content from Web Pages

HTML files, often containing structured content from web pages, are a valuable resource for many AI applications. LangChain supports HTML parsing with tools like:

The fake-content.html file is available here.

Using UnstructuredHTMLLoader: Extracts plain text content from HTML files.

Example - HTML Using UnstructuredHTMLLoader

loader = UnstructuredHTMLLoader('/path/to/fake-content.html')
data = loader.load()
data

Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/fake-content.html'}, page_content='My First Heading\n\nMy first paragraph.')]
'''

Using BSHTMLLoader with BeautifulSoup: Leverages BeautifulSoup to extract structured elements, such as titles and metadata, for deeper insights.

Example - HTML Using BSHTMLLoader with BeautifulSoup

loader = UnstructuredHTMLLoader('/path/to/HTML_LINK')
data = loader.load()
data

Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/fake-content.html', 'title': 'Test Title'}, page_content='\nTest Title\n\n\nMy First Heading\nMy first paragraph.\n\n\n')]
'''

4. JSON: Parsing and Structuring Complex Data

JSON (JavaScript Object Notation) is widely used for structured data exchange. LangChain’s JSONLoader allows for precise and efficient data extraction.

The facebook_chat.json file can be found here. and the facebook_chat_messages.jsonl file is located here.

Basic JSON Loading: For robust loading, especially with diverse file types, consider these options:

Example - JSON (Basic JSON Loading)

file_path = '/path/to/facebook_chat.json'
data = json.loads(Path(file_path).read_text())

pprint(data)

Output:

'''
{'image': {'creation_timestamp': 1675549016, 'uri': 'image_of_the_chat.jpg'},
'is_still_participant': True,
'joinable_mode': {'link': '', 'mode': 1},
'magic_words': [],
'messages': [{'content': 'Bye!',
'sender_name': 'User 2',
'timestamp_ms': 1675597571851},
{'content': 'Oh no worries! Bye',
'sender_name': 'User 1',
'timestamp_ms': 1675597435669},
{'content': 'No Im sorry it was my mistake, the blue one is not '
'for sale',
'sender_name': 'User 2',
'timestamp_ms': 1675596277579},
{'content': 'I thought you were selling the blue one!',
'sender_name': 'User 1',
'timestamp_ms': 1675595140251},
{'content': 'Im not interested in this bag. Im interested in the '
'blue one!',
'sender_name': 'User 1',
'timestamp_ms': 1675595109305},
{'content': 'Here is $129',
'sender_name': 'User 2',
'timestamp_ms': 1675595068468},
{'content': '',
'photos': [{'creation_timestamp': 1675595059,
'uri': 'url_of_some_picture.jpg'}],
'sender_name': 'User 2',
'timestamp_ms': 1675595060730},
{'content': 'Online is at least $100',
'sender_name': 'User 2',
'timestamp_ms': 1675595045152},
{'content': 'How much do you want?',
'sender_name': 'User 1',
'timestamp_ms': 1675594799696},
{'content': 'Goodmorning! $50 is too low.',
'sender_name': 'User 2',
'timestamp_ms': 1675577876645},
{'content': 'Hi! Im interested in your bag. Im offering $50. Let '
'me know if you are interested. Thanks!',
'sender_name': 'User 1',
'timestamp_ms': 1675549022673}],
'participants': [{'name': 'User 1'}, {'name': 'User 2'}],
'thread_path': 'inbox/User 1 and User 2 chat',
'title': 'User 1 and User 2 chat'}
'''

Using JSONLoader for Structured Retrieval: Use jq_schema to specify the data structure and extract only the required fields (Schema-Based Retrieval).

Example - JSON (Using JSONLoader for Structured Retrieval)

loader = JSONLoader(
file_path='/path/to/facebook_chat.json',
jq_schema='.messages[].content', # Specify path to content
text_content=False
)

data = loader.load()
pprint(data)

Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 1}, page_content='Bye!'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 2}, page_content='Oh no worries! Bye'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 3}, page_content='No Im sorry it was my mistake, the blue one is not for sale'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 4}, page_content='I thought you were selling the blue one!'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 5}, page_content='Im not interested in this bag. Im interested in the blue one!'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 6}, page_content='Here is $129'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 7}, page_content=''),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 8}, page_content='Online is at least $100'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 9}, page_content='How much do you want?'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 10}, page_content='Goodmorning! $50 is too low.'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 11}, page_content='Hi! Im interested in your bag. Im offering $50. Let me know if you are interested. Thanks!')]
'''

Processing JSON Lines (JSONL): Seamlessly handle files where each line represents a separate JSON object by setting json_lines=True.

Example - JSON (Processing JSON Lines)

loader = JSONLoader(
file_path='/path/to/facebook_chat_messages.jsonl',
jq_schema='.content',
text_content=False,
json_lines=True
)

data = loader.load()
pprint(data)

Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat_messages.jsonl', 'seq_num': 1}, page_content='Bye!'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat_messages.jsonl', 'seq_num': 2}, page_content='Oh no worries! Bye'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat_messages.jsonl', 'seq_num': 3}, page_content='No Im sorry it was my mistake, the blue one is not for sale')]
'''

Example - JSON (Use jq_schema='.' and content_key for simpler extraction)

loader = JSONLoader(
file_path='/path/to/facebook_chat_messages.jsonl',
jq_schema=".", # Select the entire JSON object on each line
content_key="sender_name", # Use "sender_name" as content
json_lines=True,
)

data = loader.load()
pprint(data)

Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat_messages.jsonl', 'seq_num': 1}, page_content='User 2'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat_messages.jsonl', 'seq_num': 2}, page_content='User 1'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat_messages.jsonl', 'seq_num': 3}, page_content='User 2')]
'''

Adding Metadata from JSON: Use custom functions to extract additional metadata, enhancing data context and traceability.

Example - JSON (Adding Metadata from JSON)

def metadata_func(record: dict, metadata: dict) -> dict:
metadata["sender_name"] = record.get("sender_name")
metadata["timestamp_ms"] = record.get("timestamp_ms")
return metadata

loader = JSONLoader(
file_path='/path/to/facebook_chat.json',
jq_schema='.messages[]',
content_key="content",
metadata_func=metadata_func # Add metadata from JSON
)

data = loader.load()
pprint(data)

Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 1, 'sender_name': 'User 2', 'timestamp_ms': 1675597571851}, page_content='Bye!'),
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 2, 'sender_name': 'User 1', 'timestamp_ms': 1675597435669}, page_content='Oh no worries! Bye'),
...
Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/facebook_chat.json', 'seq_num': 11, 'sender_name': 'User 1', 'timestamp_ms': 1675549022673}, page_content='Hi! Im interested in your bag. Im offering $50. Let me know if you are interested. Thanks!')]
'''

5. Markdown: Simplifying Documentation Integration

Markdown files are commonly used for technical documentation. LangChain’s UnstructuredMarkdownLoader efficiently processes Markdown content for AI workflows.

The example.md file can be accessed here.

Example - Markdown (Loading Markdown Files)

markdown_path = "/path/to/example.md"
loader = UnstructuredMarkdownLoader(markdown_path)

data = loader.load()
print(data[0].page_content)

Example of Output:

'''
Sample Markdown Document

Introduction

Welcome to this sample Markdown document. Markdown is a lightweight markup language used for formatting text. It's widely used for documentation, readme files, and more.
'''

6. PDF: Processing Structured Documents

PDFs are a staple for sharing structured information, and LangChain offers a suite of tools for processing PDF files:

The layout-parser-paper.pdf file can be accessed here.

PyPDFLoader: Retains the original PDF structure, including page breaks. Ideal for simple PDFs.

Example - PDF (Using PyPDFLoader)

loader = PyPDFLoader("/path/to/layout-parser-paper.pdf")
pages = loader.load_and_split()

print(pages[0].page_content)

Example of Output:

'''
LayoutParser: A Uniﬁed Toolkit for Deep
Learning Based Document Image Analysis
Zejiang Shen1 ( ), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain
'''

Combine PyPDFLoader with FAISS (Facebook AI Similarity Search) for advanced semantic search within PDF content.

Example - PDF (Advanced Retrieval with FAISS)

faiss_index = FAISS.from_documents(pages, OpenAIEmbeddings(api_key=OPENAI_API_KEY))
docs = faiss_index.similarity_search("How will the community be engaged?", k=2)

for doc in docs:
print(f"Page {doc.metadata['page']}: {doc.page_content[:300]}")

Example of Output:

'''
Page 9: 10 Z. Shen et al.
Fig. 4: Illustration of (a) the original historical Japanese document with layout
detection results and (b) a recreated version of the document image that achieves
much better character recognition recall. The reorganization algorithm rearranges
the tokens based on the their detect ...
'''

Extract images embedded within the PDF (note: images are not saved to disk but embedded within the Document object).

Example - PDF (Extract Images from PDF)

loader = PyPDFLoader("https://arxiv.org/pdf/2103.15348.pdf", extract_images=True) # Extract images (requires poppler)
pages = loader.load()
pages[4].page_content

Example of Output:

'''
LayoutParser: A Uniﬁed Toolkit for DL-Based DIA 5\nTable 1: Current layout detection models in the LayoutParser model zoo\nDataset Base Model1 Large ModelNotes\nPubLayNet [38] F / M M Layouts of modern scientiﬁc documents\nPRImA [3] ...
'''

PyMuPDFLoader: A faster alternative for straightforward PDFs.

Example - PDF (PyMuPDFLoader)

loader = PyMuPDFLoader("/path/to/layout-parser-paper.pdf")
data = loader.load()
print(data[0]) # Access and print the first document

Example of Output:

'''
page_content='LayoutParser: A Uniﬁed Toolkit for Deep
Learning Based Document Image Analysis
Zejiang Shen1 (), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain
Lee4, Jacob Carlson3, and Weining Li5
1 Allen Institute for AI
shannons@allenai.org
'''

UnstructuredPDFLoader: Excels at handling complex layouts, extracting text content reliably.

Example - PDF (UnstructuredPDFLoader)

loader = UnstructuredPDFLoader("/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/layout-parser-paper.pdf")
data = loader.load()
data

Output:

'''
page_content='1 2 0 2' metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/layout-parser-paper.pdf', 'coordinates': {'points': ((16.34, 213.36), (16.34, 253.36), (36.34, 253.36), (36.34, 213.36)), 'system': 'PixelSpace', 'layout_width': 612, 'layout_height': 792}, 'file_directory': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web', 'filename': 'layout-parser-paper.pdf', 'languages': ['eng'], 'last_modified': '2024-09-13T11:42:35', 'page_number': 1, 'filetype': 'application/pdf', 'category': 'UncategorizedText', 'element_id': 'd3ce55f220dfb75891b4394a18bcb973'}
'''

Example - PDF (Load in "elements" mode (more granular control))

Load in "elements" mode (more granular control)

loader = UnstructuredPDFLoader("/path/to/layout-parser-paper.pdf", mode="elements") # Split into individual elements
data = loader.load()

print(data[0])

Example of Output:

Example - PDF (Load PDF from a URL)

Load in "elements" mode (more granular control)

loader = UnstructuredPDFLoader("/path/to/layout-parser-paper.pdf", mode="elements") # Split into individual elements
data = loader.load()

print(data[0])

Example of Output:

'''
[Document(metadata={'source': '/tmp/tmpmtno2k0o/tmp.pdf'}, page_content='3 2 0 2\n\nb e F 7\n\n]\n\nG A . h t a m\n\n[\n\n1 v 3 0 8 3 0 . 2 0 3 2 : v i X r a\n\nA WEAK (k, k)-LEFSCHETZ THEOREM FOR PROJECTIVE TORIC ORBIFOLDS\n\nWilliam D. Montoya\n\nInstituto de Matem´atica, Estat´ıstica e Computa¸c˜ao Cient´ıﬁca, Universidade Estadual de Campinas (UNICAMP),\n\nRua S´ergio Buarque de Holanda 651, 13083-859, Campinas, SP, Brazil\n\nFebruary 9, 2023\n\nAbstract\n\nFirstly we show a generalization of the (1, 1)-Lefschetz theorem for projective toric orbifolds and secondly we prove that on 2k-dimensional quasi-smooth hyper- surfaces coming from quasi-smooth intersection surfaces, under the Cayley trick, every rational (k, k)-cohomology class is algebraic, i.e., the Hodge conjecture holds on them.\n\n1\n\nIntroduction\n\nIn [3] we proved that, under suitable conditions, on a very general codimension s quasi- smooth intersection subvariety X in a projective toric orbifold Pd Σ with d + s = 2(k + 1) the Hodge conjecture holds, that is, every (p, p)-cohomology class, under the Poincar´e duality is a rational linear combination of fundamental classes of algebraic subvarieties of X. The proof of the above-mentioned result relies, for p ≠ d + 1 − s, on a Lefschetz\n\nDate: February 9, 2023 2020 Mathematics Subject Classiﬁcation: 14C30, 14M10, 14J70, 14M25 Keywords: (1,1)- Lefschetz theorem, Hodge conjecture, toric varieties, complete intersection Email: wmontoya@ime.unicamp.br\n\n1\n\ntheorem ([7]) and the Hard Lefschetz theorem for projective orbifolds ([11]). When p = d + 1 − s the proof relies on the Cayley trick, a trick which associates to X a quasi-smooth hypersurface Y in a projective vector bundle, and the Cayley Proposition (4.3) which gives an isomorphism of some primitive cohomologies (4.2) of X and Y . ...
'''

PyPDFium2Loader: A wrapper around the PDFium library, suitable for advanced PDF manipulation.

Example - PDF (PyPDFium2Loader)

loader = PyPDFium2Loader("/path/to/layout-parser-paper.pdf")
data = loader.load()
print(data)

Example of Output:

PDFMinerLoader: Extracts both text and rich metadata, providing a comprehensive representation.

Example - PDF (PDFMinerLoader)

loader = PDFMinerLoader("/path/to/layout-parser-paper.pdf")
data = loader.load()
print(data)

Example of Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/layout-parser-paper.pdf'}, page_content='1\n2\n0\n2\n\nn\nu\nJ\n\n1\n2\n\n]\n\nV\nC\n.\ns\nc\n[\n\n2\nv\n8\n4\n3\n5\n1\n.\n3\n0\n1\n2\n:\nv\ni\nX\nr\na\n\nLayoutParser: A Uniﬁed Toolkit for Deep\nLearning Based Document Image Analysis\n\nZejiang Shen1 ((cid:0)), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\nLee4, Jacob Carlson3, and Weining Li5\n\n1 Allen Institute for AI\nshannons@allenai.org\n2 Brown University\nruochen zhang@brown.edu\n3 Harvard University\n{melissadell,jacob @fas.harvard.edu">carlson}@fas.harvard.edu\n4 University of Washington\nbcgl@cs.washington.edu\n5 University of Waterloo\nw422li@uwaterloo.ca\n\nAbstract. Recent advances in document image analysis (DIA) have been\nprimarily driven by the application of neural networks. Ideally, research\noutcomes could be easily deployed in production and extended for further\ninvestigation. However, various factors like loosely organized codebases\nand sophisticated model conﬁgurations complicate the easy reuse of im-\nportant innovations by a wide audience. ...
'''

PyPDFDirectoryLoader: Loads all PDFs from a specified directory.

Example - PDF (PDFMinerPDFasHTMLLoader)

loader = PyPDFDirectoryLoader("/path/to/Langchain_web/") # Replace with your directory
docs = loader.load()
print(docs)

Example of Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/layout-parser-paper.pdf', 'page': 0}, page_content='LayoutParser: A Uniﬁed Toolkit for Deep\nLearning Based Document Image Analysis\nZejiang Shen1 (\x00 ), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\nLee4, Jacob Carlson3, and Weining Li5\n1 Allen Institute for AI\nshannons@allenai.org\n2 Brown University\nruochen zhang@brown.edu\n3 Harvard University\n{melissadell,jacob @fas.harvard.edu">carlson}@fas.harvard.edu\n4 University of Washington\nbcgl@cs.washington.edu\n5 University of Waterloo\nw422li@uwaterloo.ca\nAbstract. Recent advances in document image analysis (DIA) have been\nprimarily driven by the application of neural networks. Ideally, research\noutcomes could be easily deployed in production and extended for further\ninvestigation. However, various factors like loosely organized codebases\nand sophisticated model conﬁgurations complicate the easy reuse of im-\nportant innovations by a wide audience. Though there have been on-going\neﬀorts to improve reusability and simplify deep learning (DL) model\ndevelopment in disciplines like natural language processing and computer\nvision, none of them are optimized for challenges in the domain of DIA.\nThis represents a major gap in the existing toolkit, as DIA is central to\nacademic research across a wide range of disciplines in the social sciences\nand humanities. This paper introduces LayoutParser, an open-source\nlibrary for streamlining the usage of DL in DIA research and applica-\ntions. The core LayoutParser library comes with a set of simple and\nintuitive interfaces for applying and customizing DL models for layout de-\ntection, character recognition, and many other document processing tasks.\nTo promote extensibility, LayoutParser also incorporates a community\nplatform for sharing both pre-trained models and full document digiti-\nzation pipelines. We demonstrate that LayoutParser is helpful for both\nlightweight and large-scale digitization pipelines in real-word use cases. ...
'''

PDFPlumberLoader: Focuses on extracting tabular data from PDFs.

Example - PDF (PDFPlumberLoader)

loader = PDFPlumberLoader("/path/to/layout-parser-paper.pdf")
data = loader.load()
print(data)

Example of Output:

'''
[Document(metadata={'source': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/layout-parser-paper.pdf', 'file_path': '/content/drive/MyDrive/0. Colab Notebooks/4. LLM/Langchain_web/layout-parser-paper.pdf', 'page': 0, 'total_pages': 16, 'Author': '', 'CreationDate': 'D:20210622012710Z', 'Creator': 'LaTeX with hyperref', 'Keywords': '', 'ModDate': 'D:20210622012710Z', 'PTEX.Fullbanner': 'This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) kpathsea version 6.3.2', 'Producer': 'pdfTeX-1.40.21', 'Subject': '', 'Title': '', 'Trapped': 'False'}, page_content='LayoutParser: A Unified Toolkit for Deep\nLearning Based Document Image Analysis\nZejiang Shen1 ((cid:0)), Ruochen Zhang2, Melissa Dell3, Benjamin Charles Germain\nLee4, Jacob Carlson3, and Weining Li5\n1 Allen Institute for AI\nshannons@allenai.org\n2 Brown University\nruochen zhang@brown.edu\n3 Harvard University\n{melissadell,jacob @fas.harvard.edu">carlson}@fas.harvard.edu ...
'''

LangChain’s document loaders provide robust and versatile solutions for transforming raw data into AI-ready formats. By supporting a wide range of file types and offering customization options, they are indispensable for any workflow involving Retrieval Augmented Generation (RAG).

In the next episode, we’ll explore advanced retrieval techniques, from embedding models to vector stores. Subscribe to the Data Mastery Series and continue mastering LangChain, one step at a time. 🚀

Data Science Explore the world of data science with Donato_Story

Dashboard Discover the power of data visualization with Donato_Story

Donato_Journey Join me on my journey (Thai version)

Course_Review Discover the training courses with Donato_Story (Thai version)