---

Unpacking Text Splitter with LangChain

Data Mastery Series — Episode 35: LangChain Website (Part 10)

Connect with me and follow our journey: Linkedin, Facebook

---

Welcome to Episode 35 of the Data Mastery Series, where we continue our exploration of LangChain, a transformative tool for integrating AI into real-world applications. If you’ve been following along, you’ve already seen how LangChain elevates data workflows. Here’s a recap of the journey so far:

• Part 1: LangChain Model I/O Basics
• Part 2–3: Prompt Templates and Few-Shot Prompts
• Part 4–6: Deep Dive into Chat Models (Part 1, Part 2, Part 3)
• Part 7: LLM Fundamentals
• Part 8: Output Parsers
• Part 9: Document Loaders

Note:This post reflects my ongoing learning journey with LangChain, drawing insights from the official documentation and related resources. The content is based on resources found link. I hope you find it valuable!

Today, we’ll unpack various text splitting techniques offered by LangChain. These techniques enable us to break down large text documents into smaller, digestible chunks, which are essential for LLMs to process information effectively.

Types of Text Splitters in LangChain

LangChain provides a diverse set of text splitters, each designed to handle different text structures and formats. Let’s explore some of the most useful options:

1. Splitting by HTML Headers

When working with structured web content, HTMLHeaderTextSplitter allows you to break text into sections based on HTML headers (<h1>, <h2>, <h3>, etc.). This approach ensures that related information stays together, much like chapters in a book.

An easy-to-read version of html_string

Example - Splitting by HTML Headers 1

html_string = """

headers_to_split_on = [
("h1", "Header 1"),
("h2", "Header 2"),
("h3", "Header 3"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
html_header_splits = html_splitter.split_text(html_string)
html_header_splits

Output

'''
[Document(metadata={}, page_content='Foo'),
Document(metadata={'Header 1': 'Foo'}, page_content='Some intro text about Foo. \nBar main section Bar subsection 1 Bar subsection 2'),
Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section'}, page_content='Some intro text about Bar.'),
Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 1'}, page_content='Some text about the first subtopic of Bar.'),
Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 2'}, page_content='Some text about the second subtopic of Bar.'),
Document(metadata={'Header 1': 'Foo'}, page_content='Baz'),
Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='Some text about Baz'),
Document(metadata={'Header 1': 'Foo'}, page_content='Some concluding text about Foo')]
'''

Example - Splitting by HTML Headers 2

url = "https://plato.stanford.edu/entries/goedel/"

headers_to_split_on = [
("h1", "Header 1"),
("h2", "Header 2"),
("h3", "Header 3"),
("h4", "Header 4"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)

for local file use html_splitter.split_text_from_file(<path_to_file>)

html_header_splits = html_splitter.split_text_from_url(url)

chunk_size = 500
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

Split

splits = text_splitter.split_documents(html_header_splits)
splits[80:85]

Output

'''
[Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='We see that Gödel first tried to reduce the consistency problem for analysis to that of arithmetic. This seemed to require a truth definition for arithmetic, which in turn led to paradoxes, such as the Liar paradox (“This sentence is false”) and Berry’s paradox (“The least number not defined by an expression consisting of just fourteen English words”). Gödel then noticed that such paradoxes would not necessarily arise if truth were replaced by provability. But this means that arithmetic truth'),
Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='means that arithmetic truth and arithmetic provability are not co-extensive — whence the First Incompleteness Theorem.'),
Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='This account of Gödel’s discovery was told to Hao Wang very much after the fact; but in Gödel’s contemporary correspondence with Bernays and Zermelo, essentially the same description of his path to the theorems is given. (See Gödel 2003a and Gödel 2003b respectively.) From those accounts we see that the undefinability of truth in arithmetic, a result credited to Tarski, was likely obtained in some form by Gödel by 1931. But he neither publicized nor published the result; the biases logicians'),
Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.1 The First Incompleteness Theorem'}, page_content='result; the biases logicians had expressed at the time concerning the notion of truth, biases which came vehemently to the fore when Tarski announced his results on the undefinability of truth in formal systems 1935, may have served as a deterrent to Gödel’s publication of that theorem.'),
Document(metadata={'Header 1': 'Kurt Gödel', 'Header 2': '2. Gödel’s Mathematical Work', 'Header 3': '2.2 The Incompleteness Theorems', 'Header 4': '2.2.2 The proof of the First Incompleteness Theorem'}, page_content='We now describe the proof of the two theorems, formulating Gödel’s results in Peano arithmetic. Gödel himself used a system related to that defined in Principia Mathematica, but containing Peano arithmetic. In our presentation of the First and Second Incompleteness Theorems we refer to Peano arithmetic as P, following Gödel’s notation.')]
'''

Example - Splitting by HTML Headers 3

url = "https://www.cnn.com/2023/09/25/weather/el-nino-winter-us-climate/index.html"

headers_to_split_on = [
("h1", "Header 1"),
("h2", "Header 2"),
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
html_header_splits = html_splitter.split_text_from_url(url)
print(html_header_splits[1].page_content[:500])

Output

'''
No two El Niño winters are the same, but many have temperature and precipitation trends in common.
Average conditions during an El Niño winter across the continental US.
One of the major reasons is the position of the jet stream, which often shifts south during an El Niño winter. This shift typically brings wetter and cooler weather to the South while the North becomes drier and warmer, according to NOAA.
Because the jet stream is essentially a river of air that storms flow through, they c
'''

2. Splitting by HTML Sections

If you need larger logical groupings, HTMLSectionSplitter splits content into bigger sections while preserving context.

Example - Splitting by HTML Sections 1

Same html_string as above

headers_to_split_on = [("h1", "Header 1"), ("h2", "Header 2")]

html_splitter = HTMLSectionSplitter(headers_to_split_on=headers_to_split_on)
html_section_splits = html_splitter.split_text(html_string)
html_section_splits

Output

'''
[Document(metadata={'Header 1': 'Foo'}, page_content='Foo \n Some intro text about Foo.'),
Document(metadata={'Header 2': 'Bar main section'}, page_content='Bar main section \n Some intro text about Bar. \n Bar subsection 1 \n Some text about the first subtopic of Bar. \n Bar subsection 2 \n Some text about the second subtopic of Bar.'),
Document(metadata={'Header 2': 'Baz'}, page_content='Baz \n Some text about Baz \n \n \n Some concluding text about Foo')]
'''

Example - Splitting by HTML Sections 2

Same html_string as above

headers_to_split_on = [
("h1", "Header 1"),
("h2", "Header 2"),
("h3", "Header 3"),
("h4", "Header 4"),
]

html_splitter = HTMLSectionSplitter(headers_to_split_on=headers_to_split_on)
html_section_splits = html_splitter.split_text(html_string)

chunk_size = 500
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)

แยกข้อความตามส่วนและขนาดที่กำหนด

splits = text_splitter.split_documents(html_section_splits)
splits

Output

'''
[Document(metadata={'Header 1': 'Foo'}, page_content='Foo \n Some intro text about Foo.'),
Document(metadata={'Header 2': 'Bar main section'}, page_content='Bar main section \n Some intro text about Bar.'),
Document(metadata={'Header 3': 'Bar subsection 1'}, page_content='Bar subsection 1 \n Some text about the first subtopic of Bar.'),
Document(metadata={'Header 3': 'Bar subsection 2'}, page_content='Bar subsection 2 \n Some text about the second subtopic of Bar.'),
Document(metadata={'Header 2': 'Baz'}, page_content='Baz \n Some text about Baz \n \n \n Some concluding text about Foo')]
'''

3. Splitting by Characters

For unstructured text, CharacterTextSplitter divides text based on character counts. You can customize chunk sizes and overlaps to ensure that chunks retain necessary context.

The state_of_the_union.txt file can be found here.

Example - Splitting by Character 1

with open("/path/to/state_of_the_union.txt") as f:
state_of_the_union = f.read()

text_splitter = CharacterTextSplitter(
separator="\n\n", # แบ่งตามย่อหน้า
chunk_size=1000, # ขนาดชิ้นข้อความ
chunk_overlap=200, # มีการซ้อนทับระหว่างชิ้น
length_function=len, # ใช้ฟังก์ชัน len ในการวัดขนาด
is_separator_regex=False, # ตัวแบ่งไม่ใช่ regex
)

texts = text_splitter.create_documents([state_of_the_union])
texts[0]

Output

'''
Document(metadata={}, page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.')
'''

Example - Add Metadata

metadatas = [{"document": 1}, {"document": 2}]
documents = text_splitter.create_documents(
[state_of_the_union, state_of_the_union], metadatas=metadatas
)
documents[0]

Output

'''
Document(metadata={'document': 1}, page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \n\nLast year COVID-19 kept us apart. This year we are finally together again. \n\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n\nWith a duty to one another to the American people to the Constitution. \n\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \n\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n\nHe met the Ukrainian people. \n\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.')
'''

Example - Separate Text

text_splitter.split_text(state_of_the_union)[0]

Output

'''
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.
Last year COVID-19 kept us apart. This year we are finally together again.
Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.
With a duty to one another to the American people to the Constitution.
And with an unwavering resolve that freedom will always triumph over tyranny.
Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated.
He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined.
He met the Ukrainian people.
From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.
'''

4. Splitting Code

LangChain also supports splitting code into logical chunks using CodeTextSplitter, which is tailored for specific programming languages like Python, JavaScript, and TypeScript.

• Python:

Example - Python

PYTHON_CODE = """
def hello_world():
print("Hello, World!")

Call the function

hello_world()
"""

python_splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.PYTHON, chunk_size=50, chunk_overlap=0
)

python_docs = python_splitter.create_documents([PYTHON_CODE])
python_docs

Output

'''
[Document(metadata={}, page_content='def hello_world():\n print("Hello, World!")'),
Document(metadata={}, page_content='# Call the function\nhello_world()')]
'''

• JavaScript (JS):

Example - JavaScript (JS)

JS_CODE = """
function helloWorld() {
console.log("Hello, World!");
}

// Call the function
helloWorld();
"""

js_splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.JS, chunk_size=60, chunk_overlap=0
)
js_docs = js_splitter.create_documents([JS_CODE])
js_docs

Output

'''
[Document(metadata={}, page_content='function helloWorld() {\n console.log("Hello, World!");\n}'),
Document(metadata={}, page_content='// Call the function\nhelloWorld();')]
'''

• TypeScript (TS):

Example - TypeScript (TS)

TS_CODE = """
function helloWorld(): void {
console.log("Hello, World!");
}

// Call the function
helloWorld();
"""

ts_splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.TS, chunk_size=60, chunk_overlap=0
)
ts_docs = ts_splitter.create_documents([TS_CODE])
ts_docs

Output

'''
[Document(metadata={}, page_content='function helloWorld(): void {'),
Document(metadata={}, page_content='console.log("Hello, World!");\n}'),
Document(metadata={}, page_content='// Call the function\nhelloWorld();')]
'''

• Markdown:

An easy-to-read version of markdown_text

Example - Markdown

markdown_text = """

🦜️🔗 LangChain

⚡ Building applications with LLMs through composability ⚡

Quick Install

# Hopefully this code block isn't split  
pip install langchain  
  
As an open-source project in a rapidly developing field, we are extremely open to contributions. """  
  
md_splitter = RecursiveCharacterTextSplitter.from_language(  
    language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0  
)  
  
md_docs = md_splitter.create_documents([markdown_text])  
md_docs  
  
# Output  
'''  
[Document(metadata={}, page_content='# 🦜️🔗 LangChain'),  
 Document(metadata={}, page_content='⚡ Building applications with LLMs through composability ⚡'),  
 Document(metadata={}, page_content='## Quick Install\n\n```bash'),  
 Document(metadata={}, page_content="# Hopefully this code block isn't split"),  
 Document(metadata={}, page_content='pip install langchain'),  
 Document(metadata={}, page_content='As an open-source project in a rapidly developing field, we'),  
 Document(metadata={}, page_content='are extremely open to contributions.')]  
'''

* **Latex:**

![](https://cdn-images-1.medium.com/max/800/1*05ZXsEac39m3n1himf9oLw.png)

An easy-to-read version of **latex\_text**

# Example - Latex  
  
latex_text = """  
\documentclass{article}  
  
\begin{document}  
  
\maketitle  
  
\section{Introduction}  
Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.  
  
\subsection{History of LLMs}  
The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.  
  
\subsection{Applications of LLMs}  
LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.  
  
\end{document}  
"""  
  
latex_splitter = RecursiveCharacterTextSplitter.from_language(  
    language=Language.MARKDOWN, chunk_size=250, chunk_overlap=0  
)  
latex_docs = latex_splitter.create_documents([latex_text])  
latex_docs  
  
# Output  
'''  
[Document(metadata={}, page_content='\\documentclass{article}\n\n\x08egin{document}\n\n\\maketitle'),  
 Document(metadata={}, page_content='\\section{Introduction}'),  
 Document(metadata={}, page_content='Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing'),  
 Document(metadata={}, page_content='tasks, including language translation, text generation, and sentiment analysis.'),  
 Document(metadata={}, page_content='\\subsection{History of LLMs}'),  
 Document(metadata={}, page_content='The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made'),  
 Document(metadata={}, page_content='it possible to train LLMs on massive datasets, leading to significant improvements in performance.'),  
 Document(metadata={}, page_content='\\subsection{Applications of LLMs}\nLLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.'),  
 Document(metadata={}, page_content='\\end{document}')]  
'''

* **HTML:**

![](https://cdn-images-1.medium.com/max/800/1*Xzxg7YXSSzysbLXfqpKjMw.png)

A plain-text version of **html\_text**

# Example - HTML  
  
html_text = """  
<!DOCTYPE html>  
<html>  
    <head>  
        <title>🦜️🔗 LangChain</title>  
        <style>  
            body {  
                font-family: Arial, sans-serif;  
            }  
            h1 {  
                color: darkblue;  
            }  
        </style>  
    </head>  
    <body>  
        <div>  
            <h1>🦜️🔗 LangChain</h1>  
            <p>⚡ Building applications with LLMs through composability ⚡</p>  
        </div>  
        <div>  
            As an open-source project in a rapidly developing field, we are extremely open to contributions.  
        </div>  
    </body>  
</html>  
"""  
  
html_splitter = RecursiveCharacterTextSplitter.from_language(  
    language=Language.HTML, chunk_size=250, chunk_overlap=0  
)  
html_docs = html_splitter.create_documents([html_text])  
html_docs  
  
# Output  
'''  
[Document(metadata={}, page_content='<!DOCTYPE html>\n<html>'),  
 Document(metadata={}, page_content='<head>\n        <title>🦜️🔗 LangChain</title>\n        <style>\n            body {\n                font-family: Arial, sans-serif;\n            }\n            h1 {\n                color: darkblue;\n            }\n        </style>\n    </head>'),  
 Document(metadata={}, page_content='<body>\n        <div>\n            <h1>🦜️🔗 LangChain</h1>\n            <p>⚡ Building applications with LLMs through composability ⚡</p>\n        </div>'),  
 Document(metadata={}, page_content='<div>\n            As an open-source project in a rapidly developing field, we are extremely open to contributions.\n        </div>\n    </body>\n</html>')]  
'''

* **Solidity:**

# Example - Solidity  
  
SOL_CODE = """  
pragma solidity ^0.8.20;  
contract HelloWorld {  
   function add(uint a, uint b) pure public returns(uint) {  
       return a + b;  
   }  
}  
"""  
  
sol_splitter = RecursiveCharacterTextSplitter.from_language(  
    language=Language.SOL, chunk_size=128, chunk_overlap=0  
)  
sol_docs = sol_splitter.create_documents([SOL_CODE])  
sol_docs  
  
# Output  
'''  
[Document(metadata={}, page_content='pragma solidity ^0.8.20;'),  
 Document(metadata={}, page_content='contract HelloWorld {\n   function add(uint a, uint b) pure public returns(uint) {\n       return a + b;\n   }\n}')]  
'''

* **C:**

# Example - C  
  
C_CODE = """  
using System;  
class Program  
{  
    static void Main()  
    {  
        int age = 30; // Change the age value as needed  
  
        // Categorize the age without any console output  
        if (age < 18)  
        {  
            // Age is under 18  
        }  
        else if (age >= 18 && age < 65)  
        {  
            // Age is an adult  
        }  
        else  
        {  
            // Age is a senior citizen  
        }  
    }  
}  
"""  
c_splitter = RecursiveCharacterTextSplitter.from_language(  
    language=Language.CSHARP, chunk_size=128, chunk_overlap=0  
)  
c_docs = c_splitter.create_documents([C_CODE])  
c_docs  
  
# Output  
'''  
[Document(metadata={}, page_content='using System;'),  
 Document(metadata={}, page_content='class Program\n{\n    static void Main()\n    {\n        int age = 30; // Change the age value as needed'),  
 Document(metadata={}, page_content='// Categorize the age without any console output\n        if (age < 18)\n        {\n            // Age is under 18'),  
 Document(metadata={}, page_content='}\n        else if (age >= 18 && age < 65)\n        {\n            // Age is an adult\n        }\n        else\n        {'),  
 Document(metadata={}, page_content='// Age is a senior citizen\n        }\n    }\n}')]  
'''

* **Haskell:**

# Example - Haskell  
  
HASKELL_CODE = """  
main :: IO ()  
main = do  
    putStrLn "Hello, World!"  
-- Some sample functions  
add :: Int -> Int -> Int  
add x y = x + y  
"""  
haskell_splitter = RecursiveCharacterTextSplitter.from_language(  
    language=Language.HASKELL, chunk_size=50, chunk_overlap=0  
)  
haskell_docs = haskell_splitter.create_documents([HASKELL_CODE])  
haskell_docs  
  
# Output  
'''  
[Document(metadata={}, page_content='main :: IO ()'),  
 Document(metadata={}, page_content='main = do\n    putStrLn "Hello, World!"\n-- Some'),  
 Document(metadata={}, page_content='sample functions\nadd :: Int -> Int -> Int\nadd x y'),  
 Document(metadata={}, page_content='= x + y')]  
'''

#### 5\. Splitting by Markdown Headers (MarkdownHeaderTextSplitter)

Similar to HTML, Markdown documents can be split by header levels using

![](https://cdn-images-1.medium.com/max/800/1*_h7dU69o9EcvXXY2Wnblrw.png)

An easy-to-read version of **markdown\_document**

# Example - Splitting by Markdown Headers 1  
  
markdown_document = "# Foo\n\n    ## Bar\n\nHi this is Jim\n\nHi this is Joe\n\n ### Boo \n\n Hi this is Lance \n\n ## Baz\n\n Hi this is Molly"  
  
headers_to_split_on = [  
    ("#", "Header 1"),  
    ("##", "Header 2"),  
    ("###", "Header 3"),  
]  
  
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)  
md_header_splits = markdown_splitter.split_text(markdown_document)  
md_header_splits  
  
# Output  
'''  
[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='Hi this is Jim  \nHi this is Joe'),  
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='Hi this is Lance'),  
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='Hi this is Molly')]  
'''

# Example - Splitting by Markdown Headers 2 (strip_headers=False)  
  
# Same html_string as above  
  
markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on, strip_headers=False)  
md_header_splits = markdown_splitter.split_text(markdown_document)  
md_header_splits  
  
# Output  
"""  
[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='# Foo  \n## Bar  \nHi this is Jim  \nHi this is Joe'),  
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='### Boo  \nHi this is Lance'),  
 Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='## Baz  \nHi this is Molly')]  
"""

![](https://cdn-images-1.medium.com/max/800/1*KcYwIdv_ggHb5fMJP533Lg.png)

An easy-to-read version of **markdown\_document**

# Example - Splitting by Markdown Headers 3  
  
markdown_document = "# Intro \n\n    ## History \n\n Markdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9] \n\n Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files. \n\n ## Rise and divergence \n\n As Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for \n\n additional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks. \n\n #### Standardization \n\n From 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort. \n\n ## Implementations \n\n Implementations of Markdown are available for over a dozen programming languages."  
  
headers_to_split_on = [  
    ("#", "Header 1"),  
    ("##", "Header 2"),  
]  
  
markdown_splitter = MarkdownHeaderTextSplitter(  
    headers_to_split_on=headers_to_split_on, strip_headers=False  
)  
md_header_splits = markdown_splitter.split_text(markdown_document)  
  
chunk_size = 250  
chunk_overlap = 30  
text_splitter = RecursiveCharacterTextSplitter(  
    chunk_size=chunk_size, chunk_overlap=chunk_overlap  
)  
  
splits = text_splitter.split_documents(md_header_splits)  
splits  
  
# Output  
"""  
[Document(metadata={'Header 1': 'Intro', 'Header 2': 'History'}, page_content='# Intro  \n## History  \nMarkdown[9] is a lightweight markup language for creating formatted text using a plain-text editor. John Gruber created Markdown in 2004 as a markup language that is appealing to human readers in its source code form.[9]'),  
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'History'}, page_content='Markdown is widely used in blogging, instant messaging, online forums, collaborative software, documentation pages, and readme files.'),  
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'Rise and divergence'}, page_content='## Rise and divergence  \nAs Markdown popularity grew rapidly, many Markdown implementations appeared, driven mostly by the need for  \nadditional features such as tables, footnotes, definition lists,[note 1] and Markdown inside HTML blocks.'),  
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'Rise and divergence'}, page_content='#### Standardization  \nFrom 2012, a group of people, including Jeff Atwood and John MacFarlane, launched what Atwood characterised as a standardisation effort.'),  
 Document(metadata={'Header 1': 'Intro', 'Header 2': 'Implementations'}, page_content='## Implementations  \nImplementations of Markdown are available for over a dozen programming languages.')]  
"""

#### 6\. **Recursively Splitting JSON**

For complex, nested JSON data, **RecursiveJsonSplitter** breaks it down into smaller units while maintaining structural integrity.

# Example - Splitting JSON Recursively  
  
json_data = requests.get("https://api.smith.langchain.com/openapi.json").json()  
  
splitter = RecursiveJsonSplitter(max_chunk_size=250)  
json_chunks = splitter.split_json(json_data=json_data)  
docs = splitter.create_documents(texts=[json_data])  
texts = splitter.split_text(json_data=json_data)  
  
print(texts[0])  
print(texts[1])  
  
# Output  
"""  
{"openapi": "3.1.0", "info": {"title": "LangSmith", "version": "0.1.0"}}  
{"paths": {"/api/v1/sessions/{session_id}": {"get": {"tags": ["tracer-sessions"], "summary": "Read Tracer Session", "description": "Get a specific session.", "operationId": "read_tracer_session_api_v1_sessions__session_id__get"}}}  
"""

#### 7\. **Recursive Character Splitting**

The **RecursiveCharacterTextSplitter** offers more fine-grained control over splitting, allowing custom separators for handling various text formats and languages.

The **state\_of\_the\_union.txt** file can be found [here](https://github.com/hwchase17/chat-your-data/blob/master/state%5Fof%5Fthe%5Funion.txt).

# Example - Recursive Character Splitting  
  
with open("/path/to/state_of_the_union.txt") as f:  
    state_of_the_union = f.read()  
  
text_splitter = RecursiveCharacterTextSplitter(  
    chunk_size=100,       
    chunk_overlap=20,     
    length_function=len,  
    is_separator_regex=False,  
)  
  
texts = text_splitter.create_documents([state_of_the_union])  
print(texts[0])  
print(texts[1])  
  
# Output  
"""  
page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and'  
page_content='of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.'  
"""

# Example - Recursive Character Splitting (custome separators)  
  
text_splitter = RecursiveCharacterTextSplitter(  
    separators=[  
        "\n\n",  # ย่อหน้า  
        "\n",    # บรรทัดใหม่  
        " ",     # ช่องว่าง  
        ".",     # จุด  
        ",",     # เครื่องหมายจุลภาค  
        "\u200b",  # Zero-width space (ใช้ในภาษาไทย)  
        "\uff0c",  # Fullwidth comma (ใช้ในภาษาจีน)  
        "\u3001",  # Ideographic comma (ใช้ในภาษาจีนและญี่ปุ่น)  
        "\uff0e",  # Fullwidth full stop (ใช้ในภาษาจีน)  
        "\u3002",  # Ideographic full stop (ใช้ในภาษาจีนและญี่ปุ่น)  
        "",  
    ],  
    chunk_size=100,       
    chunk_overlap=20,     
)

#### 8\. Semantic Chunking

For tasks requiring deeper context, **SemanticChunker** uses embeddings to group text by meaning. This advanced technique provides semantically coherent chunks, ideal for summarization or advanced retrieval systems.

# Example - Semantic Chunking 1  
  
with open("/path/to/state_of_the_union.txt") as f:  
    state_of_the_union = f.read()  
  
text_splitter = SemanticChunker(OpenAIEmbeddings(api_key=OPENAI_API_KEY))  
docs = text_splitter.create_documents([state_of_the_union])  
print(docs[0].page_content)  
print(len(docs[0].page_content))  
  
# Output  
"""  
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. And with an unwavering resolve that freedom will always triumph over tyranny. Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. He met the Ukrainian people. From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland. In this struggle as President Zelenskyy said in his speech to the European Parliament “Light will win over darkness.” The Ukrainian Ambassador to the United States is here tonight. Let each of us here tonight in this Chamber send an unmistakable signal to Ukraine and to the world. Please rise if you are able and show that, Yes, we the United States of America stand with the Ukrainian people. Throughout our history we’ve learned this lesson when dictators do not pay a price for their aggression they cause more chaos. They keep moving.  
1601  
"""

# Example - Semantic Chunking 2 (breakpoint_threshold_type = "percentile" or "Standard Deviation" pr "interquartile")  
  
with open("/path/to/state_of_the_union.txt") as f:  
    state_of_the_union = f.read()  
  
text_splitter = SemanticChunker(OpenAIEmbeddings(api_key=OPENAI_API_KEY), breakpoint_threshold_type="percentile")  
docs = text_splitter.create_documents([state_of_the_union])  
print(docs[0].page_content)  
print(len(docs[0].page_content))  
  
# Output  
"""  
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. And with an unwavering resolve that freedom will always triumph over tyranny. Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. He met the Ukrainian people. From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland. In this struggle as President Zelenskyy said in his speech to the European Parliament “Light will win over darkness.” The Ukrainian Ambassador to the United States is here tonight. Let each of us here tonight in this Chamber send an unmistakable signal to Ukraine and to the world. Please rise if you are able and show that, Yes, we the United States of America stand with the Ukrainian people. Throughout our history we’ve learned this lesson when dictators do not pay a price for their aggression they cause more chaos. They keep moving.  
1601  
"""  
  
text_splitter = SemanticChunker(OpenAIEmbeddings(api_key=OPENAI_API_KEY), breakpoint_threshold_type="standard_deviation")  
docs = text_splitter.create_documents([state_of_the_union])  
print(docs[0].page_content)  
print(len(docs[0].page_content))  
  
# Output  
"""  
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. And with an unwavering resolve that freedom will always triumph over tyranny. Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. He met the Ukrainian people. From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland. In this struggle as President Zelenskyy said in his speech to the European Parliament “Light will win over darkness.” The Ukrainian Ambassador to the United States is here tonight. Let each of us here tonight in this Chamber send an unmistakable signal to Ukraine and to the world. Please rise if you are able and show that, Yes, we the United States of America stand with the Ukrainian people. Throughout our history we’ve learned this lesson when dictators do not pay a price for their aggression they cause more chaos. They keep moving. And the costs and the threats to America and the world keep rising. That’s why the NATO Alliance was created to secure peace and stability in Europe after World War 2. The United States is a member along with 29 other nations. It matters. American diplomacy matters. American resolve matters. Putin’s latest attack on Ukraine was premeditated and unprovoked. He rejected repeated efforts at diplomacy. He thought the West and NATO wouldn’t respond. And he thought he could divide us at home. Putin was wrong. We were ready. Here is what we did. We prepared extensively and carefully. We spent months building a coalition of other freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin. I spent countless hours unifying our European allies. We shared with the world in advance what we knew Putin was planning and precisely how he would try to falsely justify his aggression. We countered Russia’s lies with truth. And now that he has acted the free world is holding him accountable. Along with twenty-seven members of the European Union including France, Germany, Italy, as well as countries like the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland. We are inflicting pain on Russia and supporting the people of Ukraine. Putin is now isolated from the world more than ever. Together with our allies –we are right now enforcing powerful economic sanctions. We are cutting off Russia’s largest banks from the international financial system. Preventing Russia’s central bank from defending the Russian Ruble making Putin’s $630 Billion “war fund” worthless. We are choking off Russia’s access to technology that will sap its economic strength and weaken its military for years to come. Tonight I say to the Russian oligarchs and corrupt leaders who have bilked billions of dollars off this violent regime no more. The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs. We are joining with our European allies to find and seize your yachts your luxury apartments your private jets. We are coming for your ill-begotten gains. And tonight I am announcing that we will join our allies in closing off American air space to all Russian flights – further isolating Russia – and adding an additional squeeze –on their economy. The Ruble has lost 30% of its value. The Russian stock market has lost 40% of its value and trading remains suspended. Russia’s economy is reeling and Putin alone is to blame. Together with our allies we are providing support to the Ukrainians in their fight for freedom. Military assistance. Economic assistance. Humanitarian assistance. We are giving more than $1 Billion in direct assistance to Ukraine. And we will continue to aid the Ukrainian people as they defend their country and to help ease their suffering. Let me be clear, our forces are not engaged and will not engage in conflict with Russian forces in Ukraine. Our forces are not going to Europe to fight in Ukraine, but to defend our NATO Allies – in the event that Putin decides to keep moving west. For that purpose we’ve mobilized American ground forces, air squadrons, and ship deployments to protect NATO countries including Poland, Romania, Latvia, Lithuania, and Estonia. As I have made crystal clear the United States and our Allies will defend every inch of territory of NATO countries with the full force of our collective power. And we remain clear-eyed. The Ukrainians are fighting back with pure courage. But the next few days weeks, months, will be hard on them. Putin has unleashed violence and chaos. But while he may make gains on the battlefield – he will pay a continuing high price over the long run. And a proud Ukrainian people, who have known 30 years  of independence, have repeatedly shown that they will not tolerate anyone who tries to take their country backwards. To all Americans, I will be honest with you, as I’ve always promised. A Russian dictator, invading a foreign country, has costs around the world. And I’m taking robust action to make sure the pain of our sanctions  is targeted at Russia’s economy. And I will use every tool at our disposal to protect American businesses and consumers. Tonight, I can announce that the United States has worked with 30 other countries to release 60 Million barrels of oil from reserves around the world. America will lead that effort, releasing 30 Million barrels from our own Strategic Petroleum Reserve. And we stand ready to do more if necessary, unified with our allies. These steps will help blunt gas prices here at home. And I know the news about what’s happening can seem alarming.  
6289  
"""  
  
text_splitter = SemanticChunker(OpenAIEmbeddings(api_key=OPENAI_API_KEY), breakpoint_threshold_type="interquartile")  
docs = text_splitter.create_documents([state_of_the_union])  
print(docs[0].page_content)  
print(len(docs[0].page_content))  
  
# Output  
"""  
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. Last year COVID-19 kept us apart. This year we are finally together again. Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. With a duty to one another to the American people to the Constitution. And with an unwavering resolve that freedom will always triumph over tyranny. Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. He met the Ukrainian people. From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. Groups of citizens blocking tanks with their bodies. Everyone from students to retirees teachers turned soldiers defending their homeland. In this struggle as President Zelenskyy said in his speech to the European Parliament “Light will win over darkness.” The Ukrainian Ambassador to the United States is here tonight. Let each of us here tonight in this Chamber send an unmistakable signal to Ukraine and to the world. Please rise if you are able and show that, Yes, we the United States of America stand with the Ukrainian people. Throughout our history we’ve learned this lesson when dictators do not pay a price for their aggression they cause more chaos. They keep moving.  
1601  
"""

#### 9\. Splitting by Tokens

TokenTextSplitter splits text based on token counts, crucial for managing input length for LLMs. You can utilize tokenizers from libraries like tiktoken, spaCy, NLTK, etc.

* **tiktoken**

# Example - Splitting by Tokens (tiktoken)  
  
with open("/path/to/state_of_the_union.txt") as f:  
    state_of_the_union = f.read()  
  
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(  
    model_name="gpt-4",  
    chunk_size=100,  
    chunk_overlap=0,  
)  
  
texts = text_splitter.split_text(state_of_the_union)  
print(texts[0])  
  
# Output  
"""  
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.    
Last year COVID-19 kept us apart. This year we are finally together again.   
Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.   
With a duty to one another to the American people to the Constitution.  
"""

* **spaCy**

# Example - Splitting by Tokens (spaCy)  
  
with open("/path/to/state_of_the_union.txt") as f:  
    state_of_the_union = f.read()  
  
text_splitter = SpacyTextSplitter(chunk_size=1000)  
texts = text_splitter.split_text(state_of_the_union)  
print(texts[0])  
  
# Output  
"""  
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.  
  
Members of Congress and the Cabinet.  
  
Justices of the Supreme Court.  
  
My fellow Americans.    
  
  
Last year COVID-19 kept us apart.  
  
This year we are finally together again.   
  
  
Tonight, we meet as Democrats Republicans and Independents.  
  
But most importantly as Americans.   
  
  
With a duty to one another to the American people to the Constitution.   
  
  
And with an unwavering resolve that freedom will always triumph over tyranny.   
  
  
Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.  
  
But he badly miscalculated.   
  
  
He thought he could roll into Ukraine and the world would roll over.  
  
Instead he met a wall of strength he never imagined.   
  
  
He met the Ukrainian people.   
  
  
From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.  
"""

* **NLTK**

# Example - NLTK  
  
text_splitter = NLTKTextSplitter(chunk_size=1000)  
texts = text_splitter.split_text(state_of_the_union)  
print(texts[0])  
  
# Output  
"""  
Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.  
  
Members of Congress and the Cabinet.  
  
Justices of the Supreme Court.  
  
My fellow Americans.  
  
Last year COVID-19 kept us apart.  
  
This year we are finally together again.  
  
Tonight, we meet as Democrats Republicans and Independents.  
  
But most importantly as Americans.  
  
With a duty to one another to the American people to the Constitution.  
  
And with an unwavering resolve that freedom will always triumph over tyranny.  
  
Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.  
  
But he badly miscalculated.  
  
He thought he could roll into Ukraine and the world would roll over.  
  
Instead he met a wall of strength he never imagined.  
  
He met the Ukrainian people.  
  
From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.  
  
Groups of citizens blocking tanks with their bodies.  
"""

Choosing the appropriate text splitter is vital for effective LLM usage. By understanding the different options provided by LangChain, you can optimize your text processing pipeline and ensure that your LLMs receive input in the most digestible format

In the next episode, we’ll delve into more features and functionalities of LangChain. Stay tuned to the Data Mastery Series and continue your journey to mastering LangChain, one step at a time. 🚀

---

[**Data Science** _Explore the world of data science with Donato\_Story_](https://medium.com/@designbynattapong/list/6f598ace4d84 "https://medium.com/@designbynattapong/list/6f598ace4d84")[](https://medium.com/@designbynattapong/list/6f598ace4d84)

[**Dashboard** Discover the power of data visualization with Donato\_Story](https://medium.com/@designbynattapong/list/3c36adde1efa "https://medium.com/@designbynattapong/list/3c36adde1efa")[](https://medium.com/@designbynattapong/list/3c36adde1efa)

[**Donato\_Journey** _Join me on my journey (Thai version)_](https://medium.com/@designbynattapong/list/977257e3785b "https://medium.com/@designbynattapong/list/977257e3785b")[](https://medium.com/@designbynattapong/list/977257e3785b)

[**Course\_Review** _Discover the training courses with Donato\_Story (Thai version)_](https://medium.com/@designbynattapong/list/d130d68e8081 "https://medium.com/@designbynattapong/list/d130d68e8081")[](https://medium.com/@designbynattapong/list/d130d68e8081)

#### Let’s Connect!

Your feedback is invaluable. Feel free to share your thoughts or questions in the comments below. You can also connect with me on:

* _Medium:_ [_medium.com/donato-story_](https://medium.com/donato-story)
* _Facebook:_ [_web.facebook.com/DonatoStory_](https://web.facebook.com/DonatoStory)
* _Linkedin:_ [_linkedin.com/in/nattapong-thanngam_](http://www.linkedin.com/in/nattapong-thanngam)