GraphRAG 输入与分块指南：支持的数据格式、统一 documents Schema 与元数据前置（Field Prepending）

GraphRAG 输入与分块指南：支持的数据格式、统一 documents Schema 与元数据前置（Field Prepending）

【免费下载链接】graphragA modular graph-based Retrieval-Augmented Generation (RAG) system项目地址: https://gitcode.com/GitHub_Trending/gr/graphrag

导读：输入数据如何被读取、标准化和切分，直接决定了 GraphRAG 图索引的质量与可追溯性。本文以 docs/index/inputs.md 为主线，系统讲解 GraphRAG 支持的多种输入格式（纯文本、CSV、JSON、JSONL、Parquet、MarkItDown）、统一的documentsDataFrame 列结构、自备 DataFrame 的 Bring-Your-Own 接入方式，以及分块（Chunking）阶段通过prepend_metadata将文档元数据重复复制到每个文本块中的配置方法。读完你将能够：为自己的语料选择最合适的输入格式、正确配置settings.yaml的input/chunking段、并让每个文本块携带文档级共享信息以提升下游实体提取与检索效果。

输入在整个索引流水线中的位置

在 GraphRAG 的默认索引流程中（详见 docs/index/default_dataflow.md），输入处理发生在最前端的Phase 1: Compose TextUnits：原始文件先被读取为documents，再按配置被切分为更小的TextUnit（文本块），后续的实体/关系提取、社区检测、社区报告等阶段全部基于这些文本块展开。

Documents --> Chunk --> Text Units

因此"如何把各种形态的源文件变成一张结构统一的documents表"是全流程的第一道关卡，也是本文的主题。输入加载在索引流水线中由两个核心 workflow 完成：

• load_input_documents：负责将存储中的文件读入并解析成统一格式，实现见 load_input_documents.py；
• create_base_text_units：负责把documents切分为text_units，并将prepend_metadata配置作用于切分过程，实现见 create_base_text_units.py。

统一 Documents 模式（Input Loading and Schema）

无论采用哪种输入格式，所有文件都会在 GraphRAG 内部被加载为一个documentsDataFrame，每个文档占一行，并遵循如下共享列结构：

该TextDocument数据结构的仓库实现位于 text_document.py，其中：

• TextDocument.get(field)支持标准字段（id、title、text、creation_date）直接取属性；其余字段名则会到raw_data中按字段名（甚至支持点号嵌套字段）查找——这正是下文"字段前置"能引用结构化源数据任意列的实现基础；
• creation_date由存储层通过storage.get_creation_date(path)从文件系统中采集（见 text.py 中对纯文本文件的用法）；
• id默认使用 SHA-512 对文本内容进行哈希生成，纯文本读取器直接调用gen_sha512_hash({"text": text}, ["text"])（见 text.py 与 hashing.py），这也是文档强调"ID 基于文本内容哈希以跨运行稳定"的源码依据。

流水线运行结束后，最终的 documents 表会以 parquet 形式落盘，其最终表结构可参考 docs/index/outputs.md 中关于 final documents 表的说明。

输入加载链路：可注入的 InputReader 架构

所有输入格式的解析都统一收敛到同一个架构：InputReader基类 +InputReaderFactory工厂。理解这一点有助于你判断"遇到不支持的格式怎么办"。

InputReader 抽象基类

input_reader.py 定义了InputReader：

• 构造函数接收storage（文件来源）、file_pattern（文件匹配正则）与encoding（默认utf-8）；
• 它实现了异步迭代协议__aiter__/_iterate_files，用self._storage.find(re.compile(self._file_pattern))找到所有匹配文件，逐个调用抽象方法read_file(path)把单个文件解析成list[TextDocument]；
• 单个文件解析失败不会中断整体流程——代码会打印 warning 并跳过该文件继续处理，具有较高的健壮性。

InputReaderFactory 与注册机制

input_reader_factory.py 中的create_input_reader(config, storage)依据config.type分发到对应的 Reader 实现：

同时仓库还导出了register_input_reader(type, initializer)公共注册函数，开发者可以把自己实现的 Reader 类按标准 Factory 模式注册进去（见 input_reader_factory.py），这部分即文档中"Custom File Handling"所述能力。

配置模型说明：输入相关配置由 graph_rag_config.py 中的input: InputConfig承载，其字段（type、encoding、file_pattern、id_column、title_column、text_column）定义在 input_config.py，并且InputConfig开启了extra="allow"，以便自定义 Reader 增加私有字段。

支持的输入格式详解（Formats）

下面逐类说明每种格式的加载语义与可配置列映射。若你的数据形态不在其中，文档建议两条路：一是自行实现InputReader注册进工厂，二是写脚本把数据先转换为以下某一种格式。

纯文本（Plain Text）

以.txt结尾的文本文件（默认input.type即text）。加载规则：

• 整文件内容作为text字段；
• title始终为文件名；
• 每个文件产出一行TextDocument，raw_data为空（None）。

从源码可以看到纯文本读取器把文件名作为 title、把文件内容哈希作为 id、从文件系统采集创建时间，见 text.py。

CSV（逗号分隔）

以.csv结尾的文件，加载语义：

• 采用csv.DictReader逐行解析（csv.py），CSV 的每一行被当作一篇独立文档；
• 输入目录下若有多个 CSV 文件，会被拼接成一个统一的documentsDataFrame；
• 可通过input配置块指定text_column与title_column，让结构化内容中的指定列充当正文与标题；若未配置，title使用文件名、text_column默认取名为text的列；
• 若数据中存在id列则直接使用，否则按前文规则基于文本内容哈希生成 ID。

需要留意的是，CSV 正文单元格内容通常需要做引号转义以符合 CSV 规范，正文中含有换行或逗号的字段务必正确包裹。

JSON

以.json结尾、内容符合 JSON 规范 的结构化文件。加载语义：

• 使用 Python 标准库的json.loads解析，文件内容必须是合法 JSON；
• 文件根部可以是单个对象，也可以是对象数组，两种形态都会被自动识别处理；
• 多个文件会被拼接成最终表；text_column、title_column配置会应用到每个加载对象的属性上。

JSON Lines（JSONL）

以.jsonl结尾的文件，每行一个独立 JSON 对象。空行与纯空白行会被跳过。

Parquet

以.parquet结尾的列式文件，作为结构化行加载，与 CSV/JSON 一样支持text_column、title_column与id_column配置。

MarkItDown

通过 MarkItDown 转换器读取文档（如 PDF、Office 文档等非纯文本格式），即input.type: markitdown。在 input_reader_factory.py 中对应MarkItDownFileReader。

仓库配套的输入解析测试用例可进一步参考 tests/unit/indexing/input 目录下的test_csv_loader.py、test_json_loader.py、test_jsonl_loader.py、test_parquet_loader.py、test_markitdown_loader.py、test_text_loader.py等文件。

Bring-Your-Own DataFrame：跳过解析，直接喂表

如果你已有现成的 pandas DataFrame，或数据存放在仓库内置格式之外的位置，可以直接调用索引 API 传入自己的表，完全绕过上述输入加载/解析环节。

GraphRAG 的索引 APIbuild_index提供了input_documents: pd.DataFrame | None参数（见 api/index.py），其文档字符串明确说明其作用是 "Override document loading and parsing and supply your own dataframe of documents to index"（覆盖文档加载与解析流程，提供自定义文档表用于索引）。

前提约束：你传入的 DataFrame 必须符合前文给出的documents统一 Schema（id、text、title、creation_date、raw_data）。满足该前提下，后面所有分块行为与走内置读取器完全一致。

元数据前置（Field Prepending）的背景动机

为什么要在分块时把元数据反复复制进每个文本块？文档给出了一个典型场景：你在索引一批新闻文章，每篇文章以标题（headline）和作者（author）开头，然后是正文。默认分块按配置的chunk_size从头到尾均匀切分——即前n个 token 进入第一个文本块，再下一个n个进入第二块，依此类推。

这意味着位于文档开头的"导语/头部信息"不会出现在每个文本块中，只会留在第一个块里。当后续阶段对这些文本块进行实体提取或摘要时，中间与末尾的块就会缺少"这篇文章是谁、标题是什么"这类全局共享信息。为避免该问题，GraphRAG 提供配置，把这类重复性内容复制到每个文本单元中。

分块配置与字段前置配置

input 配置块：声明可引用字段

如上文所述，结构化输入源的字段保存在raw_data中，可在分块前置阶段按字段名直接引用。id、title、text、creation_date这些标准字段（即使是非结构化文本）也可以被引用。

chunking 配置块：控制前置行为

prepend_metadata用于指示导入器把选中的文档字段复制到每个文本块的开头：

• 值以key: value的键值对形式写入，每对一行；
• 若未配置该设置，元数据在切分时默认被忽略；
• 对于结构化输入，raw_data对象内含源文件中出现的其余字段（对 CSV 而言即除被用作text/title/id之外的所有列）。

配置模型ChunkingConfig中该字段被定义为prepend_metadata: list[str] | None（见 chunking_config.py），即一组待前置字段名。

从实现看，create_base_text_unitsworkflow 会把config.chunking.prepend_metadata传入分块函数，并在切分前对每个文档执行document.collect(prepend_metadata)收集字段，再用add_metadata(metadata=..., line_delimiter=".\n")将其序列化为键值对文本写入每个块（见 create_base_text_units.py 与 transformers.py）。

关于小节命名的一致性提醒：settings.yaml中承载分块配置的顶级小节为chunking（对应GraphRagConfig.chunking，见 graph_rag_config.py），由graphrag init生成的 settings 文件也使用该键名；本文转录的早期版本文档示例中曾写作chunks，请以你所使用版本实际生成的 settings.yaml 为准。

一个最小示例：给每个块带上title与tag

假设源文件software.csv内容如下：

text,title,tag My first program,Hello World,tutorial An early space shooter game,Space Invaders,arcade

配置：

chunking: prepend_metadata: [title,tag]

则加载出的documentsDataFrame 为：

分块时，title与tag两个字段会被拼接到每个文本块起始处，使模型始终能看到该软件条目的标题与分类标签。

三种格式的分块完整示例

下文三个示例使用"单词数"模拟 token 以便演示。需要说明的是：LLM 的 token 并不等同于单词，实际切分以所选编码模型的 token 数为准（token 概念与计数方式参考）。

纯文本文件：前置文件名（title）

两篇独立新闻文本文件。

文件：US to lift most federal COVID-19 vaccine mandates.txt

内容：

WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Biden's administration is moving to treat COVID-19 as a routine, endemic illness. "While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down," White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday.

文件：NY lawmakers begin debating budget 1 month after due date.txt

内容：

ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains "significant wins" for New Yorkers. "I would have liked to have done this sooner. I think we would all agree to that," Cousins told reporters before voting began. "This has been a very policy-laden budget and a lot of the policies had to parsed through." Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the "least restrictive" means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. That's up from $15 in the city and $14.20 upstate.

settings.yaml

input: type: text metadata: [title] chunking: size: 100 overlap: 0 prepend_metadata: true

Documents DataFrame

Raw Text Chunks

该示例要点：

• 两篇输入文档共被切分成5 个输出文本块（第一篇 2 块，第二篇 3 块）；
• 每篇文档的文件名（title）被前置到该文档所有文本块的开头；
• 前置的 title 不计入 chunk size（length只统计正文部分的词数）；
• 每篇文档的最后一个文本块通常小于 chunk size，因为它只包含剩余的最后若干 token。

CSV：借助text_column/title_column做列映射

同一篇文章作为 CSV 的两行（注意：为便于阅读，正文未按实际 CSV 规范转义）。

文件：articles.csv

内容

headline,article US to lift most federal COVID-19 vaccine mandates,WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Biden's administration is moving to treat COVID-19 as a routine, endemic illness. "While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down," White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday. NY lawmakers begin debating budget 1 month after due date,ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains "significant wins" for New Yorkers. "I would have liked to have done this sooner. I think we would all agree to that," Cousins told reporters before voting began. "This has been a very policy-laden budget and a lot of the policies had to parsed through." Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the "least restrictive" means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. That's up from $15 in the city and $14.20 upstate.

（CSV 场景可把headline配置为title_column、article配置为text_column，原理与下文 JSON 示例一致，这里不再赘述切分细节。）

JSON：列映射 + overlap 重叠演示

本示例为两篇文章各建一个 JSON 文件，只配置读取字段，不向前置块追加元数据。

文件：article1.json

内容

{ "headline": "US to lift most federal COVID-19 vaccine mandates", "content": "WASHINGTON (AP) The Biden administration will end most of the last remaining federal COVID-19 vaccine requirements next week when the national public health emergency for the coronavirus ends, the White House said Monday. Vaccine requirements for federal workers and federal contractors, as well as foreign air travelers to the U.S., will end May 11. The government is also beginning the process of lifting shot requirements for Head Start educators, healthcare workers, and noncitizens at U.S. land borders. The requirements are among the last vestiges of some of the more coercive measures taken by the federal government to promote vaccination as the deadly virus raged, and their end marks the latest display of how President Joe Biden's administration is moving to treat COVID-19 as a routine, endemic illness. \"While I believe that these vaccine mandates had a tremendous beneficial impact, we are now at a point where we think that it makes a lot of sense to pull these requirements down,\" White House COVID-19 coordinator Dr. Ashish Jha told The Associated Press on Monday." }

文件：article2.json

内容

{ "headline": "NY lawmakers begin debating budget 1 month after due date", "content": "ALBANY, N.Y. (AP) New York lawmakers began voting Monday on a $229 billion state budget due a month ago that would raise the minimum wage, crack down on illicit pot shops and ban gas stoves and furnaces in new buildings. Negotiations among Gov. Kathy Hochul and her fellow Democrats in control of the Legislature dragged on past the April 1 budget deadline, largely because of disagreements over changes to the bail law and other policy proposals included in the spending plan. Floor debates on some budget bills began Monday. State Senate Majority Leader Andrea Stewart-Cousins said she expected voting to be wrapped up Tuesday for a budget she said contains \"significant wins\" for New Yorkers. \"I would have liked to have done this sooner. I think we would all agree to that,\" Cousins told reporters before voting began. \"This has been a very policy-laden budget and a lot of the policies had to parsed through.\" Hochul was able to push through a change to the bail law that will eliminate the standard that requires judges to prescribe the \"least restrictive\" means to ensure defendants return to court. Hochul said judges needed the extra discretion. Some liberal lawmakers argued that it would undercut the sweeping bail reforms approved in 2019 and result in more people with low incomes and people of color in pretrial detention. Here are some other policy provisions that will be included in the budget, according to state officials. The minimum wage would be raised to $17 in New York City and some of its suburbs and $16 in the rest of the state by 2026. That's up from $15 in the city and $14.20 upstate." }

settings.yaml

input: type: json title_column: headline text_column: content chunking: size: 100 overlap: 10

Documents DataFrame

Raw Text Chunks

该示例要点：

• 两篇 JSON 文档同样被切分成 5 个输出文本块；
• 本次没有做元数据前置，因此每个块的内容紧贴配置的 chunk size（100），仅每篇文档的最后一个块不足 100；
• 配置了overlap: 10，可以观察到相邻文本块之间共享了末尾 10 个 token——例如块 1 结尾的 "as" 与块 2 开头重叠。overlap 用于降低因硬切分而把语义割裂在块边界处的风险。

分块参数：chunk_size 与 overlap 的取舍

• chunking.size（chunk_size）：按 token 计数的块大小。分块是必须的，因为文档内容往往超过所用语言模型的上下文窗口。默认流程中该值默认配置为 1200 token（见 docs/index/default_dataflow.md）；更大的块会带来更低的提取保真度与更不具参考价值的上下文文本，但通常能显著加快处理速度，需要按语料与成本权衡。
• chunking.overlap：相邻文本块之间共享的 token 数，用于缓解切点处的语义断裂。

分块相关的单元测试可参考 tests/unit/chunking（如test_chunker.py、test_prepend_metadata.py），workflow 层面的测试在 test_create_base_text_units.py。

本文配套可继续深入阅读的仓库资料

• 输入 Schema 的运行时载体： text_document.py
• 各格式 Reader 实现：packages/graphrag-input/graphrag_input（text.py、csv.py、json.py、jsonl.py、parquet.py、markitdown.py）
• Reader 注册与工厂分发：input_reader_factory.py
• 输入加载与分块 workflow：load_input_documents.py 与 create_base_text_units.py
• 分块元数据变换工具：transformers.py
• 自定义组件架构说明：docs/index/architecture.md
• 默认数据流整体概览：docs/index/default_dataflow.md
• 完整配置总览：docs/index/overview.md 与 docs/config/overview.md

最后再次强调一条贯穿全文的原则：文档 ID 由文本内容哈希生成（SHA-512），目的是跨运行稳定；当使用 CSV/JSON 等结构化格式时请优先保留或显式指定稳定id列，这直接影响增量索引与溯源在多次重建间的可靠性。

【免费下载链接】graphragA modular graph-based Retrieval-Augmented Generation (RAG) system项目地址: https://gitcode.com/GitHub_Trending/gr/graphrag