【RAG】【retrievers08】基于Together.ai长上下文嵌入的混合检索-开发者社区

案例目标

本案例展示如何使用Together.ai的长上下文嵌入模型实现混合检索系统，结合块级别和文档级别的相似度计算，提高RAG系统的检索质量。通过将文档嵌入与块嵌入相结合，利用文档级别的语义信息辅助块级别的检索，从而获得更准确的检索结果。

技术栈与核心依赖

llama-index-embeddings-together
llama-index-llms-openai
llama-index-readers-web
llama-index-readers-file
llama-index-core
BeautifulSoup4
llama-index-vector-stores-chroma
openai

环境配置

# 安装必要的依赖
pip install llama-index-embeddings-together llama-index-llms-openai
pip install llama-index-readers-web llama-index-readers-file
pip install llama-index-core beautifulsoup4
pip install llama-index-vector-stores-chroma openai
# 设置API密钥
import os
os.environ["TOGETHER_API_KEY"] = "your_together_api_key"
os.environ["OPENAI_API_KEY"] = "your_openai_api_key"

案例实现

1. 数据准备

步骤 1

使用BeautifulSoup爬取LlamaIndex文档，获取120个链接的内容：

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

def get_all_links(url):
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
links = []
for link in soup.find_all('a'):
href = link.get('href')
if href and href.startswith('/') and not href.startswith('//'):
full_url = urljoin(url, href)
links.append(full_url)
return links

步骤 2

使用AsyncHtmlLoader和Html2TextTransformer处理网页内容：

from llama_index.readers.web import AsyncHtmlLoader
from llama_index.core.node_parser import HTML2TextTransformer

loader = AsyncHtmlLoader(urls[:120])
docs = await loader.aload_data()

transformer = HTML2TextTransformer()
transformed_docs = transformer.transform_documents(docs)

2. 混合检索器实现

步骤 3

初始化Together嵌入模型：

from llama_index.embeddings.together import TogetherEmbedding

embed_model = TogetherEmbedding(
model_name="togethercomputer/m2-bert-80M-8k-retrieval",
api_key=os.environ["TOGETHER_API_KEY"]
)

步骤 4

创建自定义混合检索器类：

class HybridRetriever(BaseRetriever):
def __init__(
self,
vector_retriever,
doc_retriever,
alpha=0.5, # 平衡块和文档相似度的权重
):
self.vector_retriever = vector_retriever
self.doc_retriever = doc_retriever
self.alpha = alpha
super().__init__()

def _retrieve(self, query_bundle: QueryBundle):
# 获取块级别的检索结果
vector_nodes = self.vector_retriever.retrieve(query_bundle)

# 获取文档级别的检索结果
doc_nodes = self.doc_retriever.retrieve(query_bundle)

# 创建文档ID到文档节点的映射
doc_id_to_node = {node.node_id: node for node in doc_nodes}

# 计算混合分数
hybrid_nodes = []
for node in vector_nodes:
# 获取节点所属文档的ID
doc_id = node.metadata.get("doc_id")
if doc_id and doc_id in doc_id_to_node:
# 获取文档级别的相似度
doc_node = doc_id_to_node[doc_id]
doc_similarity = doc_node.score

# 计算混合分数
node_score = node.score
hybrid_score = (1 - self.alpha) * node_score + self.alpha * doc_similarity

# 更新节点分数
node.score = hybrid_score

hybrid_nodes.append(node)

# 按混合分数排序
hybrid_nodes.sort(key=lambda x: x.score, reverse=True)

return hybrid_nodes

3. 构建检索系统

步骤 5

创建文档级别的向量存储：

from llama_index.core import Document, VectorStoreIndex, StorageContext
from llama_index.core.node_parser import SentenceSplitter

# 创建文档对象
documents = [Document(text=doc.text, metadata={"doc_id": str(i)}) for i, doc in enumerate(transformed_docs)]

# 创建文档级别的节点解析器（不分割文档）
doc_parser = SentenceSplitter(chunk_size=1000000) # 设置一个很大的值，确保不分割

# 创建文档级别的索引
doc_index = VectorStoreIndex.from_documents(documents, embed_model=embed_model, transformations=[doc_parser])
doc_retriever = doc_index.as_retriever(similarity_top_k=10)

步骤 6

创建块级别的向量存储：

# 创建块级别的节点解析器
chunk_parser = SentenceSplitter(chunk_size=512)

# 创建块级别的索引
chunk_index = VectorStoreIndex.from_documents(documents, embed_model=embed_model, transformations=[chunk_parser])
vector_retriever = chunk_index.as_retriever(similarity_top_k=10)

步骤 7

创建混合检索器：

# 创建混合检索器
hybrid_retriever = HybridRetriever(
vector_retriever=vector_retriever,
doc_retriever=doc_retriever,
alpha=0.5 # 平衡块和文档相似度的权重
)

4. 查询与评估

步骤 8

执行查询并比较结果：

from llama_index.core.query_engine import RetrieverQueryEngine

# 创建查询引擎
query_engine = RetrieverQueryEngine(hybrid_retriever)
base_query_engine = chunk_index.as_query_engine(similarity_top_k=10)

# 执行查询
query_str = "What is the LLM interface in LlamaIndex?"
response = query_engine.query(query_str)
base_response = base_query_engine.query(query_str)

# 打印结果
print("混合检索结果:")
print(str(response))
print("\n基础检索结果:")
print(str(base_response))