面包屑图标 当前位置: 首页
AI资讯
热点详情

吴恩达DeepLearning.AI大模型RAG第三部分:向量数据库与嵌入

AI热点日报
AI热点日报时间:2026-05-30
热点解读

文本切分提升向量数据库查询精准度、效率和信息覆盖。向量化将文本转为高维数字编码,语义相似的向量距离更近。所有chunk向量存入高维空间构成向量数据库,用户查询向量化后匹配最近向量,检索对应文本块给大语言模型,结合外部数据提升回答专业性和准确性。

在之前的课程中,我们已经完成了文档导入与文本切分这两步。你可能会好奇:传统的数据库明明不需要这么复杂,为什么非得切成小块?答案就藏在向量数据库的特性中——正是这种“非结构化”的存储方式,才让切分成为必不可少的一环。

  • 首先,切分能大幅提升查询的精准度。将长篇内容拆解成多个小片段,每个片段都能更细腻地反映文档中的具体信息。对向量数据库而言,这些小片段在向量化时会形成更准确的语义表示,进行相似度搜索时,自然能匹配到更相关、更精确的内容。
  • 其次,切分让向量化处理更加高效。文本越长,特征提取的难度和复杂度就越高;而短文本的向量化过程更简洁、计算量更小,生成的向量也更具代表性。
  • 第三,检索效率也随之提高。向量数据库通常存储数十万条数据,每次相似度计算都需要消耗资源。切分后的小块能降低单次计算复杂度,同时让用户的查询需求与小块信息快速精准地匹配。
  • 最后,切分还能实现更好的信息覆盖。长文档往往涵盖多个主题,直接整体向量化容易稀释或忽略某些关键信息点。切成小块后,每个块代表一个独立的主题或信息点,数据库里的信息覆盖面更全,查询时给出的响应也自然更丰富。

所以说,当技术把文本划分成一个个 chunk 之后,下一步就是存储。存储这件事,在信息管理里是最基础也最关键的一环——信息不被存下来,后续所有检索都无从谈起。将这些 chunk 存入向量数据库,就是为后续精准检索和高效利用打下坚实基础。

向量化处理

每个 chunk 的第一步,都是 Embeddings(向量化)。Embeddings 的本质,是把一段文字转化成一长串数字——说白了,就是把人类语言翻译成机器能理解和操作的数字语言。当这些高维向量被投射到一个高维空间后,你会发现一个有趣的现象:语义越相似的两段话,它们在空间里的距离就越近。这个距离的相近性,正是向量化的核心特征,也是我们后续进行语义匹配的基础。

构建向量数据库

所有 chunk 都变成向量后,把它们统一放进一个高维空间,就构成了一个向量数据库。数据库里每个向量对应一个文本块的语义特征。当用户抛出一个问题,这个问题也会先被转化为向量,然后放入这个空间。通过计算,找到与它距离最近的几个向量——这些向量代表的文本块,在语义上与用户问题最相似。然后我们拿这些向量当索引,找到对应的文本内容,再把这些内容喂给大语言模型。

提升大语言模型的回答质量

这样一来,大模型给出的回答就不只是依赖它预训练时学到的知识了,还能结合向量数据库里丰富的外部信息。结果就是:回答更专业、更贴合实际场景,也更有上下文感。这种外部数据的支撑,让模型的实用性大大提升,回答的准确性也上了几个台阶。

这种架构在处理复杂问题时特别管用——尤其是那些需要实时更新数据或者对特定领域有高精度要求的场景。通过内部知识与外部数据库的结合,我们能在更广泛的场景下提供更高质量、更个性化的响应。

实战演练

下面直接进入实战,看看怎么实现文本向量化。环境版本如下:

langchain                0.3.0
langchain-community      0.3.0
pypdf                    5.0.0
openai                   1.47.0
beautifulsoup4           4.12.3
chromadb                 0.5.15

做词向量嵌入之前,先导入文件并切分。这里我们还是用网页内容,最后打印一下切分后的块数。

from langchain_community.document_loaders import WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

# 文件导入
loader = WebBaseLoader("https://zh.d2l.ai/")
docs = loader.load()

# 文本切分
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size = 1500,
    chunk_overlap = 150
)
splits = text_splitter.split_documents(docs)
print(len(splits))

运行后可以看到,按照我们的切分方式,这段内容被分成了18个块。接下来就是找个合适的 embedding 模型,把这些文本转化成数字表达并存入向量空间。Langchain 对 OpenAI 的 embedding 模型支持最好,但因为国内网络问题没法直接调用 OpenAI,所以需要去 Langchain 官网找其他可用的模型。

搜索一圈下来,发现 Langchain 里已经支持了百度千帆和百川的 embedding 模型。下面以百川的模型为例做演示。先到百川智能官网注册账户,跟着新手引导创建自己的 API Key。

之后还需要完成实名认证。认证后就能看到账户上有80元赠送金,足够完成后续的文本嵌入操作了。

先用一个简单的句子测试一下向量数据库能不能跑通。

from langchain_community.embeddings import BaichuanTextEmbeddings

embeddings = BaichuanTextEmbeddings(baichuan_api_key="sk-*") # 这里写入刚刚创建的API Key
text_1 = "今天天气不错"

query_result = embeddings.embed_query(text_1)
print(query_result)

如果运行后顺利显示出一长串数字列表,说明模型配置成功。

[-0.0075800973, 0.05336676, -0.017781364, 0.035153043, 0.05473809, -0.0034863392, -0.017725622, 0.03400442, -0.0015618298, 0.037048902, 0.023895627, -0.045087088, -0.0064525804, -0.0049004983, -0.064887315, -0.015825018, 0.022772776, 0.0060217036, 0.025914272, 0.024085524, -0.014776563, -0.050150223, -0.026934989, -0.016943432, -0.015303897, 0.0023684907, 0.056052454, 0.0498096, 0.054810315, 0.014616517, -0.0108804125, 0.042461645, 0.015064584, 0.056263994, 0.027121587, -0.022202492, -0.027525024, -0.008664351, -0.036344998, -0.01657127, 0.014527989, 0.028132828, -0.014738482, 0.009414009, 0.07236609, 0.012243288, -0.022873089, -0.0010711104, 0.00669326, -0.07445394, 0.010032093, -0.0126827555, -0.013846456, -0.03963767, -0.0015490066, -0.014372803, 0.013366017, 0.035709277, -0.00043608135, 0.03880523, -0.029518811, 0.17903462, -0.013301915, 0.039601598, 0.007936066, -0.022787375, -0.043096073, -0.015623084, -0.006984037, -0.05443407, 0.014602836, -0.010637717, -0.024930496, -0.005672862, -0.030609699, -0.04613513, -0.005200756, -0.0019113886, -0.0010369268, 0.06364622, 0.0017903518, -0.013281618, -0.049308456, 0.0055903164, -0.039054055, 0.038039792, -0.037285045, 0.090783335, -0.015622992, 0.016203934, -0.054214057, 0.03961282, 0.026360435, -0.018353252, 0.016519671, 0.041751813, -0.014477077, 0.0127489995, -0.030455282, 0.024851138, 0.019062009, -0.017065959, -0.02627655, 0.02258509, -0.04847564, 0.073734246, -0.041269477, -0.01692317, 0.022543054, -0.0034427822, -0.014681885, 0.02256325, -0.010713156, -0.005427022, -0.05921746, -0.011502023, 0.02275171, 0.037494913, 0.047674343, -0.026584001, 0.04412184, 0.055725418, 0.007986744, -0.021016385, -0.023211442, -0.036163304, 0.016720304, 0.033081684, -0.0032843906, -0.042655814, 0.007911628, -0.024465615, 0.015406191, -0.010756438, -0.03702144, -0.02317689, -0.054836072, -0.031834766, -0.02790807, -0.032853704, 0.019062275, -0.046842992, 0.017380653, 0.014115385, 0.014931091, 0.010515374, -0.05607991, -0.0019574235, 0.013854338, 0.012391423, 0.018231213, 0.076454654, 0.008188162, -0.037639007, 0.045298122, -0.073848456, 0.068050385, -0.02705975, 0.023622544, -0.0042208578, 0.0337943, -0.022992376, 0.05826827, 0.011858104, 0.01569607, 0.042661842, -0.017209984, 0.021694295, 0.007505209, 0.011397206, -0.022396868, 0.006295061, -0.02817121, -0.020757983, -0.029239543, 0.017580206, 0.0053575137, 0.0047285426, -0.008169297, -0.025038041, 0.06669833, 0.03582289, 0.0050918288, -0.027723635, 0.00014384017, -0.02224188, -0.015801586, -0.01848588, 0.050851304, -0.09774136, -0.017576136, 0.057299227, 0.000442171, 0.013498493, 0.055637747, 0.0124135325, 0.048408154, 0.007920959, 0.0095843775, -0.02551911, 0.078900434, -0.018005438, -0.04383461, 0.006133353, 0.0077908253, -0.015221063, -0.0068961675, -0.004510117, -0.003239107, -0.016756145, -0.0010812985, -0.012540754, 0.06154694, 0.033851653, -0.03275014, -0.031958163, 0.034976978, -0.0070638503, -0.0107440045, -0.01419383, -0.020839222, 0.017075969, 0.022541199, -0.022424642, -0.008391753, 0.027798742, -0.016050762, 0.0074032117, 0.0127465585, 0.054057844, 0.02783872, -0.02091366, 0.03957936, 3.540524e-05, -0.02064022, 0.016737267, -0.020010721, -0.028651457, 0.012814642, 0.012334396, 0.01962399, 0.009970842, -0.0016417224, 0.019976128, 0.02243525, 0.008874889, 0.016382845, 0.053224027, 0.015183802, -0.027056694, -0.017940272, -0.052429568, -0.021074902, 0.0082086325, 0.013914038, -0.0017846578, -0.0028791567, 0.04976489, -0.02925111, 0.051794793, 0.039445613, 0.00066636136, -0.020006057, -0.028057193, 0.009057354, 0.032207295, 0.03994987, -0.009276522, 0.004099356, -0.0017198246, 0.008118259, 0.037628453, 0.003725293, 0.004188389, 0.010137537, 0.009433222, -0.017439801, -0.013881865, 0.0361468, 0.02582073, -0.016355189, -0.04893795, 0.006175331, -0.002328138, 0.031224497, 0.021124806, -0.047953952, 0.018411173, 0.0062507354, 0.032359783, -0.016102508, 0.016811898, 0.0061463136, 0.019790461, 0.0749888, 0.0231207, 0.037050426, 0.0382117, -0.006130921, 0.033440713, 0.04484506, 0.0064165182, 0.014606872, 0.031042341, 0.04283885, 0.025414506, 0.010775507, 0.037886404, 0.048966765, 0.024765946, -0.014064301, -0.03577636, 0.03747178, -0.026369339, 0.01162005, 0.008283756, -0.035896856, 0.018269759, -0.0018216838, 0.037973937, 0.0071238307, -0.021558587, -0.056917723, -0.036986127, -0.018191688, 0.003929742, 0.014920644, 0.046500735, -0.003899504, -0.025425287, 0.06907293, 0.0033958552, -0.011529156, -0.010054631, -0.041107662, 0.026493138, 0.019427532, -0.008603713, -0.021817626, 0.08313295, 0.019652348, 0.04430555, 0.016818708, 0.008762839, 0.024853677, -0.006410369, 0.028246468, 0.003120029, 0.026448244, 0.021368962, 0.020341473, 0.032707144, 0.010048966, -0.0052957716, 0.011840406, 0.005509251, 0.027914446, -0.031125596, -0.055385284, 0.03086642, 0.017625086, 0.0071002766, -0.028331919, 0.009388892, 0.011837206, -0.02456342, -0.07309796, -0.00057860994, -0.015253635, 0.038293343, 0.021359684, 0.029337738, -0.019162798, -0.046192612, -0.007225378, 0.002991033, 0.022218604, 0.027269544, 0.019665781, 0.045284394, -0.0138705885, -0.02697327, -0.02061466, -0.02047067, 0.027559392, -0.02104657, -0.0058107083, -0.029897332, -0.020770589, 0.015403516, -0.008580397, -0.026312502, -0.019879675, -0.030739132, 0.036861204, -0.008701295, 0.017366812, -0.056029975, 0.01807307, 0.049722634, -0.02142505, -0.025946185, -0.054207556, -0.0023204167, 0.016083129, -0.034265004, 0.06520241, -0.0015073599, -0.048764326, -0.013347777, -0.02209387, 0.039798964, -0.020732056, 0.029141719, -0.031482995, 0.032682046, 0.0010684143, -0.0097546335, -0.042786293, 0.014075801, 0.07067692, 0.037168015, 0.03665133, 0.013099383, 0.045177884, -0.004970739, -0.011883725, -0.017102279, -0.003542377, -0.0055756634, 0.003154168, 0.018256243, -0.003425703, -0.022192966, 0.017587043, -0.0498685, -0.036788683, 0.0065018726, 0.05418345, -0.0061499123, 0.023049217, -0.045025963, 0.03354438, 0.004722013, 0.025999319, 0.0023555926, -0.025582422, 0.010080429, -0.007307532, 0.0012863537, 0.019033063, -0.070112355, -0.013475271, -0.0065755583, 0.11608823, 0.004442146, -0.04703502, 0.001603611, -0.0015136485, 0.049734406, -0.002725904, -0.034830157, -0.0061635147, -0.05292667, 0.0022673467, -0.05968527, 0.020396495, 0.042536885, 0.035687473, -0.008441101, -0.0050368826, -0.010241215, 0.026221493, -0.012606776, 0.0052525415, -0.013098624, 0.021647796, -0.032603852, -0.01961021, -0.0014802065, -0.013694488, -0.026213555, -0.004319692, -0.0024521998, -0.033362728, -0.015006008, -0.00011668134, 0.006287778, -0.026308479, 0.0051860493, -0.006457781, 0.022501219, -0.060934205, 0.02888547, -0.013671586, 0.014401986, -0.013942933, 0.023061465, -0.016080467, 0.046032842, 0.009503975, -0.04311421, -0.029868264, 0.005392035, 0.077726334, -0.0032545268, -0.030466318, -0.036948357, 0.021784065, -0.013924415, 0.0051887096, -0.024259165, -0.0034383263, -0.006792153, -0.01465431, -0.0028333322, 0.031919274, -0.004374, 0.009561171, 0.008825877, 0.03902738, 0.02316122, -0.0044253934, 0.009838579, -0.05157412, -0.016854126, -0.03643707, 0.0051199044, -0.033243462, -0.020101415, 0.0062606363, 0.0029247815, 0.045445547, -0.0033785962, -0.019610366, 0.03553738, 0.08627263, -0.017463442, -0.032748982, 0.026960844, 0.034985423, 0.024329964, -0.021733815, 0.04659081, 0.07016787, 0.029160311, 0.009318909, -0.04597192, 0.009287834, 0.01196339, 0.00562987, 0.0034188589, 0.03260338, -0.017949251, 0.0062056463, -0.00607117, -0.034285963, 0.004724186, 0.006547158, -0.05276999, -0.0029403407, 0.028996695, 0.0063203387, -0.0128724035, -0.011359581, 0.010519107, 0.023802942, 0.022068769, 0.053198587, 0.038477365, 0.008498884, -0.029403154, -0.044804808, 0.0014762741, -0.010704774, -0.015829897, 0.007617258, 0.013373788, -0.06088202, 0.034654226, -0.020530708, 0.038792215, -0.027131623, -0.023142476, -0.009830022, 0.00865907, -0.03165655, -0.014819898, -0.005636912, 0.015855074, 0.04296683, -0.0061140233, 0.00010367572, -0.007218054, 0.01453267, -0.026057784, 0.015334749, -0.067188405, -0.021128211, -0.041068982, -0.025423573, -0.042716593, -0.020645829, 0.0042654662, 0.0046950225, -0.018072188, 0.00816476, -0.018691676, -0.043077044, -0.012576883, 0.04691853, -0.008885239, 0.014005581, -0.03384052, 0.013673596, 0.006526697, 0.05935537, -0.005123774, -0.034266487, -0.007336818, 0.02392226, 0.040810507, -0.025057884, 0.06466703, -0.03366888, -0.041856468, -0.01415845, 0.048406634, -0.053556327, 0.018411247, -0.0024865929, 0.0051267263, 0.005148555, 0.0076659825, -0.007355344, 0.026327092, 0.015124973, 0.01254597, -0.03510743, 0.013027197, 0.021324873, 0.0037396972, 0.011446163, -0.049029667, -0.06374169, -0.035224453, -0.008530957, 0.013979153, -0.052223757, -0.07793377, 0.0019591607, 0.0036665122, 0.018972708, -0.054722056, 0.011637819, -0.04576508, 0.020068552, -0.014040663, 0.0057487655, -0.02871472, 0.0003392991, -0.05118145, -0.008924686, 0.055410165, -0.014164663, 0.012071178, -0.089032665, 0.061980903, -0.013205378, 0.011523452, -0.010561611, -0.031480946, 0.017789777, 0.015061673, -0.01870068, 0.0050982414, -0.037708025, -0.037060946, -0.024205728, -0.0051698796, -0.047261383, -0.010686779, -0.009199801, 0.026395503, 0.014988068, -0.056384664, -0.019924317, -0.020795995, 0.029633936, -0.05803649, 0.013602117, -0.026799895, 0.02065851, -0.037712716, 0.0012044552, 0.022238478, -0.03026828, 0.01717357, 0.0122454455, -0.022047512, -0.030403102, -0.03818517, 0.0022959309, 0.05757244, 0.04599548, -0.034720577, -0.04541671, 0.035721175, 0.025337683, 0.015104129, -0.024553472, -0.0120457215, 0.011136674, 0.011347994, 0.011413503, -0.05628646, -0.036509324, -0.0073100314, -0.015757952, 0.025875537, 0.024211854, 0.033534613, -0.0070441454, -0.0112673305, -0.01562657, 0.018628063, 0.0117082475, 0.0069683115, 0.0028892006, -0.032603897, 0.04310698, -0.030078558, 0.019854952, 0.0020202359, 0.011382138, 0.067307346, 0.0153360125, -0.036706623, -0.06758915, -0.009290535, 0.0032376533, 0.021220103, -0.04974581, 0.006176343, 0.02083126, 0.008996937, -0.012759147, -0.022663713, -0.05233211, 0.05656895, -0.03491627, -0.07367662, 0.0012407355, -0.021305252, 0.0013611852, -0.002151927, -0.010974503, -0.018530773, -0.015555192, -0.0039501754, -0.034966897, -0.05811783, 0.003567218, -0.03514525, 0.02981192, -0.031437088, -7.947596e-05, 0.06417666, -0.01710917, 0.042598553, -0.024223894, -0.005406725, 0.004249046, -0.0027075368, -0.048399337, -0.067284755, -0.026074586, -0.00038019277, -0.006811091, -0.07566606, 0.08235222, 0.053955197, -0.03657168, -0.003544574, -0.079615615, 0.03951712, 0.0024060132, 0.038426884, -0.02753584, -0.06644118, -0.031597123, 0.04755233, -0.023949284, -0.032563534, 0.04564931, 0.06101583, -0.052196596, 0.01139768, -0.047797628, -0.039955243, -0.05175579, 0.01297511, -0.041249238, 0.019837338, 0.049250647, -0.014128847, 0.04320848, 0.010422725, -0.07393222, -0.006160438, 0.0027281432, 0.0030357142, 0.008153918, -0.011437956, 0.03710885, -0.0060646483, 0.041081015, -0.0504446, -0.0077678473, -0.018858869, -0.0012665257, 0.0011934272, 0.008902389, -0.004539913, 0.01709035, 0.03139153, -0.015624287, 0.019661324, 0.031275284, -0.027015321, -0.050437905, -0.0012837746, 0.008730318, 0.039631095, -0.03803519, 0.027657144, 0.0046184133, 0.047920577, 0.026898373, -0.013888336, 0.024467615, 0.046619438, 0.0073392773, 0.026615879, 0.0051005054, -0.0010000536, -0.02616397, -0.019414691, -0.021826822, 0.06719448, -0.04384584, -0.014913537, 0.018902838, -0.03909702, -0.037313778, -0.025058893, 0.040213257, -0.05010741, -0.009008712, -0.03383378, -0.01292646, 0.017064018, -0.02641823, -0.001377406, 0.04626737, -0.016107187, -0.042900816, -0.032694705, -0.035868037, 0.053755976, 0.040513847, 0.039083917, 0.016084548, -0.021831522, -0.018332051, -0.0018600253, 0.017321626, 0.015134231, 0.010182116, -0.033388954, 0.013745612, 0.03670928, -0.0120792445, -0.034577135, -0.03743215, 0.019235548, 0.05369886, 0.0060843416, 0.028200045, 0.02975724, 0.030495575, 0.0010913435, -0.007207034, -0.019063968, -0.03966763, 0.045496132, 0.021001477, 0.023116166, 0.0066026025, -0.03898141, -0.0073256157, -0.04920679, -0.045882054, -0.02638399, -0.022420716, 0.008425027, 0.022284094, -0.03773097, -0.025559437, -0.103974216, -0.005242592, -0.0671696, 0.026114486, 0.025766661, 0.0066235885, -0.027907312, -0.023954913, -0.07919421, 0.022901187, -0.04137246, -0.0090224035, -0.00075775903, 0.05223202, -0.023998972, 0.013845849, 0.020067148, 0.027350433, 0.010106185, 0.027008096, 0.004266425, 0.0053291633, -0.023529287, 0.0036859394, 0.022047544, -0.005255481, 0.02131725, -0.010966137, 0.021549305, -0.032396805, -0.006565986, -0.004006749, -0.0071045044, -0.07429621, -0.0021682882, 0.036354702, 0.026676627, 0.004047951, -0.013582781, 0.035064716, 0.0057615186, 0.020303765, 0.044283222, 0.022294967, 0.005740548, 0.02587493, -0.017269071, 0.0017926773, -0.026692772, 0.03899112, -0.0014920657, 0.01090971, -0.024606658, 0.0037900629, 0.065766014, -0.02223905, -0.017547399, -0.0033335802, 0.07088441, 0.015989626, -0.011706954, -0.0348461, 0.016082006, 0.03237533, 0.017515019, -0.037049048, -0.028268043, -0.05145202, -0.0076028937, -0.020903181, 0.00521495, 0.02538109, -0.041564934, 0.084722616, 0.02844224, 0.013816085, -0.018994967, -0.016160535, 0.010936769, 0.004741091, -0.008952183, 0.01748413, -0.015781391, -0.0027252352, 0.009749333, 0.002431545, -0.016886959, -0.016557302, -0.008530143, 0.046074543, -0.0033596302, 0.0054472997, -0.0338974, -0.0041198754, -0.051638987, -0.010516797, -0.0013898545, 0.025756834, -0.027647143, 0.01751096, 0.0040743006, 0.017228596, 0.041711632, 0.002354791, -0.009751058, -0.013770569, 0.035232715, 0.023409935, -0.03652178, -0.0030414755, -0.047194887, 0.004529896, -0.042122014, -0.01235932, 0.027848661]

准备好 embedding 模型后,还需要选一个向量数据库来存储向量。Langchain 官方文档里列出了四个最常用的:Chroma、Pinecone、FAISS 和 Lance。这里以 Chroma 为例展示具体用法。

先按官方文档要求安装 Chroma 库。

pip install langchain-chroma

安装成功后,设置向量数据库的存放路径,然后开始创建。

from langchain_chroma import Chroma
# 存放文件路径设置
persist_directory = r'D:langchain'

# 创建向量数据库
vectordb = Chroma.from_documents(
    documents=splits,
    embedding=embeddings,
    persist_directory=persist_directory
)
print(vectordb._collection.count())

如果最后也打印出18,说明向量数据库创建成功。在存放路径下会看到一个名为 chroma.sqlite3 的文件和一个内容文件夹,这就是我们构建的向量数据库。

试着对数据库做一次检索,看看能不能找到最相关的块。

# 检索
question = "图像识别"
docs = vectordb.similarity_search(question,k=3)
print(len(docs))
print(docs[0].page_content)

结果里包含了计算机视觉的相关章节,说明成功找到了相关内容。

13. 计算机视觉
13.1. 图像增广
13.2. 微调
13.3. 目标检测和边界框
13.4. 锚框
13.5. 多尺度目标检测
13.6. 目标检测数据集
13.7. 单发多框检测(SSD)
13.8. 区域卷积神经网络(R-CNN)系列
13.9. 语义分割和数据集
13.10. 转置卷积
13.11. 全卷积网络
13.12. 风格迁移
13.13. 实战 Kaggle 比赛:图像分类 (CIFAR-10)
13.14. 实战Kaggle比赛:狗的品种识别(ImageNet Dogs)

本节课完整的代码如下:

from langchain_community.document_loaders import WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import BaichuanTextEmbeddings
from langchain_chroma import Chroma
# 文件导入
loader = WebBaseLoader("https://zh.d2l.ai/")
docs = loader.load()

# 文本切分
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size = 1500,
    chunk_overlap = 150
)
splits = text_splitter.split_documents(docs)
print(len(splits))

# 文本嵌入
embeddings = BaichuanTextEmbeddings(baichuan_api_key="sk-83842453061e34d80b392edba11f62fe")

# 测试
# text_1 = "今天天气不错"

# query_result = embeddings.embed_query(text_1)
# print(query_result)

# 路径设置
persist_directory = r'D:langchain'

# 向量库创建
vectordb = Chroma.from_documents(
    documents=splits,
    embedding=embeddings,
    persist_directory=persist_directory
)
print(vectordb._collection.count())

# 检索
question = "图像识别"
docs = vectordb.similarity_search(question,k=3)
print(len(docs))
print(docs[0].page_content)

结语

以上就是本节课的核心内容:我们梳理了向量数据库和嵌入的概念,并通过 Langchain 与百川 API 的结合,成功搭建了一个向量数据库。这样一来,我们就能在高效管理语义信息的基础上,为后续的大模型对话系统提供强有力的支撑。

热点追踪提示词
你是一名 AI 行业编辑,请围绕下面这条热点输出一份资讯解读:
热点:吴恩达DeepLearning.AI大模型RAG第三部分:向量数据库与嵌入要求:
1. 先用一句话解释这条热点在讲什么
2. 再总结它为什么重要
3. 说明会影响哪些 AI 产品或内容方向
4. 最后给出 3 个适合资讯站使用的标题
来源:https://www.53ai.com/news/RAG/2025010485673.html
ai 人工智能

游乐网为非赢利性网站,所展示的游戏/软件/文章内容均来自于互联网或第三方用户上传分享,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系youleyoucom@outlook.com。

相关热点
AI热点2026-07-15 23:00
Sora国风短片提示词细节混乱的解决方法

Sora生成国风短片时细节错乱源于提示词缺乏物理结构、材质逻辑和视觉锚点的显性约束。通过锁定人物服饰结构、控制场景材质逻辑、统一风格动态节奏,并对易错部位做负向限制,可有效稳定画面。提示词越精确,AI越不易跑偏。

AI热点2026-07-15 23:00
Devin AI重复工作处理自动化:日常高频任务提效汇总

DevinAI是面向高频、规则半明确日常工程化任务的自主执行引擎,支持自然语言配置定时任务、多源数据联合分析、模板复用及异常自动恢复,可将重复性琐事彻底自动化闭环,显著提升效率。

AI热点2026-07-15 23:00
PhysForge框架:让静态3D模型变为可交互对象

PhysForge由香港大学与腾讯混元等机构提出,仅需单张输入图像即可生成具备部件结构、物理属性、功能语义与运动学参数的可交互3D资产,直接用于机器人仿真与虚拟世界,相关工作已被ICML2026接收。

AI热点2026-07-15 23:00
ACL 2026美团论文精选 能力评测到推理优化构建生成新范式

美团6篇论文被计算语言学顶级会议ACL2026收录,研究方向覆盖大模型评测、复杂流程推理、竞赛级数学思维优化、强化学习及生成式推荐,旨在提升推理能力并探索AI在本地生活服务中的新范式。

延伸阅读