 在 TensorFlow.org 上查看
|
 在 Google Colab 中运行
|
 在 GitHub 上查看
|
 下载笔记本
|
本教程介绍了词嵌入（word embeddings）。你将使用简单的 Keras 模型为一个情感分类任务训练自己的词嵌入，然后将它们可视化在 Embedding Projector（嵌入投影仪）中（如下图所示）。
将文本表示为数字
机器学习模型以向量（数字数组）作为输入。在处理文本时，首要任务是在将其馈送到模型之前，想出一种将字符串转换为数字（即“向量化”文本）的策略。本节将介绍三种实现这一目标的策略。
独热编码 (One-hot encodings)
最初的想法是，你可能会对词汇表中的每个单词进行“独热”（one-hot）编码。以句子“The cat sat on the mat”为例。该句子的词汇表（或唯一单词）为 (cat, mat, on, sat, the)。为了表示每个单词，你将创建一个长度等于词汇表大小的零向量，然后在对应于该单词的索引处放置一个 1。这种方法如下图所示。
要创建一个包含该句子编码的向量，你可以连接每个单词的独热向量。
用唯一数字对每个单词进行编码
你可能尝试的第二种方法是使用唯一数字来编码每个单词。继续上面的例子，你可以将 1 分配给 "cat"，2 分配给 "mat"，依此类推。然后，你可以将句子“The cat sat on the mat”编码为密集向量，如 [5, 1, 4, 3, 5, 2]。这种方法效率很高。你不再使用稀疏向量，而是使用密集向量（所有元素都已填充）。
然而，这种方法有两个缺点：
整数编码是任意的（它不能捕获单词之间的任何关系）。
整数编码对于模型来说可能难以解释。例如，线性分类器为每个特征学习单个权重。由于任意两个单词的相似性与它们编码的相似性之间没有关系，这种特征-权重组合是没有意义的。
词嵌入
词嵌入为我们提供了一种使用高效密集表示的方法，其中相似的单词具有相似的编码。重要的是，你无需手动指定此编码。嵌入是一个浮点值的密集向量（向量的长度是你指定的参数）。与其手动指定嵌入的值，不如将它们作为可训练参数（模型在训练过程中学习到的权重，就像模型学习密集层权重的方式一样）。通常，对于小数据集，词嵌入为 8 维；在处理大数据集时，可高达 1024 维。更高维的嵌入可以捕获单词之间细粒度的关系，但需要更多的数据来学习。
上图是词嵌入的示意图。每个单词被表示为 4 维的浮点值向量。看待嵌入的另一种方式是将其视为“查找表”。一旦这些权重被学习出来，你就可以通过在表中查找其对应的密集向量来编码每个单词。
设置
import io
import os
import re
import shutil
import string
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Embedding, GlobalAveragePooling1D
from tensorflow.keras.layers import TextVectorization
2024-07-19 12:44:24.531008: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-07-19 12:44:24.551991: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-07-19 12:44:24.558387: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
下载 IMDb 数据集
你将在本教程中使用 大型电影评论数据集。你将在此数据集上训练一个情感分类模型，并在此过程中从零开始学习嵌入。要阅读更多关于从零开始加载数据集的内容，请参阅加载文本教程。
使用 Keras 文件实用程序下载数据集，并查看目录。
url = "https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz"
dataset = tf.keras.utils.get_file("aclImdb_v1.tar.gz", url,
untar=True, cache_dir='.',
cache_subdir='')
dataset_dir = os.path.join(os.path.dirname(dataset), 'aclImdb')
os.listdir(dataset_dir)
Downloading data from https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz 84125825/84125825 ━━━━━━━━━━━━━━━━━━━━ 7s 0us/step ['imdb.vocab', 'README', 'imdbEr.txt', 'test', 'train']
查看 train/ 目录。它有 pos 和 neg 文件夹，其中包含标记为正面和负面的电影评论。你将使用 pos 和 neg 文件夹中的评论来训练一个二元分类模型。
train_dir = os.path.join(dataset_dir, 'train')
os.listdir(train_dir)
['labeledBow.feat', 'pos', 'unsupBow.feat', 'unsup', 'urls_unsup.txt', 'neg', 'urls_pos.txt', 'urls_neg.txt']
train 目录中还有其他文件夹，在创建训练数据集之前应将其删除。
remove_dir = os.path.join(train_dir, 'unsup')
shutil.rmtree(remove_dir)
接下来，使用 tf.keras.utils.text_dataset_from_directory 创建一个 tf.data.Dataset。您可以阅读这篇文本分类教程，了解更多关于使用此工具的信息。
使用 train 目录创建训练数据集和验证数据集，验证集占比 20%。
batch_size = 1024
seed = 123
train_ds = tf.keras.utils.text_dataset_from_directory(
'aclImdb/train', batch_size=batch_size, validation_split=0.2,
subset='training', seed=seed)
val_ds = tf.keras.utils.text_dataset_from_directory(
'aclImdb/train', batch_size=batch_size, validation_split=0.2,
subset='validation', seed=seed)
Found 25000 files belonging to 2 classes. Using 20000 files for training. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1721393095.413443 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.417346 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.421095 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.424822 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.436454 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.440093 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.443563 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.446951 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.450497 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.454040 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.457469 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393095.460790 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.665280 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.667438 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.669455 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.671425 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.673431 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.675386 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.677332 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.679230 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.681131 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.683097 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.684987 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.686875 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.725532 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.727543 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.729506 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.731434 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.733464 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.735421 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.737350 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.739265 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.741169 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.743546 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.745852 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 I0000 00:00:1721393096.748170 36419 cuda_executor.cc:1015] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 Found 25000 files belonging to 2 classes. Using 5000 files for validation.
查看训练数据集中的几条电影评论及其标签 (1: 正面, 0: 负面)。
for text_batch, label_batch in train_ds.take(1):
for i in range(5):
print(label_batch[i].numpy(), text_batch.numpy()[i])
0 b"Oh My God! Please, for the love of all that is holy, Do Not Watch This Movie! It it 82 minutes of my life I will never get back. Sure, I could have stopped watching half way through. But I thought it might get better. It Didn't. Anyone who actually enjoyed this movie is one seriously sick and twisted individual. No wonder us Australians/New Zealanders have a terrible reputation when it comes to making movies. Everything about this movie is horrible, from the acting to the editing. I don't even normally write reviews on here, but in this case I'll make an exception. I only wish someone had of warned me before I hired this catastrophe" 1 b'This movie is SOOOO funny!!! The acting is WONDERFUL, the Ramones are sexy, the jokes are subtle, and the plot is just what every high schooler dreams of doing to his/her school. I absolutely loved the soundtrack as well as the carefully placed cynicism. If you like monty python, You will love this film. This movie is a tad bit "grease"esk (without all the annoying songs). The songs that are sung are likable; you might even find yourself singing these songs once the movie is through. This musical ranks number two in musicals to me (second next to the blues brothers). But please, do not think of it as a musical per say; seeing as how the songs are so likable, it is hard to tell a carefully choreographed scene is taking place. I think of this movie as more of a comedy with undertones of romance. You will be reminded of what it was like to be a rebellious teenager; needless to say, you will be reminiscing of your old high school days after seeing this film. Highly recommended for both the family (since it is a very youthful but also for adults since there are many jokes that are funnier with age and experience.' 0 b"Alex D. Linz replaces Macaulay Culkin as the central figure in the third movie in the Home Alone empire. Four industrial spies acquire a missile guidance system computer chip and smuggle it through an airport inside a remote controlled toy car. Because of baggage confusion, grouchy Mrs. Hess (Marian Seldes) gets the car. She gives it to her neighbor, Alex (Linz), just before the spies turn up. The spies rent a house in order to burglarize each house in the neighborhood until they locate the car. Home alone with the chicken pox, Alex calls 911 each time he spots a theft in progress, but the spies always manage to elude the police while Alex is accused of making prank calls. The spies finally turn their attentions toward Alex, unaware that he has rigged devices to cleverly booby-trap his entire house. Home Alone 3 wasn't horrible, but probably shouldn't have been made, you can't just replace Macauley Culkin, Joe Pesci, or Daniel Stern. Home Alone 3 had some funny parts, but I don't like when characters are changed in a movie series, view at own risk." 0 b"There's a good movie lurking here, but this isn't it. The basic idea is good: to explore the moral issues that would face a group of young survivors of the apocalypse. But the logic is so muddled that it's impossible to get involved.<br /><br />For example, our four heroes are (understandably) paranoid about catching the mysterious airborne contagion that's wiped out virtually all of mankind. Yet they wear surgical masks some times, not others. Some times they're fanatical about wiping down with bleach any area touched by an infected person. Other times, they seem completely unconcerned.<br /><br />Worse, after apparently surviving some weeks or months in this new kill-or-be-killed world, these people constantly behave like total newbs. They don't bother accumulating proper equipment, or food. They're forever running out of fuel in the middle of nowhere. They don't take elementary precautions when meeting strangers. And after wading through the rotting corpses of the entire human race, they're as squeamish as sheltered debutantes. You have to constantly wonder how they could have survived this long... and even if they did, why anyone would want to make a movie about them.<br /><br />So when these dweebs stop to agonize over the moral dimensions of their actions, it's impossible to take their soul-searching seriously. Their actions would first have to make some kind of minimal sense.<br /><br />On top of all this, we must contend with the dubious acting abilities of Chris Pine. His portrayal of an arrogant young James T Kirk might have seemed shrewd, when viewed in isolation. But in Carriers he plays on exactly that same note: arrogant and boneheaded. It's impossible not to suspect that this constitutes his entire dramatic range.<br /><br />On the positive side, the film *looks* excellent. It's got an over-sharp, saturated look that really suits the southwestern US locale. But that can't save the truly feeble writing nor the paper-thin (and annoying) characters. Even if you're a fan of the end-of-the-world genre, you should save yourself the agony of watching Carriers." 0 b'I saw this movie at an actual movie theater (probably the \\(2.00 one) with my cousin and uncle. We were around 11 and 12, I guess, and really into scary movies. I remember being so excited to see it because my cool uncle let us pick the movie (and we probably never got to do that again!) and sooo disappointed afterwards!! Just boring and not scary. The only redeeming thing I can remember was Corky Pigeon from Silver Spoons, and that wasn\'t all that great, just someone I recognized. I\'ve seen bad movies before and this one has always stuck out in my mind as the worst. This was from what I can recall, one of the most boring, non-scary, waste of our collective \\)6, and a waste of film. I have read some of the reviews that say it is worth a watch and I say, "Too each his own", but I wouldn\'t even bother. Not even so bad it\'s good.'
配置数据集以获得高性能
在加载数据时，你应该使用这两种重要的方法，以确保 I/O 不会发生阻塞。
.cache() 在数据从磁盘加载后将其保留在内存中。这将确保在训练模型时数据集不会成为瓶颈。如果你的数据集太大而无法放入内存，你也可以使用此方法创建高性能的磁盘缓存，这比读取许多小文件更有效。
.prefetch() 在训练时重叠数据预处理和模型执行。
你可以在数据性能指南中了解有关这两种方法的更多信息，以及如何将数据缓存到磁盘。
AUTOTUNE = tf.data.AUTOTUNE
train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
使用 Embedding 层
Keras 使使用词嵌入变得简单。查看 Embedding 层。
Embedding 层可以被理解为一个查找表，它将整数索引（代表特定单词）映射到密集向量（它们的嵌入）。嵌入的维度（或宽度）是一个你可以通过实验来确定的参数，看看什么对你的问题效果最好，这和你尝试 Dense 层中的神经元数量的方式非常相似。
# Embed a 1,000 word vocabulary into 5 dimensions.
embedding_layer = tf.keras.layers.Embedding(1000, 5)
当你创建一个 Embedding 层时，嵌入的权重是随机初始化的（就像任何其他层一样）。在训练过程中，它们通过反向传播逐渐调整。一旦训练完成，学习到的词嵌入将粗略地编码单词之间的相似性（因为它们是针对你的模型所训练的特定问题而学习的）。
如果你将一个整数传递给嵌入层，结果将用嵌入表中的向量替换每个整数。
result = embedding_layer(tf.constant([1, 2, 3]))
result.numpy()
array([[ 0.03431678, 0.00738569, 0.00151139, 0.04062479, 0.00129622],
[ 0.02842152, -0.01932666, 0.0173661 , 0.0037834 , 0.0244248 ],
[ 0.04234162, -0.00971337, 0.04991848, 0.04969274, 0.02653965]],
dtype=float32)
对于文本或序列问题，Embedding 层接受一个形状为 (samples, sequence_length) 的 2D 整数张量，其中每个条目都是一个整数序列。它可以嵌入可变长度的序列。你可以将形状为 (32, 10)（32 个长度为 10 的序列的批次）或 (64, 15)（64 个长度为 15 的序列的批次）的批次馈送到上面的嵌入层。
返回的张量比输入多一个轴，嵌入向量沿新的最后一个轴对齐。传入一个 (2, 3) 的输入批次，输出则为 (2, 3, N)。
result = embedding_layer(tf.constant([[0, 1, 2], [3, 4, 5]]))
result.shape
TensorShape([2, 3, 5])
当给定一个序列批次作为输入时，嵌入层返回一个形状为 (samples, sequence_length, embedding_dimensionality) 的 3D 浮点张量。要将这种可变长度序列转换为固定表示，有多种标准方法。你可以在将其传递给 Dense 层之前使用 RNN、注意力机制或池化层。本教程使用池化，因为它是最简单的。 使用 RNN 进行文本分类教程是下一步的好选择。
文本预处理
接下来，定义情感分类模型所需的数据集预处理步骤。初始化一个带有所需参数的 TextVectorization 层以向量化电影评论。你可以在文本分类教程中了解更多关于使用此层的信息。
# Create a custom standardization function to strip HTML break tags '<br />'.
def custom_standardization(input_data):
lowercase = tf.strings.lower(input_data)
stripped_html = tf.strings.regex_replace(lowercase, '<br />', ' ')
return tf.strings.regex_replace(stripped_html,
'[%s]' % re.escape(string.punctuation), '')
# Vocabulary size and number of words in a sequence.
vocab_size = 10000
sequence_length = 100
# Use the text vectorization layer to normalize, split, and map strings to
# integers. Note that the layer uses the custom standardization defined above.
# Set maximum_sequence length as all samples are not of the same length.
vectorize_layer = TextVectorization(
standardize=custom_standardization,
max_tokens=vocab_size,
output_mode='int',
output_sequence_length=sequence_length)
# Make a text-only dataset (no labels) and call adapt to build the vocabulary.
text_ds = train_ds.map(lambda x, y: x)
vectorize_layer.adapt(text_ds)
创建分类模型
使用 Keras Sequential API 定义情感分类模型。在这种情况下，它是一个“连续词袋”（Continuous bag of words）风格的模型。
TextVectorization层将字符串转换为词汇表索引。你已经初始化了vectorize_layer作为 TextVectorization 层，并通过在text_ds上调用adapt构建了其词汇表。现在，vectorize_layer 可以作为你的端到端分类模型的第一层，将转换后的字符串馈送到 Embedding 层中。Embedding层获取整数编码的词汇表，并查找每个单词索引的嵌入向量。这些向量在模型训练时被学习。向量为输出数组增加了一个维度。产生的维度是：(batch, sequence, embedding)。GlobalAveragePooling1D层通过对序列维度进行平均，为每个示例返回一个固定长度的输出向量。这允许模型以最简单的方式处理可变长度的输入。固定长度的输出向量通过一个具有 16 个隐藏单元的全连接 (
Dense) 层。最后一层是全连接层，带有一个输出节点。
embedding_dim=16
model = Sequential([
vectorize_layer,
Embedding(vocab_size, embedding_dim, name="embedding"),
GlobalAveragePooling1D(),
Dense(16, activation='relu'),
Dense(1)
])
编译并训练模型
你将使用 TensorBoard 来可视化包括损失和准确率在内的指标。创建一个 tf.keras.callbacks.TensorBoard。
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir="logs")
使用 Adam 优化器和 BinaryCrossentropy 损失函数编译并训练模型。
model.compile(optimizer='adam',
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=['accuracy'])
model.fit(
train_ds,
validation_data=val_ds,
epochs=15,
callbacks=[tensorboard_callback])
Epoch 1/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 3s 69ms/step - accuracy: 0.5037 - loss: 0.6918 - val_accuracy: 0.4886 - val_loss: 0.6848 Epoch 2/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 52ms/step - accuracy: 0.5038 - loss: 0.6823 - val_accuracy: 0.4886 - val_loss: 0.6715 Epoch 3/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.5042 - loss: 0.6676 - val_accuracy: 0.4892 - val_loss: 0.6520 Epoch 4/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.5093 - loss: 0.6453 - val_accuracy: 0.5242 - val_loss: 0.6256 Epoch 5/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.5647 - loss: 0.6154 - val_accuracy: 0.6028 - val_loss: 0.5936 Epoch 6/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.6464 - loss: 0.5801 - val_accuracy: 0.6638 - val_loss: 0.5594 Epoch 7/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.7129 - loss: 0.5423 - val_accuracy: 0.7090 - val_loss: 0.5263 Epoch 8/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.7518 - loss: 0.5057 - val_accuracy: 0.7330 - val_loss: 0.4966 Epoch 9/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.7789 - loss: 0.4725 - val_accuracy: 0.7570 - val_loss: 0.4714 Epoch 10/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.7991 - loss: 0.4437 - val_accuracy: 0.7744 - val_loss: 0.4507 Epoch 11/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.8145 - loss: 0.4190 - val_accuracy: 0.7862 - val_loss: 0.4338 Epoch 12/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.8254 - loss: 0.3978 - val_accuracy: 0.7956 - val_loss: 0.4202 Epoch 13/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.8345 - loss: 0.3794 - val_accuracy: 0.8032 - val_loss: 0.4092 Epoch 14/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 51ms/step - accuracy: 0.8419 - loss: 0.3634 - val_accuracy: 0.8092 - val_loss: 0.4003 Epoch 15/15 20/20 ━━━━━━━━━━━━━━━━━━━━ 1s 52ms/step - accuracy: 0.8480 - loss: 0.3493 - val_accuracy: 0.8146 - val_loss: 0.3931 <keras.src.callbacks.history.History at 0x7fab81be6f40>
通过这种方法，模型的验证准确率达到约 78%（注意，由于训练准确率较高，模型存在过拟合）。
你可以查看模型摘要以了解有关模型每一层的更多信息。
model.summary()
在 TensorBoard 中可视化模型指标。
#docs_infra: no_execute
%load_ext tensorboard
%tensorboard --logdir logs
检索训练好的词嵌入并将它们保存到磁盘
接下来，检索训练过程中学习到的词嵌入。嵌入是模型中 Embedding 层的权重。权重矩阵的形状为 (vocab_size, embedding_dimension)。
使用 get_layer() 和 get_weights() 从模型中获取权重。get_vocabulary() 函数提供词汇表，以便构建每行一个标记的元数据文件。
weights = model.get_layer('embedding').get_weights()[0]
vocab = vectorize_layer.get_vocabulary()
将权重写入磁盘。要使用 Embedding Projector，你将上传两个制表符分隔格式的文件：一个向量文件（包含嵌入）和一个元数据文件（包含单词）。
out_v = io.open('vectors.tsv', 'w', encoding='utf-8')
out_m = io.open('metadata.tsv', 'w', encoding='utf-8')
for index, word in enumerate(vocab):
if index == 0:
continue # skip 0, it's padding.
vec = weights[index]
out_v.write('\t'.join([str(x) for x in vec]) + "\n")
out_m.write(word + "\n")
out_v.close()
out_m.close()
如果你正在 Colaboratory 中运行此教程，可以使用以下代码片段将这些文件下载到你的本地机器（或者使用文件浏览器，View -> Table of contents -> File browser）。
try:
from google.colab import files
files.download('vectors.tsv')
files.download('metadata.tsv')
except Exception:
pass
可视化嵌入
要可视化嵌入，请将它们上传到嵌入投影仪。
打开 Embedding Projector（这也可以在本地 TensorBoard 实例中运行）。
点击“Load data”。
上传你上面创建的两个文件：
vecs.tsv和meta.tsv。
你训练的嵌入现在将显示出来。你可以搜索单词以找到它们的最近邻居。例如，尝试搜索“beautiful”。你可能会看到像“wonderful”这样的邻居。
后续步骤
本教程向你展示了如何在小型数据集上从零开始训练和可视化词嵌入。
要使用 Word2Vec 算法训练词嵌入，请尝试 Word2Vec 教程。
要了解更多关于高级文本处理的信息，请阅读用于语言理解的 Transformer 模型。