在 TensorFlow.org 上查看
在 Google Colab 中运行
在 GitHub 上查看源代码
下载笔记本在本基于笔记本的教程中，我们将创建一个 TFX 管道，该管道将创建一个简单的分类模型，并分析其在多个运行中的性能。本笔记本基于我们在 简单 TFX Pipeline 教程 中构建的 TFX 管道。如果您尚未阅读该教程，请在继续本笔记本之前阅读它。
当您调整模型或使用新数据集训练模型时，您需要检查模型是否有所改进或变差。仅仅检查诸如准确率之类的顶级指标可能还不够。每个训练后的模型都应在推送到生产环境之前进行评估。
我们将向之前教程中创建的管道添加一个 Evaluator 组件。Evaluator 组件对您的模型执行深度分析，并将新模型与基线模型进行比较，以确定它们是否“足够好”。它使用 TensorFlow Model Analysis 库实现。
请参阅 了解 TFX 管道，以详细了解 TFX 中的各种概念。
设置
设置过程与之前教程相同。
首先，我们需要安装 TFX Python 包并下载我们将用于模型的数据集。
升级 Pip
为了避免在本地运行时升级系统中的 Pip，请检查我们是否在 Colab 中运行。本地系统当然可以单独升级。
try:
import colab
!pip install --upgrade pip
except:
pass
安装 TFX
pip install -U tfx
您是否重新启动了运行时？
如果您使用的是 Google Colab，则第一次运行上面的单元格时，必须通过点击上面的“重新启动运行时”按钮或使用“运行时 > 重新启动运行时...”菜单来重新启动运行时。这是因为 Colab 加载包的方式。
检查 TensorFlow 和 TFX 版本。
import tensorflow as tf
print('TensorFlow version: {}'.format(tf.__version__))
from tfx import v1 as tfx
print('TFX version: {}'.format(tfx.__version__))
2024-05-08 09:12:22.461208: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-05-08 09:12:22.461259: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-05-08 09:12:22.462861: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered TensorFlow version: 2.15.1 TFX version: 1.15.0
设置变量
有一些变量用于定义管道。您可以根据需要自定义这些变量。默认情况下，管道的所有输出都将在当前目录下生成。
import os
PIPELINE_NAME = "penguin-tfma"
# Output directory to store artifacts generated from the pipeline.
PIPELINE_ROOT = os.path.join('pipelines', PIPELINE_NAME)
# Path to a SQLite DB file to use as an MLMD storage.
METADATA_PATH = os.path.join('metadata', PIPELINE_NAME, 'metadata.db')
# Output directory where created models from the pipeline will be exported.
SERVING_MODEL_DIR = os.path.join('serving_model', PIPELINE_NAME)
from absl import logging
logging.set_verbosity(logging.INFO) # Set default logging level.
准备示例数据
我们将使用相同的数据集 Palmer Penguins 数据集。
此数据集中有四个数值特征，这些特征已标准化为范围 [0,1]。我们将构建一个分类模型，该模型预测企鹅的 species。
由于 TFX ExampleGen 从目录中读取输入，因此我们需要创建一个目录并将数据集复制到其中。
import urllib.request
import tempfile
DATA_ROOT = tempfile.mkdtemp(prefix='tfx-data') # Create a temporary directory.
_data_url = 'https://raw.githubusercontent.com/tensorflow/tfx/master/tfx/examples/penguin/data/labelled/penguins_processed.csv'
_data_filepath = os.path.join(DATA_ROOT, "data.csv")
urllib.request.urlretrieve(_data_url, _data_filepath)
('/tmpfs/tmp/tfx-datakcma5ryu/data.csv',
<http.client.HTTPMessage at 0x7fc5d80acb80>)
创建管道
我们将向我们在 简单 TFX Pipeline 教程 中创建的管道添加一个 Evaluator 组件。
Evaluator 组件需要来自 ExampleGen 组件的输入数据，以及来自 Trainer 组件的模型，以及一个 tfma.EvalConfig 对象。我们可以选择提供一个基线模型，该模型可用于将指标与新训练的模型进行比较。
评估器会创建两种输出工件，ModelEvaluation 和 ModelBlessing。ModelEvaluation 包含详细的评估结果，可以使用 TFMA 库对其进行进一步调查和可视化。ModelBlessing 包含一个布尔结果，指示模型是否通过了给定的标准，可以在后面的组件（如 Pusher）中用作信号。
编写模型训练代码
我们将使用与 简单 TFX Pipeline 教程 中相同的模型代码。
_trainer_module_file = 'penguin_trainer.py'
%%writefile {_trainer_module_file}
# Copied from https://tensorflowcn.cn/tfx/tutorials/tfx/penguin_simple
from typing import List
from absl import logging
import tensorflow as tf
from tensorflow import keras
from tensorflow_transform.tf_metadata import schema_utils
from tfx.components.trainer.executor import TrainerFnArgs
from tfx.components.trainer.fn_args_utils import DataAccessor
from tfx_bsl.tfxio import dataset_options
from tensorflow_metadata.proto.v0 import schema_pb2
_FEATURE_KEYS = [
'culmen_length_mm', 'culmen_depth_mm', 'flipper_length_mm', 'body_mass_g'
]
_LABEL_KEY = 'species'
_TRAIN_BATCH_SIZE = 20
_EVAL_BATCH_SIZE = 10
# Since we're not generating or creating a schema, we will instead create
# a feature spec. Since there are a fairly small number of features this is
# manageable for this dataset.
_FEATURE_SPEC = {
**{
feature: tf.io.FixedLenFeature(shape=[1], dtype=tf.float32)
for feature in _FEATURE_KEYS
},
_LABEL_KEY: tf.io.FixedLenFeature(shape=[1], dtype=tf.int64)
}
def _input_fn(file_pattern: List[str],
data_accessor: DataAccessor,
schema: schema_pb2.Schema,
batch_size: int = 200) -> tf.data.Dataset:
"""Generates features and label for training.
Args:
file_pattern: List of paths or patterns of input tfrecord files.
data_accessor: DataAccessor for converting input to RecordBatch.
schema: schema of the input data.
batch_size: representing the number of consecutive elements of returned
dataset to combine in a single batch
Returns:
A dataset that contains (features, indices) tuple where features is a
dictionary of Tensors, and indices is a single Tensor of label indices.
"""
return data_accessor.tf_dataset_factory(
file_pattern,
dataset_options.TensorFlowDatasetOptions(
batch_size=batch_size, label_key=_LABEL_KEY),
schema=schema).repeat()
def _build_keras_model() -> tf.keras.Model:
"""Creates a DNN Keras model for classifying penguin data.
Returns:
A Keras Model.
"""
# The model below is built with Functional API, please refer to
# https://tensorflowcn.cn/guide/keras/overview for all API options.
inputs = [keras.layers.Input(shape=(1,), name=f) for f in _FEATURE_KEYS]
d = keras.layers.concatenate(inputs)
for _ in range(2):
d = keras.layers.Dense(8, activation='relu')(d)
outputs = keras.layers.Dense(3)(d)
model = keras.Model(inputs=inputs, outputs=outputs)
model.compile(
optimizer=keras.optimizers.Adam(1e-2),
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=[keras.metrics.SparseCategoricalAccuracy()])
model.summary(print_fn=logging.info)
return model
# TFX Trainer will call this function.
def run_fn(fn_args: TrainerFnArgs):
"""Train the model based on given args.
Args:
fn_args: Holds args used to train the model as name/value pairs.
"""
# This schema is usually either an output of SchemaGen or a manually-curated
# version provided by pipeline author. A schema can also derived from TFT
# graph if a Transform component is used. In the case when either is missing,
# `schema_from_feature_spec` could be used to generate schema from very simple
# feature_spec, but the schema returned would be very primitive.
schema = schema_utils.schema_from_feature_spec(_FEATURE_SPEC)
train_dataset = _input_fn(
fn_args.train_files,
fn_args.data_accessor,
schema,
batch_size=_TRAIN_BATCH_SIZE)
eval_dataset = _input_fn(
fn_args.eval_files,
fn_args.data_accessor,
schema,
batch_size=_EVAL_BATCH_SIZE)
model = _build_keras_model()
model.fit(
train_dataset,
steps_per_epoch=fn_args.train_steps,
validation_data=eval_dataset,
validation_steps=fn_args.eval_steps)
# The result of the training should be saved in `fn_args.serving_model_dir`
# directory.
model.save(fn_args.serving_model_dir, save_format='tf')
Writing penguin_trainer.py
编写管道定义
我们将定义一个函数来创建一个 TFX 管道。除了上面提到的 Evaluator 组件之外，我们还将添加一个名为 Resolver 的节点。为了检查新模型是否比之前的模型更好，我们需要将其与之前发布的模型（称为基线模型）进行比较。 ML Metadata(MLMD) 会跟踪管道的所有先前工件，Resolver 可以使用名为 LatestBlessedModelStrategy 的策略类从 MLMD 中找到最新的已批准模型（通过评估器成功通过的模型）。
import tensorflow_model_analysis as tfma
def _create_pipeline(pipeline_name: str, pipeline_root: str, data_root: str,
module_file: str, serving_model_dir: str,
metadata_path: str) -> tfx.dsl.Pipeline:
"""Creates a three component penguin pipeline with TFX."""
# Brings data into the pipeline.
example_gen = tfx.components.CsvExampleGen(input_base=data_root)
# Uses user-provided Python function that trains a model.
trainer = tfx.components.Trainer(
module_file=module_file,
examples=example_gen.outputs['examples'],
train_args=tfx.proto.TrainArgs(num_steps=100),
eval_args=tfx.proto.EvalArgs(num_steps=5))
# NEW: Get the latest blessed model for Evaluator.
model_resolver = tfx.dsl.Resolver(
strategy_class=tfx.dsl.experimental.LatestBlessedModelStrategy,
model=tfx.dsl.Channel(type=tfx.types.standard_artifacts.Model),
model_blessing=tfx.dsl.Channel(
type=tfx.types.standard_artifacts.ModelBlessing)).with_id(
'latest_blessed_model_resolver')
# NEW: Uses TFMA to compute evaluation statistics over features of a model and
# perform quality validation of a candidate model (compared to a baseline).
eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec(label_key='species')],
slicing_specs=[
# An empty slice spec means the overall slice, i.e. the whole dataset.
tfma.SlicingSpec(),
# Calculate metrics for each penguin species.
tfma.SlicingSpec(feature_keys=['species']),
],
metrics_specs=[
tfma.MetricsSpec(per_slice_thresholds={
'sparse_categorical_accuracy':
tfma.PerSliceMetricThresholds(thresholds=[
tfma.PerSliceMetricThreshold(
slicing_specs=[tfma.SlicingSpec()],
threshold=tfma.MetricThreshold(
value_threshold=tfma.GenericValueThreshold(
lower_bound={'value': 0.6}),
# Change threshold will be ignored if there is no
# baseline model resolved from MLMD (first run).
change_threshold=tfma.GenericChangeThreshold(
direction=tfma.MetricDirection.HIGHER_IS_BETTER,
absolute={'value': -1e-10}))
)]),
})],
)
evaluator = tfx.components.Evaluator(
examples=example_gen.outputs['examples'],
model=trainer.outputs['model'],
baseline_model=model_resolver.outputs['model'],
eval_config=eval_config)
# Checks whether the model passed the validation steps and pushes the model
# to a file destination if check passed.
pusher = tfx.components.Pusher(
model=trainer.outputs['model'],
model_blessing=evaluator.outputs['blessing'], # Pass an evaluation result.
push_destination=tfx.proto.PushDestination(
filesystem=tfx.proto.PushDestination.Filesystem(
base_directory=serving_model_dir)))
components = [
example_gen,
trainer,
# Following two components were added to the pipeline.
model_resolver,
evaluator,
pusher,
]
return tfx.dsl.Pipeline(
pipeline_name=pipeline_name,
pipeline_root=pipeline_root,
metadata_connection_config=tfx.orchestration.metadata
.sqlite_metadata_connection_config(metadata_path),
components=components)
我们需要通过 eval_config 向评估器提供以下信息
- 要配置的其他指标（如果需要比模型中定义的指标更多指标）。
- 要配置的切片
- 模型验证阈值，用于验证是否包含验证
因为 SparseCategoricalAccuracy 已经包含在 model.compile() 调用中，它将自动包含在分析中。因此，我们在这里不添加任何额外的指标。 SparseCategoricalAccuracy 将用于决定模型是否足够好。
我们计算整个数据集和每个企鹅物种的指标。 SlicingSpec 指定了我们如何聚合声明的指标。
新模型需要通过两个阈值，一个是 0.6 的绝对阈值，另一个是相对于基线模型更高的相对阈值。当您第一次运行管道时， change_threshold 将被忽略，只检查 value_threshold。如果您多次运行管道， Resolver 将找到先前运行的模型，并将其用作比较的基线模型。
有关更多信息，请参阅 评估器组件指南。
运行管道
我们将使用 LocalDagRunner，就像在之前的教程中一样。
tfx.orchestration.LocalDagRunner().run(
_create_pipeline(
pipeline_name=PIPELINE_NAME,
pipeline_root=PIPELINE_ROOT,
data_root=DATA_ROOT,
module_file=_trainer_module_file,
serving_model_dir=SERVING_MODEL_DIR,
metadata_path=METADATA_PATH))
INFO:absl:Generating ephemeral wheel package for '/tmpfs/src/temp/docs/tutorials/tfx/penguin_trainer.py' (including modules: ['penguin_trainer']).
INFO:absl:User module package has hash fingerprint version 1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.
INFO:absl:Executing: ['/tmpfs/src/tf_docs_env/bin/python', '/tmpfs/tmp/tmprw4uskdx/_tfx_generated_setup.py', 'bdist_wheel', '--bdist-dir', '/tmpfs/tmp/tmp380pw4r5', '--dist-dir', '/tmpfs/tmp/tmp3ooau66m']
/tmpfs/src/tf_docs_env/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************
!!
self.initialize_options()
INFO:absl:Successfully built user code wheel distribution at 'pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl'; target user module is 'penguin_trainer'.
INFO:absl:Full user module path is 'penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl'
INFO:absl:Using deployment config:
executor_specs {
key: "CsvExampleGen"
value {
beam_executable_spec {
python_executor_spec {
class_path: "tfx.components.example_gen.csv_example_gen.executor.Executor"
}
}
}
}
executor_specs {
key: "Evaluator"
value {
beam_executable_spec {
python_executor_spec {
class_path: "tfx.components.evaluator.executor.Executor"
}
}
}
}
executor_specs {
key: "Pusher"
value {
python_class_executable_spec {
class_path: "tfx.components.pusher.executor.Executor"
}
}
}
executor_specs {
key: "Trainer"
value {
python_class_executable_spec {
class_path: "tfx.components.trainer.executor.GenericExecutor"
}
}
}
custom_driver_specs {
key: "CsvExampleGen"
value {
python_class_executable_spec {
class_path: "tfx.components.example_gen.driver.FileBasedDriver"
}
}
}
metadata_connection_config {
database_connection_config {
sqlite {
filename_uri: "metadata/penguin-tfma/metadata.db"
connection_mode: READWRITE_OPENCREATE
}
}
}
INFO:absl:Using connection config:
sqlite {
filename_uri: "metadata/penguin-tfma/metadata.db"
connection_mode: READWRITE_OPENCREATE
}
INFO:absl:Component CsvExampleGen is running.
INFO:absl:Running launcher for node_info {
type {
name: "tfx.components.example_gen.csv_example_gen.component.CsvExampleGen"
}
id: "CsvExampleGen"
}
contexts {
contexts {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
contexts {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
contexts {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.CsvExampleGen"
}
}
}
}
outputs {
outputs {
key: "examples"
value {
artifact_spec {
type {
name: "Examples"
properties {
key: "span"
value: INT
}
properties {
key: "split_names"
value: STRING
}
properties {
key: "version"
value: INT
}
base_type: DATASET
}
}
}
}
}
parameters {
parameters {
key: "input_base"
value {
field_value {
string_value: "/tmpfs/tmp/tfx-datakcma5ryu"
}
}
}
parameters {
key: "input_config"
value {
field_value {
string_value: "{\n \"splits\": [\n {\n \"name\": \"single_split\",\n \"pattern\": \"*\"\n }\n ]\n}"
}
}
}
parameters {
key: "output_config"
value {
field_value {
string_value: "{\n \"split_config\": {\n \"splits\": [\n {\n \"hash_buckets\": 2,\n \"name\": \"train\"\n },\n {\n \"hash_buckets\": 1,\n \"name\": \"eval\"\n }\n ]\n }\n}"
}
}
}
parameters {
key: "output_data_format"
value {
field_value {
int_value: 6
}
}
}
parameters {
key: "output_file_format"
value {
field_value {
int_value: 5
}
}
}
}
downstream_nodes: "Evaluator"
downstream_nodes: "Trainer"
execution_options {
caching_options {
}
}
INFO:absl:MetadataStore with DB connection initialized
running bdist_wheel
running build
running build_py
creating build
creating build/lib
copying penguin_trainer.py -> build/lib
installing to /tmpfs/tmp/tmp380pw4r5
running install
running install_lib
copying build/lib/penguin_trainer.py -> /tmpfs/tmp/tmp380pw4r5
running install_egg_info
running egg_info
creating tfx_user_code_Trainer.egg-info
writing tfx_user_code_Trainer.egg-info/PKG-INFO
writing dependency_links to tfx_user_code_Trainer.egg-info/dependency_links.txt
writing top-level names to tfx_user_code_Trainer.egg-info/top_level.txt
writing manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'
reading manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'
writing manifest file 'tfx_user_code_Trainer.egg-info/SOURCES.txt'
Copying tfx_user_code_Trainer.egg-info to /tmpfs/tmp/tmp380pw4r5/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3.9.egg-info
running install_scripts
creating /tmpfs/tmp/tmp380pw4r5/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/WHEEL
creating '/tmpfs/tmp/tmp3ooau66m/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl' and adding '/tmpfs/tmp/tmp380pw4r5' to it
adding 'penguin_trainer.py'
adding 'tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/METADATA'
adding 'tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/WHEEL'
adding 'tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/top_level.txt'
adding 'tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703.dist-info/RECORD'
removing /tmpfs/tmp/tmp380pw4r5
INFO:absl:[CsvExampleGen] Resolved inputs: ({},)
INFO:absl:select span and version = (0, None)
INFO:absl:latest span and version = (0, None)
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Going to run a new execution 1
INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=1, input_dict={}, output_dict=defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
custom_properties {
key: "input_fingerprint"
value {
string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548"
}
}
custom_properties {
key: "span"
value {
int_value: 0
}
}
, artifact_type: name: "Examples"
properties {
key: "span"
value: INT
}
properties {
key: "split_names"
value: STRING
}
properties {
key: "version"
value: INT
}
base_type: DATASET
)]}), exec_properties={'output_file_format': 5, 'output_config': '{\n "split_config": {\n "splits": [\n {\n "hash_buckets": 2,\n "name": "train"\n },\n {\n "hash_buckets": 1,\n "name": "eval"\n }\n ]\n }\n}', 'input_base': '/tmpfs/tmp/tfx-datakcma5ryu', 'input_config': '{\n "splits": [\n {\n "name": "single_split",\n "pattern": "*"\n }\n ]\n}', 'output_data_format': 6, 'span': 0, 'version': None, 'input_fingerprint': 'split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548'}, execution_output_uri='pipelines/penguin-tfma/CsvExampleGen/.system/executor_execution/1/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/CsvExampleGen/.system/stateful_working_dir/71dcf2a2-5c57-4eda-b836-d76ab760acc0', tmp_dir='pipelines/penguin-tfma/CsvExampleGen/.system/executor_execution/1/.temp/', pipeline_node=node_info {
type {
name: "tfx.components.example_gen.csv_example_gen.component.CsvExampleGen"
}
id: "CsvExampleGen"
}
contexts {
contexts {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
contexts {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
contexts {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.CsvExampleGen"
}
}
}
}
outputs {
outputs {
key: "examples"
value {
artifact_spec {
type {
name: "Examples"
properties {
key: "span"
value: INT
}
properties {
key: "split_names"
value: STRING
}
properties {
key: "version"
value: INT
}
base_type: DATASET
}
}
}
}
}
parameters {
parameters {
key: "input_base"
value {
field_value {
string_value: "/tmpfs/tmp/tfx-datakcma5ryu"
}
}
}
parameters {
key: "input_config"
value {
field_value {
string_value: "{\n \"splits\": [\n {\n \"name\": \"single_split\",\n \"pattern\": \"*\"\n }\n ]\n}"
}
}
}
parameters {
key: "output_config"
value {
field_value {
string_value: "{\n \"split_config\": {\n \"splits\": [\n {\n \"hash_buckets\": 2,\n \"name\": \"train\"\n },\n {\n \"hash_buckets\": 1,\n \"name\": \"eval\"\n }\n ]\n }\n}"
}
}
}
parameters {
key: "output_data_format"
value {
field_value {
int_value: 6
}
}
}
parameters {
key: "output_file_format"
value {
field_value {
int_value: 5
}
}
}
}
downstream_nodes: "Evaluator"
downstream_nodes: "Trainer"
execution_options {
caching_options {
}
}
, pipeline_info=id: "penguin-tfma"
, pipeline_run_id='2024-05-08T09:12:28.606391', top_level_pipeline_run_id=None, frontend_url=None)
INFO:absl:Generating examples.
WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.
INFO:absl:Processing input csv data /tmpfs/tmp/tfx-datakcma5ryu/* to TFExample.
WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
INFO:absl:Examples generated.
INFO:absl:Value type <class 'NoneType'> of key version in exec_properties is not supported, going to drop it
INFO:absl:Value type <class 'list'> of key _beam_pipeline_args in exec_properties is not supported, going to drop it
INFO:absl:Cleaning up stateless execution info.
INFO:absl:Execution 1 succeeded.
INFO:absl:Cleaning up stateful execution info.
INFO:absl:Deleted stateful_working_dir pipelines/penguin-tfma/CsvExampleGen/.system/stateful_working_dir/71dcf2a2-5c57-4eda-b836-d76ab760acc0
INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'examples': [Artifact(artifact: uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
custom_properties {
key: "input_fingerprint"
value {
string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548"
}
}
custom_properties {
key: "span"
value {
int_value: 0
}
}
, artifact_type: name: "Examples"
properties {
key: "span"
value: INT
}
properties {
key: "split_names"
value: STRING
}
properties {
key: "version"
value: INT
}
base_type: DATASET
)]}) for execution 1
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Component CsvExampleGen is finished.
INFO:absl:Component latest_blessed_model_resolver is running.
INFO:absl:Running launcher for node_info {
type {
name: "tfx.dsl.components.common.resolver.Resolver"
}
id: "latest_blessed_model_resolver"
}
contexts {
contexts {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
contexts {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
contexts {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.latest_blessed_model_resolver"
}
}
}
}
inputs {
inputs {
key: "_generated_model_3"
value {
channels {
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
artifact_query {
type {
name: "Model"
base_type: MODEL
}
}
}
hidden: true
}
}
inputs {
key: "_generated_modelblessing_4"
value {
channels {
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
artifact_query {
type {
name: "ModelBlessing"
}
}
}
hidden: true
}
}
inputs {
key: "model"
value {
input_graph_ref {
graph_id: "graph_1"
key: "model"
}
}
}
inputs {
key: "model_blessing"
value {
input_graph_ref {
graph_id: "graph_1"
key: "model_blessing"
}
}
}
input_graphs {
key: "graph_1"
value {
nodes {
key: "dict_2"
value {
output_data_type: ARTIFACT_MULTIMAP
dict_node {
node_ids {
key: "model"
value: "input_3"
}
node_ids {
key: "model_blessing"
value: "input_4"
}
}
}
}
nodes {
key: "input_3"
value {
output_data_type: ARTIFACT_LIST
input_node {
input_key: "_generated_model_3"
}
}
}
nodes {
key: "input_4"
value {
output_data_type: ARTIFACT_LIST
input_node {
input_key: "_generated_modelblessing_4"
}
}
}
nodes {
key: "op_1"
value {
output_data_type: ARTIFACT_MULTIMAP
op_node {
op_type: "tfx.dsl.input_resolution.strategies.latest_blessed_model_strategy.LatestBlessedModelStrategy"
args {
node_id: "dict_2"
}
}
}
}
result_node: "op_1"
}
}
}
downstream_nodes: "Evaluator"
execution_options {
caching_options {
}
}
INFO:absl:Running as an resolver node.
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:[latest_blessed_model_resolver] Resolved inputs: ({'model_blessing': [], 'model': []},)
INFO:absl:Component latest_blessed_model_resolver is finished.
INFO:absl:Component Trainer is running.
INFO:absl:Running launcher for node_info {
type {
name: "tfx.components.trainer.component.Trainer"
base_type: TRAIN
}
id: "Trainer"
}
contexts {
contexts {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
contexts {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
contexts {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Trainer"
}
}
}
}
inputs {
inputs {
key: "examples"
value {
channels {
producer_node_query {
id: "CsvExampleGen"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.CsvExampleGen"
}
}
}
artifact_query {
type {
name: "Examples"
base_type: DATASET
}
}
output_key: "examples"
}
min_count: 1
}
}
}
outputs {
outputs {
key: "model"
value {
artifact_spec {
type {
name: "Model"
base_type: MODEL
}
}
}
}
outputs {
key: "model_run"
value {
artifact_spec {
type {
name: "ModelRun"
}
}
}
}
}
parameters {
parameters {
key: "custom_config"
value {
field_value {
string_value: "null"
}
}
}
parameters {
key: "eval_args"
value {
field_value {
string_value: "{\n \"num_steps\": 5\n}"
}
}
}
parameters {
key: "module_path"
value {
field_value {
string_value: "penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl"
}
}
}
parameters {
key: "train_args"
value {
field_value {
string_value: "{\n \"num_steps\": 100\n}"
}
}
}
}
upstream_nodes: "CsvExampleGen"
downstream_nodes: "Evaluator"
downstream_nodes: "Pusher"
execution_options {
caching_options {
}
}
INFO:absl:MetadataStore with DB connection initialized
WARNING:absl:ArtifactQuery.property_predicate is not supported.
INFO:absl:[Trainer] Resolved inputs: ({'examples': [Artifact(artifact: id: 1
type_id: 15
uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
properties {
key: "split_names"
value {
string_value: "[\"train\", \"eval\"]"
}
}
custom_properties {
key: "file_format"
value {
string_value: "tfrecords_gzip"
}
}
custom_properties {
key: "input_fingerprint"
value {
string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548"
}
}
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "payload_format"
value {
string_value: "FORMAT_TF_EXAMPLE"
}
}
custom_properties {
key: "span"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "Examples"
create_time_since_epoch: 1715159549765
last_update_time_since_epoch: 1715159549765
, artifact_type: id: 15
name: "Examples"
properties {
key: "span"
value: INT
}
properties {
key: "split_names"
value: STRING
}
properties {
key: "version"
value: INT
}
base_type: DATASET
)]},)
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Going to run a new execution 3
INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=3, input_dict={'examples': [Artifact(artifact: id: 1
type_id: 15
uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
properties {
key: "split_names"
value {
string_value: "[\"train\", \"eval\"]"
}
}
custom_properties {
key: "file_format"
value {
string_value: "tfrecords_gzip"
}
}
custom_properties {
key: "input_fingerprint"
value {
string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548"
}
}
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "payload_format"
value {
string_value: "FORMAT_TF_EXAMPLE"
}
}
custom_properties {
key: "span"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "Examples"
create_time_since_epoch: 1715159549765
last_update_time_since_epoch: 1715159549765
, artifact_type: id: 15
name: "Examples"
properties {
key: "span"
value: INT
}
properties {
key: "split_names"
value: STRING
}
properties {
key: "version"
value: INT
}
base_type: DATASET
)]}, output_dict=defaultdict(<class 'list'>, {'model': [Artifact(artifact: uri: "pipelines/penguin-tfma/Trainer/model/3"
, artifact_type: name: "Model"
base_type: MODEL
)], 'model_run': [Artifact(artifact: uri: "pipelines/penguin-tfma/Trainer/model_run/3"
, artifact_type: name: "ModelRun"
)]}), exec_properties={'module_path': 'penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl', 'custom_config': 'null', 'train_args': '{\n "num_steps": 100\n}', 'eval_args': '{\n "num_steps": 5\n}'}, execution_output_uri='pipelines/penguin-tfma/Trainer/.system/executor_execution/3/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/Trainer/.system/stateful_working_dir/cc01f07b-5e10-46fd-89d4-e0e7d78fb6fe', tmp_dir='pipelines/penguin-tfma/Trainer/.system/executor_execution/3/.temp/', pipeline_node=node_info {
type {
name: "tfx.components.trainer.component.Trainer"
base_type: TRAIN
}
id: "Trainer"
}
contexts {
contexts {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
contexts {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
contexts {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Trainer"
}
}
}
}
inputs {
inputs {
key: "examples"
value {
channels {
producer_node_query {
id: "CsvExampleGen"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.CsvExampleGen"
}
}
}
artifact_query {
type {
name: "Examples"
base_type: DATASET
}
}
output_key: "examples"
}
min_count: 1
}
}
}
outputs {
outputs {
key: "model"
value {
artifact_spec {
type {
name: "Model"
base_type: MODEL
}
}
}
}
outputs {
key: "model_run"
value {
artifact_spec {
type {
name: "ModelRun"
}
}
}
}
}
parameters {
parameters {
key: "custom_config"
value {
field_value {
string_value: "null"
}
}
}
parameters {
key: "eval_args"
value {
field_value {
string_value: "{\n \"num_steps\": 5\n}"
}
}
}
parameters {
key: "module_path"
value {
field_value {
string_value: "penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl"
}
}
}
parameters {
key: "train_args"
value {
field_value {
string_value: "{\n \"num_steps\": 100\n}"
}
}
}
}
upstream_nodes: "CsvExampleGen"
downstream_nodes: "Evaluator"
downstream_nodes: "Pusher"
execution_options {
caching_options {
}
}
, pipeline_info=id: "penguin-tfma"
, pipeline_run_id='2024-05-08T09:12:28.606391', top_level_pipeline_run_id=None, frontend_url=None)
INFO:absl:Train on the 'train' split when train_args.splits is not set.
INFO:absl:Evaluate on the 'eval' split when eval_args.splits is not set.
INFO:absl:udf_utils.get_fn {'module_path': 'penguin_trainer@pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl', 'custom_config': 'null', 'train_args': '{\n "num_steps": 100\n}', 'eval_args': '{\n "num_steps": 5\n}'} 'run_fn'
INFO:absl:Installing 'pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl' to a temporary directory.
INFO:absl:Executing: ['/tmpfs/src/tf_docs_env/bin/python', '-m', 'pip', 'install', '--target', '/tmpfs/tmp/tmpflpa4y_q', 'pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl']
Processing ./pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl
INFO:absl:Successfully installed 'pipelines/penguin-tfma/_wheels/tfx_user_code_Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703-py3-none-any.whl'.
INFO:absl:Training model.
INFO:absl:Feature body_mass_g has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_depth_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_length_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature flipper_length_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature species has a shape dim {
size: 1
}
. Setting to DenseTensor.
Installing collected packages: tfx-user-code-Trainer
Successfully installed tfx-user-code-Trainer-0.0+1e19049dced0ccb21e0af60dae1c6e0ef09b63d1ff0e370d7f699920c2735703
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tfx_bsl/tfxio/tf_example_record.py:343: parse_example_dataset (from tensorflow.python.data.experimental.ops.parsing_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map(tf.io.parse_example(...))` instead.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tfx_bsl/tfxio/tf_example_record.py:343: parse_example_dataset (from tensorflow.python.data.experimental.ops.parsing_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map(tf.io.parse_example(...))` instead.
INFO:absl:Feature body_mass_g has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_depth_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_length_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature flipper_length_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature species has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature body_mass_g has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_depth_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_length_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature flipper_length_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature species has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature body_mass_g has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_depth_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature culmen_length_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature flipper_length_mm has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Feature species has a shape dim {
size: 1
}
. Setting to DenseTensor.
INFO:absl:Model: "model"
INFO:absl:__________________________________________________________________________________________________
INFO:absl: Layer (type) Output Shape Param # Connected to
INFO:absl:==================================================================================================
INFO:absl: culmen_length_mm (InputLay [(None, 1)] 0 []
INFO:absl: er)
INFO:absl:
INFO:absl: culmen_depth_mm (InputLaye [(None, 1)] 0 []
INFO:absl: r)
INFO:absl:
INFO:absl: flipper_length_mm (InputLa [(None, 1)] 0 []
INFO:absl: yer)
INFO:absl:
INFO:absl: body_mass_g (InputLayer) [(None, 1)] 0 []
INFO:absl:
INFO:absl: concatenate (Concatenate) (None, 4) 0 ['culmen_length_mm[0][0]',
INFO:absl: 'culmen_depth_mm[0][0]',
INFO:absl: 'flipper_length_mm[0][0]',
INFO:absl: 'body_mass_g[0][0]']
INFO:absl:
INFO:absl: dense (Dense) (None, 8) 40 ['concatenate[0][0]']
INFO:absl:
INFO:absl: dense_1 (Dense) (None, 8) 72 ['dense[0][0]']
INFO:absl:
INFO:absl: dense_2 (Dense) (None, 3) 27 ['dense_1[0][0]']
INFO:absl:
INFO:absl:==================================================================================================
INFO:absl:Total params: 139 (556.00 Byte)
INFO:absl:Trainable params: 139 (556.00 Byte)
INFO:absl:Non-trainable params: 0 (0.00 Byte)
INFO:absl:__________________________________________________________________________________________________
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1715159557.217578 10799 device_compiler.h:186] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
100/100 [==============================] - 2s 5ms/step - loss: 0.4620 - sparse_categorical_accuracy: 0.8455 - val_loss: 0.1694 - val_sparse_categorical_accuracy: 0.9400
INFO:absl:Function `_wrapped_model` contains input name(s) resource with unsupported characters which will be renamed to model_dense_2_biasadd_readvariableop_resource in the SavedModel.
INFO:tensorflow:Assets written to: pipelines/penguin-tfma/Trainer/model/3/Format-Serving/assets
INFO:tensorflow:Assets written to: pipelines/penguin-tfma/Trainer/model/3/Format-Serving/assets
INFO:absl:Writing fingerprint to pipelines/penguin-tfma/Trainer/model/3/Format-Serving/fingerprint.pb
INFO:absl:Training complete. Model written to pipelines/penguin-tfma/Trainer/model/3/Format-Serving. ModelRun written to pipelines/penguin-tfma/Trainer/model_run/3
INFO:absl:Cleaning up stateless execution info.
INFO:absl:Execution 3 succeeded.
INFO:absl:Cleaning up stateful execution info.
INFO:absl:Deleted stateful_working_dir pipelines/penguin-tfma/Trainer/.system/stateful_working_dir/cc01f07b-5e10-46fd-89d4-e0e7d78fb6fe
INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'model': [Artifact(artifact: uri: "pipelines/penguin-tfma/Trainer/model/3"
, artifact_type: name: "Model"
base_type: MODEL
)], 'model_run': [Artifact(artifact: uri: "pipelines/penguin-tfma/Trainer/model_run/3"
, artifact_type: name: "ModelRun"
)]}) for execution 3
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Component Trainer is finished.
INFO:absl:Component Evaluator is running.
INFO:absl:Running launcher for node_info {
type {
name: "tfx.components.evaluator.component.Evaluator"
base_type: EVALUATE
}
id: "Evaluator"
}
contexts {
contexts {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
contexts {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
contexts {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Evaluator"
}
}
}
}
inputs {
inputs {
key: "baseline_model"
value {
channels {
producer_node_query {
id: "latest_blessed_model_resolver"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.latest_blessed_model_resolver"
}
}
}
artifact_query {
type {
name: "Model"
base_type: MODEL
}
}
output_key: "model"
}
}
}
inputs {
key: "examples"
value {
channels {
producer_node_query {
id: "CsvExampleGen"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.CsvExampleGen"
}
}
}
artifact_query {
type {
name: "Examples"
base_type: DATASET
}
}
output_key: "examples"
}
min_count: 1
}
}
inputs {
key: "model"
value {
channels {
producer_node_query {
id: "Trainer"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Trainer"
}
}
}
artifact_query {
type {
name: "Model"
base_type: MODEL
}
}
output_key: "model"
}
}
}
}
outputs {
outputs {
key: "blessing"
value {
artifact_spec {
type {
name: "ModelBlessing"
}
}
}
}
outputs {
key: "evaluation"
value {
artifact_spec {
type {
name: "ModelEvaluation"
}
}
}
}
}
parameters {
parameters {
key: "eval_config"
value {
field_value {
string_value: "{\n \"metrics_specs\": [\n {\n \"per_slice_thresholds\": {\n \"sparse_categorical_accuracy\": {\n \"thresholds\": [\n {\n \"slicing_specs\": [\n {}\n ],\n \"threshold\": {\n \"change_threshold\": {\n \"absolute\": -1e-10,\n \"direction\": \"HIGHER_IS_BETTER\"\n },\n \"value_threshold\": {\n \"lower_bound\": 0.6\n }\n }\n }\n ]\n }\n }\n }\n ],\n \"model_specs\": [\n {\n \"label_key\": \"species\"\n }\n ],\n \"slicing_specs\": [\n {},\n {\n \"feature_keys\": [\n \"species\"\n ]\n }\n ]\n}"
}
}
}
parameters {
key: "example_splits"
value {
field_value {
string_value: "null"
}
}
}
parameters {
key: "fairness_indicator_thresholds"
value {
field_value {
string_value: "null"
}
}
}
}
upstream_nodes: "CsvExampleGen"
upstream_nodes: "Trainer"
upstream_nodes: "latest_blessed_model_resolver"
downstream_nodes: "Pusher"
execution_options {
caching_options {
}
}
INFO:absl:MetadataStore with DB connection initialized
WARNING:absl:ArtifactQuery.property_predicate is not supported.
WARNING:absl:ArtifactQuery.property_predicate is not supported.
INFO:absl:[Evaluator] Resolved inputs: ({'examples': [Artifact(artifact: id: 1
type_id: 15
uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
properties {
key: "split_names"
value {
string_value: "[\"train\", \"eval\"]"
}
}
custom_properties {
key: "file_format"
value {
string_value: "tfrecords_gzip"
}
}
custom_properties {
key: "input_fingerprint"
value {
string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548"
}
}
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "payload_format"
value {
string_value: "FORMAT_TF_EXAMPLE"
}
}
custom_properties {
key: "span"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "Examples"
create_time_since_epoch: 1715159549765
last_update_time_since_epoch: 1715159549765
, artifact_type: id: 15
name: "Examples"
properties {
key: "span"
value: INT
}
properties {
key: "split_names"
value: STRING
}
properties {
key: "version"
value: INT
}
base_type: DATASET
)], 'model': [Artifact(artifact: id: 2
type_id: 18
uri: "pipelines/penguin-tfma/Trainer/model/3"
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "Model"
create_time_since_epoch: 1715159558908
last_update_time_since_epoch: 1715159558908
, artifact_type: id: 18
name: "Model"
base_type: MODEL
)], 'baseline_model': []},)
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Going to run a new execution 4
INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=4, input_dict={'examples': [Artifact(artifact: id: 1
type_id: 15
uri: "pipelines/penguin-tfma/CsvExampleGen/examples/1"
properties {
key: "split_names"
value {
string_value: "[\"train\", \"eval\"]"
}
}
custom_properties {
key: "file_format"
value {
string_value: "tfrecords_gzip"
}
}
custom_properties {
key: "input_fingerprint"
value {
string_value: "split:single_split,num_files:1,total_bytes:25648,xor_checksum:1715159548,sum_checksum:1715159548"
}
}
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "payload_format"
value {
string_value: "FORMAT_TF_EXAMPLE"
}
}
custom_properties {
key: "span"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "Examples"
create_time_since_epoch: 1715159549765
last_update_time_since_epoch: 1715159549765
, artifact_type: id: 15
name: "Examples"
properties {
key: "span"
value: INT
}
properties {
key: "split_names"
value: STRING
}
properties {
key: "version"
value: INT
}
base_type: DATASET
)], 'model': [Artifact(artifact: id: 2
type_id: 18
uri: "pipelines/penguin-tfma/Trainer/model/3"
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "Model"
create_time_since_epoch: 1715159558908
last_update_time_since_epoch: 1715159558908
, artifact_type: id: 18
name: "Model"
base_type: MODEL
)], 'baseline_model': []}, output_dict=defaultdict(<class 'list'>, {'evaluation': [Artifact(artifact: uri: "pipelines/penguin-tfma/Evaluator/evaluation/4"
, artifact_type: name: "ModelEvaluation"
)], 'blessing': [Artifact(artifact: uri: "pipelines/penguin-tfma/Evaluator/blessing/4"
, artifact_type: name: "ModelBlessing"
)]}), exec_properties={'eval_config': '{\n "metrics_specs": [\n {\n "per_slice_thresholds": {\n "sparse_categorical_accuracy": {\n "thresholds": [\n {\n "slicing_specs": [\n {}\n ],\n "threshold": {\n "change_threshold": {\n "absolute": -1e-10,\n "direction": "HIGHER_IS_BETTER"\n },\n "value_threshold": {\n "lower_bound": 0.6\n }\n }\n }\n ]\n }\n }\n }\n ],\n "model_specs": [\n {\n "label_key": "species"\n }\n ],\n "slicing_specs": [\n {},\n {\n "feature_keys": [\n "species"\n ]\n }\n ]\n}', 'example_splits': 'null', 'fairness_indicator_thresholds': 'null'}, execution_output_uri='pipelines/penguin-tfma/Evaluator/.system/executor_execution/4/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/Evaluator/.system/stateful_working_dir/156cd629-f1d0-4e6d-8519-c9ad5128ceba', tmp_dir='pipelines/penguin-tfma/Evaluator/.system/executor_execution/4/.temp/', pipeline_node=node_info {
type {
name: "tfx.components.evaluator.component.Evaluator"
base_type: EVALUATE
}
id: "Evaluator"
}
contexts {
contexts {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
contexts {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
contexts {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Evaluator"
}
}
}
}
inputs {
inputs {
key: "baseline_model"
value {
channels {
producer_node_query {
id: "latest_blessed_model_resolver"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.latest_blessed_model_resolver"
}
}
}
artifact_query {
type {
name: "Model"
base_type: MODEL
}
}
output_key: "model"
}
}
}
inputs {
key: "examples"
value {
channels {
producer_node_query {
id: "CsvExampleGen"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.CsvExampleGen"
}
}
}
artifact_query {
type {
name: "Examples"
base_type: DATASET
}
}
output_key: "examples"
}
min_count: 1
}
}
inputs {
key: "model"
value {
channels {
producer_node_query {
id: "Trainer"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Trainer"
}
}
}
artifact_query {
type {
name: "Model"
base_type: MODEL
}
}
output_key: "model"
}
}
}
}
outputs {
outputs {
key: "blessing"
value {
artifact_spec {
type {
name: "ModelBlessing"
}
}
}
}
outputs {
key: "evaluation"
value {
artifact_spec {
type {
name: "ModelEvaluation"
}
}
}
}
}
parameters {
parameters {
key: "eval_config"
value {
field_value {
string_value: "{\n \"metrics_specs\": [\n {\n \"per_slice_thresholds\": {\n \"sparse_categorical_accuracy\": {\n \"thresholds\": [\n {\n \"slicing_specs\": [\n {}\n ],\n \"threshold\": {\n \"change_threshold\": {\n \"absolute\": -1e-10,\n \"direction\": \"HIGHER_IS_BETTER\"\n },\n \"value_threshold\": {\n \"lower_bound\": 0.6\n }\n }\n }\n ]\n }\n }\n }\n ],\n \"model_specs\": [\n {\n \"label_key\": \"species\"\n }\n ],\n \"slicing_specs\": [\n {},\n {\n \"feature_keys\": [\n \"species\"\n ]\n }\n ]\n}"
}
}
}
parameters {
key: "example_splits"
value {
field_value {
string_value: "null"
}
}
}
parameters {
key: "fairness_indicator_thresholds"
value {
field_value {
string_value: "null"
}
}
}
}
upstream_nodes: "CsvExampleGen"
upstream_nodes: "Trainer"
upstream_nodes: "latest_blessed_model_resolver"
downstream_nodes: "Pusher"
execution_options {
caching_options {
}
}
, pipeline_info=id: "penguin-tfma"
, pipeline_run_id='2024-05-08T09:12:28.606391', top_level_pipeline_run_id=None, frontend_url=None)
INFO:absl:udf_utils.get_fn {'eval_config': '{\n "metrics_specs": [\n {\n "per_slice_thresholds": {\n "sparse_categorical_accuracy": {\n "thresholds": [\n {\n "slicing_specs": [\n {}\n ],\n "threshold": {\n "change_threshold": {\n "absolute": -1e-10,\n "direction": "HIGHER_IS_BETTER"\n },\n "value_threshold": {\n "lower_bound": 0.6\n }\n }\n }\n ]\n }\n }\n }\n ],\n "model_specs": [\n {\n "label_key": "species"\n }\n ],\n "slicing_specs": [\n {},\n {\n "feature_keys": [\n "species"\n ]\n }\n ]\n}', 'example_splits': 'null', 'fairness_indicator_thresholds': 'null'} 'custom_eval_shared_model'
INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
label_key: "species"
}
slicing_specs {
}
slicing_specs {
feature_keys: "species"
}
metrics_specs {
per_slice_thresholds {
key: "sparse_categorical_accuracy"
value {
thresholds {
slicing_specs {
}
threshold {
value_threshold {
lower_bound {
value: 0.6
}
}
}
}
}
}
}
INFO:absl:Using pipelines/penguin-tfma/Trainer/model/3/Format-Serving as model.
INFO:absl:The 'example_splits' parameter is not set, using 'eval' split.
INFO:absl:Evaluating model.
INFO:absl:udf_utils.get_fn {'eval_config': '{\n "metrics_specs": [\n {\n "per_slice_thresholds": {\n "sparse_categorical_accuracy": {\n "thresholds": [\n {\n "slicing_specs": [\n {}\n ],\n "threshold": {\n "change_threshold": {\n "absolute": -1e-10,\n "direction": "HIGHER_IS_BETTER"\n },\n "value_threshold": {\n "lower_bound": 0.6\n }\n }\n }\n ]\n }\n }\n }\n ],\n "model_specs": [\n {\n "label_key": "species"\n }\n ],\n "slicing_specs": [\n {},\n {\n "feature_keys": [\n "species"\n ]\n }\n ]\n}', 'example_splits': 'null', 'fairness_indicator_thresholds': 'null'} 'custom_extractors'
INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
label_key: "species"
}
slicing_specs {
}
slicing_specs {
feature_keys: "species"
}
metrics_specs {
model_names: ""
per_slice_thresholds {
key: "sparse_categorical_accuracy"
value {
thresholds {
slicing_specs {
}
threshold {
value_threshold {
lower_bound {
value: 0.6
}
}
}
}
}
}
}
INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
label_key: "species"
}
slicing_specs {
}
slicing_specs {
feature_keys: "species"
}
metrics_specs {
model_names: ""
per_slice_thresholds {
key: "sparse_categorical_accuracy"
value {
thresholds {
slicing_specs {
}
threshold {
value_threshold {
lower_bound {
value: 0.6
}
}
}
}
}
}
}
INFO:absl:eval_shared_models have model_types: {'tf_keras'}
INFO:absl:Request was made to ignore the baseline ModelSpec and any change thresholds. This is likely because a baseline model was not provided: updated_config=
model_specs {
label_key: "species"
}
slicing_specs {
}
slicing_specs {
feature_keys: "species"
}
metrics_specs {
model_names: ""
per_slice_thresholds {
key: "sparse_categorical_accuracy"
value {
thresholds {
slicing_specs {
}
threshold {
value_threshold {
lower_bound {
value: 0.6
}
}
}
}
}
}
}
INFO:absl:Evaluation complete. Results written to pipelines/penguin-tfma/Evaluator/evaluation/4.
INFO:absl:Checking validation results.
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:112: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
WARNING:tensorflow:From /tmpfs/src/tf_docs_env/lib/python3.9/site-packages/tensorflow_model_analysis/writers/metrics_plots_and_validations_writer.py:112: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
INFO:absl:Blessing result True written to pipelines/penguin-tfma/Evaluator/blessing/4.
INFO:absl:Cleaning up stateless execution info.
INFO:absl:Execution 4 succeeded.
INFO:absl:Cleaning up stateful execution info.
INFO:absl:Deleted stateful_working_dir pipelines/penguin-tfma/Evaluator/.system/stateful_working_dir/156cd629-f1d0-4e6d-8519-c9ad5128ceba
INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'evaluation': [Artifact(artifact: uri: "pipelines/penguin-tfma/Evaluator/evaluation/4"
, artifact_type: name: "ModelEvaluation"
)], 'blessing': [Artifact(artifact: uri: "pipelines/penguin-tfma/Evaluator/blessing/4"
, artifact_type: name: "ModelBlessing"
)]}) for execution 4
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Component Evaluator is finished.
INFO:absl:Component Pusher is running.
INFO:absl:Running launcher for node_info {
type {
name: "tfx.components.pusher.component.Pusher"
base_type: DEPLOY
}
id: "Pusher"
}
contexts {
contexts {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
contexts {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
contexts {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Pusher"
}
}
}
}
inputs {
inputs {
key: "model"
value {
channels {
producer_node_query {
id: "Trainer"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Trainer"
}
}
}
artifact_query {
type {
name: "Model"
base_type: MODEL
}
}
output_key: "model"
}
}
}
inputs {
key: "model_blessing"
value {
channels {
producer_node_query {
id: "Evaluator"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Evaluator"
}
}
}
artifact_query {
type {
name: "ModelBlessing"
}
}
output_key: "blessing"
}
}
}
}
outputs {
outputs {
key: "pushed_model"
value {
artifact_spec {
type {
name: "PushedModel"
base_type: MODEL
}
}
}
}
}
parameters {
parameters {
key: "custom_config"
value {
field_value {
string_value: "null"
}
}
}
parameters {
key: "push_destination"
value {
field_value {
string_value: "{\n \"filesystem\": {\n \"base_directory\": \"serving_model/penguin-tfma\"\n }\n}"
}
}
}
}
upstream_nodes: "Evaluator"
upstream_nodes: "Trainer"
execution_options {
caching_options {
}
}
INFO:absl:MetadataStore with DB connection initialized
WARNING:absl:ArtifactQuery.property_predicate is not supported.
WARNING:absl:ArtifactQuery.property_predicate is not supported.
INFO:absl:[Pusher] Resolved inputs: ({'model_blessing': [Artifact(artifact: id: 5
type_id: 22
uri: "pipelines/penguin-tfma/Evaluator/blessing/4"
custom_properties {
key: "blessed"
value {
int_value: 1
}
}
custom_properties {
key: "current_model"
value {
string_value: "pipelines/penguin-tfma/Trainer/model/3"
}
}
custom_properties {
key: "current_model_id"
value {
int_value: 2
}
}
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "ModelBlessing"
create_time_since_epoch: 1715159563475
last_update_time_since_epoch: 1715159563475
, artifact_type: id: 22
name: "ModelBlessing"
)], 'model': [Artifact(artifact: id: 2
type_id: 18
uri: "pipelines/penguin-tfma/Trainer/model/3"
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "Model"
create_time_since_epoch: 1715159558908
last_update_time_since_epoch: 1715159558908
, artifact_type: id: 18
name: "Model"
base_type: MODEL
)]},)
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Going to run a new execution 5
INFO:absl:Going to run a new execution: ExecutionInfo(execution_id=5, input_dict={'model_blessing': [Artifact(artifact: id: 5
type_id: 22
uri: "pipelines/penguin-tfma/Evaluator/blessing/4"
custom_properties {
key: "blessed"
value {
int_value: 1
}
}
custom_properties {
key: "current_model"
value {
string_value: "pipelines/penguin-tfma/Trainer/model/3"
}
}
custom_properties {
key: "current_model_id"
value {
int_value: 2
}
}
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "ModelBlessing"
create_time_since_epoch: 1715159563475
last_update_time_since_epoch: 1715159563475
, artifact_type: id: 22
name: "ModelBlessing"
)], 'model': [Artifact(artifact: id: 2
type_id: 18
uri: "pipelines/penguin-tfma/Trainer/model/3"
custom_properties {
key: "is_external"
value {
int_value: 0
}
}
custom_properties {
key: "tfx_version"
value {
string_value: "1.15.0"
}
}
state: LIVE
type: "Model"
create_time_since_epoch: 1715159558908
last_update_time_since_epoch: 1715159558908
, artifact_type: id: 18
name: "Model"
base_type: MODEL
)]}, output_dict=defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: "pipelines/penguin-tfma/Pusher/pushed_model/5"
, artifact_type: name: "PushedModel"
base_type: MODEL
)]}), exec_properties={'custom_config': 'null', 'push_destination': '{\n "filesystem": {\n "base_directory": "serving_model/penguin-tfma"\n }\n}'}, execution_output_uri='pipelines/penguin-tfma/Pusher/.system/executor_execution/5/executor_output.pb', stateful_working_dir='pipelines/penguin-tfma/Pusher/.system/stateful_working_dir/12b47904-285c-43d9-bb2e-9b1b59dce2f0', tmp_dir='pipelines/penguin-tfma/Pusher/.system/executor_execution/5/.temp/', pipeline_node=node_info {
type {
name: "tfx.components.pusher.component.Pusher"
base_type: DEPLOY
}
id: "Pusher"
}
contexts {
contexts {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
contexts {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
contexts {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Pusher"
}
}
}
}
inputs {
inputs {
key: "model"
value {
channels {
producer_node_query {
id: "Trainer"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Trainer"
}
}
}
artifact_query {
type {
name: "Model"
base_type: MODEL
}
}
output_key: "model"
}
}
}
inputs {
key: "model_blessing"
value {
channels {
producer_node_query {
id: "Evaluator"
}
context_queries {
type {
name: "pipeline"
}
name {
field_value {
string_value: "penguin-tfma"
}
}
}
context_queries {
type {
name: "pipeline_run"
}
name {
field_value {
string_value: "2024-05-08T09:12:28.606391"
}
}
}
context_queries {
type {
name: "node"
}
name {
field_value {
string_value: "penguin-tfma.Evaluator"
}
}
}
artifact_query {
type {
name: "ModelBlessing"
}
}
output_key: "blessing"
}
}
}
}
outputs {
outputs {
key: "pushed_model"
value {
artifact_spec {
type {
name: "PushedModel"
base_type: MODEL
}
}
}
}
}
parameters {
parameters {
key: "custom_config"
value {
field_value {
string_value: "null"
}
}
}
parameters {
key: "push_destination"
value {
field_value {
string_value: "{\n \"filesystem\": {\n \"base_directory\": \"serving_model/penguin-tfma\"\n }\n}"
}
}
}
}
upstream_nodes: "Evaluator"
upstream_nodes: "Trainer"
execution_options {
caching_options {
}
}
, pipeline_info=id: "penguin-tfma"
, pipeline_run_id='2024-05-08T09:12:28.606391', top_level_pipeline_run_id=None, frontend_url=None)
INFO:absl:Model version: 1715159563
INFO:absl:Model written to serving path serving_model/penguin-tfma/1715159563.
INFO:absl:Model pushed to pipelines/penguin-tfma/Pusher/pushed_model/5.
INFO:absl:Cleaning up stateless execution info.
INFO:absl:Execution 5 succeeded.
INFO:absl:Cleaning up stateful execution info.
INFO:absl:Deleted stateful_working_dir pipelines/penguin-tfma/Pusher/.system/stateful_working_dir/12b47904-285c-43d9-bb2e-9b1b59dce2f0
INFO:absl:Publishing output artifacts defaultdict(<class 'list'>, {'pushed_model': [Artifact(artifact: uri: "pipelines/penguin-tfma/Pusher/pushed_model/5"
, artifact_type: name: "PushedModel"
base_type: MODEL
)]}) for execution 5
INFO:absl:MetadataStore with DB connection initialized
INFO:absl:Component Pusher is finished.
当管道完成时，您应该能够看到类似以下内容
INFO:absl:Blessing result True written to pipelines/penguin-tfma/Evaluator/blessing/4.
或者您也可以手动检查生成的工件存储的输出目录。如果您使用文件浏览器访问 pipelines/penguin-tfma/Evaluator/blessing/，您可以看到一个名为 BLESSED 或 NOT_BLESSED 的文件，具体取决于评估结果。
如果祝福结果为 False，Pusher 将拒绝将模型推送到 serving_model_dir，因为该模型不够好，无法用于生产。
您可以再次运行管道，可能使用不同的评估配置。即使您使用完全相同的配置和数据集运行管道，训练后的模型也可能略有不同，因为模型训练固有的随机性会导致 NOT_BLESSED 模型。
检查管道的输出
您可以使用 TFMA 来调查和可视化模型评估工件中的评估结果。
从输出工件获取分析结果
您可以使用 MLMD API 以编程方式定位这些输出。首先，我们将定义一些实用程序函数来搜索刚刚生成的输出工件。
from ml_metadata.proto import metadata_store_pb2
# Non-public APIs, just for showcase.
from tfx.orchestration.portable.mlmd import execution_lib
# TODO(b/171447278): Move these functions into the TFX library.
def get_latest_artifacts(metadata, pipeline_name, component_id):
"""Output artifacts of the latest run of the component."""
context = metadata.store.get_context_by_type_and_name(
'node', f'{pipeline_name}.{component_id}')
executions = metadata.store.get_executions_by_context(context.id)
latest_execution = max(executions,
key=lambda e:e.last_update_time_since_epoch)
return execution_lib.get_output_artifacts(metadata, latest_execution.id)
我们可以找到 Evaluator 组件的最新执行，并获取其输出工件。
# Non-public APIs, just for showcase.
from tfx.orchestration.metadata import Metadata
from tfx.types import standard_component_specs
metadata_connection_config = tfx.orchestration.metadata.sqlite_metadata_connection_config(
METADATA_PATH)
with Metadata(metadata_connection_config) as metadata_handler:
# Find output artifacts from MLMD.
evaluator_output = get_latest_artifacts(metadata_handler, PIPELINE_NAME,
'Evaluator')
eval_artifact = evaluator_output[standard_component_specs.EVALUATION_KEY][0]
INFO:absl:MetadataStore with DB connection initialized
Evaluator 始终返回一个评估工件，我们可以使用 TensorFlow 模型分析库对其进行可视化。例如，以下代码将呈现每个企鹅物种的准确性指标。
import tensorflow_model_analysis as tfma
eval_result = tfma.load_eval_result(eval_artifact.uri)
tfma.view.render_slicing_metrics(eval_result, slicing_column='species')
SlicingMetricsViewer(config={'weightedExamplesColumn': 'example_count'}, data=[{'slice': 'species:0', 'metrics…
如果您在 Show 下拉列表中选择 'sparse_categorical_accuracy'，您可以看到每个物种的准确性值。您可能希望添加更多切片并检查您的模型是否适合所有分布，以及是否存在任何可能的偏差。
下一步
在 TensorFlow 模型分析库教程 中了解更多关于模型分析的信息。
您可以在 https://tensorflowcn.cn/tfx/tutorials 上找到更多资源
请参阅 了解 TFX 管道，以详细了解 TFX 中的各种概念。