工程化实践：如何基于自定义数据进行Ptuning实验？如何部署预测？

 关注微信公众号&＃xff1a;NLP分享汇。【喜欢的扫波关注&＃xff0c;每天都在更新自己之前的积累】

· 背景说明

谁说GPT只擅长生成&＃xff0c;GPT也能自然语言理解。利用 P-tuning 方法&＃xff0c;GPT 的自然语言能力可以匹敌 BERT。2021年&＃xff0c;清华、智源等机构的一项研究打破了这一刻板印象。

这一切源于这篇论文&＃xff1a;《GPT Understands, Too》

论文原文&＃xff1a;https://arxiv.org/pdf/2103.10385.pdf

GitHub&＃xff1a;https://github.com/THUDM/P-tuning

但今天这篇文章并非要向大家赘述P-tuning原理。关于上述论文大家可以搜搜博客自行消化&＃xff0c;我们这里主要要借助百度PaddleNLP去介绍一下P-tuning小样本模型的一些工程化干货。

· 前言

小样本学习&＃xff08;Few-Shot Learning&＃xff09; 旨在研究如何从少量有监督的训练样本中学习出具有良好泛化性的模型&＃xff0c;对训练数据很少或监督数据获取成本极高的应用场景有很大价值。百度将中文预训练模型renie1.0和prompt方法结合&＃xff0c;为我们提供了一个P-tuning一站式的应用方法&＃xff0c;代码用起来超级方便。但是&＃xff0c;直接用百度github提供的内置数据、README去按部就班跑起来程序似乎对我们的帮助就很鸡肋&＃xff0c;更多的我们希望该开源能够应用到我们着手的具体小样本项目场景中&＃xff0c;比如&＃xff1a;风控、质检等。这篇文章则希望帮助读者们解决如下3个问题&＃xff1a;1&＃xff09;复现可能存在的环境安装问题和代码运行问题&＃xff0c;并奉上解决方法&＃xff1b;2&＃xff09;如何基于自定义数据实验P-tuning&＃xff1b;3&＃xff09;如何部署训练好的模型进行单句预测。

· 基础准备

在基于自定义数据进行P-tuning实验前&＃xff0c;小编希望读者能先按github上的README先跑起其内置的FewCLUE数据集&＃xff08;小样本学习测评基准-中文版&＃xff09;。

1&＃xff09;P-tuning github&＃xff1a;https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/few_shot/p-tuning

代码结构&＃xff1a;


|—— ptuning.py # P-tuning 策略的训练、评估主脚本
|—— dataset.py # P-tuning 策略针对 FewCLUE 9 个数据集的任务转换逻辑&＃xff0c;以及明文 -> 训练数据的转换
|—— model.py # P-tuning 的网络结构
|—— evaluate.py # 针对 FewCLUE 9 个数据集的评估函数
|—— predict.py # 针对 FewCLUE 9 个数据集进行预测

2&＃xff09;FewCLUE数据集

github: https://github.com/CLUEbenchmark/FewCLUE

paper&＃xff1a;https://arxiv.org/abs/2107.07498

· 工程实践分析

1&＃xff09;环境安装和代码运行BUG

【BUG-1】ImportError: /usr/lib/x86_64-linux-gnu/libstdc&＃43;&＃43;.so.6: VERSION &＃39;GLIBCXX_3.4.22&＃39; not found
【解决方法】https://blog.csdn.net/u014577702/article/details/123453453?spm&＃61;1001.2014.3001.5502

【BUG-2】request.exceptions.ConnectionError: HTTPSConnectionPool(host&＃61;&＃39;paddlenlp.bj.bcebos.com&＃39;,port&＃61;443): Max retries exceeded w: /model/transformers/ernie/ernie_v1_chn_base.pdparams (Caused by New Connection)
【解决方法】自行下载相应安装内容&＃xff0c;导入相应位置。

【BUG-3】找不到libcublas.so
【解决方法】找不到libcublas.so&＃xff0c;则需要链接到该软链接所在位置&＃xff0c;使用如下命令
export LD_LIBRARY_PATH&＃61;/home/pafl/anaconda3/envs/mypaddle/lib:$PATH
export LD_LIBRARY_PATH&＃61;conda环境所在位置:$PATH

【BUG-4】
INFO 2021-10-26 19:01:38,701 launch_utils.py:327] terminate all the procs
ERROR 2021-10-26 19:01:38,702 launch_utils.py:584] ABORT!!! Out of all 1 trainers, the trainer process with rank&＃61;[0] was aborted. Please check its log.
INFO 2021-10-26 19:01:41,705 launch_utils.py:327] terminate all the procs
【解决方法】https://github.com/PaddlePaddle/PaddleNLP/issues/1238

【BUG-5】训练报错ABORT!!! Out of all 1 trainers, the trainer process with rank&＃61;[0] was aborted. Please check its log.
【解决方法】单卡不需要 paddle.distributed.launch

【BUG-6】ln: failed to create symbolic link &＃39;libcudnn.so&＃39; : File exits
【解决方法】https://www.jianshu.com/p/b308d3bbde8a


【BUG-7】fatal error: &＃39;Segmentation fault&＃39; is detected by the operating system
【解决方法】https://blog.csdn.net/u014577702/article/details/123453508

【BUG-8】Fatal Error : &＃39;Access to an undefined portion of a memory object&＃39; is detected by the operating system
【解决方法】一般就是cudatoolkit、cudann的版本不符合环境要求&＃xff0c;升级到对应版本就行。我的cudatoolkit从8.0升级到1.0就能正常运行。conda install cudatoolkit&＃61;10.0

2&＃xff09;如何基于自定义数据实验P-tuning?

step-1  实现读取函数&＃xff0c;以字典形式返回明文数据&＃xff0c;并在def train()中更改加载数据的方式 eg:

# 假设明文数据为 \t 分隔的 2 列数据: text \t label
def read_fn(data_path):example &＃61; []with open(data_path, &＃39;r&＃39;, encoding&＃61;&＃39;utf-8&＃39;) as f:for line in f:text, label &＃61; line.rstrip().split(&＃39;\t&＃39;)yield {"text":text, ";label":label}

def do_train():...# 原直接利用内置数据fewclue的数据加载方式# train_ds, dev_ds, public_test_ds &＃61; load_dataset(# "fewclue",# name&＃61;args.task_name,# splits&＃61;("train_0", "dev_0", "test_public"))# 自己的数据加载方式train_ds &＃61; load_dataset(read_fn,data_path &＃61; "../train.txt",lazy&＃61;False)dev_ds &＃61; load_dataset(read_fn,data_path &＃61; "../dev.txt",lazy&＃61;False)public_test_ds &＃61; load_dataset(read_fn,data_path &＃61; "../test.txt",lazy&＃61;False)

step-2  在 ./label_normalized/ 目录下创建名为 mytask.json 的 Label map 词典&＃xff0c;负责对 Label 进行标准化, eg:

# 以 label 为 "Positive"、"Negative"的分类任务为例。
{"Negative":"负例","Positive":"正例"}

3&＃xff09;如何部署训练好的模型进行单句预测&＃xff1f;

# 单独写个py进行单句预测
import argparse
import os
import sys
import random
import time
import json
from functools import partialimport numpy as np
import paddle
import paddle.nn.functional as Fimport paddlenlp as ppnlp
from model import ErnieForPretraining
from paddlenlp.data import Stack, Tuple, Pad
from paddlenlp.datasets import load_datasetfrom data import create_dataloader, transform_fn_dict
from data import convert_example, convert_chid_example
from evaluate import do_evaluate, do_evaluate_chid
import timeparser &＃61; argparse.ArgumentParser()
args &＃61; parser.parse_args()def set_seed(seed):"""sets random seed"""random.seed(seed)np.random.seed(seed)paddle.seed(seed)&＃64;paddle.no_grad()
def do_predict(model, tokenizer, data_loader, label_normalize_dict):model.eval()normed_labels &＃61; [normalized_lablefor origin_lable, normalized_lable in label_normalize_dict.items()]origin_labels &＃61; [origin_lablefor origin_lable, normalized_lable in label_normalize_dict.items()]label_length &＃61; len(normed_labels[0])y_pred_labels &＃61; []for batch in data_loader:src_ids, token_type_ids, masked_positions &＃61; batch# [bs * label_length, vocab_size]prediction_probs &＃61; model.predict(input_ids&＃61;src_ids,token_type_ids&＃61;token_type_ids,masked_positions&＃61;masked_positions)batch_size &＃61; len(src_ids)vocab_size &＃61; prediction_probs.shape[1]# prediction_probs: [batch_size, label_lenght, vocab_size]prediction_probs &＃61; paddle.reshape(prediction_probs, shape&＃61;[batch_size, -1, vocab_size]).numpy()# [label_num, label_length]label_ids &＃61; np.array([tokenizer(label)["input_ids"][1:-1] for label in normed_labels])y_pred &＃61; np.ones(shape&＃61;[batch_size, len(label_ids)])# Calculate joint distribution of candidate labelsfor index in range(label_length):y_pred *&＃61; prediction_probs[:, index, label_ids[:, index]]# Get max probs label&＃39;s indexy_pred_index &＃61; np.argmax(y_pred, axis&＃61;-1)for index in y_pred_index:y_pred_labels.append(origin_labels[index])return y_pred_labelspredict_file &＃61; {"mytask": "mytask_predict.json"
}def read_fn(data_path):example &＃61; []with open(data_path, &＃39;r&＃39;, encoding&＃61;&＃39;utf-8&＃39;) as f:for line in f:sid, text &＃61; line.rstrip().split(&＃39;\t&＃39;)yield {"id":int(sid), "sentence":text}def write_my(task_name, output_file, pred_labels):test_ds &＃61; load_dataset(read_fn, data_path &＃61; "../test.txt", lazy&＃61;False)test_example &＃61; {}with open(output_file, &＃39;w&＃39;, encoding&＃61;&＃39;utf-8&＃39;) as f:for idx, example in enumerate(test_ds):test_example["id"] &＃61; example["id"]test_example["label"] &＃61; pred_labels[idx]str_test_example &＃61; json.dumps(test_example)f.write(str_test_example &＃43; "\n")write_fn &＃61; {"mytask": write_my
}if __name__ &＃61;&＃61; "__main__":paddle.set_device(&＃39;cpu&＃39;)set_seed(1000)label_normalize_json &＃61; os.path.join("./label_normalized","mytask.json")init_from_ckpt &＃61; "../model_state.pdparams"label_norm_dict &＃61; Nonewith open(label_normalize_json, encoding&＃61;&＃39;utf-8&＃39;) as f:label_norm_dict &＃61; json.load(f)convert_example_fn &＃61; convert_examplepredict_fn &＃61; do_predictprint("model paramas loading ...")model &＃61; ErnieForPretraining.from_pretrained(&＃39;ernie-1.0&＃39;)tokenizer &＃61; ppnlp.transformers.ErnieTokenizer.from_pretrained(&＃39;ernie-1.0&＃39;)# Load parameters of best model on test_public.json of current taskif args.init_from_ckpt and os.path.isfile(args.init_from_ckpt):state_dict &＃61; paddle.load(args.init_from_ckpt)model.set_dict(state_dict)print("Loaded parameters from %s" % args.init_from_ckpt)else:raise ValueError("Please set --params_path with correct pretrained model file")# [src_ids, token_type_ids, masked_positions, masked_lm_labels]batchify_fn &＃61; lambda samples, fn&＃61;Tuple(Pad(axis&＃61;0, pad_val&＃61;tokenizer.pad_token_id), # src_idsPad(axis&＃61;0, pad_val&＃61;tokenizer.pad_token_type_id), # token_type_idsStack(dtype&＃61;"int64"), # masked_positions): [data for data in fn(samples)]trans_func &＃61; partial(convert_example_fn,tokenizer&＃61;tokenizer,max_seq_length&＃61;args.max_seq_length,p_embedding_num&＃61;args.p_embedding_num,is_test&＃61;True)while True:print("Input sentence: ")sentence &＃61; sys.stdin.readline().strip()# 如下只是一种最简单粗暴的方式&＃xff0c;大家有时间可以自行更改with open("../test.txt","w",encoding&＃61;&＃39;utf-8&＃39;) as ftest:ftest.write(&＃39;1&＃39; &＃43; &＃39;\t&＃39; &＃43; sentence)ftest.close()test_ds &＃61; load_dataset(read_fn,data_path&＃61;"../test.txt",lazy&＃61;False)# Task related transform operations, eg: numbert label -> text_label, english -> chinesetransform_fn &＃61; partial(transform_fn_dict[args.task_name],label_normalize_dict&＃61;label_norm_dict,is_test&＃61;True)test_ds &＃61; test_ds.map(transform_fn, lazy&＃61;False)test_data_loader &＃61; create_dataloader(test_ds,mode&＃61;&＃39;eval&＃39;,batch_size&＃61;1,batchify_fn&＃61;batchify_fn,trans_fn&＃61;trans_func)y_pred_labels &＃61; predict_fn(model, tokenizer, test_data_loader,label_norm_dict)if y_pred_labels[0] &＃61;&＃61; &＃39;Poscase&＃39;:print("正例")else:print("负例")

如果你在进行P-tuning实验中遇到了问题&＃xff0c;可以在公众号给我发消息&＃xff0c;定期查看回复。