iomgaa/Minimind
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update data preprocessing methods

Browse Source
This commit is contained in:
gongjy 2024-09-27 22:34:30 +08:00
parent 75753ea765
commit 2981d3ea86
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
0-eval_pretrain.py
View File

@ -107,6 +107,7 @@ if __name__ == "__main__":
print('问题：', prompt) print('问题：', prompt)
qa_index += 1 qa_index += 1
prompt = tokenizer.bos_token + prompt
x = tokenizer(prompt).data['input_ids'] x = tokenizer(prompt).data['input_ids']
x = (torch.tensor(x, dtype=torch.long, device=device)[None, ...]) x = (torch.tensor(x, dtype=torch.long, device=device)[None, ...])