先把文献的PDF老老实实下载下来

想办法把PDF的内容粘贴到文本文件里

给文本文件里的内容分成句子

spaCY

1
2
3
4
5
6
7
8
9
10
11
12
import spacy

file = 'test.md'
out_file = file.split('.')[0] + '.translated.md'

with open(file,'r',encoding='utf-8') as f:
    text = f.read()

nlp = spacy.load("en_core_web_sm")
doc = nlp(text)

sentences = [sent.text.strip() for sent in doc.sents]

挨个句子套翻译模型翻译成中文

HuggingFace
模型官网装，但是记得有几个git clone不下来的文件，手动下载一下。
或者 用推荐的方案：

1
2
# Make sure you have git-lfs installed (https://git-lfs.com)
git lfs install

然后就这样：

1
2
3
4
5
6
7
8
9
10
11
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from transformers import pipeline

model_path = '.\\opus-mt-en-zh'
tokenizer = AutoTokenizer.from_pretrained(model_path)  
model = AutoModelForSeq2SeqLM.from_pretrained(model_path)
pipeline = pipeline("translation", model=model, tokenizer=tokenizer)

sentence = sentences[0]
finaltext = pipeline(sentence)
translated = finaltext[0]['translation_text']

总结个完整流程

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import spacy  
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from transformers import pipeline

file = input('INPUT YOUR FILE:')

out_file = file.split('.')[0] + '.translated.md'
with open(file,'r',encoding='utf-8') as f:
    text = f.read()

nlp        = spacy.load("en_core_web_sm")
doc        = nlp(text)
sentences  = [sent.text.strip() for sent in doc.sents]
model_path = '.\\opus-mt-en-zh'
tokenizer  = AutoTokenizer.from_pretrained(model_path)  
model      = AutoModelForSeq2SeqLM.from_pretrained(model_path)
pipeline   = pipeline("translation", model=model, tokenizer=tokenizer)

with open(out_file, 'w', encoding='utf-8') as fw:
    for sentence in sentences:
        if sentence.startswith('['): # 省略题注
            fw.write(sentence)
            continue

        finaltext = pipeline(sentence)
        fw.write(f"【{finaltext[0]['translation_text']}】\n{sentence}\n")