软件需求:

首先你必须要有Moses(废话哈哈)、然后要有GIZA++用作词对齐(traning-model.perl的时候会用到)、IRSTLM产生语言模型

大致步骤:

大体的步骤如下:

  1. 准备Parallerl data(需要句子对齐):对语料进行tokenisation、truecasing和cleaning步骤之后才能使用于我们的机器翻译系统(哈哈,都快忍不住直接写详细步骤了)
  2. 训练你的语言模型(使用IRSTLM):当然也有几步,详细叙述再说
  3. 然后就是训练你的翻译系统啦(可能要花一两个小时):(2) run GIZA
    (3) align words
    (4) learn lexical translation
    (5) extract phrases
    (6) score phrases
    (7) learn reordering model
    (8) learn generation model
    (9) create decoder config file
  4. 最后是蛋疼的Tuning(当然你也可以自己手动的Tuning),大概要花几个小时
  5. 最后你就可以跑了,如果你嫌启动慢,可以把模型转化为Binarised-model来进行,会更快,当然这会需要你改动一些东西,不过很简单

详细步骤以及说明:

一、准备语料:

  首先我们需要你找到你要创建的翻译系统匹配的平行语料,例如英法新闻平行语料,然后对语料进行三步处理后才可以使用:tokenisation、truecasing和cleaning

  首先是tokenisation,就是词例化或者说分词,要使用 :mosesdecoder/scripts/tokenizer/tokenizer.perl进行词例化。 作用是将平行语料(其实是两个文本文件),中的每个句子进行词例化

例子如下:

 ~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en \
    < ~/corpus/training/news-commentary-v8.fr-en.en    \
    > ~/corpus/news-commentary-v8.fr-en.tok.en
 ~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l fr \
    < ~/corpus/training/news-commentary-v8.fr-en.fr    \
    > ~/corpus/news-commentary-v8.fr-en.tok.fr

说明:-l 后指定文本的语言,看了一下文件,似乎只支持 en de fr 和 it,如要给中文分词的话,可能需要更多的配置,然后要指定输入的文本和输出的地点,切记 “<”  ">"是必须的!

完整参数说明如下:

Usage ./tokenizer.perl (-l [en|de|...]) (-threads 4) < textfile > tokenizedfile
Options:
  -q     ... quiet.
  -a     ... aggressive hyphen splitting.
  -b     ... disable Perl buffering.
  -time  ... enable processing time calculation.
  -penn  ... use Penn treebank-like tokenization.
  -protected FILE  ... specify file with patters to be protected in tokenisation.
  -no-escape ... don't perform HTML escaping on apostrophy, quotes, etc.

然后我们还要进行truecasing,也就是对词汇的大小写进行调整

truecasing: The initial words in each sentence are converted to their most probable casing. This helps reduce data sparsity.,使用的脚本是:

/mosesdecoder/scripts/recaser/train-truecaser.perl和 /scripts/recaser/truecase.perl例子如下
~/mosesdecoder/scripts/recaser/train-truecaser.perl \
     --model ~/corpus/truecase-model.en --corpus     \
     ~/corpus/news-commentary-v8.fr-en.tok.en
 ~/mosesdecoder/scripts/recaser/train-truecaser.perl \
     --model ~/corpus/truecase-model.fr --corpus     \
     ~/corpus/news-commentary-v8.fr-en.tok.fr
 ~/mosesdecoder/scripts/recaser/truecase.perl \
   --model ~/corpus/truecase-model.en         \
   < ~/corpus/news-commentary-v8.fr-en.tok.en \
   > ~/corpus/news-commentary-v8.fr-en.true.en
 ~/mosesdecoder/scripts/recaser/truecase.perl \
   --model ~/corpus/truecase-model.fr         \
   < ~/corpus/news-commentary-v8.fr-en.tok.fr \
   > ~/corpus/news-commentary-v8.fr-en.true.fr

其中的train-truecaser脚本用来训练truecaser的model,输入文件仍然是你的corpus的已tok文件,作用是修改每一句子的首字母,输出是每个不同单词的形式和频率

manual中的USER GUIDE写到:

Instead of lowercasing all training and test data, we may also want to keep words in their nat-
ural case, and only change the words at the beginning of their sentence to their most frequent
form. This is what we mean by truecasing. Again, this requires first the training of a truecasing
model, which is a list of words and the frequency of their different forms.

然后最后一部就是用刚才的模型进行truecase啦:

truecase.perl --model MODEL [-b] < in > out

 -b代表 unbuffered,不清楚用途目前

 在中文处理中应该不需要truecasing这一步

 

最后的词处理clean,去除一些过长的单词

Finally we clean, limiting sentence length to 80:

 ~/mosesdecoder/scripts/training/clean-corpus-n.perl \
    ~/corpus/news-commentary-v8.fr-en.true fr en \
    ~/corpus/news-commentary-v8.fr-en.clean 1 80

二、训练语言模型(使用IRSTLM)

终于搞定了我们的语料了,下面我们要进入更深入的话题:训练语言模型

语言模型最朴实的作用在于让你的output更加流畅,更加像母语,为了达到这一效果,我们需要另外的句子对齐的平行语料来训练我们的语言模型(如果你用training model的同一个语料来训练,未免感觉会无效?)

参看manual中base系统的描述,我们需要用到以下几个工具来训练我们的语言模型。

首先:add-start-end.sh

 ~/irstlm/bin/add-start-end.sh                 \
   < ~/corpus/news-commentary-v8.fr-en.true.en \
   > news-commentary-v8.fr-en.sb.en

  用于把你的语料添加上开始结束标记(实际上就是<s></s>标签对)

然后使用:build-lm.sh build一个语言模型源文件,输出lm源文件

export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
   -i news-commentary-v8.fr-en.sb.en                  \
   -t ./tmp  -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en

最后compile:

~/irstlm/bin/compile-lm  \

软件需求:

首先你必须要有Moses(废话哈哈)、然后要有GIZA++用作词对齐(traning-model.perl的时候会用到)、IRSTLM产生语言模型

大致步骤:

大体的步骤如下:

  1. 准备Parallerl data(需要句子对齐):对语料进行tokenisation、truecasing和cleaning步骤之后才能使用于我们的机器翻译系统(哈哈,都快忍不住直接写详细步骤了)
  2. 训练你的语言模型(使用IRSTLM):当然也有几步,详细叙述再说
  3. 然后就是训练你的翻译系统啦(可能要花一两个小时):(2) run GIZA
    (3) align words
    (4) learn lexical translation
    (5) extract phrases
    (6) score phrases
    (7) learn reordering model
    (8) learn generation model
    (9) create decoder config file
  4. 最后是蛋疼的Tuning(当然你也可以自己手动的Tuning),大概要花几个小时
  5. 最后你就可以跑了,如果你嫌启动慢,可以把模型转化为Binarised-model来进行,会更快,当然这会需要你改动一些东西,不过很简单

详细步骤以及说明:

一、准备语料:

  首先我们需要你找到你要创建的翻译系统匹配的平行语料,例如英法新闻平行语料,然后对语料进行三步处理后才可以使用:tokenisation、truecasing和cleaning

  首先是tokenisation,就是词例化或者说分词,要使用 :mosesdecoder/scripts/tokenizer/tokenizer.perl进行词例化。 作用是将平行语料(其实是两个文本文件),中的每个句子进行词例化

例子如下:

 ~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en \
    < ~/corpus/training/news-commentary-v8.fr-en.en    \
    > ~/corpus/news-commentary-v8.fr-en.tok.en
 ~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l fr \
    < ~/corpus/training/news-commentary-v8.fr-en.fr    \
    > ~/corpus/news-commentary-v8.fr-en.tok.fr

说明:-l 后指定文本的语言,看了一下文件,似乎只支持 en de fr 和 it,如要给中文分词的话,可能需要更多的配置,然后要指定输入的文本和输出的地点,切记 “<”  ">"是必须的!

完整参数说明如下:

Usage ./tokenizer.perl (-l [en|de|...]) (-threads 4) < textfile > tokenizedfile
Options:
  -q     ... quiet.
  -a     ... aggressive hyphen splitting.
  -b     ... disable Perl buffering.
  -time  ... enable processing time calculation.
  -penn  ... use Penn treebank-like tokenization.
  -protected FILE  ... specify file with patters to be protected in tokenisation.
  -no-escape ... don't perform HTML escaping on apostrophy, quotes, etc.

然后我们还要进行truecasing,也就是对词汇的大小写进行调整

truecasing: The initial words in each sentence are converted to their most probable casing. This helps reduce data sparsity.,使用的脚本是:

/mosesdecoder/scripts/recaser/train-truecaser.perl和 /scripts/recaser/truecase.perl例子如下

~/mosesdecoder/scripts/recaser/train-truecaser.perl \
     --model ~/corpus/truecase-model.en --corpus     \
     ~/corpus/news-commentary-v8.fr-en.tok.en
 ~/mosesdecoder/scripts/recaser/train-truecaser.perl \
     --model ~/corpus/truecase-model.fr --corpus     \
     ~/corpus/news-commentary-v8.fr-en.tok.fr
 ~/mosesdecoder/scripts/recaser/truecase.perl \
   --model ~/corpus/truecase-model.en         \
   < ~/corpus/news-commentary-v8.fr-en.tok.en \
   > ~/corpus/news-commentary-v8.fr-en.true.en
 ~/mosesdecoder/scripts/recaser/truecase.perl \
   --model ~/corpus/truecase-model.fr         \
   < ~/corpus/news-commentary-v8.fr-en.tok.fr \
   > ~/corpus/news-commentary-v8.fr-en.true.fr

其中的train-truecaser脚本用来训练truecaser的model,输入文件仍然是你的corpus的已tok文件,作用是修改每一句子的首字母,输出是每个不同单词的形式和频率

manual中的USER GUIDE写到:

Instead of lowercasing all training and test data, we may also want to keep words in their nat-
ural case, and only change the words at the beginning of their sentence to their most frequent
form. This is what we mean by truecasing. Again, this requires first the training of a truecasing
model, which is a list of words and the frequency of their different forms.

然后最后一部就是用刚才的模型进行truecase啦:

truecase.perl --model MODEL [-b] < in > out

 -b代表 unbuffered,不清楚用途目前

 在中文处理中应该不需要truecasing这一步

 

最后的词处理clean,去除一些过长的单词

Finally we clean, limiting sentence length to 80:

 ~/mosesdecoder/scripts/training/clean-corpus-n.perl \
    ~/corpus/news-commentary-v8.fr-en.true fr en \
    ~/corpus/news-commentary-v8.fr-en.clean 1 80

二、训练语言模型(使用IRSTLM)

终于搞定了我们的语料了,下面我们要进入更深入的话题:训练语言模型

语言模型最朴实的作用在于让你的output更加流畅,更加像母语,为了达到这一效果,我们需要另外的句子对齐的平行语料来训练我们的语言模型(如果你用training model的同一个语料来训练,未免感觉会无效?)

参看manual中base系统的描述,我们需要用到以下几个工具来训练我们的语言模型。

首先:add-start-end.sh

 ~/irstlm/bin/add-start-end.sh                 \
   < ~/corpus/news-commentary-v8.fr-en.true.en \
   > news-commentary-v8.fr-en.sb.en

  用于把你的语料添加上开始结束标记(实际上就是<s></s>标签对)

然后使用:build-lm.sh build一个语言模型源文件,输出lm源文件

export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
   -i news-commentary-v8.fr-en.sb.en                  \
   -t ./tmp  -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en

最后compile:

~/irstlm/bin/compile-lm  \
   --text  \
   news-commentary-v8.fr-en.lm.en.gz \
   news-commentary-v8.fr-en.arpa.en

注意这里的 --text 后不要加 yes manual中写错了,这样就生成了一个arpa文件(可以用于query和生成二进制的IRSTLM模型以及KenLM模型,这里一直用生成KenLM解决,因为不知道为何IRSTLM不好用)

You can directly create an IRSTLM binary LM (for faster loading in Moses) by replacing the last command with the following:

 ~/irstlm/bin/compile-lm news-commentary-v8.fr-en.lm.en.gz \
   news-commentary-v8.fr-en.blm.en

You can transform an arpa LM (*.arpa.en file) into an IRSTLM binary LM as follows:

 ~/irstlm/bin/compile-lm \
   news-commentary-v8.fr-en.arpa.en \
   news-commentary-v8.fr-en.blm.en

or viceversa, you can transform an IRSTLM binary LM into an arpa LM as follows:

 ~/irstlm/bin/compile-lm \
   --text yes \
   news-commentary-v8.fr-en.blm.en \
   news-commentary-v8.fr-en.arpa.en

This instead binarises (for faster loading) the *.arpa.en file using KenLM:

 ~/mosesdecoder/bin/build_binary \
   news-commentary-v8.fr-en.arpa.en \
   news-commentary-v8.fr-en.blm.en

You can check the language model by querying it, e.g.

 $ echo "is this an English sentence ?"                       \
   | ~/mosesdecoder/bin/query news-commentary-v8.fr-en.blm.en
   --text  \
   news-commentary-v8.fr-en.lm.en.gz \
   news-commentary-v8.fr-en.arpa.en

注意这里的 --text 后不要加 yes manual中写错了,这样就生成了一个arpa文件(可以用于query和生成二进制的IRSTLM模型以及KenLM模型,这里一直用生成KenLM解决,因为不知道为何IRSTLM不好用)

You can directly create an IRSTLM binary LM (for faster loading in Moses) by replacing the last command with the following:

 ~/irstlm/bin/compile-lm news-commentary-v8.fr-en.lm.en.gz \
   news-commentary-v8.fr-en.blm.en

You can transform an arpa LM (*.arpa.en file) into an IRSTLM binary LM as follows:

 ~/irstlm/bin/compile-lm \
   news-commentary-v8.fr-en.arpa.en \
   news-commentary-v8.fr-en.blm.en

or viceversa, you can transform an IRSTLM binary LM into an arpa LM as follows:

 ~/irstlm/bin/compile-lm \
   --text yes \
   news-commentary-v8.fr-en.blm.en \
   news-commentary-v8.fr-en.arpa.en

This instead binarises (for faster loading) the *.arpa.en file using KenLM:

 ~/mosesdecoder/bin/build_binary \
   news-commentary-v8.fr-en.arpa.en \
   news-commentary-v8.fr-en.blm.en

You can check the language model by querying it, e.g.

 $ echo "is this an English sentence ?"                       \
   | ~/mosesdecoder/bin/query news-commentary-v8.fr-en.blm.en

三、训练翻译模型

首先看看参数:


Reference: All Training Parameters

  • --root-dir -- root directory, where output files are stored
  • --corpus -- corpus file name (full pathname), excluding extension
  • --e -- extension of the English corpus file
  • --f -- extension of the foreign corpus file
  • --lm -- language model: <factor>:<order>:<filename> (option can be repeated)
  • --first-step -- first step in the training process (default 1)
  • --last-step -- last step in the training process (default 7)
  • --parts -- break up corpus in smaller parts before GIZA++ training
  • --corpus-dir -- corpus directory (default $ROOT/corpus)
  • --lexical-dir -- lexical translation probability directory (default $ROOT/model)
  • --model-dir -- model directory (default $ROOT/model)
  • --extract-file -- extraction file (default $ROOT/model/extract)
  • --giza-f2e -- GIZA++ directory (default $ROOT/giza.$F-$E)
  • --giza-e2f -- inverse GIZA++ directory (default $ROOT/giza.$E-$F)
  • --alignment -- heuristic used for word alignment: intersect, union, grow, grow-final, grow-diag, grow-diag-final (default), grow-diag-final-and, srctotgt, tgttosrc
  • --max-phrase-length -- maximum length of phrases entered into phrase table (default 7)
  • --giza-option -- additional options for GIZA++ training
  • --verbose -- prints additional word alignment information
  • --no-lexical-weighting -- only use conditional probabilities for the phrase table, not lexical weighting
  • --parts -- prepare data for GIZA++ by running snt2cooc in parts
  • --direction -- run training step 2 only in direction 1 or 2 (for parallelization)
  • --reordering -- specifies which reordering models to train using a comma-separated list of config-strings, see FactoredTraining.BuildReorderingModel. (default distance)
  • --reordering-smooth -- specifies the smoothing constant to be used for training lexicalized reordering models. If the letter "u" follows the constant, smoothing is based on actual counts. (default 0.5)
  • --alignment-factors --
  • --translation-factors --
  • --reordering-factors --
  • --generation-factors --
  • --decoding-steps --

Moses创建一个翻译系统的基本过程记录,以后会按照每个过程详细说明,并给出每个步骤的参数说明的更多相关文章

  1. linux内核分析作业6:分析Linux内核创建一个新进程的过程

    task_struct结构: struct task_struct {   volatile long state;进程状态  void *stack; 堆栈  pid_t pid; 进程标识符  u ...

  2. 第六周——分析Linux内核创建一个新进程的过程

    "万子恵 + 原创作品转载请注明出处 + <Linux内核分析>MOOC课程http://mooc.study.163.com/course/USTC-1000029000 &q ...

  3. 分析Linux内核创建一个新进程的过程

    一.原理分析 1.进程的描述 进程控制块PCB——task_struct,为了管理进程,内核必须对每个进程进行清晰的描述,进程描述符提供了内核所需了解的进程信息. struct task_struct ...

  4. 分析Linux内核创建一个新进程的过程【转】

    转自:http://www.cnblogs.com/MarkWoo/p/4420588.html 前言说明 本篇为网易云课堂Linux内核分析课程的第六周作业,本次作业我们将具体来分析fork系统调用 ...

  5. 实验六:分析Linux内核创建一个新进程的过程

    原创作品转载请注明出处 + <Linux内核分析>MOOC课程http://mooc.study.163.com/course/USTC-1000029000 题目自拟,内容围绕对Linu ...

  6. 动手实践记录(利用django创建一个博客系统)

    1.添加一个分类的标签,和主表的关系是 外键 class Category(models.Model): """ 分类 """ name = ...

  7. 《Entity Framework 6 Recipes》翻译系列 (3) -----第二章 实体数据建模基础之创建一个简单的模型

    第二章 实体数据建模基础 很有可能,你才开始探索实体框架,你可能会问“我们怎么开始?”,如果你真是这样的话,那么本章就是一个很好的开始.如果不是,你已经建模,并在实体分裂和继承方面感觉良好,那么你可以 ...

  8. 无废话Android之listview入门,自定义的数据适配器、采用layoutInflater打气筒创建一个view对象、常用数据适配器ArrayAdapter、SimpleAdapter、使用ContentProvider(内容提供者)共享数据、短信的备份、插入一条记录到系统短信应用(3)

    1.listview入门,自定义的数据适配器 <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/and ...

  9. Linux内核创建一个新进程的过程

    “平安的祝福 + 原创作品转载请注明出处 + <Linux内核分析>MOOC课程http://mooc.study.163.com/course/USTC-1000029000 ” 进程在 ...

随机推荐

  1. CSS3 滤镜

    了解摄影活着美图秀秀之类美图软件的同学对滤镜肯定不陌生,CSS3对各种滤镜效果有了支持,可以做出很多好玩儿效果,走马观花了解一下 语法 filter: function(param); 很很多CSS3 ...

  2. iOS Xcode, 解决“Could not insert new outlet connection: Could not find any information for the class named”的问题。

    在Xcode中,我们可以在StoryBoard编辑界面或者是xib编辑界面中通过“Control键+拖拽“的方式将某个界面元素和对应的代码文件连接起来,在代码文件中创建outlet. 不过,如果你的运 ...

  3. 启动本地Oracle

    net start OracleOraDb10g_home1TNSListenernet start OracleServiceORCL第一个是监听服务第二个是数据库服务

  4. DB_MYSQL_mysql-5.7.10-winx64解压版安装笔记

    1.http://dev.mysql.com/downloads/mysql/ 里面下载Windows (x86, 64-bit), ZIP Archive mysql-5.7.10-winx64.z ...

  5. iOS:基于AVPlayer实现的视频播放器

    最近在学习AVFoundation框架的相关知识,写了一个基于AVPlayer的视频播放器,相关功能如下图: 代码github:https://github.com/wzpziyi1/VideoPla ...

  6. eclipse一直卡住,出现 “android sdk content loader 0%” 卡住的错误分析及解决方法

    分析:这种问题之前没有遇到过,也不知道什么原因,直接去网上查询,打开www.stackoverflow.com,输入要查询问题的关键词,我们输入 “android sdk content loader ...

  7. NumberFormat类的用法

    NumberFormat.getInstance()方法返回NumberFormat的一个实例(实际上是NumberFormat具体的一个子类,例如DecimalFormat), 这适合根据本地设置格 ...

  8. 锋利的jQuery-6--序列化函数serialize()和serializeArray()在表单提交中的作用

    在通过jQuery ajax提交表单的时候,通常用下边的方法获取表单内容. var form = 'add-account-form'; //表单id $('#' + form).submit(fun ...

  9. 替换SearchBar 键盘上的 搜索 按钮

    for (UIView *searchBarSubview in [searchBar subviews]) {        if ([searchBarSubview conformsToProt ...

  10. POJ #2479 - Maximum sum

    Hi, I'm back. This is a realy classic DP problem to code. 1. You have to be crystal clear about what ...