https://github.com/yandex/faster-rnnlm

 
 

Gdb ./rnnlm

r -rnnlm model-good.faster -train thread.title.good.train.txt -valid thread.title.good.valid.txt -hidden 5- -direct-order 3 -direct 200 -bptt 4 -bptt-block 10 -threads 1

[root@cq01-forum-rstree01.cq01.baidu.com faster-rnnlm]# more thread.title.good.train.txt

唉        稳        凉菜        干货        批发        稳        左成        个        月        都

咦        丢        图        跑

毕竟        新人

我        想去旅行

昨天        玩        个        满        深渊        人马                 才        踩        了        55

这        状态        还        不如        温网

新型        投资项目

晒        早饭        就        酱

渣土        哥        真是        太        放肆        了

推荐        就是        有        这样        的

白素贞        水        漫        文水        城

我知道        那些夏天        就像        你        一样        回        不

渑池        至        洛阳        最早        的        车        几        点        哪里        坐        到        洛阳        几点

宏观        方面        大        的        流动性        格局        虽无        明显        变化        但        眼下        地方        政府        债务        限

电工        行业        竞争        大        锦力        电器        有        优势

兄弟        啊                 影技        1        班        q        群        是        多少

你们        家乡        话        叫        什么

深深        的        孤独感        与        挫败        感        感觉        个人

一起去        旅游        吧

谁知道        四会        那里        有        修        打火机        的

[root@cq01-forum-rstree01.cq01.baidu.com faster-rnnlm]# pwd

/home/users/chenghuige/other/faster-rnnlm.debug/faster-rnnlm

  1. 统计词频建立vocabulary

void Vocabulary::BuildFromCorpus(const std::string& fpath, bool show_progress)

首先添加一个 </s>

AddWord(kEOSTag);
只是编号0

 
 

然后逐个添加每行

每行处理的时候按照IsSpace切分

inline bool IsSpace(char c) {

return c == ' ' || c == '\r' || c == '\t' || c == '\n';

 
 

然后其实就是对每个词
类似 Identifer.h那样顺序编号,没出现的词
叫做oov 编号 -1

 
 

除了编号之外
同时统计频次

最后按照频次排序
从大到小
同时更新编号
也就是频次最大的
这里 </s> 编号为0

(gdb) p words_

, word = 0x6ae1c0 "</s>"}, {

freq = 258246, word = 0x6aef20 "\265\304"}, {freq = 126910, word = 0x6aeff0 "\301\313"}, {

freq = 101904, word = 0x6aedc0 "\316\322"}, {freq = 67328, word = 0x6aeee0 "\323\320"}, {

freq = 62290, word = 0x6aec10 "\270\366"}, {freq = 60866, word = 0x6afb20 "\322\273"}, {

 
 

[root@cq01-forum-rstree01.cq01.baidu.com faster-rnnlm]# wc -l thread.title.good.train.txt

thread.title.good.train.txt

 
 

gdb) p cfg

$2 = {layer_size = 5, layer_count = 1, maxent_hash_size = 199947228, maxent_order = 3, use_nce = false, nce_lnz = 9, reverse_sentence = false, layer_type = "sigmoid"}

 
 

  1. 构建网格结构

main_nnet = new NNet(vocab, cfg, use_cuda, use_cuda_memory_efficient);

构造函数调用Init 在这里

 
 

embeddings.resize(vocab.size(), cfg.layer_size);

//(word_num, hidden_size) 二维数组

 
 

rec_layer = CreateLayer(cfg.layer_type, cfg.layer_size, cfg.layer_count);

//隐层
建立一个layer 默认layer_type是sigmoid

 
 

maxent_layer.Init(cfg.maxent_hash_size);

//最大熵
@TODO

 
 

softmax_layer = HSTree::CreateHuffmanTree(vocab, cfg.layer_size);

//输出层 softmax 采用huffman树

 
 

 
 

Faster-rnnlm代码分析1 - 词表构建,Nnet成员的更多相关文章

  1. 完整全面的Java资源库(包括构建、操作、代码分析、编译器、数据库、社区等等)

    构建 这里搜集了用来构建应用程序的工具. Apache Maven:Maven使用声明进行构建并进行依赖管理,偏向于使用约定而不是配置进行构建.Maven优于Apache Ant.后者采用了一种过程化 ...

  2. Android代码分析工具lint学习

    1 lint简介 1.1 概述 lint是随Android SDK自带的一个静态代码分析工具.它用来对Android工程的源文件进行检查,找出在正确性.安全.性能.可使用性.可访问性及国际化等方面可能 ...

  3. 常用 Java 静态代码分析工具的分析与比较

    常用 Java 静态代码分析工具的分析与比较 简介: 本文首先介绍了静态代码分析的基 本概念及主要技术,随后分别介绍了现有 4 种主流 Java 静态代码分析工具 (Checkstyle,FindBu ...

  4. angular代码分析之异常日志设计

    angular代码分析之异常日志设计 错误异常是面向对象开发中的记录提示程序执行问题的一种重要机制,在程序执行发生问题的条件下,异常会在中断程序执行,同时会沿着代码的执行路径一步一步的向上抛出异常,最 ...

  5. [Asp.net 5] DependencyInjection项目代码分析4-微软的实现(3)

    这个系列已经写了5篇,链接地址如下: [Asp.net 5] DependencyInjection项目代码分析 [Asp.net 5] DependencyInjection项目代码分析2-Auto ...

  6. wifi display代码 分析

    转自:http://blog.csdn.net/lilian0118/article/details/23168531 这一章中我们来看Wifi Display连接过程的建立,包含P2P的部分和RTS ...

  7. Device Tree(三):代码分析【转】

    转自:http://www.wowotech.net/linux_kenrel/dt-code-analysis.html Device Tree(三):代码分析 作者:linuxer 发布于:201 ...

  8. 集成TFS Build生成与SonarQube获取代码分析结果

    软件项目在开发过程中,往往由于任务重.时间紧等原因忽略软件代码的质量和规范检查,只注重软件功能的开发和交付.等软件交付上线以后,由于代码质量导致的问题会耗费开发和运维人员的大量时间.研发表明,项目上线 ...

  9. Device Tree(三):代码分析

    一.前言 Device Tree总共有三篇,分别是: 1.为何要引入Device Tree,这个机制是用来解决什么问题的?(请参考引入Device Tree的原因) 2.Device Tree的基础概 ...

随机推荐

  1. UNITY 之FixedUpdate

    这个机制的加入 比 AS3好了很多 AS3的EnterFrame相当于UNITY的Update 但是FLASH做不了也是因为浏览器的限制吧! Here's how the fixed time ste ...

  2. sqlserver 分离和附加

    1.何为数据库分离和附加 数据库分离: 分离数据库就是将某个数据库(如student_Mis)从SQL Server数据库列表中删除,使其不再被SQL Server管理和使用,但该数据库的文件(.MD ...

  3. ubuntu 解决中文zip乱码问题

    用到的工具是The Unarchiver项目提供的lsar/unar工具.The Unarchiver项目主页:http://code.google.com/p/theunarchiver/ 安装(1 ...

  4. UIButton 点击后变灰

    +(UIButton *)getBlueButtonWithTitle:(NSString *)aTitle{ UIButton *button = [UIButton buttonWithType: ...

  5. Varnish 4.0 实战(转)

    简介 Varnish 是一款高性能且开源的反向代理服务器和 HTTP 加速器,其采用全新的软件体系机构,和现在的硬件体系紧密配合,与传统的 squid 相比,varnish 具有性能更高.速度更快.管 ...

  6. CS小分队第一阶段冲刺站立会议(5月6日)

    冲刺阶段第一天 今日任务:完成游戏2048退出自动保存和进入自动读取功能,完善其他功能.

  7. js运动 模仿淘宝幻灯

    <!doctype html> <html> <head> <meta charset="utf-8"> <title> ...

  8. 【转】C#传委托给C的函数指针调用问题

    C#传委托给C的函数指针调用问题C代码如下: #include "stdio.h" __declspec(dllexport) int Call(int (*qq)(int num ...

  9. jqGrid API 相关

    取消所有选中的行: $("jqgridtableid").trigger("reloadGrid"): 设定选中行,可设定多行选中: $("jqgri ...

  10. vs之Nuget

    1.http://msdn.microsoft.com/zh-cn/magazine/hh547106.aspx 2.在Nuget控制台中安装特定版本: Install-Package <程序包 ...