今天为了求解hiveserver占用内存过大的问题,特地加了hive在apache的邮件列表,讨论半天。特别说的是 里面的人确实很热情啊 ,外国人做事确实很认真,讨论帖发的时候都狠详细。

粘出一些记录:

Did you update your JDK in last time? A java-dev told me that could be
a issue in JDK _26
(https://forums.oracle.com/forums/thread.jspa?threadID=2309872), some
devs report a memory decrease when they use GC - flags. I'm quite not
sure, sounds for me to far away. The stacks have a lot waitings, but I see nothing special. - Alex 2011/12/12 王锋 <wfeng1982@163.com>:
>
> The hive log:
>
> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121840_767713480.txt
> 8159.581: [GC [PSYoungGen: 1927208K->688K(2187648K)]
> 9102425K->7176256K(9867648K), 0.0765670 secs] [Times: user=0.36 sys=0.00,
> real=0.08 secs]
> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121841_451939518.txt
> 8219.455: [GC [PSYoungGen: 1823477K->608K(2106752K)]
> 8999046K->7176707K(9786752K), 0.0719450 secs] [Times: user=0.66 sys=0.01,
> real=0.07 secs]
> Hive history file=/tmp/hdfs/hive_job_log_hdfs_201112121842_1930999319.txt
>
> Now we have 3 hiveservers and I set the concurrent job num to 4,but the Mem
> still be so large .I'm mad, God
>
> have other suggestions ?
>
> 在 2011-12-12 17:59:52,"alo alt" <wget.null@googlemail.com
>> 写道:
>>When you start a high-load hive query can you watch the stack-traces?
>>Its possible over the webinterface:
>>http://jobtracker:50030/stacks
>>
>>- Alex
>>
>>
>>2011/12/12 王锋 <wfeng1982@163.com>
>>>
>>> hiveserver will throw oom after several hours .
>>>
>>>
>>> At 2011-12-12 17:39:21,"alo alt" <wget.null@googlemail.com> wrote:
>>>
>>> what happen when you set xmx=2048m or similar? Did that have any negative effects for running queries?
>>>
>>> 2011/12/12 王锋 <wfeng1982@163.com>
>>>>
>>>> I have modify hive jvm args.
>>>> the new args is -Xmx15000m -XX:NewRatio=1 -Xms2000m .
>>>>
>>>> but the memory used by hiveserver is still large.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> At 2011-12-12 16:20:54,"Aaron Sun" <aaron.sun82@gmail.com> wrote:
>>>>
>>>> Not from the running jobs, what I am saying is the heap size of the Hadoop really depends on the number of files, directories on the HDFS. Remove old files periodically or merge small files would bring in some performance boost.
>>>>
>>>> On the Hive end, the memory consumed also depends on the queries that are executed. Monitor the reducers of the Hadoop job, and my experiences are that reduce part could be the bottleneck here.
>>>>
>>>> It's totally okay to host multiple Hive servers on one machine.
>>>>
>>>> 2011/12/12 王锋 <wfeng1982@163.com>
>>>>>
>>>>> is the files you said the files from runned jobs of our system? and them can't be so much large.
>>>>>
>>>>> why is the cause of namenode. what are hiveserver doing when it use so large memory?
>>>>>
>>>>> how do you use hive? our method using hiveserver is correct?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> 在 2011-12-12 14:27:09,"Aaron Sun" <aaron.sun82@gmail.com> 写道:
>>>>>
>>>>> Not sure if this is because of the number of files, since the namenode would track each of the file and directory, and blocks.
>>>>> See this one. http://www.cloudera.com/blog/2009/02/the-small-files-problem/
>>>>>
>>>>> Please correct me if I am wrong, because this seems to be more like a hdfs problem which is actually irrelevant to Hive.
>>>>>
>>>>> Thanks
>>>>> Aaron
>>>>>
>>>>> 2011/12/11 王锋 <wfeng1982@163.com>
>>>>>>
>>>>>>
>>>>>> I want to know why the hiveserver use so large memory,and where the memory has been used ?
>>>>>>
>>>>>> 在 2011-12-12 10:02:44,"王锋" <wfeng1982@163.com> 写道:
>>>>>>
>>>>>>
>>>>>> The namenode summary:
>>>>>>
>>>>>>
>>>>>>
>>>>>> the mr summary
>>>>>>
>>>>>>
>>>>>> and hiveserver:
>>>>>>
>>>>>>
>>>>>> hiveserver jvm args:
>>>>>> export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=1 -Xms15000m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParallelGC -XX:ParallelGCThreads=20 -XX:+UseParall
>>>>>> elOldGC -XX:-UseGCOverheadLimit -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>>>>>>
>>>>>> now we using 3 hiveservers in the same machine.
>>>>>>
>>>>>>
>>>>>> 在 2011-12-12 09:54:29,"Aaron Sun" <aaron.sun82@gmail.com> 写道:
>>>>>>
>>>>>> how's the data look like? and what's the size of the cluster?
>>>>>>
>>>>>> 2011/12/11 王锋 <wfeng1982@163.com>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm one of engieer of sina.com. We have used hive ,hiveserver several months. We have our own tasks schedule system .The system can schedule tasks running with hiveserver by jdbc.
>>>>>>>
>>>>>>> But The hiveserver use mem very large, usally large than 10g. we have 5min tasks which will be running every 5 minutes.,and have hourly tasks .total num of tasks is 40. And we start 3 hiveserver in one linux server,and be cycle connected .
>>>>>>>
>>>>>>> so why Memory of hiveserver using so large and how we do or some suggestion from you ?
>>>>>>>
>>>>>>> Thanks and Best Regards!
>>>>>>>
>>>>>>> Royce Wang

最上面   Alex发现一篇文章

https://forums.oracle.com/forums/thread.jspa?threadID=2309872说是 jdk_1.0.26存在泄露的风险,我们正在使用也正是同一个版本,看这个url里文章说的也是谁也不能确认,而oracle方自然说不由其负责。
I tried with java6u29 and java7 and they work great. Actually on the production server we are running for almost 4 days with java7 and it's stable, no crash, no slowdown, no restart in this period, and with less maximum memory. If it's going to last for a week then I trust it will go on fine.
最后是有人用java6u29 和java7 运行 稳定。特别是java7.明天尝试在hiveserver服务器换用java7试试。

append。。。。

今天改用jdk 7测试 情况基本一致,看来问题并不是 jvm问题。

使用jmap -heap 发现  hiveserver 新生代 并没有去按照ratio设置的 那样,最大容量还是默认的800m,这个对数据分析来说太小了,使用xmn配置新生代,并配置最大新生代大小,而且将gc机制改为cms,目前内存占用稳定在 2.3g左右。

最后的参数 :

export HADOOP_OPTS="$HADOOP_OPTS  -Xms5000m -Xmn4000m -XX:MaxNewSize=4000m -Xss128k  -XX:MaxHeapFreeRatio=80 -XX:MinHeapFreeRatio=40 -XX:+UseParNewGC -XX:+UseConcMarkSw
eepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:-UseGCOverheadLimit -XX:MaxTenuringThreshold=8 -XX:P
ermSize=800M -XX:MaxPermSize=800M -XX:GCTimeRatio=19 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

hiveserver 占用内存过大的问题的更多相关文章

  1. Unity3D占用内存太大的解决方法

    原地址:http://www.cnblogs.com/88999660/archive/2013/03/15/2961663.html 最近网友通过网站搜索Unity3D在手机及其他平台下占用内存太大 ...

  2. (转)Unity3D占用内存太大的解决方法

    自:http://www.cnblogs.com/88999660/archive/2013/03/15/2961663.html 最近网友通过网站搜索Unity3D在手机及其他平台下占用内存太大.  ...

  3. win7 64位的 svchost.exe 占用内存过大的问题

    svchost.exe 是用来启动系统服务的,所以某个 svchost.exe 占用内存过大,可能就是它启动的那个服务占用内存过大,所以只要停止并禁用那个服务就行了. 一般来说占用内存最大的服务是 S ...

  4. Unity3D占用内存太大怎么解决呢?

    最近网友通过网站搜索Unity3D在手机及其他平台下占用内存太大. 这里写下关于Unity3D对于内存的管理与优化. Unity3D 里有两种动态加载机制:一个是Resources.Load,另外一个 ...

  5. php 因循环数据 赋值变量 占用内存太大 提示错误

    Fatal error: Allowed memory size of 134217728 bytes exhausted 网上很多解决方法:就简单记录下 一个csv导入功能 由于数据太多 占用内存太 ...

  6. C# Winform应用程序占用内存较大解决方法整理(转)

    原文:http://www.jb51.net/article/56682.htm 背景: 微软的 .NET FRAMEWORK 现在可谓如火如荼了.但是,.NET 一直所为人诟病的就是“胃口太大”,狂 ...

  7. (转)C# Winform应用程序占用内存较大解决方法整理

    背景: 微软的 .NET FRAMEWORK 现在可谓如火如荼了.但是,.NET 一直所为人诟病的就是“胃口太大”,狂吃内存,虽然微软声称 GC 的功能和智能化都很高,但是内存的回收问题,一直存在困扰 ...

  8. C# Winform应用程序占用内存较大解决方法整理

     微软的 .NET FRAMEWORK 现在可谓如火如荼了.但是,.NET 一直所为人诟病的就是“胃口太大”,狂吃内存,虽然微软声称 GC 的功能和智能化都很高,但是内存的回收问题,一直存在困扰,尤其 ...

  9. SQL Server 2008 R2占用内存越来越大两种解决方法

    SQL Server 2008 R2运行越久,占用内存会越来越大. 第一种:有了上边的分析结果,解决方法就简单了,定期重启下SQL Server 2008 R2数据库服务即可,使用任务计划定期执行下边 ...

随机推荐

  1. oracle配置数据库连接方式

    有以下两种: (1)tnsname.ora中添加信息 例如: (2)在plsq  database下添加地址 例如:192.168.1.210:1521/orcl

  2. afterTextChanged() callback being called without the text being actually changed

    afterTextChanged() callback being called without the text being actually changed up vote8down votefa ...

  3. ES6学习笔记(九)

    1.概述 ES5的对象属性名都是字符串,这容易造成属性名的冲突.比如,你使用了一个他人提供的对象,但又想为这个对象添加新的方法(mixin模式),新方法的名字就有可能与现有方法产生冲突.如果有一种机制 ...

  4. sublime test2 快捷键

    快捷键比较全的:http://blog.useasp.net/archive/2013/06/14/sublime-text-2-all-default-Shortcuts-table-on-wind ...

  5. 细说mysql索引

    本文从如何建立mysql索引以及介绍mysql的索引类型,再讲mysql索引的利与弊,以及建立索引时需要注意的地方 首先:先假设有一张表,表的数据有10W条数据,其中有一条数据是nickname='c ...

  6. JAVA同步锁机制 wait() notify() notifyAll()

    wait() notify() notifyAll() 这3个方法用于协调多个线程对共享数据的存取,所以必须在synchronized语句块中使用. wait() 必须在synchronized函数或 ...

  7. 单调队列&amp;单调栈

    单调队列 例题: Poj 2823给定一个数列,从左至右输出每个长度为m的数列段内的最小数和最大数.数列长度:N<=106,m<=N 对于单调队列,我们这样子来定义: 1.维护区间最值 2 ...

  8. Android Volley 库通过网络获取 JSON 数据

    本文内容 什么是 Volley 库 Volley 能做什么 Volley 架构 环境 演示 Volley 库通过网络获取 JSON 数据 参考资料 Android 关于网络操作一般都会介绍 HttpC ...

  9. 浅学JavaScript

    JavaScript是互联网上最流行的脚本语言,可广泛用于服务器.PC.笔记本电脑智能手机等设备: 对事件的反应: <!DOCTYPE html> <html> <hea ...

  10. Java模拟按键

    JDK自带了Robot类,此类用于为测试自动化.自运行演示程序和其他需要控制鼠标和键盘的应用程序生成本机系统输入事件.Robot 的主要目的是便于 Java 平台实现自动测试. 详情可查看jdk1.6 ...