11G数据的hive初测试

作者：Jen覀nyfraaaaa-v- | 来源：互联网 | 2023-05-17 12:22

首先，看到标题，我想就会有人喷我，①11G对于hadoop而言不值一提②做耗时测试本没有必要，毕竟hadoop用于线下系统。我的想法：没那么复杂，只是想看下hadoop的文件处理能力和表

首先，看到标题，我想就会有人喷我，①11G对于hadoop而言不值一提②做耗时测试本没有必要，毕竟hadoop用于线下系统。

我的想法：没那么复杂，只是想看下hadoop的文件处理能力和表现出的问题，只是单纯的享受这个解决问题并且得到结果的快感。

不那么多废话啦!直接进入正题：

1、数据来源：

apache的日志，接近900万条数据，大小为11个G。

2、创建表，并用正则匹配的方式导入数据，(这里hive的正则和普通正则，路由不同，之后会发新的博客讨论这个问题：(hive的正则)http://blog.csdn.net/lengzijian/article/details/7048575)：
这里正则值提取的日志当中的：site：当前站点，get：当前站点请求页，fore：之前站点------主要用于点击流，淡然是简单版的。为了以后应付更大文件做准备。

CREATE TABLE click (site STRING, get STRING, fore STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ("input.regex" = ".*?- ([^ ]*) [GET]*[POST]* (/.*?) .*WEBTRENDS_ID=.* [http://]*(.+) .*","output.format.string" = "%1$s,%3$s,%2$s")STORED AS TEXTFILE;

3、导入数据，我这里有23个数据，分别代表一天的时间(可能少了一个小时)，可以把这23个数据连接在一起导入，但是我没有这样做，因为分别看每一个的时间，很容易的看到每个时间段的访问量，对于绘制监控图比较方便。

LOAD DATA LOCAL INPATH "/data/log/sdc_20110926_hr【00-22】.log" INTO TABLE click;

4、查询：因为刚刚做完没有经验，直接运行了

select count(*) from click;

出现以下信息：

Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapred.reduce.tasks=
Starting Job = job_201112071106_0005, Tracking URL = http://hadoop149:50030/jobdetails.jsp?jobid=job_201112071106_0005
Kill Command = /root/hadoop-0.20.2/bin/hadoop job  -Dmapred.job.tracker=hdfs://192.168.30.149:9004 -kill job_201112071106_0005
2011-12-07 17:03:55,880 Stage-1 map = 0%,  reduce = 0%
2011-12-07 17:04:04,936 Stage-1 map = 1%,  reduce = 0%
2011-12-07 17:04:15,040 Stage-1 map = 2%,  reduce = 0%
2011-12-07 17:04:17,057 Stage-1 map = 3%,  reduce = 0%
2011-12-07 17:04:24,103 Stage-1 map = 4%,  reduce = 0%
2011-12-07 17:04:26,121 Stage-1 map = 5%,  reduce = 0%
2011-12-07 17:04:36,185 Stage-1 map = 6%,  reduce = 0%
2011-12-07 17:04:38,199 Stage-1 map = 7%,  reduce = 0%
2011-12-07 17:04:44,239 Stage-1 map = 7%,  reduce = 2%
2011-12-07 17:04:50,275 Stage-1 map = 9%,  reduce = 2%
2011-12-07 17:04:59,333 Stage-1 map = 11%,  reduce = 3%
2011-12-07 17:05:08,386 Stage-1 map = 13%,  reduce = 3%
2011-12-07 17:05:14,425 Stage-1 map = 13%,  reduce = 4%
2011-12-07 17:05:17,450 Stage-1 map = 14%,  reduce = 4%
2011-12-07 17:05:21,476 Stage-1 map = 15%,  reduce = 4%
2011-12-07 17:05:25,505 Stage-1 map = 16%,  reduce = 4%
2011-12-07 17:05:28,530 Stage-1 map = 16%,  reduce = 5%
2011-12-07 17:05:34,568 Stage-1 map = 17%,  reduce = 5%
2011-12-07 17:05:37,588 Stage-1 map = 18%,  reduce = 5%
2011-12-07 17:05:43,629 Stage-1 map = 19%,  reduce = 6%
2011-12-07 17:05:46,648 Stage-1 map = 20%,  reduce = 6%
2011-12-07 17:05:55,705 Stage-1 map = 22%,  reduce = 6%
2011-12-07 17:05:58,724 Stage-1 map = 22%,  reduce = 7%
2011-12-07 17:06:04,761 Stage-1 map = 23%,  reduce = 7%
2011-12-07 17:06:07,783 Stage-1 map = 24%,  reduce = 7%
2011-12-07 17:06:16,839 Stage-1 map = 25%,  reduce = 8%
2011-12-07 17:06:18,853 Stage-1 map = 26%,  reduce = 8%
2011-12-07 17:06:25,900 Stage-1 map = 27%,  reduce = 8%
2011-12-07 17:06:31,937 Stage-1 map = 27%,  reduce = 9%
2011-12-07 17:06:33,951 Stage-1 map = 28%,  reduce = 9%
2011-12-07 17:06:36,973 Stage-1 map = 29%,  reduce = 9%
2011-12-07 17:06:43,010 Stage-1 map = 30%,  reduce = 9%
2011-12-07 17:06:47,039 Stage-1 map = 31%,  reduce = 10%
2011-12-07 17:06:55,088 Stage-1 map = 32%,  reduce = 10%
2011-12-07 17:06:56,095 Stage-1 map = 33%,  reduce = 10%
2011-12-07 17:07:03,140 Stage-1 map = 34%,  reduce = 10%
2011-12-07 17:07:08,172 Stage-1 map = 35%,  reduce = 11%
2011-12-07 17:07:14,210 Stage-1 map = 36%,  reduce = 11%
2011-12-07 17:07:18,241 Stage-1 map = 37%,  reduce = 11%
2011-12-07 17:07:23,275 Stage-1 map = 38%,  reduce = 12%
2011-12-07 17:07:29,312 Stage-1 map = 39%,  reduce = 12%
2011-12-07 17:07:32,331 Stage-1 map = 40%,  reduce = 13%
2011-12-07 17:07:38,373 Stage-1 map = 41%,  reduce = 13%
2011-12-07 17:07:41,392 Stage-1 map = 42%,  reduce = 13%
2011-12-07 17:07:47,429 Stage-1 map = 42%,  reduce = 14%
2011-12-07 17:07:48,436 Stage-1 map = 43%,  reduce = 14%
2011-12-07 17:07:51,456 Stage-1 map = 44%,  reduce = 14%
2011-12-07 17:07:56,497 Stage-1 map = 45%,  reduce = 14%
2011-12-07 17:08:02,533 Stage-1 map = 46%,  reduce = 14%
2011-12-07 17:08:08,568 Stage-1 map = 46%,  reduce = 15%
2011-12-07 17:08:09,575 Stage-1 map = 47%,  reduce = 15%
2011-12-07 17:08:12,594 Stage-1 map = 48%,  reduce = 15%
2011-12-07 17:08:18,634 Stage-1 map = 49%,  reduce = 15%
2011-12-07 17:08:23,664 Stage-1 map = 50%,  reduce = 16%
2011-12-07 17:08:29,701 Stage-1 map = 51%,  reduce = 16%
2011-12-07 17:08:32,718 Stage-1 map = 51%,  reduce = 17%
2011-12-07 17:08:35,736 Stage-1 map = 52%,  reduce = 17%
2011-12-07 17:08:39,761 Stage-1 map = 53%,  reduce = 17%
2011-12-07 17:08:44,791 Stage-1 map = 54%,  reduce = 17%
2011-12-07 17:08:51,832 Stage-1 map = 55%,  reduce = 17%
2011-12-07 17:08:53,845 Stage-1 map = 55%,  reduce = 18%
2011-12-07 17:08:56,864 Stage-1 map = 56%,  reduce = 18%
2011-12-07 17:09:00,887 Stage-1 map = 57%,  reduce = 18%
2011-12-07 17:09:05,920 Stage-1 map = 58%,  reduce = 18%
2011-12-07 17:09:08,939 Stage-1 map = 58%,  reduce = 19%
2011-12-07 17:09:14,974 Stage-1 map = 60%,  reduce = 19%
2011-12-07 17:09:24,029 Stage-1 map = 61%,  reduce = 20%
2011-12-07 17:09:27,049 Stage-1 map = 62%,  reduce = 20%
2011-12-07 17:09:33,086 Stage-1 map = 63%,  reduce = 20%
2011-12-07 17:09:36,105 Stage-1 map = 64%,  reduce = 20%
2011-12-07 17:09:39,124 Stage-1 map = 64%,  reduce = 21%
2011-12-07 17:09:45,160 Stage-1 map = 65%,  reduce = 21%
2011-12-07 17:09:46,167 Stage-1 map = 66%,  reduce = 21%
2011-12-07 17:09:54,216 Stage-1 map = 67%,  reduce = 22%
2011-12-07 17:09:58,241 Stage-1 map = 68%,  reduce = 22%
2011-12-07 17:10:03,272 Stage-1 map = 69%,  reduce = 22%
2011-12-07 17:10:09,309 Stage-1 map = 69%,  reduce = 23%
2011-12-07 17:10:10,316 Stage-1 map = 70%,  reduce = 23%
2011-12-07 17:10:15,345 Stage-1 map = 71%,  reduce = 23%
2011-12-07 17:10:21,382 Stage-1 map = 72%,  reduce = 23%
2011-12-07 17:10:24,401 Stage-1 map = 73%,  reduce = 24%
2011-12-07 17:10:30,438 Stage-1 map = 74%,  reduce = 24%
2011-12-07 17:10:33,457 Stage-1 map = 75%,  reduce = 24%
2011-12-07 17:10:39,497 Stage-1 map = 76%,  reduce = 24%
2011-12-07 17:10:42,517 Stage-1 map = 77%,  reduce = 24%
2011-12-07 17:10:45,537 Stage-1 map = 77%,  reduce = 25%
2011-12-07 17:10:51,589 Stage-1 map = 79%,  reduce = 25%
2011-12-07 17:11:00,645 Stage-1 map = 80%,  reduce = 26%
2011-12-07 17:11:06,683 Stage-1 map = 81%,  reduce = 27%
2011-12-07 17:11:09,700 Stage-1 map = 82%,  reduce = 27%
2011-12-07 17:11:18,755 Stage-1 map = 83%,  reduce = 27%
2011-12-07 17:11:21,774 Stage-1 map = 84%,  reduce = 27%
2011-12-07 17:11:30,854 Stage-1 map = 85%,  reduce = 28%
2011-12-07 17:11:33,874 Stage-1 map = 86%,  reduce = 28%
2011-12-07 17:11:39,912 Stage-1 map = 87%,  reduce = 28%
2011-12-07 17:11:45,949 Stage-1 map = 88%,  reduce = 29%
2011-12-07 17:11:48,970 Stage-1 map = 89%,  reduce = 29%
2011-12-07 17:11:55,008 Stage-1 map = 90%,  reduce = 29%
2011-12-07 17:11:58,028 Stage-1 map = 91%,  reduce = 29%
2011-12-07 17:12:01,048 Stage-1 map = 91%,  reduce = 30%
2011-12-07 17:12:04,067 Stage-1 map = 92%,  reduce = 30%
2011-12-07 17:12:07,088 Stage-1 map = 93%,  reduce = 30%
2011-12-07 17:12:13,127 Stage-1 map = 94%,  reduce = 31%
2011-12-07 17:12:19,167 Stage-1 map = 95%,  reduce = 31%
2011-12-07 17:12:22,186 Stage-1 map = 96%,  reduce = 31%
2011-12-07 17:12:25,205 Stage-1 map = 97%,  reduce = 31%
2011-12-07 17:12:28,225 Stage-1 map = 98%,  reduce = 31%
2011-12-07 17:12:31,251 Stage-1 map = 99%,  reduce = 31%
2011-12-07 17:12:34,275 Stage-1 map = 100%,  reduce = 33%
2011-12-07 17:12:40,312 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_201112071106_0005
OK
8625654
Time taken: 541.638 seconds

在执行过程中，一台电脑(子节点)一直再响，看了下cpu，双核几乎全被占满。最后结果运行了

OK8625654Time taken: 541.638 seconds

测试先做到这里，之后会继续更新。

推荐阅读

char
南邮ctf-web的writeup

本文介绍了南邮ctf-web的writeup，包括签到题和md5 collision。在CTF比赛和渗透测试中，可以通过查看源代码、代码注释、页面隐藏元素、超链接和HTTP响应头部来寻找flag或提示信息。利用PHP弱类型，可以发现md5('QNKCDZO')='0e830400451993494058024219903391'和md5('240610708')='0e462097431906509019562988736854'。 ... [详细]

蜡笔小新 2023-12-13 10:58:55
io
如何使用Java获取服务器硬件信息和磁盘负载率

本文介绍了使用Java编程语言获取服务器硬件信息和磁盘负载率的方法。首先在远程服务器上搭建一个支持服务端语言的HTTP服务，并获取服务器的磁盘信息，并将结果输出。然后在本地使用JS编写一个AJAX脚本，远程请求服务端的程序，得到结果并展示给用户。其中还介绍了如何提取硬盘序列号的方法。 ... [详细]

蜡笔小新 2023-12-14 13:56:20
input
HDFS2.x新特性

一、集群间数据拷贝scp实现两个远程主机之间的文件复制scp-rhello.txtroothadoop103:useratguiguhello.txt推pushscp-rr ... [详细]

蜡笔小新 2023-12-13 13:52:40
get
GetWindowLong函数

今天在看一个代码里头写了GetWindowLong(hwnd,0)，我当时就有点费解，靠，上网搜索函数原型说明，死活找不到第 ... [详细]

蜡笔小新 2023-12-14 17:58:15
match
PHP图片截取方法及应用实例

本文介绍了使用PHP动态切割JPEG图片的方法，并提供了应用实例，包括截取视频图、提取文章内容中的图片地址、裁切图片等问题。详细介绍了相关的PHP函数和参数的使用，以及图片切割的具体步骤。同时，还提供了一些注意事项和优化建议。通过本文的学习，读者可以掌握PHP图片截取的技巧，实现自己的需求。 ... [详细]

蜡笔小新 2023-12-14 16:44:09
input
【译】发送表单数据

这是原文链接：sendingformdata许多情况下，我们使用表单发送数据到服务器。服务器处理数据并返回响应给用户。这看起来很简单，但是 ... [详细]

蜡笔小新 2023-12-14 16:19:10
blob
数据库的存储结构及其重要性

本文介绍了数据库的存储结构及其重要性，强调了关系数据库范例中将逻辑存储与物理存储分开的必要性。通过逻辑结构和物理结构的分离，可以实现对物理存储的重新组织和数据库的迁移，而应用程序不会察觉到任何更改。文章还展示了Oracle数据库的逻辑结构和物理结构，并介绍了表空间的概念和作用。 ... [详细]

蜡笔小新 2023-12-14 16:00:02
match
android listview OnItemClickListener失效原因

最近在做listview时发现OnItemClickListener失效的问题，经过查找发现是因为button的原因。不仅listitem中存在button会影响OnItemClickListener事件的失效，还会导致单击后listview每个item的背景改变，使得item中的所有有关焦点的事件都失效。本文给出了一个范例来说明这种情况，并提供了解决方法。 ... [详细]

蜡笔小新 2023-12-14 14:25:50
io
Windows下配置PHP5.6的方法及注意事项

本文介绍了在Windows系统下配置PHP5.6的步骤及注意事项，包括下载PHP5.6、解压并配置IIS、添加模块映射、测试等。同时提供了一些常见问题的解决方法，如下载缺失的msvcr110.dll文件等。通过本文的指导，读者可以轻松地在Windows系统下配置PHP5.6，并解决一些常见的配置问题。 ... [详细]

蜡笔小新 2023-12-14 12:37:25
import
安卓select模态框样式改变_微软Office风格的多端（Web、安卓、iOS）组件库——Fabric UI...

介绍FabricUI是微软开源的一套Office风格的多端组件库，共有三套针对性的组件，分别适用于web、android以及iOS，Fab ... [详细]

蜡笔小新 2023-12-14 05:55:08
get
Linux下Perl的安装及第一个程序运行

本文介绍了在Linux下安装Perl的步骤，并提供了一个简单的Perl程序示例。同时，还展示了运行该程序的结果。 ... [详细]

蜡笔小新 2023-12-13 18:29:11
get
Web学习历程记录（七）——Tomcat基本概念和配置

本文介绍了Web学习历程记录中关于Tomcat的基本概念和配置。首先解释了Web静态Web资源和动态Web资源的概念，以及C/S架构和B/S架构的区别。然后介绍了常见的Web服务器，包括Weblogic、WebSphere和Tomcat。接着详细讲解了Tomcat的虚拟主机、web应用和虚拟路径映射的概念和配置过程。最后简要介绍了http协议的作用。本文内容详实，适合初学者了解Tomcat的基础知识。 ... [详细]

蜡笔小新 2023-12-13 17:08:24
get
ASP.NET Tips: 获取插入记录的ID的方法详解

本文详细介绍了在ASP.NET中获取插入记录的ID的几种方法，包括使用SCOPE_IDENTITY()和IDENT_CURRENT()函数，以及通过ExecuteReader方法执行SQL语句获取ID的步骤。同时，还提供了使用这些方法的示例代码和注意事项。对于需要获取表中最后一个插入操作所产生的ID或马上使用刚插入的新记录ID的开发者来说，本文提供了一些有用的技巧和建议。 ... [详细]

蜡笔小新 2023-12-13 17:03:18
io
Webmin远程命令执行漏洞复现及防护方法

本文介绍了Webmin远程命令执行漏洞CVE-2019-15107的漏洞详情和复现方法，同时提供了防护方法。漏洞存在于Webmin的找回密码页面中，攻击者无需权限即可注入命令并执行任意系统命令。文章还提供了相关参考链接和搭建靶场的步骤。此外，还指出了参考链接中的数据包不准确的问题，并解释了漏洞触发的条件。最后，给出了防护方法以避免受到该漏洞的攻击。 ... [详细]

蜡笔小新 2023-12-13 16:14:53
input
java 模拟get post请求_Java后台模拟发送http的get和post请求，并测试

个人学习使用：谨慎参考1Client类importcom.thoughtworks.gauge.Step;importcom.thoughtworks.gauge.T ... [详细]

蜡笔小新 2023-12-13 14:20:23

Jen覀nyfraaaaa-v-

这个家伙很懒，什么也没留下！

Tags | 热门标签

RankList | 热门文章