广告精准投放的代码封装优化及parquet文件分区操作案例

上一节实践DSP广告数据平台的日志文件转换成parquet文件第二种方案

现在主要对该工程进行代码封装优化&＃xff0c;在将来如果使用的是spark 1.6&＃43;可是用样例类来封装数据&＃xff0c;如下优化代码主要有&＃xff1a;

代码封装优化

1、首先&＃xff0c;编写封装创建sparkContext实例

dolphin-doit01\src\main\scala\cn\sheep\dolphin\common\DolphinAppComm.scala

package cn.sheep.dolphin.commonimport cn.sheep.dolphin.config.DolphinConfig
import org.apache.spark.{SparkConf, SparkContext}
import cn.sheep.dolphin.common.DolphinEnum._
/*** author: old sheep* Created 2021/04/03 11:24*/
object DolphinAppComm {/*** 封装创建sparkContext实例** &＃64;param appName* &＃64;param params* &＃64;return*/def createSparkContext(appName: String, params: Map[String, String] &＃61; Map.empty) &＃61; {val sparkConf &＃61; new SparkConf()sparkConf.setAppName(appName)if (DolphinConfig._power.equals(local.toString)) { // 运行模式的开关sparkConf.setMaster("local[*]")}// 封装用户传递进来的参数params.foreach {case (key, value) &＃61;> sparkConf.set(key, value)}new SparkContext(sparkConf)}
}

 2、配置全局文件

dolphin-doit01\src\main\resources\application.conf

# 设置一个程序运行时的开关&＃xff0c;控制成是否是本地运行还是集群运行模式(cluster: 集群|local:本机)
dolphin.common.power&＃61;local# mysql的配置信息
dolphin.mysql.driver&＃61;com.mysql.jdbc.Driver
dolphin.mysql.url&＃61;"jdbc:mysql:///spark?characterEncoding&＃61;utf-8"
dolphin.mysql.username&＃61;root
dolphin.mysql.password&＃61;123456

3、定义枚举类

dolphin-doit01\src\main\scala\cn\sheep\dolphin\common\DolphinEnum.scala

package cn.sheep.dolphin.common/*** author: old sheep* Created 2021/04/03 11:31*/
object DolphinEnum extends Enumeration{type DolphinEnum &＃61; Valueval local, cluster &＃61; Value
}

4、编写解析application.conf配置文件

dolphin-doit01\src\main\scala\cn\sheep\dolphin\config\DolphinConfig.scala

package cn.sheep.dolphin.configimport java.util.Properties
import com.typesafe.config.{Config, ConfigFactory}
/*** 解析application.conf配置文件* author: old sheep* Created 2021/04/03 11:29*/
object DolphinConfig {private lazy val application &＃61; ConfigFactory.load()/* 解析运行模式 */val _power &＃61; application.getString("dolphin.common.power")/* 解析mysql配置 */val _driver &＃61; application.getString("dolphin.mysql.driver")val _url &＃61; application.getString("dolphin.mysql.url")val _username &＃61; application.getString("dolphin.mysql.username")val _password &＃61; application.getString("dolphin.mysql.password")/* mysql的属性参数封装 */val props &＃61; new Properties()props.setProperty("driver", _driver)props.setProperty("user", _username)props.setProperty("password", _password)}

优化Bz2Parquet类为&＃xff1a;

//读取离线的数据文件的// val sc &＃61; new SparkContext("")val sc &＃61; DolphinAppComm.createSparkContext("将bz2日志文件转换成parquet文件")// 读取离线的bz2日志文件val data &＃61; sc.textFile(bz2InputPath)// 过滤非法数据val filteredRDD: RDD[Array[String]] &＃61; data.map(_.split(",", -1)).filter(_.size >&＃61; 85)// parquet <- DataFrame(几种创建方式) <- SQLContext <- RDDval sqlc &＃61; new SQLContext(sc)

优化Bz2ParquetV2类为&＃xff1a;

// spark序列化优化参数val sparkParams &＃61; Map[String, String]("spark.serializer" -> classOf[KryoSerializer].getName,"spark.sql.parquet.compression.codec" -> "snappy" // 设置parquet文件的压缩格式)val sc &＃61; DolphinAppComm.createSparkContext("将bz2日志文件转换成parquet文件",sparkParams)

parquet文件分区操作

在Bz2ParquetV2类&＃xff0c;进行对partitionBy 对数据按照省份进行分区设置&＃xff1a;

// partitionBy 对数据按照省份进行分区dataFrame.write.partitionBy("provincename").parquet(parquetOutPath)


运行Bz2ParquetV2类程序&＃xff0c;控制台打印输出日志信息&＃xff1a;


21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;江西省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;河北省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;河南省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;浙江省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;海南省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;宁夏回族自治区
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;湖北省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;安徽省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;山东省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;山西省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;福建省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;广东省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;西藏自治区
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;湖南省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;贵州省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;辽宁省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;上海市
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;重庆市
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;云南省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;陕西省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;青海省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;香港特别行政区
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;内蒙古自治区
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;北京市
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;台湾省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;吉林省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;四川省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;天津市
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;澳门特别行政区
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;广西壮族自治区
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;新疆维吾尔自治区
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;甘肃省
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;未知
21/04/03 12:30:46 INFO HadoopFsRelation: Listing file:/C:/Users/mrman/dolphin-doit01/parquet/provincename&＃61;黑龙江省


在自己的磁盘文件里可以查看