大数据采集与清洗

单击此处编辑母版标题样式,单击此处编辑母版文本样式,第二级,第三级,第四级,第五级,*,*,单击此处编辑母版标题样式,单击此处编辑母版文本样式,第二级,第三级,第四级,第五级,*,*,数据采集与清洗,2019|02|15,周,乐,什么是大数据,大数据处理流程,大数据的主要特征,大数据采集的概念,大数据采集应用,1,什么是大数据,淘宝推荐,依据购物行为偏好引荐,依据你最近的阅读行为和消费行为进行引荐,依据你用的设备往来不断猜特征.,依据时节改变进行引荐,2014-03,2015-08,2017-10,2016-03,2018,大数据工作首先写入政府工作报告,十三五规划纲要,提出,实施国家大数据战略,2018,年,政府工作报告,提出：实施大数据发展行动，注重用互联网、大数据等提升监管效能,国务院发布,促进大数据发展的行动纲要,十九大提出推动大数据战略，与实体经济深度融合,行业现状与前景,2019,年人社部拟最新发布,15,项新职业,1,.大数据工程技术人员,2,.云计算工程技术人员,3.,人工智能工程技术人员,4.,物联网工程技术人员,5.,什么是是大数数据,大数据据（BigData）,是指无无法使使用传传统和和常用用的软软件技技术和和工具具在一一定时时间内内完成成获取取、管管理和和处理理的数数据集集,2,大数据据的主主要特特征,大数据据主要要特征征,Volume,Velocity,Variety,Veracity,真实性性（Veracity），即即追求求高质质量的的数据据。,容量大大（Volume）,指大大规模模的数数据量量，并并且数数据量量呈持持续增增长趋趋势。,速度快快（Velocity）,指的的是数数据被被创建建和移移动的的速度度。,种类多多（Variety）,指数数据来来自多多种数数据源源，数数据种种类和和格式式。,Value,价值密密度低低（Value），指随随着数数据量量的增增长，数据据中有有意义义的信信息却却没有有成相相应比比例增增长。,3,大数据据处理理流程程,大数据据处理理流程程,数据预预处理理就是将将采集集来的的数据据从多多种数数据库库导入入到大大型的的分布布式数数据库库中（目前前主要要是hfds或或hive）,并同同时做做一些些简单单的清清洗和和预处处理工工作。,数据据统统计计分分析析就是是对对上上面面已已经经完完成成的的存存储储在在大大型型分分布布式式数数据据库库中中的的数数据据进进行行归归类类统统计计，可可以以满满足足一一般般场场景景的的分分析析需需求求。,数据据挖挖掘掘是对对数数据据进进行行基基于于各各种种算算法法的的分分析析计计算算，从从而而起起到到预预测测的的效效果果，实实现现一一些些高高级级别别数数据据分分析析的的需需求求。,数据据采采集集就是是利利用用多多种种数数据据库库（关关系系型型，NOSQL）去去存存储储不不同同来来源源的的数数据据。,数据据展展示示就是是对对以以上上处处理理完完的的结结果果进进行行分分析析，或或者者形形成成报报表表。,4,大数数据据采采集集的的概概念念,大数数据据采采集集的的概概念念,3,、大大数数据据采采集集技技术术方方法法,大数数据据采采集集技技术术就就是是对对数数据据进进行行ETL操操作作，通通过过对对数数据据进进行行提提取取、转转换换、加加载载，最最终终挖挖掘掘数数据据的的潜潜在在价价值值。ETL指指的的是是Extract-Transform-Load，也也就就是是抽抽取取、转转换换、加加载载。,抽取取-从从各各种种数数据据源源获获取取数数据据,转换换-按按需需求求格格式式将将源源数数据据转转换换为为目目标标数数据据,加载-把目标标数据加加载到数数据仓库库中,2、数据据采集与与大数据据采集的的区别,传统数据据采集：来源单单一，数数据量相相当小；结构单单一；关关系数据据库和并并行数据据库,大数据的的数据采采集：来来源广泛泛，数量量巨大；数据类类型丰富富；分布布式数据据库,1、什么么是数据据采集,数据采集集就是数数据获取取，数据据源主要要分为线线上数据据和内容容数据,大数据采采集系统统,1.日志志采集系系统（,Apache,Flume、Scribe）,3.数据据库采集集系统（关系型型、nosql等各种种数据库库）,2.网络络数据采采集系统统（Scrapy框框架,、Apache Nutch,）,5,大数据据采集集应用用,技能准准备,Python,基础,Linux,操作系系统基基本操操作,数据库库基础础（,SQL,语句操操作）,环境准准备,Python,Jdk(java,环境,),数据库库（,mysql,）,Thanks,YOURTITLE,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,YOURTITLE,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,2,OKPPT,工作室室,YOURTITLE,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,YOURTITLE,Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,YOUR TITLE,21%,9%,28%,42%,3,OKPPT,工作室,YOUR TITLE,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,YOUR TITLE,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.,YOUR TITLE,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.Nothing is difficulttotheman whowill try.,A vsB,Thanks,演讲完毕，谢谢谢观看！,