第一届自然语言处理与中文计算会议课件

微博情感分析评测总结第一届自然语言处理与中文计算会议2019年4月21日1 1北京大学计算机科学技术研究所北京大学计算机科学技术研究所2 2苏州大学计算机科学与技术学院苏州大学计算机科学与技术学院3 3哈工大深圳研究生院计算机科学与技术学科部哈工大深圳研究生院计算机科学与技术学科部xxxxxx1关注我呗2019年4月21微博情感分析评测总结第一届自然语言处理与中文计算会议2019任务设置观点句识别情感倾向性判断针对观点句情感要素抽取针对观点句2关注我呗2019年4月21任务设置观点句识别2关注我呗2019年4月21数据集来自腾讯微博数据原始数据集20话题主要是热门事件，如“三亚春节宰客”约17500条微博约32000个句子3关注我呗2019年4月21数据集来自腾讯微博数据3关注我呗2019年4月21数据集数据标注三标注者对同一数据标注数据集大小2023条微博3416个句子1209个非观点句2207个观点句407正面、1766负面、34其他2361个对象441正面、1910负面、10其他4关注我呗2019年4月21数据集数据标注4关注我呗2019年4月21数据集标注标准关于观点句个人意愿、心情不是观点句，如“我感到很高兴”表达了明显观点的反问句属于观点句，如“体育竞技有不残酷的吗？”关于观点句倾向性正面、负面，其他(Other)三类，“其他”表示中性或无法确定正负的情况不进行5关注我呗2019年4月21数据集标注标准5关注我呗2019年4月21数据集标注标准关于观点句情感对象情感对象优先从当前句子抽取，但可以跨越当前句子从前面的句子或后面的句子中抽取（前面的邻近句子优先）情感对象尽可能完整和明确，事件按细粒度标注：“ipad的屏幕很棒！”-“ipad的屏幕”“官二代陶汝坤:作案动机极其卑劣”-“作案动机”对象本身不是情感表达：”#官二代求爱不成将少女毁容#这种畜生是怎么被教育出来的啊！”-“官二代”人称代词需要尽可能在当前微博内进行指代消解：“小明就读于北京大学，他是名优秀的学生。”-”小明“抽取出句子中每个情感片段所对应的情感对象：“你根本已经不是个人了，你比蛇还冷血，你比畜生还畜生。”-“你”“你”“你”6关注我呗2019年4月21数据集标注标准6关注我呗2019年4月21参评单位34支队伍包括全国重点高校与科研机构观点句识别：53组情感倾向判断：48组情感对象抽取：22组7关注我呗2019年4月21参评单位34支队伍7关注我呗2019年4月21观点句识别任务要求识别每条微博中各个句子是观点句还是非观点句评价标准准确率、召回率、F值微平均宏平均8关注我呗2019年4月21观点句识别任务要求8关注我呗2019年4月21观点句识别微平均RecallRecallPrecisionPrecision1.0000.8000.6000.4000.2000.0000.5000.5500.6000.6500.7000.7500.8000.8500.9000.9501.000Series1,0.691,0.743Series1,0.746,0.733Series1,0.918,0.681Series1,0.577,0.707Series1,0.773,0.673Series1,0.694,0.742Series1,0.557,0.740Series1,0.528,0.734Series1,0.603,0.825Series1,0.592,0.822Series1,0.449,0.835Series1,0.726,0.738Series1,0.726,0.738Series1,0.378,0.619Series1,0.378,0.619Series1,0.757,0.647Series1,0.537,0.828Series1,0.502,0.728Series1,0.439,0.747Series1,0.406,0.781Series1,0.772,0.746Series1,0.656,0.707Series1,0.733,0.700Series1,0.536,0.737Series1,0.473,0.695Series1,0.406,0.745Series1,0.406,0.745Series1,0.717,0.714Series1,0.745,0.715Series1,0.646,0.740Series1,0.683,0.733Series1,0.944,0.671Series1,0.944,0.671Series1,0.588,0.805Series1,0.789,0.745Series1,0.891,0.674Series1,0.871,0.660Series1,0.562,0.704Series1,0.632,0.725Series1,0.649,0.708Series1,0.221,0.638Series1,0.338,0.783Series1,0.455,0.780Series1,0.348,0.696Series1,0.959,0.645Series1,0.647,0.765Series1,0.542,0.779Series1,0.802,0.756Series1,0.812,0.756Series1,0.119,0.773Series1,0.658,0.728Series1,0.716,0.716Series1,0.334,0.7019关注我呗2019年4月21观点句识别微平均9关注我呗2019年4月21观点句识别微平均F-measureF-measure百百分分比比0.00-0.050.05-0.100.10-0.150.15-0.200.20-0.250.25-0.300.30-0.350.35-0.400.40-0.450.45-0.500.50-0.550.55-0.600.60-0.650.65-0.700.70-0.750.75-0.800.80-0.850.85-0.900.90-0.950.95-1.000.0%5.0%10.0%15.0%20.0%25.0%Series1,0.00-0.05,0Series1,0.05-0.10,0Series1,0.10-0.15,0Series1,0.15-0.20,0Series1,0.20-0.25,0.018867925Series1,0.25-0.30,0Series1,0.30-0.35,0.018867925Series1,0.35-0.40,0Series1,0.40-0.45,0Series1,0.45-0.50,0.094339623Series1,0.50-0.55,0.056603774Series1,0.55-0.60,0.094339623Series1,0.60-0.65,0.113207547Series1,0.65-0.70,0.188679245Series1,0.70-0.75,0.226415094Series1,0.75-0.80,0.188679245Series1,0.80-0.85,0Series1,0.85-0.90,0Series1,0.90-0.95,0Series1,0.95-1.00,010关注我呗2019年4月21观点句识别微平均10关注我呗2019年4月21观点句识别宏平均RecallRecallPrecisionPrecision1.0000.9000.8000.7000.6000.5000.4000.3000.2000.1000.0000.5000.5500.6000.6500.7000.7500.8000.8500.9000.9501.000Series1,0.680,0.745Series1,0.735,0.734Series1,0.914,0.684Series1,0.577,0.715Series1,0.774,0.680Series1,0.684,0.706Series1,0.541,0.733Series1,0.514,0.724Series1,0.589,0.828Series1,0.581,0.824Series1,0.435,0.836Series1,0.717,0.743Series1,0.717,0.743Series1,0.383,0.601Series1,0.383,0.601Series1,0.751,0.648Series1,0.520,0.826Series1,0.502,0.736Series1,0.431,0.741Series1,0.395,0.776Series1,0.757,0.747Series1,0.651,0.713Series1,0.734,0.705Series1,0.522,0.743Series1,0.461,0.695Series1,0.394,0.742Series1,0.394,0.742Series1,0.708,0.722Series1,0.738,0.721Series1,0.639,0.744Series1,0.678,0.737Series1,0.942,0.674Series1,0.942,0.674Series1,0.581,0.807Series1,0.782,0.748Series1,0.892,0.679Series1,0.869,0.663Series1,0.557,0.699Series1,0.618,0.723Series1,0.634,0.708Series1,0.217,0.630Series1,0.337,0.792Series1,0.443,0.781Series1,0.348,0.686Series1,0.960,0.649Series1,0.641,0.760Series1,0.529,0.767Series1,0.788,0.758Series1,0.797,0.757Series1,0.112,0.766Series1,0.651,0.732Series1,0.712,0.719Series1,0.341,0.70711关注我呗2019年4月21观点句识别宏平均11关注我呗2019年4月21观点句识别宏平均F-measureF-measure百百分分比比0.00-0.050.05-0.100.10-0.150.15-0.200.20-0.250.25-0.300.30-0.350.35-0.400.40-0.450.45-0.500.50-0.550.55-0.600.60-0.650.65-0.700.70-0.750.75-0.800.80-0.850.85-0.900.90-0.950.95-1.000.0%5.0%10.0%15.0%20.0%25.0%Series1,0.00-0.05,0Series1,0.05-0.10,0Series1,0.10-0.15,0Series1,0.15-0.20,0.018867925Series1,0.20-0.25,0Series1,0.25-0.30,0Series1,0.30-0.35,0.018867925Series1,0.35-0.40,0Series1,0.40-0.45,0.056603774Series1,0.45-0.50,0.037735849Series1,0.50-0.55,0.094339623Series1,0.55-0.60,0.075471698Series1,0.60-0.65,0.113207547Series1,0.65-0.70,0.20754717Series1,0.70-0.75,0.226415094Series1,0.75-0.80,0.150943396Series1,0.80-0.85,0Series1,0.85-0.90,0Series1,0.90-0.95,0Series1,0.95-1.00,012关注我呗2019年4月21观点句识别宏平均12关注我呗2019年4月21情感倾向性判断任务要求判断微博中每条观点句的情感倾向正面、负面、其他评价标准准确率、召回率、F值微平均宏平均13关注我呗2019年4月21情感倾向性判断任务要求13关注我呗2019年4月21情感倾向性判断微平均RecallRecallPrecisionPrecision0.0000.1000.2000.3000.4000.5000.6000.7000.8000.9000.0000.1000.2000.3000.4000.5000.6000.7000.8000.9001.000Series1,0.574,0.831Series1,0.614,0.824Series1,0.698,0.761Series1,0.440,0.764Series1,0.568,0.734Series1,0.565,0.782Series1,0.403,0.724Series1,0.379,0.718Series1,0.507,0.841Series1,0.493,0.833Series1,0.426,0.426Series1,0.640,0.881Series1,0.626,0.863Series1,0.097,0.258Series1,0.099,0.261Series1,0.559,0.559Series1,0.415,0.772Series1,0.406,0.809Series1,0.262,0.598Series1,0.241,0.594Series1,0.614,0.796Series1,0.344,0.643Series1,0.379,0.803Series1,0.399,0.647Series1,0.565,0.788Series1,0.591,0.794Series1,0.478,0.740Series1,0.495,0.725Series1,0.481,0.893Series1,0.631,0.886Series1,0.587,0.587Series1,0.850,0.850Series1,0.389,0.691Series1,0.511,0.809Series1,0.480,0.740Series1,0.184,0.832Series1,0.288,0.288Series1,0.400,0.879Series1,0.266,0.803Series1,0.771,0.804Series1,0.584,0.902Series1,0.464,0.857Series1,0.675,0.842Series1,0.685,0.844Series1,0.108,0.108Series1,0.341,0.476Series1,0.341,0.476Series1,0.150,0.45014关注我呗2019年4月21情感倾向性判断微平均14关注我呗2019年4月21情感倾向性判断微平均F-measureF-measure百百分分比比0.00-0.050.05-0.100.10-0.150.15-0.200.20-0.250.25-0.300.30-0.350.35-0.400.40-0.450.45-0.500.50-0.550.55-0.600.60-0.650.65-0.700.70-0.750.75-0.800.80-0.850.85-0.900.90-0.950.95-1.000.0%2.0%4.0%6.0%8.0%10.0%12.0%14.0%16.0%Series1,0.00-0.05,0Series1,0.05-0.10,0Series1,0.10-0.15,0.0625Series1,0.15-0.20,0Series1,0.20-0.25,0.020833333Series1,0.25-0.30,0.020833333Series1,0.30-0.35,0.041666667Series1,0.35-0.40,0.083333333Series1,0.40-0.45,0.041666667Series1,0.45-0.50,0.0625Series1,0.50-0.55,0.104166667Series1,0.55-0.60,0.125Series1,0.60-0.65,0.125Series1,0.65-0.70,0.104166667Series1,0.70-0.75,0.145833333Series1,0.75-0.80,0.041666667Series1,0.80-0.85,0.020833333Series1,0.85-0.90,0Series1,0.90-0.95,0Series1,0.95-1.00,015关注我呗2019年4月21情感倾向性判断微平均15关注我呗2019年4月21情感倾向性判断宏平均RecallRecallPrecisionPrecision0.0000.1000.2000.3000.4000.5000.6000.7000.8000.9000.0000.1000.2000.3000.4000.5000.6000.7000.8000.9001.000Series1,0.563,0.823Series1,0.608,0.825Series1,0.702,0.768Series1,0.445,0.758Series1,0.574,0.738Series1,0.562,0.783Series1,0.387,0.708Series1,0.365,0.703Series1,0.497,0.849Series1,0.487,0.843Series1,0.413,0.413Series1,0.632,0.878Series1,0.619,0.860Series1,0.105,0.341Series1,0.107,0.342Series1,0.561,0.561Series1,0.404,0.776Series1,0.407,0.791Series1,0.253,0.583Series1,0.232,0.578Series1,0.600,0.789Series1,0.335,0.641Series1,0.370,0.800Series1,0.390,0.641Series1,0.562,0.780Series1,0.590,0.786Series1,0.472,0.734Series1,0.490,0.725Series1,0.481,0.895Series1,0.630,0.888Series1,0.579,0.579Series1,0.854,0.854Series1,0.387,0.693Series1,0.496,0.799Series1,0.465,0.731Series1,0.181,0.829Series1,0.289,0.289Series1,0.391,0.872Series1,0.263,0.733Series1,0.778,0.809Series1,0.578,0.899Series1,0.452,0.855Series1,0.663,0.840Series1,0.672,0.842Series1,0.102,0.102Series1,0.329,0.459Series1,0.329,0.459Series1,0.151,0.43516关注我呗2019年4月21情感倾向性判断宏平均16关注我呗2019年4月21情感倾向性判断宏平均F-measureF-measure百百分分比比0.00-0.050.05-0.100.10-0.150.15-0.200.20-0.250.25-0.300.30-0.350.35-0.400.40-0.450.45-0.500.50-0.550.55-0.600.60-0.650.65-0.700.70-0.750.75-0.800.80-0.850.85-0.900.90-0.950.95-1.000.0%2.0%4.0%6.0%8.0%10.0%12.0%14.0%16.0%Series1,0.00-0.05,0Series1,0.05-0.10,0Series1,0.10-0.15,0.0625Series1,0.15-0.20,0Series1,0.20-0.25,0.020833333Series1,0.25-0.30,0.041666667Series1,0.30-0.35,0.020833333Series1,0.35-0.40,0.083333333Series1,0.40-0.45,0.041666667Series1,0.45-0.50,0.083333333Series1,0.50-0.55,0.083333333Series1,0.55-0.60,0.145833333Series1,0.60-0.65,0.125Series1,0.65-0.70,0.125Series1,0.70-0.75,0.125Series1,0.75-0.80,0.020833333Series1,0.80-0.85,0Series1,0.85-0.90,0.020833333Series1,0.90-0.95,0Series1,0.95-1.00,017关注我呗2019年4月21情感倾向性判断宏平均17关注我呗2019年4月21情感要素抽取任务要求找出微博中每条观点句作者的评价对象判断针对情感对象的观点极性评价标准精确评价宽松评价准确率、召回率、F值18关注我呗2019年4月21情感要素抽取任务要求18关注我呗2019年4月21情感要素抽取严格评价微平均RecallRecallPrecisionPrecision0.0000.0500.1000.1500.2000.2500.3000.0000.1000.2000.3000.4000.5000.600Series1,0.147,0.066Series1,0.102,0.065Series1,0.039,0.028Series1,0.066,0.485Series1,0.077,0.425Series1,0.177,0.311Series1,0.005,0.081Series1,0.089,0.120Series1,0.100,0.182Series1,0.160,0.160Series1,0.112,0.112Series1,0.003,0.006Series1,0.078,0.135Series1,0.071,0.120Series1,0.027,0.126Series1,0.062,0.106Series1,0.164,0.260Series1,0.058,0.117Series1,0.273,0.303Series1,0.275,0.303Series1,0.063,0.241Series1,0.009,0.03119关注我呗2019年4月21情感要素抽取严格评价19关注我呗2019年4月21情感要素抽取严格评价微平均F-measureF-measure百百分分比比0.00-0.050.05-0.100.10-0.150.15-0.200.20-0.250.25-0.300.30-0.350.35-0.400.40-0.450.45-0.500.50-0.550.55-0.600.60-0.650.65-0.700.70-0.750.75-0.800.80-0.850.85-0.900.90-0.950.95-1.000.0%5.0%10.0%15.0%20.0%25.0%30.0%35.0%Series1,0.00-0.05,0.227272727Series1,0.05-0.10,0.318181818Series1,0.10-0.15,0.227272727Series1,0.15-0.20,0.045454545Series1,0.20-0.25,0.090909091Series1,0.25-0.30,0.090909091Series1,0.30-0.35,0Series1,0.35-0.40,0Series1,0.40-0.45,0Series1,0.45-0.50,0Series1,0.50-0.55,0Series1,0.55-0.60,0Series1,0.60-0.65,0Series1,0.65-0.70,0Series1,0.70-0.75,0Series1,0.75-0.80,0Series1,0.80-0.85,0Series1,0.85-0.90,0Series1,0.90-0.95,0Series1,0.95-1.00,020关注我呗2019年4月21情感要素抽取严格评价20关注我呗2019年4月21情感要素抽取严格评价宏平均RecallRecallPrecisionPrecision0.0000.0500.1000.1500.2000.2500.3000.0000.0500.1000.1500.2000.2500.3000.3500.4000.4500.500Series1,0.144,0.070Series1,0.101,0.068Series1,0.040,0.031Series1,0.066,0.474Series1,0.076,0.417Series1,0.174,0.324Series1,0.005,0.031Series1,0.090,0.133Series1,0.101,0.196Series1,0.169,0.172Series1,0.118,0.121Series1,0.003,0.006Series1,0.081,0.149Series1,0.073,0.125Series1,0.027,0.121Series1,0.064,0.109Series1,0.172,0.271Series1,0.062,0.123Series1,0.263,0.306Series1,0.265,0.305Series1,0.058,0.244Series1,0.010,0.02921关注我呗2019年4月21情感要素抽取严格评价21关注我呗2019年4月21情感要素抽取严格评价宏平均F-measureF-measure百百分分比比0.00-0.050.05-0.100.10-0.150.15-0.200.20-0.250.25-0.300.30-0.350.35-0.400.40-0.450.45-0.500.50-0.550.55-0.600.60-0.650.65-0.700.70-0.750.75-0.800.80-0.850.85-0.900.90-0.950.95-1.000.0%5.0%10.0%15.0%20.0%25.0%30.0%Series1,0.00-0.05,0.227272727Series1,0.05-0.10,0.272727273Series1,0.10-0.15,0.272727273Series1,0.15-0.20,0.045454545Series1,0.20-0.25,0.090909091Series1,0.25-0.30,0.090909091Series1,0.30-0.35,0Series1,0.35-0.40,0Series1,0.40-0.45,0Series1,0.45-0.50,0Series1,0.50-0.55,0Series1,0.55-0.60,0Series1,0.60-0.65,0Series1,0.65-0.70,0Series1,0.70-0.75,0Series1,0.75-0.80,0Series1,0.80-0.85,0Series1,0.85-0.90,0Series1,0.90-0.95,0Series1,0.95-1.00,022关注我呗2019年4月21情感要素抽取严格评价22关注我呗2019年4月21情感要素抽取宽松评价微平均RecallRecallPrecisionPrecision0.0000.0500.1000.1500.2000.2500.3000.3500.4000.0000.1000.2000.3000.4000.5000.6000.700Series1,0.325,0.264Series1,0.213,0.241Series1,0.073,0.086Series1,0.086,0.636Series1,0.099,0.569Series1,0.223,0.404Series1,0.007,0.137Series1,0.138,0.198Series1,0.140,0.279Series1,0.220,0.290Series1,0.189,0.287Series1,0.028,0.107Series1,0.105,0.232Series1,0.114,0.254Series1,0.040,0.257Series1,0.098,0.236Series1,0.249,0.398Series1,0.110,0.201Series1,0.354,0.388Series1,0.356,0.387Series1,0.082,0.422Series1,0.019,0.09123关注我呗2019年4月21情感要素抽取宽松评价23关注我呗2019年4月21情感要素抽取宽松评价微平均F-measureF-measure百百分分比比0.00-0.050.05-0.100.10-0.150.15-0.200.20-0.250.25-0.300.30-0.350.35-0.400.40-0.450.45-0.500.50-0.550.55-0.600.60-0.650.65-0.700.70-0.750.75-0.800.80-0.850.85-0.900.0%5.0%10.0%15.0%20.0%25.0%Series1,0.136363636Series1,0.090909091Series1,0.00-0.05,0.181818182Series1,0.05-0.10,0.227272727Series1,0.10-0.15,0.136363636Series1,0.15-0.20,0.090909091Series1,0.20-0.25,0.045454545Series1,0.25-0.30,0.090909091Series1,0.30-0.35,0Series1,0.35-0.40,0Series1,0.40-0.45,0Series1,0.45-0.50,0Series1,0.50-0.55,0Series1,0.55-0.60,0Series1,0.60-0.65,0Series1,0.65-0.70,0Series1,0.70-0.75,0Series1,0.75-0.80,0Series1,0.80-0.85,0Series1,0.85-0.90,024关注我呗2019年4月21情感要素抽取宽松评价24关注我呗2019年4月21情感要素抽取宽松评价宏平均RecallRecallPrecisionPrecision0.0000.0500.1000.1500.2000.2500.3000.3500.4000.0000.1000.2000.3000.4000.5000.6000.700Series1,0.330,0.269Series1,0.219,0.246Series1,0.074,0.092Series1,0.086,0.643Series1,0.098,0.572Series1,0.219,0.420Series1,0.007,0.071Series1,0.139,0.219Series1,0.138,0.286Series1,0.229,0.302Series1,0.196,0.300Series1,0.029,0.110Series1,0.106,0.243Series1,0.113,0.252Series1,0.040,0.239Series1,0.102,0.241Series1,0.258,0.400Series1,0.115,0.201Series1,0.342,0.393Series1,0.344,0.391Series1,0.075,0.387Series1,0.020,0.09325关注我呗2019年4月21情感要素抽取宽松评价25关注我呗2019年4月21情感要素抽取宽松评价宏平均F-measureF-measure百百分分比比0.00-0.050.05-0.100.10-0.150.15-0.200.20-0.250.25-0.300.30-0.350.35-0.400.40-0.450.45-0.500.50-0.550.55-0.600.60-0.650.65-0.700.70-0.750.75-0.800.80-0.850.85-0.900.90-0.950.95-1.000.0%5.0%10.0%15.0%20.0%25.0%Series1,0.00-0.05,0.136363636Series1,0.05-0.10,0.090909091Series1,0.10-0.15,0.227272727Series1,0.15-0.20,0.181818182Series1,0.20-0.25,0.090909091Series1,0.25-0.30,0.136363636Series1,0.30-0.35,0.045454545Series1,0.35-0.40,0.090909091Series1,0.40-0.45,0Series1,0.45-0.50,0Series1,0.50-0.55,0Series1,0.55-0.60,0Series1,0.60-0.65,0Series1,0.65-0.70,0Series1,0.70-0.75,0Series1,0.75-0.80,0Series1,0.80-0.85,0Series1,0.85-0.90,0Series1,0.90-0.95,0Series1,0.95-1.00,026关注我呗2019年4月21情感要素抽取宽松评价26关注我呗2019年4月21小结微博情感分析相当具有挑战性不同于产品评论数据，微博数据领域多样，话题广泛，表达自由本次评测中微博观点句识别、倾向性分析的最好结果F值接近于80%，情感要素抽取的最好结果精确评价F值接近于30%，比预想的结果要好，但仍有较大的提升空间本次评测的标注数据可作为相关研究的科研数据，已免费公开发布标注规范和任务设置有待进一步的改进27关注我呗2019年4月21小结微博情感分析相当具有挑战性27关注我呗2019年4月2128关注我呗2019年4月2128关注我呗2019年4月21致谢感谢腾讯微博提供数据！感谢北京大学、苏州大学、哈工大深圳研究生院的老师和同学进行数据标注与测评！感谢评测参与单位的支持和意见反馈！29关注我呗2019年4月21致谢感谢腾讯微博提供数据！29关注我呗2019年4月2130关注我呗2019年4月2130关注我呗2019年4月21