玩转Kaggle：ClassifyLeaves（叶子分类）——数据分析篇

一、Classify Leaves竞赛介绍

描述&＃xff1a;(叶子种类分类&＃xff0c;总共176类&＃xff0c;训练数据18353张图&＃xff0c;测试数据8800张图片&＃xff0c;每一类至少有50张图片)

The task is predicting categories of leaf images. This dataset contains 176 categories, 18353 training images, 8800 test images. Each category has at least 50 images for training. The test set is split evenly into the public and private leaderboard.

The evaluation metric for this competition is Classification Accuracy.

Good luck and have fun!

kanggle竞赛地址&＃xff1a;
https://www.kaggle.com/c/classify-leaves

二、数据分析

import pandas as pd
import numpy as np
from d2l import torch as d2l
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder,OneHotEncoder
import os
from PIL import Image
from torchvision import transforms


2.1 训练数据信息统计和查看

# 加载原始数据的统计目录
train_data &＃61; pd.read_csv(&＃39;./data/classify-leaves/train.csv&＃39;)
train_data.head()


# 训练数据共18353张图片&＃xff0c;176类叶子
train_data.describe()


# 统计训练数据label的类别和数量
train_data[&＃39;label&＃39;].value_counts()


maclura_pomifera 353
ulmus_rubra 235
prunus_virginiana 223
acer_rubrum 217
broussonettia_papyrifera 214...
cedrus_deodara 58
ailanthus_altissima 58
crataegus_crus-galli 54
evodia_daniellii 53
juniperus_virginiana 51
Name: label, Length: 176, dtype: int64


# 树叶的名字统计
labels_unique &＃61; train_data[&＃39;label&＃39;].unique()
labels_unique


array([&＃39;maclura_pomifera&＃39;, &＃39;ulmus_rubra&＃39;, &＃39;broussonettia_papyrifera&＃39;,&＃39;prunus_virginiana&＃39;, &＃39;acer_rubrum&＃39;, &＃39;cryptomeria_japonica&＃39;,&＃39;staphylea_trifolia&＃39;, &＃39;asimina_triloba&＃39;, &＃39;diospyros_virginiana&＃39;,&＃39;tilia_cordata&＃39;, &＃39;ulmus_pumila&＃39;, &＃39;quercus_muehlenbergii&＃39;,&＃39;juglans_cinerea&＃39;, &＃39;cercis_canadensis&＃39;, &＃39;ptelea_trifoliata&＃39;,&＃39;acer_palmatum&＃39;, &＃39;catalpa_speciosa&＃39;, &＃39;abies_concolor&＃39;,&＃39;eucommia_ulmoides&＃39;, &＃39;quercus_montana&＃39;, &＃39;koelreuteria_paniculata&＃39;,..., &＃39;sassafras_albidum&＃39;, &＃39;acer_griseum&＃39;,&＃39;ailanthus_altissima&＃39;, &＃39;pinus_thunbergii&＃39;, &＃39;crataegus_crus-galli&＃39;,&＃39;juniperus_virginiana&＃39;], dtype&＃61;object)


# 对label进行编码&＃xff0c;并将映射表保存下来
labelencoder &＃61; LabelEncoder()
labelencoder.fit(train_data[&＃39;label&＃39;])
train_data[&＃39;label&＃39;] &＃61; labelencoder.transform(train_data[&＃39;label&＃39;])
label_map &＃61; dict(zip(labelencoder.classes_,labelencoder.transform(labelencoder.classes_)))
label_inv_map &＃61; {v:k for k,v in label_map.items()}
label_map


{&＃39;abies_concolor&＃39;: 0,&＃39;abies_nordmanniana&＃39;: 1,&＃39;acer_campestre&＃39;: 2,&＃39;acer_ginnala&＃39;: 3,&＃39;acer_griseum&＃39;: 4,&＃39;acer_negundo&＃39;: 5,&＃39;acer_palmatum&＃39;: 6,&＃39;acer_pensylvanicum&＃39;: 7,&＃39;acer_platanoides&＃39;: 8,&＃39;acer_pseudoplatanus&＃39;: 9,&＃39;acer_rubrum&＃39;: 10,...&＃39;zelkova_serrata&＃39;: 175}


#查看前20类的统计数量和绘图查看
top20_trainData &＃61; train_data[&＃39;label&＃39;].value_counts().sort_values(ascending&＃61;False).head(20)
print(top20_trainData)
plt.figure(figsize&＃61;(15,10))
sns.barplot(x&＃61;top20_trainData.index,y&＃61;top20_trainData)
plt.xticks(rotation&＃61;70)
plt.title("Top 20 categories of leaf statistics")
plt.show()


2.2 测试数据统计和分析

test_data &＃61; pd.read_csv(&＃39;./data/classify-leaves/test.csv&＃39;)
test_data


8800 rows × 1 columns

# 测试数据共8800张图
test_data.describe()


2.3 可视化训练数据

folder_path &＃61; "./data/classify-leaves/"# 拿出几张照片看看
fig, ax &＃61; plt.subplots(nrows&＃61;3,ncols&＃61;4,sharex&＃61;True,sharey&＃61;True, figsize&＃61;(18,12)
)ax &＃61; ax.flatten()
transform &＃61; transforms.Compose([transforms.Resize((224,224)),transforms.ToTensor()])
for i in range(12):img_path &＃61; os.path.join(folder_path,train_data[&＃39;image&＃39;][i])data &＃61; Image.open(img_path)data &＃61; transform(data)ax[i].imshow(data.permute((2,1,0))) # 记住将维度变换一下ax[i].set(title&＃61;train_data[&＃39;label&＃39;][i])ax[i].title.set_size(25)ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
plt.show()


三、整理

1. 训练数据和测试数据都是RG图&＃xff0c;可以考虑转换为灰度图进行识别&＃xff08;判断颜色特征对数据集是否特别重要&＃xff09;RGB图
2. 给定的训练数据和测试数据都是规整的大小&＃xff0c;但是叶子占据的比例较小&＃xff0c;可以考虑进行图片裁减
3. 训练数据数量不是特别大&＃xff0c;可以考虑进行数据增强&＃xff0c;扩大数据集数据量小
4. 由于数据较小&＃xff0c;使用k折交叉验证可以得到一个更好的结果