我是scrakit-learn的新手,目前正在学习NaïveBayes(Multinomial).现在,我正在研究sklearn.feature_extraction.text中的文本向量化,出于某种原因,当我向某些文本进行矢量化时,单词"I"不会出现在输出的数组中.
码:
x_train = ['I am a Nigerian hacker', 'I like puppies'] # convert x_train to vectorized text vectorizer_train = CountVectorizer(min_df=0) vectorizer_train.fit(x_train) x_train_array = vectorizer_train.transform(x_train).toarray() # print vectorized text, feature names print x_train_array print vectorizer_train.get_feature_names()
输出:
1 1 0 1 0 0 0 1 0 1 [u'am', u'hacker', u'like', u'nigerian', u'puppies']
为什么"我"似乎没有出现在功能名称中?当我将其更改为"Ia"或类似的其他内容时,它确实会显示出来.
这是由默认值token_pattern
for 引起的CountVectorizer
,它会删除单个字符的标记:
>>> vectorizer_train CountVectorizer(analyzer=u'word', binary=False, charset=None, charset_error=None, decode_error=u'strict', dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content', lowercase=True, max_df=1.0, max_features=None, min_df=0, ngram_range=(1, 1), preprocessor=None, stop_words=None, strip_accents=None, token_pattern=u'(?u)\\b\\w\\w+\\b', tokenizer=None, vocabulary=None) >>> pattern = re.compile(vectorizer_train.token_pattern, re.UNICODE) >>> print(pattern.match("I")) None
要保留"I",请使用不同的模式,例如
>>> vectorizer_train = CountVectorizer(min_df=0, token_pattern=r"\b\w+\b") >>> vectorizer_train.fit(x_train) CountVectorizer(analyzer=u'word', binary=False, charset=None, charset_error=None, decode_error=u'strict', dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content', lowercase=True, max_df=1.0, max_features=None, min_df=0, ngram_range=(1, 1), preprocessor=None, stop_words=None, strip_accents=None, token_pattern='\\b\\w+\\b', tokenizer=None, vocabulary=None) >>> vectorizer_train.get_feature_names() [u'a', u'am', u'hacker', u'i', u'like', u'nigerian', u'puppies']
请注意,现在也保留了无信息的单词"a".