python 词云UnicodeEncodeError错误 python中文词云代码

1. 词云简介

词云，又称文字云、标签云，是对文本数据中出现频率较高的“关键词”在视觉上的突出呈现，形成关键词的渲染形成类似云一样的彩色图片，从而一眼就可以领略文本数据的主要表达意思。常见于博客、微博、文章分析等。

除了网上现成的Wordle、Tagxedo、Tagul、Tagcrowd等词云制作工具，在python中也可以用wordcloud包比较轻松地实现（官网、github项目）：

from wordcloud import WordCloud
import matplotlib.pyplot as plt

# Read the whole text.
text = open('constitution.txt').read()

# Generate a word cloud image
wordcloud = WordCloud().generate(text)

# Display the generated image:
# the matplotlib way:
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis("off")

生成的词云如下：

还可以设置图片作为mask：

alice_mask = np.array(Image.open(path.join(d, "alice_mask.png")))
wc = WordCloud(background_color="white", max_words=2000, mask=alice_mask, stopwords=stopwords, contour_width=3, contour_color='steelblue')
wc.generate(text)

 

2. 安装

pip install wordcloud

词云：解决pip install wordcloud安装过程中报错“error: command 'x86_64-linux-gnu-gcc' failed with exit status 1”问题

  

3. 根据源码分析wordcloud的实现原理

总的来说，wordcloud做的是三件事：

(1) 文本预处理

(2) 词频统计

(3) 将高频词以图片形式进行彩色渲染

 

从上面的代码可以看到，用 wordcloud.generate(text)

源码：

def generate(self, text):
    """Generate wordcloud from text.

    The input "text" is expected to be a natural text. If you pass a sorted
    list of words, words will appear in your output twice. To remove this
    duplication, set ``collocations=False``.

    Alias to generate_from_text.

    Calls process_text and generate_from_frequencies.

    Returns
    -------
    self
    """
    return self.generate_from_text(text)

def generate_from_text(self, text):
    """Generate wordcloud from text.

    The input "text" is expected to be a natural text. If you pass a sorted
    list of words, words will appear in your output twice. To remove this
    duplication, set ``collocations=False``.

    Calls process_text and generate_from_frequencies.

    ..versionchanged:: 1.2.2
        Argument of generate_from_frequencies() is not return of
        process_text() any more.

    Returns
    -------
    self
    """
    words = self.process_text(text)
    self.generate_from_frequencies(words)
    return self

generate()和generate_from_text()

 它的调用顺序是：

generate(self, text)
=>
self.generate_from_text(text)
=>
words = self.process_text(text)
self.generate_from_frequencies(words)

其中 process_text(text) 对应的是文本预处理和词频统计，而 generate_from_frequencies(words) 对应的是根据词频中生成词云。

 

(1) process_text(text)　主要是进行分词和去噪。

具体地，它做了以下操作：

• 检测文本编码
• 分词(根据规则进行tokenize)、保留单词字符(A-Za-z0-9_)和单引号(')、去除单字符
• 去除停用词
• 去除后缀('s) -- 针对英文
• 去除纯数字
• 统计一元和二元词频计数(unigrams_and_bigrams) -- 可选

返回的结果是一个字典 dict(string, int) ，表示的是分词后的token以及对应出现的次数。

这里有一些需要注意的地方，文章后面会再提到。 

源码如下：

def process_text(self, text):
    """Splits a long text into words, eliminates the stopwords.

    Parameters
    ----------
    text : string
        The text to be processed.

    Returns
    -------
    words : dict (string, int)
        Word tokens with associated frequency.

    ..versionchanged:: 1.2.2
        Changed return type from list of tuples to dict.

    Notes
    -----
    There are better ways to do word tokenization, but I don't want to
    include all those things.
    """

    stopwords = set([i.lower() for i in self.stopwords])

    flags = (re.UNICODE if sys.version < '3' and type(text) is unicode
             else 0)
    regexp = self.regexp if self.regexp is not None else r"\w[\w']+"

    words = re.findall(regexp, text, flags)
    # remove stopwords
    words = [word for word in words if word.lower() not in stopwords]
    # remove 's
    words = [word[:-2] if word.lower().endswith("'s") else word
             for word in words]
    # remove numbers
    words = [word for word in words if not word.isdigit()]

    if self.collocations:
        word_counts = unigrams_and_bigrams(words, self.normalize_plurals)
    else:
        word_counts, _ = process_tokens(words, self.normalize_plurals)

    return word_counts

def process_text(self, text)

 

(2) generate_from_frequencies(words)　主要是根据上一步的结果生成词云分布。

具体地，它做了以下操作：

• 对词计数结果进行排序，并归一化(normalized)到0~1之间，得到词频
• 创建图像并确定font_size初始值
• 给self.words_赋值，记录的是出现频率最高的前max_words个词，以及对应的归一化后的词频，即dict(token, normalized_frequency)
• 画出灰度图：词频越大，font_size越大；根据生成的随机数来决定字的水平/垂直方向

• 若随机数小于self.prefer_horizontal则为水平方向，否则为垂直方向；
• 如果空间不足，优先考虑旋转方向，其次考虑将字体变小

• 给self.layout_赋值，记录的是词和词频、字体大小、位置、方向、以及颜色，即list(zip(frequencies, font_sizes, positions, orientations, colors)) 

可以看到，这个函数的主要目的在于得到self.layout_的值，记录了要生成词云分布图所需要的信息。

后面wordcloud.to_file(filename)或者plt.imshow(wordcloud)会把结果以图像的形式呈现出来。其中to_file()函数就会先检测是否已经给self.layout_赋值，如果没有的话会报错。

源码如下：

def generate_from_frequencies(self, frequencies, max_font_size=None):
    """Create a word_cloud from words and frequencies.

    Parameters
    ----------
    frequencies : dict from string to float
        A contains words and associated frequency.

    max_font_size : int
        Use this font-size instead of self.max_font_size

    Returns
    -------
    self

    """
    # make sure frequencies are sorted and normalized
    frequencies = sorted(frequencies.items(), key=itemgetter(1), reverse=True)
    if len(frequencies) <= 0:
        raise ValueError("We need at least 1 word to plot a word cloud, "
                         "got %d." % len(frequencies))
    frequencies = frequencies[:self.max_words]

    # largest entry will be 1
    max_frequency = float(frequencies[0][1])

    frequencies = [(word, freq / max_frequency)
                   for word, freq in frequencies]

    if self.random_state is not None:
        random_state = self.random_state
    else:
        random_state = Random()

    if self.mask is not None:
        mask = self.mask
        width = mask.shape[1]
        height = mask.shape[0]
        if mask.dtype.kind == 'f':
            warnings.warn("mask image should be unsigned byte between 0"
                          " and 255. Got a float array")
        if mask.ndim == 2:
            boolean_mask = mask == 255
        elif mask.ndim == 3:
            # if all channels are white, mask out
            boolean_mask = np.all(mask[:, :, :3] == 255, axis=-1)
        else:
            raise ValueError("Got mask of invalid shape: %s"
                             % str(mask.shape))
    else:
        boolean_mask = None
        height, width = self.height, self.width
    occupancy = IntegralOccupancyMap(height, width, boolean_mask)

    # create image
    img_grey = Image.new("L", (width, height))
    draw = ImageDraw.Draw(img_grey)
    img_array = np.asarray(img_grey)
    font_sizes, positions, orientations, colors = [], [], [], []

    last_freq = 1.

    if max_font_size is None:
        # if not provided use default font_size
        max_font_size = self.max_font_size

    if max_font_size is None:
        # figure out a good font size by trying to draw with
        # just the first two words
        if len(frequencies) == 1:
            # we only have one word. We make it big!
            font_size = self.height
        else:
            self.generate_from_frequencies(dict(frequencies[:2]),
                                           max_font_size=self.height)
            # find font sizes
            sizes = [x[1] for x in self.layout_]
            try:
                font_size = int(2 * sizes[0] * sizes[1] 
                                / (sizes[0] + sizes[1]))
            # quick fix for if self.layout_ contains less than 2 values
            # on very small images it can be empty
            except IndexError:
                try:
                    font_size = sizes[0]
                except IndexError:
                    raise ValueError('canvas size is too small')
    else:
        font_size = max_font_size

    # we set self.words_ here because we called generate_from_frequencies
    # above... hurray for good design?
    self.words_ = dict(frequencies)

    # start drawing grey image
    for word, freq in frequencies:
        # select the font size
        rs = self.relative_scaling
        if rs != 0:
            font_size = int(round((rs * (freq / float(last_freq))
                                   + (1 - rs)) * font_size))
        if random_state.random() < self.prefer_horizontal:
            orientation = None
        else:
            orientation = Image.ROTATE_90
        tried_other_orientation = False
        while True:
            # try to find a position
            font = ImageFont.truetype(self.font_path, font_size)
            # transpose font optionally
            transposed_font = ImageFont.TransposedFont(
                font, orientation=orientation)
            # get size of resulting text
            box_size = draw.textsize(word, font=transposed_font)
            # find possible places using integral image:
            result = occupancy.sample_position(box_size[1] + self.margin,
                                               box_size[0] + self.margin,
                                               random_state)
            if result is not None or font_size < self.min_font_size:
                # either we found a place or font-size went too small
                break
            # if we didn't find a place, make font smaller
            # but first try to rotate!
            if not tried_other_orientation and self.prefer_horizontal < 1:
                orientation = (Image.ROTATE_90 if orientation is None else
                               Image.ROTATE_90)
                tried_other_orientation = True
            else:
                font_size -= self.font_step
                orientation = None

        if font_size < self.min_font_size:
            # we were unable to draw any more
            break

        x, y = np.array(result) + self.margin // 2
        # actually draw the text
        draw.text((y, x), word, fill="white", font=transposed_font)
        positions.append((x, y))
        orientations.append(orientation)
        font_sizes.append(font_size)
        colors.append(self.color_func(word, font_size=font_size,
                                      position=(x, y),
                                      orientation=orientation,
                                      random_state=random_state,
                                      font_path=self.font_path))
        # recompute integral image
        if self.mask is None:
            img_array = np.asarray(img_grey)
        else:
            img_array = np.asarray(img_grey) + boolean_mask
        # recompute bottom right
        # the order of the cumsum's is important for speed ?!
        occupancy.update(img_array, x, y)
        last_freq = freq

    self.layout_ = list(zip(frequencies, font_sizes, positions,
                            orientations, colors))
    return self

def generate_from_frequencies(self, frequencies, max_font_size=None)

 

4. 应用到中文语料应该要注意的点

wordcloud包是由Andreas Mueller在2015-03-20发布1.0.0版本，现在最新的是2018-03-13发布的1.4.1版本。

英文语料可以直接输入到wordcloud中，但是对于中文语料，仅仅用wordcloud不能直接生成中文词云图。

原因：

英文单词以空格分隔，而我们从前面process_text(text)看到源码中是直接用正则表达式(默认为r"\w[\w']+")进行处理：

In  : re.findall(r"\w[\w']+", "It's Monday today.")
Out: ["It's", 'Monday', 'today']

但是中文里面词与词之间一般不用字符分隔：

In : re.findall(r"\w[\w']+", "今天天气不错，蓝天白云，还有温暖的阳光 哈　哈哈")
Out: ['今天天气不错', '蓝天白云', '还有温暖的阳光', '哈哈']

可以看出，原生的wordcloud是为英文服务的，去除标点符号（单符号'除外)并分割成token；

而应用到中文语料上的时候，注意要先分好词，再用空格分隔连接成字符串，最后输入到wordcloud。

另外要注意的是，无论是对英文还是中文，默认是把单字符剔除掉（因为 regexp = self.regexp if self.regexp is not None else r"\w[\w']+" ），如果想要保留单字符，将regexp参数讲表达式设置为 r"\w[\w']*"

from wordcloud import WordCloud
from scipy.misc import imread

def generate_wordcloud(text, max_words=200, pic_path=None):
    """
    生成词云
    :param text: 一段以空格为间断的字符串
    :param max_words: 词数目上限
    :param pic_path: 输出图片路径
    :return:
    """
    mk = imread("tuoyuan.jpg")
    wc = WordCloud(font_path="/usr/share/fonts/myfonts/msyh.ttf", background_color="white", max_words=max_words,
                   mask=mk, width=1000, height=500, max_font_size=100, prefer_horizontal=0.95, collocations=False)
    wc.generate(text=text)
    if pic_path:
        wc.to_file(pic_path)
    else:
        plt.imshow(wc)
        plt.axis("off")
        plt.show()
    return wc.words_

def run_wordcloud(corpus, max_words, pic_path=None):
    text = " ".join([" ".join(line) for line in corpus])   # 将分词后的结果用空格连接
    word2weight = generate_wordcloud(text=text, max_words=max_words, pic_path=pic_path)
    word2weight_sorted = sorted(word2weight.items(), key=lambda x: x[1], reverse=True)
    logging.info([(k, float("%.5f" % v)) for k, v in word2weight_sorted])

更多参考：word_cloud/examples/wordcloud_cn.py

 

5. 重写代码

用词云是为了直观地看语料的关键信息，在本人的实际工作应用中，主要目的在于获取关键信息，而不太关注界面的呈现方式。

所以在了解wordcloud源码实现原理之后，决定自己用代码实现。

一方面，使得代码的实现更公开透明，在效率相当的情况下尽量避免使用第三方库，效果可控，甚至还可以提升效率；

另一方面，能结合实际情况更灵活地处理问题。

针对中文的预处理，可以和分词结合一起完成。这里主要进行：分词和词性标注、小写化、去停用词、去数字、去单字符、以及保留指定词性。

import jieba
import jieba.posseg as pseg

class Utils(object):
    def __init__(self, utils_data=None):
        self.stopwords = self.init_utils(utils_data)
        self.pos_save = {
            "n", "an", "Ng", "nr", "ns", "nt", "nz", "vn", "un",  # 名
            "v", "vg", "vd",  # 动
            "a", "ag", "ad",  # 形
            "j", "l", "i", "z", "b", "g", "s", "h",  # j简称略语、l习用语、i成语、z状态词、b区别词、g语素、s处所词、h前接成分
            "zg", "eng",
            "x"}  # 未知（自定义词）

    def _init_utils(self, utils_data):
        for wd in utils_data["user_dict"]:
            jieba.add_word(wd)
        return set(utils_data["stopwords"])

    def _token_filter(self, token):  # 去停用词; 去数字; 去单字
        return token not in self.stopwords and not token.isdigit() and len(token) >= 2

    def _token_filter_with_flag(self, pair_word_flag):  # 保留指定词性
        return self.token_filter(pair_word_flag.word) and pair_word_flag.flag in self.pos_save

    def cut(self, text):
        return list(filter(self._token_filter, list(jieba.cut(text.lower()))))  # 分词; 小写化;

    def cut_with_flag(self, text):
        pairs = list(filter(self._token_filter_with_flag,  list(pseg.cut(text.lower()))))  # 分词和词性标注; 小写化;
        return [p.word for p in pairs]

 

做完文本分词和其它预处理之后，直接统计词及对应的出现次数即可。为了更直观，这里输出的是词计数，而不是归一化后的词频。排序结果与wordcloud等同。

def word_count(corpus, n_gram=1, n=None):
        counter = Counter()
        if n_gram == 1:
            for line in corpus:
                counter.update(line)
        elif n_gram == 2:
            for line in corpus:
                size = len(line)
                counter.update(["%s_%s" % (line[idx], line[idx + 1]) for idx in range(size) if idx + 1 < size])  # 有序
        else:
            logging.info("[Error] Invalid value of param n_gram: %s (only 1 or 2 accepted)" % n_gram)
        return counter.most_common(n=n)

另外还可以统计高频词的共现情况、把高频词/词共现反向映射到对应的句子等等，便于从高频词层面到高频句子类型层面的归纳。

 

参考：

https://pypi.org/project/wordcloud/

https://github.com/amueller/word_cloud

http://python.jobbole.com/87496/

https://www.jianshu.com/p/ead991a08563