尽可能使字符串相等 python python中字符串可以相加吗

我们可以通过几种方式加速匹配.我假设在您的代码中,str1是数据集中的名称,str2是一个geoname字符串.为了测试代码,我从你问题中的数据中创建了两个微小的数据集.我编写了两个匹配函数best_match和first_match,它们使用您当前的string_similarity函数,因此我们可以看到我的策略给出了相同的结果. best_match检查所有geoname字符串&如果超过给定阈值分数,则返回分数最高的字符串,否则返回None. first_match(可能)更快：它只返回超过阈值的第一个geoname字符串,如果找不到,则返回None,所以如果它找不到匹配,那么它仍然必须搜索整个geoname列表.

在我的改进版本中,我们为每个str1生成一次bigrams,而不是为我们比较的每个str2重新生成str1的bigrams.我们事先计算所有geoname bigrams,将它们存储在由字符串索引的字典中,这样我们就不必为每个str重新生成它们.此外,我们将geoname bigrams存储为集合.这使得计算hit_count的速度更快,因为集合成员资格测试比对字符串列表进行线性扫描要快得多. geodict还需要存储每个bigram的长度：一个集合不包含重复的项目,因此bigrams集合的长度可能小于bigrams列表,但我们需要列表长度来正确计算得分.

# Some fake data
geonames = [
'Slettmarkmountains Jotunheimen Norway',
'Fairy Glen Skye Scotland UK',
'Emigrant Wilderness California',
'Yosemite National Park',
'Half Dome Yosemite National Park',
]
mynames = [
'Jotunheimen Norway',
'Fairy Glen',
'Slettmarkmountains Jotunheimen Norway',
'Bryce Canyon',
'Half Dome',
]
def get_bigrams(string):
"""
Take a string and return a list of bigrams.
"""
s = string.lower()
return [s[i:i+2] for i in range(len(s) - 1)]
def string_similarity(str1, str2):
"""
Perform bigram comparison between two strings
and return a percentage match in decimal form.
"""
pairs1 = get_bigrams(str1)
pairs2 = get_bigrams(str2)
union = len(pairs1) + len(pairs2)
hit_count = 0
for x in pairs1:
for y in pairs2:
if x == y:
hit_count += 1
break
return (2.0 * hit_count) / union
# Find the string in geonames which is the best match to str1
def best_match(str1, thresh=0.2):
score, str2 = max((string_similarity(str1, str2), str2) for str2 in geonames)
if score < thresh:
str2 = None
return score, str2
# Find the 1st string in geonames that matches str1 with a score >= thresh
def first_match(str1, thresh=0.2):
for str2 in geonames:
score = string_similarity(str1, str2)
if score >= thresh:
return score, str2
return None
print('Best')
for mystr in mynames:
print(mystr, ':', best_match(mystr))
print()
print('First')
for mystr in mynames:
print(mystr, ':', best_match(mystr))
print()
# Put all the geoname bigrams into a dict
geodict = {}
for s in geonames:
bigrams = get_bigrams(s)
geodict[s] = (set(bigrams), len(bigrams))
def new_best_match(str1, thresh=0.2):
pairs1 = get_bigrams(str1)
pairs1_len = len(pairs1)
score, str2 = max((2.0 * sum(x in pairs2 for x in pairs1) / (pairs1_len + pairs2_len), str2)
for str2, (pairs2, pairs2_len) in geodict.items())
if score < thresh:
str2 = None
return score, str2
def new_first_match(str1, thresh=0.2):
pairs1 = get_bigrams(str1)
pairs1_len = len(pairs1)
for str2, (pairs2, pairs2_len) in geodict.items():
score = 2.0 * sum(x in pairs2 for x in pairs1) / (pairs1_len + pairs2_len)
if score >= thresh:
return score, str2
return None
print('New Best')
for mystr in mynames:
print(mystr, ':', new_best_match(mystr))
print()
print('New First')
for mystr in mynames:
print(mystr, ':', new_first_match(mystr))
print()
产量
Best
Jotunheimen Norway : (0.6415094339622641, 'Slettmarkmountains Jotunheimen Norway')
Fairy Glen : (0.5142857142857142, 'Fairy Glen Skye Scotland UK')
Slettmarkmountains Jotunheimen Norway : (1.0, 'Slettmarkmountains Jotunheimen Norway')
Bryce Canyon : (0.1875, None)
Half Dome : (0.41025641025641024, 'Half Dome Yosemite National Park')
First
Jotunheimen Norway : (0.6415094339622641, 'Slettmarkmountains Jotunheimen Norway')
Fairy Glen : (0.5142857142857142, 'Fairy Glen Skye Scotland UK')
Slettmarkmountains Jotunheimen Norway : (1.0, 'Slettmarkmountains Jotunheimen Norway')
Bryce Canyon : (0.1875, None)
Half Dome : (0.41025641025641024, 'Half Dome Yosemite National Park')
New Best
Jotunheimen Norway : (0.6415094339622641, 'Slettmarkmountains Jotunheimen Norway')
Fairy Glen : (0.5142857142857142, 'Fairy Glen Skye Scotland UK')
Slettmarkmountains Jotunheimen Norway : (1.0, 'Slettmarkmountains Jotunheimen Norway')
Bryce Canyon : (0.1875, None)
Half Dome : (0.41025641025641024, 'Half Dome Yosemite National Park')
New First
Jotunheimen Norway : (0.6415094339622641, 'Slettmarkmountains Jotunheimen Norway')
Fairy Glen : (0.5142857142857142, 'Fairy Glen Skye Scotland UK')
Slettmarkmountains Jotunheimen Norway : (1.0, 'Slettmarkmountains Jotunheimen Norway')
Bryce Canyon : None
Half Dome : (0.41025641025641024, 'Half Dome Yosemite National Park')
new_first_match相当简单.这条线
for str2, (pairs2, pairs2_len) in geodict.items():

循环遍历geodict中的每个项目,提取每个字符串,bigram集和真正的bigram长度.

sum(x in pairs2 for x in pairs1)

计算对1中有多少双佬是成对2集的成员.

因此,对于每个地理名称字符串,我们计算相似性得分并返回它,如果它是> =阈值,其默认值为0.2.您可以给它一个不同的默认阈值,或者在调用它时传递一个阈值.

new_best_match有点复杂. 😉

((2.0 * sum(x in pairs2 for x in pairs1) / (pairs1_len + pairs2_len), str2)
for str2, (pairs2, pairs2_len) in geodict.items())

是一个生成器表达式.它遍历geodict项并为每个geoname字符串创建一个(score,str2)元组.然后,我们将该生成器表达式提供给max函数,该函数返回具有最高分数的元组.

这是new_first_match的一个版本,它实现了juvian在评论中提出的建议.它可以节省一点时间.如果两个bigram都为空,这个版本也可以避免测试.

def new_first_match(str1, thresh=0.2):
pairs1 = get_bigrams(str1)
pairs1_len = len(pairs1)
if not pairs1_len:
return None
hiscore = 0
for str2, (pairs2, pairs2_len) in geodict.items():
if not pairs2_len:
continue
total_len = pairs1_len + pairs2_len
bound = 2.0 * pairs1_len / total_len
if bound >= hiscore:
score = 2.0 * sum(x in pairs2 for x in pairs1) / total_len
if score >= thresh:
return score, str2
hiscore = max(hiscore, score)
return None

一个更简单的变化是不打扰计算他的核心&只是比较绑定到thresh.