1 GIL(Global Interpretor Lock,全局解释器锁)
see:
如果其他条件不变，Python程序的执行速度直接与解释器的“速度”相关。不管你怎样优化自己的程序，你的程序的执行速度还是依赖于解释器执行你的程序的效率。
目前来说，多线程执行还是利用多核系统最常用的方式。尽管多线程编程大大好于“顺序”编程，不过即便是仔细的程序员也没法在代码中将并发性做到最好。
对于任何Python程序，不管有多少的处理器，任何时候都总是只有一个线程在执行。
事实上，这个问题被问得如此频繁以至于Python的专家们精心制作了一个标准答案：”不要使用多线程，请使用多进程。“但这个答案比那个问题更加让人困惑。
GIL对诸如当前线程状态和为垃圾回收而用的堆分配对象这样的东西的访问提供着保护。然而，这对Python语言来说没什么特殊的，它需要使用一个GIL。这是该实现的一种典型产物。现在也有其它的Python解释器（和编译器）并不使用GIL。虽然，对于CPython来说，自其出现以来已经有很多不使用GIL的解释器。
不管某一个人对Python的GIL感觉如何，它仍然是Python语言里最困难的技术挑战。想要理解它的实现需要对操作系统设计、多线程编程、C语言、解释器设计和CPython解释器的实现有着非常彻底的理解。单是这些所需准备的就妨碍了很多开发者去更彻底的研究GIL。
2 threading
threading 模块提供比/基于 thread 模块更高层次的接口；如果此模块由于 thread 丢失而无法使用，可以使用 dummy_threading 来代替。
CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.
举例:
1 import threading, zipfile
2
3 class AsyncZip(threading.Thread):
4 def __init__(self, infile, outfile):
5 threading.Thread.__init__(self)
6 self.infile = infile
7 self.outfile = outfile
8 def run(self):
9 f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED)
10 f.write(self.infile)
11 f.close()
12 print 'Finished background zip of: ', self.infile
13
14 background = AsyncZip('mydata.txt', 'myarchive.zip')
15 background.start()
16 print 'The main program continues to run in foreground.'
17
18 background.join() # Wait for the background task to finish
19 print 'Main program waited until background was done.'
2.1 创建线程
1 import threading
2 import datetime
3
4 class ThreadClass(threading.Thread):
5 def run(self):
6 now = datetime.datetime.now()
7 print "%s says Hello World at time: %s" % (self.getName(), now)
8
9 for i in range(2):
10 t = ThreadClass()
11 t.start()
2.2 使用线程队列
1 import Queue
2 import threading
3 import urllib2
4 import time
5 from BeautifulSoup import BeautifulSoup
6
7 hosts = ["http://yahoo.com", "http://", "http://amazon.com",
8 "http://ibm.com", "http://apple.com"]
9
10 queue = Queue.Queue()
11 out_queue = Queue.Queue()
12
13 class ThreadUrl(threading.Thread):
14 """Threaded Url Grab"""
15 def __init__(self, queue, out_queue):
16 threading.Thread.__init__(self)
17 self.queue = queue
18 self.out_queue = out_queue
19
20 def run(self):
21 while True:
22 #grabs host from queue
23 host = self.queue.get()
24
25 #grabs urls of hosts and then grabs chunk of webpage
26 url = urllib2.urlopen(host)
27 chunk = url.read()
28
29 #place chunk into out queue
30 self.out_queue.put(chunk)
31
32 #signals to queue job is done
33 self.queue.task_done()
34
35 class DatamineThread(threading.Thread):
36 """Threaded Url Grab"""
37 def __init__(self, out_queue):
38 threading.Thread.__init__(self)
39 self.out_queue = out_queue
40
41 def run(self):
42 while True:
43 #grabs host from queue
44 chunk = self.out_queue.get()
45
46 #parse the chunk
47 soup = BeautifulSoup(chunk)
48 print soup.findAll(['title'])
49
50 #signals to queue job is done
51 self.out_queue.task_done()
52
53 start = time.time()
54 def main():
55
56 #spawn a pool of threads, and pass them queue instance
57 for i in range(5):
58 t = ThreadUrl(queue, out_queue)
59 t.setDaemon(True)
60 t.start()
61
62 #populate queue with data
63 for host in hosts:
64 queue.put(host)
65
66 for i in range(5):
67 dt = DatamineThread(out_queue)
68 dt.setDaemon(True)
69 dt.start()
70
71
72 #wait on the queue until everything has been processed
73 queue.join()
74 out_queue.join()
75
76 main()
77 print "Elapsed Time: %s" % (time.time() - start)
3 dummy_threading(threading的备用方案)
dummy_threading 模块提供完全复制了threading模块的接口，如果无法使用thread，则可以用这个模块替代.
使用方法:
1 try:
2 import threading as _threading
3 except ImportError:
4 import dummy_threading as _threading
4 thread
在Python3中叫 _thread,应该尽量使用 threading 模块替代。
5 dummy_thread(thead的备用方案)
dummy_thread 模块提供完全复制了thread模块的接口，如果无法使用thread，则可以用这个模块替代.
在Python3中叫 _dummy_thread, 使用方法:
try:
import thread as _thread
except ImportError:
import dummy_thread as _thread
最好使用 dummy_threading 来代替.
6 multiprocessing(基于thread接口的多进程)
see:
使用 multiprocessing 模块创建子进程而不是线程来克服GIL引起的问题.
举例:
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
p = Pool(5)
print(p.map(f, [1, 2, 3]))
6.1 Process类
创建进程是使用Process类:
from multiprocessing import Process
def f(name):
print 'hello', name
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
6.2 进程间通信
Queue 方式:
from multiprocessing import Process, Queue
def f(q):
q.put([42, None, 'hello'])
if __name__ == '__main__':
q = Queue()
p = Process(target=f, args=(q,))
p.start()
print q.get() # prints "[42, None, 'hello']"
p.join()
Pipe 方式:
from multiprocessing import Process, Pipe
def f(conn):
conn.send([42, None, 'hello'])
conn.close()
if __name__ == '__main__':
parent_conn, child_conn = Pipe()
p = Process(target=f, args=(child_conn,))
p.start()
print parent_conn.recv() # prints "[42, None, 'hello']"
6.3 同步
添加锁:
from multiprocessing import Process, Lock
def f(l, i):
l.acquire()
print 'hello world', i
l.release()
if __name__ == '__main__':
lock = Lock()
for num in range(10):
Process(target=f, args=(lock, num)).start()
6.4 共享状态
应该尽量避免共享状态.
共享内存方式:
1 from multiprocessing import Process, Value, Array
2
3 def f(n, a):
4 n.value = 3.1415927
5 for i in range(len(a)):
6 a[i] = -a[i]
7
8 if __name__ == '__main__':
9 num = Value('d', 0.0)
10 arr = Array('i', range(10))
11
12 p = Process(target=f, args=(num, arr))
13 p.start()
14 p.join()
15
16 print num.value
17 print arr[:]
Server进程方式:
from multiprocessing import Process, Manager
def f(d, l):
d[1] = '1'
d['2'] = 2
d[0.25] = None
l.reverse()
if __name__ == '__main__':
manager = Manager()
d = manager.dict()
l = manager.list(range(10))
p = Process(target=f, args=(d, l))
p.start()
p.join()
print d
print l
第二种方式支持更多的数据类型，如list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Queue, Value ,Array.
6.5 Pool类
通过Pool类可以建立进程池:
1 from multiprocessing import Pool
2
3 def f(x):
4 return x*x
5
6 if __name__ == '__main__':
7 pool = Pool(processes=4) # start 4 worker processes
8 result = pool.apply_async(f, [10]) # evaluate "f(10)" asynchronously
9 print result.get(timeout=1) # prints "100" unless your computer is *very* slow
10 print pool.map(f, range(10)) # prints "[0, 1, 4,..., 81]"
7 multiprocessing.dummy
在官方文档只有一句话:
multiprocessing.dummy replicates the API of multiprocessing but is no more than a wrapper around the threading module.
-
multiprocessing.dummy是 multiprocessing 模块的完整克隆，唯一的不同在于 multiprocessing 作用于进程，而 dummy 模块作用于线程; - 可以针对 IO 密集型任务和 CPU 密集型任务来选择不同的库.
IO 密集型任务选择multiprocessing.dummy，CPU 密集型任务选择multiprocessing.
举例:
1 import urllib2
2 from multiprocessing.dummy import Pool as ThreadPool
3
4 urls = [
5 'http://www.python.org',
6 'http://www.python.org/about/',
7 'http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html',
8 'http://www.python.org/doc/',
9 'http://www.python.org/download/',
10 'http://www.python.org/getit/',
11 'http://www.python.org/community/',
12 'https://wiki.python.org/moin/',
13 'http://planet.python.org/',
14 'https://wiki.python.org/moin/LocalUserGroups',
15 'http://www.python.org/psf/',
16 'http://docs.python.org/devguide/',
17 'http://www.python.org/community/awards/'
18 # etc..
19 ]
20
21 # Make the Pool of workers
22 pool = ThreadPool(4)
23 # Open the urls in their own threads
24 # and return the results
25 results = pool.map(urllib2.urlopen, urls)
26 #close the pool and wait for the work to finish
27 pool.close()
28 pool.join()
29
30 results = []
31 for url in urls:
32 result = urllib2.urlopen(url)
33 results.append(result)
8 后记
- 如果选择多线程，则应该尽量使用
threading模块，同时注意GIL的影响- 如果多线程没有必要，则使用多进程模块
multiprocessing，此模块也通过multiprocessing.dummy支持多线程.- 分析具体任务是I/O密集型，还是CPU密集型
