基于Celery写爬虫却频繁退出，求救！

我们目前在基于 Celery 队列做定向爬虫。目标网页很简单，但存储的时候要将目标页上的各种信息转换、分解为结构化数据。在这期间需要查询很多次数据库。
采用 “发出任务 -> 队列 -> 在 Celery 中执行” 的方式。目前遇到以下问题

1. 使用 supervisor 管理 Celery 的运行。但发现每隔一段时间，启动的多个 Celery 进程就好一起退出（同时）。

   supervisor.log 如下（部分）

   1 2 3 4 5 6 7 8 9 10

   2015-05-09 03:02:56,462 INFO stopped: celerya (exit status 0) 2015-05-09 03:02:57,457 INFO stopped: celeryb (exit status 0) 2015-05-09 03:04:38,275 INFO spawned: 'celerya' with pid 3547 2015-05-09 03:04:48,529 INFO success: celerya entered RUNNING state, process has stayed up for > than 10 seconds (startsecs) 2015-05-09 03:07:55,995 INFO spawned: 'celeryb' with pid 3926 2015-05-09 03:08:06,337 INFO success: celeryb entered RUNNING state, process has stayed up for > than 10 seconds (startsecs) 2015-05-09 03:09:01,861 INFO stopped: celerya (exit status 0) 2015-05-09 03:11:29,792 INFO stopped: celeryb (exit status 0) 2015-05-09 03:12:02,037 INFO spawned: 'celerya' with pid 4706 2015-05-09 03:12:03,044 INFO spawned: 'celeryb' with pid 4710

   
   

   退出时 celery 的异常信息如下：

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

   Traceback (most recent call last):   File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start     self.blueprint.start(self)   File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start     step.start(parent)   File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start     return self.obj.start()   File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start     blueprint.start(self)   File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start     step.start(parent)   File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start     c.loop(*c.loop_args())   File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/celery/worker/loops.py", line 97, in synloop     connection.drain_events(timeout=2.0)   File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events     return self.transport.drain_events(self.connection, **kwargs)   File "/usr/local/opt/pyenv/versions/calvino/lib/python2.7/site-packages/kombu/transport/virtual/__init__.py", line 840, in drain_events     message, queue = item TypeError: 'NoneType' object is not iterable

   
   
   



2. 在运行 4 小时之后，celery 进程虽然活着，但却停止工作，无 log，手动重启之后恢复工作。但 80% 的任务出现数据库异常，提示

   1	IntegrityError: (IntegrityError) (1062, u"Duplicate entry '201173000000006801' for key 'PRIMARY'")

   错误。手动查找 改主键却不存在。



3. 在运行 2 小时左右之后，开始出现 陆续出现主键重复错误，手动查找，该数据确实存在。但实际上应该不存在。因为该条数据就是在本次任务中创建的。

附： celery 运行参数：