问题描述:pyspider运行一段时间(一天到三天不等)后调度器自动退出.重启pyspider能继续运行
运行环境:linux mint 17.2 通过pip安装的pyspider.使用mongodb做为数据库.
启动配置:
{ "taskdb": "mongodb+taskdb://localhost/taskdb", "projectdb": "mongodb+projectdb://localhost/projectdb", "resultdb": "mongodb+resultdb://localhost/resultdb", "webui": { "username": "admin", "password": "123456", "need-auth": true } }
终端报错:
[E 160522 09:37:38 scheduler:499] localhost:27017: [Errno -2] Name or service not known Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 494, in run self.run_once() File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 1061, in run_once super(ThreadBaseScheduler, self).run_once() File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 478, in run_once self._update_projects() File "/usr/local/lib/python2.7/dist-packages/pyspider/scheduler/scheduler.py", line 93, in _update_projects for project in self.projectdb.check_update(self._last_update_project): File "/usr/local/lib/python2.7/dist-packages/pyspider/database/mongodb/projectdb.py", line 62, in check_update for project in self.get_all(fields=('updatetime', 'name')): File "/usr/local/lib/python2.7/dist-packages/pyspider/database/mongodb/projectdb.py", line 50, in get_all for each in self.collection.find({}, fields): File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1090, in next if len(self.__data) or self._refresh(): File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1012, in _refresh self.__read_concern)) File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 850, in __send_message **kwargs) File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 798, in _send_message_with_response exhaust) File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 809, in _reset_on_error return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/pymongo/server.py", line 88, in send_message_with_response with self.get_socket(all_credentials, exhaust) as sock_info: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/pymongo/server.py", line 152, in get_socket with self.pool.get_socket(all_credentials, checkout) as sock_info: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/pymongo/pool.py", line 539, in get_socket sock_info = self._get_socket_no_auth() File "/usr/local/lib/python2.7/dist-packages/pymongo/pool.py", line 573, in _get_socket_no_auth sock_info, from_pool = self.connect(), False File "/usr/local/lib/python2.7/dist-packages/pymongo/pool.py", line 512, in connect _raise_connection_failure(self.address, error) File "/usr/local/lib/python2.7/dist-packages/pymongo/pool.py", line 65, in _raise_connection_failure raise AutoReconnect(msg) AutoReconnect: localhost:27017: [Errno -2] Name or service not known [I 160522 09:37:38 scheduler:505] scheduler exiting...
应该是pyspider连接不上mangodb导致的。
Name or service not known,可能是pyspider识别不了mangodb的HOST。
查看是否mangodb的原因,查看mangodb的日志,看是否有异常
查看是否网络原因,1. 把localhost换成127.0.0.1或者本机IP 2. 参考http://blog.csdn.net/sahenzhang/article/details/8181685