爬取的内容：书籍名称，作者名称，书籍简介，全书网5041页大约16万条数据,写入mysql数据库和.txt文件
1，创建scrapy项目
scrapy startproject numberone
2，创建爬虫主程序
cd numberone scrapy genspider quanshuwang www.quanshuwang.com
3，setting中设置请求头
user_agent = "mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, like gecko) chrome/39.0.2171.71 safari/537.36"
4，item中设置要爬取的字段
class numberoneitem(scrapy.item):
# define the fields for your item here like:
# name = scrapy.field()
book_author = scrapy.field()
book_name = scrapy.field()
book_desc = scrapy.field()
5，quanshuwang.py主程序中写获取数据的主代码
# -*- coding: utf-8 -*-
import scrapy
from numberone.items import numberoneitem
class qiubaispider(scrapy.spider):
name = 'quanshuwang'
# 这句话是定义爬虫爬取的范围，最好注释掉
# allowed_domains = ['www.qiushibaike.com']
# 开始爬取的路由
start_urls = ['http://www.quanshuwang.com/list/0_1.html']
def parse(self, response):
book_list = response.xpath('//ul[@class="seewell cf"]/li')
for i in book_list:
item = numberoneitem()
item['book_name'] = i.xpath('./span/a/text()').extract_first()
item['book_author'] = i.xpath('./span/a[2]/text()').extract_first()
item['book_desc'] = i.xpath('./span/em/text()').extract_first()
yield item
next = response.xpath('//a[@class="next"]/@href').extract_first()
if next:
yield scrapy.request(next, callback=self.parse)
6，pipelines.py管道文件中文件中写持久化保存.txt和mysql。
# -*- coding: utf-8 -*-
# define your item pipelines here
#
# don't forget to add your pipeline to the item_pipelines setting
# see: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
import pymysql
# 写入文件的类
class numberonepipeline(object):
f = none
def open_spider(self,spider):
self.f = open('全书网.txt','a+',encoding='utf-8')
def process_item(self, item, spider):
print(item['book_name']+'：正在写入文件...')
book_name = item['book_name']
book_author = item['book_author']
book_desc = item['book_desc']
self.f.write('书名：'+book_name+'\n'+'作者：'+book_author+'\n'+'书籍简介：'+book_desc+'\n\n')
return item
def close_spider(self,spider):
self.f.close()
# 写入数据库的类
class mysqlpipeline(object):
conn = none
mycursor = none
def open_spider(self,spider):
self.conn = pymysql.connect(host='172.16.25.4',user='root',password='root',db='quanshuwang')
self.mycursor = self.conn.cursor()
def process_item(self, item, spider):
print(item['book_name'] + '：正在写数据库...')
book_name = item['book_name']
book_author = item['book_author']
book_desc = item['book_desc']
self.mycursor = self.conn.cursor()
sql = 'insert into qsw values (null,"%s","%s","%s")'%(book_name,book_author,book_desc)
bool = self.mycursor.execute(sql)
self.conn.commit()
return item
def close_spider(self,spider):
self.conn.close()
self.mycursor.close()
7，setting.py文件中打开管道文件。
item_pipelines = {
'numberone.pipelines.numberonepipeline': 300,
'numberone.pipelines.mysqlpipeline': 400,
}
8，执行运行爬虫的命令
scrapy crawl quanshuwang --nolog
9,控制台输出
贵府嫡女：正在写数据库... 随身空间农女翻身记：正在写入文件... 随身空间农女翻身记：正在写数据库... 阴间商人：正在写入文件... 阴间商人：正在写数据库... 我的美味有属性：正在写入文件... 我的美味有属性：正在写数据库... 剑仙修炼纪要：正在写入文件... 剑仙修炼纪要：正在写数据库... 在阴间上班的日子：正在写入文件... 在阴间上班的日子：正在写数据库... 轮回之鸿蒙传说：正在写入文件... 轮回之鸿蒙传说：正在写数据库... 末日星城：正在写入文件... 末日星城：正在写数据库... 异域神州道：正在写入文件... 异域神州道：正在写数据库...
10，打开文件和数据库查看是否写入成功
done。
