[問題] 爬蟲錯誤
應用隨機休息再繼續，還是會出現這樣的錯誤
error: [Errno 10054] 遠端主機已強制關閉一個現存的連線。
進行except例外處理，continue繼續的話
後面nameTag對應到的genre,rating 全部都會打亂
這樣應該要怎麼處理?
謝謝
for i in idlist:
headers = {'User-Agent':'Mozilla/5.0 (Windows; U; Windows NT
6.1;en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'}
req =
urllib2.Request("http://www.imdb.com/title/tt"+i+"/",headers=headers)
try:
html =
urllib2.urlopen("http://www.imdb.com/title/tt"+i+"/",timeout = 30)
htmls = html.read()
html.close
soup = BeautifulSoup(htmls)
nameTag = [a.get_text() for a in soup.find_all("title")]
genreTag = [a.get_text() for a in
soup.find_all("span",{"itemprop":"genre"})]
ratingTag = soup.find_all("span",{"itemprop":"ratingValue"})
for tag in nameTag:
titlelist.append(nameTag)
for tag in genreTag:
genrelist.append(genreTag)
break
for tag in ratingTag:
val = ''.join(tag.find(text=True))
valuelist.append(val)
except HTTPError, e:
print e.code
print e.read()
#continue
except URLError, e:
print 'Reason: ', e.reason
#continue
rsleep = random.randint(10, 40)
time.sleep(rsleep)
return zip(titlelist, genrelist, valuelist)
--
※ 發信站: 批踢踢實業坊(ptt.cc)
◆ From: 118.160.190.62
→
05/19 22:30, , 1F
05/19 22:30, 1F
→
05/20 01:09, , 2F
05/20 01:09, 2F
→
05/20 12:58, , 3F
05/20 12:58, 3F
→
05/20 15:38, , 4F
05/20 15:38, 4F
→
05/20 15:39, , 5F
05/20 15:39, 5F
→
05/20 15:44, , 6F
05/20 15:44, 6F
→
05/20 15:51, , 7F
05/20 15:51, 7F
→
05/20 15:52, , 8F
05/20 15:52, 8F
討論串 (同標題文章)
Python 近期熱門文章
PTT數位生活區 即時熱門文章
