[問題] 爬蟲 - requests後的資料查詢 (已解決)

看板Python作者truth1999 (真)時間7年前 (2018/03/17 10:48)推噓8(8推 0噓 34→)

留言42則, 5人參與討論串1/1

大家好，小弟剛接觸python幾天... 有個資料查詢的問題一直無法解決 (http://amis.afa.gov.tw/m_coop/CoopVegFullMarketMainProd.aspx ) 該網站查詢的資料結果是url不變並以js呈現(請見諒非本科系的形容@@") 且結果若超過60筆，才會出現下一頁的按鈕可以點選所以碰到的問題是在顯示資料之前第2、3頁...下一頁的按鈕不會出現而按鈕的方式是以javascripl：__doPostBack的方式寫入因此問題有兩個： 1.我該如何才能獲得第1頁之後的資料目前查到的方法好像要利用requests.session方法但是怎麼寫都跑不出來 2.有辦法可以在查詢後，直接利用"下載"這個按鈕下載資料嗎爬文後不知道能否利用request後，再配合selenium 只是找到的文章大多是利用cookie帶入，好像不適合這個網頁不知道是否有善心版友可以協助指導新手... 以下附上查詢的code跟試著加入session的code 單純查詢一次性資料： # -*- coding: utf-8 -*- import requests from bs4 import BeautifulSoup import pandas as pd resp = requests.get('http://amis.afa.gov.tw/m_coop/CoopVegFullMarketMainProd.aspx') soup = BeautifulSoup(resp.text, 'html5lib') view_state = soup.find(id='__VIEWSTATE')['value'] event_validation = soup.find(id='__EVENTVALIDATION')['value'] viewstate_generator = soup.find(id='__VIEWSTATEGENERATOR')['value'] form_data = { '__VIEWSTATE': view_state, '__VIEWSTATEGENERATOR': viewstate_generator, '__EVENTVALIDATION': event_validation, 'ctl00$contentPlaceHolder$txtStartDate':'106/12/01', 'ctl00$contentPlaceHolder$txtEndDate':'107/03/15', 'ctl00$contentPlaceHolder$ucCoopVegFruitMarket$radlMarketRange':'P', 'ctl00$contentPlaceHolder$ucCoopVegFruitMarket$lstMarket':'109', 'ctl00$contentPlaceHolder$ucCoopVegFruitMainProduct$radlProductType':'V', 'ctl00$contentPlaceHolder$ucCoopVegFruitMainProduct$lstProduct':'FA0', 'ctl00$contentPlaceHolder$dropProductClass':'1', 'ctl00$contentPlaceHolder$btnQuery':'查詢' } resp = requests.post('http://amis.afa.gov.tw/m_coop/CoopVegFullMarketMainProd.aspx', data=form_data) df = pd.read_html(resp.text)[6].dropna(axis=0, how='any') print(df) --------------------------------------------------------------------------- 加上session，但是資料一直無法跳到第2頁> < ： import requests from bs4 import BeautifulSoup import pandas as pd resp = requests.get('http://amis.afa.gov.tw/m_coop/CoopVegFullMarketMainProd.aspx') soup = BeautifulSoup(resp.text, 'html5lib') with requests.session() as s: s.headers['user-agent'] = 'Mozilla/5.0' soup = BeautifulSoup(resp.content, 'html5lib') target = ( 'ctl00$contentPlaceHolder$dataPager$ctl02$ctl{:02d}' ) # unsupported CSS Selector 'input[name^=ctl00][value]' data = { tag['name']: tag['value'] for tag in soup.select('input[name^=ctl00]') if tag.get('value') } state = { tag['name']: tag['value'] for tag in soup.select('input[name^=__]') } data.update(state) view_state = soup.find(id='__VIEWSTATE')['value'] event_validation = soup.find(id='__EVENTVALIDATION')['value'] viewstate_generator = soup.find(id='__VIEWSTATEGENERATOR')['value'] form_data = { '__VIEWSTATE': view_state, '__VIEWSTATEGENERATOR': viewstate_generator, '__EVENTVALIDATION': event_validation, 'ctl00$contentPlaceHolder$txtStartDate':'106/12/01', 'ctl00$contentPlaceHolder$txtEndDate':'107/03/15', 'ctl00$contentPlaceHolder$ucCoopVegFruitMarket$radlMarketRange':'P', 'ctl00$contentPlaceHolder$ucCoopVegFruitMarket$lstMarket':'109', 'ctl00$contentPlaceHolder$ucCoopVegFruitMainProduct$radlProductType':'V', 'ctl00$contentPlaceHolder$ucCoopVegFruitMainProduct$lstProduct':'FA0', 'ctl00$contentPlaceHolder$dropProductClass':'1', 'ctl00$contentPlaceHolder$btnQuery':'查詢' } # data['ctl00$FhMainContent$FhContent$ctl00$AnalysesCourse$CustomPager$total'] resp = requests.post('http://amis.afa.gov.tw/m_coop/CoopVegFullMarketMainProd.aspx', data=form_data) soup = BeautifulSoup(resp.text, 'html5lib') sl = soup.find_all("span") last_page = '+'.join([span.get_text() for span in sl][-2]) # for page in range(last_page + 1): for page in range(int(last_page)): data['__EVENTTARGET'] = target.format(page) df = pd.read_html(resp.text)[7].dropna(axis=0, how='any') print(df) 謝謝大家 -- ※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 180.217.170.50 ※ 文章網址: https://www.ptt.cc/bbs/Python/M.1521254934.A.7ED.html ※ 編輯: truth1999 (180.217.170.50), 03/17/2018 10:51:12

→