[問題] httpwebrequest請求逾時
目前的專案是需要對多個網站抓數據下來
有將httpwebrequest包成一個方法,並使用多線程去跑
方法最後有將request abort以及response close還有設定為null
方法最前有GC.Collect();
request的keep-alive設定為false
ServicePointManager.DefaultConnectionLimit設定為512
十多個網站中,有兩個常會出現逾時,其他網站不會
軟體單獨對這兩個網站取數據不會出現
同時對十多個網站抓取數據才會發生
每個網站各自的抓取間隔皆為2秒
原本想說是不是網站本身問題,但是不全部一起抓就不會發生逾時
想請問板上前輩,還有什麼問題是可能造成這樣的狀況發生?
感謝
---------------------------------------------------------
補充程式碼
public class MyRequest
{
string UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.2; zh-TW;
rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 ( .NET CLR 3.5.30729)";
string Accept = "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg,
application/x-shockwave-flash, application/vnd.ms-excel,
application/vnd.ms-powerpoint, application/msword, application/json,
text/javascript,text/html, application/xhtml+xml, */*";
int Timeout = 60000;
private bool CheckValidationResult(object sender, X509Certificate
certificate, X509Chain chain, SslPolicyErrors errors)
{
return true;
}
public List<string> Get(string Url, string Referer, CookieContainer
CC, bool UsingXMLHttpRequest, bool UsingProxy, ReturnType RT, string
HeaderName)
{
List<string> ReturnList = new List<string>();
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(Url);
if (Url.StartsWith("https", StringComparison.OrdinalIgnoreCase))
{
ServicePointManager.ServerCertificateValidationCallback = new
RemoteCertificateValidationCallback(CheckValidationResult);
req.ProtocolVersion = HttpVersion.Version10;
}
req.Referer = Referer;
req.CookieContainer = CC;
req.UserAgent = UserAgent;
req.Accept = Accept;
req.Timeout = Timeout;
req.Headers["Accept-Language"] = "zh-TW";
req.AutomaticDecompression = DecompressionMethods.GZip;
req.KeepAlive = false;
if (UsingXMLHttpRequest)
{
req.Headers["X-Requested-With"] = "XMLHttpRequest";
}
if (UsingProxy)
{
req.Proxy = Proxy;
}
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(res.GetResponseStream());
if (RT == ReturnType.ResponseContent)
{
string str = sr.ReadToEnd();
ReturnList.Add(str);
}
else if (RT == ReturnType.ResponseUrl)
{
string str = res.ResponseUri.ToString();
ReturnList.Add(str);
}
else if (RT == ReturnType.ResponseHeader)
{
string HeaderStr = string.Empty;
if (HeaderName.Contains(","))
{
string[] HNarray = HeaderName.Split(',');
for (int i = 0; i < HNarray.Length; i++)
{
HeaderStr += res.GetResponseHeader(HNarray[i]);
if (i != HNarray.Length - 1)
{
HeaderStr += "|";
}
}
ReturnList.Add(HeaderStr);
}
else
{
HeaderStr = res.GetResponseHeader(HeaderName);
ReturnList.Add(HeaderStr);
}
}
else if (RT == ReturnType.All)
{
string str1 = sr.ReadToEnd();
string str2 = res.ResponseUri.ToString();
ReturnList.Add(str1);
ReturnList.Add(str2);
string HeaderStr = string.Empty;
if (HeaderName.Contains(","))
{
string[] HNarray = HeaderName.Split(',');
for (int i = 0; i < HNarray.Length; i++)
{
HeaderStr += res.GetResponseHeader(HNarray[i]);
if (i != HNarray.Length - 1)
{
HeaderStr += "|";
}
}
ReturnList.Add(HeaderStr);
}
else
{
HeaderStr = res.GetResponseHeader(HeaderName);
ReturnList.Add(HeaderStr);
}
}
sr.Close();
res.Close();
req.Abort();
return ReturnList;
}
public enum ReturnType
{
ResponseContent,
ResponseUrl,
ResponseHeader,
All
}
}
個人將上面方法包成一個DLL
專案本身使用backgroundwork來跑
Dowork內使用Get()取資料
結束後thread.sleep(2000);
之後進到RunWorkerCompleted再重新呼叫BGW.RunWorkerAsync();
----------------------------------------------------------
更新進度
怎樣修改都沒有好轉
原本測試的平台是Server2003 X86 種花固I
轉移到Win7 X64 種花浮動
什麼問題都沒有了....
在Server2008 X64上也都沒有問題
專案編輯是32位元的
也看過工作管理員是*32在執行
想請問這樣可能會是什麼問題?
對了,專案是.net 4.0
--
※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 111.240.231.238
※ 文章網址: https://www.ptt.cc/bbs/C_Sharp/M.1440193774.A.9B5.html
→
08/22 08:24, , 1F
08/22 08:24, 1F
→
08/22 08:30, , 2F
08/22 08:30, 2F
→
08/22 08:30, , 3F
08/22 08:30, 3F
→
08/22 08:31, , 4F
08/22 08:31, 4F
→
08/22 08:33, , 5F
08/22 08:33, 5F
→
08/22 08:35, , 6F
08/22 08:35, 6F
→
08/22 08:35, , 7F
08/22 08:35, 7F
→
08/22 08:37, , 8F
08/22 08:37, 8F
※ 編輯: unstoppable (111.240.231.238), 08/22/2015 15:14:21
→
08/22 15:24, , 9F
08/22 15:24, 9F
→
08/22 21:08, , 10F
08/22 21:08, 10F
→
08/22 21:31, , 11F
08/22 21:31, 11F
→
08/22 21:31, , 12F
08/22 21:31, 12F
→
08/22 21:32, , 13F
08/22 21:32, 13F
→
08/22 21:32, , 14F
08/22 21:32, 14F
感謝fo細心回應
用BGW主要是因為更新UI界面太方便了~XD(明明就是自己懶得寫thread委派)
頻寬應該不會是主要的原因,數據都是1~2X K的大小
目前看線程同時間最多7個BGW在跑
GetResponseAsyn這個我研究下,第一次看到
※ 編輯: unstoppable (111.240.231.238), 08/22/2015 23:52:32
※ 編輯: unstoppable (111.240.231.238), 08/23/2015 07:17:40
C_Sharp 近期熱門文章
PTT數位生活區 即時熱門文章
