[問題] 透過R使用Rhdfs package讀取超大CSV檔
[問題類型]:
程式諮詢(我想用R 做某件事情，但是我不知道要怎麼用R 寫出來)
[軟體熟悉度]:
入門(寫過其他程式，只是對語法不熟悉)
[問題敘述]:
使用R透過RHadoop的Rhdfs 1.0.8
[程式範例]:
我目前的實驗環境，需要讀取很大的CSV File（存放在Hadoop的HDFS上，檔案大小幾乎
都大於20GB），
我使用了RHdoop的rhdfs R Package
Ref.
https://github.com/RevolutionAnalytics/RHadoop/wiki
使用Rstudio Web版開發，原始碼如下
*************************************************************************************************
Sys.setenv(HADOOP_CMD="/usr/lib/hadoop/bin/hadoop")
Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-mapreduce/hadoop-streaming-2.2.0.2.0.6.0-101.jar")
Sys.setenv(HADOOP_COMMON_LIB_NATIVE_DIR="/usr/lib/hadoop/lib/native/")
library(rmr2);
library(rhdfs);
library(lubridate);
hdfs.init();
f = hdfs.file("/bigdata/rawdata/201312.csv","r",buffersize=104857600);
m = hdfs.read(f);
c = rawToChar(m);
data = read.table(textConnection(c), sep = ",");
*************************************************************************************************
讀完後，發現它只讀進了前一千五百多筆的資料，正確應該有一億多筆
*************************************************************************************************
去Google了一下，有查到下列這個解的方向
“rhdfs uses the java api for reading files stored in hdfs.
That api will not necessarily read the entire file in one shot.
It will return some number of bytes for each read.
When it reaches the end of the file it returns -1.
In the case of rhdfs, and end of the file will return NULL.
So, you need to loop on the hdfs.read call until NULL is returned”
不過看了rhdfs的手冊，並沒有仔細提到上面解法關於hdfs.read()的行為:<
不知道有人有這方面經驗嗎？
[關鍵字]:
R, Large Scale Data Set, Big Data, Hadoop, RHadoop, CSV, HDFS, rhdfs
Thanks in advance!
--
※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 60.251.38.1
※ 文章網址: http://www.ptt.cc/bbs/R_Language/M.1400472388.A.FDB.html
※ 編輯: dppman (60.251.38.1), 05/19/2014 12:07:32
※ 編輯: dppman (60.251.38.1), 05/19/2014 12:18:34
推
05/19 13:01, , 1F
05/19 13:01, 1F
→
05/19 15:13, , 2F
05/19 15:13, 2F
→
05/19 15:14, , 3F
05/19 15:14, 3F
推
05/19 15:46, , 4F
05/19 15:46, 4F
→
05/19 16:11, , 5F
05/19 16:11, 5F
→
05/19 16:22, , 6F
05/19 16:22, 6F
→
05/19 16:23, , 7F
05/19 16:23, 7F
→
05/19 16:25, , 8F
05/19 16:25, 8F
※ 編輯: dppman (59.120.150.53), 05/19/2014 16:26:22
→
05/19 16:33, , 9F
05/19 16:33, 9F
→
05/19 16:33, , 10F
05/19 16:33, 10F
→
05/19 16:33, , 11F
05/19 16:33, 11F
→
05/19 16:33, , 12F
05/19 16:33, 12F
→
05/26 10:31, , 13F
05/26 10:31, 13F
→
05/29 11:58, , 14F
05/29 11:58, 14F
→
05/29 11:58, , 15F
05/29 11:58, 15F
→
05/29 13:50, , 16F
05/29 13:50, 16F
→
05/29 13:51, , 17F
05/29 13:51, 17F
討論串 (同標題文章)
R_Language 近期熱門文章
PTT數位生活區 即時熱門文章
