Scraping dal sito Web di aspx utilizzando R

Question

Sto cercando di eseguire un'attività utilizzando R per raschiare i dati su un sito Web.

1. Vorrei passare attraverso ogni link sulla pagina seguente: http://capitol.hawaii.gov/advreports/advreport.aspx?year=2013&report=deadline&rpt_type=&measuretype=hb&title=House fatture

2. selezionare solo gli elementi con lo stato attuale, che indichi "trasmessa al governatore". Ad esempio, http://capitol.hawaii.gov/measure_indiv.aspx?billtype=HB&billnumber=17&year=2013

3. E quindi rottamare le celle all'interno di STATUS TEXT per la seguente clausola "Lettura finale passata". Ad esempio: Lettura finale passata come modificata in SD 2 con i rappresentanti Fale, Jordan, Tsuji che vota con riserve; Rappresentante (i) Cabanilla, Morikawa, Oshiro, Tokioka hanno votato no (4) e nessuno ha scusato (0).

Ho provato con esempi precedenti con i pacchetti Rcurl e XML (in R), ma non so come usarli correttamente per i siti aspx. Quindi quello che mi piacerebbe avere è: 1. Alcuni suggerimenti su come costruire un tale codice. 2. Raccomandazione su come apprendere le conoscenze necessarie per svolgere tale compito.

Grazie per qualsiasi aiuto,

Tom

Answer 1

require(httr) 
require(XML) 

basePage <- "http://capitol.hawaii.gov" 

h <- handle(basePage) 

GET(handle = h) 

res <- GET(handle = h, path = "/advreports/advreport.aspx?year=2013&report=deadline&rpt_type=&measuretype=hb&title=House") 

# parse content for "Transmitted to Governor" text 
resXML <- htmlParse(content(res, as = "text")) 
resTable <- getNodeSet(resXML, '//*/table[@id ="GridViewReports"]/tr/td[3]') 
appRows <-sapply(resTable, xmlValue) 
include <- grepl("Transmitted to Governor", appRows) 
resUrls <- xpathSApply(resXML, '//*/table[@id ="GridViewReports"]/tr/td[2]//@href') 

appUrls <- resUrls[include] 

# look at just the first 

res <- GET(handle = h, path = appUrls[1]) 

resXML <- htmlParse(content(res, as = "text")) 


xpathSApply(resXML, '//*[text()[contains(.,"Passed Final Reading")]]', xmlValue) 

[1] "Passed Final Reading as amended in SD 2 with Representative(s) Fale, Jordan, 
Tsuji voting aye with reservations; Representative(s) Cabanilla, Morikawa, Oshiro, 
Tokioka voting no (4) and none excused (0)." 

pacchetto Let httr gestire tutto il lavoro di fondo attraverso la creazione di un handle.

Se si desidera eseguire su tutti i 92 collegamenti:

# get all the links returned as a list (will take sometime) 
# print statement included for sanity 
res <- lapply(appUrls, function(x){print(sprintf("Got url no. %d",which(appUrls%in%x))); 
            GET(handle = h, path = x)}) 
resXML <- lapply(res, function(x){htmlParse(content(x, as = "text"))}) 
appString <- sapply(resXML, function(x){ 
        xpathSApply(x, '//*[text()[contains(.,"Passed Final Reading")]]', xmlValue) 
         }) 


head(appString) 

> head(appString) 
$href 
[1] "Passed Final Reading as amended in SD 2 with Representative(s) Fale, Jordan, Tsuji voting aye with reservations; Representative(s) Cabanilla, Morikawa, Oshiro, Tokioka voting no (4) and none excused (0)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 25 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 0 Excused: none."             
[2] "Passed Final Reading as amended in CD 1 with Representative(s) Cullen, Har voting aye with reservations; Representative(s) McDermott voting no (1) and none excused (0)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 25 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 0 Excused: none."         
[2] "Passed Final Reading as amended in CD 1 with none voting aye with reservations; Representative(s) Hashem, McDermott voting no (2) and none excused (0)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 24 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 1 Excused: Ige."      
[2] "Passed Final Reading as amended in CD 1 with none voting aye with reservations; none voting no (0) and Representative(s) Say excused (1)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 25 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 0 Excused: none."       
[2] "Passed Final Reading as amended in CD 1 with Representative(s) Johanson voting aye with reservations; none voting no (0) and none excused (0)." 

$href 
[1] "Passed Final Reading, as amended (CD 1). 25 Aye(s); Aye(s) with reservations: none . 0 No(es): none. 0 Excused: none." 
[2] "Passed Final Reading as amended in CD 1 with none voting aye with reservations; none voting no (0) and none excused (0)."