Getting a sample from a large data file with R

I'm working on a little project to attempt to cluster the voters in North Carolina into congressional districts.  My goal is to see if there's a way to have a computer draw the districts instead of relying on people with potential biases.

I quickly ran into quite a big wall when I reviewed the file listing all the voters in North Carolina.  I should have suspected that it would be huge!  A file consisting of around 7.5 million voters (both active and invalid) and around 60 columns is about 4.5 gigabytes.  Considering I have 4 gigabytes of RAM, I needed an alternative plan.

Well, I know that I just wanted a sample of this file.  I'm going to try and geocode these addresses and use the lat/long coordinates for the clustering.  As I've stated in previous posts, geocoding has daily limits. Geocoding tons of addresses can take serious time.

I didn't have too much success using the standard read.table function in R.  There are skip and nrow parameters, but they didn't seem to help too much when dealing with my RAM woes.  I also tried the Fread package, but my data had some flaws and Fread wasn't too flexible working around it.

I took a really simplistic approach to my problem by utilizing the lowly file command that comes standard with R.  First, a loop with the file command went through each line and copied only rows that didn't have problems.  In my situation, there were extra quote symbols in some of the lines. Those lines weren't worth it.  So, I skipped them.

 I also took out voters that weren't listed as ACTIVE or INACTIVE.

v <- file("c:\\users\\doug shartzer\\documents\\data\\ncvoter_Statewide.txt")

open(v)

while(length(line <- readLines(v,1)) > 0) {

if (sum(table(strsplit(line,'"'))) == 140) {

if (strsplit(line,'"')[[1]][[10]] == 'ACTIVE' | strsplit(line,'"')[[1]][[10]] == 'INACTIVE' ) {

write(line, 'c:\\users\\doug shartzer\\documents\\data\\voter_good_all.txt',append=T)

}

}

if (sum(table(strsplit(line,'"'))) != 140) {

print(line)

}

}

close(v)

q()

Although it did take a while to run (16 hours), I didn't run into any problems with memory.

After that, I collected a sample and wrote those voters to another file.

s <- sample(7500000, 75000)

v <- file("c:\\users\\doug shartzer\\documents\\data\\voter_good_all.txt")

open(v)

while(length(line <- readLines(v,1)) > 0) {

if (x %in% s){

write(line, 'c:\\users\\doug shartzer\\documents\\data\\voter_sample_03142014.txt',append=T)

}

}

shartzer\\documents\\data\\voter_run_status.txt',append=T)

close(v)

q()

After this process, I had a much more manageable file to play around with.

Best Football Conference for Getting Drafted

A few weeks ago I was talking to my coworker about how there's a perception that SEC teams dominate college football.  If this were true, I speculated this may be due to better recruiting.  Winning schools typically recruit better.  Recruits want to play for winners. Teams with more exposure and evidence of success are more likely to have their players get drafted into the NFL.

However, is this really the case?  Do players in winning conferences, specifically the big, bad SEC, get drafted higher than players on "weaker" conferences?  Does the best recruiting conference perform the best on the field?

According to scout.com,  the SEC conference is, in fact, typically the best recruiting school.

As you can see from the chart above, the SEC typically gets more top 100 recruits than the other big six conferences (ACC, Big Ten, PAC 12, Big 12 and Big East).  The SEC's only been bested ONCE since 2002.

Does recruiting equate to wins?  They're pretty good each year, but it's not as clear cut as you would think.  However, the SEC is always one of the more highly ranked conferences when the dust settles at the end of the season, according to the Associated Press rankings.

You can see here the SEC teams usually fare better with top 25 votes than schools in other conferences.  However, there have a few years where other conferences have outperformed SEC schools.

Does the SEC pay dividends to the recruits who commit to their schools and help collect top 25 rankings when compared to the other conferences?  NFL teams do draft more SEC players than the other conferences in the majority of years since 2000, but it's certainly not a safe bet.

 This chart show that both the ACC and Big 10 have had more draft picks than the SEC on two occasions since 2000.  In most years, the margins between conference schools is pretty narrow.

So, Recruits fare pretty well on NFL draft day when committing to SEC schools, but it isn't necessarily always THE safest conference to bet on.  The SEC does the best job recruiting, but it doesn't ALWAYS equate to being the dominate conference every year.