Showing posts with label R. Show all posts
Showing posts with label R. Show all posts

Thursday, September 4, 2014

Collecting Camera (EXIF) Data through R

Slowly, but surely, I'm working on a way to organize all my photos on the various folders on my computer.  It's a mess.  I've got folders with thousands of pictures, copies of folders, files with different names, etc.

I've been trying to come up with a way to create a nice, organized folder with only one copy of each of my photos with them all organized into subfolders.  It's tricky, though, because I don't really know what I have.

Enter EXIF data.  This is data that the camera applies to each file it creates when it captures a photo.  There's potentially a lot of data available, but it depends on the camera manufacturer.  This can include the photos creation date, the dimensions of the photo, the camera make and model, and gps data, such as latitude, longitude and altitude.

There's supposedly a unique ID value cameras can assign, which is supposed to be globally unique.  Unfortunately, most of the pictures I took didn't have that available.  However, there's plenty of other data that I can combine to check for uniqueness.

There's this great, free tool that you can use to read EXIF data.  You use the tool through a command prompt window.  This really works great since you can interact with a command prompt window using R.

Building on what I learned from this blog post, I built a function that I can utilize to go through a bunch of my photo files.  It's pretty basic.

#This function calls exiftool, a command line application that returns exif data from photo files, searches the file for pertinent data, and returns those values in a

#vector.  If nothing is found, ‘UNKNOWN’ is returned.


getexifdata <- function(filename){

cmd <- paste('exiftool -c ' ,shQuote('%.6f'), shQuote(filename)) #create the MSDOS command we’ll be using.

exifdata <-  system(cmd,intern=T)  #use the MSDOS command using the system function.


#the system command that calls exiftool returns a vector.  Each line consists of one property value of the photo.  It’s starts with the name of the property and the actual value starts at the 35th character.

#these next few lines are searching the returned vector for specific property value using the name found at the beginning.  

#If found, collect the property value starting at character 35.  Since there can be multiple matches, collect only the first item found. Search different possible labels as camera companies name stuff differently.


imageheight <- substring(exifdata [grep('^Exif Image Height        |Image Height  ',exifdata )[1]],35,nchar(exifdata [grep('^Exif Image Height       |Image Height  ',exifdata )[1]]))

imagewidth <- substring(exifdata [grep('^Exif Image Width      |Image Width ',exifdata )[1]],35,nchar(exifdata [grep('^Exif Image Width      |Image Width ',exifdata )[1]]))

gpslatitude <- substring(exifdata [grep('^GPS Latitude      ',exifdata )[1]],35,nchar(exifdata [grep('^GPS Latitude       ',exifdata )[1]]))

gpslongitude <- substring(exifdata [grep('^GPS Longitude      ',exifdata )[1]],35,nchar(exifdata [grep('^GPS Longitude       ',exifdata )[1]]))

cameramodel <- substring(exifdata [grep('^Camera Model Name      ',exifdata )[1]],35,nchar(exifdata [grep('^Camera Model Name       ',exifdata )[1]]))

createdate <- substring(exifdata [grep('^Create Date      |File Creation Date',exifdata )[1]],35,nchar(exifdata [grep('^Create Date      |File Creation Date',exifdata )[1]]))


#If no value is found, NA is returned. Set to ‘UNKNOWN’


if (is.na(imagewidth)){imagewidth <- 'UNKNOWN'}

if (is.na(imageheight)){imageheight <- 'UNKNOWN'}

if (is.na(gpslatitude)){gpslatitude <- 'UNKNOWN'}

if (is.na(gpslongitude)){gpslongitude <- 'UNKNOWN'}

if (is.na(cameramodel)){cameramodel <- 'UNKNOWN'}

if (is.na(createdate)){createdate <- 'UNKNOWN'}

#return values as a vector.

return(c(imagewidth,imageheight, gpslatitude, gpslongitude,cameramodel, createdate))


}


NOTE: Your mileage may definitely vary on this one.  Camera models assign labels for their data differently.  If you're like me and you have pictures in your folders from lots of different camera manufacturers, you've got to take that into account and change the labels you are looking for. Some also put in their EXIF data values that others don't.

Another good place to get data on photo files is with the file.info function.  I talk about that here.

Wednesday, July 16, 2014

Get a custom file list with R

I'm working on a way to better manage my files on my computer.  I've got tons of duplicate photos and mp3s that I've copied to remote drives.  Additionally, I've got folders with hundreds of poorly named files.  Opening one of these folders is a nightmare if I happen to open one with a thumbnail view!

Because I've been playing around with it recently, I thought I would see what R could do to help me.  I've got a ways to go, but it looks like R has a great function that can put the specifics of a file within a data set, file.info.

file.info("C:\\users\\doug shartzer\\messy folder\\song1.mp3")

Additionally, you can provide file.info with more than one file at a time within a vector.  In fact, you can pass it an entire folder using the dir function on a folder, which will create a vector containing all the files within the provided folder.  Be sure to use the full.names argument on the dir function to get the full path, which file.info needs.

file.info(dir("c:\\users\\doug shartzer\\messy folder\\",full.names=T))

What's even better?  You can provide the dir function with a vector of folder names to get a giant dataset of file details with just one line of code.  It also has a recursive argument that'll include files within each folder's subfolder.  

file.info(dir(c("c:\\","d:\\","e:\\"), full.names=T, recursive=T)

Lastly, dir also allows you to limit your list further by accepting regular expressions in its pattern argument to limit the list of files returned. I can limit my list of files to just pictures and music files. 

 file.info(dir(c('c:\\','d:\\','e:\\'),recursive=T,full.names=T,pattern='+mpg$|+mp3$|+jpg$'))

So, building a list of the files I've got to go through appears to be a breeze!  Now I've got to figure out where to go from here...


Monday, March 17, 2014

Getting a sample from a large data file with R

I'm working on a little project to attempt to cluster the voters in North Carolina into congressional districts.  My goal is to see if there's a way to have a computer draw the districts instead of relying on people with potential biases.

I quickly ran into quite a big wall when I reviewed the file listing all the voters in North Carolina.  I should have suspected that it would be huge!  A file consisting of around 7.5 million voters (both active and invalid) and around 60 columns is about 4.5 gigabytes.  Considering I have 4 gigabytes of RAM, I needed an alternative plan.

Well, I know that I just wanted a sample of this file.  I'm going to try and geocode these addresses and use the lat/long coordinates for the clustering.  As I've stated in previous posts, geocoding has daily limits. Geocoding tons of addresses can take serious time.

I didn't have too much success using the standard read.table function in R.  There are skip and nrow parameters, but they didn't seem to help too much when dealing with my RAM woes.  I also tried the Fread package, but my data had some flaws and Fread wasn't too flexible working around it.

I took a really simplistic approach to my problem by utilizing the lowly file command that comes standard with R.  First, a loop with the file command went through each line and copied only rows that didn't have problems.  In my situation, there were extra quote symbols in some of the lines. Those lines weren't worth it.  So, I skipped them.

 I also took out voters that weren't listed as ACTIVE or INACTIVE.

v <- file("c:\\users\\doug shartzer\\documents\\data\\ncvoter_Statewide.txt")

open(v)

while(length(line <- readLines(v,1)) > 0) {

if (sum(table(strsplit(line,'"'))) == 140) {

if (strsplit(line,'"')[[1]][[10]] == 'ACTIVE' | strsplit(line,'"')[[1]][[10]] == 'INACTIVE' ) {

write(line, 'c:\\users\\doug shartzer\\documents\\data\\voter_good_all.txt',append=T)

}

}

if (sum(table(strsplit(line,'"'))) != 140) {

print(line)

}

}

close(v)

q()


Although it did take a while to run (16 hours), I didn't run into any problems with memory.

After that, I collected a sample and wrote those voters to another file.

s <- sample(7500000, 75000)

v <- file("c:\\users\\doug shartzer\\documents\\data\\voter_good_all.txt")

open(v)

while(length(line <- readLines(v,1)) > 0) {

if (x %in% s){

write(line, 'c:\\users\\doug shartzer\\documents\\data\\voter_sample_03142014.txt',append=T)

}

}

shartzer\\documents\\data\\voter_run_status.txt',append=T)

close(v)


q()

After this process, I had a much more manageable file to play around with.

Monday, November 11, 2013

Random Maps of Wake County Demographics

I've been playing around a lot with mapping data recently.  Here are some maps that represent voters in Wake County, North Carolina.  Each map represents a sample of 35000 voters collected from the North Carolina State Board of Elections in October 2013.  Each dot is a single voter and their residential location.





My data was collected from the following sources:

Voter registration information from the NC Board of Elections:  ftp://www.app.sboe.state.nc.us/
Mapping shapefiles from Wake county:  http://www.wakegov.com/gis/services/pages/data.aspx

Geocoding the addresses was done by Texas A&M's Geoservices:  http://geoservices.tamu.edu/

Sunday, November 10, 2013

A Map of Registered Republicans and Democrats in Wake County


Here's my R code.  I utilized the R rgdal package for creating the maps.
This assumes that you've already got your voter data loaded into R.

roads <- readOGR("c:\\data\\poly\\wake_streets\\streets.shp","streets")
roadmap <- spTransform(roads, CRS("+proj=longlat +datum=WGS84"))
county <- readOGR("C:\\data\\poly\\nc_counties\\NC_State_County_Boundary_NAD83HARN.shp",'NC_State_County_Boundary_NAD83HARN')
countymap <- spTransform(county, CRS("+proj=longlat +datum=WGS84"))

plot(roadmap[roadmap$CLASSNAME == 'INT',],col='black',border='black', lwd=.5,axes=F,xlim=c(-79,-78.2),ylim=c(35.5,36.1))
plot(countymap[countymap$County == 'Wake',], add=T)
plot(roadmap[roadmap$CLASSNAME == 'USHWY',],col='black',border='black', lwd=.5, add=T)
points(vtx[vtx$party == 'REP','lng'],vtx[vtx$party == 'REP','lat'],col = rgb(255,0,0,50,maxColorValue=255),cex=.2,pch=20)
points(vtx[vtx$party == 'DEM','lng'],vtx[vtx$party == 'DEM','lat'],col = rgb(0,0,255,50,maxColorValue=255),cex=.2,pch=20)
title("Registered Wake County Republican or Democrats \n (sample of 35,000) - Oct 2013")

My data was collected from the following sources:
Voter registration information from the NC Board of Elections:  ftp://www.app.sboe.state.nc.us/
Mapping shapefiles from Wake county:  http://www.wakegov.com/gis/services/pages/data.aspx

Geocoding the addresses was done by Texas A&M's Geoservices:  http://geoservices.tamu.edu/




Friday, November 1, 2013

Do Running Backs Drafted in Earlier Rounds Perform Better in the NFL?

It appears there is a decent correlation  (0.5, according to my calculations) between an NFL running back's performance and their their draft position.  I'm measuring performance using their career yard total.

This data does NOT include yardage for the 2013 season or players drafted in 2013.


Monday, October 28, 2013

Mapping Shapefiles from the State of North Carolina

I've been looking forever on how to use shapefiles originating from North Carolina for making maps.  Normally, I'll get crazy latitude and longitude coordinates if I plot the shapefiles using the default parameters through R's rgdal package.

Tonight, I stumbled upon this StackOverflow post, which shows that TWO functions are needed in order to fully utilize NC shapefiles: readOGR and spTransform.

I've always just used readOGR, which works fine for shapefile originating from other places, like the US Census bureau.  I've always had problems with files from my state, however, until I came across the aforementioned StackOverflow post.

Here's what I put into R to make a simple map of all roads in Wake County:

roads <- readOGR("c:\\data\\poly\\wake_streets\\streets.shp","streets", p4s = CRS("+proj=lcc +lat_1=34.33333333333334 +lat_2=36.16666666666666 +lat_0=33.75 +lon_0=-79 +x_0=609601.2199999997 +y_0=0 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=us-ft +no_defs"))
roadmap <- spTransform(roads, CRS("+proj=longlat +datum=WGS84"))
plot(roadmap,axes=T)
title("Roads in Wake County, North Carolina")

This'll produce a map like so:



If you want to try this at home, you can get a bunch of shapefiles from North Carolina's State Board of Elections FTP site that seem to work with this code.

Now that this mystery has finally been uncovered, my mapping options are much improved!

Wednesday, October 2, 2013

Getting Geocodes through R and Google's Web Service

Part of my new job as a Data Integration Analyst is learning how to study and manipulate data.  So far, I've really enjoyed this new challenge and I love having the opportunity to learn something new.  

I learned pretty quickly that R is a pretty popular programming language within the realm of data and analytics.  By itself, R can perform some complex data analysis. However, packages provided by other R enthusiasts can be loaded into the R interface to make it more powerful.  I've spent the last few months getting more familiar with the language and additional packages and learning to appreciate it.  Although I still have a lot to learn, I can already see that R can do a lot of really cool stuff.

One aspect of analytics that I've been particularly fascinated with involves analyzing data through geography.  R has a lot of packages that make this pretty straightforward.  The ones I've seen so far are great, but, in order to map a specific place, you need geocoordinates (latitude and longitude points).  Providing just an address to R and one of these mapping packages won't do.

I really want to map some data regarding voters in my home county, Wake county, North Carolina.  I think I figured out how to do it.

Google provides a free web service that allows you to collect geocoordinates for any address. All you have to do is provide Google with a residential address through a URL.

R has a function that allows you to collect data through the web.  It's as easy as this:

getweb <- url('http://maps.googleapis.com/maps/api/geocode/xml?address=1600+Pennsylvania+Avenue,+20500&sensor=true')
getaddress <- readLines(getweb)
close(getweb)

I've just requested the geocoordinates of the White House, placed the results in another object, then closed the connection with Google.

Google returns the data in an XML string, which is now in my 'getaddress' object.  Google can also return JSON, but R has a package that can interpret XML for you. Once you install the package, you can collect the coordinates from the XML like so:

lng <- xmlValue(getNodeSet(xmlParse(y),'//result//geometry//location//lng')[[1]])
lat <- xmlValue(getNodeSet(xmlParse(y),'//result//geometry//location//lat')[[1]])

You now have coordinates!  Using one of R's available mapping packages, you can plot it like so.  




This simple map was created using one of the easier of R's maps packages to create a map. Here's the process:

map('usa',bg='lightblue',col='tan',fill=T)
points(lng,lat,pch='*',cex=10,col='red')

This is really just a glimpse into the world of mapping through R.  There's a ton of resources out there that allow you to map all sorts of regions, locations, boundaries, and landmarks. 

The possibilities are endless.

NOTE:  Google is very generous to provide geocoordinates for free.  However, they do limit the number of daily queries for each person to 2500.  

Monday, September 23, 2013

Best Football Conference for Getting Drafted

A few weeks ago I was talking to my coworker about how there's a perception that SEC teams dominate college football.  If this were true, I speculated this may be due to better recruiting.  Winning schools typically recruit better.  Recruits want to play for winners. Teams with more exposure and evidence of success are more likely to have their players get drafted into the NFL.

However, is this really the case?  Do players in winning conferences, specifically the big, bad SEC, get drafted higher than players on "weaker" conferences?  Does the best recruiting conference perform the best on the field?

According to scout.com,  the SEC conference is, in fact, typically the best recruiting school.



As you can see from the chart above, the SEC typically gets more top 100 recruits than the other big six conferences (ACC, Big Ten, PAC 12, Big 12 and Big East).  The SEC's only been bested ONCE since 2002.

Does recruiting equate to wins?  They're pretty good each year, but it's not as clear cut as you would think.  However, the SEC is always one of the more highly ranked conferences when the dust settles at the end of the season, according to the Associated Press rankings.


You can see here the SEC teams usually fare better with top 25 votes than schools in other conferences.  However, there have a few years where other conferences have outperformed SEC schools.

Does the SEC pay dividends to the recruits who commit to their schools and help collect top 25 rankings when compared to the other conferences?  NFL teams do draft more SEC players than the other conferences in the majority of years since 2000, but it's certainly not a safe bet.


 This chart show that both the ACC and Big 10 have had more draft picks than the SEC on two occasions since 2000.  In most years, the margins between conference schools is pretty narrow.

So, Recruits fare pretty well on NFL draft day when committing to SEC schools, but it isn't necessarily always THE safest conference to bet on.  The SEC does the best job recruiting, but it doesn't ALWAYS equate to being the dominate conference every year.