Installation

You have my 100% money-back guarantee that the most difficult part of using this package is installing the requirements. Once you’ve got OpenCV installed, the rest will be smooth sailing.

1. Install OpenCV

On Mac, use homebrew to install OpenCV.

brew install opencv

Installing OpenCV with brew did work for me, but it also broke my previous Python installation and all my virtual environments. So uhh, good luck with that. I don’t have experience installing OpenCV on Windows, but you can try this tutorial.

2. Clone the project repository

After installing OpenCV, we can clone the project repository from GitHub:

git clone https://github.com/johnwmillr/facer.git Facer

The git clone command will create a new folder, ./Facer, and populate it with files from the GitHub repository. After cloning the repo, cd into the new directory.

3. Install the Python libraries

After installing OpenCV and cloning the project repository, we just need to add a few more helper libraries: numpy, matplotlib, and dlib.

pip install -r requirements.txt

4. Download the pre-trained detection model

The face landmark detection relies on a pre-trained model that must be downloaded separately from the dlib package itself.

wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2

Unzip the compressed file after it finishes downloading and move it into the ./Facer/dlib directory.

Congrats, you’ve installed everything you need to run Facer!

Creating average faces

Having installed Facer (and completed the hardest parts of this tutorial!), we can begin creating average faces.

The process is actually quite straightforward. Here’s how to load images from a directory and create an average face:

from facer import facer

# Load face images
path_to_images = "./face_images"
images = facer.load_images(path_to_images)

# Detect landmarks for each face
landmarks, faces = facer.detect_face_landmarks(images)

# Use  the detected landmarks to create an average face
average_face = facer.create_average_face(faces, landmarks, save_image=True)

# View the composite image
plt.imshow(average_face)
plt.show()

That’s it! The facer.create_average_face function will have saved a JPG image containing the average face output.

The Facer package also supports creating animated GIFs that display each face used to create an average:

from facer import facer

# Load face images
path_to_images = "./face_images"
images = facer.load_images(path_to_images)

# Create the animated GIF
gif, average_face = facer.create_animated_gif(images)

Let me know if you have any questions! I’ll be most responsive to new issues or pull requests on the GitHub repository.

You’ve probably seen average faces before—they’re not too uncommon. The concept is simple: take a collection of pictures of faces and blend them together to create a single “average face” that is representative of the whole group. The actual blending process involves some complicated math, but it essentially boils down to aligning the faces to make sure the nose, eyes, mouth, etc. all line up.

I’d been playing with average faces for the last few weeks with—initially—only amusing results. After updating code from a slightly outdated (but extremely useful) tutorial and wrapping it into my own Python package, I was eager to find an application worth writing about.

After having moderate success with averages of musicians’ faces from different genres, my brother suggested I target a source where I could count on the composition of the photos being mostly consistent: Fortune 500 company executive leadership pages. So, the immediate goal of this post is simple: for each of the top 50 Fortune 500 companies, determine the average face of the company’s executive leadership.

Methods

My source code for this project is available on GitHub.

Downloading faces

I began by scraping the names of the top 100 companies on the Fortune 500 list. I used Selenium (an automated browsing tool) to load with the Fortune website and scrape the company names. Selenium is better suited than BeautifulSoup for handling pages that use JavaScript, a feature I knew would more robustly handle individual company websites.

After collecting the top 100 companies, I automated a Google search for each company: “company name corporate leadership page.” For each searched, I saved the URL for the first hit from the same domain as the company website.

Next I scraped images from each company’s corporate leadership page. This turned out to be more challenging than I expected. I was familiar with using BeautifulSoup to scrape images from websites, but simply finding all the <img> tags wasn’t enough. For example, coming up with a specific solution for Apple would have been easy enough, but the method would be no good for me if it didn’t work for all other websites on the list as well. I ended up with an approach that monitors all network traffic while Selenium loads a website and extracts and saves any image data.

Unfortunately, the fully automated scraping wasn’t successful on about 15 of the top 50 websites. The most common issue was a corporate leadership page not including any images, only a list of names. For these websites, I ran Google image searches that included the individual and company’s names and collected the first image to include a single face.

Averaging faces

Once corporate leadership images were downloaded for each company, I used my Facer package to create the average faces. Facer is largely just a reworking of preexisting code from Satya Mallick. The majority of my work was updating the tutorial’s code for Python 3 and the latest version of OpenCV. Also, for some reason, installing OpenCV broke my Mac’s Python install and virtual environments :( So watch out!

Point Facer at a directory of images and it:

1. Detects faces in images
2. Determines face landmarks locations (eyes, ears, mouth, etc.)
3. Warps each face’s landmarks to a standard central location (i.e. make sure each face is facing forward, toward the camera)
4. Averages the overlapping faces together into a single image

Fortune 500 (Top 50)

Well—as we might have guessed—the results are as unsurprising as they are uninspiring. I was expecting fifty images of mostly white men, but I didn’t expect just how similar each of those images was going to appear. Apart from a handful of standouts (e.g. Target, Bank of America, Alphabet), I honestly thought I had made a mistake with my averaging code; how could I end up with so many nearly identical faces? Alas, the code’s not the issue. Even when looking through company pages such as General Motors where there are a few noticeable women or people of color, their appearance is quickly blended out by the overwhelmingly white male majority.

Take some time and look through the average faces for the top 50 companies. My favorite might be #25, Bank of America, a dead ringer for Mrs. Doubtfire.

Grouping by industry

The Fortune 500 list categorizes each company into an industry. After being a bit underwhelmed at the lack of any diversity when comparing each face in the top 50, I was curious whether we might see any trends when grouping the faces by industry. Would the average face of Finance appear older than the average face of Tech?

Discussion

I hope this project inspires conversation. What questions does it bring to mind for you? Are these results interesting? Surprising? Useful? Important? What other areas could we apply this sort of work to?

Let me know what you think!

Example interactive plot

After analyzing 12k+ country lyrics, I’ve gone and downloaded about 40k rap and rock lyrics using my LyricsGenius Python package. I’m still working on my analysis of those lyrics, but I think this first plot is pretty interesting.

Move your mouse over the points in the plot to see more detail about the artists they represent.

For whatever reason, measuring how often an artist uses the word like is a pretty decent way to differentiate between country, rock, and rap artists. Rap artists are much more likely than country artists to use the word like in their songs.

There is also a clear separation on gender between country artists for the word love, but the effect doesn’t appear to hold for rap artists. Unfortunately, there weren’t enough female rock artists in my dataset to notice any relationships with gender in that genre.

Creating the plot

As I mentioned above, I used the mpld3 library to make the interactive plot. The library went unmaintained for awhile, but has been recently switched to new maintainers, which will hopefully revitalize the project. Before this switch in ownership, I forked the repo to continue making small tweaks to suit my needs.

The process of creating an interactive plot with mpld3 is actually fairly straightforward. The library is well documented and provides a number of helpful tutorials and examples for getting started. I use the library’s tooltip feature with some extra CSS to make the artist info and image show up on the plot above.

Here is the relevant code excerpt from the tooltip tutorial:

fig, ax = plt.subplots(figsize=(10,10))

labels = []
for i in range(N):
    label = df.ix[[i], :].T
    label.columns = ['Row {0}'.format(i)]
    labels.append(str(label.to_html()))

points = ax.plot(df.x, df.y, 'o', color='b',
                 mec='k', ms=15, mew=1, alpha=.6)

tooltip = plugins.PointHTMLTooltip(points[0], labels,
                                   voffset=10, hoffset=10, css=css)
plugins.connect(fig, tooltip)
mpld3.show()

Using the mpld3.show() function will display the graph within a Jupyter notebook. However, we’re interested in exporting the plot for use in a Jekyll blog post, so we’ll use the mpld3.save_html() function instead of the show() command:

filename = 'example_plot.html'
mpld3.save_html(fig, filename, template_type='simple')

For some reason, the plot wouldn’t render in my blog post unless I set the template_type variable to 'simple'.

Adding the plot to a post

After exporting the plot as an HTML file, add it to the _includes/ directory within your Jekyll site. After putting the file in place, we just need to include the HTML within the Markdown file for a blog post:

Here is some example text for an exciting blog post. Deep blockchain learning is the future, blah, blah, etc.

{% include example_plot.html %}

And then you keep writing your exciting blog post.

That’s it! The interactive plot should show up once the Jekyll site is built and renders the page.

Up next

I’d love to make the plots even more interactive. It would be very cool if users were able to specify the actual lyrics data that gets displayed in the plot. If anybody knows a way to do this with a Jekyll blog, let met know!

The odd thing about this dataset is that roughly 60% of its 200k+ recipes are identical. For whatever reason, there are more than 120,000 copies of a Johnsonville Three Cheese Italian Style Chicken Sausage Skillet Pizza recipe.

Why there are so many copies of this recipe – or, for that matter, why the dish exists in the first place – is beyond me. I was more concerned with how all the duplicate recipes would affect my analysis.

The task at hand

As part of my food recipes project, I’m interested in finding different versions of recipes for the same type of food (e.g. chocolate chip cookies) and comparing how the combinations of ingredients vary between recipes. To properly compare different recipes I need to know how many servings each is supposed to make. For example, two different cookie recipes may both require a cup of sugar, but you’ll end up with completely different cookies if the first recipe is for a batch of ten while the second makes three dozen.

Unfortunately, the recipes in the pre-cleaned JSON dataset did not contain serving size information. The raw HTML files did contain serving sizes, but I would have to iterate through each of the 200k+ files to extract the information. I knew my work would be considerably faster if I could throw out the duplicate Johnsonville recipes and reduce the size of my dataset by 60%.

Python approach

My first instinct was to write a simple Python script to iterate through the HTML files and delete any that contained the Johnsonville recipe. The code itself was easy enough to write. I got a Jupyter notebook going that looped through the raw HTML files, deleted any Johnsonville duplicates, and estimated the script’s remaining time.

# Loop through the HTML recipe files, deleting the Johnsonville duplicates
import os, time

def calcTimeRemaining(elapsed, n, total):
    return (total-n)/(n/elapsed)

htmlFiles = os.listdir('./Data/Recipes/O2/recipes_html/')
numFiles = len(htmlFiles)
start = time.time()
last = time.time() - start
for n, filename in enumerate(htmlFiles):
    with open(filename, 'r') as infile:
        html = infile.read()
    if 'italian style chicken sausage skillet pizza' in html:
        os.remove(filename)
    if (time.time() - last)/60 > 0.05:  # Print a progress update
        last = time.time()
        elapsed = (time.time() - start)/60
        print("File ({n}/{t}) | Remaining: {r:.1f} min | Files/min: {fpm:.1f}".format(
            n=n+1, t=numFiles, fpm=(n+1)/elapsed,
            r=calcTimeRemaining(elapsed, n+1, numFiles)))

After letting the Jupyter notebook run for a few minutes, I was shocked to see an estimated 12 hours remaining.

File (1531/225624) | Remaining: 12.1 hours | Files/min: 303.1

There had to be a better way!

Unix to the rescue

My moment of clarity came while listening to DataCamp’s DataFramed Podcast in the shower. Hugo Bowne-Anderson, the show’s enthusiastic host and interviewer, was doing a Language Corner interlude with Spencer Boucher, a Data Science Curriculum Lead at Data Camp, about the often overlooked utility of Unix in data science. Hugo and Spencer stressed a few key advantages command-line tools have over full-featured programming languages when it comes to data science work:

• Most Unix tools have rich APIs, optimized for the “80% use case” (i.e. optimal commands for one-off tasks)
• Unix is available in nearly all scientific computing environments, even when full programming languages aren’t
• Unix tools can be easily chained together to create powerful, sophisticated commands
• Using Unix can help avoid loading entire datasets into memory for processing
• Many Unix commands are easily parallelizable

During Spencer’s call to action to use Unix tools for data science (especially while cleaning data) it hit me – I could use Unix to delete the Johnsonville recipes!

A quick Google search led me to a Stack Overflow post that recommended piping the output of a grep command into awk, creating an executable .sh file that would then delete the duplicate recipes. After visiting a few more Stack Overflow forums and testing the commands in a directory with fewer files, I ended up with a string of commands that did the trick:

$grep -rl . -e 'Italian Style Chicken Sausage Skillet Pizza' | awk '{print "rm -v "$1}' > deleteRecipes.sh
$source deleteRecipes.sh

I’ve added an appendix to this post that provides a full description of how each of the Unix tools are working together in the command above.

Summary

The whole process of identifying and deleting the culprit recipes took about 1.5 hours with the Unix code above, nearly 90% faster than the estimated time for my Python code! There are bound to be better solutions to this problem than the ones I’ve described here. I’m sure there’s a proper way to solve this sort of problem in Python, and I’d love to hear in the comments below what approach that may be.

But for me, this was a great opportunity to try out a new technique and expand my set of data science skills ever so slightly. The next time I’m tackling a 40 GB list of unruly HTML files, I’ll be less likely to chuck my keyboard across the room in frustration and more quick to pull this powerful grep awk combo from my data science tool belt.

A big thanks to Hugo and Spencer from the DataFramed Podcast. If you aren’t already listening to DataFramed, I strongly recommend checking it out. Each episode covers a different area of data science, and Hugo is an engaging, articulate interviewer.

Update: Guess I should have done a bit more Googling and just used fdupes. 😑

Appendix

Full description of the Unix tools ⇪

The Unix code I ended up with makes use of a couple of really helpful command-line tools:

grep

The grep tool uses regular expressions to parse and print text within the command line. In the command above I’m telling grep to search the current directory for any files containing the “Skillet Pizza” text and to then print their file names.

The -rl option tells grep to search recursively and print the matched file names, rather than the actual matched text. The e option is used to specify the regular expression to search with. The . tells grep to search within the current directory.

pipe and redirect

The | character in-between the grep and awk commands is a pipe and is used to send the output from one command into another. The > character redirects the output of a command to be stored as text in a file (and vice versa). In my example above, the output from grep (a list of filenames) is piped into the awk command whose output (a list of rm commands) is then redirected into the deleteRecipes.sh file. Be careful with > as it will overwrite any existing files with the same name. The >> operator is a safer alternative that appends to rather than overwrites existing files.

awk

The awk command is actually calling an entire programming language (named AWK) that is helpful for filtering text, especially data stored in a columnar format (e.g. CSV). Serendipitously, my friend Micah sent me a helpful article while I was working on this post that makes a case for learning AWK and provides a few useful examples.

I’m using awk to generate a line of text – rm -v FILENAME.html – for each Johnsonville recipe identified by grep. The $1 is a variable in awk that will take on the file names piped in from grep. The rm statement deletes the specified file, and -v makes the command verbose.

“argument list too long”

Because the directory I was working in contained so many files, I kept running into an “argument list too long” error when I would try the command I found on the original Stack Overflow post. Unix has a limited buffer for the length of any given command, and so it was unable to string together the roughly 120,000 names of files containing the Johnsonville recipe. In the end, outputting the rm commands to a file to be executed later (rather than maintaining a list of all the rm within the command line buffer) allowed me to get around the error.

Finally, I call use the source command to run the rm commands and delete the offending Johnsonville recipes.

Finished my #DataVisualization post about #SharkTank. Looked at each Shark's funding history, broken down by gender. @BarbaraCorcoran and @robertherjavec are the most biased! @mcuban and @TheSharkDaymond are pretty fair. #DataSciencehttps://t.co/G7LbXDmpCy

— John W. Miller (@johnwmillr) August 6, 2018

Introduction

My brother and I are big fans of Shark Tank. After watching a few episodes together on vacation recently, I started to wonder whether it’d be possible to identify any patterns in the way Sharks make deals on the show. For instance—are male Sharks more likely to make deals with men than women? Or, for that matter, are male entrepreneurs more likely to make it on the show in the first place? With a few of these questions in mind, I started looking for the data I’d need to come up with answers.

Collecting the data

As I suspected, I wasn’t the first to ask these sorts of questions about Shark Tank. In a post on Quora, Halle Tecco described her own detailed analysis of the show. Amongst other insights, Halle found that across the first six seasons of the show, the two female Sharks (Barbara Corcoran and Lori Greiner) made a larger percentage of their deals with women than their male counterparts.

Halle made the data she collected available online. Due to a lack of information in the dataset on entrepreneurs with non-binary gender identity, I only used male and female labels in my analysis. The code I wrote for this analysis is available on GitHub.

Posing the question

It’s worth noting that the number of men and women appearing on the show is not equal: the ratio of all-male teams to all-female teams is roughly 70/30.

I was curious to see how each Shark’s funding probabilities might look after accounting for this skewed ratio. As I thought through different ways to measure this effect, I kept asking myself the same question — what percentage of pitches from either gender did each Shark make a deal with?

I needed a bit more data to be able to answer my question. Luckily, there is a Wikipedia page that provides short descriptions of each Shark Tank episode. I used Beautiful Soup to scrape the attendance records for each of the main Sharks on the show. By combining Halle’s deal data with the attendance records I collected, I was able to determine how often each Shark made deals with the male and female entrepreneurs they heard pitches from.

Each bar represents the percent of entrepreneurs of a given gender that the Shark made deals with. For example, Mark Cuban made deals with 20.2% of the women and 17.1% of the men he heard pitches from.

Discussion

The main conclusion to draw from the graph is that certain Sharks do not make deals with men and women at equal rates. The next step is to ask what might be contributing to this imbalance. The wrong step is to jump to the conclusion that the Sharks are acting on sexist motivations.

Clearly, there are many factors at play here. For one, Barbara Corcoran and Robert Herjavec — the Sharks with the largest funding biases — tend to invest in completely different industries (food/beverage and tech, respectively). Additionally, the data I collected do not contain information on which Sharks made offers that were rejected, meaning my graph does not account for potential biases on behalf of the entrepreneurs.

Season 10 of Shark Tank premiers this October. ABC’s promo for the show claims that “[The Sharks] will once again give people from all walks of life the chance to chase the American dream”. I’m looking forward to the premiere as much as anyone else. But with this analysis in mind, I’ll think a bit more about who gets to walk onto the show — and leave with a deal.

Collecting the data

I’ve been trying to get more practice interfacing with APIs, so I was hoping to find a soccer API rather than simply a precompiled dataset. After doing some Googling for World Cup APIs, I came across the Sportradar API. Sportradar provides 15 APIs for soccer data alone. Luckily, they provide a free trial for the APIs that allows a total of 1,000 queries.

I wrote a Python package that makes it easy to interface with the Sportradar APIs. The code is available on GitHub. The easiest way to get started is to install the package using pip:

$pip install sportradar

After pip finishes the installation you can use the package to download World Cup data.

from sportradar import Soccer

# Create an instance of the Sportradar Soccer API class
sr = Soccer.Soccer("paste your api key here")

# Get a list of all tournaments
tournaments = sr.get_tournaments().json()

# Get info on the 2018 World Cup (Teams, Rounds, etc.)
worldcup = sr.get_tournament_info(tournaments['tournaments'][4]['id']).json()

# Get more information on each team in the World Cup
teams = []
team_counter = 0
for group in worldcup['groups']:
    for team in group['teams']:
        team_counter += 1
        team_id = team['id']
        team_name = team['name']
        print("({}): {}, {}".format(team_counter, team_name, team_id))
        try:
            teams.append(sportsradar.get_team_profile(team_id).json())
        except Exception as e:
            print("Error: {}".format(e))
        time.sleep(5) # wait 5 seconds before next API call

# Save the team data to a .json file
print("Saving the data...", end="", flush=True)
with open("world_cup_team_data.json", "w") as outfile:
    json.dump(teams, outfile)
print(" Done.")

And now we have the data we need! Definitely take a look at Sportradar’s API documentation to get a sense for what all is available (e.g. NBA, NFL, other soccer tournaments, etc.).

Visualizations

Below are some of the plots I’ve made using the World Cup data. For each of the box-whisker plots, the middle line indicates the median of the team’s height distribution. The edges of the boxes extend to the first and third quartiles of the data. The whiskers extend to show the range of the data, excluding any extreme outliers. You can find the code I wrote to generate these plots on GitHub.

Distributions of player height sorted by country

Distributions of player height sorted by position

This is an interactive plot. Move your mouse over the scatter plot points to see details on each player.

This plot did quite well on /r/dataisbeautiful, although people were pretty upset I didn’t use metric units.

Player heights and goals scored

Height vs. weight by player position

The plot below compares the relationship between the lexical diversity of an album’s lyrical content and the review score the album received on Pitchfork. Move your mouse over the points in the plot to see more detail about the albums they represent.

I was surprised to find a small negative correlation between the album review score and the lexical diversity of the album’s corresponding review. The Pitchfork reviews for higher-rated albums are less lexically diverse.

Here’s a prettier, static version of the same plot:

Below is an excerpt from one of the chapters from my thesis covering the theory behind classical decision trees and random forests. I hope my descriptions of the concepts here can be of some use to those hoping to better understand tree-based classification.

Decision trees and random forests

This chapter first describes the theory behind random forest classifiers, then details the specific implementation of a classifier trained on retinal layer shapes derived from volumetric SD-OCT images. Finally, the results of the classifier are presented along with a brief discussion of the work.

Random forest classifiers are an ensemble learning method built from decision trees. Decision trees often follow the Classification and Regression Tree (CART) framework, and, as the name implies, can be used for either classification or regression tasks [1]. This thesis focuses solely on binary classification trees and will use the general term decision trees to refer specifically to binary classification trees. Random forests, along with other improvements which will be detailed in below, combine the outputs of many decision tree models in order to produce an output that is both more accurate and less sensitive to noise than traditional decision trees.

Classical decision trees

Decision trees operate with a simple objective of partitioning a dataset into two categories using a sequence of binary “Yes” or “No” splits. Decision trees achieve this objective by sequentially identifying features on which to split the data into subsets that are more homogeneous than the current subset. For example, when diagnosing a fever, a decision tree may classify a patient as having or not having a fever by answering a series of Yes or No questions, such as “is the temperature greater than 100F?”, “are the palms sweaty?”, etc, and arriving at a classification of having or not having a fever. The process of training a decision tree determines which Yes or No questions to ask (i.e. which features to split on) and at what values to set each Yes/No threshold.

There are a few parameters to select while training the binary tree: which features do we split on? How do we determine the threshold for splitting? When do we stop splitting the data (i.e. stop adding nodes to the tree)? The first two questions can be addressed through the use of an impurity metric.

Impurity can be thought of as a measure of the heterogeneity of dataset; a set of data with an equal distribution of two classes would be highly impure. The Gini impurity index, \(I(t)\), is a common measure of impurity in random forests [2-4]: \begin{align} I(t) = \sum_{m=1}^{M} P(\omega_m |t)(1-P(\omega_m|t)) \end{align} where \(P(\omega_m |t)\) is the probability of finding a point, \(\omega_m\), from class \(m\) in a subset \(X_t\) and \(M\) is the total number of classes. \(X_t\) is the current set of points at node \(t\), before the data are split into two purer subsets. The probability is approximated as \begin{align} P(\omega_m |t) = \frac{N_t^m}{N_t}, m = 1,2,\ldots,M \end{align} where \(N_t^m\) is the number of points from class \(m\) in a potential new subset \(X_{tY}\) and \(N_t\) is the total number of points in \(X_t\).

The Gini impurity is maximal when \(P(\omega_m \vert t)\) is uniform. Decision trees identify the optimal feature and threshold at each split by iterating through each of the features, testing different threshold values, and picking the combination of feature and threshold that results in the biggest decrease in \(I(t)\). By minimizing the Gini impurity, decision trees identify subsets of the data that will be maximally pure.

The third question, determining when to stop splitting the data, is typically addressed through a heuristic approach known as pruning [3]. It is difficult to stop training a tree at precisely the right point; stopping early would result in poor classification results and stopping too late may result in overfitting. One common practice is to allow the tree to grow to a large size – potentially overfitting to the data – and then removing nodes to balance prediction error and model complexity [3].

Decision trees provide a number of advantages over other standard classification techniques. If a tree remains small, its results are easily interpretable. Decision trees can be adapted for regression or classification. And in general, decision trees are computationally simple and quick to fit.

However, there are also a number of substantial disadvantages associated with decision trees. Compared to support vector machines and other ensemble techniques, decision trees have low accuracy and are highly sensitive to small changes in their training data [3]. Additionally, decision trees are only able to split data parallel to the feature axes and are forced to approximate diagonal lines as many horizontal and vertical splits [2]. Random forests improve substantially on both of the issues of accuracy and instability while making justifiable concessions in interpretability of the model.

Improvements of random forests over classical decision trees

Random forests were introduced in 2001 by Breiman to address a number of shortcomings associated with traditional decision trees, such as low accuracy and high sensitivity to variations in training data [5]. Random forests use a bootstrapped ensemble of decision trees along with a random sampling of features during training to achieve good accuracy while remaining robust to noise [5]. Random forests are now widely used in a variety of disciplines and generally applicable to a number of problems [6,7].

Bagging (bootstrap aggregating) refers to the practice of training multiple trees on random subsets (with replacement) of the training data with the intent of reducing variance in the final model [8]. The final output of the ensemble of classifiers is then typically determined through an averaging or winner-takes-all vote of the output of each tree [3].

Traditional bagging trains trees on distributions that are identically distributed but not necessarily independent. The variance of \(N\) identically distributed random variables with variance \(\sigma\) is \(\rho\sigma^2 + \frac{1-\rho}{N}\sigma^2\), whereas the variance of an average of \(N\) independent identically distributed random variables is \(\frac{1}{N}\sigma^2\) [6]. Random forests attempt to achieve the smaller predictor variance by producing trees that are independent as well as identically distributed. This is achieved by randomly selecting a subset of features at each split of the data (i.e. tree node).

As the trees comprising a random forest are trained using bagging, which selects a random subset of data for each tree, it is possible to identify samples that were not included in the training of a given tree. These samples are referred to as out-of-bag (OOB) and can be treated like an internal test set to evaluate the model [6]. The OOB error estimate is the mean prediction error of a sample \(x_i\) calculated from the trees which did not include \(x_i\) in their subset of training data [5,6].

Random forests (and decision trees in general) lend themselves to straightforward calculation of feature importance. A common approach for calculating feature importance in random forests is to simply rank features based on their total reduction in the Gini impurity at each split node accumulated across each of the trees used to build the forest [6]. It is common practice to normalize the importance of all features to range from 0 to 1 or 0 to 100 [4].

References

1. L. Breiman, Classification and Regression Trees, Wadsworth statistics/probability series. Wadsworth International Group, 1984.
2. C. M. Bishop, Pattern Recognition and Machine Learning. Secaucus, NJ, USA: Springer-Verlag New York, Inc., 2006.
3. S. Theodoridis, Machine Learning: A Bayesian and Optimization Perspective, .NET Developers Series. Elsevier Science, 2015.
4. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: machine learning in python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
5. L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001.
6. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 12th printing, Springer Series in Statistics. New York, NY, USA: Springer New York Inc., 2017.
7. J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake, “Real-time human pose recognition in parts from single depth images,” CVPR 2011, pp. 1297–1304, 2011.
8. L. Breiman, “Bagging predictors,” Machine Learning, vol. 24, no. 2, pp. 123–140, 1996.

This post walks through how to export raw data from the iOS Health app and then demonstrates a simple analysis of the step count data in Python. You can find the code for this post in my GitHub repo.

Exporting the health data

Apple makes it pretty easy to export raw data from the Health app. You’ll find the export button at the bottom of the user profile.

Choose a sharing option that’ll be easiest for you to send the data to your computer. I shared the data into the Notes app which then synced with my laptop over iCloud. The data exports as a ZIP file, export.zip. The ZIP file contains two XML files: export_cda.xml and export.xml.

The export.xml file was the larger of the two (11 MB), so I figured that was the one which contained the health data.

Reading the data in Python

It’s actually a pretty straightforward process to read the raw XML data in Python.

First we load the XML file using Python’s XML library.

# Reads the XML file exported by Apple's Health iOS app
import xml.etree.ElementTree
xDoc = xml.etree.ElementTree.parse('./data/apple_health_export/export.xml')
items = list(xDoc.getroot()) # Convert the XML items to a list

With the XML items stored as a list, we can simply loop through and extract the data we’re interested in. The Health app exports more than just step counts, so we need to specify which type of data we want from the XML file. For an analysis of step counts we’re interested in the "HKQuantityTypeIdentifierStepCount" items, but we could also select "HKQuantityTypeIdentifierBodyMassIndex" or "HKCategoryTypeIdentifierSleepAnalysis" amongst others.

# Loop through the XML items, appending samples with the requested identifier
tmp_data = []
item_type_identifier='HKQuantityTypeIdentifierStepCount' # Desired data type
for i,item in enumerate(items):
    if 'type' in item.attrib and item.attrib['type'] == item_type_identifier:
        # Attributes to extract from the current item
        tmp_data.append((item.attrib['creationDate'],
                         item.attrib['startDate'],
                         item.attrib['endDate'],
                         item.attrib['value']))

# Convert to data frame and numpy arrays
import numpy as np
import pandas as pd
data = pd.DataFrame(tmp_data, columns = ['creationDate','startDate','endDate','value'])
all_step_counts = np.array(data.values[:,-1], dtype=int)
all_step_dates  = np.array(data.values[:,0])

We now have the data we’re interested in: step counts and their corresponding sample times. In my case, with about a year’s worth of data, I had 12,456 samples, or roughly 34 samples per day. I assume the phone isn’t sampling at a constant rate – I’d be curious to know what determines how often it generates a sample in this data file.

Displaying the data

As a sanity check, we can start by recreating one of the graph’s generated by the iPhone.

What happens if we just plot all the raw data from the xml file?

Hmm… that doesn’t look like what we want. What’s going on? Well, obviously, the plot from the iPhone is a bar graph. But even if we plot the raw data as a bar graph instead of a scatter we won’t end up with what we want. The important difference between the two plots is that the iPhone is doing a monthly averaging of the raw data. Let’s do the same in our Python code.

Calculating steps per month

We have a different number of samples for each month so we can’t just divide each month’s total step counts by a fixed number. Luckily, the iPhone records the date and time of each sample. We can use this information to group each sample by its corresponding month. The monthrange function from Python’s calendar library tells us the number of days in each month, which we’ll use to divide steps by days to arrive at an daily average for each month.

from datetime import datetime
from calendar import monthrange

steps_per_month, month_labels = [],[]
current_month = datetime.strptime(all_step_dates[0][:7], '%Y-%m')
running_step_count = 0
for n, date, step_count in zip(range(len(all_step_dates)), all_step_dates, all_step_counts):
    new_month = datetime.strptime(date[:7], '%Y-%m')

    if new_month > current_month or n == len(all_step_dates)-1:
        # How many days are in the current month?
        if date==all_step_dates[-1]:
            days_in_month = int(date[8:10])
        else:
            days_in_month = monthrange(current_month.year, current_month.month)[1]

        # Average step count for current month
        steps_per_month.append(running_step_count/days_in_month)
        month_labels.append(current_month.strftime('%b-%Y'))

        # Reset the running step count and current month
        current_month = new_month
        running_step_count = step_count
    else:
        running_step_count += step_count

# Convert to numpy arrays
steps_per_month = np.array(steps_per_month)
month_labels = np.array(month_labels)

So what’s it look like if we plot the monthly averages we’ve calculated?

That’s more like it! If you compare this plot to the one generated by the iPhone above, you’ll see that they are essentially identical. Now that we’ve finished this sanity check and know that we’re properly handling the data, we can move on to more interesting analyses.

As I mentioned at the top of this post, I wanted to export and analyze my step count data because I had started to notice a relationship between daily activity level and the weather in Iowa City, where I live. I scraped the average monthly temperatures for Iowa City over the last year and plotted them on top of my average daily step counts for each month.

As somebody pointed out in my post on /r/dataisbeautiful, I either take fewer steps when it’s cold out, or my daily activity levels are having a serious affect on local weather patterns. The code for the above plot is contained within the same GitHub repository I linked to above.

Summary

By this point you should be able to export raw data from Apple’s Health app on iOS and produce a plot of the step counts in Python. Let me know in the comments section below if you have any questions or comments!

More specifically, I scraped Ranker.com for a list of the top female and male country artists of the last 100 years and used my python wrapper for the Genius API to download the lyrics to each song by every artist on the list. After my script ran for about six hours I was left with the lyrics to 12,446 songs by 83 artists stored in a 105 MB JSON file. As a bit of an outsider to the world of country music, I was curious whether some of the preconceived notions I had about the genre were true.

Some pertinent questions:

• Which artist mentions trucks in their songs most often?
• Does an artist’s affinity for trucks predict any other features? Their gender, for example? Or their favorite drink?
• How has the genre’s vocabulary changed over time?
• Of all the artists, whose language is most diverse? Whose is most repetitive?

You can find my code for this project on GitHub.

Analysis

If you like beer, you may also like…

I’m interested in whether an artist’s tendency to use certain terms correlate together. For example, if an artist is more likely to mention beer in their songs, are they more likely to also mention trucks?

It turns out that yes, they are.

Each point on the two plots below represents a single artist. The values for each point were calculated as the percentage of times the given artist mentions a particular term across all of their songs. For example, I had 46 songs by Cole Swindell in my database, and he mentioned beer in 24 of them, arriving at a percentage of 52%.

Think about that. Cole Swindell mentions beer in more than half of his songs. You can also count on him referencing trucks once every five songs. Dustin Lynch turned out to be the artist that sang about trucks most often, with the word truck appearing in 23.8% of his songs.

I’ve also added the artists’ genders to the plot. I’ll need to do some more analysis, but there does appear to be a relationship between an artist’s gender and their tendency to sing about trucks and beer. It’s hard to tell from this plot, but roughly half of the female artists in my dataset are actually stacked on top of each other at the origin, meaning they didn’t mention beer or trucks in any of their songs.

Check out the relationship between an artist’s use of the words girl and love. There’s an even more obvious trend with gender here. The more often a male country singer uses the word girl in his songs, the less likely he is to mention love. Interesting.

Love is falling out of fashion

I also wanted to look at how vocabulary changes over time for all country artists. We know from the plot above that male country artists are less likely to sing about love and more likely to sing about girls. The above plots combined songs from all years – I wonder if we’ll see different effects after separating songs into the years they were published.

The plot below displays the percentage of songs mentioning a given term for each year, excluding years that had less than ten songs in my database. It looks like it’s becoming less common for country artists to sing about love. The correlation isn’t all that strong, but there is a noticeable downward trend.

I also found it funny that as love gets mentioned less frequently, it’s becoming more common for country songs to include the word girl. And, if we remember from the plot of girl vs. love above, we know it’s primarily men who are driving the rise in popularity of this term. The popularity of the word boy hasn’t changed much over time, appearing in about 12% of songs each year.

Country music is getting more repetitive

After reading Kaylin Walker’s excellent post “50 Years of Pop Music”, I decided to look at how vocabulary sizes have changed over time. Similar to what Kaylin found when looking at Bilboard hits from the last 50 years, it appears that, while the correlation is weak, the average total word count for country songs has increased with time. The average unique word count has also increased with time but only slightly.

In other words, country lyrics have become more repetitive over the years.

To dig a little deeper, I took a look at the lexical diversity of the lyrics over time. Lexical diversity is a fancy term for a simple concept: what percentage of the words in a body of text are unique? A body of text where each word is only used once would have the highest possible lexical diversity with a value of 1.

Sure enough, if we look from year 2002 to the present (where the effect is most pronounced), there is a clear downward trend in lexical diversity. This trend holds for pop music in general, as illustrated by this excellent post from The Pudding on the repetitive nature of pop lyrics. Might this be an indication that country music has gotten more poppy over time?

And the most repetitive artist is…

At the start of this post I asked which artists have the most and least diverse lyrics. Lexical diversity can give us some insight into this question. For each artist, I calculated the average lexical diversity across each of their songs. The figure below shows the distribution of different lexical diversity values. The average country artist has a lexical diversity of 0.50. There would be a number of possible ways to arrive at 0.50, but one intuition for the value is that about half the words in a song are unique.

So, to answer the question, “whose lyrics are most diverse?”, we can just look at the extremes of the distribution. Kane Brown comes in at the bottom with a lexical diversity score of 0.39, and Kellie Pickler is sitting on top with 0.59. To try and get a sense for what the two artists sing about, I generated word clouds from their respective sets of lyrics.

It’s difficult to draw much from the word clouds by themselves, but it’s a fun way to get a quick impression of which words an artist uses most often. Kane Brown’s favorite words are girl, back, know, and yeah. Kellie Pickler’s are love, know, want, and go.

Days of the week

This one is pretty straightforward. Looks like we’re all just livin’ for the weekend.

Making predictions

So, we’ve collected a bunch of country lyrics, and we’ve started to notice some interesting trends in the data. Can the trends we’ve identified provide any further insight into the genre? Might we be able to use the trends we’ve identified in the lyrics to make inferences about the artists? Could we, for example, given a set of song lyrics guess the gender of the artist who wrote them?

To make these sorts of predictions and inferences from the lyrics, we’ll need to identify useful features and train a classifier, both of which I’ll go into detail on in my next post. Stay tuned!

To my complete surprise, this post was picked as a winner in the 2018 Pudding Cup!

If you aren’t already familiar with The Pudding’s incredible data storytelling work, go check them out.