The blog of Cecilia Lee

Introduction to R Programming

2017-12-05T00:00:00+00:00

For someone like me, who has only had some programming experience in Python, the syntax of R feels alienating initially. However, I believe it’s just a matter of time before adapting to the unique logicality of a new language. And indeed, the grammar of R flows more naturally to me after having to practice for a while, and I began to grasp its kind of remarkable beauty, that has captivated the heart of countless statisticians throughout the years.

If you don’t know what R is, it’s basically a programming language created for statistician by statistician. Hence, it easily becomes one of the most fluid and powerful tools in the field of Data Science.

Here I’d like to walk through my study notes with the most explicit step-by-step directions to introduce you to the world of R.

Why Learn R for Data Science?

Before diving in, you might want to know why should you learn R for Data Science. There are two major reasons:

Powerful Analytic Packages for Data Science

Firstly, R has an extremely vast package ecosystem. It provides robust tools to master all the core skill sets of Data Science, from data manipulation, data visualization, to machine learning. The vivid community keeps the R language’s functionalities growing and improving.

High Industry Popularity and Demand

With its great analytical power, R is becoming the lingua franca for data science. It is widely used in the industry and is in heavy use at several of the best companies who are hiring Data Scientists including Google and Facebook. It is one of the highly sought after skills for a Data Science job.

Quickstart Installation Guide

To start programming with R on your computer, you need two things: R and RStudio.

Install R Language

You have to first install the R language itself to your computer (It doesn’t come by default). To download R, go to CRAN, https://cloud.r-project.org/ (the comprehensive R archive network). Choose your system and select the latest version to install.

Install RStudio

You also need a hefty tool to write and compile R codes. And RStudio is the most robust and popular IDE (integrated development environment) for R programming. It is available on http://www.rstudio.com/download. (open source and for free!)

Overview of RStudio

Now you have everything ready. Let’s have a brief overview at RStudio. Fire up RStudio, the interface looks as such:

Go to File > New File > R Script to open a new script file. You’ll see a new section appear at the top left side of your interface. A typical RStudio workspace composes of the 4 panels you’re seeing right now:

RStudio Interface

Here’s a brief explanation of the use of the 4 panels in the RStudio interface:

Script
This is where your main R script located.

Console
This area shows the output of code you run from script. You can also directly write codes in the console.

Environment
This space displays the set of external elements added, including dataset, variables, vectors, functions etc.

Output
This space displays the graphs created during exploratory data analysis. You can also seek help with embedded R’s documentation here.

Running R Codes

After knowing your IDE, the first thing you want to do is to write some codes.

Using the Console Panel

You can use the console panel directly to write your codes. Hit Enter, the output of your codes will be returned and displayed immediately after. However, codes entered in the console cannot be traced later. (i.e. you can’t save your codes) This is where script comes to use. But console is good for quick experiment before formatting your codes in script.

Using the Script Panel

To write proper R codes, you start with a new script by going to File > New File > R Script, or hit Shift + Ctrl + N. You can then write your codes in the script panel. Select the line(s) to run and press Ctrl + Enter. The output will be shown in the console section beneath. You can also click on little Run button located at the top right corner of this panel. Codes written in script can be saved for later review (File > Save or Ctrl + S).

Basics of R Programming

Finally, with all the set-ups, you can write your first piece of R script. The following paragraphs introduce you to the basics of R programming.

A quick tip before going: all lines after the symbol # will be treated as a comment and will not be rendered in the output.

Arithmetics

Let’s start with some basic arithmetics. You can do some simple calculations with the arithmetic operators:

Operator	Function
`+`	Addition
`-`	Subtraction
`*`	Multiplication
`/`	Division
`^`	Exponentiation
`%%`	Modulo
`%/%`	Integer Division

Addition +, subtraction -, multiplication *, division / should be intuitive.

# Addition
1 + 1
#[1] 2

# Subtraction
2 - 2
#[1] 0

# Multiplication
3 * 2
#[1] 6

# Division
4 / 2
#[1] 2

The exponentiation operator ^ raises the number to its left to the power of the number to its right: for example 3 ^ 2 is 9.

# Exponentiation
2 ^ 4
#[1] 16

The modulo operator %% returns the remainder of the division of the number to the left by the number on its right, for example 5 modulo 3 or 5 %% 3 is 2.

# Modulo
5 %% 2
#[1] 1

Lastly, the integer division operator %/% returns the maximum times the number on the left can be divided by the number on its right, the fractional part is discarded, for example, 9 %/% 4 is 2.

# Integer division
5 %/% 2
#[1] 2

You can also add brackets () to change the order of operation. Order of operations is the same as in mathematics (from highest to lowest precedence):

Brackets
Exponentiation
Division
Multiplication
Addition
Subtraction

# Brackets
(3 + 5) * 2
#[1] 16

Variable Assignment

A basic concept in (statistical) programming is called a variable.

A variable allows you to store a value (e.g. 4) or an object (e.g. a function description) in R. You can then later use this variable’s name to easily access the value or the object that is stored within this variable.

Create New Variables

Create a new object with the assignment operator <-. All R statements where you create objects and assignment statements have the same form: object_name <- value.

num_var <- 10

chr_var <- "Ten"

To access the value of the variable, simply type the name of the variable in the console.

num_var
#[1] 10

chr_var
#[1] "Ten"

You can access the value of the variable anywhere you call it in the R script, and perform further operations on them.

first_var <- 1
second_var <- 2

first_var + second_var
#[1] 3

sum_var <- first_var + second_var
sum_var
#[1] 3

Naming Variables

Not all kinds of names are accepted in R. Variable names must start with a letter, and can only contain letters, numbers, . and _. Also, bear in mind that R is case-sensitive, i.e. Cat would not be identical to cat.

Your object names should be descriptive, so you’ll need a convention for multiple words. It is recommended to snake_case where you separate lowercase words with _.

i_use_snake_case
otherPeopleUseCamelCase
some.people.use.periods
And_aFew.People_RENOUNCEconvention

Assignment Operators

If you’ve been programming in other languages before, you’ll notice that the assignment operator in R is quite strange as it uses <- instead of the commonly used equal sign = to assign objects.

Indeed, using = will still work in R, but it will cause confusion later. So you should always follow the convention and use <- for assignment.

<- is a pain to type as you’ll have to make lots of assignments. To make life easier, you should remember RStudio’s awesome keyboard shortcut Alt + - (the minus sign) and incorporate it in your regular workflow.

Environments

Look at the environment panel at the upper right corner, you’ll find all of the objects that you’ve created.

Basic Data Types

You’ll work with numerous data types in R. Here are some of the most basic ones:

Data Type	Explanation
Numerics	Decimals values like `4.5` are called numerics.
Integers	Natural numbers like `4` are called integers. Integers are also numerics.
Logical	Boolean values (`TRUE` or `FALSE`) are called logical.
Characters	Text (or string) values like `'Dog'` are called characters.

Knowing the data type of an object is important, as different data types work with different functions, and you perform different operations on them. For example, adding a numeric and a character together will throw an error.

To check an object’s data type, you can use the class() function.

# usage: class(x)
# description: Prints the vector of names of classes an object inherits from.
# arguments: x : An R object.

Here is an example:

int_var <- 10
class(int_var)
#[1] "numeric"

dbl_var <- 10.11
class(dbl_var)
#[1] "numeric"

lgl_var <- TRUE
class(lgl_var)
#[1] "logical"

chr_var <- "Hello"
class(chr_var)
#[1] "character"

Functions

Functions are the fundamental building blocks of R. In programming, a named section of a program that performs a specific task is a function. In this sense, a function is a type of procedure or routine.

R comes with a prewritten set of functions that are kept in a library. (class() as demonstrated in the previous section is a built-in function.) You can use additional functions in other libraries by installing packages. You can also write your own functions to perform specialized tasks.

Here is the typical form of an R function:

function_name(arg1 = val1, arg2 = val2, ...)

function_name is the name of the function. arg1 and arg2 are arguments. They’re variables to be passed into the function. The type and number of arguments depend on the definition of the function. val1 and val2 are values of the arguments correspondingly.

Passing Arguments

R can match arguments both by position and by name. So you don’t necessarily have to supply the names of the arguments if you have the positions of the arguments placed correctly.

class(x = 1)
#[1] "numeric"

class(1)
#[1] "numeric"

Functions are always accompanied with loads of arguments for configurations. However, you don’t have to supply all of the arguments for a function to work.

Here is the documentation of the sum() function:

# usage: sum(..., na.rm = FALSE)
# description: Returns the sum of all the values present in its arguments.
# arguments: ... : Numeric or complex or logical vectors.; na.rm : Logical. Should missing values (including NaN) be removed?

From the documentation, we learned that there’re two arguments for the sum() function: ... and na.rm. Notice that na.rm contains a default value FALSE. This makes it an optional argument. If you don’t supply any values to the optional arguments, the function will automatically fill in the default value to proceed.

sum(2, 10)
#[1] 12

sum(2, 10, NaN)
#[1] NaN

sum(2, 10, NaN, na.rm = TRUE)
#[1] 12

Getting Help

There is a large collection of functions in R and you’ll never remember all of them. Hence, knowing how to get help is important.

RStudio has a handy tool ? to help you in recalling the use of the functions:

?function_name

Look how magical it is to show the R documentation directly at the output panel for quick reference:

Last but not least, if you get stuck, Google it! For beginners like us, our confusions must have been gone through by numerous R learners before and there will always be something helpful and insightful on the web.

Happy learning!

Starting with R for Data Science

2017-12-04T00:00:00+00:00

R and Python, both solid and robust for data analytics and machine learning, are the most popular languages used in Data Science.¹ I embarked my learning journey with Python (indeed it’s my first programming language), and fell in love immediately with the graceful language for its elegant syntax, extensive flexibility, and powerful data science packages: Numpy, Pandas, Matplolib, Seaborn, and more. It enables me to navigate the world in a way that I’ve never imagined.

There’re a lot of discussions regarding the choice of the first language in Data Science for people who have completely no experience in the field. I’m incompetent in making detail comparisons of the pros and cons of the languages with legitimate justifications.² However, in summary of what I’ve learnt, Python is ideal for programmers, as well as people without any programming experience (like me!); while R is perfect for statistician, and also anyone with a solid mathematical background.

We’ve always heard programming advice for beginners to first become an expert of one language, then to proceed in studying another one, and soon for more. In Data Science, it is important to learn one of the two essential languages extremely well to sharpen the swiss-army knife in one’s analytic toolbox.

I haven’t been learning Python and its Data Science packages for long, but R was definitely on my future roadmap. However, due to a recent change of events, I’m preparing myself to land a junior data scientist position, and the work requires the application of R. Thus, I begin my study of R with the reputed interactive courses at DataCamp and the notorious book R for Data Science written by Hadley Wickham, one of the best R programmers in the world.

I always enjoy writing programming notes, and I reckon writing is an incredible way to consolidate knowledge and to enhance understanding of a domain. Hence, despite a myriad of R tutorials out there, I hope to share my notes and journey here to other passionate Data Science learners.

From “The State of Data Science & Machine Learning 2017”, an industry-wide survey by Kaggle, within 7,955 valid responses, 76.3% voted Python, while 59.2% voted R as the tool most used at work. ↩
“Everyone data scientist has an opinions on what language you should learn first. As it turns out, people who solely use Python or R feel like they made the right choice. But if you ask people that use both R and Python, they are twice as likely to recommend Python.”, quoted from “The State of Data Science & Machine Learning 2017”. For comparison of Data Science languages in a nutshell, read “Which language should you learn for Data Science?”. ↩

Fix Github 'GH001: Large files detected' with BFG Repo-Cleaner

2017-10-20T00:00:00+00:00

Recently I’ve accidentally committed a large data file to Git while working on a data wrangling project. When I tried to push it to Github using push -u origin master, it failed of remote error: GH001: Large files detected because the large data file exceeded Github’s 100MB file size limit.

Here’s a view of the full error code:

remote: error: GH001: Large files detected. You may want to try Git Large File Storage — https://git-lfs.github.com.
remote: error: Trace: s9116d865251981c96d8dd2cdf7ef464
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File sample/large/data/file/path.json is 679.95 MB; this exceeds GitHub’s file size limit of 100.00 MB

I can’t solve this by simply deleting the large file repository, as it has already been commited. The situation was even worsen since I’ve made few more commits ahead of the origin after this faultful commit.

Fortunately, there is a quick fix solution using a simple script called BFG Repo-Cleaner. The main objective of BFG Repo-Cleaner is to remove large or troublesome files from our Git repository history. It works like git-filter-branch, but simpler, easier, and faster.

Here is a step by step guide:

First we download the cleaner from it’s website. Here is a link to the latest version: bfg-1.12.16.jar.
The only requirement to run the script is to have the Java Runtime Environment (Java 7 or above) installed. Note that it’s a script to be run in the command line. The following commands should be submitted in terminal (or equivalent).
Before cleaning, we make a backup of our repository to ensure we don’t lose anything by running git clone --mirror git://example.com/some-big-repo.git. This is a bare repo, which means our normal files won’t be visible, but it’s a full copy of the Git database of our repository.
Now we can clean our Git repository history by running java -jar bfg.jar --strip-blobs-bigger-than 100M some-big-repo.git.

Get Exchange Rates with currencylayer API in Python

2017-10-08T00:00:00+00:00

There’re a fairly large number of APIs offering real-time and historical currency exchange rate out there. currencylayer API is one of the best. It’s an easy-to-use JSON API supporting 168 currencies, delivering accurate, reliable and consistent currency exchange rates.

This article aims to share the basics of using the currencylayer API in Python with the Requests library to get live and historical quotes.

(For comparisons in currency exchange rate APIs, see Free and Paid Currency Converter API Comparison.)

Get currencylayer API

To get started, we first need to get our currencylayer API access key from its website.

The sign up procedure is very simple. From currencylayer’s homepage, we navigate to Start using the API and select the Free Plan. The free plan supports live and historical quotes of 1,000 API Requests per month.

After creating an account, we log in to our Account Dashboard and there we find our API Access Key.

Import Requests Library

To access the API in python, we need to import the Requests library.

import requests

For those who are new to Requests, it is a simple and elegant Python HTTP library to grab, post, stream, and connect to secure web pages. If you don’t have it installed, you may easily install it with the pip install requests command.

We also store our currenclayer API key as a variable for easier access.

api = '2639ccac02d7c15359d45f9a2bc9d8ea'

Get Live Quote

Now we’re all set. Let’s try to get a live quote with our API key.

This is the “live” request URL as documented on currencylayer’s website.

http://apilayer.net/api/live

    ? access_key = YOUR_ACCESS_KEY
    & currencies = USD,AUD,CAD,PLN,MXN
    & format = 1

We’ll use the requests.get method to get the JSON response.

First, we store all the values of the required parameters in a dict. We should input all our target currencies as a string, seperated by a ‘,’.

params = {'access_key': api, 'currencies': 'USD,EUR,CNY,HKD', 'format': 1}

We create a response object r using requests.get.

r = requests.get('http://apilayer.net/api/live', params = params)

We encode our response in json using the r.json method. And here we get our live quote stored in a dict.

livequote = r.json()

# livequote
{u'privacy': u'https://currencylayer.com/privacy',
 u'quotes': {u'USDCNY': 6.653204,
  u'USDEUR': 0.852304,
  u'USDHKD': 7.804504,
  u'USDUSD': 1},
 u'source': u'USD',
 u'success': True,
 u'terms': u'https://currencylayer.com/terms',
 u'timestamp': 1507467546}

We can easily access our stored data using Python dict methods.

livequote['quotes']['USDCNY']

Get Historical Quote

It’s equally easy to get a historical quote with our API key.

The request URL is changed as below. We just need to change the url and pass an extra parameter of date to our request.

http://apilayer.net/api/historical

    ? access_key = YOUR_ACCESS_KEY
    & date = YYYY-MM-DD
    & currencies = USD,AUD,CAD,PLN,MXN
    & format = 1

We use the same requests method to get the JSON-encoded dict of the exchange rate on a historical date.

params = {'access_key': api, 'date': '2017-10-01', 'currencies': 'USD,EUR,CNY,HKD', 'format': 1}
r = requests.get('http://apilayer.net/api/historical', params = params)
histquote = r.json()

# histquote
{u'date': u'2017-10-01',
 u'historical': True,
 u'privacy': u'https://currencylayer.com/privacy',
 u'quotes': {u'USDCNY': 6.653198,
  u'USDEUR': 0.846697,
  u'USDHKD': 7.810304,
  u'USDUSD': 1},
 u'source': u'USD',
 u'success': True,
 u'terms': u'https://currencylayer.com/terms',
 u'timestamp': 1506902399}