1) Cleaning the data

We are mostly interested in the tweets of the dataset and their timestamps, hence we would like to clean them so that we will only get the sentiments to obtain the document-term matrix. We will use the tm package to achieve that.

The following code is used to obtain the document-term matrix:

# Obtaining a corpus object

corpus = as.character(tweets_mutated[, "tweets"]) %>%
  removeURL %>%
  removeSpecial %>%
  stemWord %>%
  VectorSource %>%
  VCorpus

# Transforming the corpus
 
cleaned_corpus = tm_map(corpus, tolower) %>%
  tm_map(PlainTextDocument) %>%
  tm_map(removePunctuation) %>%
  tm_map(removeNumbers) %>%
  tm_map(removeWords, stopwords("english")) %>%
  tm_map(stripWhitespace)
 
# Obtaining the Document Term Matrix
 
DTM = DocumentTermMatrix(cleaned_corpus) %>% 
  removeSparseTerms(0.99) %>%
  as.matrix

# Merging the data

res_H = cbind.data.frame(DTM, "hits" = tweets_mutated$hillary_hits/100 )
res_T = cbind.data.frame(DTM, "hits" = tweets_mutated$trump_hits/100 )

The idea is to pipeline the tweets from the dataset through a process of transformations. An initial glimpse of the data reveals that the some of the tweets have links and special characters (such as emojis) that needed to be removed, hence we created the functions removeURL and removeSpecial to do just that. Furthermore, we recognized that, depending on the context, some words mean the same thing, hence we had to “stem” the words so that they will return to their original word. A function called hunspell_stem() from the hunspell package allowed us to do just that (and more efficiently than a similar from the tm package). The function is included in the functional stemWord so that the tweets that go through it will return the word in its root form.

Suppose a vector of characters:

[1] "loving" "love" "loved" "lovely"

Depending on the context, we would want a vector that would be expressed as following:

[1] "love" "love" "loved" "love"

This process is important so that the document term matrix can pick up on the frequency at which the sentiments appear since it usually returns a matrix that is sparse and that that would be problematic when interpreting the results. Furthermore, we let the tweets ran through some more functions like tolower,removePunctuation, removeNumbers, removeWords (to remove English stopwords) and stripWhiteSpace to obtain tweets of only sentiments.

Note that before we had to go through this process, we had to concatenate the strings by each day to match with the number of hits we have since we have too have them by day. This is in line with our hypothesis that the search hits are not only affected by the sentiments invoked by one candidate individually, but rather collectively.

After transforming the tweets, we transformed them into a document-term matrix and remove sparse terms.

The dimension of our document-term matrix is:

dim(DTM)

## [1] 268 226

And now, we are ready to do our analysis.

2) Logistic Regression

With the dataset, we attempt to fit a logistic regression since our relative popularity (i.e. search hits) is designed to be bounded from 0 to 1 and signify proportions. First, we will crudely fit the response with all of the features. Once we do that, we will get the following results:

# MSE for Hillary's model (adjusted for scale)
mse(logit_H$fitted.values,res_H$hits)*100^2

## [1] 1.043019

# MSE for Trump's model (adjusted for scale)
mse(logit_T$fitted.values,res_T$hits)*100^2

## [1] 22.39258

# Number of features used

## On Hillary's GTrends
length(AIC.glm_H$coefficients) - 1 

## [1] 226

## On Trump's GTrends 
length(AIC.glm_T$coefficients) - 1 

## [1] 226

3) LASSO vs Ridge Regressions

We may also be interested in doing a penalized linear regressions to find out which variables matter more in the estimate. Two of the most common penalized regressions are the LASSO and Ridge regressions, with the former penalizes the coefficients more. Since our design matrix is a document-term matrix, some terms may not be that useful, hence using the penalized regressions can help us ignore those variables. We attempt to tune the penalty for LASSO first:

# LASSO fit for Hillary
cvH.fit=cv.glmnet(as.matrix(res_H[,1:227]),as.matrix(res_H$hits),type.measure="mse")
plot(cvH.fit)

# LASSO fit for Trump
cvT.fit=cv.glmnet(as.matrix(res_T[,1:227]),as.matrix(res_T$hits),type.measure="mse")
plot(cvT.fit)

# The tuning parameters

## For Hillary
cvH.fit$lambda.1se

## [1] 0.001972102

## For Trump
cvT.fit$lambda.1se

## [1] 0.0046166

The parameters chosen here are the ones that minimize the MSE, hence we will be using them as the parameter to fit our models.

## MSE for Hillary's LASSO model (adjusted for scale)
mse(yhat111,res_H$hits)*100^2

## [1] 0.03889185

## MSE for Trump's LASSO model (adjusted for scale)
mse(yhat222,res_T$hits)*100^2

## [1] 0.21313

We see that the MSE for both of the plots seem to fit perfectly, which is a concern since we have a sparse matrix. Due to this, performing a gradient descent will not be meaningful.

Futhermore, we may now wish to perform a Ridge regression.

# Define a sequence of lambdas
l = seq(0, 1000, by = 0.01)

# Fitting the models with lambdas
ridge.H = lm.ridge(hits~., res_H, lambda = l)
ridge.T = lm.ridge(hits~., res_T, lambda = l)

# Plotting GCV against lambdas

plot(ridge.H$lambda, ridge.H$GCV, col = "darkorange" ,type = "l", ylab = "GCV", xlab = "lambda", main = "GCV against lambdas for Hillary's Ridge Regression")

plot(ridge.T$lambda, ridge.T$GCV, col = "darkorange" ,type = "l", ylab = "GCV", xlab = "lambda", main = "GCV against lambdas for Trump's Ridge Regression")

Similar to LASSO, the Ridge regression has the same tuning parameter for penalty, which we will tune differently than LASSO by using the GCV metric. After obtaining the minimum GCV that corresponds with the lambdas, we will then use that lambda to fit our models.

The following are the MSE’s for both models:

## MSE for Hillary's Ridge model (adjusted for scale)
mse(y.pred.H,res_H$hits)*100^2

## [1] 14.93831

## MSE for Trump's Ridge model (adjusted for scale)
mse(y.pred.T, res_T$hits)*100^2

## [1] 130.2933

We see that Ridge regressions perform badly for Trump’s Google Trends hits relative to that of Hillary. We say that this might be due to the sharp peaks that the Trump’s hits may have, or that maybe it is due to the dataset being collected since Trump’s tweets are fewer than Hillary’s.

4) Random Forest

With Random forest we first try the default version, and realize that by simply doing randomForest() and setting up the ntree we still have a pretty bad model. Therefore we start tunning the parameters of mtry and ‘nodesize’:

We can see that Clinton ’s optimal mtry is 80 and optimal nodesize is 32. Trump’s optimal mtry is 77 and its optimal nodesize is 37.

From the variable importance plots of both models, the term candidates seem to show up the most in both.

The result of the Mean Squared Errors are listed as follows:

## MSE for Hillary's RF model (adjusted for scale)
mse(rf.H$predicted, res_H$hits)*100^2

## [1] 35.92742

## MSE for Trump's RF model (adjusted for scale)
mse(rf.T$predicted, res_T$hits)*100^2

## [1] 224.5075

5) Feature Selection

We could attempt to find the features that greatly affect the prediction. However, since we have features of only factor variables, computing the distance matrices may be a problem. As such, we could use the FactoMineR package to solve this problem since it would allow us to compute the principal components for mixed variables

fact = res_H[,1:226] %>% lapply(as.factor) %>% do.call(cbind, .)
res.pca=PCA(fact[,1:226])

We see that doing a feature selection by PCA from each of the plots since they did not contribute too much to the explanations of the variance for both dimensions. Therefore, we say that using PCA on a document-term matrix may not work that well since our variables are not continuous.