Creating Annotated Data Frames from GEO with the GEOquery package

In this post, we will go over how to use the GEOquery package to download a data matrix (or eset object) directly into R and append specific probe annotation information to this matrix for it to be exported as a csv file for easy manipulation in Excel or spreadsheet tools. This is especially useful for sharing data with collaborators who are not familiar with R and would rather look up there favorite genes in a spreadsheet format.

Mining gene expression data from publicly available databases is a great way to find evidence to support you working hypothesis that gene X is relevant in condition Y. You may also want to mine publicly available data to build on an existing hypothesis or simply to find additional support for your favorite gene in a different animal model or experimental condition. In this post, we will go over how to use the GEOquery package to download a data matrix (or eset object) directly into R and append specific probe annotation information to this matrix for it to be exported as a csv file for easy manipulation in Excel or spreadsheet tools. This is especially useful for sharing data with collaborators who are not familiar with R and would rather look up there favorite genes in a spreadsheet format.

First, let’s start by opening an R session and creating a function to return the eset (ExpressionSet) object or the original list object downloaded by the getGEO() function in R.

getGEOdataObjects <- function(x, getGSEobject=FALSE){
# Make sure the GEOquery package is installed
# Use the getGEO() function to download the GEO data for the id stored in x
GSEDATA <- getGEO(x, GSEMatrix=T, AnnotGPL=FALSE)
# Inspect the object by printing a summary of the expression values for the first 2 columns
print(summary(exprs(GSEDATA[[1]])[, 1:2]))

# Get the eset object
eset <- GSEDATA[[1]]
# Save the objects generated for future use in the current working directory
save(GSEDATA, eset, file=paste(x, ".RData", sep=""))

# check whether we want to return the list object we downloaded on GEO or
# just the eset object with the getGSEobject argument
if(getGSEobject) return(GSEDATA) else return(eset)

We can test this function on a GEO dataset such as GSE73835 as follows:

# Store the dataset ids in a vector GEO_DATASETS just in case you want to loop through several GEO ids
GEO_DATASETS <- c("GSE73835")

# Use the function we created to return the eset object
eset <- getGEOdataObjects(GEO_DATASETS[1])
# Inspect the eset object to get the annotation GPL id

You will see the following output:

ExpressionSet (storageMode: lockedEnvironment)
assayData: 45281 features, 6 samples
element names: exprs
protocolData: none
sampleNames: GSM1904293 GSM1904294 … GSM1904298 (6 total)
varLabels: title geo_accession … data_row_count (35 total)
varMetadata: labelDescription
featureNames: ILMN_1212602 ILMN_1212603 … ILMN_3163582 (45281 total)
fvarLabels: ID Species … ORF (30 total)
fvarMetadata: Column Description labelDescription
experimentData: use ‘experimentData(object)’
Annotation: GPL6887

We will first need to download the annotation file for GPL6887. Then we can create a data frame with the probe annotation categories we are most interested in as follows:

# Get the annotation GPL id (see Annotation: GPL10558)
gpl <- getGEO('GPL6887', destdir=".")

# Inspect the table of the gpl annotation object

# Get the gene symbol and entrez ids to be used for annotations
Table(gpl)[1:10, c(1, 6, 9, 12)]

# Get the gene expression data for all the probes with a gene symbol
geneProbes <- which(!$Symbol))
probeids <- as.character(Table(gpl)$ID[geneProbes])

probes <- intersect(probeids, rownames(exprs(eset)))

geneMatrix <- exprs(eset)[probes, ]

inds <- which(Table(gpl)$ID %in% probes)
# Check you get the same probes

# Create the expression matrix with gene ids
geneMatTable <- cbind(geneMatrix, Table(gpl)[inds, c(1, 6, 9, 12)])

# Save a copy of the expression matrix as a csv file
write.csv(geneMatTable, paste(GEO_DATASETS[1], "_DataMatrix.csv", sep=""), row.names=T)

Let’s take a look at the first 6 lines of the data frame we just created with the head() function.

example1As you can see once we export this data frame as a csv file, it is much easier for others to open this file as a spreadsheet and get useful information such as the gene symbol or entrez id with the expression values across the samples.

Hope this helps and happy collaborations!

Creating color palettes in R

In the R post, we will present how to create your own color palettes and how to work with other palettes such as RColorBrewer, wesanderson and hex codes from for exciting color palettes.

There are several color palettes available in R such as rainbow(), heat.colors(), terrain.colors(), and topo.colors(). We can visualize these  as 3D pie charts using the plotrix R package.

# Let's create a pie chart with n=7 colors using each palette
sliceValues <- rep(10, 7) # each slice value=10 for proportionate slices
pie3D(sliceValues,explode=0, theta=1.2, col=rainbow(n=7), main="rainbow()")

# Let's create a figure with all 4 base color palettes
par(mfrow=c(1, 4))
pie3D(sliceValues,explode=0, theta=1.2, col=rainbow(n=7), main="rainbow()")
pie3D(sliceValues,explode=0, theta=1.2, col=heat.colors(n=7), main="heat.colors()")
pie3D(sliceValues,explode=0, theta=1.2, col=terrain.colors(n=7), main="terrain.colors()")
pie3D(sliceValues,explode=0, theta=1.2, col=topo.colors(n=7), main="topo.colors()")

Screen Shot 2016-07-10 at 9.01.30 AM

Other popular color palettes include the RColorBrewer package that has a variety of sequential, divergent and qualitative palettes and the wesanderson package that has color palettes derived from his films.


# To see all palettes available in this package
par(mfrow=c(1, 1))

# To create pie charts from a sequential, divergent and qualitative RColorBrewer palette
par(mfrow=c(1, 4))
pie3D(sliceValues,explode=0, theta=1.2, col=brewer.pal(7, "RdPu"), main="Sequential RdPu")
pie3D(sliceValues,explode=0, theta=1.2, col=brewer.pal(7, "RdGy"), main="Divergent RdGy")
pie3D(sliceValues,explode=0, theta=1.2, col=brewer.pal(7, "Set1"), main="Qualitative Set1")

# And add pie chart with a wes_anderson palette
# we will only use 5 slices in the example since the Darjeeling palette only has 5 colors
pie3D(sliceValues[1:5],explode=0, theta=1.2, col=wes_palette("Darjeeling2"), main="Darjeeling2")

Screen Shot 2016-07-10 at 9.01.45 AMYou can also create your own color palettes in R with your colors of choice with the colors() function or creating a vector with the color names. A great review and cheat sheet for R colors can be found at

# To get an idea of the colors available
length(colors()) # 657

# To see all 657 colors as a color chart you can source the R script to generate a pdf version in your working directory

Screen Shot 2016-07-09 at 5.18.32 PM

# We can create choose a palette based on the R chart as follow:
mycols <- colors()[c(8, 5, 30, 53, 118, 72)] #
# or you could enter the color names directly
# mycols <- c("aquamarine", "antiquewhite2", "blue4", "chocolate1", "deeppink2", "cyan4")

# You could also get and store all distinct colors in the cl object and use the sample function to select colors at random
cl <- colors(distinct = TRUE)
set.seed(15887) # to set random generator seed
mycols2 <- sample(cl, 7)

You can also create color palettes with hex color codes. A great example of this is to work with popular color palettes available on the website. This website has various palettes you can choose from and even derive color palettes from your favorite websites. For example, let’s grab the color palette from the website at .

Screen Shot 2016-07-09 at 5.36.02 PM

After entering the URL of our website, we will receive the hex codes for the color scheme used on the website.

Screen Shot 2016-07-09 at 5.38.34 PM

We can even export the colors as little pencils 🙂


You can also choose from hundred of color schemes based on your color of choice. For example, we will also create a color palette based on the color olive – ColorCombo382.


# For the color palette
mycols3 <- c("#c6d4e1", "#2f2016", "#fcfaea", "#456789")

# For ColorCombos382 palette
mycols4 <- c("#C3D938", "#772877", "#7C821E", "#D8B98B", "#7A4012")

# Now to get the pie charts for the last four palettes
pie3D(sliceValues,explode=0, theta=1.2, col=mycols, main="colors()")
pie3D(sliceValues,explode=0, theta=1.2, col=mycols2, main="sample(colors(distinct=TRUE)")
pie3D(sliceValues[1:4],explode=0, theta=1.2, col=mycols3, main=" color grab")
pie3D(sliceValues[1:5],explode=0, theta=1.2, col=mycols4, main="ColorCombos382")

Screen Shot 2016-07-10 at 9.01.56 AM

We can also create a color palette with the colorRampPalette() to use for heatmaps and other plots. For this example, we will use the leukemia dataset available in the GSVAdata package, which corresponds to microarray data from 37 human acute leukemias where 20 of these cases are Acute lymphoblastic leukemia (ALL) and the other 17 are ALL’s with Mixed leukemia gene rearrangements. For more information on the study please see Armstrong et al. Nat Genet 30:41-47, 2002.

data(leukemia) # loads leukemia_eset

# Create a matrix from the gene expression eset object
M1 <- exprs(leukemia_eset)

# Get a matrix of the top 50 most variable probes accros the samples
topVarGenes <- head(order(-rowVars(M1)), 50)
mat <- M1[ topVarGenes, ]
mat <- mat - rowMeans(mat)

# For sample annotation information

# Get sample group as a factor the ColSideColors
ALLgroup <- as.factor(pData(leukemia_eset)[colnames(M1), 1])

# Get the colors for the ALL subtype
sidecols <- c("#4FD5D6", "#FF0000")

# Here is a fancy color palette inspired by
cool = rainbow(50, start=rgb2hsv(col2rgb('cyan'))[1], end=rgb2hsv(col2rgb('blue'))[1])
warm = rainbow(50, start=rgb2hsv(col2rgb('red'))[1], end=rgb2hsv(col2rgb('yellow'))[1])
cols = c(rev(cool), rev(warm))
mypalette <- colorRampPalette(cols)(255)

library("gplots") # for the heatmap.2 function

png(filename="Heatmap_Example.png", width=12, height=10, units = 'in', res = 300)
heatmap.2(mat, trace="none", col=mypalette, ColSideColors=sidecols[ALLgroup],
labRow=FALSE, labCol=FALSE, mar=c(6,12), scale="row", key.title="")
legend("topright", legend=levels(ALLgroup), fill=sidecols, title="", cex=1.2)


Now you are all set to work with and create your own awesome color palettes! Happy R programing 🙂


Converting Gene Names in R with AnnotationDbi

There are many ways to convert gene accession numbers or ids to gene symbols or other types of ids in R and several R/Bioconductor packages to facilitate this process including the AnnotationDbi, annotate, and biomaRt packages. In this post, we are going to learn how to convert gene ids with the AnnotationDbi and package.

There are many ways to convert gene accession numbers or ids to gene symbols or other types of ids in R and several R/Bioconductor packages to facilitate this process including the AnnotationDbi, annotate, and biomaRt packages. In this post, we are going to learn how to convert gene ids with the AnnotationDbi and package. You could potentially modify this code to work with other species such as mice with the package.

For example, say we have a gene expression matrix stored in M1 created from an eset object you downloaded from GEO. The study I will be using for this example is A Leukemic Stem Cell Expression Signature is Associated with Clinical Outcomes in Acute Myeloid Leukemia deposited on GEO with the accession id GSE24006. To view the script on how to generate the expression set (eset) object see the post – Retrieving Gene Expression Data  Objects & Matrices From GEO.

# Convert you eset object to a matrix with the exprs() function
M1 <- exprs(eset)

# Convert the row names to entrez ids

geneSymbols <- mapIds(, keys=rownames(M1), column="SYMBOL", keytype="ENTREZID", multiVals="first")

The mapIds() function from the AnnotationDbi package returns a named vector making it simple to retrieve entrez id for a given gene as follows: <- c("658", "1360")

# returns the gene symbols of the entrez
# "BMPR1B" "CPB1"

We can create a function to return a matrix with gene symbols instead of entrez ids as follows:

getMatrixWithSymbols <- function(df){

geneSymbols <- mapIds(, keys=rownames(df), column="SYMBOL", keytype="ENTREZID", multiVals="first")

# get the entrez ids with gene symbols i.e. remove those with NA's for gene symbols
inds <- which(!
found_genes <- geneSymbols[inds]

# subset your data frame based on the found_genes
df2 <- df[names(found_genes), ]
rownames(df2) <- found_genes

# Now, let's use the function to create a matrix for the genes with gene symbols
M1symb <- getMatrixWithSymbols(M1)

We can generalize this function to go back and forth between gene symbols and entrez ids (or other ids) as follows:

We can generalize this function to go back and forth between gene symbols and entrez ids (or other ids) as follows:

# This function can take any of the columns( as type and keys as long as the row names are in the format of the keys argument
getMatrixWithSelectedIds <- function(df, type, keys){

geneSymbols <- mapIds(, keys=rownames(df), column=type, keytype=keys, multiVals="first")

# get the entrez ids with gene symbols i.e. remove those with NA's for gene symbols
inds <- which(!
found_genes <- geneSymbols[inds]

# subset your data frame based on the found_genes
df2 <- df[names(found_genes), ]
rownames(df2) <- found_genes

# for example, going from SYMBOL to ENTREZID
M1entrez <- getMatrixWithSelectedIds(M1symb, type="ENTREZID", keys="SYMBOL")

Stay tuned for more posts on Converting Gene Names in R with the annotation and biomaRt package.

Pre-processing .CEL files in R

This post shows you how to compare data from two separate studies without the hassle of tackling batch effects, etc. By scaling and centring the data in both studies, you can look for trends in the data and look for gene expression changes that go in a similar direction.

One of the most efficient ways to pre-process microarray data in R, is to use the oligo R/Bioconductor package. In a few lines of code you can go from raw .CEL files to a normalized data matrix you can work with for downstream analysis. It is particularly useful, if you wish to reanalyse a subset of .CEL files from a previously published dataset.

For example, say we are interested in comparing relative gene expression levels of Atf1, Atf3, Brca1 i in the lung, liver, and bone marrow of CB17 mice to previously published mouse tissue expression data on GEO. From our study, we have a data matrix of normalized expression values we obtained from our microarray study stored in CB17mat object.

First, we will scale all gene expression values by row to obtain center and scale these values to have an idea which organ expresses the highest levels of Atf1, Atf3, and Brca1 relative to the others in our study. To do this we will use the scale() with the default settings center=TRUE and scale=TRUE. Since the scale default function scales and centers columns we will need to transpose our matrix before proceeding.

# From your gene expression matrix stored in CB17mat
genes = c("Atf1", "Atf3", "Brca1")
CB17scal <- t(apply(CB17mat[genes, ], 1, scale))

# We will also add the missing column names to our scaled matrix
colnames(CB17scal) <- colnames(CB17mat)

# You can also plot a heatmap to look at the effects of the scaling on the expression levels across the tissues using the gplots package

# Clusters the rows (and potentially columns) by Pearson correlation as distance method
corrdist = function(x) as.dist(1-cor(t(x), method="pearson"))

# and Ward method as the agglomeration method
hclust.avl = function(x) hclust(x, method="ward.D2")

# with dendrogram="row" and Colv=NA, we are only clustering the rows i.e. genes
png(filename="example1.png", width=10, height=10, units = 'in', res = 300)
heatmap.2(CB17scal, dendrogram="row", Colv=NA, scale="none", key.title="", col=rev(redblue(250)), trace='none', cexCol=1.2, cexRow=1.5, hclustfun=hclust.avl, distfun=corrdist, margins=c(8, 12))


As you can see Atf1 and Brca1 are relatively higher in bone marrow, whereas Atf3 is relatively higher in the lung compared to the other organs.

Now let’s analyze the liver, lung and bone marrow data from the Large-scale analysis of the human and mouse transcriptomes study from Su et al, 2002, PNAS Apr 2;99(7):4465-70. The individual files from the liver, lung and bone marrow were downloaded from GSE97 and the data was normalized with the oligo package as follows:

# Move the CEL files to ~/MY_WORKING_DIRECTORY/filesToAnalyse
# Set the working directory to the folder you would like to save your results

# Load the libraries needed for the analysis

# Load the packages needed for the analysis
# You can choose to save the CEL files for your tissues of interest
geneCELs <- list.celfiles("~/MY_WORKING_DIRECTORY/filesToAnalyse", full.names=TRUE)

affyGeneFS <- read.celfiles(geneCELs)

# RMA at the probet level
geneCore <- rma(affyGeneFS)

# Inspect the eset object

# For the featureData info for the array used in this study

probeids <- featureNames(geneCore)
geneAnnotation <- select(mgu74a.db, probeids, c("SYMBOL", "ENTREZID", "GENENAME"), multiVals="first")

# Save the ESET data and annotation
save(geneCore, geneAnnotation, file="geneCoreTissueV2.RData")
saveRDS(geneCore, "geneCore.rds")
saveRDS(geneAnnotation, "geneAnnotation.rds")

# Convert the eset object to a matrix to get the gene expression values
# for Atf1, Atf3, and Brca1
M1 <- exprs(geneCore)

Gene_Symbols <- sapply(rownames(M1), getGeneSymbol, df=geneAnnotation)
DM1 <- data.frame(M1, Gene_Symbols=Gene_Symbols[rownames(M1)], stringsAsFactors=FALSE)

# Use can create a function to select the probe with the top interquantile range with IQR()
# to represent the gene expression value or use the TopIqrSymbolMatrix()
# in the "Useful Functions to Work with Microarrays" post (coming soon!)

tissueM1 <- TopIqrSymbolMatrix(DM1)
tissueExprs <- tissueM1[, 1:ncol(M1)]
rownames(tissueExprs) <- tissueM1$Gene_Symbols
head(tissueExprs[, 1:4])

# To get the geo file ids
geoColnames <- gsub(".CEL", "", colnames(tissueExprs))

# Get the new colnames based on tissue
# Create an excel sheet with the GEO .CEL file id and tissue sample names you would like to use
# Load the excel sheet as a data frame using the gdata package
TissueNames <- read.xls("GSE97_FileSampleIdentifier.xlsx", sheet=1, stringsAsFactors=FALSE, row.names=1)
TissueSamples <- TissueNames[geoColnames, 2]

# Replace the tissue exprs with the tissue names
colnames(tissueExprs) <- TissueSamples
saveRDS(tissueExprs, "tissueExprs.rds")

# Now create a matrix for our 3 genes of interest
genes = c("Atf1", "Atf3", "Brca1")
genesMat <- as.matrix(M1[genes, ])

# Let's scale and center the expression values
genesMatScal <- t(apply(genesMat, 1, scale))

# Add the column names
colnames(genesMatScal) <- rep(c("liver", "lung", "bone marrow"), each=2)

png(filename="example1b.png", width=10, height=10, units = 'in', res = 300)
heatmap.2(genesMatScal, dendrogram="row", Colv=NA, scale="none", key.title="", col=rev(redblue(250)), trace='none', cexCol=1.2, cexRow=1.5, hclustfun=hclust.avl, distfun=corrdist, margins=c(8, 12))


By comparing both heatmaps, we can see that Atf1 and Brca1 have more similar expression patterns across the tissues than Atf3 in these mice compared to our CB17 mice tissue data.
Therefore, Brca1 and Atf1 tissue difference might be more consistent among the tissues from different mice.

Alternatively, it could just be an artifact of the small sample size and/or how the samples were processed before running the arrays. That being said, it’s a good thing to reanalyse studies and compare with your arrays to get an idea of sample variability and how the experimental design, pre- and post- processing affects the overall interpretation of the results.

Feel free to leave comments or email me at