1 Introduction

For a general practical introduction to pRoloc, readers are referred to the tutorial, available using vignette("pRoloc-tutorial", package = "pRoloc"). The following document provides a overview of the algorithms available in the package. The respective section describe unsupervised machine learning (USML), supervised machine learning (SML), semi-supervised machine learning (SSML) as implemented in the novelty detection algorithm and transfer learning.

2 Data sets

We provide 75 test data sets in the pRolocdata package that can be readily used with pRoloc. The data set can be listed with pRolocdata and loaded with the data function. Each data set, including its origin, is individually documented.

The data sets are distributed as MSnSet instances. Briefly, these are dedicated containers for quantitation data as well as feature and sample meta-data. More details about MSnSets are available in the pRoloc tutorial and in the MSnbase package, that defined the class.

library("pRolocdata")
data(tan2009r1)
tan2009r1
## MSnSet (storageMode: lockedEnvironment)
## assayData: 888 features, 4 samples 
##   element names: exprs 
## protocolData: none
## phenoData
##   sampleNames: X114 X115 X116 X117
##   varLabels: Fractions
##   varMetadata: labelDescription
## featureData
##   featureNames: P20353 P53501 ... P07909 (888 total)
##   fvarLabels: FBgn Protein.ID ... markers.tl (16 total)
##   fvarMetadata: labelDescription
## experimentData: use 'experimentData(object)'
##   pubMedIds: 19317464 
## Annotation:  
## - - - Processing information - - -
## Added markers from  'mrk' marker vector. Thu Jul 16 22:53:44 2015 
##  MSnbase version: 1.17.12

Other omics data

While our primary biological domain is quantitative proteomics, with special emphasis on spatial proteomics, the underlying class infrastructure on which pRoloc and implemented in the Bioconductor MSnbase package enables the conversion from/to transcriptomics data, in particular microarray data available as ExpressionSet objects using the as coercion methods (see the MSnSet section in the MSnbase-development vignette). As a result, it is straightforward to apply the methods summarised here in detailed in the other pRoloc vignettes to these other data structures.

3 Unsupervised machine learning

Unsupervised machine learning refers to clustering, i.e. finding structure in a quantitative, generally multi-dimensional data set of unlabelled data.

Currently, unsupervised clustering facilities are available through the plot2D function and the MLInterfaces package (Carey et al., n.d.). The former takes an MSnSet instance and represents the data on a scatter plot along the first two principal components. Arbitrary feature meta-data can be represented using different colours and point characters. The reader is referred to the manual page available through ?plot2D for more details and examples.

pRoloc also implements a MLean method for MSnSet instances, allowing to use the relevant infrastructure with the organelle proteomics framework. Although provides a common interface to unsupervised and numerous supervised algorithms, we refer to the pRoloc tutorial for its usage to several clustering algorithms.

Note Current development efforts in terms of clustering are described on the Clustering infrastructure wiki page (https://github.com/lgatto/pRoloc/wiki/Clustering-infrastructure) and will be incorporated in future version of the package.

4 Supervised machine learning

Supervised machine learning refers to a broad family of classification algorithms. The algorithms learns from a modest set of labelled data points called the training data. Each training data example consists of a pair of inputs: the actual data, generally represented as a vector of numbers and a class label, representing the membership to exactly 1 of multiple possible classes. When there are only two possible classes, on refers to binary classification. The training data is used to construct a model that can be used to classifier new, unlabelled examples. The model takes the numeric vectors of the unlabelled data points and return, for each of these inputs, the corresponding mapped class.

4.1 Algorithms used

k-nearest neighbour (KNN) Function knn from package class. For each row of the test set, the \(k\) nearest (in Euclidean distance) training set vectors are found, and the classification is decided by majority vote over the \(k\) classes, with ties broken at random. This is a simple algorithm that is often used as baseline classifier. %% If there are ties for the \(k^{th}\) nearest vector, all candidates are included in the vote.

Partial least square DA (PLS-DA) Function plsda from package . Partial least square discriminant analysis is used to fit a standard PLS model for classification.

Support vector machine (SVM) A support vector machine constructs a hyperplane (or set of hyperplanes for multiple-class problem), which are then used for classification. The best separation is defined as the hyperplane that has the largest distance (the margin) to the nearest data points in any class, which also reduces the classification generalisation error. To assure liner separation of the classes, the data is transformed using a kernel function into a high-dimensional space, permitting liner separation of the classes.

pRoloc makes use of the functions svm from package and ksvm from .

Artificial neural network (ANN) Function nnet from package . Fits a single-hidden-layer neural network, possibly with skip-layer connections.

Naive Bayes (NB) Function naiveBayes from package . Naive Bayes classifier that computes the conditional a-posterior probabilities of a categorical class variable given independent predictor variables using the Bayes rule. Assumes independence of the predictor variables, and Gaussian distribution (given the target class) of metric predictors.

Random Forest (RF) Function randomForest from package .

Chi-square (\(\chi^2\)) Assignment based on squared differences between a labelled marker and a new feature to be classified. Canonical protein correlation profile method (PCP) uses squared differences between a labelled marker and new features. In (Andersen et al. 2003), \(\chi^2\) is defined as , i.e. \(\chi^{2} = \frac{\sum_{i=1}^{n} (x_i - m_i)^{2}}{n}\), whereas (Wiese et al. 2007) divide by the value the squared value by the value of the reference feature in each fraction, i.e. \(\chi^{2} = \sum_{i=1}^{n}\frac{(x_i - m_i)^{2}}{m_i}\), where \(x_i\) is normalised intensity of feature \(x\) in fraction \(i\), \(m_i\) is the normalised intensity of marker \(m\) in fraction \(i\) and \(n\) is the number of fractions available. We will use the former definition.

PerTurbo From (Courty, Burger, and Laurent 2011): PerTurbo, an original, non-parametric and efficient classification method is presented here. In our framework, the manifold of each class is characterised by its Laplace-Beltrami operator, which is evaluated with classical methods involving the graph Laplacian. The classification criterion is established thanks to a measure of the magnitude of the spectrum perturbation of this operator. The first experiments show good performances against classical algorithms of the state-of-the-art. Moreover, from this measure is derived an efficient policy to design sampling queries in a context of active learning. Performances collected over toy examples and real world datasets assess the qualities of this strategy.

The PerTurbo implementation comes from the pRoloc packages.

4.2 Estimating algorithm parameters

It is essential when applying any of the above classification algorithms, to wisely set the algorithm parameters, as these can have an important effect on the classification. Such parameters are for example the width \(sigma\) of the Radial Basis Function (Gaussian kernel) \(exp(-\sigma \| x - x' \|^2 )\) and the \(cost\) (slack) parameter (controlling the tolerance to mis-classification) of our SVM classifier. The number of neighbours \(k\) of the KNN classifier is equally important as will be discussed in this sections.

The next figure illustrates the effect of different choices of \(k\) using organelle proteomics data from (Dunkley et al. 2006) (dunkley2006 from pRolocdata). As highlighted in the squared region, we can see that using a low \(k\) (\(k = 1\) on the left) will result in very specific classification boundaries that precisely follow the contour or our marker set as opposed to a higher number of neighbours (\(k = 8\) on the right). While one could be tempted to believe that optimised classification boundaries are preferable, it is essential to remember that these boundaries are specific to the marker set used to construct them, while there is absolutely no reason to expect these regions to faithfully separate any new data points, in particular proteins that we wish to classify to an organelle. In other words, the highly specific \(k = 1\) classification boundaries are over-fitted for the marker set or, in other words, lack generalisation to new instances. We will demonstrate this using simulated data taken from (James et al. 2013) and show what detrimental effect over-fitting has on new data.

Classification boundaries using k=1 or k=8 on the dunkley2006 data.

Classification boundaries using \(k=1\) or \(k=8\) on the dunkley2006 data.

The figure below uses 2 \(\times\) 100 simulated data points belonging to either of the orange or blue classes. The genuine classes for all the points is known (solid lines) and the KNN algorithm has been applied using \(k=1\) (left) and \(k=100\) (right) respectively (purple dashed lines). As in our organelle proteomics examples, we observe that when k = 1, the decision boundaries are overly flexible and identify patterns in the data that do not reflect to correct boundaries (in such cases, the classifier is said to have low bias but very high variance). When a large \(k\) is used, the classifier becomes inflexible and produces approximate and nearly linear separation boundaries (such a classifier is said to have low variance but high bias). On this simulated data set, neither \(k = 1\) nor \(k = 100\) give good predictions and have test error rates (i.e. the proportion of wrongly classified points ) of 0.1695 and 0.1925, respectively.

The KNN classifier using k = 1 (left, solid classification boundaries) and k = 100 (right, solid classification boundaries) compared the Bayes decision boundaries (see original material for details). Reproduced with permission from [James et al. (2013)].

The KNN classifier using \(k = 1\) (left, solid classification boundaries) and \(k = 100\) (right, solid classification boundaries) compared the Bayes decision boundaries (see original material for details). Reproduced with permission from [James et al. (2013)].

To quantify the effect of flexibility and lack thereof in defining the classification boundaries, (James et al. 2013) calculate the classification error rates using training data (training error rate) and testing data (testing error rate). The latter is completely new data that was not used to assess the model error rate when defining algorithm parameters; one often says that the model used for classification has not seen this data. If the model performs well on new data, it is said to generalise well. This is a quality that is required in most cases, and in particular in our organelle proteomics experiments where the training data corresponds to our marker sets. Figure @ref{fig:ISL2} plots the respective training and testing error rates as a function of \(1/k\) which is a reflection of the flexibility/complexity of our model; when \(1/k = 1\), i.e. \(k = 1\) (far right), we have a very flexible model with the risk of over-fitting. Greater values of \(k\) (towards the left) characterise less flexible models. As can be seen, high values of \(k\) produce poor performance for both training and testing data. However, while the training error steadily decreases when the model complexity increases (smaller \(k\)), the testing error rate displays a typical U-shape: at a value around \(k = 10\), the testing error rate reaches a minimum and then starts to increase due to over-fitting to the training data and lack of generalisation of the model.

Effect of train and test error with respect to model complexity. The former decreases for lower values of k while the test error reaches a minimum around k = 10 before increasing again. Reproduced with permission from [James et al. (2013)].

Effect of train and test error with respect to model complexity. The former decreases for lower values of \(k\) while the test error reaches a minimum around \(k = 10\) before increasing again. Reproduced with permission from [James et al. (2013)].

These results show that adequate optimisation of the model parameters are essential to avoid either too flexible models (that do not generalise well to new data) or models that do not describe the decision boundaries adequately. Such parameter selection is achieved by cross validation, where the initial marker proteins are separated into training data used to build classification models and independent testing data used to assess the model on new data.

We recommend the book An Introduction to Statistical Learning (http://www-bcf.usc.edu/~gareth/ISL/) by (James et al. 2013) for a more detailed introduction of machine learning.

4.3 Default analysis scheme

Below, we present a typical classification analysis using pRoloc. The analysis typically consists of two steps. The first one is to optimise the classifier parameters to be used for training and testing (see above). A range of parameters are tested using the labelled data, for which the labels are known. For each set of parameters, we hide the labels of a subset of labelled data and use the other part to train a model and apply in on the testing data with hidden labels. The comparison of the estimated and expected labels enables to assess the validity of the model and hence the adequacy if the parameters. Once adequate parameters have been identified, they are used to infer a model on the complete organelle marker set and used to infer the sub-cellular location of the unlabelled examples.

Parameter optimisation

Algorithmic performance is estimated using a stratified 20/80 partitioning. The 80% partitions are subjected to 5-fold cross-validation in order to optimise free parameters via a grid search, and these parameters are then applied to the remaining 20%. The procedure is repeated \(n = 100\) times to sample \(n\) accuracy metrics (see below) values using \(n\), possibly different, optimised parameters for evaluation.

Models accuracy is evaluated using the F1 score, \(F1 = 2 ~ \frac{precision \times recall}{precision + recall}\), calculated as the harmonic mean of the precision (\(precision = \frac{tp}{tp+fp}\), a measure of exactness – returned output is a relevant result) and recall (\(recall=\frac{tp}{tp+fn}\), a measure of completeness – indicating how much was missed from the output). What we are aiming for are high generalisation accuracy, i.e high \(F1\), indicating that the marker proteins in the test data set are consistently correctly assigned by the algorithms.

The results of the optimisation procedure are stored in an GenRegRes object that can be inspected, plotted and best parameter pairs can be extracted.

For a given algorithm alg, the corresponding parameter optimisation function is names algOptimisation or, equivalently, algOptimization. See table 1 for details. A description of each of the respective model parameters is provided in the optimisation function manuals, available through ?algOptimisation.

params <- svmOptimisation(tan2009r1, times = 10,
                          xval = 5, verbose = FALSE)
params
## Object of class "GenRegRes"
## Algorithm: svm 
## Hyper-parameters:
##  cost: 0.0625 0.125 0.25 0.5 1 2 4 8 16
##  sigma: 0.001 0.01 0.1 1 10 100
## Design:
##  Replication: 10 x 5-fold X-validation
##  Partitioning: 0.2/0.8 (test/train)
## Results
##  macro F1:
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##  0.7564  0.8074  0.8704  0.8631  0.9266  0.9385 
##  best sigma: 0.1 1   
##  best cost: 8 16 4   
## Use getWarnings() to see warnings.

Classification

tan2009r1 <- svmClassification(tan2009r1, params)
tan2009r1
## MSnSet (storageMode: lockedEnvironment)
## assayData: 888 features, 4 samples 
##   element names: exprs 
## protocolData: none
## phenoData
##   sampleNames: X114 X115 X116 X117
##   varLabels: Fractions
##   varMetadata: labelDescription
## featureData
##   featureNames: P20353 P53501 ... P07909 (888 total)
##   fvarLabels: FBgn Protein.ID ... svm.scores (18 total)
##   fvarMetadata: labelDescription
## experimentData: use 'experimentData(object)'
##   pubMedIds: 19317464 
## Annotation:  
## - - - Processing information - - -
## Added markers from  'mrk' marker vector. Thu Jul 16 22:53:44 2015 
## Performed svm prediction (sigma=1 cost=16) Mon Sep 24 19:43:11 2018 
##  MSnbase version: 1.17.12

4.4 Customising model parameters

Below we illustrate how to weight different classes according to the number of labelled instances, where large sets are down weighted. This strategy can help with imbalanced designs.

w <- table(fData(markerMSnSet(dunkley2006))$markers)
wpar <- svmOptimisation(dunkley2006, class.weights = w)
wres <- svmClassification(dunkley2006, wpar, class.weights = w)
parameter optimisation classification algorithm package
knnOptimisation knnClassification nearest neighbour class
knntlOptimisation knntlClassification nearest neighbour transfer learning pRoloc
ksvmOptimisation ksvmClassification support vector machine kernlab
nbOptimisation nbClassification naive bayes e1071
nnetOptimisation nnetClassification neural networks nnet
perTurboOptimisation perTurboClassification PerTurbo pRoloc
plsdaOptimisation plsdaClassification partial least square caret
rfOptimisation rfClassification random forest randomForest
svmOptimisation svmClassification support vector machine e1071
Table:(#tab:mltab) Machi ne learning methods in *p Roloc*.

5 Comparison of different classifiers

Several supervised machine learning algorithms have already been applied to organelle proteomics data classification: partial least square discriminant analysis in (Dunkley et al. 2006, Tan2009), support vector machines (SVMs) in (Trotter et al. 2010), random forest in (Ohta et al. 2010), neural networks in (Tardif et al. 2012), naive Bayes (Nikolovski et al. 2012). In our HUPO 2011 poster1 Gatto, Laurent; Breckels, Lisa M.; Trotter, Matthew W.B.; Lilley, Kathryn S. (2011): pRoloc - A unifying bioinformatics framework for organelle proteomics. https://doi.org/10.6084/m9.figshare.5042965.v1, we show that different classification algorithms provide very similar performance. We have extended this comparison on various datasets distributed in the pRolocdata package. On figure @ref{fig:f1box}, we illustrate how different algorithms reach very similar performances on most of our test datasets.

Comparison of classification performances of several contemporary classification algorithms on data from the pRolocdata package.

Comparison of classification performances of several contemporary classification algorithms on data from the pRolocdata package.

6 Semi-supervised machine learning

The phenoDisco algorithm is a semi-supervised novelty detection method by (Breckels et al. 2013) (figure below). It uses the labelled (i.e. markers, noted \(D_L\)) and unlabelled (i.e. proteins of unknown localisation, noted \(D_U\)) sets of the input data. The algorithm is repeated \(N\) times (the times argument in the phenoDisco function). At each iteration, each organelle class \(D_{L}^{i}\) and the unlabelled complement are clustered using Gaussian mixture modelling. While unlabelled members that systematically cluster with \(D_{L}^{i}\) and pass outlier detection are labelled as new putative members of class \(i\), any example of \(D_U\) which are not merged with any any of the \(D_{L}^{i}\) and are consistently clustered together throughout the \(N\) iterations are considered members of a new phenotype.

The PhenoDisco iterative algorithm.

The PhenoDisco iterative algorithm.

7 Transfer learning

When multiple sources of data are available, it is often beneficial to take all or several into account with the aim of increasing the information to tackle a problem of interest. While it is at times possible to combine these different sources of data, this can lead to substantially harm to performance of the analysis when the different data sources are of variable signal-to-noise ratio or the data are drawn from different domains and recorded by different encoding (quantitative and binary, for example). If we defined the following two data source

  1. primary data, of high signal-to-noise ratio, but general available in limited amounts;
  2. auxiliary data, of limited signal-to-noise, and available in large amounts;

then, a transfer learning algorithm will efficiently support/complement the primary target domain with auxiliary data features without compromising the integrity of our primary data.

We have developed a transfer learning framework (Breckels et al. 2016) and applied to the analysis of spatial proteomics data, as described in the pRoloc-transfer-learning vignette.

8 Session information

All software and respective versions used to produce this document are listed below.

## R version 3.5.1 Patched (2018-07-12 r74967)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 16.04.5 LTS
## 
## Matrix products: default
## BLAS: /home/biocbuild/bbs-3.7-bioc/R/lib/libRblas.so
## LAPACK: /home/biocbuild/bbs-3.7-bioc/R/lib/libRlapack.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats4    parallel  stats     graphics  grDevices utils     datasets 
## [8] methods   base     
## 
## other attached packages:
##  [1] knitr_1.20           pRolocdata_1.18.0    pRoloc_1.20.2       
##  [4] MLInterfaces_1.60.1  cluster_2.0.7-1      annotate_1.58.0     
##  [7] XML_3.98-1.16        AnnotationDbi_1.42.1 IRanges_2.14.12     
## [10] S4Vectors_0.18.3     MSnbase_2.6.4        ProtGenerics_1.12.0 
## [13] BiocParallel_1.14.2  mzR_2.14.0           Rcpp_0.12.18        
## [16] Biobase_2.40.0       BiocGenerics_0.26.0  BiocStyle_2.8.2     
## 
## loaded via a namespace (and not attached):
##   [1] backports_1.1.2       plyr_1.8.4            igraph_1.2.2         
##   [4] lazyeval_0.2.1        splines_3.5.1         ggvis_0.4.3          
##   [7] crosstalk_1.0.0       ggplot2_3.0.0         digest_0.6.17        
##  [10] foreach_1.4.4         BiocInstaller_1.30.0  htmltools_0.3.6      
##  [13] GO.db_3.6.0           viridis_0.5.1         gdata_2.18.0         
##  [16] magrittr_1.5          memoise_1.1.0         doParallel_1.0.14    
##  [19] sfsmisc_1.1-2         limma_3.36.5          recipes_0.1.3        
##  [22] gower_0.1.2           rda_1.0.2-2.1         dimRed_0.1.0         
##  [25] lpSolve_5.6.13        prettyunits_1.0.2     colorspace_1.3-2     
##  [28] blob_1.1.1            xfun_0.3              dplyr_0.7.6          
##  [31] crayon_1.3.4          RCurl_1.95-4.11       hexbin_1.27.2        
##  [34] genefilter_1.62.0     bindr_0.1.1           impute_1.54.0        
##  [37] DRR_0.0.3             survival_2.42-6       iterators_1.0.10     
##  [40] glue_1.3.0            gtable_0.2.0          ipred_0.9-7          
##  [43] zlibbioc_1.26.0       ddalpha_1.3.4         kernlab_0.9-27       
##  [46] prabclus_2.2-6        DEoptimR_1.0-8        abind_1.4-5          
##  [49] scales_1.0.0          vsn_3.48.1            mvtnorm_1.0-8        
##  [52] DBI_1.0.0             viridisLite_0.3.0     xtable_1.8-3         
##  [55] progress_1.2.0        magic_1.5-9           bit_1.1-14           
##  [58] proxy_0.4-22          mclust_5.4.1          preprocessCore_1.42.0
##  [61] lava_1.6.3            prodlim_2018.04.18    sampling_2.8         
##  [64] htmlwidgets_1.2       httr_1.3.1            threejs_0.3.1        
##  [67] FNN_1.1.2.1           RColorBrewer_1.1-2    fpc_2.1-11.1         
##  [70] modeltools_0.2-22     pkgconfig_2.0.2       flexmix_2.3-14       
##  [73] nnet_7.3-12           caret_6.0-80          tidyselect_0.2.4     
##  [76] rlang_0.2.2           reshape2_1.4.3        later_0.7.5          
##  [79] munsell_0.5.0         mlbench_2.1-1         tools_3.5.1          
##  [82] RSQLite_2.1.1         pls_2.7-0             broom_0.5.0          
##  [85] geometry_0.3-6        evaluate_0.11         stringr_1.3.1        
##  [88] mzID_1.18.0           yaml_2.2.0            ModelMetrics_1.2.0   
##  [91] bit64_0.9-7           robustbase_0.93-3     randomForest_4.6-14  
##  [94] purrr_0.2.5           dendextend_1.8.0      bindrcpp_0.2.2       
##  [97] nlme_3.1-137          whisker_0.3-2         mime_0.5             
## [100] RcppRoll_0.3.0        biomaRt_2.36.1        compiler_3.5.1       
## [103] curl_3.2              e1071_1.7-0           affyio_1.50.0        
## [106] tibble_1.4.2          stringi_1.2.4         highr_0.7            
## [109] lattice_0.20-35       trimcluster_0.1-2.1   Matrix_1.2-14        
## [112] gbm_2.1.4             pillar_1.3.0          MALDIquant_1.18      
## [115] data.table_1.11.6     bitops_1.0-6          httpuv_1.4.5         
## [118] R6_2.2.2              pcaMethods_1.72.0     affy_1.58.0          
## [121] hwriter_1.3.2         bookdown_0.7          promises_1.0.1       
## [124] gridExtra_2.3         codetools_0.2-15      MASS_7.3-50          
## [127] gtools_3.8.1          assertthat_0.2.0      CVST_0.2-2           
## [130] rprojroot_1.3-2       withr_2.1.2           diptest_0.75-7       
## [133] hms_0.4.2             grid_3.5.1            rpart_4.1-13         
## [136] timeDate_3043.102     tidyr_0.8.1           class_7.3-14         
## [139] rmarkdown_1.10        lubridate_1.7.4       shiny_1.1.0          
## [142] base64enc_0.1-3

References

Andersen, Jens S., Christopher J. Wilkinson, Thibault Mayor, Peter Mortensen, Erich A. Nigg, and Matthias Mann. 2003. “Proteomic Characterization of the Human Centrosome by Protein Correlation Profiling.” Nature 426 (6966). Center for Experimental BioInformatics, Department of Biochemistry; Molecular Biology, University of Southern Denmark, Campusvej 55, DK-5230 Odense M, Denmark.:570–74. https://doi.org/10.1038/nature02166.

Breckels, L M, S B Holden, D Wojnar, C M Mulvey, A Christoforou, A Groen, M W Trotter, O Kohlbacher, K S Lilley, and L Gatto. 2016. “Learning from Heterogeneous Data Sources: An Application in Spatial Proteomics.” PLoS Comput Biol 12 (5):e1004920. https://doi.org/10.1371/journal.pcbi.1004920.

Breckels, Lisa M, Laurent Gatto, Andy Christoforou, Arnoud J Groen, Kathryn S Lilley, and Matthew W B Trotter. 2013. “The Effect of Organelle Discovery Upon Sub-Cellular Protein Localisation.” J Proteomics, March. https://doi.org/10.1016/j.jprot.2013.02.019.

Carey, Vince, Robert Gentleman, Jess Mar, and contributions from Jason Vertrees, and Laurent Gatto. n.d. MLInterfaces: Uniform Interfaces to R Machine Learning Procedures for Data in Bioconductor Containers.

Courty, Nicolas, Thomas Burger, and Johann Laurent. 2011. “PerTurbo: A New Classification Algorithm Based on the Spectrum Perturbations of the Laplace-Beltrami Operator.” In In the Proceedings of Ecml/Pkdd (1), edited by Dimitrios Gunopulos, Thomas Hofmann, Donato Malerba, and Michalis Vazirgiannis, 6911:359–74. Lecture Notes in Computer Science. Springer.

Dunkley, Tom P. J., Svenja Hester, Ian P. Shadforth, John Runions, Thilo Weimar, Sally L. Hanton, Julian L. Griffin, et al. 2006. “Mapping the Arabidopsis Organelle Proteome.” Proc Natl Acad Sci USA 103 (17). Department of Biochemistry, University of Cambridge, Downing Site, Cambridge CB2 1QW, United Kingdom.:6518–23. https://doi.org/10.1073/pnas.0506958103.

James, G., D. Witten, T. Hastie, and R. Tibshirani. 2013. An Introduction to Statistical Learning: With Applications in R. Springer Texts in Statistics. Springer.

Nikolovski, N, D Rubtsov, M P Segura, G P Miles, T J Stevens, T P Dunkley, S Munro, K S Lilley, and P Dupree. 2012. “Putative Glycosyltransferases and Other Plant Golgi Apparatus Proteins Are Revealed by LOPIT Proteomics.” Plant Physiol 160 (2):1037–51. https://doi.org/10.1104/pp.112.204263.

Ohta, S, J C Bukowski-Wills, L Sanchez-Pulido, Fde L Alves, L Wood, Z A Chen, M Platani, et al. 2010. “The Protein Composition of Mitotic Chromosomes Determined Using Multiclassifier Combinatorial Proteomics.” Cell 142 (5):810–21. https://doi.org/10.1016/j.cell.2010.07.047.

Tardif, M, A Atteia, M Specht, G Cogne, N Rolland, S Brugière, M Hippler, et al. 2012. “PredAlgo: A New Subcellular Localization Prediction Tool Dedicated to Green Algae.” Mol Biol Evol 29 (12):3625–39. https://doi.org/10.1093/molbev/mss178.

Trotter, Matthew W. B., Pawel G. Sadowski, Tom P. J. Dunkley, Arnoud J. Groen, and Kathryn S. Lilley. 2010. “Improved Sub-Cellular Resolution via Simultaneous Analysis of Organelle Proteomics Data Across Varied Experimental Conditions.” PROTEOMICS 10 (23). WILEY-VCH Verlag:4213–9. https://doi.org/10.1002/pmic.201000359.

Wiese, Sebastian, Thomas Gronemeyer, Rob Ofman, Markus Kunze, Cláudia P. Grou, José A. Almeida, Martin Eisenacher, et al. 2007. “Proteomics Characterization of Mouse Kidney Peroxisomes by Tandem Mass Spectrometry and Protein Correlation Profiling.” Mol Cell Proteomics 6 (12). Medizinisches Proteom-Center, Ruhr-Universitaet Bochum, Universitaetsstrasse 150, 44780 Bochum, Germany.:2045–57. https://doi.org/10.1074/mcp.M700169-MCP200.