Step 3: Post-Processing

Tyler J Burns

October 2, 2017

The post-processing function:

This vignette covers what takes place following the generation of SCONE output detailed in TheSconeWorkflow.Rmd. The obvious step that needs to take place is the Scone generated columns being merged into the original input data. The user gets the option of log base 10 transforming q values, which is easier to visualize. The user also gets the option to run t-SNE on the data, such that said maps can be colored by SCONE generated values. In this case, t-SNE is run utilizing the Rtsne package, using the same markers that were used as input for the KNN. generation.

library(Sconify)
wand.final <- PostProcessing(scone.output = wand.scone,
                         cell.data = wand.combined,
                         input = input.markers)
## Read the 1000 x 27 data matrix successfully!
## OpenMP is working. 1 threads.
## Using no_dims = 2, perplexity = 30.000000, and theta = 0.500000
## Computing input similarities...
## Building tree...
## Done in 0.32 seconds (sparsity = 0.126418)!
## Learning embedding...
## Iteration 50: error is 62.831950 (50 iterations in 0.27 seconds)
## Iteration 100: error is 60.443447 (50 iterations in 0.21 seconds)
## Iteration 150: error is 60.157523 (50 iterations in 0.22 seconds)
## Iteration 200: error is 60.039016 (50 iterations in 0.23 seconds)
## Iteration 250: error is 59.980867 (50 iterations in 0.23 seconds)
## Iteration 300: error is 1.377899 (50 iterations in 0.17 seconds)
## Iteration 350: error is 1.260727 (50 iterations in 0.15 seconds)
## Iteration 400: error is 1.225463 (50 iterations in 0.15 seconds)
## Iteration 450: error is 1.214288 (50 iterations in 0.15 seconds)
## Iteration 500: error is 1.204625 (50 iterations in 0.14 seconds)
## Iteration 550: error is 1.197417 (50 iterations in 0.15 seconds)
## Iteration 600: error is 1.192732 (50 iterations in 0.15 seconds)
## Iteration 650: error is 1.189537 (50 iterations in 0.15 seconds)
## Iteration 700: error is 1.187751 (50 iterations in 0.15 seconds)
## Iteration 750: error is 1.186144 (50 iterations in 0.16 seconds)
## Iteration 800: error is 1.185609 (50 iterations in 0.15 seconds)
## Iteration 850: error is 1.184140 (50 iterations in 0.15 seconds)
## Iteration 900: error is 1.183601 (50 iterations in 0.16 seconds)
## Iteration 950: error is 1.182368 (50 iterations in 0.16 seconds)
## Iteration 1000: error is 1.180294 (50 iterations in 0.16 seconds)
## Fitting performed in 3.46 seconds.
wand.combined # input data
## # A tibble: 1,000 x 51
##    `CD3(Cd110)Di` `CD3(Cd111)Di` `CD3(Cd112)Di` `CD235-61-7-15(…
##             <dbl>          <dbl>          <dbl>            <dbl>
##  1         -0.439       -0.0941        -0.458             -1.46 
##  2         -0.495       -0.218          0.923              0.285
##  3         -1.83        -1.69          -1.40              -0.926
##  4         -0.323       -0.151          1.22              -0.215
##  5         -0.125       -0.148         -0.00157            0.554
##  6         -0.181       -0.166         -0.0167            -0.848
##  7         -0.191       -0.263         -0.250             -0.303
##  8         -0.135       -0.00296        0.420              0.791
##  9         -0.158       -0.0951        -0.235             -0.369
## 10          0.288        0.0174         1.39               0.107
## # … with 990 more rows, and 47 more variables: `CD3(Cd114)Di` <dbl>,
## #   `CD45(In115)Di` <dbl>, `CD19(Nd142)Di` <dbl>, `CD22(Nd143)Di` <dbl>,
## #   `IgD(Nd145)Di` <dbl>, `CD79b(Nd146)Di` <dbl>, `CD20(Sm147)Di` <dbl>,
## #   `CD34(Nd148)Di` <dbl>, `CD179a(Sm149)Di` <dbl>, `CD72(Eu151)Di` <dbl>,
## #   `IgM(Eu153)Di` <dbl>, `Kappa(Sm154)Di` <dbl>, `CD10(Gd156)Di` <dbl>,
## #   `Lambda(Gd157)Di` <dbl>, `CD24(Dy161)Di` <dbl>, `TdT(Dy163)Di` <dbl>,
## #   `Rag1(Dy164)Di` <dbl>, `PreBCR(Ho165)Di` <dbl>, `CD43(Er167)Di` <dbl>,
## #   `CD38(Er168)Di` <dbl>, `CD40(Er170)Di` <dbl>, `CD33(Yb173)Di` <dbl>,
## #   `HLA-DR(Yb174)Di` <dbl>, Time <dbl>, Cell_length <dbl>,
## #   `cPARP(La139)Di` <dbl>, `pPLCg2(Pr141)Di` <dbl>,
## #   `pSrc(Nd144)Di` <dbl>, `pSTAT5(Nd150)Di` <dbl>, `Ki67(Sm152)Di` <dbl>,
## #   `pErk12(Gd155)Di` <dbl>, `pSTAT3(Gd158)Di` <dbl>,
## #   `pAKT(Tb159)Di` <dbl>, `pBLNK(Gd160)Di` <dbl>, `pSyk(Dy162)Di` <dbl>,
## #   `tIkBa(Er166)Di` <dbl>, `pP38(Tm169)Di` <dbl>, `pBTK(Yb171)Di` <dbl>,
## #   `pS6(Yb172)Di` <dbl>, `pCrkL(Lu175)Di` <dbl>, `pCREB(Yb176)Di` <dbl>,
## #   `DNA1(Ir191)Di` <dbl>, `DNA2(Ir193)Di` <dbl>,
## #   `Viability1(Pt195)Di` <dbl>, `Viability2(Pt196)Di` <dbl>,
## #   wanderlust <dbl>, condition <chr>
wand.scone # scone-generated data
## # A tibble: 1,000 x 34
##    `pCrkL(Lu175)Di… `pCREB(Yb176)Di… `pBTK(Yb171)Di.… `pS6(Yb172)Di.I…
##               <dbl>            <dbl>            <dbl>            <dbl>
##  1            1                    1            1                0.699
##  2            1                    1            1                0.672
##  3            1                    1            1                0.545
##  4            1                    1            1                0.545
##  5            1                    1            1                1    
##  6            0.948                1            1                0.845
##  7            1                    1            0.999            0.976
##  8            0.948                1            1                0.300
##  9            1                    1            1                0.924
## 10            0.978                1            1                0.602
## # … with 990 more rows, and 30 more variables:
## #   `cPARP(La139)Di.IL7.qvalue` <dbl>, `pPLCg2(Pr141)Di.IL7.qvalue` <dbl>,
## #   `pSrc(Nd144)Di.IL7.qvalue` <dbl>, `Ki67(Sm152)Di.IL7.qvalue` <dbl>,
## #   `pErk12(Gd155)Di.IL7.qvalue` <dbl>,
## #   `pSTAT3(Gd158)Di.IL7.qvalue` <dbl>, `pAKT(Tb159)Di.IL7.qvalue` <dbl>,
## #   `pBLNK(Gd160)Di.IL7.qvalue` <dbl>, `pP38(Tm169)Di.IL7.qvalue` <dbl>,
## #   `pSTAT5(Nd150)Di.IL7.qvalue` <dbl>, `pSyk(Dy162)Di.IL7.qvalue` <dbl>,
## #   `tIkBa(Er166)Di.IL7.qvalue` <dbl>, `pCrkL(Lu175)Di.IL7.change` <dbl>,
## #   `pCREB(Yb176)Di.IL7.change` <dbl>, `pBTK(Yb171)Di.IL7.change` <dbl>,
## #   `pS6(Yb172)Di.IL7.change` <dbl>, `cPARP(La139)Di.IL7.change` <dbl>,
## #   `pPLCg2(Pr141)Di.IL7.change` <dbl>, `pSrc(Nd144)Di.IL7.change` <dbl>,
## #   `Ki67(Sm152)Di.IL7.change` <dbl>, `pErk12(Gd155)Di.IL7.change` <dbl>,
## #   `pSTAT3(Gd158)Di.IL7.change` <dbl>, `pAKT(Tb159)Di.IL7.change` <dbl>,
## #   `pBLNK(Gd160)Di.IL7.change` <dbl>, `pP38(Tm169)Di.IL7.change` <dbl>,
## #   `pSTAT5(Nd150)Di.IL7.change` <dbl>, `pSyk(Dy162)Di.IL7.change` <dbl>,
## #   `tIkBa(Er166)Di.IL7.change` <dbl>, IL7.fraction.cond.2 <dbl>,
## #   density <dbl>
wand.final # the data after post-processing
## # A tibble: 1,000 x 87
##    `CD3(Cd110)Di` `CD3(Cd111)Di` `CD3(Cd112)Di` `CD235-61-7-15(…
##             <dbl>          <dbl>          <dbl>            <dbl>
##  1         -0.439       -0.0941        -0.458             -1.46 
##  2         -0.495       -0.218          0.923              0.285
##  3         -1.83        -1.69          -1.40              -0.926
##  4         -0.323       -0.151          1.22              -0.215
##  5         -0.125       -0.148         -0.00157            0.554
##  6         -0.181       -0.166         -0.0167            -0.848
##  7         -0.191       -0.263         -0.250             -0.303
##  8         -0.135       -0.00296        0.420              0.791
##  9         -0.158       -0.0951        -0.235             -0.369
## 10          0.288        0.0174         1.39               0.107
## # … with 990 more rows, and 83 more variables: `CD3(Cd114)Di` <dbl>,
## #   `CD45(In115)Di` <dbl>, `CD19(Nd142)Di` <dbl>, `CD22(Nd143)Di` <dbl>,
## #   `IgD(Nd145)Di` <dbl>, `CD79b(Nd146)Di` <dbl>, `CD20(Sm147)Di` <dbl>,
## #   `CD34(Nd148)Di` <dbl>, `CD179a(Sm149)Di` <dbl>, `CD72(Eu151)Di` <dbl>,
## #   `IgM(Eu153)Di` <dbl>, `Kappa(Sm154)Di` <dbl>, `CD10(Gd156)Di` <dbl>,
## #   `Lambda(Gd157)Di` <dbl>, `CD24(Dy161)Di` <dbl>, `TdT(Dy163)Di` <dbl>,
## #   `Rag1(Dy164)Di` <dbl>, `PreBCR(Ho165)Di` <dbl>, `CD43(Er167)Di` <dbl>,
## #   `CD38(Er168)Di` <dbl>, `CD40(Er170)Di` <dbl>, `CD33(Yb173)Di` <dbl>,
## #   `HLA-DR(Yb174)Di` <dbl>, Time <dbl>, Cell_length <dbl>,
## #   `cPARP(La139)Di` <dbl>, `pPLCg2(Pr141)Di` <dbl>,
## #   `pSrc(Nd144)Di` <dbl>, `pSTAT5(Nd150)Di` <dbl>, `Ki67(Sm152)Di` <dbl>,
## #   `pErk12(Gd155)Di` <dbl>, `pSTAT3(Gd158)Di` <dbl>,
## #   `pAKT(Tb159)Di` <dbl>, `pBLNK(Gd160)Di` <dbl>, `pSyk(Dy162)Di` <dbl>,
## #   `tIkBa(Er166)Di` <dbl>, `pP38(Tm169)Di` <dbl>, `pBTK(Yb171)Di` <dbl>,
## #   `pS6(Yb172)Di` <dbl>, `pCrkL(Lu175)Di` <dbl>, `pCREB(Yb176)Di` <dbl>,
## #   `DNA1(Ir191)Di` <dbl>, `DNA2(Ir193)Di` <dbl>,
## #   `Viability1(Pt195)Di` <dbl>, `Viability2(Pt196)Di` <dbl>,
## #   wanderlust <dbl>, condition <dbl>, `pCrkL(Lu175)Di.IL7.change` <dbl>,
## #   `pCREB(Yb176)Di.IL7.change` <dbl>, `pBTK(Yb171)Di.IL7.change` <dbl>,
## #   `pS6(Yb172)Di.IL7.change` <dbl>, `cPARP(La139)Di.IL7.change` <dbl>,
## #   `pPLCg2(Pr141)Di.IL7.change` <dbl>, `pSrc(Nd144)Di.IL7.change` <dbl>,
## #   `Ki67(Sm152)Di.IL7.change` <dbl>, `pErk12(Gd155)Di.IL7.change` <dbl>,
## #   `pSTAT3(Gd158)Di.IL7.change` <dbl>, `pAKT(Tb159)Di.IL7.change` <dbl>,
## #   `pBLNK(Gd160)Di.IL7.change` <dbl>, `pP38(Tm169)Di.IL7.change` <dbl>,
## #   `pSTAT5(Nd150)Di.IL7.change` <dbl>, `pSyk(Dy162)Di.IL7.change` <dbl>,
## #   `tIkBa(Er166)Di.IL7.change` <dbl>, IL7.fraction.cond.2 <dbl>,
## #   density <dbl>, `bh-SNE1` <dbl>, `bh-SNE2` <dbl>,
## #   `pCrkL(Lu175)Di.IL7.qvalue` <dbl>, `pCREB(Yb176)Di.IL7.qvalue` <dbl>,
## #   `pBTK(Yb171)Di.IL7.qvalue` <dbl>, `pS6(Yb172)Di.IL7.qvalue` <dbl>,
## #   `cPARP(La139)Di.IL7.qvalue` <dbl>, `pPLCg2(Pr141)Di.IL7.qvalue` <dbl>,
## #   `pSrc(Nd144)Di.IL7.qvalue` <dbl>, `Ki67(Sm152)Di.IL7.qvalue` <dbl>,
## #   `pErk12(Gd155)Di.IL7.qvalue` <dbl>,
## #   `pSTAT3(Gd158)Di.IL7.qvalue` <dbl>, `pAKT(Tb159)Di.IL7.qvalue` <dbl>,
## #   `pBLNK(Gd160)Di.IL7.qvalue` <dbl>, `pP38(Tm169)Di.IL7.qvalue` <dbl>,
## #   `pSTAT5(Nd150)Di.IL7.qvalue` <dbl>, `pSyk(Dy162)Di.IL7.qvalue` <dbl>,
## #   `tIkBa(Er166)Di.IL7.qvalue` <dbl>
# tSNE map shows highly responsive population of interest
TsneVis(wand.final, 
        "pSTAT5(Nd150)Di.IL7.change", 
        "IL7 -> pSTAT5 change")

# tSNE map now colored by q value
TsneVis(wand.final, 
        "pSTAT5(Nd150)Di.IL7.qvalue", 
        "IL7 -> pSTAT5 -log10(qvalue)")

# tSNE map colored by KNN density estimation
TsneVis(wand.final, "density")

Subsampling your data prior to running t-SNE:

If one has a large number of cells in the dataset (>100K), then t-SNE can become time-consuming and produce results that are less clean. As such, I provide a wrapper that allows one to subsample the final data and run t-SNE on the subsampled data, producing a new tibble that contains the subsampled data along with two t-SNE dimensions added to it. Note the two added dimensions at the end of the tibble are called “bh-SNE11” and “bh-SNE21”. This is because the dimensions “bh-SNE1” and “bh-SNE2” are already in the data, because t-SNE was run during the post processing step in this example. As I have stated, a user would realistically use this function with a much larger number of cells, in which case the user would have selected “tsne = FALSE” in the post.processing function detailed above in this vignette.

wand.final.sub <- SubsampleAndTsne(dat = wand.final, 
                                   input = input.markers, 
                                   numcells = 500)
## Read the 500 x 27 data matrix successfully!
## OpenMP is working. 1 threads.
## Using no_dims = 2, perplexity = 30.000000, and theta = 0.500000
## Computing input similarities...
## Building tree...
## Done in 0.06 seconds (sparsity = 0.240000)!
## Learning embedding...
## Iteration 50: error is 54.815336 (50 iterations in 0.12 seconds)
## Iteration 100: error is 53.741806 (50 iterations in 0.10 seconds)
## Iteration 150: error is 53.568818 (50 iterations in 0.09 seconds)
## Iteration 200: error is 53.488847 (50 iterations in 0.09 seconds)
## Iteration 250: error is 53.503826 (50 iterations in 0.09 seconds)
## Iteration 300: error is 0.931602 (50 iterations in 0.09 seconds)
## Iteration 350: error is 0.875091 (50 iterations in 0.08 seconds)
## Iteration 400: error is 0.858721 (50 iterations in 0.07 seconds)
## Iteration 450: error is 0.851106 (50 iterations in 0.11 seconds)
## Iteration 500: error is 0.847492 (50 iterations in 0.10 seconds)
## Iteration 550: error is 0.843782 (50 iterations in 0.07 seconds)
## Iteration 600: error is 0.840146 (50 iterations in 0.07 seconds)
## Iteration 650: error is 0.837472 (50 iterations in 0.07 seconds)
## Iteration 700: error is 0.836191 (50 iterations in 0.08 seconds)
## Iteration 750: error is 0.835587 (50 iterations in 0.11 seconds)
## Iteration 800: error is 0.833976 (50 iterations in 0.10 seconds)
## Iteration 850: error is 0.832570 (50 iterations in 0.08 seconds)
## Iteration 900: error is 0.832333 (50 iterations in 0.07 seconds)
## Iteration 950: error is 0.832682 (50 iterations in 0.08 seconds)
## Iteration 1000: error is 0.831867 (50 iterations in 0.07 seconds)
## Fitting performed in 1.73 seconds.
wand.final.sub
## # A tibble: 500 x 89
##    `CD3(Cd110)Di` `CD3(Cd111)Di` `CD3(Cd112)Di` `CD235-61-7-15(…
##             <dbl>          <dbl>          <dbl>            <dbl>
##  1         -0.503      0.406             -0.865            0.649
##  2          0.253      0.492             -0.367           -0.737
##  3         -0.887     -0.329             -0.459           -0.982
##  4         -0.207     -0.184             -0.562           -2.39 
##  5          0.862     -0.0213            -0.235            0.232
##  6         -0.115     -0.172             -0.158           -1.99 
##  7         -0.242     -0.443             -0.414           -1.10 
##  8         -0.248     -0.0562            -0.132            0.771
##  9         -0.224     -0.0000293          1.05             0.836
## 10         -0.356      1.72               2.54             0.413
## # … with 490 more rows, and 85 more variables: `CD3(Cd114)Di` <dbl>,
## #   `CD45(In115)Di` <dbl>, `CD19(Nd142)Di` <dbl>, `CD22(Nd143)Di` <dbl>,
## #   `IgD(Nd145)Di` <dbl>, `CD79b(Nd146)Di` <dbl>, `CD20(Sm147)Di` <dbl>,
## #   `CD34(Nd148)Di` <dbl>, `CD179a(Sm149)Di` <dbl>, `CD72(Eu151)Di` <dbl>,
## #   `IgM(Eu153)Di` <dbl>, `Kappa(Sm154)Di` <dbl>, `CD10(Gd156)Di` <dbl>,
## #   `Lambda(Gd157)Di` <dbl>, `CD24(Dy161)Di` <dbl>, `TdT(Dy163)Di` <dbl>,
## #   `Rag1(Dy164)Di` <dbl>, `PreBCR(Ho165)Di` <dbl>, `CD43(Er167)Di` <dbl>,
## #   `CD38(Er168)Di` <dbl>, `CD40(Er170)Di` <dbl>, `CD33(Yb173)Di` <dbl>,
## #   `HLA-DR(Yb174)Di` <dbl>, Time <dbl>, Cell_length <dbl>,
## #   `cPARP(La139)Di` <dbl>, `pPLCg2(Pr141)Di` <dbl>,
## #   `pSrc(Nd144)Di` <dbl>, `pSTAT5(Nd150)Di` <dbl>, `Ki67(Sm152)Di` <dbl>,
## #   `pErk12(Gd155)Di` <dbl>, `pSTAT3(Gd158)Di` <dbl>,
## #   `pAKT(Tb159)Di` <dbl>, `pBLNK(Gd160)Di` <dbl>, `pSyk(Dy162)Di` <dbl>,
## #   `tIkBa(Er166)Di` <dbl>, `pP38(Tm169)Di` <dbl>, `pBTK(Yb171)Di` <dbl>,
## #   `pS6(Yb172)Di` <dbl>, `pCrkL(Lu175)Di` <dbl>, `pCREB(Yb176)Di` <dbl>,
## #   `DNA1(Ir191)Di` <dbl>, `DNA2(Ir193)Di` <dbl>,
## #   `Viability1(Pt195)Di` <dbl>, `Viability2(Pt196)Di` <dbl>,
## #   wanderlust <dbl>, condition <dbl>, `pCrkL(Lu175)Di.IL7.change` <dbl>,
## #   `pCREB(Yb176)Di.IL7.change` <dbl>, `pBTK(Yb171)Di.IL7.change` <dbl>,
## #   `pS6(Yb172)Di.IL7.change` <dbl>, `cPARP(La139)Di.IL7.change` <dbl>,
## #   `pPLCg2(Pr141)Di.IL7.change` <dbl>, `pSrc(Nd144)Di.IL7.change` <dbl>,
## #   `Ki67(Sm152)Di.IL7.change` <dbl>, `pErk12(Gd155)Di.IL7.change` <dbl>,
## #   `pSTAT3(Gd158)Di.IL7.change` <dbl>, `pAKT(Tb159)Di.IL7.change` <dbl>,
## #   `pBLNK(Gd160)Di.IL7.change` <dbl>, `pP38(Tm169)Di.IL7.change` <dbl>,
## #   `pSTAT5(Nd150)Di.IL7.change` <dbl>, `pSyk(Dy162)Di.IL7.change` <dbl>,
## #   `tIkBa(Er166)Di.IL7.change` <dbl>, IL7.fraction.cond.2 <dbl>,
## #   density <dbl>, `bh-SNE1` <dbl>, `bh-SNE2` <dbl>,
## #   `pCrkL(Lu175)Di.IL7.qvalue` <dbl>, `pCREB(Yb176)Di.IL7.qvalue` <dbl>,
## #   `pBTK(Yb171)Di.IL7.qvalue` <dbl>, `pS6(Yb172)Di.IL7.qvalue` <dbl>,
## #   `cPARP(La139)Di.IL7.qvalue` <dbl>, `pPLCg2(Pr141)Di.IL7.qvalue` <dbl>,
## #   `pSrc(Nd144)Di.IL7.qvalue` <dbl>, `Ki67(Sm152)Di.IL7.qvalue` <dbl>,
## #   `pErk12(Gd155)Di.IL7.qvalue` <dbl>,
## #   `pSTAT3(Gd158)Di.IL7.qvalue` <dbl>, `pAKT(Tb159)Di.IL7.qvalue` <dbl>,
## #   `pBLNK(Gd160)Di.IL7.qvalue` <dbl>, `pP38(Tm169)Di.IL7.qvalue` <dbl>,
## #   `pSTAT5(Nd150)Di.IL7.qvalue` <dbl>, `pSyk(Dy162)Di.IL7.qvalue` <dbl>,
## #   `tIkBa(Er166)Di.IL7.qvalue` <dbl>, `bh-SNE11` <dbl>, `bh-SNE21` <dbl>