IE 582 - Final
ilaydacelenk
06/02/2021
1 Introduction
Normally, models learn from single instances, on the other hand, in Multiple Instance Learning (MIL) the aim is to classify bags of instances. The aim of this Exam is to come up with two alternative bag-level representation approaches in MIL for a given data. The original data set that will be used can be found here and the updated version with bag classes and bag Id’s can be found here. This data is created from 92 molecules of which 47 are musks and rest are not. Ultimate goal is to find a model to predict whether a molecule is musk or non-musk. After some manipulations, the data we have is created.
I will be using the bag-level representations that I have suggested in my supervised training models in order to be able to make predictions on the bag classes. Afterwards, I will compare the representations along with the models using area under the ROC curve (AUC) based on 10-fold cross validation.
Since the task is to find a set of representative instances based on bags, the approach is called instance-based approach.
Libraries and dataset are imported.
library(skimr)
library(dplyr)
library(data.table)
library(caret)
library(glmnet)
library(cvAUC)
library(cluster)
library(factoextra)
library(TunePareto)
library(rpart)
musk_data <- read.csv("https://raw.githubusercontent.com/BU-IE-582/fall20-ilaydacelenk/master/files/Musk1.csv", header=FALSE)Characteristics of the Musk dataset are given as follows:
- Binary Classification
- 166 Features
- First Column is the Bag Class: Labeled V1
- Second Column is the Bag ID: Labeled V2
- All Numerical Features: From V3 to V168
- Bag Classes: 0(negative), 1(positive)
Using skim function, I observe that there is no NA values in this dataset and all attributes are numeric.
In MIL, bags are labeled as 0 if all the instances in this bag are 0 and bags are labeled as 1 if at least one of the instances in this bag is 1. For example, Bag 1 has 4 instances and we know that at least one of them is 1 and at most 3 of them are 0 in reality. Bag 50 is labeled as 0 and it has 2 instances, which means they are both 0 in reality.
Since I will be using distance, first I will scale the data.
skim(musk_data)| Name | musk_data |
| Number of rows | 476 |
| Number of columns | 168 |
| _______________________ | |
| Column type frequency: | |
| numeric | 168 |
| ________________________ | |
| Group variables | None |
Variable type: numeric
| skim_variable | n_missing | complete_rate | mean | sd | p0 | p25 | p50 | p75 | p100 | hist |
|---|---|---|---|---|---|---|---|---|---|---|
| V1 | 0 | 1 | 0.43 | 0.50 | 0 | 0.00 | 0.0 | 1.00 | 1 | ▇▁▁▁▆ |
| V2 | 0 | 1 | 52.39 | 26.91 | 1 | 29.00 | 56.5 | 81.00 | 92 | ▃▅▅▅▇ |
| V3 | 0 | 1 | 38.73 | 18.09 | -9 | 31.00 | 42.0 | 50.00 | 130 | ▂▇▅▁▁ |
| V4 | 0 | 1 | -120.14 | 88.31 | -199 | -193.00 | -144.5 | -101.00 | 98 | ▇▅▁▁▂ |
| V5 | 0 | 1 | -79.24 | 69.17 | -166 | -144.00 | -108.0 | -21.00 | 83 | ▇▂▅▂▁ |
| V6 | 0 | 1 | 15.90 | 75.52 | -115 | -53.00 | 28.0 | 38.00 | 157 | ▅▃▇▁▅ |
| V7 | 0 | 1 | -112.30 | 28.70 | -117 | -117.00 | -117.0 | -117.00 | 238 | ▇▁▁▁▁ |
| V8 | 0 | 1 | -9.76 | 92.03 | -184 | -89.00 | 11.0 | 70.25 | 200 | ▃▆▅▇▁ |
| V9 | 0 | 1 | -16.63 | 106.56 | -170 | -159.00 | 41.0 | 51.00 | 214 | ▆▁▇▃▁ |
| V10 | 0 | 1 | 7.76 | 75.79 | -231 | -9.25 | 18.0 | 57.00 | 188 | ▁▂▆▇▁ |
| V11 | 0 | 1 | -91.05 | 96.32 | -242 | -215.00 | -41.0 | -30.00 | 135 | ▅▁▇▁▁ |
| V12 | 0 | 1 | -76.08 | 72.47 | -284 | -118.25 | -73.5 | -28.00 | 218 | ▁▆▇▁▁ |
| V13 | 0 | 1 | -94.34 | 89.69 | -327 | -112.00 | -57.0 | -46.00 | 104 | ▂▁▃▇▁ |
| V14 | 0 | 1 | -70.08 | 82.95 | -326 | -104.00 | -69.5 | -29.00 | 196 | ▁▂▇▁▁ |
| V15 | 0 | 1 | -71.63 | 66.27 | -302 | -86.00 | -62.0 | -42.00 | 117 | ▁▁▇▅▁ |
| V16 | 0 | 1 | -114.68 | 83.23 | -348 | -125.25 | -97.0 | -82.00 | 72 | ▂▁▇▇▁ |
| V17 | 0 | 1 | -106.64 | 76.27 | -292 | -109.00 | -94.0 | -77.00 | 77 | ▂▁▇▁▁ |
| V18 | 0 | 1 | -265.09 | 63.87 | -318 | -296.25 | -285.5 | -266.75 | 13 | ▇▁▁▁▁ |
| V19 | 0 | 1 | -41.32 | 115.72 | -225 | -168.25 | 28.0 | 50.00 | 125 | ▅▂▁▇▃ |
| V20 | 0 | 1 | -71.46 | 65.40 | -312 | -88.00 | -69.5 | -46.75 | 139 | ▁▁▇▂▁ |
| V21 | 0 | 1 | -38.69 | 81.63 | -291 | -73.25 | -31.0 | 5.25 | 214 | ▁▂▇▁▁ |
| V22 | 0 | 1 | -48.36 | 69.19 | -243 | -61.00 | -39.0 | -18.00 | 273 | ▂▇▇▁▁ |
| V23 | 0 | 1 | -73.22 | 77.84 | -298 | -106.25 | -66.5 | -40.75 | 196 | ▁▃▇▂▁ |
| V24 | 0 | 1 | -63.57 | 87.53 | -187 | -179.00 | -22.0 | -14.00 | 147 | ▅▁▇▁▁ |
| V25 | 0 | 1 | -57.50 | 93.54 | -255 | -143.00 | -36.5 | 13.25 | 187 | ▅▃▇▆▁ |
| V26 | 0 | 1 | 33.93 | 117.35 | -70 | -61.00 | -44.0 | 154.75 | 312 | ▇▁▂▂▁ |
| V27 | 0 | 1 | -18.29 | 93.78 | -102 | -97.00 | -88.0 | 77.00 | 254 | ▇▁▅▁▁ |
| V28 | 0 | 1 | -73.64 | 89.42 | -243 | -173.00 | -38.0 | -6.75 | 162 | ▅▁▇▃▁ |
| V29 | 0 | 1 | -15.57 | 92.36 | -205 | -114.25 | 27.0 | 53.00 | 157 | ▃▁▂▇▁ |
| V30 | 0 | 1 | -98.58 | 63.85 | -164 | -150.25 | -117.5 | -63.00 | 147 | ▇▅▁▁▁ |
| V31 | 0 | 1 | 2.61 | 58.08 | -140 | -31.25 | 7.0 | 25.00 | 181 | ▁▃▇▂▁ |
| V32 | 0 | 1 | -40.53 | 118.03 | -154 | -142.00 | -127.0 | 91.00 | 258 | ▇▁▂▃▁ |
| V33 | 0 | 1 | -100.03 | 46.68 | -118 | -117.00 | -117.0 | -115.75 | 190 | ▇▁▁▁▁ |
| V34 | 0 | 1 | -12.79 | 93.92 | -149 | -101.00 | -38.0 | 97.75 | 143 | ▇▅▅▂▆ |
| V35 | 0 | 1 | -8.98 | 62.50 | -124 | -39.00 | -20.0 | 17.25 | 219 | ▂▇▂▂▁ |
| V36 | 0 | 1 | -127.89 | 111.20 | -278 | -224.00 | -182.0 | -5.75 | 163 | ▇▂▂▅▁ |
| V37 | 0 | 1 | -47.59 | 79.61 | -156 | -129.25 | -35.0 | -0.75 | 227 | ▇▇▃▂▁ |
| V38 | 0 | 1 | 78.96 | 40.88 | 16 | 47.00 | 73.0 | 95.00 | 291 | ▇▆▃▁▁ |
| V39 | 0 | 1 | -109.09 | 114.47 | -175 | -174.00 | -168.0 | -153.00 | 224 | ▇▁▁▂▁ |
| V40 | 0 | 1 | -31.93 | 81.51 | -192 | -110.25 | -1.5 | 25.00 | 165 | ▅▂▇▅▁ |
| V41 | 0 | 1 | -75.96 | 72.05 | -145 | -142.00 | -110.5 | -23.00 | 105 | ▇▁▅▁▁ |
| V42 | 0 | 1 | -6.88 | 80.87 | -181 | -72.25 | 20.0 | 48.00 | 189 | ▅▂▇▅▁ |
| V43 | 0 | 1 | -95.76 | 84.91 | -190 | -174.00 | -124.5 | -49.00 | 82 | ▇▂▅▁▂ |
| V44 | 0 | 1 | 8.72 | 77.35 | -151 | -43.00 | 17.0 | 75.00 | 201 | ▅▆▇▇▁ |
| V45 | 0 | 1 | -67.97 | 64.26 | -291 | -80.50 | -58.0 | -34.00 | 168 | ▁▁▇▁▁ |
| V46 | 0 | 1 | -101.63 | 88.89 | -343 | -99.25 | -75.0 | -64.00 | 81 | ▂▁▂▇▁ |
| V47 | 0 | 1 | -108.74 | 79.52 | -315 | -125.00 | -99.0 | -82.00 | 65 | ▂▁▇▃▂ |
| V48 | 0 | 1 | -148.01 | 88.95 | -338 | -183.00 | -121.5 | -101.00 | 115 | ▂▂▇▁▁ |
| V49 | 0 | 1 | -89.50 | 67.12 | -157 | -147.00 | -103.0 | -83.00 | 110 | ▇▅▁▂▁ |
| V50 | 0 | 1 | -93.91 | 71.16 | -294 | -123.25 | -75.0 | -55.75 | 99 | ▁▂▇▃▁ |
| V51 | 0 | 1 | -18.62 | 74.30 | -269 | -44.00 | -13.0 | 25.00 | 180 | ▁▂▇▇▁ |
| V52 | 0 | 1 | -75.82 | 66.36 | -273 | -106.00 | -86.0 | -32.00 | 214 | ▁▇▆▁▁ |
| V53 | 0 | 1 | -81.79 | 77.61 | -329 | -111.00 | -87.0 | -65.00 | 218 | ▁▃▇▂▁ |
| V54 | 0 | 1 | -73.46 | 90.94 | -205 | -196.00 | -30.0 | -18.00 | 148 | ▅▁▇▁▁ |
| V55 | 0 | 1 | -59.53 | 94.16 | -206 | -178.00 | -20.0 | -1.00 | 159 | ▅▁▇▂▁ |
| V56 | 0 | 1 | 29.28 | 82.82 | -143 | -8.50 | 51.0 | 72.00 | 286 | ▂▃▇▁▁ |
| V57 | 0 | 1 | 34.06 | 87.81 | -113 | -50.00 | 20.0 | 112.00 | 259 | ▇▃▅▆▁ |
| V58 | 0 | 1 | -67.60 | 62.45 | -282 | -82.25 | -58.0 | -40.00 | 199 | ▁▂▇▁▁ |
| V59 | 0 | 1 | 5.85 | 111.39 | -168 | -125.00 | 56.5 | 91.25 | 231 | ▅▂▃▇▁ |
| V60 | 0 | 1 | -108.29 | 79.46 | -178 | -155.25 | -139.0 | -113.75 | 111 | ▇▁▁▁▁ |
| V61 | 0 | 1 | 9.06 | 66.48 | -173 | -20.00 | 5.0 | 49.00 | 181 | ▁▂▇▃▁ |
| V62 | 0 | 1 | -82.60 | 91.79 | -254 | -162.00 | -112.5 | 7.25 | 178 | ▅▇▃▆▁ |
| V63 | 0 | 1 | -8.09 | 99.33 | -102 | -98.00 | -22.5 | 46.25 | 243 | ▇▅▁▂▂ |
| V64 | 0 | 1 | -82.59 | 110.75 | -195 | -192.00 | -130.0 | 51.00 | 173 | ▇▁▁▅▁ |
| V65 | 0 | 1 | 10.91 | 27.87 | -69 | -13.25 | 20.0 | 30.00 | 110 | ▁▃▇▁▁ |
| V66 | 0 | 1 | -63.91 | 132.99 | -197 | -188.00 | -131.5 | 95.25 | 220 | ▇▂▁▅▁ |
| V67 | 0 | 1 | -37.55 | 81.61 | -159 | -121.00 | -17.0 | -7.00 | 108 | ▆▁▇▁▃ |
| V68 | 0 | 1 | 53.99 | 22.02 | 6 | 42.00 | 57.0 | 63.00 | 197 | ▃▇▁▁▁ |
| V69 | 0 | 1 | -161.49 | 24.12 | -166 | -165.00 | -165.0 | -164.00 | 262 | ▇▁▁▁▁ |
| V70 | 0 | 1 | -40.79 | 58.05 | -196 | -68.25 | -41.0 | -16.75 | 157 | ▁▅▇▂▁ |
| V71 | 0 | 1 | -32.36 | 97.49 | -134 | -133.00 | -33.0 | 64.75 | 205 | ▇▂▃▃▁ |
| V72 | 0 | 1 | -28.59 | 62.19 | -192 | -61.00 | -35.0 | 2.25 | 168 | ▁▅▇▂▁ |
| V73 | 0 | 1 | -86.52 | 77.74 | -174 | -158.00 | -121.0 | -33.75 | 83 | ▇▁▅▁▂ |
| V74 | 0 | 1 | -82.99 | 65.27 | -150 | -146.00 | -121.0 | -35.00 | 98 | ▇▁▅▁▁ |
| V75 | 0 | 1 | -85.70 | 80.03 | -322 | -104.00 | -68.0 | -43.00 | 133 | ▂▂▇▆▁ |
| V76 | 0 | 1 | -92.66 | 81.88 | -338 | -116.25 | -83.0 | -60.00 | 112 | ▁▂▇▅▂ |
| V77 | 0 | 1 | -73.35 | 71.37 | -272 | -92.00 | -63.5 | -40.00 | 132 | ▁▁▇▂▁ |
| V78 | 0 | 1 | -176.83 | 11.96 | -191 | -182.00 | -180.0 | -177.00 | -65 | ▇▁▁▁▁ |
| V79 | 0 | 1 | -81.88 | 113.77 | -255 | -235.25 | -15.0 | 0.00 | 154 | ▆▁▃▇▁ |
| V80 | 0 | 1 | -77.77 | 85.29 | -318 | -93.00 | -56.0 | -30.00 | 158 | ▂▁▇▃▁ |
| V81 | 0 | 1 | -56.39 | 84.39 | -310 | -93.00 | -51.0 | -13.00 | 185 | ▁▁▇▂▁ |
| V82 | 0 | 1 | 16.95 | 82.93 | -197 | 5.75 | 47.0 | 57.00 | 233 | ▂▁▇▂▁ |
| V83 | 0 | 1 | -74.42 | 87.37 | -258 | -143.25 | -53.5 | -10.00 | 183 | ▃▂▇▂▁ |
| V84 | 0 | 1 | -42.08 | 94.63 | -176 | -167.00 | 2.0 | 16.25 | 192 | ▅▁▇▁▁ |
| V85 | 0 | 1 | -62.78 | 79.41 | -301 | -114.00 | -58.0 | -45.00 | 208 | ▁▅▇▂▁ |
| V86 | 0 | 1 | 51.76 | 97.50 | -98 | -37.00 | 63.5 | 137.00 | 251 | ▇▆▃▇▃ |
| V87 | 0 | 1 | -89.85 | 98.66 | -222 | -174.00 | -146.0 | 14.00 | 183 | ▇▂▂▃▁ |
| V88 | 0 | 1 | -43.11 | 105.51 | -200 | -184.00 | 11.5 | 28.00 | 189 | ▅▁▇▁▁ |
| V89 | 0 | 1 | -58.93 | 79.26 | -208 | -106.00 | -74.0 | -41.25 | 113 | ▂▇▇▁▃ |
| V90 | 0 | 1 | -99.08 | 68.77 | -213 | -149.00 | -112.0 | -56.00 | 105 | ▅▇▅▁▂ |
| V91 | 0 | 1 | 5.33 | 59.84 | -111 | -46.00 | 16.0 | 30.00 | 163 | ▅▃▇▂▁ |
| V92 | 0 | 1 | -23.07 | 127.82 | -159 | -146.00 | -96.5 | 123.00 | 272 | ▇▁▂▃▁ |
| V93 | 0 | 1 | -148.55 | 85.96 | -202 | -202.00 | -198.0 | -114.75 | 168 | ▇▁▁▁▁ |
| V94 | 0 | 1 | 44.68 | 46.44 | -20 | 11.00 | 28.0 | 95.00 | 167 | ▇▇▂▃▂ |
| V95 | 0 | 1 | -14.30 | 84.12 | -121 | -107.00 | 5.0 | 19.00 | 233 | ▇▇▂▃▁ |
| V96 | 0 | 1 | -122.87 | 75.32 | -326 | -135.00 | -113.0 | -92.00 | 178 | ▂▃▇▁▁ |
| V97 | 0 | 1 | 8.43 | 32.61 | -77 | -9.00 | 4.0 | 28.00 | 149 | ▂▇▆▁▁ |
| V98 | 0 | 1 | 34.22 | 99.27 | -70 | -47.00 | -25.0 | 118.00 | 336 | ▇▂▂▂▁ |
| V99 | 0 | 1 | -46.54 | 97.89 | -191 | -108.00 | -67.0 | 3.00 | 171 | ▆▇▆▂▃ |
| V100 | 0 | 1 | -41.11 | 101.69 | -191 | -177.00 | 9.0 | 35.00 | 194 | ▅▁▇▂▁ |
| V101 | 0 | 1 | -92.30 | 64.82 | -155 | -153.00 | -141.0 | -26.00 | 48 | ▇▁▂▃▁ |
| V102 | 0 | 1 | 2.51 | 103.76 | -155 | -124.00 | 40.0 | 80.00 | 235 | ▆▁▇▅▁ |
| V103 | 0 | 1 | -109.04 | 91.36 | -210 | -194.00 | -120.0 | -70.00 | 88 | ▇▃▅▁▂ |
| V104 | 0 | 1 | 76.74 | 47.27 | -37 | 33.00 | 73.0 | 129.00 | 265 | ▃▇▇▁▁ |
| V105 | 0 | 1 | -88.44 | 78.21 | -296 | -103.00 | -66.0 | -47.75 | 96 | ▂▁▇▇▁ |
| V106 | 0 | 1 | -98.32 | 85.91 | -320 | -112.00 | -68.5 | -57.00 | 95 | ▂▁▅▇▁ |
| V107 | 0 | 1 | -118.16 | 80.59 | -319 | -117.00 | -104.5 | -91.00 | 72 | ▂▁▇▂▁ |
| V108 | 0 | 1 | -95.77 | 80.20 | -287 | -135.00 | -92.0 | -44.00 | 179 | ▂▅▇▂▁ |
| V109 | 0 | 1 | -134.89 | 61.07 | -194 | -174.00 | -162.0 | -111.50 | 144 | ▇▂▁▁▁ |
| V110 | 0 | 1 | -110.94 | 75.29 | -293 | -158.00 | -92.0 | -56.00 | 52 | ▂▃▇▇▂ |
| V111 | 0 | 1 | -68.86 | 77.01 | -250 | -77.00 | -66.0 | -18.00 | 183 | ▂▂▇▁▁ |
| V112 | 0 | 1 | -112.81 | 74.54 | -295 | -126.00 | -107.0 | -91.00 | 53 | ▂▁▇▂▁ |
| V113 | 0 | 1 | -70.03 | 73.04 | -252 | -69.00 | -61.0 | -47.00 | 117 | ▂▁▇▁▁ |
| V114 | 0 | 1 | -54.40 | 88.91 | -268 | -120.25 | -43.0 | 0.00 | 211 | ▂▃▇▁▁ |
| V115 | 0 | 1 | -62.78 | 97.98 | -223 | -179.50 | -33.0 | 16.00 | 188 | ▆▃▇▃▁ |
| V116 | 0 | 1 | -58.64 | 96.21 | -206 | -181.25 | -24.5 | 17.25 | 160 | ▇▂▇▆▁ |
| V117 | 0 | 1 | -83.79 | 97.87 | -251 | -166.00 | -129.0 | 19.25 | 201 | ▇▇▃▆▁ |
| V118 | 0 | 1 | -36.25 | 64.91 | -252 | -63.25 | -24.0 | -2.00 | 187 | ▁▂▇▂▁ |
| V119 | 0 | 1 | 11.03 | 120.44 | -104 | -91.00 | -81.0 | 154.00 | 315 | ▇▁▁▃▁ |
| V120 | 0 | 1 | -36.54 | 106.23 | -209 | -151.75 | 7.5 | 53.00 | 137 | ▅▂▂▇▁ |
| V121 | 0 | 1 | -7.79 | 118.30 | -193 | -153.25 | 66.0 | 81.00 | 215 | ▅▁▁▇▁ |
| V122 | 0 | 1 | -38.07 | 94.64 | -201 | -150.00 | 1.0 | 30.00 | 152 | ▆▁▆▇▁ |
| V123 | 0 | 1 | -29.36 | 102.34 | -123 | -114.00 | -107.0 | 78.00 | 254 | ▇▁▅▁▁ |
| V124 | 0 | 1 | 16.32 | 87.81 | -118 | -67.00 | -2.0 | 91.25 | 270 | ▇▂▆▃▁ |
| V125 | 0 | 1 | 3.35 | 105.84 | -131 | -101.00 | -28.5 | 110.00 | 279 | ▇▁▃▃▁ |
| V126 | 0 | 1 | -2.33 | 72.35 | -121 | -69.00 | 9.0 | 49.00 | 200 | ▇▃▇▃▁ |
| V127 | 0 | 1 | -49.01 | 97.50 | -145 | -131.00 | -99.0 | 51.00 | 149 | ▇▂▁▂▂ |
| V128 | 0 | 1 | 25.63 | 34.51 | -44 | -4.00 | 17.5 | 55.00 | 137 | ▃▇▆▂▁ |
| V129 | 0 | 1 | -84.80 | 78.24 | -285 | -142.25 | -88.0 | -22.00 | 123 | ▁▇▇▇▁ |
| V130 | 0 | 1 | -73.85 | 124.04 | -219 | -183.00 | -146.0 | 65.25 | 238 | ▇▁▂▃▁ |
| V131 | 0 | 1 | -93.42 | 71.70 | -305 | -123.00 | -82.0 | -51.00 | 159 | ▁▃▇▂▁ |
| V132 | 0 | 1 | -68.17 | 96.86 | -188 | -185.00 | -49.0 | -16.00 | 152 | ▇▂▇▂▂ |
| V133 | 0 | 1 | 11.94 | 52.80 | -94 | -20.00 | 23.0 | 40.00 | 140 | ▃▃▇▂▂ |
| V134 | 0 | 1 | -50.21 | 87.18 | -140 | -120.00 | -103.0 | 16.00 | 170 | ▇▁▂▂▁ |
| V135 | 0 | 1 | -82.76 | 65.72 | -322 | -97.00 | -83.0 | -61.00 | 176 | ▁▁▇▁▁ |
| V136 | 0 | 1 | -86.15 | 75.56 | -341 | -120.00 | -86.0 | -51.00 | 187 | ▁▁▇▂▁ |
| V137 | 0 | 1 | -89.71 | 78.76 | -342 | -112.00 | -84.0 | -42.00 | 111 | ▁▂▇▇▁ |
| V138 | 0 | 1 | -45.76 | 62.47 | -197 | -91.00 | -36.0 | -23.00 | 137 | ▂▅▇▂▁ |
| V139 | 0 | 1 | -19.79 | 98.28 | -201 | -126.00 | 20.0 | 52.00 | 186 | ▅▁▃▇▁ |
| V140 | 0 | 1 | -39.03 | 77.07 | -200 | -91.00 | -43.5 | -19.00 | 127 | ▁▅▇▁▃ |
| V141 | 0 | 1 | -70.73 | 86.65 | -246 | -141.25 | -45.0 | -7.00 | 138 | ▅▃▇▆▁ |
| V142 | 0 | 1 | -54.39 | 65.37 | -286 | -69.00 | -51.5 | -34.00 | 207 | ▁▂▇▂▁ |
| V143 | 0 | 1 | -56.02 | 75.96 | -294 | -91.00 | -55.0 | -39.00 | 238 | ▁▃▇▂▁ |
| V144 | 0 | 1 | -24.26 | 77.38 | -186 | -90.00 | -0.5 | 27.25 | 177 | ▃▂▇▃▁ |
| V145 | 0 | 1 | -14.14 | 97.59 | -158 | -142.25 | 29.0 | 58.00 | 213 | ▆▁▇▂▁ |
| V146 | 0 | 1 | -37.47 | 93.14 | -173 | -164.00 | 11.0 | 23.00 | 158 | ▆▁▇▃▁ |
| V147 | 0 | 1 | -174.16 | 24.83 | -179 | -178.00 | -178.0 | -177.00 | 178 | ▇▁▁▁▁ |
| V148 | 0 | 1 | -83.62 | 62.23 | -107 | -103.00 | -102.0 | -101.00 | 240 | ▇▁▁▁▁ |
| V149 | 0 | 1 | -113.95 | 13.96 | -131 | -121.00 | -118.0 | -115.00 | -3 | ▇▁▁▁▁ |
| V150 | 0 | 1 | -32.79 | 67.24 | -201 | -75.25 | -33.0 | -3.00 | 134 | ▁▅▇▂▂ |
| V151 | 0 | 1 | -46.04 | 74.16 | -215 | -93.00 | -54.0 | -27.00 | 121 | ▁▅▇▁▂ |
| V152 | 0 | 1 | -3.24 | 86.07 | -191 | -77.50 | 34.0 | 58.00 | 174 | ▃▂▃▇▁ |
| V153 | 0 | 1 | -24.10 | 70.17 | -126 | -91.00 | -24.0 | 30.00 | 243 | ▇▇▃▂▁ |
| V154 | 0 | 1 | -37.48 | 69.01 | -125 | -120.00 | -16.0 | 3.00 | 154 | ▇▃▇▃▁ |
| V155 | 0 | 1 | -29.83 | 66.57 | -112 | -110.00 | -1.5 | 13.00 | 207 | ▇▇▅▁▁ |
| V156 | 0 | 1 | -83.16 | 71.55 | -171 | -166.00 | -58.0 | -46.00 | 84 | ▇▁▇▁▂ |
| V157 | 0 | 1 | 16.22 | 78.50 | -145 | -50.25 | 3.0 | 104.50 | 161 | ▂▇▆▂▇ |
| V158 | 0 | 1 | -78.94 | 94.27 | -197 | -188.00 | -83.0 | 23.00 | 113 | ▇▅▂▇▁ |
| V159 | 0 | 1 | -221.73 | 63.02 | -252 | -240.00 | -236.0 | -232.00 | 141 | ▇▁▁▁▁ |
| V160 | 0 | 1 | -204.41 | 89.57 | -324 | -264.00 | -236.0 | -154.75 | 72 | ▇▅▂▂▁ |
| V161 | 0 | 1 | -108.26 | 121.79 | -217 | -208.00 | -189.5 | 30.00 | 173 | ▇▁▁▃▁ |
| V162 | 0 | 1 | -25.23 | 65.59 | -135 | -91.00 | -15.0 | 22.00 | 185 | ▆▇▇▁▁ |
| V163 | 0 | 1 | 37.43 | 87.32 | -126 | -39.25 | 31.5 | 128.00 | 253 | ▅▇▆▇▁ |
| V164 | 0 | 1 | 79.75 | 49.27 | -78 | 33.00 | 84.0 | 119.00 | 291 | ▁▅▇▁▁ |
| V165 | 0 | 1 | 85.36 | 47.49 | 35 | 54.00 | 69.0 | 99.00 | 302 | ▇▂▁▁▁ |
| V166 | 0 | 1 | -34.13 | 16.02 | -132 | -42.25 | -36.0 | -28.00 | 24 | ▁▁▃▇▁ |
| V167 | 0 | 1 | -26.24 | 58.01 | -258 | -28.00 | -11.0 | 7.00 | 82 | ▁▁▁▇▁ |
| V168 | 0 | 1 | 33.71 | 54.19 | -72 | -12.00 | 35.0 | 74.00 | 235 | ▆▇▇▁▁ |
#scale
data <- musk_data
data[,3:168] <- scale(data[,3:168])
euclidean <- data %>% dist("euclidean") %>% as.matrix() %>% as.data.table()
manhattan <- data %>% dist("manhattan") %>% as.matrix() %>% as.data.table()I will be clustering the instances using K-medoids algorithm. This means number of clusters will be a parameter to tune. Optimum number of clusters can be found by factoextra package.
Optimal number of clusters is found as 5. Then I will try to find the medoids using euclidean distance with 3 clusters then calculate the distances to medoids. The aim is to find the average distance between instances and centroids.
xdata <- data[,3:168]
fviz_nbclust(xdata, FUN = kmeans, method="wss")
1.1 Euclidean Distance
First, I will consider Euclidean distance.
medoid_euc <- pam(xdata, k=5, diss=FALSE, metric='euclidean')
medoids_id<-medoid_euc$id.med
med_distances <- c()
for(i in 1:5){
med_distances<-cbind(med_distances,euclidean[,get(as.character(as.numeric(medoids_id[i])))])
}
med_distances <- med_distances %>% as.data.table(med_distances)
setnames(med_distances, old=colnames(med_distances), new=c("C1","C2","C3","C4","C5")) #5 clusters
med_bag <- cbind(musk_data[,1:2], med_distances) %>% as.data.table()
bag_average <- med_bag %>% aggregate(., list(med_bag$V2), mean) %>% select(-Group.1) # now the columns represent the average distances to the center of the clusters
print(head(bag_average))FALSE V1 V2 C1 C2 C3 C4 C5
FALSE 1 1 1 8.725194 27.60158 23.03847 25.40421 30.73681
FALSE 2 1 2 9.883008 26.48365 21.60155 24.52460 30.53410
FALSE 3 1 3 9.772072 25.89624 21.96623 24.00592 30.13954
FALSE 4 1 4 5.867434 25.32948 21.61557 23.64801 28.88497
FALSE 5 1 5 15.435006 27.03491 23.47088 25.49654 31.02967
FALSE 6 1 6 10.277841 24.20578 20.36060 23.17971 28.193631.2 Manhattan Distance
My second approach is to consider Manhattan distance.
medoid_man <- pam(xdata, k=5, diss=FALSE, metric='manhattan')
medoids_id<-medoid_euc$id.med
med_distances <- c()
for(i in 1:5){
med_distances<-cbind(med_distances,manhattan[,get(as.character(as.numeric(medoids_id[i])))])
}
med_distances <- med_distances %>% as.data.table(med_distances)
setnames(med_distances, old=colnames(med_distances), new=c("C1","C2","C3","C4","C5")) #5 clusters
med_bag <- cbind(musk_data[,1:2], med_distances) %>% as.data.table()
bag_average2 <- med_bag %>% aggregate(., list(med_bag$V2), mean) %>% select(-Group.1) # now the columns represent the average distances to the center of the clusters
print(head(bag_average2))FALSE V1 V2 C1 C2 C3 C4 C5
FALSE 1 1 1 68.33971 189.1221 209.5723 206.0008 179.8548
FALSE 2 1 2 81.18382 180.9932 195.7443 202.4480 193.1192
FALSE 3 1 3 84.22995 178.0037 207.7646 201.4983 202.0210
FALSE 4 1 4 47.90798 183.1800 206.7360 202.9134 197.1458
FALSE 5 1 5 103.21577 195.9836 216.6387 221.1038 217.7994
FALSE 6 1 6 82.56845 185.2568 198.7261 216.1075 206.75742 Analysis
2.1 Euclidean Distance
Decision Tree model with parameter tuning and 10-Fold Cross Validation is used to make predictions. The AUC is 1 and accuracy is also 1.
## Decision Tree MODEL
trainn <- bag_average %>% select(-V2)
X_train <- bag_average %>% select(-V1)
y_train <- bag_average%>% select(V1) %>% as.factor()
set.seed(1)
complexity=c(0.01,0.03,0.05)
min_number_of_observations=c(10,15,25)
cv_summary=data.table()
#for balanced folds: stratified =TRUE
index <- generateCVRuns(trainn$V1, ntimes=5, nfold=10, stratified =TRUE)
acc <- 0
best_complexity <- 0
best_min_number_of_observations <- 0
for(i in 1:10){
for(c in complexity){
for(m in min_number_of_observations){
test_index <- index$`Run 1`[i]
train_cv <- trainn[-test_index[[1]],]
test_cv <- trainn[test_index[[1]],]
fitControl <- rpart.control(cp = c, minbucket = m)
model_DT <- rpart(factor(V1)~., trainn, control=fitControl)
pred <- predict(model_DT, test_cv, type="class")
cm <- confusionMatrix(pred, factor(test_cv$V1))
acc_temp <- cm$overall[1:1]
auc_temp <- AUC(as.numeric(pred),as.numeric(test_cv$V1))
cv_summary <- rbind(cv_summary,data.table(Fold=i, Complexity=c, Min_Observations_per_Leaf=m, Accuracy=acc_temp, AUC=auc_temp))
if(acc_temp>acc){
acc<-acc_temp
best_complexity <- c
best_min_number_of_observations <- m
auc<-auc_temp
}
}
}
}
cv_summary <- cv_summary %>% group_by(Complexity, Min_Observations_per_Leaf) %>% summarise(mean_accuracy = mean(Accuracy), mean_AUC=mean(AUC))
print(cv_summary)FALSE # A tibble: 9 x 4
FALSE # Groups: Complexity [3]
FALSE Complexity Min_Observations_per_Leaf mean_accuracy mean_AUC
FALSE <dbl> <dbl> <dbl> <dbl>
FALSE 1 0.01 10 1 1
FALSE 2 0.01 15 1 1
FALSE 3 0.01 25 1 1
FALSE 4 0.03 10 1 1
FALSE 5 0.03 15 1 1
FALSE 6 0.03 25 1 1
FALSE 7 0.05 10 1 1
FALSE 8 0.05 15 1 1
FALSE 9 0.05 25 1 1Logistic Regression model with parameter tuning and 10-Fold Cross Validation is used to make predictions. The AUC is 1 and accuracy is also 1.
## Logistic Regression MODEL
set.seed(1)
#for balanced folds: stratified =TRUE
index <- generateCVRuns(trainn$V1, ntimes=5, nfold=10, stratified =TRUE)
generate_lambda <- glmnet(as.matrix(trainn), trainn$V1, family="binomial", alpha = 1, nlambda=50, standardize = TRUE)
lambdas <- generate_lambda$lambda
cv_summary <- data.table()
for(i in 1:10){
test_index <- index$`Run 1`[i]
train_cv <- trainn[-test_index[[1]],]
test_cv <- trainn[test_index[[1]],]
model_lr <- glmnet(as.matrix(trainn), as.factor(trainn$V1), family="binomial", alpha = 1,lambda=lambdas,standardize = TRUE)
pred <- predict(model_lr, as.matrix(test_cv), s = lambdas, type = "class")
pred <- pred[,2]
cm <- confusionMatrix(factor(pred), factor(test_cv$V1))
acc_temp <- cm$overall[1:1]
auc_temp <- AUC(as.numeric(pred),as.numeric(test_cv$V1))
cv_summary <- rbind(cv_summary,data.table(Fold=i, Accuracy=acc_temp, AUC=auc_temp))
}
print(cv_summary)FALSE Fold Accuracy AUC
FALSE 1: 1 1 1
FALSE 2: 2 1 1
FALSE 3: 3 1 1
FALSE 4: 4 1 1
FALSE 5: 5 1 1
FALSE 6: 6 1 1
FALSE 7: 7 1 1
FALSE 8: 8 1 1
FALSE 9: 9 1 1
FALSE 10: 10 1 12.2 Manhattan Distance
Decision Tree model with parameter tuning and 10-Fold Cross Validation is used to make predictions. The AUC and Accuracy are almost 0.5.
## Decision Tree MODEL
trainn2 <- bag_average2 %>% select(-V2)
X_train <- bag_average2 %>% select(-V1)
y_train <- bag_average2 %>% select(V1) %>% as.factor()
set.seed(1)
complexity=c(0.01,0.03,0.05)
min_number_of_observations=c(10,15,25)
cv_summary=data.table()
#for balanced folds: stratified =TRUE
index <- generateCVRuns(trainn$V1, ntimes=5, nfold=10, stratified =TRUE)
acc <- 0
best_complexity <- 0
best_min_number_of_observations <- 0
for(i in 1:10){
for(c in complexity){
for(m in min_number_of_observations){
test_index <- index$`Run 1`[i]
train_cv <- trainn2[-test_index[[1]],]
test_cv <- trainn2[test_index[[1]],]
fitControl <- rpart.control(cp = c, minbucket = m)
model_DT <- rpart(factor(V1)~., trainn, control=fitControl)
pred <- predict(model_DT, test_cv, type="class")
cm <- confusionMatrix(pred, factor(test_cv$V1))
acc_temp <- cm$overall[1:1]
auc_temp <- AUC(as.numeric(pred),as.numeric(test_cv$V1))
cv_summary <- rbind(cv_summary,data.table(Fold=i, Complexity=c, Min_Observations_per_Leaf=m, Accuracy=acc_temp, AUC=auc_temp))
if(acc_temp>acc){
acc<-acc_temp
best_complexity <- c
best_min_number_of_observations <- m
auc<-auc_temp
}
}
}
}
cv_summary <- cv_summary %>% group_by(Complexity, Min_Observations_per_Leaf) %>% summarise(mean_accuracy = mean(Accuracy), mean_AUC=mean(AUC))
print(cv_summary)FALSE # A tibble: 9 x 4
FALSE # Groups: Complexity [3]
FALSE Complexity Min_Observations_per_Leaf mean_accuracy mean_AUC
FALSE <dbl> <dbl> <dbl> <dbl>
FALSE 1 0.01 10 0.489 0.5
FALSE 2 0.01 15 0.489 0.5
FALSE 3 0.01 25 0.489 0.5
FALSE 4 0.03 10 0.489 0.5
FALSE 5 0.03 15 0.489 0.5
FALSE 6 0.03 25 0.489 0.5
FALSE 7 0.05 10 0.489 0.5
FALSE 8 0.05 15 0.489 0.5
FALSE 9 0.05 25 0.489 0.5Logistic Regression model with parameter tuning and 10-Fold Cross Validation is used to make predictions. The AUC is 1 and accuracy is also 1.
## Logistic Regression MODEL
set.seed(1)
#for balanced folds: stratified =TRUE
index <- generateCVRuns(trainn$V1, ntimes=5, nfold=10, stratified =TRUE)
generate_lambda <- glmnet(as.matrix(trainn2), trainn2$V1, family="binomial", alpha = 1, nlambda=50, standardize = TRUE)
lambdas <- generate_lambda$lambda
cv_summary <- data.table()
for(i in 1:10){
test_index <- index$`Run 1`[i]
train_cv <- trainn2[-test_index[[1]],]
test_cv <- trainn2[test_index[[1]],]
model_lr <- glmnet(as.matrix(trainn2), as.factor(trainn2$V1), family="binomial", alpha = 1, lambda=lambdas, standardize = TRUE)
pred <- predict(model_lr, as.matrix(test_cv), s = lambdas, type = "class")
pred <- pred[,2]
cm <- confusionMatrix(factor(pred), factor(test_cv$V1))
acc_temp <- cm$overall[1:1]
auc_temp <- AUC(as.numeric(pred),as.numeric(test_cv$V1))
cv_summary <- rbind(cv_summary,data.table(Fold=i, Accuracy=acc_temp, AUC=auc_temp))
}
print(cv_summary)FALSE Fold Accuracy AUC
FALSE 1: 1 1 1
FALSE 2: 2 1 1
FALSE 3: 3 1 1
FALSE 4: 4 1 1
FALSE 5: 5 1 1
FALSE 6: 6 1 1
FALSE 7: 7 1 1
FALSE 8: 8 1 1
FALSE 9: 9 1 1
FALSE 10: 10 1 13 Comparison
I used Euclidean and Manhattan distance metrics for clustering, then I trained Decision Tree and Logistic Regression Models on the representative data with parameter tuning and 10-Fold Cross-Validation. All the models except Decision Tree for Manhattan distance, resulted in AUC as 1. Decision Tree for Manhattan distance resulted in AUC as almost 0.5.
4 Conclusion
In conclusion, cross validation results are very high and I would like to see how the models work on whole data to see if there is overfitting. But this is not desired as stated in the discussion board on Moodle.
5 Extra Model with Max Distance
The following model is given as extra to show that I have tried other ways and did not succeed. This is not a good model since accuracy on train data is very high and accuracy on test data is very low which means there is overfitting.
Another method that I have tried was to select a representative instance from each bag considering the distances to other bags. I want the representative instance for a positive labeled bag to be far away from the negatives bags. So, for each instance in a bag, I will calculate the distances to all the negative bags and select the instance that is the farthest from the negatives bags. So I need an approach to decide on instance-bag distance. I will define it as the minimum distance between the selected instance and all the instances from the bag.
As an example consider Bag 1 with 4 instances which is labeled as positive. In order to find a representative instance for Bag 1, for each of the 4 instances, I will compute a distance vector to all the instances from the negative bags. Let N be the set of instances from negative bags. My aim is to find the instance in Bag 1 that is farthest from the elements of N. Then I will compare the 4 distances and select the one with the largest distance as the representative.
To be more clear, Bag 1 has 4 instances and labeled as 1, Bag 50 has 2 instances and labeled as 0. Let’s call Bag 1 instances as b1-1, b1-2, b1-3, b1-4 and Bag 50 instances as b50-1, b50-2. Then, dist(b1-1,Bag 50)= min(dist(b1-1,b50-1), dist(b1-1,b50-2)) and dist(b1-1,N)=min(dist(b1-1,Bag i)) for all i. We find argmax(dist(b1-j,N)) for all j=1, 2, 3, 4 and it gives our representative. In summary, I will use Euclidean distance computing instance-instance and take the minimum of them for instance-bag and then take the minimum for instance-negative. This corresponds to argmax(dist(b1-i,N)) for all i.
The one with the largest instance-negative distance will be the representative. Instance-positive distances are calculated in the same manner.
#scale
musk <- musk_data
musk[,3:168] <- scale(musk[,3:168])
musk<- musk %>% mutate(instance = 1:n()) %>% select(instance, everything())
#distance instance-instance
ins_ins <- as.matrix(dist(musk[,4:169], method = "euclidean"))
labeled_ins_ins <- cbind(musk[,1:3], ins_ins)
#Bag ID's of class 0 bags
ID_0 <- musk %>% filter(V1==0) %>% summarise(id=unique(V2)) %>% unlist() %>% as.numeric()
#Bag ID's of class 1 bags
ID_1 <- musk %>% filter(V1==1) %>% summarise(id=unique(V2)) %>% unlist() %>% as.numeric()
#indices of class 0 instances
ins_0 <- musk %>% filter(V1==0) %>% summarise(id=unique(instance)) %>% unlist() %>% as.numeric()
#indices of class 1 instances
ins_1 <- musk %>% filter(V1==1) %>% summarise(id=unique(instance)) %>% unlist() %>% as.numeric()
# number of instances in each negative bag, 45 positive bags
n_neg <- musk %>% filter(V1==0) %>% group_by(V2) %>% summarise(count=n())
# number of instances in each positive bag, 47 positive bags
n_pos <- musk %>% filter(V1==1) %>% group_by(V2) %>% summarise(count=n())
#for 1(positive) bags
representative_indices_1 <- c()
for (i in ID_1){
bag <- labeled_ins_ins %>% filter(V2==i) %>% select(ins_0+3) %>% t() %>% data.frame()
ind <- labeled_ins_ins %>% filter(V2==i) %>% summarise(id=unique(instance)) %>% unlist() %>% as.numeric()
min_dist = c()
for (j in 1:length(bag)){
x<-min(bag[[j]]) # her instance için min uzaklık bulduk
min_dist <- c(min_dist,x) # her bag için bir vektör bulduk
}
order <- rbind(ind, min_dist) %>% t() %>% data.frame() %>% arrange(desc(min_dist)) # sıraladık
representative_index <- order[1,1] #for each instance in a bag, there will be one representative index
representative_indices_1 <- c(representative_indices_1,representative_index)
}
representative_indices_1FALSE [1] 4 8 9 11 15 19 21 23 28 30 35 44 51 52 54 58 64 66 70
FALSE [20] 79 85 87 90 94 102 106 108 115 121 124 126 130 138 140 147 149 154 158
FALSE [39] 166 168 173 178 183 186 190 194 202#for 0(negative) bags
representative_indices_0 <- c()
for (i in ID_0){
bag <- labeled_ins_ins %>% filter(V2==i) %>% select(ins_1+3) %>% t() %>% data.frame()
ind <- labeled_ins_ins %>% filter(V2==i) %>% summarise(id=unique(instance)) %>% unlist() %>% as.numeric()
min_dist = c()
for (j in 1:length(bag)){
x<-min(bag[[j]]) # her instance için min uzaklık bulduk
min_dist <- c(min_dist,x) # her bag için bir vektör bulduk
order <- rbind(ind, min_dist) %>% t() %>% data.frame() %>% arrange(desc(min_dist)) # sıraladık
representative_index <- order[1,1] #for each instance in a bag, there will be one representative index
}
representative_indices_0 <- c(representative_indices_0,representative_index)
}
representative_indices_0FALSE [1] 208 212 216 219 223 230 234 236 238 240 244 248 281 289 290 292 296 300 302
FALSE [20] 305 307 308 313 315 317 320 323 324 327 333 339 340 349 387 398 431 432 434
FALSE [39] 437 438 457 458 462 468 474representative_indices <- c(representative_indices_1, representative_indices_0)
train <- musk[representative_indices, ] %>% select(-V2, -instance) %>% as.data.frame()
test <- musk[-representative_indices, ] %>% select(-V2, -instance) %>% as.data.frame()
Xtrain <- train %>% select(-V1)
ytrain <- train %>% select(V1)
Xtest <- test %>% select(-V1)
ytest <- test %>% select(V1)
set.seed(2)
model_pra<-cv.glmnet(as.matrix(Xtrain), train$V1, type.measure = "class", family="binomial", nfolds=10)
pred_pra <- predict(model_pra, newx=as.matrix(Xtest),s=c("lambda.min"), type = "class")
cm_pra_test <- confusionMatrix(factor(pred_pra), factor(ytest$V1))
cm_pra_test[["overall"]][["Accuracy"]]FALSE [1] 0.7005208cm_pra_train <- confusionMatrix(factor(predict(model_pra, newx=as.matrix(Xtrain), s=c("lambda.min"), type = "class")), factor(train$V1))
cm_pra_train[["overall"]][["Accuracy"]]FALSE [1] 0.9347826print(model_pra)FALSE
FALSE Call: cv.glmnet(x = as.matrix(Xtrain), y = train$V1, type.measure = "class", nfolds = 10, family = "binomial")
FALSE
FALSE Measure: Misclassification Error
FALSE
FALSE Lambda Measure SE Nonzero
FALSE min 0.02953 0.1522 0.03794 22
FALSE 1se 0.05161 0.1848 0.04415 17plot(model_pra)
AUC(as.numeric(pred_pra),as.numeric(test$V1))FALSE [1] 0.7272321nb <- c(3, 5, 7)
knn_grid <- expand.grid(k = nb)
fitControl <- trainControl(method = "repeatedcv", number = 10, repeats = 1, classProbs = FALSE, search="random")
model_knn <- train(factor(V1) ~ ., data = train, method = "knn", trControl = fitControl, tuneGrid = knn_grid)
print(model_knn[["results"]])FALSE k Accuracy Kappa AccuracySD KappaSD
FALSE 1 3 0.8511111 0.6938938 0.10064265 0.2138613
FALSE 2 5 0.6950000 0.3829445 0.08711384 0.1715547
FALSE 3 7 0.6750000 0.3465265 0.09831398 0.1659169pred_knn <- predict(model_knn, newdata=Xtest)
cm_knn_test <- confusionMatrix(factor(pred_knn), factor(test$V1))
cm_knn_test[["overall"]][["Accuracy"]]FALSE [1] 0.625cm_knn_train <- confusionMatrix(factor(predict(model_knn, newdata=Xtrain)), factor(train$V1))
cm_knn_train[["overall"]][["Accuracy"]]FALSE [1] 0.9565217AUC(as.numeric(pred_knn),as.numeric(test$V1))FALSE [1] 0.675This html file found here and Rmd codes can be found here as well.