Here is a copy of the 2023 exams where I have added brief comments on what I expected to see in the answers to each question.

Beyond the comments, I expected to see clean text and figures with captions. I also expected all results to be commented on and discussed. Just plots without comment do not results in full marks.

EXAM May 22nd - June 5th, 2023

The take-home exam comprises three questions. To pass the exam you have to provide complete answers to Questions 1 as well as adequate answers to one of Questions 2 and 3. For higher grades you have to provide complete answers to all three questions.

You should work alone for this exam. The reports will be checked with Urkund. You can discuss coding issues or bugs with other students sparingly but don't discuss how to intepret figures or tables.

"Can I use ChatGPT?" It's been my experience that, while ChatGPT can be helpful, it is also overconfident in providing answers that are in fact wrong. It takes time to check the output from ChatGPT and, depending on your own skillset, this may result in an overall waste of time rather than a saving time, especially if you want to connect multiple tasks. Your main task is to analyze and interpret the results. ChatGPT might help you with setting up the analysis but not interpreting and spinning the analysis in the way I have requested.

So I would say that you should use ChatGPT with caution and only for simple coding questions like generating plots, proposed functions calls (which may contain bugs), etc. Don't use ChatGPT for generating text - I will run this through GPTZero.

In addition, I will randomly select a subset of students for an oral presentation of your results and codes.

GOOD LUCK!

• The report should not be longer than 1 page of text + max 1/2 page conclusion/main summary per sub-task (sub-tasks = a,b,c etc).
• You do not have to discuss the methods and assumptions unless this pertains to a result you present - i.e. you don't need to write a "lab report" with introduction, methods, analysis and conclusions. Here, it's only analysis and conclusions that I am interested in.
• Figures and Tables: maximum of 4 figures and 2 tables per sub-task.
• Make sure you provide captions for the tables and figures - these should clearly state what the figure/table contains as well as a 1-2 conclusion statement about what can be learnt from the figure/table. Refer to the figures and tables in the text!
• MOTIVATE YOUR ANSWERS!
• Don't contradict your results! If something is unexpected say so and explain why you thought you would see something different.
• Please think about your presentation. Spell check, grammar check. "The arm-length check": meaning, skim through the report as it appears on the screen - does it look nice and neat or is it messy with mis-alignments, cut-off sentences, plots that are tiny etc?
• No cut and paste code mixed with the text.
• No, jupyter notebooks are not accepted as a report. Preferably use latex (overleaf).
• Yes, you can use bulletpoints to present your findings. This could be especially useful for the subtask conclusion.
• Stress your main findings in the 1/2 page subtask conclusion.
• Make sure you put the figures with each subtask after the subtask discussion and before the next subtask so we don't have to scroll back and forth between tasks when we grade.

IMPORTANT

Wherever the exam task contains a question ("Does X differ from Y?") it is not sufficient to answer "Yes" or "No" - you need to explain how you arrived to this conclusion, which results provide you with these answers, as well as a statement trying explain why the results are what they are. Likewise, when a task is phrased as "Can you do X to get Y?" I, of course, mean for you to attempt to answer this question by doing something, not just hypothesizing about it.

To answer the questions it is for the most part inadequate to perform the task once on the full data set. Make use of re-sampling techniques to support your findings.

Question 1 Data set

For this exam question you will use a data set with 5000 observations and 1000 features. The data set is contained in a tab-del file called "DataMatrix.txt" posted on the class home page.

The first column of the data matrix contains a 4-class label vector.

The data set contains missing values and these are encoded as -1000.

For all subtasks, make sure you motivate your answers and provide a take-home message summarizing your main conclusions.

Question 1 (50p)

1a (10p)

For now, let us ignore the missing values. That is, drop the observations that contain any missing values.

Train multiple classifiers (at least 6, of varying degrees of complexity) on multiple training-test splits of the data. Compare classification performance.

Pay particular attention to the following;

• balance between classes
• classification accuracy, specificity and sensitivity overall and the class level.

Be careful about using the full data set for any training/validation steps. For example, exploration and dimension reduction, if you choose to pursue this approach, should be done of the training data only.

Make sure to discuss:

Overall performance and how you assess this

Are all or only some classes well separated

Class imbalance, modeling assumptions, training metrics

The full training pipeline you set up and motivate all your choices

Expectation

• discuss the pipeline - ensure no data leakage if you use e.g. pca
• stratified sampling since the data set is imbalanced
• dropping missing values exacerbated the imbalance (by construction by me) - expected you to notice this effect
• investigate different metrics of classification performance and notice that e.g. class 2 is very difficult to detect while class 3 and 4 are well separated
• report the results in side-by-side boxplots (for example) to compare classification performance.
• use class specific metrics
• consider upsampling or other techniques to counter the class imbalance

1b (10p)

Let us turn our attention to feature selection. Again, we will ignore (drop) missing values for now.

Use at least 3 different techniques for selecting features for classification. (These should be truly different and not just the same wrapper applied to different classifiers. I.e., don't use variable importance on rf, logistic regression and knn for example, but explore other feature selection approaches.)

You can limit yourself to the most promising 3 classifiers from Question 1a for this task.

Pay particular attention to;

• selection stability
• how many features are selected with different methods/techniques/criteria

Make sure to discuss:

Method comparisons

Are the same features predictive for all classes or can performance be improved if you can select different features for different classification tasks?

Can you find an underlying reasons for why this particular task may be challenges (based on this data set, not in all generality)?

Sources of, and approaches to improve, selection instability

Expectation

• repeated train-test splits
• comment on the number of features "needed" for each of the classifiers - based on stability selection, importance statistics across the runs
• look at overlap between top selected features, or the list of top features, between the classifiers
• notice that some of the classifiers+selection are stable and some are not (here lasso was more stable than RF)
• comment on the performance drop off for the classifiers as you reduced the number of features (including plots of course)
• in terms of improving selection stability, elastic net was better than lasso, running on PCs is more stable than on feautures
• the imbalance was a challenge and that the classes were separated in different subspaces

1c (10p)

There are 4 classes in the data set. Could this be converted to a 3 class problem and result in an improved performance?

What about a 5 class problem?

Explain and implement your strategy for assessing this (statistically).

Make sure to discuss:

How you chose/constructed/approached the 3/5 classification problem

How you compare the performance between the 3/5 and original 4 class problem to assess if performance is affected

How is feature selection impacted by the new class division?

Expectation

• One could approach this problem many ways.
• The obvious thing was to collapse class 1 and 2 because they were difficult to classify
• For part 2, many made the mistake of creating a data leakage situation by clustering the full data first and relabeling the data set with 5 classes (clusters).
• You could investigate subclasses based on what you see in training data
• You could also explore mixture discriminant methods
• By exploring the data in a reduced dimension you can see a subclass that I created. It was important to plot the data and not blindly use all featuress.

1d (10p)

How many features are related to the class labels (assess statistically)?

Does this differ depending which class-comparisons you consider (you don't have to try all pairwise comparisons but compare at least 3 different class-comparisons (all-vs-all, one-vs-one, one-vs-rest, etc)?

Make sure to discuss (and investigate):

How this depends on sample size

How this question differs from 1b

How you adress the sizeable number of features in the data set

Expectation

• Here, it was important to notice the phrasing of the question - "significant" and "related" - which is not a feature selection question but one of testing.
• Perform multiple testing - and as the question states that can be done in many ways
• I expected to see multiple testing adjusted feature lists for the different tests and comment on the number of significant features.
• Notice that there were fewer signficant features in the Class 1 vs 2 comparison - the classes that overlap.
• Notice that there was partial overlap between signficant features between the classes/tests
• Explore the impact of reduced sample sizes and notice that fewer feature are found to be significant then

1e (10p)

Without using any class label information, perform clustering of the data set.

How many clusters do you detect? - explain which (at least 3) metrics you use. Remember! these metrics cannot depend on the true class labels.

What about the clustering the features? Relate the outcome of feature clustering to Questions 1b and 1d. What can you conclude?

Make sure to discuss;

How the result vary when you use different metrics and clustering methods

How the results differ if you use linear or nonlinear dimension reduction techniques

The full analysis pipeline, motivating all your choices

Expectation

• Many made the mistake of not plotting the data and clusters found for a reality check
• Also - make sure you know how to use the selection criteria (e.g. you look for maximum silhoutte score, not an elbow etc).
• I expected to see an application of several clustering methods and methods for selecting the number of clusters
• The results depended very much on how you transformed the data
• Depending on the pipeline from dimension reduction, clustering method and selecting the number of clusters you could get between 2 and 8 clusters
• Assessing cluster stability across multiple runs indicated that the 2-3 clusters may be stable

Question 2 (25p)

For the second question we will consider how to deal with the missing values in Question 1. As we have discussed in class, you can choose to drop the observations that contain missing values. However, this may affect the outcome if the missingness is not completely at random.

2a (10p)

Investigate the missing value distribution in the data set.

Summarize your findings and discuss risks, if any, you see with dropping observations from the data set if any features for that observation contain missing values.

On the small, clean, data set you used for Question 1 (after dropping observations), mimic at least one missing value distribution/source you observed on the full data set, i.e., generate even more missing values according to the same distribution you detected on the full data set. How is classification performance affected if you drop the observations with missing values generated by you according to the detected mechanism?

Expectation

• I expected you to notice that the missingness was uneven across classes which exacerbated the imbalance if observations were dropped - this was a clear risk.
• I expected a simulation study where you dropped randomly or more in one class and see how much that affected the classification performance - which varied between classifiers.

2b (15)

Implement at least 3 different strategies for imputing the missing values in the data set.

Pay particular attention to how you impute data on the test set - you cannot use the test set to learn how to impute on the training set, nor to tune parameters of imputation, so you have to figure out how to learn imputation from the training set only and apply it on the test set. If test data is used to learn/tune have to impute it's called data leakage.

Make sure you discuss;

How class-specific performance (accuracy, sensitivity, specificity) is affected by imputation

How the different imputation strategies perform

Concerns you might have given what you discovered in 2a.

Expectation

• You should see a slight improvement in performance if you impute the missing values, especially for methods that need more data or can't handle imbalance well
• Mean imputation was not a good method is paired with a distance based classifier like kNN
• Sensitivity is increased through imputation, but class 2 was still very difficult to detect
• The concern is that this is not missing at random.

Question 3 (25p)

In class we have discussed Ensemble methods. For example, RandomForest is one ensemble technique that uses bootstrap and feature sampling to build multiple classfication trees that are then aggregated with simple voting. Boosting is another technique some of you have looked into on your own. Here, we build a set of smaller trees (usually) that on their own may not result in a good performance but learnt sequentially while putting more and more emphasis on misclassified observations may perform really well as an ensemble. Extensions to parallel updates for scalability are also available.

Model Stacking

For this question you will explore model stacking. The benefit of model stacking is that you can create ensembles of completely different methods and pipelines.

You need to split the data into 3 components; training, validation and testing. On the same training data split, train different classifiers and tune using validation data created from the training data only. Extract predictions on the validation data for all the classifiers. These predictions can be the labels but preferably you should extract the soft predictions, i.e. class posteriors.

Now you take these class predictions from all classifiers and stack them, i.e. creating new features. If you used L classifiers there are $L*C$ predictors, where C is the number of classes. If your classifiers $l$ is a good classifier, the class probability vector for observation i, $(p_{i1}, p_{i2}, \cdots, p_{iC})$ will look close to a 0-1 vector with the large probability value at the position of the true class label $y_i$. That is, features generatated by the stacking procedure may resemble 0-1 encoding of the outcomes!

Now we come to the final step. Run logistic regression with the stacked features as new features and the validation true labels as outcome. This generates a new predictive model for the outcome. Here, it is common to use regularization to not only get weights for the different classifiers but also select only a subset of the, and to address the co-linearities that are bound to be created by stacking predictions from classifiers applied to the same validation set.

To predict on the test set you have to predict new features (posterior probabilities or class predictions) from each classifer on the test set, as well as feed these new features through your ensemble model.

There are packages available but I want you to implement a model stacking procedure from scratch using the principles above. The goal is to dig in and investigate pros and cons and the logic behind the approach. One can implement a full cross-validation scheme instead of a fixed train/validation split but you don't have to do this for the exam.

3a (10p)

Implement and apply model stacking to the Question 2 data set (or Question 1 data if you did not do Question 2 - and in that case you don't include imputation techniques below. If you like, you can impute the data and use for Question 3 even if you didn't do Question 2).

Is classification improved? Discuss.

Use at least 10 classifiers of varying character, but preferably many more. Make the most use of model stacking by exploring a rich set of pipeline choices to stack. For example, in model stacking you can include classifiers using different imputation approaches, dimension reduction techniques, number of reduced dimensions, feature transformation and selection, etc.

Make sure to discuss;

Which methods contribute the most to the ensemble?

Can you conclude anything about important features, number of dimensions, pre-processing steps etc, based on the ensemble?

Benefits and limitations based on your results

Expectation

• The main idea is to go beyond model averaging or a standard random forest so I expected to see a compination of classifiers and dimension reduction techniques and imputation techniques in the stack.
• The performance was only modestly improved by stacking
• However, if you explored class-specific weights (e.g. multinomial regression) you could see that some combinations were more favourable for the class 1 vs 2 classification for example.

3b (5p)

What if you only used 1 type of classifier but with lots of different tuning values (dimension reduction, tuning parameters, etc)?

What can you conclude?

Expectation

• There were many ways to approach this task
• The fun part was to think of this as an alternative to tuning - the weights in the stack tell you which tuning parameter the data prefers...
• For example, if you used the number of PCs as a tuning value and included say 3 levels in the stack, you might notice that some of the classes strongly prefered many PCs and some not as many
• I expected to see a demonstration of class-specific preferences or overall indications that certain tuning values were identified through the stack.

3c (10p)

Contaminate (mislabel) the training/validation data in Question 3a.

Re-run the procedure and discuss the impact of this - which methods contribute most the ensemble now? Why?

Expectation

• Some of the members of your ensemble is more sensitive to mislabeling - so you should expect their contribution to the stack to decrease as you include mislabeling
• Depending on the methods you included in the stack, you could see a cross-over where a method that contributed a lot to the stack for clean data is overtaken by another method when the data contains mislabeling
• Example: k in kNN might have a high weight for small k initially and then a much smaller weight as the mislabeling increases.
• I expected to see multiple test-train runs to draw conclusions