[This article was first published on mlr-org, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don’t.
We will go beyond resampling single learners. We will learn how to compare a large number of different models using benchmarking. In this exercise, we will not show you how to tune a learner. Instead, we will compare identical learners with different hyperparameters that are set manually. In particular, we will learn how to set up benchmarking instances in mlr3.
We create the task as for the resampling exercise: Again, we make us of our work horse: The German Credit Data set.
library(mlr3verse)
Loading required package: mlr3
task = tsk("german_credit")
set.seed(20220801)
We are going to compare a range of different KNN models ranging from a 
of 3 to 30. Furthermore, we want to assess the performance of a logistic regression.
Create the learners
Create a logistic regression learner and many KNN learners. You should cover all KNNs with a between 3 and 30. Save all learners in a list. Give the KNN learners an appropriate id that reflects their .
Show Hint 1:
Use the lapply function or a for-loop to create the list of learners with between 3 and 30. Don’t forget to also include the logistic regression learner in your list (the append function might be helpful here to extend a created list). The lrn function has an argument id that can be used to change the name of the learner (here, you should give the KNN learners an appropriate id that reflects their value of to be able to distinguish the learners).
Show Hint 2:
To create a list of KNN learners, you can use this template: lapply(..., function(i) lrn("classif.kknn", k = i, id = paste0("classif.knn", i))
log_reg = lrn("classif.log_reg")
knn = lapply(3:30, function(i) lrn("classif.kknn", k = i, id = paste0("classif.knn", i)))
lrns = append(log_reg, knn)
Create the resampling
Create a 4-fold cross-validation resampling. Create a list that only contains this resampling (this is needed later for the benchmark_grid function).
Show Hint:
See the previous resampling use case.
cv4 = list(rsmp("cv", folds = 4))
Create a benchmarking design
To design your benchmark experiment consisting of tasks, learners and resampling technique, you can use the benchmark_grid function from mlr3. Here, we will use only one task and one resampling technique but multiple learners. Use the previously created task (german credit), learners (the list of many KNN learners and a single logistic regression learner) and resampling (4 fold CV) as input.
Show Hint 1:
Also make sure that the task is included in a list as the arguments of the benchmark_grid function requires lists as input.
Show Hint 2:
benchmark_grid(...
design = benchmark_grid(list(task), lrns, cv4)
Run the benchmark
Now you still need to run all experiments specified in the design. Do so by using the benchmark function. This may take some time. (Still less than a minute.) Make sure to store the benchmark in a new object called bmr as you will reuse and inspect the benchmark result in the subsequent exercises.
Show Hint 1:
bmr = benchmark(design)
INFO [19:02:59.886] [mlr3] Running benchmark with 116 resampling iterations INFO [19:03:00.011] [mlr3] Applying learner 'classif.log_reg' on task 'german_credit' (iter 1/4) INFO [19:03:00.119] [mlr3] Applying learner 'classif.log_reg' on task 'german_credit' (iter 2/4) INFO [19:03:00.222] [mlr3] Applying learner 'classif.log_reg' on task 'german_credit' (iter 3/4) INFO [19:03:00.323] [mlr3] Applying learner 'classif.log_reg' on task 'german_credit' (iter 4/4) INFO [19:03:00.423] [mlr3] Applying learner 'classif.knn3' on task 'german_credit' (iter 1/4) INFO [19:03:00.530] [mlr3] Applying learner 'classif.knn3' on task 'german_credit' (iter 2/4) INFO [19:03:00.651] [mlr3] Applying learner 'classif.knn3' on task 'german_credit' (iter 3/4) INFO [19:03:00.776] [mlr3] Applying learner 'classif.knn3' on task 'german_credit' (iter 4/4) INFO [19:03:00.908] [mlr3] Applying learner 'classif.knn4' on task 'german_credit' (iter 1/4) INFO [19:03:01.035] [mlr3] Applying learner 'classif.knn4' on task 'german_credit' (iter 2/4) INFO [19:03:01.064] [mlr3] Applying learner 'classif.knn4' on task 'german_credit' (iter 3/4) INFO [19:03:01.123] [mlr3] Applying learner 'classif.knn4' on task 'german_credit' (iter 4/4) INFO [19:03:01.172] [mlr3] Applying learner 'classif.knn5' on task 'german_credit' (iter 1/4) INFO [19:03:01.227] [mlr3] Applying learner 'classif.knn5' on task 'german_credit' (iter 2/4) INFO [19:03:01.298] [mlr3] Applying learner 'classif.knn5' on task 'german_credit' (iter 3/4) INFO [19:03:01.359] [mlr3] Applying learner 'classif.knn5' on task 'german_credit' (iter 4/4) INFO [19:03:01.408] [mlr3] Applying learner 'classif.knn6' on task 'german_credit' (iter 1/4) INFO [19:03:01.463] [mlr3] Applying learner 'classif.knn6' on task 'german_credit' (iter 2/4) INFO [19:03:01.510] [mlr3] Applying learner 'classif.knn6' on task 'german_credit' (iter 3/4) INFO [19:03:01.558] [mlr3] Applying learner 'classif.knn6' on task 'german_credit' (iter 4/4) INFO [19:03:01.575] [mlr3] Applying learner 'classif.knn7' on task 'german_credit' (iter 1/4) INFO [19:03:01.619] [mlr3] Applying learner 'classif.knn7' on task 'german_credit' (iter 2/4) INFO [19:03:01.664] [mlr3] Applying learner 'classif.knn7' on task 'german_credit' (iter 3/4) INFO [19:03:01.706] [mlr3] Applying learner 'classif.knn7' on task 'german_credit' (iter 4/4) INFO [19:03:01.748] [mlr3] Applying learner 'classif.knn8' on task 'german_credit' (iter 1/4) INFO [19:03:01.789] [mlr3] Applying learner 'classif.knn8' on task 'german_credit' (iter 2/4) INFO [19:03:01.831] [mlr3] Applying learner 'classif.knn8' on task 'german_credit' (iter 3/4) INFO [19:03:01.873] [mlr3] Applying learner 'classif.knn8' on task 'german_credit' (iter 4/4) INFO [19:03:01.915] [mlr3] Applying learner 'classif.knn9' on task 'german_credit' (iter 1/4) INFO [19:03:01.956] [mlr3] Applying learner 'classif.knn9' on task 'german_credit' (iter 2/4) INFO [19:03:01.998] [mlr3] Applying learner 'classif.knn9' on task 'german_credit' (iter 3/4) INFO [19:03:02.040] [mlr3] Applying learner 'classif.knn9' on task 'german_credit' (iter 4/4) INFO [19:03:02.082] [mlr3] Applying learner 'classif.knn10' on task 'german_credit' (iter 1/4) INFO [19:03:02.123] [mlr3] Applying learner 'classif.knn10' on task 'german_credit' (iter 2/4) INFO [19:03:02.167] [mlr3] Applying learner 'classif.knn10' on task 'german_credit' (iter 3/4) INFO [19:03:02.209] [mlr3] Applying learner 'classif.knn10' on task 'german_credit' (iter 4/4) INFO [19:03:02.249] [mlr3] Applying learner 'classif.knn11' on task 'german_credit' (iter 1/4) INFO [19:03:02.291] [mlr3] Applying learner 'classif.knn11' on task 'german_credit' (iter 2/4) INFO [19:03:02.333] [mlr3] Applying learner 'classif.knn11' on task 'german_credit' (iter 3/4) INFO [19:03:02.375] [mlr3] Applying learner 'classif.knn11' on task 'german_credit' (iter 4/4) INFO [19:03:02.416] [mlr3] Applying learner 'classif.knn12' on task 'german_credit' (iter 1/4) INFO [19:03:02.458] [mlr3] Applying learner 'classif.knn12' on task 'german_credit' (iter 2/4) INFO [19:03:02.500] [mlr3] Applying learner 'classif.knn12' on task 'german_credit' (iter 3/4) INFO [19:03:02.543] [mlr3] Applying learner 'classif.knn12' on task 'german_credit' (iter 4/4) INFO [19:03:02.582] [mlr3] Applying learner 'classif.knn13' on task 'german_credit' (iter 1/4) INFO [19:03:02.625] [mlr3] Applying learner 'classif.knn13' on task 'german_credit' (iter 2/4) INFO [19:03:02.669] [mlr3] Applying learner 'classif.knn13' on task 'german_credit' (iter 3/4) INFO [19:03:02.709] [mlr3] Applying learner 'classif.knn13' on task 'german_credit' (iter 4/4) INFO [19:03:02.751] [mlr3] Applying learner 'classif.knn14' on task 'german_credit' (iter 1/4) INFO [19:03:02.792] [mlr3] Applying learner 'classif.knn14' on task 'german_credit' (iter 2/4) INFO [19:03:02.834] [mlr3] Applying learner 'classif.knn14' on task 'german_credit' (iter 3/4) INFO [19:03:02.878] [mlr3] Applying learner 'classif.knn14' on task 'german_credit' (iter 4/4) INFO [19:03:02.919] [mlr3] Applying learner 'classif.knn15' on task 'german_credit' (iter 1/4) INFO [19:03:02.961] [mlr3] Applying learner 'classif.knn15' on task 'german_credit' (iter 2/4) INFO [19:03:03.000] [mlr3] Applying learner 'classif.knn15' on task 'german_credit' (iter 3/4) INFO [19:03:03.041] [mlr3] Applying learner 'classif.knn15' on task 'german_credit' (iter 4/4) INFO [19:03:03.083] [mlr3] Applying learner 'classif.knn16' on task 'german_credit' (iter 1/4) INFO [19:03:03.124] [mlr3] Applying learner 'classif.knn16' on task 'german_credit' (iter 2/4) INFO [19:03:03.168] [mlr3] Applying learner 'classif.knn16' on task 'german_credit' (iter 3/4) INFO [19:03:03.210] [mlr3] Applying learner 'classif.knn16' on task 'german_credit' (iter 4/4) INFO [19:03:03.254] [mlr3] Applying learner 'classif.knn17' on task 'german_credit' (iter 1/4) INFO [19:03:03.296] [mlr3] Applying learner 'classif.knn17' on task 'german_credit' (iter 2/4) INFO [19:03:03.338] [mlr3] Applying learner 'classif.knn17' on task 'german_credit' (iter 3/4) INFO [19:03:03.379] [mlr3] Applying learner 'classif.knn17' on task 'german_credit' (iter 4/4) INFO [19:03:03.421] [mlr3] Applying learner 'classif.knn18' on task 'german_credit' (iter 1/4) INFO [19:03:03.462] [mlr3] Applying learner 'classif.knn18' on task 'german_credit' (iter 2/4) INFO [19:03:03.504] [mlr3] Applying learner 'classif.knn18' on task 'german_credit' (iter 3/4) INFO [19:03:03.545] [mlr3] Applying learner 'classif.knn18' on task 'german_credit' (iter 4/4) INFO [19:03:03.587] [mlr3] Applying learner 'classif.knn19' on task 'german_credit' (iter 1/4) INFO [19:03:03.629] [mlr3] Applying learner 'classif.knn19' on task 'german_credit' (iter 2/4) INFO [19:03:03.672] [mlr3] Applying learner 'classif.knn19' on task 'german_credit' (iter 3/4) INFO [19:03:03.715] [mlr3] Applying learner 'classif.knn19' on task 'german_credit' (iter 4/4) INFO [19:03:03.757] [mlr3] Applying learner 'classif.knn20' on task 'german_credit' (iter 1/4) INFO [19:03:03.799] [mlr3] Applying learner 'classif.knn20' on task 'german_credit' (iter 2/4) INFO [19:03:03.841] [mlr3] Applying learner 'classif.knn20' on task 'german_credit' (iter 3/4) INFO [19:03:03.883] [mlr3] Applying learner 'classif.knn20' on task 'german_credit' (iter 4/4) INFO [19:03:03.924] [mlr3] Applying learner 'classif.knn21' on task 'german_credit' (iter 1/4) INFO [19:03:03.966] [mlr3] Applying learner 'classif.knn21' on task 'german_credit' (iter 2/4) INFO [19:03:04.007] [mlr3] Applying learner 'classif.knn21' on task 'german_credit' (iter 3/4) INFO [19:03:04.049] [mlr3] Applying learner 'classif.knn21' on task 'german_credit' (iter 4/4) INFO [19:03:04.091] [mlr3] Applying learner 'classif.knn22' on task 'german_credit' (iter 1/4) INFO [19:03:04.133] [mlr3] Applying learner 'classif.knn22' on task 'german_credit' (iter 2/4) INFO [19:03:04.176] [mlr3] Applying learner 'classif.knn22' on task 'german_credit' (iter 3/4) INFO [19:03:04.218] [mlr3] Applying learner 'classif.knn22' on task 'german_credit' (iter 4/4) INFO [19:03:04.259] [mlr3] Applying learner 'classif.knn23' on task 'german_credit' (iter 1/4) INFO [19:03:04.300] [mlr3] Applying learner 'classif.knn23' on task 'german_credit' (iter 2/4) INFO [19:03:04.343] [mlr3] Applying learner 'classif.knn23' on task 'german_credit' (iter 3/4) INFO [19:03:04.385] [mlr3] Applying learner 'classif.knn23' on task 'german_credit' (iter 4/4) INFO [19:03:04.426] [mlr3] Applying learner 'classif.knn24' on task 'german_credit' (iter 1/4) INFO [19:03:04.468] [mlr3] Applying learner 'classif.knn24' on task 'german_credit' (iter 2/4) INFO [19:03:04.509] [mlr3] Applying learner 'classif.knn24' on task 'german_credit' (iter 3/4) INFO [19:03:04.551] [mlr3] Applying learner 'classif.knn24' on task 'german_credit' (iter 4/4) INFO [19:03:04.593] [mlr3] Applying learner 'classif.knn25' on task 'german_credit' (iter 1/4) INFO [19:03:04.634] [mlr3] Applying learner 'classif.knn25' on task 'german_credit' (iter 2/4) INFO [19:03:04.678] [mlr3] Applying learner 'classif.knn25' on task 'german_credit' (iter 3/4) INFO [19:03:04.720] [mlr3] Applying learner 'classif.knn25' on task 'german_credit' (iter 4/4) INFO [19:03:04.762] [mlr3] Applying learner 'classif.knn26' on task 'german_credit' (iter 1/4) INFO [19:03:04.804] [mlr3] Applying learner 'classif.knn26' on task 'german_credit' (iter 2/4) INFO [19:03:04.846] [mlr3] Applying learner 'classif.knn26' on task 'german_credit' (iter 3/4) INFO [19:03:04.889] [mlr3] Applying learner 'classif.knn26' on task 'german_credit' (iter 4/4) INFO [19:03:04.934] [mlr3] Applying learner 'classif.knn27' on task 'german_credit' (iter 1/4) INFO [19:03:04.975] [mlr3] Applying learner 'classif.knn27' on task 'german_credit' (iter 2/4) INFO [19:03:05.016] [mlr3] Applying learner 'classif.knn27' on task 'german_credit' (iter 3/4) INFO [19:03:05.058] [mlr3] Applying learner 'classif.knn27' on task 'german_credit' (iter 4/4) INFO [19:03:05.099] [mlr3] Applying learner 'classif.knn28' on task 'german_credit' (iter 1/4) INFO [19:03:05.141] [mlr3] Applying learner 'classif.knn28' on task 'german_credit' (iter 2/4) INFO [19:03:05.185] [mlr3] Applying learner 'classif.knn28' on task 'german_credit' (iter 3/4) INFO [19:03:05.226] [mlr3] Applying learner 'classif.knn28' on task 'german_credit' (iter 4/4) INFO [19:03:05.268] [mlr3] Applying learner 'classif.knn29' on task 'german_credit' (iter 1/4) INFO [19:03:05.310] [mlr3] Applying learner 'classif.knn29' on task 'german_credit' (iter 2/4) INFO [19:03:05.352] [mlr3] Applying learner 'classif.knn29' on task 'german_credit' (iter 3/4) INFO [19:03:05.394] [mlr3] Applying learner 'classif.knn29' on task 'german_credit' (iter 4/4) INFO [19:03:05.435] [mlr3] Applying learner 'classif.knn30' on task 'german_credit' (iter 1/4) INFO [19:03:05.477] [mlr3] Applying learner 'classif.knn30' on task 'german_credit' (iter 2/4) INFO [19:03:05.518] [mlr3] Applying learner 'classif.knn30' on task 'german_credit' (iter 3/4) INFO [19:03:05.559] [mlr3] Applying learner 'classif.knn30' on task 'german_credit' (iter 4/4) INFO [19:03:05.587] [mlr3] Finished benchmark
Evaluate the benchmark
Choose two appropriate metrics to evaluate the different learners performance on the task. Compute these metrics and also visualize at least one of them using the autoplot function.
Show Hint 1:
The previously stored benchmark object has a method $aggregate(...) just like the objects created with the resample function from the previous use case.
Show Hint 2:
autoplot(..., measure = msr(...))
res = bmr$aggregate(measures = c(msr("classif.fp"), msr("classif.acc")))
head(res)
nr task_id learner_id resampling_id iters classif.fp classif.acc <int> <char> <char> <char> <int> <num> <num> 1: 1 german_credit classif.log_reg cv 4 38.75 0.747 2: 2 german_credit classif.knn3 cv 4 44.25 0.683 3: 3 german_credit classif.knn4 cv 4 44.25 0.683 4: 4 german_credit classif.knn5 cv 4 44.00 0.724 5: 5 german_credit classif.knn6 cv 4 44.75 0.728 6: 6 german_credit classif.knn7 cv 4 44.75 0.728 Hidden columns: resample_result
autoplot(bmr, measure = msr("classif.acc"))
Interpret the results
Interpret the plot. Which seems to work well given the task? Would you prefer a logistic regression over a KNN learner? ::: {.content-hidden unless-meta=“params.showsolution”} ::: {.callout-note collapse=“true”} ### Solution A of approx. 12 seems to perform best (in terms of accuracy). A too small underfits, a large one overfits. Not knowing the true , a logistic regression seems preferable. If is too small, the average performance of the logistic regression is much better. If is too large, the variance of the performance is much higher compared to the logistic regression. However, a KNN an optimal has a consistently high accuracy.
::: ::: # Extra: Parallelize your efforts
Benchmarking is embarassingly parallel. That means it is very easy to run the experiments of the benchmarking on different machines or cores. In many cases (not all!), this can significantly speed up computation time. We recommend to do this using the future::plan function when paralellizing mlr3 benchmarks.
Show Hint 1:
You need to use the plan function twice. Once to set up a multisession, then go back to parallel.
Show Hint 2:
library(future) plan(multisession) # your code plan(sequential)
# load the packages library(mlr3) library(future) library(future.apply) library(mlr3learners) library(kknn) # parallel plan plan(multisession) set.seed(100) # it is good practice to set a seed before bmr_par = benchmark(design)
INFO [19:03:07.101] [mlr3] Running benchmark with 116 resampling iterations INFO [19:03:07.219] [mlr3] Applying learner 'classif.log_reg' on task 'german_credit' (iter 1/4) INFO [19:03:07.326] [mlr3] Applying learner 'classif.log_reg' on task 'german_credit' (iter 2/4) INFO [19:03:07.428] [mlr3] Applying learner 'classif.log_reg' on task 'german_credit' (iter 3/4) INFO [19:03:07.533] [mlr3] Applying learner 'classif.log_reg' on task 'german_credit' (iter 4/4) INFO [19:03:07.633] [mlr3] Applying learner 'classif.knn3' on task 'german_credit' (iter 1/4) INFO [19:03:07.744] [mlr3] Applying learner 'classif.knn3' on task 'german_credit' (iter 2/4) INFO [19:03:07.865] [mlr3] Applying learner 'classif.knn3' on task 'german_credit' (iter 3/4) INFO [19:03:07.998] [mlr3] Applying learner 'classif.knn3' on task 'german_credit' (iter 4/4) INFO [19:03:08.119] [mlr3] Applying learner 'classif.knn4' on task 'german_credit' (iter 1/4) INFO [19:03:08.245] [mlr3] Applying learner 'classif.knn4' on task 'german_credit' (iter 2/4) INFO [19:03:08.269] [mlr3] Applying learner 'classif.knn4' on task 'german_credit' (iter 3/4) INFO [19:03:08.328] [mlr3] Applying learner 'classif.knn4' on task 'german_credit' (iter 4/4) INFO [19:03:08.384] [mlr3] Applying learner 'classif.knn5' on task 'german_credit' (iter 1/4) INFO [19:03:08.442] [mlr3] Applying learner 'classif.knn5' on task 'german_credit' (iter 2/4) INFO [19:03:08.499] [mlr3] Applying learner 'classif.knn5' on task 'german_credit' (iter 3/4) INFO [19:03:08.550] [mlr3] Applying learner 'classif.knn5' on task 'german_credit' (iter 4/4) INFO [19:03:08.616] [mlr3] Applying learner 'classif.knn6' on task 'german_credit' (iter 1/4) INFO [19:03:08.681] [mlr3] Applying learner 'classif.knn6' on task 'german_credit' (iter 2/4) INFO [19:03:08.727] [mlr3] Applying learner 'classif.knn6' on task 'german_credit' (iter 3/4) INFO [19:03:08.775] [mlr3] Applying learner 'classif.knn6' on task 'german_credit' (iter 4/4) INFO [19:03:08.795] [mlr3] Applying learner 'classif.knn7' on task 'german_credit' (iter 1/4) INFO [19:03:08.838] [mlr3] Applying learner 'classif.knn7' on task 'german_credit' (iter 2/4) INFO [19:03:08.881] [mlr3] Applying learner 'classif.knn7' on task 'german_credit' (iter 3/4) INFO [19:03:08.924] [mlr3] Applying learner 'classif.knn7' on task 'german_credit' (iter 4/4) INFO [19:03:08.966] [mlr3] Applying learner 'classif.knn8' on task 'german_credit' (iter 1/4) INFO [19:03:09.007] [mlr3] Applying learner 'classif.knn8' on task 'german_credit' (iter 2/4) INFO [19:03:09.048] [mlr3] Applying learner 'classif.knn8' on task 'german_credit' (iter 3/4) INFO [19:03:09.090] [mlr3] Applying learner 'classif.knn8' on task 'german_credit' (iter 4/4) INFO [19:03:09.131] [mlr3] Applying learner 'classif.knn9' on task 'german_credit' (iter 1/4) INFO [19:03:09.173] [mlr3] Applying learner 'classif.knn9' on task 'german_credit' (iter 2/4) INFO [19:03:09.215] [mlr3] Applying learner 'classif.knn9' on task 'german_credit' (iter 3/4) INFO [19:03:09.256] [mlr3] Applying learner 'classif.knn9' on task 'german_credit' (iter 4/4) INFO [19:03:09.298] [mlr3] Applying learner 'classif.knn10' on task 'german_credit' (iter 1/4) INFO [19:03:09.340] [mlr3] Applying learner 'classif.knn10' on task 'german_credit' (iter 2/4) INFO [19:03:09.384] [mlr3] Applying learner 'classif.knn10' on task 'german_credit' (iter 3/4) INFO [19:03:09.426] [mlr3] Applying learner 'classif.knn10' on task 'german_credit' (iter 4/4) INFO [19:03:09.467] [mlr3] Applying learner 'classif.knn11' on task 'german_credit' (iter 1/4) INFO [19:03:09.509] [mlr3] Applying learner 'classif.knn11' on task 'german_credit' (iter 2/4) INFO [19:03:09.550] [mlr3] Applying learner 'classif.knn11' on task 'german_credit' (iter 3/4) INFO [19:03:09.592] [mlr3] Applying learner 'classif.knn11' on task 'german_credit' (iter 4/4) INFO [19:03:09.633] [mlr3] Applying learner 'classif.knn12' on task 'german_credit' (iter 1/4) INFO [19:03:09.675] [mlr3] Applying learner 'classif.knn12' on task 'german_credit' (iter 2/4) INFO [19:03:09.717] [mlr3] Applying learner 'classif.knn12' on task 'german_credit' (iter 3/4) INFO [19:03:09.756] [mlr3] Applying learner 'classif.knn12' on task 'german_credit' (iter 4/4) INFO [19:03:09.799] [mlr3] Applying learner 'classif.knn13' on task 'german_credit' (iter 1/4) INFO [19:03:09.843] [mlr3] Applying learner 'classif.knn13' on task 'german_credit' (iter 2/4) INFO [19:03:09.885] [mlr3] Applying learner 'classif.knn13' on task 'german_credit' (iter 3/4) INFO [19:03:09.926] [mlr3] Applying learner 'classif.knn13' on task 'german_credit' (iter 4/4) INFO [19:03:09.968] [mlr3] Applying learner 'classif.knn14' on task 'german_credit' (iter 1/4) INFO [19:03:10.010] [mlr3] Applying learner 'classif.knn14' on task 'german_credit' (iter 2/4) INFO [19:03:10.051] [mlr3] Applying learner 'classif.knn14' on task 'german_credit' (iter 3/4) INFO [19:03:10.096] [mlr3] Applying learner 'classif.knn14' on task 'german_credit' (iter 4/4) INFO [19:03:10.138] [mlr3] Applying learner 'classif.knn15' on task 'german_credit' (iter 1/4) INFO [19:03:10.180] [mlr3] Applying learner 'classif.knn15' on task 'german_credit' (iter 2/4) INFO [19:03:10.219] [mlr3] Applying learner 'classif.knn15' on task 'german_credit' (iter 3/4) INFO [19:03:10.261] [mlr3] Applying learner 'classif.knn15' on task 'german_credit' (iter 4/4) INFO [19:03:10.305] [mlr3] Applying learner 'classif.knn16' on task 'german_credit' (iter 1/4) INFO [19:03:10.344] [mlr3] Applying learner 'classif.knn16' on task 'german_credit' (iter 2/4) INFO [19:03:10.386] [mlr3] Applying learner 'classif.knn16' on task 'german_credit' (iter 3/4) INFO [19:03:10.427] [mlr3] Applying learner 'classif.knn16' on task 'german_credit' (iter 4/4) INFO [19:03:10.472] [mlr3] Applying learner 'classif.knn17' on task 'german_credit' (iter 1/4) INFO [19:03:10.513] [mlr3] Applying learner 'classif.knn17' on task 'german_credit' (iter 2/4) INFO [19:03:10.555] [mlr3] Applying learner 'classif.knn17' on task 'german_credit' (iter 3/4) INFO [19:03:10.594] [mlr3] Applying learner 'classif.knn17' on task 'german_credit' (iter 4/4) INFO [19:03:10.636] [mlr3] Applying learner 'classif.knn18' on task 'german_credit' (iter 1/4) INFO [19:03:10.677] [mlr3] Applying learner 'classif.knn18' on task 'german_credit' (iter 2/4) INFO [19:03:10.719] [mlr3] Applying learner 'classif.knn18' on task 'german_credit' (iter 3/4) INFO [19:03:10.763] [mlr3] Applying learner 'classif.knn18' on task 'german_credit' (iter 4/4) INFO [19:03:10.804] [mlr3] Applying learner 'classif.knn19' on task 'german_credit' (iter 1/4) INFO [19:03:10.844] [mlr3] Applying learner 'classif.knn19' on task 'german_credit' (iter 2/4) INFO [19:03:10.886] [mlr3] Applying learner 'classif.knn19' on task 'german_credit' (iter 3/4) INFO [19:03:10.927] [mlr3] Applying learner 'classif.knn19' on task 'german_credit' (iter 4/4) INFO [19:03:10.969] [mlr3] Applying learner 'classif.knn20' on task 'german_credit' (iter 1/4) INFO [19:03:11.011] [mlr3] Applying learner 'classif.knn20' on task 'german_credit' (iter 2/4) INFO [19:03:11.053] [mlr3] Applying learner 'classif.knn20' on task 'german_credit' (iter 3/4) INFO [19:03:11.095] [mlr3] Applying learner 'classif.knn20' on task 'german_credit' (iter 4/4) INFO [19:03:11.137] [mlr3] Applying learner 'classif.knn21' on task 'german_credit' (iter 1/4) INFO [19:03:11.179] [mlr3] Applying learner 'classif.knn21' on task 'german_credit' (iter 2/4) INFO [19:03:11.223] [mlr3] Applying learner 'classif.knn21' on task 'german_credit' (iter 3/4) INFO [19:03:11.265] [mlr3] Applying learner 'classif.knn21' on task 'german_credit' (iter 4/4) INFO [19:03:11.307] [mlr3] Applying learner 'classif.knn22' on task 'german_credit' (iter 1/4) INFO [19:03:11.348] [mlr3] Applying learner 'classif.knn22' on task 'german_credit' (iter 2/4) INFO [19:03:11.391] [mlr3] Applying learner 'classif.knn22' on task 'german_credit' (iter 3/4) INFO [19:03:11.432] [mlr3] Applying learner 'classif.knn22' on task 'german_credit' (iter 4/4) INFO [19:03:11.474] [mlr3] Applying learner 'classif.knn23' on task 'german_credit' (iter 1/4) INFO [19:03:11.517] [mlr3] Applying learner 'classif.knn23' on task 'german_credit' (iter 2/4) INFO [19:03:11.558] [mlr3] Applying learner 'classif.knn23' on task 'german_credit' (iter 3/4) INFO [19:03:11.600] [mlr3] Applying learner 'classif.knn23' on task 'german_credit' (iter 4/4) INFO [19:03:11.642] [mlr3] Applying learner 'classif.knn24' on task 'german_credit' (iter 1/4) INFO [19:03:11.686] [mlr3] Applying learner 'classif.knn24' on task 'german_credit' (iter 2/4) INFO [19:03:11.727] [mlr3] Applying learner 'classif.knn24' on task 'german_credit' (iter 3/4) INFO [19:03:11.769] [mlr3] Applying learner 'classif.knn24' on task 'german_credit' (iter 4/4) INFO [19:03:11.811] [mlr3] Applying learner 'classif.knn25' on task 'german_credit' (iter 1/4) INFO [19:03:11.853] [mlr3] Applying learner 'classif.knn25' on task 'german_credit' (iter 2/4) INFO [19:03:11.895] [mlr3] Applying learner 'classif.knn25' on task 'german_credit' (iter 3/4) INFO [19:03:11.937] [mlr3] Applying learner 'classif.knn25' on task 'german_credit' (iter 4/4) INFO [19:03:11.979] [mlr3] Applying learner 'classif.knn26' on task 'german_credit' (iter 1/4) INFO [19:03:12.021] [mlr3] Applying learner 'classif.knn26' on task 'german_credit' (iter 2/4) INFO [19:03:12.063] [mlr3] Applying learner 'classif.knn26' on task 'german_credit' (iter 3/4) INFO [19:03:12.105] [mlr3] Applying learner 'classif.knn26' on task 'german_credit' (iter 4/4) INFO [19:03:12.149] [mlr3] Applying learner 'classif.knn27' on task 'german_credit' (iter 1/4) INFO [19:03:12.191] [mlr3] Applying learner 'classif.knn27' on task 'german_credit' (iter 2/4) INFO [19:03:12.233] [mlr3] Applying learner 'classif.knn27' on task 'german_credit' (iter 3/4) INFO [19:03:12.274] [mlr3] Applying learner 'classif.knn27' on task 'german_credit' (iter 4/4) INFO [19:03:12.316] [mlr3] Applying learner 'classif.knn28' on task 'german_credit' (iter 1/4) INFO [19:03:12.358] [mlr3] Applying learner 'classif.knn28' on task 'german_credit' (iter 2/4) INFO [19:03:12.400] [mlr3] Applying learner 'classif.knn28' on task 'german_credit' (iter 3/4) INFO [19:03:12.442] [mlr3] Applying learner 'classif.knn28' on task 'german_credit' (iter 4/4) INFO [19:03:12.483] [mlr3] Applying learner 'classif.knn29' on task 'german_credit' (iter 1/4) INFO [19:03:12.525] [mlr3] Applying learner 'classif.knn29' on task 'german_credit' (iter 2/4) INFO [19:03:12.567] [mlr3] Applying learner 'classif.knn29' on task 'german_credit' (iter 3/4) INFO [19:03:12.611] [mlr3] Applying learner 'classif.knn29' on task 'german_credit' (iter 4/4) INFO [19:03:12.653] [mlr3] Applying learner 'classif.knn30' on task 'german_credit' (iter 1/4) INFO [19:03:12.694] [mlr3] Applying learner 'classif.knn30' on task 'german_credit' (iter 2/4) INFO [19:03:12.736] [mlr3] Applying learner 'classif.knn30' on task 'german_credit' (iter 3/4) INFO [19:03:12.779] [mlr3] Applying learner 'classif.knn30' on task 'german_credit' (iter 4/4) INFO [19:03:12.798] [mlr3] Finished benchmark
plan(sequential)
We learnt how to set benchmark in mlr3. While we only looked at a single task and a single resampling, the procedure easily applies to more complex benchmarks with many tasks. Additionally, we learnt how to understand benchmark results. Last but not least, you may have parallelized your benchmark if you still had some time left.
Related