> two submissions have exactly the same AUC numbers

Hi,

A strange thing just happened: my two submissions have different models. But the AUC scores are exactly the same, even to the 10th digit after the decimal point... Their ROC curves are not exactly the same though.

I can't think of any possible explanations. Any idea what happened?

Thanks,
Ran

Posted by: rzhang229 @ Oct. 12, 2022, 9:28 p.m.

Hi Ran,
it is possible for 2 ROC curves to be different and have the same AUC but it is indeed a but odd that the results are identical to so many digits. How about the log-loss, are they the same as well? Log-loss is the performance metric that will be used to break ties amongst participants, if needed.

Posted by: kdrukker @ Oct. 12, 2022, 9:31 p.m.

I don't know how to find the log-loss for my previous submission. I believe they are quite different. I was wondering if there might be a bug somewhere that caused two submissions have the same AUC.

Posted by: rzhang229 @ Oct. 12, 2022, 9:37 p.m.

I am unsure which prior submission you are comparing the one from today to, but a previous submission of yours with AUC=0.65 had a slightly different 95% confidence interval for AUC (as seen in the png with the ROC plot) than the one from today so I am inclined to think that it is a coincidence not a bug.

Posted by: kdrukker @ Oct. 12, 2022, 9:40 p.m.

Hi Ran,

I'm running submission_1003 and submission_1012_HF_model_ensemble again just to see what comes out. You can see them in your submission area. I will delete them if they match their corresponding previous runs. If not then we should sort of figure out why and delete the ones you think are bad runs. There is a limit of 10 for this phase so I don't want to take up your 10 possible submissions, but I want to check out why the results were so similar between these two.

Posted by: challenge-organizer @ Oct. 13, 2022, 8:49 p.m.

Hi,
Thanks for the help. Would you mind deleting the last two submissions? They are indeed the same as my previous ones.
Thanks,
Ran

Posted by: rzhang229 @ Oct. 14, 2022, 9:35 p.m.
Post in this thread