Software Programming

Kunuk Nykjaer

Probability of getting certified by guessing

leave a comment »


I am curious about the probability of passing a certification exam by guessing.
Therefore I made a quick implementation to calculate some probabilities.

In this example I will use
Website .NET Developer Certification for Sitecore CMS as the test experiment.

The exam is a multiple-choice format with m possible answers where one answer is correct. Let’s assume m is 3.

In total there are N questions and you have to get at least p to pass the exam. Let’s say N is 40 and p is 28.
You have 55 minutes to answer all the questions (82.5 seconds per answer).
Almost always two possible answers look very similar and one is usually easier to spot as a wrong candidate from the other two.

The math example of how to calculate this kind of probabilities can be read here.

The probability of passing the exam by only guessing
AtLeastProbability(0.33, 40, 28) ~ 0%;

The probability of passing the exam 
Knowing 10 answers and guessing the rest minimum 18 correct
AtLeastProbability(0.33, 30, 18) ~ 0%;

Knowing 20 answers and guessing the rest minimum 8 correct
AtLeastProbability(0.33, 20, 8) ~ 34%;

Knowing 22 answers and guessing the rest minimum 6 correct
AtLeastProbability(0.33, 18, 6) ~ 59%;


As you can see, even if you know 22 correct answers out of 40. Then the chance of passing is only about 59% if you guess the rest of the questions where you need to guess 6 correct. That is not promising if your strategy is to pass by guessing.

Now let’s assume you will be able to spot one wrong candidate and are left with 50% of correct guessing. The calculations are now:

The probability of passing the exam by only guessing
AtLeastProbability(0.5, 40, 28) ~ 1%;

The probability of passing the exam 
Knowing 10 answers and guessing the rest minimum 18 correct
AtLeastProbability(0.5, 30, 18) ~ 18%;

Knowing 20 answers and guessing the rest minimum 8 correct
AtLeastProbability(0.5, 20, 8) ~ 87%;

Knowing 22 answers and guessing the rest minimum 6 correct
AtLeastProbability(0.5, 18, 6) ~ 95%;


If you are able to spot one wrong candidate, then you should at least know 22 correct answers out of 28 to pass the exam.

The strategy of passing a certification exam by guessing seems like a failing strategy.

Out of the box this program will calculate the probability of having at least 3 successes out of 5 where the success probability is 33%.
AtLeastProbability(0.33, 5, 3) ~ 21%;

Program.cs

using System;
using System.Collections.Generic;
using System.Diagnostics;

/// <summary>
/// Author: Kunuk Nykjaer
/// Binomial probability
/// At least x success out of n tries with probability p.
/// </summary>
public class Program
{
    public static void Main(string[] args)
    {
        const double T = 1.0 / 3; // odds of success        
        const int knownCorrect = 0;
        long n = 5; // total of chances
        long atleast = 3; // must have at least this number of success        
        n -= knownCorrect;
        atleast -= knownCorrect;
        AtLeastProbability(T, n, atleast);
    }

    // http://www.regentsprep.org/Regents/math/algtrig/ATS7/BLesson2.htm
    static void AtLeastProbability(double T, long n, long atleast)
    {
        if (T < 0 || T > 1 || n < 0 || atleast < 0 || atleast > n)
            throw new ApplicationException("AtLeastProbability invalid");

        double F = 1.0 - T; // odds of failure
        Console.WriteLine(string.Format("AtLeastProbability({0}, {1}, {2}, {3})", 
            Math.Round(T,2), Math.Round(F,2), n, atleast));
        var stopwatch = new System.Diagnostics.Stopwatch();
        stopwatch.Start();

        double odds = 0;
        for (long r = atleast; r <= n; r++)
            odds += C(n, r) * Math.Pow(T, r) * Math.Pow(F, n - r);

        stopwatch.Stop();
        odds = Math.Round(odds * 100, 3); // %
        Console.WriteLine(string.Format(
            "odds: {0}%\nTime elapsed msec: {1}", odds, stopwatch.ElapsedMilliseconds));
        Console.WriteLine("press a key to exit"); Console.ReadKey();
    }


    // Binomial    
    // using tricks to avoid too big numbers in factorisation, 
    // runtime might be slow for really big numbers
    // nCr see http://en.wikipedia.org/wiki/Binomial_probability
    static long C(long n, long r)
    {
        if (r < 0 || n < r) throw new ApplicationException("C(n,r) invalid n,r");
        if (n <= 1) return 1;

        long bigr = n - r;
        if (bigr < r) bigr = r;

        var uppers = new System.Collections.Generic.List<long>();
        var lowers = new System.Collections.Generic.List<long>();

        for (long i = n; i > bigr; i--) uppers.Add(i);

        for (long i = n - bigr; i > 1; i--) lowers.Add(i);

        // O(n^2)
        // trick to reduce big numbers, datatype long has a limit
        for (int i = 0; i < uppers.Count; i++)
        {
            for (int j = 0; j < lowers.Count; j++)
            {
                long u = uppers[i];
                long l = lowers[j];
                long gcd = Gcd(u, l);
                if (gcd <= 1) continue;

                // reduce number value
                uppers[i] = u / gcd;
                lowers[j] = l / gcd;
            }
        }

        long upper = 1;
        long lower = 1;
        foreach (var i in lowers) lower *= i;
        foreach (var i in uppers) upper *= i;        
       
        //long datatype has a limit
        if (upper < 0 || lower < 0){           
            throw new ApplicationException("C(n,r) negative, datatype limit issue"); 
        }

        long result = upper / lower;
        return result;
    }

    // greatest common divisor
    public static long Gcd(long a, long b)
    {
        return b == 0 ? a : Gcd(b, a % b);
    }
}
Advertisements

Written by kunuk Nykjaer

November 12, 2011 at 6:08 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: