Q-Learning Updates

Q-learning learns how good each action is in each state, Q(s, a), from experience alone. After taking action a in state s, receiving reward and landing in state s':

target = reward, if the episode ended (done = 1)
target = reward + gamma × max over a' of Q(s', a'), otherwise
Q(s, a) ← Q(s, a) + alpha × (target - Q(s, a))

Start with every Q value at 0 and apply the transitions in order.

Implement q_learning(num_states, num_actions, transitions, alpha, gamma).

Input

num_states, num_actions: The table size.

transitions: One row [state, action, reward, next_state, done] per step.

alpha: The learning rate.

gamma: The discount factor.

Output

Return the Q table as a matrix with one row per state and one column per action.

Constraints

1 ≤ num_states, num_actions ≤ 100, 0 ≤ number of transitions ≤ 10,000

-100 ≤ reward ≤ 100, done is 0 or 1

0 < alpha ≤ 1, 0 ≤ gamma ≤ 1

Examples

Example 1

Input:
alpha = 0.5
gamma = 0.9
num_states = 2
num_actions = 2
transitions = [
    [0, 1, 0, 1, 0],
    [1, 0, 10, 0, 1]
]

Expected output:
[
    [0.0, 0.0],
    [5.0, 0.0]
]

Explanation

The first step finds no reward yet: target = 0 + 0.9 × max(Q[1]) = 0, so Q[0][1] stays 0. The second step ends the episode with reward 10: Q[1][0] = 0 + 0.5 × (10 - 0) = 5.

Example 2

Input:
alpha = 0.5
gamma = 0.9
num_states = 2
num_actions = 2
transitions = [
    [1, 0, 10, 0, 1],
    [0, 1, 0, 1, 0]
]

Expected output:
[
    [0.0, 2.25],
    [5.0, 0.0]
]

Explanation

Same steps in the other order. Q[1][0] becomes 5 first, so the step from state 0 now sees it: target = 0 + 0.9 × 5 = 4.5 and Q[0][1] = 0.5 × 4.5 = 2.25. The reward has flowed back one state.

References

Run the sample tests to check your code without submitting.

Example 1 Not run

Input:

alpha = 0.5
gamma = 0.9
num_states = 2
num_actions = 2
transitions = [
    [0, 1, 0, 1, 0],
    [1, 0, 10, 0, 1]
]

Expected output:

[
    [0.0, 0.0],
    [5.0, 0.0]
]

Your output

Example 2 Not run

Input:

alpha = 0.5
gamma = 0.9
num_states = 2
num_actions = 2
transitions = [
    [1, 0, 10, 0, 1],
    [0, 1, 0, 1, 0]
]

Expected output:

[
    [0.0, 2.25],
    [5.0, 0.0]
]

Your output

4 hidden tests also run when you submit.

Online C# Compiler & Debugger

Write, compile, run, and debug C# code online. Set breakpoints, step into or over code, and inspect variables and nested objects in a full IDE with multiple files, saved workspaces, sharing, and live collaboration—no setup required.

Version

Pre-installed Libs

🚀 350,747 total executions (1 this month)

Output

Practice C# with coding challenges

Solve hands-on problems right in this editor: run the sample tests, submit, and get an AI code review.

All C# challenges

Easy

Average of Three

Average three whole numbers.

You'll learn ArithmeticDivision

BeginnerLearning-pathVariables

View solutions (1)

Easy

Rectangle Perimeter

Compute the perimeter of a rectangle from its sides.

You'll learn ArithmeticReturn values

BeginnerLearning-pathVariables

View solutions (1)

Medium

One Generation of a Genetic Algorithm

Breed the next generation of bit strings with tournament selection, single-point crossover and mutation.

You'll learn Genetic algorithmsEvolutionary search

OptimizationSimulationai

No solutions yet

Easy

Epsilon-Greedy Multi-Armed Bandit

Balance exploration and exploitation on a multi-armed bandit with epsilon-greedy action selection.

You'll learn Explore vs exploitBandits

Reinforcement learningaimachine learning

No solutions yet

Easy

Markov Chain State Distribution

Propagate a probability distribution through a Markov chain's transition matrix for a number of steps.

You'll learn Markov chainsProbability

Simulationaimath

No solutions yet

Medium

Value Iteration

Compute how valuable each state of a Markov decision process is by repeatedly applying the Bellman optimality update.

You'll learn Bellman equationMDPs

Reinforcement learningaimachine learning

No solutions yet

Medium

Alpha-Beta Pruning

Evaluate a game tree with minimax and alpha-beta pruning, and count how many leaves the search really looks at.

You'll learn Alpha-beta pruningGame search

RecursionTreesai

No solutions yet

Medium

Minimax Tic-Tac-Toe

Pick the perfect tic-tac-toe move by searching the whole game tree with minimax.

You'll learn MinimaxGame trees

Recursionai

No solutions yet

Medium

A* Search on a Weighted Grid

Find the cheapest path across a grid of terrain costs with A*, the search algorithm behind game and robot pathfinding.

You'll learn A* searchHeuristics

GraphsGridsShortest-pathai

No solutions yet

Medium

Temperature and Top-K Sampling

Choose the next token like a language model does: keep the top k logits, apply temperature, and sample.

You'll learn Text generationSampling

aimachine learning

No solutions yet

Hard

Multi-Head Self-Attention

Project the input into queries, keys and values, run attention in several heads in parallel, and combine them.

You'll learn Multi-head attentionMatrix projections

aimachine learning

No solutions yet

Medium

Layer Normalization

Normalize each row to zero mean and unit variance, then scale and shift it, as every Transformer block does.

You'll learn Layer normTransformers

Neural networksaimachine learning

No solutions yet

🔷 About This C# / .NET Online Executor

The CodeUtility C# Executor lets you write and run real C# code directly in your browser - no installation, Visual Studio setup, or .NET SDK required. It’s powered by a secure sandbox that supports the .NET runtime and modern C# language versions, including C# 9 and later.

This tool uses dotnet-script under the hood to compile and execute your code with true .NET behavior, allowing you to experiment with both basic and advanced C# features like LINQ, async/await, pattern matching, and generics.

You can test simple scripts, functions, and logic flows, or explore concepts like classes, structs, inheritance, and interfaces - all in a clean, distraction-free coding environment.

Whether you’re learning C#, testing algorithms, or experimenting with .NET features, the CodeUtility C# Executor gives you a fast, secure, and reliable way to run C# code instantly from any browser.

More than a code runner

Use a multi-file editor, an isolated execution terminal, runtime-specific libraries, persistent workspaces, sharing, and real-time collaboration from the same executor.

How to use this executor?

Choose a runtime, open or create a file, write your code, and select Run. Output appears in the terminal, while signed-in users can keep files in a workspace, install libraries, and collaborate through shares.

Select a version and edit the current file or open another workspace file.
Run the code, provide input in the terminal when requested, and review saved run history.
Use libraries, workspace tools, sharing, and chat when your task needs more than one snippet.

Start coding

VersionSelect a runtime before running. Installed libraries belong to the selected language version.
Open files lists unopened text files. Select one to open it in an available editor tab.
Run executes the active file with its workspace available to the program.
Stop ends the current execution before its time limit.
Advanced Mode opens the IDE layout with a file explorer and movable, resizable terminal.

Workspace and libraries

New file creates a file in the selected parent directory and opens it in the editor.
New folder creates a directory, up to the supported nesting depth.
Upload files adds one or more safe files to a directory.
Download saves a file or a zipped directory to your device.
Copy duplicates a workspace file or directory.
Rename changes a file name and updates its open tab and history.
Move relocates an item; files and directories can also be dragged.
Delete removes an item after confirmation and closes its open editor tab.
Pre-installed Libs shows packages already included in this runtime.
Add Lib installs a private package for your account and selected runtime.

Terminal and assistance

Run history opens previous and next saved outputs.
Copy output copies the current terminal text.
Clear output removes all saved run outputs after confirmation.
Explain summarizes the active code.
Recents opens recent explanations and results.
Settings changes fonts, themes, and visible toolbar buttons.

Sharing and collaboration

Share publishes the current file, a directory, or the whole workspace as a snapshot or live workspace. You control link access and who may save changes.
Logs records saved collaborator actions in a shared workspace.
Share chat lets signed-in participants message, mention people, share code blocks, and see online or typing members.
Live collaboration shows collaborator cursors and saved file updates. View-only users can load the latest version or keep temporary edits.

Quotas and limits

Execution time is limited to 20 seconds per server run. Use Stop when you want to end it sooner.
Workspace storage allows 100 files, 100 directories, and 10 MiB total for each signed-in user workspace and language.
Files and folders are limited to 1 MiB per file and 5 nested directory levels. Files generated by your code follow the same quota.
Private libraries are limited to 10 installed packages for each account and runtime.
Share snapshots can contain up to 100 files, 1 MiB per file, and 2 MiB total.
Share invitations support up to 25 people per share, with access and saving permissions managed separately.
Share chat allows 2,000 characters per message and 10,000 messages per share.
Quota enforcement also applies while code is running. A write that exceeds file count, size, storage, or nesting limits is rejected safely.

Good to know Only run code and install packages you trust. Execution is isolated and limited, but your program can still modify files inside a writable workspace up to its enforced quota.

💡 C# Basics & Examples You Can Try Above

1. Declaring Variables and Constants

C# uses strongly-typed variable declarations. Use const for compile-time constants and readonly for runtime constants.

int age = 30;
double pi = 3.14159;
char grade = 'A';
string name = "Alice";
bool isActive = true;

// Constants
const int MaxUsers = 100;
const string Company = "CodeUtility";

2. Conditionals (if / switch)

Use if, else if, and switch for control flow.

int x = 2;
if (x == 1)
{
    Console.WriteLine("One");
}
else if (x == 2)
{
    Console.WriteLine("Two");
}
else
{
    Console.WriteLine("Other");
}

switch (x)
{
    case 1:
        Console.WriteLine("One");
        break;
    case 2:
        Console.WriteLine("Two");
        break;
    default:
        Console.WriteLine("Other");
        break;
}

3. Loops

C# supports for, while, and foreach loops.

for (int i = 0; i < 3; i++)
{
    Console.WriteLine(i);
}

int n = 3;
while (n > 0)
{
    Console.WriteLine(n);
    n--;
}

4. Arrays

Arrays store fixed-size collections of elements of the same type.

int[] numbers = { 10, 20, 30 };
Console.WriteLine(numbers[1]);

5. List Manipulation

Use List<T> for dynamic collections.

List<int> nums = new List<int> { 1, 2, 3 };
nums.Add(4);
nums.Remove(2);

foreach (int n in nums)
{
    Console.Write(n + " ");
}

6. Console Input/Output

Use Console.WriteLine and Console.ReadLine for basic I/O.

Console.Write("Enter your name: ");
string name = Console.ReadLine();
Console.WriteLine($"Hello, {name}!");

7. Functions

Define methods using a return type, name, and parameters.

int Add(int a, int b)
{
    return a + b;
}

Console.WriteLine(Add(3, 4));

8. Dictionaries

Dictionary<TKey, TValue> stores key-value pairs.

Dictionary<string, int> ages = new Dictionary<string, int>();
ages["Alice"] = 30;
Console.WriteLine(ages["Alice"]);

9. Exception Handling

Handle runtime errors using try, catch, and finally.

try
{
    throw new Exception("Something went wrong");
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
}

10. File I/O

Use File and StreamReader/StreamWriter for file operations.

File.WriteAllText("file.txt", "Hello File");
string text = File.ReadAllText("file.txt");
Console.WriteLine(text);

11. String Manipulation

C# strings support methods like Length, Substring, Contains.

string message = "Hello World";
Console.WriteLine(message.Length);
Console.WriteLine(message.Substring(0, 5));
Console.WriteLine(message.Contains("World"));

12. Classes & Objects

C# supports object-oriented programming through classes and objects.

class Person
{
    public string Name;
    public Person(string name) => Name = name;
    public void Greet() => Console.WriteLine($"Hi, I'm {Name}");
}

Person p = new Person("Alice");
p.Greet();