String Literal Tokens

Tokenizing Strings

Real languages have more than numbers. Let's add string literals — text enclosed in double quotes:

"hello"   =>  { type: "STRING", value: "hello" }

Scanning a String

When we encounter a ", we consume characters until we find the closing ":

if (input[i] === '"') {
    i++; // skip opening quote
    let str = "";
    while (i < input.length && input[i] !== '"') {
        str += input[i];
        i++;
    }
    i++; // skip closing quote
    tokens.push({ type: "STRING", value: str });
}

The value stores the string contents without the surrounding quotes.

Identifiers

We also need identifiers — names like print or x. An identifier starts with a letter and continues with letters or digits:

if (isAlpha(input[i])) {
    let name = "";
    while (i < input.length && isAlphaNumeric(input[i])) {
        name += input[i];
        i++;
    }
    tokens.push({ type: "IDENT", value: name });
}

Your Task

Write a tokenize(input) function that handles: numbers (NUMBER), strings in double quotes (STRING), identifiers (IDENT), + (PLUS), whitespace (skip), and EOF.