pcrov/unicode

Miscellaneous Unicode utility functions

Unicode

, _(*1)

Miscellaneous Unicode utility functions., _(*2)

Functions

Namespace pcrov\Unicode., _(*3)

surrogate_pair_to_code_point(int $high, int $low): int

Translates a UTF-16 surrogate pair into a single code point. [Wikipedia's UTF-16 article]0 explains what this is fairly well., _(*4)

utf8_find_invalid_byte_sequence(string $string): ?int

Returns the position of the first invalid byte sequence or null if the input is valid., _(*5)

utf8_get_invalid_byte_sequence(string $string): ?string

Returns the first invalid byte sequence or null if the input is valid., _(*6)

utf8_get_state_machine(): array

Provides a state machine letting you walk a (potentially endless) UTF-8 sequence byte by byte., _(*7)

It is in the form of [byte => [valid next byte => ...,], ...], _(*8)

Example use:, _(*9)

function utf8_generate_all_code_points(): string
{
    $generator = function (array $machine, string $buffer = "") use (&$generator) {
        // Completed a UTF-8 encoded code point.
        if ($buffer !== "" && isset($machine["\x0"])) {
            return $buffer;
        }

        $out = "";
        foreach ($machine as $byte => $next) {
            $out .= $generator($next, $buffer . $byte);
        }

        return $out;
    };

    return $generator(utf8_get_state_machine());
}

utf8_validate(string $string): bool

Does what it says on the box., _(*10)

Data

The test/data directory holds two files containing all possible UTF-8 encoded characters. All 1,112,064 of them. One as plain text, the other as json. These are not included in packaged stable releases but can be generated with the example utf8_generate_all_code_points() function above (returns the plain text string.), _(*11)

Excerpts from the [Unicode 10.0.0 standard][1]:

Recreated here for ease of reference. Nobody likes PDFs., _(*12)

Table 3-6. UTF-8 Bit Distribution

Table 3-7. Well-Formed UTF-8 Byte Sequences