unicode issue

[veksha]

Hi, i have an issue when ident contains non-latin characters.
for example, one character Ы will be parsed as two characters: Р«

[glipari]

Hello, I am sorry, for the moment there is no Unicode support, just plain ASCII. Introducing Unicode should not too difficult, however, but at the moment I have no time to work on this project.
I may consider Unicode support for the next version, though.
Can you please provide one or more test cases? Thanks in advance.

Best regards,

[veksha]

here is my test:

unicode_test.cpp

#include <sstream>
#include <tinyparser.hpp>
using namespace tipa;
using namespace std;

struct Record {
    string name, unlock, message;
    void clear() { name = ""; unlock = ""; message = ""; }
};

int main()
{
    const token tk_ident = create_lib_token("^[^\\d\\W][\\w\\s\\.,!?:\\\\]*");
    
    rule unlock  =   rule("unlock") >> rule(":") >> rule("\"") >> rule(tk_ident) >> rule("\"");
    rule message =  rule("message") >> rule(":") >> rule("\"") >> rule(tk_ident) >> rule("\"");
    rule property = unlock | message;
    rule button = rule("\"") >> rule(tk_ident) >> rule("\"") >> rule('{') >> *property >> rule('}');
    rule root = *button;

    vector<Record> records;
    Record temp;

     unlock.read_vars(temp.unlock);
    message.read_vars(temp.message);
    button.set_action([&temp, &records](parser_context &pc) {
            vector<string> v;
            pc.collect_tokens(1, back_inserter(v));
            temp.name = v[0];
            records.push_back(temp);
            temp.clear();
        });

    stringstream s(R"(
        "door ascii" {
            unlock: "silver key"
            message: "This place is unknown to you. Would you like to enter?"
        }
        "door ascii_2" {
            unlock: "test key"
            message: "test"
        }
        "door unicode" {
            unlock: "silver key"
            message: "Это место не изведано. Желаете войти?"
        }
    )");
    
    parser_context pc;
    pc.set_stream(s);

    bool f = false;
    try { f = parse_all(root, pc); }
    catch(parse_exc exc) { cout << "exception caught" << endl; cout << exc.what() << endl; f = false; }

    cout << "Parser status : " << boolalpha << f << endl;
    if (!f) { cout << pc.get_formatted_err_msg() << endl; }
    
    for (auto x: records)
        cout << "> " << x.name << ", unlock: " << x.unlock << ", message: " << x.message << endl;     
}

                                                    /*
g++ unicode_test.cpp tinyparser.a -Itipa/src && a.exe
                                                    */

[glipari]

Thank you :) I should have some free time next week, let's see what I can do.