Problem with cyrillic symbols

[nickla]

I use postgresql as database server. i created database:

CREATE DATABASE qtaxi
WITH OWNER = postgres
ENCODING = 'UTF8'
TABLESPACE = pg_default
LC_COLLATE = 'ru_RU.UTF-8'
LC_CTYPE = 'ru_RU.UTF-8'
CONNECTION LIMIT = -1;

When i try to SELECT cyrillic strings from database using qxorm i receive '???????' instead, but everything is ok when i try to use QSqlQuery.

I looked into postgresql log file and see that pgadmin and qxorm make same request before select statement:

select version()
SET CLIENT_ENCODING TO 'UNICODE'
SET DATESTYLE TO 'ISO'

This means that problem is not in QSqlQuery or QSqlDatabase. It is in qxorm.

I tried to find out where qxorm fetches data from query into data structure but failed. Can you provide me with this information?

[qxorm]

Hi,

When i try to SELECT cyrillic strings from database using qxorm i receive '???????' instead, but everything is ok when i try to use QSqlQuery.

Waouhhhh, really strange !!!

I tried to find out where qxorm fetches data from query into data structure but failed. Can you provide me with this information?

Take a look at this file : ./inl/QxDao/QxSqlQueryHelper_FetchAll.inl
And put a breakpoint into resolveOutput() methods...

[nickla]

Solved.

Added following into client and server main.cpp:

Code: Select all

#include <QTextCodec>

QTextCodec* codec = QTextCodec::codecForName("UTF-8");
QTextCodec::setCodecForLocale(codec);
QTextCodec::setCodecForCStrings(codec);
QTextCodec::setCodecForTr(codec);


[qxorm]

Thx for providing the solution

[nickla]

This is not right solution b\c there is wrong encoding in database now. I see in pgadmin '?????' but everything is ok in client program. I`ll post solution later.

[remico]

All appearances, I have same trouble.
I try to create a client-server app.
sqlite DB encoding is UTF-8.
When I use "serialization_xml" I receive "?????" instead of cyrillic string on the client side.
But note:
- immediately after "fetch" operation on the server side the class's field of QString type contains normal cyrillic symbols.
- but on the client side - after de-/serialization - the QString-field contains "?????" symbols.

And note when I use "serialization_binary", all ok with cyrillic symbols.

I have found following info in the boost docs:

The current library defines in 3 formats (text, binary, and XML), wide and narrow characters, an attempts to be portable between compiler libraries.
...
- character XML archives (i.e. xml_oarchive) will produce XML output with characters encoded according to the current stream locale.
- wide character XML archives (i.e. xml_woarchive) will produce files encoded in UTF-8.

This character encoding is implemented by changing the locale of the i/o stream used by an archive when the archive is constructed, the stream local is changed back to its original value. This action can be overridden by specifying boost::archive::no_codecvt when the archive is opened. In this case, the stream locale will not be changed by the serialization library.
...

http://www.boost.org/doc/libs/1_51_0/li ... arencoding

I suppose this can be the cause of our trouble. But I don't know, how to check my guess. Can you tell me if my guess is correct?

[qxorm]

You could try to change the boost serialization engine (to use "wide" serialization or "polymorphic" serialization), it's very easy to do, just open the file "./include/QxCommon/QxConfig.h" and choose what serialization you want :

Code: Select all

#define _QX_SERIALIZE_POLYMORPHIC            0

#define _QX_SERIALIZE_BINARY                 (! _QX_SERIALIZE_POLYMORPHIC && 1)
#define _QX_SERIALIZE_TEXT                   (! _QX_SERIALIZE_POLYMORPHIC && 0)
#define _QX_SERIALIZE_XML                    (! _QX_SERIALIZE_POLYMORPHIC && 1)
#define _QX_SERIALIZE_PORTABLE_BINARY        (! _QX_SERIALIZE_POLYMORPHIC && 0)

/* -- Link error with VC++ 9.0 => Qt uses "-Zc:wchar_t-" option to compile and boost serialization library is compiled without this option -- */
#define _QX_SERIALIZE_WIDE_BINARY            (! _QX_SERIALIZE_POLYMORPHIC && 0)
#define _QX_SERIALIZE_WIDE_TEXT              (! _QX_SERIALIZE_POLYMORPHIC && 0)
#define _QX_SERIALIZE_WIDE_XML               (! _QX_SERIALIZE_POLYMORPHIC && 0)

Note : you must rebuild QxOrm library and your project to take into account your modification.

[remico]

really thanks for your advice.
But I tried to use "wide_xml" still yesterday. And I got very strange server's behavior (it crashed with fallowing message:

libs/serialization/src/basic_serializer_map.cpp:85: const boost::archive::detail::basic_serializer* boost::archive::detail::basic_serializer_map::find(const boost::serialization::extended_type_info&) const: Assertion `it != m_map.end()' failed.
The program has unexpectedly finished.

at any client request). So I thought that this method won't work for me for some strange reason

But today I finally found the reason of the crashes.
Everything was obscenely easy, however as always.
I rebuild QxOrm lib with modified QxConfig.h file ("wide_xml" option enabled) in separate directory, and then I just copied the dll into my project dir.
But I FORGOT to modify QxConfig.h file in my project's "INCLUDEPATH" directory.
I know I am really stupid )))

The "cyrillic symbols and serialization" problem SOLVED now. Everything works fine.

[qxorm]

The "cyrillic symbols and serialization" problem SOLVED now. Everything works fine.

Good news !

[nickla]

I have problem with this.

I changed QxConfig.h like this:

Code: Select all

#define _QX_SERIALIZE_POLYMORPHIC            0

#define _QX_SERIALIZE_BINARY                 (! _QX_SERIALIZE_POLYMORPHIC && 1)
#define _QX_SERIALIZE_TEXT                   (! _QX_SERIALIZE_POLYMORPHIC && 0)
#define _QX_SERIALIZE_XML                    (! _QX_SERIALIZE_POLYMORPHIC && 1)
#define _QX_SERIALIZE_PORTABLE_BINARY        (! _QX_SERIALIZE_POLYMORPHIC && 0)

/* -- Link error with VC++ 9.0 => Qt uses "-Zc:wchar_t-" option to compile and boost serialization library is compiled without this option -- */
#define _QX_SERIALIZE_WIDE_BINARY            (! _QX_SERIALIZE_POLYMORPHIC && 0)
#define _QX_SERIALIZE_WIDE_TEXT              (! _QX_SERIALIZE_POLYMORPHIC && 0)
#define _QX_SERIALIZE_WIDE_XML               (! _QX_SERIALIZE_POLYMORPHIC && 1)


And in projects:

Code: Select all

qx::service::QxConnect::getSingleton()->setSerializationType(qx::service::QxConnect::serialization_wide_xml);


After that i tried to rebuild all my projects and got this error:

Code: Select all

-lQxOrmd -lqxServiceServerd -lboost_serialization -lQtSql -lQtXml -lQtNetwork -lQtCore -lpthread
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_iarchive<boost::archive::xml_wiarchive>::load_override(boost::archive::object_id_type&, int)'
make[1]: Leaving directory `/home/kna/qxorm/qxTaxiServer'
make: Leaving directory `/home/kna/qxorm/qxTaxiServer'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_end(char const*)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::end_preamble()'
../debug/libQxOrmd.so: undefined reference to `boost::archive::xml_wiarchive_impl<boost::archive::xml_wiarchive>::~xml_wiarchive_impl()'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_text_oprimitive<std::basic_ostream<wchar_t, std::char_traits<wchar_t> > >::~basic_text_oprimitive()'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_override(boost::archive::class_id_optional_type const&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_override(boost::archive::object_reference_type const&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_iarchive<boost::archive::xml_wiarchive>::load_start(char const*)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::detail::archive_serializer_map<boost::archive::xml_wiarchive>::insert(boost::archive::detail::basic_serializer const*)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::xml_wiarchive_impl<boost::archive::xml_wiarchive>::xml_wiarchive_impl(std::basic_istream<wchar_t, std::char_traits<wchar_t> >&, unsigned int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_override(boost::archive::tracking_type const&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_override(boost::archive::object_id_type const&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::detail::archive_serializer_map<boost::archive::xml_woarchive>::find(boost::serialization::extended_type_info const&)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_iarchive<boost::archive::xml_wiarchive>::load_override(boost::archive::class_id_type&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::xml_wiarchive_impl<boost::archive::xml_wiarchive>::load(std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >&)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::xml_woarchive_impl<boost::archive::xml_woarchive>::save(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::detail::archive_serializer_map<boost::archive::xml_woarchive>::insert(boost::archive::detail::basic_serializer const*)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_override(boost::archive::version_type const&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_iarchive<boost::archive::xml_wiarchive>::load_end(char const*)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::~basic_xml_oarchive()'
../debug/libQxOrmd.so: undefined reference to `boost::archive::xml_woarchive_impl<boost::archive::xml_woarchive>::save(std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> > const&)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_iarchive<boost::archive::xml_wiarchive>::load_override(boost::archive::tracking_type&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::detail::archive_serializer_map<boost::archive::xml_wiarchive>::erase(boost::archive::detail::basic_serializer const*)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::detail::archive_serializer_map<boost::archive::xml_woarchive>::erase(boost::archive::detail::basic_serializer const*)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_override(boost::archive::class_id_type const&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::detail::archive_serializer_map<boost::archive::xml_wiarchive>::find(boost::serialization::extended_type_info const&)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::xml_woarchive_impl<boost::archive::xml_woarchive>::xml_woarchive_impl(std::basic_ostream<wchar_t, std::char_traits<wchar_t> >&, unsigned int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_start(char const*)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::xml_wiarchive_impl<boost::archive::xml_wiarchive>::load_override(boost::archive::class_name_type&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_override(boost::archive::class_name_type const&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_oarchive<boost::archive::xml_woarchive>::save_override(boost::archive::class_id_reference_type const&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::basic_xml_iarchive<boost::archive::xml_wiarchive>::load_override(boost::archive::version_type&, int)'
../debug/libQxOrmd.so: undefined reference to `boost::archive::xml_wiarchive_impl<boost::archive::xml_wiarchive>::load(std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
collect2: ld returned 1 exit status
make[1]: *** [../debug/qxTaxiServerd] Error 1
make: *** [debug] Error 2


There is no boost_archive lib in ubuntu. How can i fix it?

Also, there is not boost_serialization-mt-d library in my system. I changed this in QxOrm on boost_serialization-mt.