Postgres QVariant Serialization

Open discussion on QxOrm library

Postgres QVariant Serialization

Postby Kasuax » Fri Aug 03, 2018 9:19 pm

We've turned off BOOST Serialization and are left with the stock QDataStream serialization.
Postgres by default will store BYTEA as hex because apparently that is the preferered format. I doesn't appear that the qxlibrary does anything special outgoing and this implicitly happens when the write happens. Reading and deserialization has become a big problem because the data is overlooked because it is not decoded from hex before it goes back through this function.

Code: Select all
template <class T>
inline qx_bool from_byte_array(T & obj, const QByteArray & data, unsigned int flags = 1 /* boost::archive::no_header */)
{
   Q_UNUSED(flags);
   qx_bool result = false;
   if (data.isEmpty()) { return qx_bool(false, "input binary data is empty"); }
   QDataStream stream(data);
   quint32 magic = 0; stream >> magic;
   if (magic != 9438) { return qx_bool(false, "input binary data is not valid"); }
   try { stream >> obj; result = true; }
   catch (const std::exception & e) { result.setDesc(QString("deserialization error '%ERR%'").replace("%ERR%", e.what())); }
   catch (...) { result.setDesc(QString("deserialization error '%ERR%'").replace("%ERR%", "unknown error")); }
   if (! result.getDesc().isEmpty()) { QString msg = result.getDesc(); qDebug("[QxOrm] qx::serialization::qt::from_byte_array() : %s", qPrintable(msg)); }
   return result;
}


The function before this looks like we can setup format flags in the orm so that we can handle this use case.

Code: Select all
template <typename T> struct QxConvert_FromVariant< QList<T> > {
static inline qx_bool fromVariant(const QVariant & v, QList<T> & t, const QString & format, int index, qx::cvt::context::ctx_type ctx)
{
    Q_UNUSED(index);
    Q_UNUSED(ctx);

    if (format.compare("hex", Qt::CaseInsensitive) == 0)
    {
        QByteArray local = QByteArray::fromHex(v.toByteArray());
        return QX_CVT_DEFAULT_ARCHIVE::from_byte_array(t, local);
    }
    else
        return QX_CVT_DEFAULT_ARCHIVE_FROM_VARIANT;
}


Is this the best way we should be handling this scenario? Looking for some advice here, I just need to be able to reliably serialize and deserialize QList<QString> to and from the db.
If we wanted to register our own Converter for custom datatypes how would we do that?

Thanks!
Kasuax
 
Posts: 62
Joined: Mon Jun 20, 2016 6:42 pm

Re: Postgres QVariant Serialization

Postby qxorm » Mon Aug 20, 2018 9:45 am

Hello,

Postgres by default will store BYTEA as hex because apparently that is the preferered format.

I'm surprised about that. There are several customers which work with Postgres and this is the first time I see that.
Do you have a link with more details about that ?
qxorm
Site Admin
 
Posts: 481
Joined: Mon Apr 12, 2010 7:45 am

Re: Postgres QVariant Serialization

Postby Kasuax » Mon Aug 20, 2018 3:45 pm

I'm running out of time to implement a solution. Any advice would be very welcome.
Thank you!

https://www.postgresql.org/docs/9.1/sta ... inary.html
Kasuax
 
Posts: 62
Joined: Mon Jun 20, 2016 6:42 pm

Re: Postgres QVariant Serialization

Postby qxorm » Tue Aug 21, 2018 7:47 am

I think you are using an old version of Qt with recent version of PostgreSQL.
In this link : https://code.woboq.org/qt5/qtbase/src/p ... l.cpp.html
You can see :
Code: Select all
void QPSQLDriverPrivate::setByteaOutput()
{
    if (pro >= QPSQLDriver::Version9) {
        // Server version before QPSQLDriver::Version9 only supports escape mode for bytea type,
        // but bytea format is set to hex by default in PSQL 9 and above. So need to force the
        // server to use the old escape mode when connects to the new server.
        PGresult *result = exec("SET bytea_output TO escape");
        int status = PQresultStatus(result);
        if (status != PGRES_COMMAND_OK)
            qWarning("%s", PQerrorMessage(connection));
        PQclear(result);
    }
}


Or there is still a bug in the Qt QPSQL driver.
This is strange.

Anyway, you could also imagine create your own C++ type and persist it as you want.
Here is a template you could follow :
Code: Select all
#ifndef _MY_CUSTOM_PERSISTABLE_TYPE_H_
#define _MY_CUSTOM_PERSISTABLE_TYPE_H_

#ifdef _MSC_VER
#pragma once
#endif

#include <QxOrm.h>

class MyPersistableType
{
   /* What you want here */
};

QX_REGISTER_CLASS_NAME(MyPersistableType)
QX_CLASS_VERSION(MyPersistableType, 0)

QDataStream & operator<< (QDataStream & stream, const MyPersistableType & t)
{
   /* Your implementation here */
}

QDataStream & operator>> (QDataStream & stream, MyPersistableType & t)
{
   /* Your implementation here */
}

namespace qx {
namespace cvt {
namespace detail {

template <> struct QxConvert_ToVariant< MyPersistableType > {
static inline QVariant toVariant(const MyPersistableType & t, const QString & format, int index, qx::cvt::context::ctx_type ctx)
{
   /* Here I convert from MyPersistableType to QVariant */
} };

template <> struct QxConvert_FromVariant< MyPersistableType > {
static inline qx_bool fromVariant(const QVariant & v, MyPersistableType & t, const QString & format, int index, qx::cvt::context::ctx_type ctx)
{
   /* Here I convert from QVariant to MyPersistableType */
   return qx_bool(true);
} };

} // namespace detail
} // namespace cvt
} // namespace qx

#ifndef _QX_NO_JSON

namespace qx {
namespace cvt {
namespace detail {

template <>
struct QxConvert_ToJson< MyPersistableType >
{
   static inline QJsonValue toJson(const MyPersistableType & t, const QString & format)
   {
      /* Your implementation here */
   }
};

template <>
struct QxConvert_FromJson< MyPersistableType >
{
   static inline qx_bool fromJson(const QJsonValue & j, MyPersistableType & t, const QString & format)
   {
      /* Your implementation here */
   }
};

} // namespace detail
} // namespace cvt
} // namespace qx

#endif // _QX_NO_JSON

// ------------------------------------
// If you are using boost serialization, you have to specialize other template : see here for more details : https://www.qxorm.com/qxorm_en/manual.html#manual_460
// ------------------------------------

#endif // _MY_CUSTOM_PERSISTABLE_TYPE_H_
qxorm
Site Admin
 
Posts: 481
Joined: Mon Apr 12, 2010 7:45 am


Return to QxOrm - Open discussion

Who is online

Users browsing this forum: No registered users and 5 guests

cron