实现了一个通用的内存流模板类,支持与标准流(stl)的高效直接通信,具有线程策略选择机制,也可用于对COM接口IStream的封装中。

早先简单封装了一个CByteStream类(继承了IStream COM接口),但是很低效,且不够灵活。利用周末的两天实现了如下定义的通用内存流类。
template
<
 class _Ty,
 template < class > class _ThreadingModel = DefaultThreadingModel,
 template < class > class _Allocator = ::std::allocator,
 class _Size = long,
 _Size _block_size = 4096
>
class MemoryStream: public _ThreadingModel< MemoryStream< _Ty, _ThreadingModel, _Allocator > >
{
public:
 typedef _Ty      value_type;
 typedef value_type*    pointer;
 typedef const value_type*  const_pointer;
 typedef value_type&    reference;
 typedef const value_type&  const_reference;
 typedef _ThreadingModel< MemoryStream< _Ty, _ThreadingModel, _Allocator > > ThreadingModel;
 typedef _Allocator< _Ty >  block_allocator;
 typedef _Size     size_t;
//
// …
//
}


除了内存管理器(Allocator)外与内存相关访问的封装:
template < class _Ty, class _Size = unsigned int >
struct MemCopier
{
 _Ty* operator()( _Ty* pDest, const _Ty* pSrc, _Size cbCopy )
 {
  ::memcpy( pDest, pSrc, cbCopy * sizeof( _Ty ) );
  return pDest;
 }
};


提供如下公开的调用接口:
函数:
// Read a specified number of type_sizes from the stream object
 // into memory, starting at the current seek pointer.
 // Return the actual number of type_sizes read from the stream
 // object.

 size_t Read( pointer pBuf, size_t cbRead )


 // Write a specified number of type_sizes into the stream object
 // starting at the current seek pointer.
 // Return the actual number of type_sizes written into the stream
 // object.

 size_t Write( const_pointer pBuf, size_t cbWrite )


 // Copy a specified number of bytes from the current seek pointer
 // in the stream to the current seek pointer in another stream.

 size_t CopyTo( MemoryStream& stream, size_t cbCopy )


 // Generic version of CopyTo function
 template < class _OutStream >
  size_t CopyTo( _OutStream& stream, size_t cbCopy )


 // Getneric CopyFrom function
 template < class _InStream >
  size_t CopyFrom( const _InStream& stream, size_t cbCopy )


 // Move the stream pointer to a specified location.
 // Return the new position of the stream pointer.

 size_t Seek( size_t offset, int nSeekMode = MemoryBase::seek_cur )


 // Release the memory space occupied by the stream object.
 void Clear()


 // Return the number of type_sizes of the data existing in the
 // stream object.

 size_t GetSize() const


 // Return the current position of the pointer of the stream object.
 size_t Tell() const


 // Judge whether the stream pointer is at the end of the stream
 // object.

 bool End() const


 // Judge whether the stream object has data.
 bool Empty() const
操作符:
 MemoryStream& operator = ( const MemoryStream& stream )


 // operator >>
 MemoryStream& operator >> ( const_reference vInput )
 MemoryStream& operator >> ( const_pointer pInput )
 template < class _InStream >
  MemoryStream& operator >> ( const _InStream& stream )
 MemoryStream& operator >> ( MemoryStream& stream )


 // operator <<
 MemoryStream& operator << ( reference vOutput )
 MemoryStream& operator << ( pointer pOutput )
 template < class _OutStream >
  MemoryStream& operator << ( _OutStream& stream )
 MemoryStream& operator << ( MemoryStream& stream )


几个测试用例如下:
void Test1()
{
 fstream infile;
 infile.open( “E:\\yxd\\fuzhuang.mpg”, ios_base::in | ios_base::binary );
 MemoryStream< char > theStream;
 theStream >> infile;
 infile.close();
 fstream outfile;
 outfile.open( “E:\\fuzhuang.mpg”, ios_base::out | ios_base::binary );
 theStream.Seek( 0, MemoryBase::seek_set );
 theStream << outfile;
 outfile.close();
}


void Test2()
{
 MemoryStream< char > theStream;
 for ( char n = ‘a’; n < ‘z’ + 1; ++ n )
  theStream >> n;
 theStream >> ‘A’;
 theStream.Seek( 0, MemoryBase::seek_set );
 theStream << cout;
 cout << endl;
 char szText[] = “this is a test.”;
 theStream.Write( szText, MemoryStream< char >::size_t( strlen( szText ) ) );
 theStream.Seek( 0, MemoryBase::seek_set );
 theStream << cout;
 cout << endl;
 theStream.Seek( 10, MemoryBase::seek_end );
 theStream << cout;
 cout << endl;
 theStream.Seek( -20, MemoryBase::seek_cur );
 theStream << cout;
 cout << endl;
 MemoryStream< char > theStream1( theStream );
 theStream1 << cout;
 cout << endl;
 MemoryStream< char > theStream2 = theStream1;
 theStream2 << cout;
 cout << endl;
 MemoryStream< char > theStream3;
 theStream2.Seek( 13, MemoryBase::seek_set );
 theStream3 >> theStream2;
 theStream3.Seek( 0, MemoryBase::seek_set );
 theStream3 << cout;
 cout << endl;
 char szRead[256] = { 0 };
 theStream.Seek( 0, MemoryBase::seek_set );
 cout << “Total read size is ” << theStream.Read( szRead, 256 ) << ” bytes:” << endl;
 cout << szRead << endl;
}


void Test3()
{
 MemoryStream< char > theStream1;
 theStream1.CopyFrom( cin, 10 );
 theStream1.Seek( -10 );
 theStream1 << cout;
 cout << endl;
 theStream1.Seek( 0, MemoryBase::seek_set );
 MemoryStream< char > theStream2;
 theStream1.CopyTo( theStream2, theStream2.GetSize() );
 theStream2.Seek( 0, MemoryBase::seek_set );
 theStream2 << cout;
 cout << endl;
 MemoryStream< char > theStream3;
 theStream1.Seek( 5, MemoryBase::seek_set );
 cout << “Copied ” << theStream1.CopyTo( theStream3, theStream1.GetSize() ) << ” bytes from theStream1 to theStream3:” << endl;
 theStream3.Seek( 0, MemoryBase::seek_set );
 theStream3 << cout;
 cout << endl;
}


void Test4()
{
 MemoryStream< char, ::Loki::ObjectLevelLockable >* theLockableStream = new MemoryStream< char, ::Loki::ObjectLevelLockable >;
 for ( unsigned char c = 0; c < 255; ++ c )
  *theLockableStream >> char( c );
 theLockableStream->Seek( 0, MemoryBase::seek_set );
 *theLockableStream << cout;
 cout << endl;
 //RunStreamPrint( theLockableStream );// Thread #1
 //RunStreamPrint( theLockableStream );// Thread #2
 //RunStreamPrint( theLockableStream );// Thread #3
 //RunStreamPrint( theLockableStream );// Thread #4
 //RunStreamPrint( theLockableStream );// Thread #5
 //RunStreamPrint( theLockableStream );// Thread #6

}


MemoryStream内部的内存组织方式采用的是基于::std::map这样的索引方式,采用散列的方式在大多数情况下性能应该会有更好的表现,但是这样代码的复杂性会增加,所以我就借用了::std::map,这层针对MemoryStream内部内存结构的组织策略原本也向上一层延迟,交给用户自己决定,但是还是复杂性太高,俺也没想出太好的方法,并且不像线程模型、allocator这样的有很多现成的东西可供选择。另外暂时还缺少对模板参数_Ty一样的MemoryStream之间的直接通信支持,不过比较容易实现。

This entry was posted in 技术. Bookmark the permalink.

6 Responses to 实现了一个通用的内存流模板类,支持与标准流(stl)的高效直接通信,具有线程策略选择机制,也可用于对COM接口IStream的封装中。

  1. zhangyv says:

    XDR是sun公司设计的数据结构传输标准,有内存流和I/O流两种,已经成为大多数客户机/服务器应用中的事实上的标准。直接使用XDR是不错的选择,可以做进一步了解。

  2. yxd says:

    嗯,在看XDR的RFC文档,XDR似乎更倾向于解决不同系统间数据流通讯的格式兼容、统一问题,是个标准制定。。。

  3. zhangyv says:

    考虑到不同系统间数据流通信的兼容性,一般在发送数据流之前要经过“网络字序转换”。XDR封装了一系列数据流接受和发送的函数,是一个标准。当然完全自己手工写也可以。你实现的STL内存流模板类,似乎无法实现“网络字序转换”,可能在不同系统间数据传输可能会出现一些问题,兼容性稍差。

    我写的部分总结:

    ------------------------

    网络数据结构传递可能存在以下问题:网络字序问题、浮点数传输、指针处理

    自定义手工处理方式:

    将待发送数据结构转换以后放入应用的发送缓冲区;

    将应用的接收缓冲区中数据结构转换以后再进行数据处理。

    代码示例:

    void send_int32_2buf(char *buf, unit32_t n)

    {

    n = htonl(n);

    bcopy((char *)&n, buf, sizeof(unit32_t));

    }

    void send_string_2buf(char *buf, char*str);

    {

    bcopy(str, buf, strlen(str));

    }

    void send2_buf(char *buf, struct u_data *ptr)

    {

    send_int32_2buf(buf, ptr->aInt);

    buf += sizeof(unit32_t);

    send_string_2buf(buf, ptr->str);

    buf += strlen(ptr->str); // * sizeof(char)

    }

    void recv_int32_from(char *buf, unit32_t *n)

    {

    bcopy(buf, (void*)n, sizeof(unit32_t));

    *n = ntohl(*n);

    }

    void recv_from_buf(char *buf, struct u_data *ptr)

    {

    recv_int32_from_buf(buf, &(ptr->aInt));

    buf += sizeof(uint32_t);

    }

    ?XDR标准和实现原理

    XDR数据结构传输标准是SUN公司设计的,已经成为大多数客户机/服务器应用中的事实上的标准。

    XDR对各种数据类型规定了编码方式。初始化函数是

    #include <rpc/xdr.h>

    extern void xdrmem_create((XDR *xdrs, const caddr_t addr, u_int size, enum xdr_op xop)); //xdrs是创建后XDR流指针,addr是存放XDR流发送缓冲区

    XDR的流转换方式和上面自定义方式类似,但是对各种数据类型的处理做了统一规定。应用程序设计配对的接收和发送,分别处理每个数据项。

    XDR有内存流和I/O流两种。可以使用内存流,进行套接字缓冲区间的数据结构传输;使用I/O流将编解码的结果输出到文件流中。

    XDR和TCP都是流的抽象,所以两者可以很好结合。另外XDR提供了面向记录的XDR抽象,应用在UDP传输。

    XDR转化函数所做操作决定于XDR流本身性质。如果XDR是编码流时,转化函数就做数据编码,如果是解码流时,转换函数就做数据解码。

  4. zhangyv says:

    上面那段代码是运行在linux下。

    网络字序转换函数族:

    头文件:<netinet/in.h>

    unsigned long int htonl(unsigned long int hostlong); //host to network long 字序转换

    unsigned long int htons(unsigned long int hostlong); //host to network short 字序转换

    unsigned long int ntohl(unsigned long int hostlong); //network to host long 字序转换

    unsigned long int ntohs(unsigned long int hostlong); //network to host short 字序转换

  5. yxd says:

    哦,我知道你的意思了,不过你可能误解了我写的内存流的用途,这个是个没有格式规定的内存使用形式,也就是说你网络字序也好还是按照xdr格式规定写也好,或者其它一些什么体系的内存结构也好,没关系,你都可以写在符合类似这样的一个内存流里(比如win32 com规范的IStream,或stl的stream等等)。我想,IStream并不需要规定你什么位置存什么东西,以什么顺序存等等,这个是应用层的事。

  6. zhangyv says:

    呵呵,欢迎访问我的BLOG: blog.csdn.net/zhangyv,下午刚改版的:)

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

*
To prove you're a person (not a spam script), type the security word shown in the picture. Click on the picture to hear an audio file of the word.
Click to hear an audio file of the anti-spam word