Octane v1.01.20 - The Open Compression Toolkit for C++ http://octane.sourceforge.net/
Homepage | Main | Modules | Class Hierarchy | Compound List | File List | Compound Members | Related Pages

OctaneCompressor Class Reference
[Compressors]

#include <compressor.hpp>

Inheritance diagram for OctaneCompressor:

OctaneClass mtfllCompressor OctaneCompressor_Statistical SampleCompressor zleCompressor SampleStatCompressor SubStrHuffCompressor List of all members.

Detailed Description

The Base Compressor class implements our generic interface to all compressors, and includes wrappers to measure compression/decompression times and diskspace.

Definition at line 45 of file compressor.hpp.

Public Member Functions

 OctaneCompressor (bool registerme=false)
 constructor

virtual ~OctaneCompressor ()
 destructor

virtual std::string GetName ()
 provide a unique name for the coder, used in some cases to automatically register the object with a manager

virtual std::string GetDescription ()
 optionally provide a longer (maybe 20-60 characters) description

virtual std::string GetHelpInformation ()
 optionally provide more information about the object on request for help

virtual unsigned short int GetCompressorStateFileGUID ()
 Return a universally unique unsigned-short-int id# for a predefined dictionary id file.

virtual OctaneCompressor * MakeCompressorInstantiation ()=0
 Instantiate a compressor from this class (like a factory).

virtual std::string GetClassName ()
 Get the assigned name of an instantiated compressor.

std::string GetInstantiatedName ()
 Get the user-set instantiated object name for this compressor.

unsigned short int GetGuid ()
 Get guid.

void SetInstantiatedName (std::string val)
 Set the user-set instantiated object name for this compressor.

void SetInfo (std::string compressorname, unsigned short int compressorguid)
 Set details (compressor name and guid) from file.

virtual void ResetState ()
 Reset the state of the compressor to its initial state.

virtual void SynchronizeStateForNewStream ()
 Synchronize state for a new stream - this will be called each time before beginning a new parsing stream.

virtual bool PrepareForCompression ()
 Prepare for parsing mode, this will be called parsing begins.

virtual void SetupAnyDefaultParser ()
 Some compressors contains parsers which begin ready to parse, without any 'training'.

virtual bool IsReadyToCompress ()
 is the compressor ready to compress? Normally this is false until a dictionary is built or loaded.

bool Compress (bitreader &from, bitwriter &to, bool needtowriteheader, bool savedecompressstateinfo)
 The Compress function is an API wrapper around a protected function which does the actual compression, in order to perform some timing/stats gathering functions.

bool Decompress (bitreader &from, bitwriter &to, bool needtoreadheader)
 The Decompress function is an API wrapper around a protected function which does the actual compression, in order to perform some timing/stats gathering functions.

bool CreateSymbolsAndModelsUsingStream (bitreader &from)
 Wrappers around the actual calls to DoProtected.., in order to perform some timing and space measurement.

bool Save (bitwriter &to, bool fortempdecompressiononly)
 Wrappers around the actual calls to DoProtected.., in order to perform some timing and space measurement.

bool Load (bitreader &from)
 Normally the compressor manager loads the header of a saved file and dynamically instantiates a compressor, but we provide this function for manually calling load on an already instantiated compressor.

bool LoadState (bitreader &from)
 Wrappers around the actual calls to DoProtected.., in order to perform some timing and space measurement.

void ResetStatistics ()
 Reset the cumulative statistics tracking.

unsigned int GetStatistics_Cumulative_Compression_BytesRead ()
 cumulative stats, can be reset with ResetCumulativeStatistics()

unsigned int GetStatistics_Cumulative_Decompression_BytesRead ()
 cumulative stats, can be reset with ResetCumulativeStatistics()

unsigned int GetStatistics_Cumulative_Compression_BytesWritten ()
 cumulative stats, can be reset with ResetCumulativeStatistics()

unsigned int GetStatistics_Cumulative_Decompression_BytesWritten ()
 cumulative stats, can be reset with ResetCumulativeStatistics()

float GetStatistics_Cumulative_Compression_Bitrate ()
 cumulative stats, can be reset with ResetCumulativeStatistics(); bit rate is in bits per bytes.

float GetStatistics_Cumulative_Decompression_Bitrate ()
 cumulative stats, can be reset with ResetCumulativeStatistics(); bit rate is in bits per bytes.

float GetStatistics_Cumulative_Compression_Ratio ()
 cumulative stats, can be reset with ResetCumulativeStatistics(); bit ratio is decompressedsize/compressedsize.

float GetStatistics_Cumulative_Decompression_Ratio ()
 cumulative stats, can be reset with ResetCumulativeStatistics(); bit ratio is decompressedsize/compressedsize.

float GetStatistics_Cumulative_Compression_ExecutionTime ()
 cumulative stats, can be reset with ResetCumulativeStatistics(); time is in seconds.

float GetStatistics_Cumulative_Decompression_ExecutionTime ()
 cumulative stats, can be reset with ResetCumulativeStatistics(); time is in seconds.

float GetStatistics_Cumulative_ExecutionTime ()
 cumulative stats, can be reset with ResetCumulativeStatistics(); this returns the sum of compression+decompression times; time is in seconds.

int GetStatistics_Cumulative_ErrorCount ()
 cumulative stats, can be reset with ResetCumulativeStatistics(); number of errors since last reset.

unsigned int GetStatistics_LastOperation_Compression_BytesRead ()
 returns the statistics for the last operation.

unsigned int GetStatistics_LastOperation_Decompression_BytesRead ()
 returns the statistics for the last operation.

unsigned int GetStatistics_LastOperation_Compression_BytesWritten ()
 returns the statistics for the last operation.

unsigned int GetStatistics_LastOperation_Decompression_BytesWritten ()
 returns the statistics for the last operation.

unsigned int GetStatistics_LastOperation_State_BytesRead ()
 returns the statistics for the last operation on state data.

unsigned int GetStatistics_LastOperation_State_BytesWritten ()
 returns the statistics for the last operation on state data.

float GetStatistics_LastOperation_Compression_Bitrate ()
 returns the statistics for the last operation; bit rate is in bits per bytes.

float GetStatistics_LastOperation_Decompression_Bitrate ()
 returns the statistics for the last operation; bit rate is in bits per bytes.

float GetStatistics_LastOperation_Compression_Ratio ()
 returns the statistics for the last operation; bit ratio is decompressedsize/compressedsize.

float GetStatistics_LastOperation_Decompression_Ratio ()
 returns the statistics for the last operation; bit ratio is decompressedsize/compressedsize.

float GetStatistics_LastOperation_Compression_ExecutionTime ()
 returns the statistics for the last operation; time is in seconds.

float GetStatistics_LastOperation_Decompression_ExecutionTime ()
 returns the statistics for the last operation; time is in seconds.

float GetStatistics_LastOperation_ExecutionTime ()
 returns the statistics for the last operation; this returns the sum of compression+decompression times; time is in seconds.

float GetStatistics_LastOperation_State_ExecutionTime ()
 returns the statistics for the last state operation; time is in seconds.

int GetStatistics_LastOperation_ErrorCount ()
 returns number of errors during the last operation.

void Add_LastOperation_Compression_BytesRead (int val)
 kludge fixup to manually adjust counts (useful for when we read a header through a separate call)

void Add_LastOperation_Compression_BytesWritten (int val)
 kludge fixup to manually adjust counts (useful for when we read a header through a separate call)

void Add_LastOperation_Decompression_BytesRead (int val)
 kludge fixup to manually adjust counts (useful for when we read a header through a separate call)

void Add_LastOperation_Decompression_BytesWritten (int val)
 kludge fixup to manually adjust counts (useful for when we read a header through a separate call)

bool SaveCompressionHeader (bitwriter &to, bool savedecompressstateinfo)
 Save header info for a compression stream (either supershort 1 byte, or long state info).

bool LoadCompressionHeader (bitreader &from)
 Load header info for a compression stream (either supershort 1 byte, or long state info).

bool SaveGUID (bitwriter &to)
 Save GUID field.


Static Public Member Functions

int LoadGUID (bitreader &from)
 Load GUID field.


Protected Member Functions

virtual bool DoProtectedCompress (bitreader &from, bitwriter &to)=0
 This is the member function which does the actual compression, and this is the function that should be subclasses by derived classes.

virtual bool DoProtectedDecompress (bitreader &from, bitwriter &to)=0
 This is the member function which does the actual decompression, and this is the function that should be subclasses by derived classes.

virtual bool DoProtectedCreateSymbolsAndModelsUsingStream (bitreader &from)
 This is the member function which does the actual training on an input in order to generate parsers/modelers/coders, and this is the function that should be subclasses by derived classes.

virtual bool DoProtectedSaveState (bitwriter &to, bool fortempdecompressiononly)
 This is the member function which does the actual saving of state, and this is the function that should be subclasses by derived classes.

virtual bool DoProtectedLoadState (bitreader &from)
 This is the member function which does the actual loading of state, and this is the function that should be subclasses by derived classes.


Protected Attributes

std::string instantiatedname
 name of instantiated instance of this class

unsigned short int guid
 unique short id for a compressor

Member Function Documentation

virtual std::string OctaneCompressor::GetName (   )  [inline, virtual]
 

provide a unique name for the coder, used in some cases to automatically register the object with a manager

Returns:
a short *unique* name

Reimplemented from OctaneClass.

Definition at line 55 of file compressor.hpp.

References GetInstantiatedName(). 

00055 {return GetInstantiatedName();};

virtual std::string OctaneCompressor::GetDescription (   )  [inline, virtual]
 

optionally provide a longer (maybe 20-60 characters) description

Returns:
a one line description

Reimplemented from OctaneClass.

Reimplemented in mtfllCompressor, SampleCompressor, SampleStatCompressor, OctaneCompressor_Statistical, SubStrHuffCompressor, and zleCompressor.

Definition at line 56 of file compressor.hpp.

Referenced by OctaneTester::ListCompressors(), and OctaneTester::ShowHelp(). 

00056 {return "Base Compressor";};

virtual std::string OctaneCompressor::GetHelpInformation (   )  [inline, virtual]
 

optionally provide more information about the object on request for help

Returns:
a long string (can be multiple
newlines)

Reimplemented from OctaneClass.

Reimplemented in OctaneCompressor_Statistical.

Definition at line 57 of file compressor.hpp. 

00057 {return "";};

virtual unsigned short int OctaneCompressor::GetCompressorStateFileGUID (   )  [inline, virtual]
 

Return a universally unique unsigned-short-int id# for a predefined dictionary id file.

This number is used to very compactly identify a preset compressor+statefile(dictionary) for use when transmission bandwidth is at a premium, such as in an online chat program with very short transmission (<<80 characters per transmission).

Returns:
0 when this compressor+configuration has no globally-registered GUID.

Definition at line 62 of file compressor.hpp.

References guid.

Referenced by OctaneTester::ListCompressors(). 

00062 {return guid;};

virtual bool OctaneCompressor::PrepareForCompression (   )  [inline, virtual]
 

Prepare for parsing mode, this will be called parsing begins.

Returns:
true on success
Todo:
is this a duplication of functionality in ResetState() and SynchronizeStateForNewStream()?

Reimplemented in OctaneCompressor_Statistical.

Definition at line 90 of file compressor.hpp. 

00090 {return true;};

virtual void OctaneCompressor::SetupAnyDefaultParser (   )  [inline, virtual]
 

Some compressors contains parsers which begin ready to parse, without any 'training'.

Here is where such a parser should be initialized.

Reimplemented in OctaneCompressor_Statistical.

Definition at line 93 of file compressor.hpp. 

00093 {;};

virtual bool OctaneCompressor::IsReadyToCompress (   )  [inline, virtual]
 

is the compressor ready to compress? Normally this is false until a dictionary is built or loaded.

Returns:
true on success

Reimplemented in SampleCompressor, and OctaneCompressor_Statistical.

Definition at line 96 of file compressor.hpp.

Referenced by OctaneTester::CheckCompressorReady(). 

00096 {return true;};

bool OctaneCompressor::Decompress (  bitreader &  from,
bitwriter &  to,
bool  needtoreadheader
) 
 

The Decompress function is an API wrapper around a protected function which does the actual compression, in order to perform some timing/stats gathering functions.

Todo:
what should we do on a LoadCompressionHeader() error?

Definition at line 77 of file compressor.cpp.

References bitwriter::align_byte(), DoProtectedDecompress(), LoadCompressionHeader(), OctaneTimer::start(), OctaneTimer::stop(), bitwriter::tell_byte(), and bitreader::tell_byte().

Referenced by OctaneTester::RunCommand_Decompress(), and OctaneTester::RunCommand_TestString(). 

00078 {
00079         // just wrap the calls to derived classes
00080         bool bretv;
00081         OctaneTimer octanetimer;
00082         size_t fromstart,tostart;
00083 
00084         // do decompression and measure it
00085         octanetimer.start();
00086         fromstart=from.tell_byte();
00087         tostart=to.tell_byte();
00088 
00089         if (needtoreadheader)
00090                 {
00091                 // this decompress could be called before the header is read, or after it.
00092                 // if the header is already read by caller, we wont try to re-read it here.
00093                 // if it hasn't been read yet, we can read it here, and it MUST match our current compressor.
00094                 // load header - this is either a supershort GUID only (usually only 1 byte), or the complete stateinfo for the compressor
00095                 bretv=LoadCompressionHeader(from);
00096                 }
00097 
00098         // do decompression
00099         bretv=DoProtectedDecompress(from,to);
00100         
00101         // align output bitwriter (this must be performed when we finish)
00102         to.align_byte();
00103 
00104         // finish measurements
00105         LastOperation_Decompression_ExecutionTime=octanetimer.stop();
00106         Cumulative_Decompression_ExecutionTime+=LastOperation_Decompression_ExecutionTime;
00107         LastOperation_Decompression_BytesRead=from.tell_byte()-fromstart;
00108         Cumulative_Decompression_BytesRead+=LastOperation_Decompression_BytesRead;
00109         LastOperation_Decompression_BytesWritten=to.tell_byte()-tostart;
00110         Cumulative_Decompression_BytesWritten+=LastOperation_Decompression_BytesWritten;
00111 
00112         // update error counts
00113         if (!bretv)
00114                 {
00115                 ++Cumulative_ErrorCount;
00116                 ++LastOperation_ErrorCount;
00117                 }
00118                 
00119         // return success
00120         return bretv;
00121 }

bool OctaneCompressor::Save (  bitwriter &  to,
bool  fortempdecompressiononly
) 
 

Wrappers around the actual calls to DoProtected.., in order to perform some timing and space measurement.

This function also saves HEADER info which identifies the compressor class, which is read by CompressorManager

Definition at line 151 of file compressor.cpp.

References DoProtectedSaveState(), GetClassName(), GetInstantiatedName(), guid, bitwriter::put(), OctaneTimer::start(), OctaneTimer::stop(), and bitwriter::tell_byte().

Referenced by OctaneTester::RunCommand_SaveCompressor(), and SaveCompressionHeader(). 

00152 {
00153         // wrap call to save state and measure it
00154         bool bretv;
00155         size_t tostart;
00156         OctaneTimer octanetimer;
00157 
00158         // do state save and measure it
00159         octanetimer.start();
00160         tostart=to.tell_byte();
00161 
00162         // save compressor header (see OctaneCompressor::Save() for where this is written)
00163         string tempstring=GetClassName();
00164         to.put(tempstring);
00165         tempstring=GetInstantiatedName();
00166         to.put(tempstring);
00167         to.put(guid);
00168         
00169         // now call derived protected method to save state
00170         bretv=DoProtectedSaveState(to,fortempdecompressiononly);
00171 
00172         // measure result
00173         LastOperation_State_BytesWritten=to.tell_byte()-tostart;
00174         LastOperation_State_ExecutionTime=octanetimer.stop();
00175         
00176         // update error counts
00177         if (!bretv)
00178                 {
00179                 ++Cumulative_ErrorCount;
00180                 ++LastOperation_ErrorCount;
00181                 }
00182 
00183         // return success
00184         return bretv;
00185 }

bool OctaneCompressor::Load (  bitreader &  from  ) 
 

Normally the compressor manager loads the header of a saved file and dynamically instantiates a compressor, but we provide this function for manually calling load on an already instantiated compressor.

It reads the header, verifies it is for proper class, and then calls LoadState().

Definition at line 188 of file compressor.cpp.

References bitreader::get(), GetClassName(), instantiatedname, and LoadState().

Referenced by LoadCompressionHeader(). 

00189         {
00190         //  Normally the compressor manager loads the header of a saved file and dynamically instantiates a compressor,
00191         //   but we provide this function for manually calling load on an already instantiated compressor.
00192         //  It reads the header, verifies it is for proper class, and then calls LoadState().
00193         bool bretv;
00194 
00195         // read standard compressor header (see OctaneCompressor::Save() for where this is written)
00196         string compressorclassname;
00197         string compressorname;
00198         short int compressorguid;
00199         from.get(compressorclassname);
00200         from.get(compressorname);
00201         from.get(compressorguid);
00202         
00203         if (compressorclassname!=GetClassName())
00204                 {
00205                 cout << "Error: class name from file does not match compressor class name being read; load aborted."<<endl;
00206                 return false;
00207                 }
00208 
00209         // set name but not guid
00210         if (instantiatedname=="")
00211                 instantiatedname=compressorname;
00212 
00213         // now load state
00214         bretv=LoadState(from);
00215         return bretv;
00216         }

virtual bool OctaneCompressor::DoProtectedCompress (  bitreader &  from,
bitwriter &  to
)  [protected, pure virtual]
 

This is the member function which does the actual compression, and this is the function that should be subclasses by derived classes.

It is wrapped by the public API call Compress() which performs some measurements on compression speed and streamsize.

Note:
this function should never be called by an outside class - the API interface should be called instead.
Returns:
true on success

Implemented in mtfllCompressor, SampleCompressor, OctaneCompressor_Statistical, and zleCompressor.

Referenced by Compress().

virtual bool OctaneCompressor::DoProtectedDecompress (  bitreader &  from,
bitwriter &  to
)  [protected, pure virtual]
 

This is the member function which does the actual decompression, and this is the function that should be subclasses by derived classes.

It is wrapped by the public API call Decompress() which performs some measurements on decompression speed and streamsize.

Note:
this function should never be called by an outside class - the API interface should be called instead.
Returns:
true on success

Implemented in mtfllCompressor, SampleCompressor, OctaneCompressor_Statistical, and zleCompressor.

Referenced by Decompress().

virtual bool OctaneCompressor::DoProtectedCreateSymbolsAndModelsUsingStream (  bitreader &  from  )  [inline, protected, virtual]
 

This is the member function which does the actual training on an input in order to generate parsers/modelers/coders, and this is the function that should be subclasses by derived classes.

It is wrapped by the public API call CreateSymbolsAndModelsUsingStream() which performs some measurements on speed and streamsize.

Note:
this function should never be called by an outside class - the API interface should be called instead.
Returns:
true on success

Reimplemented in SampleCompressor, and OctaneCompressor_Statistical.

Definition at line 142 of file compressor.hpp.

Referenced by CreateSymbolsAndModelsUsingStream(). 

00142 {return true;};

virtual bool OctaneCompressor::DoProtectedSaveState (  bitwriter &  to,
bool  fortempdecompressiononly
)  [inline, protected, virtual]
 

This is the member function which does the actual saving of state, and this is the function that should be subclasses by derived classes.

It is wrapped by the public API call SaveState() which performs some measurements on speed and streamsize.

Note:
this function should never be called by an outside class - the API interface should be called instead.
Returns:
true on success

Reimplemented from OctaneClass.

Reimplemented in SampleCompressor, and OctaneCompressor_Statistical.

Definition at line 148 of file compressor.hpp.

Referenced by Save(). 

00148 {return true;};

virtual bool OctaneCompressor::DoProtectedLoadState (  bitreader &  from  )  [inline, protected, virtual]
 

This is the member function which does the actual loading of state, and this is the function that should be subclasses by derived classes.

It is wrapped by the public API call LoadState() which performs some measurements on speed and streamsize.

Note:
this function should never be called by an outside class - the API interface should be called instead.
Returns:
true on success

Reimplemented from OctaneClass.

Reimplemented in SampleCompressor, and OctaneCompressor_Statistical.

Definition at line 154 of file compressor.hpp.

Referenced by LoadState(). 

00154 {return true;};

int OctaneCompressor::GetStatistics_LastOperation_ErrorCount (   )  [inline]
 

returns number of errors during the last operation.

Todo:
i don't think we currently reset this, so it may behave like cumulative error until we fix it.

Definition at line 217 of file compressor.hpp. 

00217 {return LastOperation_ErrorCount;};

bool OctaneCompressor::SaveCompressionHeader (  bitwriter &  to,
bool  savedecompressstateinfo
) 
 

Save header info for a compression stream (either supershort 1 byte, or long state info).

Note:
this is kind of ugly, maybe we can find a better solution.

Definition at line 283 of file compressor.cpp.

References bitwriter::put_byte(), Save(), and SaveGUID().

Referenced by Compress(). 

00284         {
00285         if (savedecompressstateinfo)
00286                 {
00287                 // save the state information  for the compressor prior to the data
00288                 // first write a special byte signifying that full state info follows
00289                 to.put_byte((unsigned char)254);
00290                 Save(to,true);
00291                 }
00292         else
00293                 {
00294                 // save only the GUID of the compressor
00295                 // if the GUID is valid, it should let receiver know which decompressor to use, or otherwise issue a warning
00296                 SaveGUID(to);
00297                 }
00298 
00299         // return success
00300         return true;
00301         }

bool OctaneCompressor::SaveGUID (  bitwriter &  to  ) 
 

Save GUID field.

Todo:
  • we will add code here so that we use only 1 byte for most guid, and 2 bytes for others, but for now we always use 1 byte for GUID

Definition at line 336 of file compressor.cpp.

References guid, and bitwriter::put_byte().

Referenced by SaveCompressionHeader(). 

00337         {
00338         // save a short guid
00339         to.put_byte((unsigned char)guid);
00340         // return success
00341         return true;
00342         }

int OctaneCompressor::LoadGUID (  bitreader &  from  )  [static]
 

Load GUID field.

Returns:
the guid or -1 if its the NOGUID constant

Definition at line 346 of file compressor.cpp.

References bitreader::get_byte().

Referenced by LoadCompressionHeader(), and OctaneTester::RunCommand_Decompress(). 

00347         {
00348         // load a short guid
00349         unsigned char c;
00350 
00351         // read first character
00352         c=from.get_byte();
00353         
00354         // special NONGUID character
00355         if (c==254)
00356                 return -1;
00357 
00358         return c;
00359         }

The documentation for this class was generated from the following files:  
Generated on 20 May 2004 by doxygen 1.3.3