lundi 24 mars 2014

Draft GDAL/OGR class hierarchy for GDAL 2.0

As a result of the first day of the OSGeo Code Sprint 2014 Vienna, I just wanted to share the outcome of my thoughts for a possible re-organisation of the GDAL/OGR class hierarchy, to achieve the mythical "Grand Unification". This is really work in progress, and I'm not even sure I will stick with it tomorrow morning... But here we go...

I have identified two principal aims :
  • adding support for metadata to OGR driver, datasource and layers (validation of creation options, etc...). That one is easy : make derive the 3 base classes from the GDALMajorObject class
  • more difficult and ambitious: making it possible to have a Dataset that contains both raster and vector data. You just open the data container once and can get both raster and vector data. Possible use cases: GeoPackage, PCIDSK, Spatialite/Rasterlite, Postgis/Postgis raster, ...
And one major constraint : avoid rewriting each of the existing 211 drivers... which represent 1.2 million lines of C/C++ code, 140 000 lines of code of Python autotests...

The class hierarchy of GDAL/OGR 1.X versions is quite simple :


So definitely 2 seperate worlds.

To achieve the first aim of getting metadata into OGR, you just have to do :



And now let's consider where the second aim could lead us :






Another way of presenting it with more details is the following pseudo-code :
/* Interface for major object */
class GDALIMajorObject
{
    public:
        virtual char      **GetMetadataDomainList() = 0;
        virtual char      **GetMetadata( const char * pszDomain = "" );
        [other methods go here]
};

/* Implementation of major object, and base class for dataset, bands, layers, etc.. */
class GDALMajorObject: public GDALIMajorObject
{
     /* existing code of GDALMajorObject */
};

/* Interface for raster functions */
class GDALIRasterDataset: public GDALIMajorObject
{
    public:
        virtual int         HandleRasterData() = 0;

        virtual int         GetRasterXSize( void ) = 0;
        virtual int         GetRasterYSize( void ) = 0;
        virtual int         GetRasterCount( void ) = 0;
        virtual GDALRasterBand *GetRasterBand( int ) = 0;
        [other methods go here]
};

/* Interface vor vector functions*/
class GDALIVectorDataset: public GDALIMajorObject
{
    public:
        virtual int         HandleVectorData() = 0;
       
        virtual int         GetLayerCount() = 0;
        virtual OGRLayer    *GetLayer(int) = 0;
        [other methods go here]
};

/* Convenience interface for both raster and vector functions */
class GDALIDataset: public GDALIRasterDataset, public GDALIVectorDataset
{
    public:
        /* That's all ! */
};

/* Partial implementation of GDALIRasterDataset */
class GDALAbstractRasterDataset : public GDALIRasterDataset, public GDALMajorObject
{
    /* Current code of GDALDataset GDAL v1 goes here */
   
    public:
        virtual int         HandleRasterData() { return TRUE; }
};

/* Convenience class used by vector only drivers */
class GDALEmptyRasterDataset : GDALAbstractRasterDataset
{
    public:
        virtual int         HandleRasterData() { return FALSE; }
};

/* Partial implementation of GDALIVectorDataset */
class GDALAbstractVectorDataset : public GDALIVectorDataset, public GDALMajorObject
{
    /* Current code of OGRDatasource GDAL v1 goes here*/
   
    public:
        virtual int         HandleVectorData() { return TRUE; }
};

/* Convenience class used by raster only drivers */
class GDALEmptyVectorDataset : public GDALIVectorDataset
{
    public:
        virtual int         HandleVectorData() { return FALSE; }
       
        virtual int         GetLayerCount() { return 0; }
        virtual OGRLayer    *GetLayer(int) { return NULL; }
        [other methods go here]
};

/* Equivalent of GDALDataset GDAL v1 (plus dummy vector interface). Existing GDAL drivers would derive from it. */
class GDALRasterDataset : public GDALAbstractRasterDataset, public virtual GDALEmptyVectorDataset, public virtual GDALIDataset
{
};

/* Equivalent of OGRDatasource GDAL v1 (plus dummy raster interface). Existing OGR drivers would derive from it. */
class GDALVectorDataset : public GDALAbstractVectorDataset, public virtual GDALEmptyRasterDataset, public virtual GDALIDataset
{
};

/* GDAL v2 base class for new drivers that need both raster and vector data */
class GDALDataset : public GDALAbstractRasterDataset, public virtual GDALAbstractVectorDataset, public virtual GDALIDataset
{
};

The impact on the existing code base is :
  • Current GDAL drivers must replace mentions of GDALDataset by GDALRasterDataset (automatic conversion)
  • Current OGR drivers must replace mentions of OGRDatasource by GDALVectorDataset (automatic conversion)
  • OGRDatasourceH becomes an alias of GDALDatasetH
  • C methods cast the opaque dataset pointer GDALDatasetH to GDALIDataset before invoking the C++ methods.
 Of course, all the above is just "nice" theory (rather complicated admitedly), and I should really try to go to the practice part of it to test if it can actually work...

jeudi 16 janvier 2014

OGR OpenFileGDB driver

Last year, I blogged about the reverse-engineering of the ESRI file geodatabase format.

What's new ?

That work, thanks to funding, has lead to the writing of a OGR OpenFileGDB driver, now available in GDAL/OGR trunk repository.

Since the initial phase of the reverse-engineering, significant advances have been made and documented :
  • .gdbtablx files (that contain offsets in the .gdbtable files of features) can use optimization when the feature ids are sparsed (if a series of 1024 consecutive feature ids do not exist)
  • .gdbindexes files contain the list of fields that have an attribute index, and the filename of that index (only the format for ArcGIS v10 geodatabases for now. The v9 format is different and more complicated)
  • .atx files are such attribute index files. The OpenFileGDB driver can use those attribute indexes to speed-up simple WHERE clauses of SQL requests, or attribute filters (SetAttributeFilter())
  • .spx files are partially deciphered, but not yet to the point of being usable. They share exactly the same base structure as attribute indexes. For each feature (or group of features in non-final pages of an index of depth greater than one), the value indexed is a 8 byte structure that describe the spatial footprint. The first 4 bytes seem to be about the y coordinate, and the next 4 bytes the x coordinate. Rather simple, no ? Except that the encoding of those bytes is still not fully understood (I think I've captured the logic for point geometries at the final page level, but non-final pages or non-point geometries are still mysterious). So, for now, we use the minimum bounding rectangle found at the beginning of geometry blob to speed-up spatial queries. And during the first full scan, we build an in-memory spatial index that is used for later spatial queries in the same session.
FileGDB vs OpenFileGDB

In the current state, if we do a comparison of the OpenFileGDB driver with the FileGDB driver using FileGDB API SDK v1.3, we find :

On the plus side :
  • Can read ArcGIS 9.X Geodatabases, and not only 10 or above.
  • Can open layers with any spatial reference system.
  • Thread-safe (i.e. datasources can be processed in parallel).
  • Uses the VSI Virtual File API, enabling the user to read a Geodatabase in a ZIP file or stored on a HTTP server.
  • Faster on databases with a big number of fields.
  • Does not depend on a third-party library. Available on any platform supported by GDAL/OGR
  • Robust against corrupted Geodatabase files.

On the minus side :
  • Read-only.
  • Cannot use spatial indexes.
And now ?

In the testing process, I discovered datasets that have layers compressed using a so-called Smart Data Compression. Which is a completely different beast from standard GDB tables. Not sure how "smart" that compression is, but the end result is particularly cryptic. The only thing that can be recognized is the field names... Those .gdbtable.sdc are neither supported by the OpenFileGDB driver, and guess what?, nor by the FileGDB API. So we are on (non-)feature parity...

I've encountered a few raster File Geodatabase datasets
(apparently tiled), and a quick inspection of the tables makes me believe
that a raster driver would be doable.

That's all for now. Testers appreciated as usual ! Windows users can for example download the builds tagged "-development" kindly provided by Tamas Szekeres on gisinternals.