jeudi 2 mai 2019

Incremental Docker builds using ccache

Those past days, I've experimented with Docker to be able to produce "official" nightly builds of GDAL
Docker Hub is supposed to have an automated build mechanism, but the cloud resources they put behind that feature seem to be insufficient to sustain the demand and builds tend to drag forever.
Hence I decided to setup a local cron job to refresh my images and push them. Of course, as there are currently 5 different Dockerfile configurations and building both PROJ and GDAL from scratch could be time consuming, I wanted this to be as most efficient as possible. One observation is that between two nightly builds, very few files changes on average, so ideally I would want to recompile only the ones that have changed, and have the minimum of updated Docker layers refreshed and pushed.

There are several approaches I combined together to optimize the builds. For those already familiar with Docker, you can probably skip to the "Use of ccache" section of this post.

Multi-stage builds

This is a Docker 17.05 feature in which you can define several steps (that will form each a separate image), and later steps can copy from the file system of the previous steps. Typically you use a two-stage approach.
The first stage installs development packages, builds the application and installs it in some /build directory.
The second stage starts from a minimal image, installs runtime dependency, and copies the binaries generated at the previous stage from the /build to the root of the final image.
This approach avoids any development packages to be in the final image, which keeps it lean.

Such Dockerfile will look like:

FROM ubuntu:18.04 AS builder
RUN apt-get install g++ make
RUN ./configure --prefix=/usr && make && make install DESTDIR=/build

FROM ubuntu:18.04 AS finalimage
RUN apt-get install libstdc+
COPY --from=builder /build/usr/ /usr/



Fine-grained layering of the final image

Each step in a Dockerfile generates a layer, which chained together form an image.
When pulling/pushing an image, layers are processed individually, and only the ones that are not present on the target system are pulled/pushed.
One important note is that the refresh/invalidation of a step/layer causes the
refresh/invalidation of later steps/layers (even if the content of the layer does
not change in a user observable way, its internal ID will change).
So one approach is to put first in the Dockerfile the steps that will change the less frequently, such as dependencies coming from the package manager, third-party dependencies whose versions rarely change, etc. And at the end, the applicative part. And even the applications refreshed as part of the nightly builds can be decomposed in fine-grained layers.
In the case of GDAL and PROJ, the installed directories are:
$prefix/usr/include
$prefix/usr/lib
$prefix/usr/bin
$prefix/usr/share/{proj|gdal}

The lib is the most varying one (each time a .cpp file changes, the .so changes).
But installed include files and resources tend to be less frequently updated.

So a better ordering of our Dockerfile is:
COPY --from=builder /build/usr/share/gdal/ /usr/share/gdal/
COPY --from=builder /build/usr/include/ /usr/include/
COPY --from=builder /build/usr/bin/ /usr/bin/
COPY --from=builder /build/usr/lib/ /usr/lib/


With one subtlety, as part of our nightly builds, the sha1sum of the HEAD of the git repository is embedded in a string of $prefix/usr/include/gdal_version.h. So in the builder stage, I separate that precise file from other include files and put it in a dedicated /build_most_varying target together with the .so files.

RUN [..] \
    && make install DESTDIR=/build \
    && mkdir -p /build_most_varying/usr/include \
    && mv /build/usr/include/gdal_version.h /build_most_varying/usr/include \
    && mv /build/usr/lib /build_most_varying/usr


And thus, the finalimage stage is slightly changed to:

COPY --from=builder /build/usr/share/gdal/ /usr/share/gdal/
COPY --from=builder /build/usr/include/ /usr/include/
COPY --from=builder /build/usr/bin/ /usr/bin/
COPY --from=builder /build_most_varying/usr/ /usr/


Layer depending on a git commit

In the builder stage, the step that refreshes the GDAL build depends on an
argument, GDAL_VERSION, that defaults to "master"

ARG GDAL_VERSION=master
RUN wget -q https://github.com/OSGeo/gdal/archive/${GDAL_VERSION}.tar.gz \
    && build instructions here...

Due to how Docker layer caching works, building several times in a row this Dockerfile would not refresh the GDAL build (unless you invoke docker build with the --no-cache switch, which disable all layer caching), so the script that triggers the docker build, gets the sha1sum of the latest git commit and passes it with:

GDAL_VERSION=$(curl -Ls https://api.github.com/repos/OSGeo/gdal/commits/HEAD -H "Accept: application/vnd.github.VERSION.sha")
docker build --build-var GDAL_VERSION=${GDAL_VERSION} -t myimage .

In the (unlikely) event where the GDAL repository would not have changed, no
new build would be even attempted.

Note: this part is not necessarily a best practice. Other Docker mechanisms,
such as using a Git URL as the build context, could potentially be used. But as
we want to be able to refresh both GDAL and PROJ, that would not be really suitable.
Another advantage of the above approach is that the Dockerfile is self sufficient
to create an image with just "docker build -t myimage ."

Use of ccache

This is the part for which I could not find an already made & easy to deploy solution.

With the previous techniques, we have a black and white situation. A GDAL build is either entirely cached by the Docker layer caching in the case the repository did not change at all, or completely done from scratch if the commit id has changed (which may be some change not affecting at all the installed file). It would be better if we could use ccache to minimize the number of files to be rebuilt.
Unfortunately it is not possible with docker build to mount a volume where the ccache directory would be stored (apparently because of security issues). There is an experimental RUN --mount=type=cache feature in Docker 18.06 that could perhaps be equivalently used, but it requires both the client and daemon to be started in experimental mode.

The trick I use, which has the benefit of working with a default Docker installation, is to download from the Docker build container the content of a ccache directory from the host, do the build and upload the modified ccache back onto the host.

I use rsync for that, as it is simple to setup. Initially, I used a rsync daemon directly running in the host, but based on inspiration given by https://github.com/WebHare/ccache-memcached-server which proposes an alternative, I've modified it to run in a Docker container, gdal_rsync_daemon, which mounts the host ccache directory. The benefit of my approach over the ccache-memcached-server one is that it does not require a patched version of ccache to run in the build instance.

So the synopsis is:

host cache directory <--> gdal_rsync_daemon (docker instance)  <------> Docker build instance
                  (docker volume mounting)                           (rsync network protocol)


You can consult here the relevant portion of the launching script which builds and launches the gdal_rsync_daemon. And the corresponding Dockerfile step in the builder stage is rather straightforward:

# for alpine. or equivalent with other package managers
RUN apk add --nocache rsync ccache

ARG RSYNC_REMOTE
ARG GDAL_VERSION=master
RUN if test "${RSYNC_REMOTE}" != ""; then \
        echo "Downloading cache..."; \
        rsync -ra ${RSYNC_REMOTE}/gdal/ $HOME/; \
        export CC="ccache gcc"; \
        export CXX="ccache g++"; \
        ccache -M 1G; \
    fi \
    # omitted: download source tree depending on GDAL_VERSION
    # omitted: build
    && if test "${RSYNC_REMOTE}" != ""; then \
        ccache -s; \
        echo "Uploading cache..."; \
        rsync -ra --delete $HOME/.ccache ${RSYNC_REMOTE}/gdal/; \
        rm -rf $HOME/.ccache; \
    fi


I also considered a simplified variation of the above that would not use rsync, where after the build, we would "docker cp" the cache from the build image to the host, and at the next build, copy the cache into the build context. But that would have two drawbacks:
  • our build layers would contain the cache
  • any chance in the cache would cause the build context to be different and subsequent builds to have their cached layers invalidated.

Summary

We have managed to create a Dockerfile that can be used in a standalone mode
to create a GDAL build from scratch, or can be integrated in a wrapper build.sh
script that offers incremental rebuild capabilities to minimize use of CPU resources. The image has fine grained layering which also minimizes upload and download times for frequent push / pull operations.

jeudi 31 janvier 2019

SRS barn raising: 8th report. Ready for your testing !

This is the 8th progress report of the GDAL SRS barn effort.

As the title implies, a decisive milestone has now been reached, with the "gdalbarn" branches of libgeotiff and GDAL having been now merged in their respective master branch.

On the PROJ side, a number of fixes and enhancements have been done:
- missing documentation for a few functions, the evolution of cs2cs and the new projinfo utility has been added
- the parser of the WKT CONCATENATEDOPERATION construct can now understand step presented in a reverse order
- a few iterations to update the syntax parsing rules of WKT2:2018 following the latest adjustments done by the OGC WKT Standard Working Group
- in my previous work, I had introduced a "PROJ 5" to export CRS using pipeline/unitconvert/axisswap as an attempt of improving the PROJ.4 format used by GDAL and other products. However after discussion with other PROJ developers, we realize that it is likely a dead-end since it is still lossy in many aspects and can cause confusion with coodinate operations. Consequently the PROJ_5 convention will be identical to PROJ_4 for CRS export. And the use of PROJ strings to express CRS themselves is discouraged. It can still makes sense if using the "early-binding" approach and specifying towgs84/nadgrids/geoidgrids parameters. But in a late-binding approach, WKT is much more powerful to capture important information like geodetic datum names.
- when examining how the new code I added those past months with the existing PROJ codebase, it became clear that there was a confusion when importing PROJ strings expressing coordinate operations versus PROJ strings expressing a CRS. So for the later use case, a "+type=crs" must be added in the PROJ string. As a consequence the proj_create_from_proj_string() and proj_create_from_user_input() functions have been removed, and proj_create() can now been used for all types of PROJ strings.
- The PROJ_LIB environment variable now supports multiple paths, separated by colon on Unix and semi-colon on Windows

On the GDAL side,
  • the OGRCoordinateTransformation now uses the PROJ API to automatically compute the best transformation pipeline, enabling late-binding capabilities. In the case where the user does not provide an explicit area of use and several coordinate operations are possible, the Transform() method can automatically switches between coordinate operations given the input coordinate values. This should offer behaviour similar to previous versions  for example for NAD27 to NAD83 conversion when PROJ had a +nadgrids=@conus,@alaska,@ntv2_0.gsb,@ntv1_can.dat hardcoded rule. This dymanic selection logic has also been moved to PROJ proj_create_crs_to_crs() function. Note that however this might not always lead to the desired results, so specifying a precise area of interest, or even a specific coordinate operation, is preferred when full control is needed.
  • gdalinfo, ogrinfo and gdalsrsinfo now outputs WKT2:2018 by default (can be changed with a command line switch). On the API side, the exportToWKT() method will still export WKT1 by default (can of course be changed). The rationale is that WKT consumers might not be ready yet for WKT2, so this should limit the backward compatibility issues. In the future (couple of years timeframe), this default WKT version might be upgraded when more consumers are WKT2-ready.
  • A RFC 73: Integration of PROJ6 for WKT2, late binding capabilities, time-support and unified CRS database document was created to document all the GDAL changes. After discussion with the community, this RFC has been approved
  • As a result, all of the above mentionned work has now been merged in GDAL master
  • Important practical discussion: GDAL master now depends on PROJ master (and ultimately PROJ 6.0 once it is released)
Consequently, on the pure development front, most of the work has now been completed.  As all those changes done those last months deeply impact SRS related functionnality in GDAL and PROJ, we rely now on your careful testing to spot the inevitable issues that have not yet been detected by their respective automatic regression test suites. The earlier they are detected, the easier they will be fixable, in particular if they impact the API.

vendredi 28 décembre 2018

SRS barn raising: 7th report

This is the 7th progress report of the GDAL SRS barn effort.

On the PROJ side, things are consolidating up. Tens of "random" fixes have been pushed due to the GDAL autotest suite triggering a number of interesting cases. The C API has also been enhanced to accommodate for the needs of GDAL and libgeotiff. We also have received feedback from an early adopter (a developer of a pyproj binding based on PROJ master). The major development items have been the move of the WKT 1 syntax validation from GDAL to PROJ, as well as the development of an equivalent WKT 2 syntax validator in PROJ (this task has been useful to uncover a few minor issues in the draft of the future WKT2:2018 standard). A reorganization of the PROJ source tree (with a conversion of most C files to C++ files) has also been done, as a preliminary step, for a pull request to better integrate the new ISO-19111 functionality I developed those last months with the existing C API.

Regarding libgeotiff, a v1.4.3 maintenance version has been released with the fixes of the last two years, before the new works for the integration of PROJ master are done. libgeotiff development has been moved from Subversion to https://github.com/OSGeo/libgeotiff. As a preliminary step, continuous integration capability has been added to test compilation under Linux/GCC and Windows/Visual Studo, with a few runtime tests.
A pull request is ready with the integration of PROJ master with libgeotiff. It features:
  • PROJ master / PROJ 6 as a required depedency of libgeotiff
  • Use of the proj.db database to resolve the various coded values, mostly in the GTIFGetDefn() "normalization" function of libgeotiff, instead of using the .csv files previously generated from the EPSG database. Typically an EPSG code identifying a projected CRS is resolved them into its base elements: projection method, projection parameters, base geodetic CRS, etc..
  • Complete removal of those .csv files and associated functionality
This work will be merged once the above mention PROJ pull request that affects naming of the new functions of the C API has been merged and taken into account. This gdalbarn branch of libgeotiff has also been succesfully integrated in the internal libgeotiff copy of the GDAL gdalbarn branch.

Regarding GDAL, a maintenance v2.3.3 and feature v2.4.0 versions have been released for the same reasons as above.
Most methods of the OGRSpatialReference class have now been re-implemented to rely on the PROJ C API to do queries and state changes, which avoids potentially lossy import / export to WKT 1. Similarly to the libgeotiff work, the ImportFromEPSG() functionality now relies on proj.db, and consequently all EPSG or ESRI related .csv files have been removed from the GDAL data directory. I've also drafted a plan regarding on how to be able to take into account WKT 2 by GDAL raster drivers, and proposed changes regarding how to better handle the gap between the axis order as mandated by the CRS authority (for example latitude first, longitude second for geographic CRS in the EPSG dataset) and the actual order of the values in raster metadata or vector geometries. The first part (use of OGRSpatialReference in raster driver) has been implemented in the gdalbarn branch and the second part is in good progress, with the drivers now advertizing their data axis to CRS axis mapping. The ongoing work is to make  OGRCoordinateTransformation use the PROJ API to automatically compute the best transformation pipeline, enabling late-binding capabilities.



vendredi 30 novembre 2018

SRS barn raising: 6th report

This is the sixth progress report of the GDAL SRS barn effort. The pace of changes has not yet slow down, with still a significant part of the work being in PROJ, and an initial integration in GDAL.

The major news item is that RFC2, implementing the new capabilities (WKT-2 support, late-binding approach, SQLite database), has now been merged into PROJ master.

An initial integration of PROJ master into GDAL has been started in a custom GDAL branch . This includes:
  • PROJ master, which will be released as 6.0, is now a required dependency of GDAL. It actually becomes the only required external third-party dependency (if we except the copies of a few libraries, such as libtiff, libgeotiff, etc. that have been traditionaly included into the GDAL source tree)
  • The dozen of continuous integration configurations have been modified to build PROJ master as a preliminary step.
  • Related to the above, we have including into PROJ a way to "version" its symbols. If PROJ is built with -DPROJ_RENAME_SYMBOLS in CFLAGS and CXXFLAGS, all its exported symbols are prefixed with "internal_". This enables GDAL to link against PROJ master, while still using pre-compiled dependencies (such as libspatialite) that link against the system PROJ version, without a risk of symbol clash. This is particularly useful to be able to run GDAL autotests on continuous integration environments that use pre-packaged dependencies (or if you want to test the new GDAL without rebuilding all reverse dependencies of GDAL). This however remains a hack, and ultimately when PROJ 6 has been released, all reverse dependencies should be built against it. (this solution has been successfully tested in the past where GDAL had a libtiff 4.0 internal copy, whereas external libtiff used by some GDAL dependencies relied on the system libtiff 3.X)
  • Compatibility mechanisms which were required to support older PROJ versions have been removed. In particular, the runtime loading (using dlopen() / LoadLibrary() mechanism) has been removed. It proved to cause code complication, and users frequently ran into headaches with different PROJ versions being loaded and clashing/crashing at runtime.
  • The OGRSpatialReference class which implements CRS manipulation in GDAL has been modified to use the new PROJ functions to import and export between WKT and PROJ strings. Previously GDAL had such code, which is now redundant with what PROJ offers. This preliminary integration caused a number of fixes to be made on PROJ to have compatibility with the input and output of GDAL for WKT.1 and PROJ strings. Besides "moving" code from GDAL to PROJ, a practical consequence is that the addition of a new projection method into PROJ will no longer require changes to be made to GDAL for it to be usable for reprojection purposes.

There have been reflections on how to use the new code developped in PROJ by the existing PROJ code. A pull request is currently under review and implements:
  • changes needed to remove from the data/ directory the now obsolete EPSG, IGNF, esri and esri.extra files to rely instead of the proj.db dataase 
  • making the proj_create_crs_to_crs() API use the new late-binding approach to create transformation pipelines
  • updating cs2cs to use that new API. 
  • list and address backward compatibility issues related to honouring official axis order
Integration phase in GDAL continus with the aim of using more of the new PROJ code. Typically the OGRSpatialReference class that models in GDAL the CRS/SRS was up to now mostly a hierarchy of WKT nodes, where setters methods of OGRSpatialReference would directly create/modify/delete nodes, and getter methods query them. This approach was fine when you had to manage just one WKT version (with the caveat that it was also easy to produce invalid WKT representions, lacking mandatory nodes). However, this is no longer appropriate now that we want to support multiple WKT versions. Our goal is to make OGRSpatialReference act rather on osgeo::proj::CRS objects (and its derived classes). Switching between the two abstractions is a non-trivial task and doing it in a bing-bang approach seemed risky, so we are progressively doing it by using a dual internal modelling. A OGRSpatialReference instance will maintain as a primary source a osgeo::proj::CRS object, and for operations not yet converted to the new approach, will fallback to translating it internally to WKT.1 to allow direct manipulation of the nodes, and then retranslate that updated WKT.1 representation back to a osgeo::proj::CRS object. Ultimately the proportion of methods using the fallback way should decrease (it is not completely clear we can remove all of them since direct node manipulation is spread in a significant number of GDAL drivers). The task is slowly progressing, because each change can subtely modify the final WKT.1 representation (nodes being added, number of significant digits changing) and cause a number of unit tests to break (GDAL autotest suite is made of 280 000 lines of Python code) and be analyzed to see if there was a bug and or just an expected result to be slightly altered.
Because of all the above impacts, we have decided to do an early release in December of GDAL master as GDAL 2.4.0 with all the new features since GDAL 2.3, in order to be able to land this PROJ integration afterwards afterwards. A GDAL 2.5.0 release will hopefully follow around May 2019 with the result of the gdalbarn work.


Other side activities regarding collecting transformation grids:
  • Following a clarification from IGN France on the open licensing of their geodesy related resources, their CRS and transformation XML registry is now processed to populate the IGNF objects in the proj.db database (the previous import used the already processed IGNF file containing PROJ string, which caused information losses). The associated vertical shift grids have also been converted from their text-based format to the PROJ digestable .gtx format, integrated in the proj-datumgrid-europe package, and they have been referenced in the database for transformations that use them.
  • The NGS GEOID 2012B vertical grids to convert between NAD83 ellipsoidal heights and NAVD88 heights have also been integrated in the proj-datumgrid-north-america package

mercredi 31 octobre 2018

SRS barn raising: 5th report

This is the fifth progress report of the GDAL SRS barn effort. A lot of activity in PROJ developments has been done this month again.

New conversion and transformation methods referenced in the EPSG database have been integrated:
  • Projection methods: Lambert Conic Conformal (2SP Michigan) and Laborde Oblique Mercator
  • "Change of Vertical Unit", "Vertical offset" and other fixes relative to vertical component handling
  • "Axis order reversal"
  • "Affine parametric transformation", and its implementation in PROJ computation core
  • "Molodensky-Badekas" transformation, and its implementation in PROJ computation core
Regarding database-related activities:
  • Make concatenate operation building from proj.db robust to inverse sub-operations (the EPSG database lists chained operations, but does not indicate if the forward or reverse path must be taken)
  • Reference transformation grids available in the proj-datumgrid-* packages, such as NAD83 -> NAD83(HPGN) grids, OSTN15_NTv2_OSGBtoETRS.gsb
  • Add a celestial_body table and celestial_body property to ellipsoid, as a provision to handle non-Earth bodies
  • Add a text_definition column to geodetic_crs and projected_crs to allow definition by PROJ string (or WKT)
  • Add  the possibility of defining in 'other_transformation' a transformation defined by a PROJ pipeline
  • The createOperations() method that returns transformations between two CRS is now able to find pivot CRS when no direct transformation path exists. One notable fact to underline is that the pivot is not necessarily WGS84, and several candidate pivots are explored. When transforming from CRS A to CRS B, the database will be searched for all CRS C for which there is a referenced transformation from/to both CRS A and CRS B (advanced users may also be able to restrict the candidate(s) pivot to use)
  • Import PROJ.4 definitions contained currently data/IGNF into the database in a relational form. A script to directly use the official registry as the source, instead of the PROJ.4 strings that derive from it, has also been developed but is not used yet.
  • Import of the CRS definitions and transformations between horizontal CRS from the CSV files of the published Esri Projection Engine Database Documentation. The projected CRS are imported with their ESRI WKT definition directly put in the database, so ongoing work is in progress to be able to ingest this WKT1 variant in the WKT2-based form used internally by PROJ.
  • Update the CRS and transformations to EPSG dataset v9.5.4
The addition of the DerivedEngineeringCRS, DerivedParametricCRS and DerivedTemporalCRS classes mark the completion of the implementation of the full ISO-19111:2018 CRS modelling

A rather tedious task has consisted in optimizing the code (mostly by avoiding a lot of unnecessary instanciation of temporary C++ objects, involving looking frequently at disassembled output to locate them) to make the generated binary lighter. This has enabled an optimized build to go down from 3.1 to 2.6 MB

The Doxygen-generated documentation of the C++ API has been integrated with the general PROJ documentation in ReST format, with the Breathe module.

An initial C API that makes part of the C++ functionality available to C users has also been added.

Finally, a RFC has been written to officially submit this work for PROJ PSC approval (doubling the size of a code base and changing its language requirements is not a mundane detail). This RFC has now been officially adopted, and will make it possible to ultimately merge this work into PROJ master. You may skim through it to look at a few examples of the use of the projinfo utility that demonstrates part of the new capabilities developed.

vendredi 28 septembre 2018

SRS barn raising: 4th report

This is the fourth progress report of the GDAL SRS barn effort.

The in-progess task at the time of the last report was the addition of a method createFromPROJString() that takes a PROJ string and builds ISO-19111 corresponding objects. It has now been completed for ProjectedCRS. Not all arbitrary valid PROJ strings can be currently mapped to ISO-19111 objects, but at least strings expressing CRS definitions can. A few constructs involving Helmert transforms are also parsed as CoordinateOperation.

A few random tasks completed:
  • Map time-dependent Helmert transforms, and fixes for existing Helmert transforms
  • Map Molodensky and Abridged Molodensky transformation methods from/to PROJ strings
  • Map Longitude rotation transformation method
  • Update to recognize VRF and TRF WKT2-2018 keywords
  • Add import/export of VERTICALEXTENT and TIMEEXTENT WKT2 elements
  • Add import/export of WKT2:2018 USAGE node
  • Progress in implementation of "esoteric" ISO-19111 classes: DatumEnsemble, DynamicVerticalReferenceFrame, OrdinalCS, DerivedProjectedCRS, EngineeringCRS, ParametricCRS, DerivedVerticalCRS
  • Several fixes

The most interesting task regarding the high-level objectives of the roadmap of the barn campaign is the creation of a proj.db SQLite3 database containing CRS (and related objects) and coordinate operations (datum shifts, ...) definitions. The workflow to build it is:
  1. Import EPSG dataset PostgreSQL .sql dumps
  2. Run scripts/build_db.py that will ingest those dumps in a temporary SQLite3 database and then extract needed information from it and marshall it a more digestable form for our final proj.db. At the end the script, outputs new .sql scripts in the data/sql subdirectory of the PROJ directory. We keep in version control those text files, for better tracability of changes.
  3. At make time, we build the proj.db database by importing those .sql scripts.
Steps 1 and 2 are done (typically by PROJ developers) each time you need to update to a newer version.
Step 3 is done automatically at PROJ build time, from a git clone or a tarball of an official release.

The proj.db structure allows for multiple authorities. Instead of having a single code column to identify and reference objects, we use a tuple (authority_name, code) as a key, both columns being of text type for better generality. So ('EPSG', '4326') or ('IGNF', 'LAMB1') are possible. At that time, only EPSG derived objects are in the database. Import of other dictionaries are task for later.

Having a database is good, but using it is better. So the next task was to implement a factory class able to build an object in our ISO-19111 modelling from its (auth_name, code). This is now completed for all object categories of the database. The in-progress task is the generalization/augmentation of the current method createOperation(sourceCRS, targetCRS) that creates coordinate method without external input than the definition of the CRS as a createOperations(sourceCRS, targetCRS, context) that also uses the proj.db database to find registered coordinate operations (typically between the geographic CRS), and take into account specified area of interest and desired accuracy, to propose a list of candidate coordinate operations (chaining for example the reverse projection, a datum shift, and a forward projection). I'm also working on a new `projinfo`utility that will be similar in purpose to the `gdalsrsinfo`, offering the possibility to ingest PROJ strings (legacy PROJ.4 format and new PROJ pipelines), WKT 1, WKT 2, authority codes and output as PROJ (legacy and new), WKT 1, WKT2:2015, WKT2:2018. A mode to list coordinate operations possible between two CRS will also be available.

One interesting statistics: the number of lines of C++ code (including blank lines and comments) added to PROJ per this work is now greater than the number of historical C code: 47 000 lines (14 K being tests) vs 43 000.

mercredi 29 août 2018

SRS barn raising: 3rd report

This is the third progress report of the GDAL SRS barn effort.

In the last month, the main task was to continue, and finish, the mapping of all GDAL currently supported projection methods between their different representations as WKT 2, WKT 1 and PROJ string: LCC_2SP, LCC_2SP_Belgium, Modified Azimuthal Equidistant, Guam Projection, Bonne, (Lambert) Cylindrical Equal Area,    GaussSchreiberTransverseMercator, CassiniSoldner, EckertI to VI, EquidistantCylindricalSpherical, Gall, GoodeHomolosine, InterruptedGoodeHomolosine, GeostationarySatelliteSweepX/Y, International Map of the World Polyconic, Krovak North Oriented and Krovak, LAEA, Mercator_1SP and Mercator_2SP, WebMercator (including GDAL WKT1 import/export tricks), Mollweide, ObliqueStereographic and Orthographic, American polyconic, PolarSterographicVariantA and PolarSterographicVariantB, Robinson and Sinusoidal, Stereographic, VanDerGrinten, Wagner I to VII, QuadrilateralizedSphericalCube, SphericalCrossTrackHeight, Aitoff, Winkel I and II, Winkel Tripel, Craster_Parabolic, Loximuthal and Quartic_Authalic

The task was tedious, but necessary.  For some cases, this involved cross-checking formulas in the EPSG "Guidance Note 7, part 2 Coordinate Conversions & Transformations including Formulas", PROJ implementation and Snyder "Map Projections - A Working Manual" because of ambiguities in some projection names. Typically the ObliqueStereographic in EPSG is not the Oblique Stereographic of Snyder. The former is implemented as the Oblique Stereographic Alternative (sterea) in PROJ, and the later as the Oblique Stereographic (stere). The parameter names in WKT 2 / EPSG tend also to be much more specific that in GDAL WKT 1. When in GDAL WKT1, you have mostly a "latitude_of_origin" parameter mapping to the lat_0 PROJ parameter, in WKT2, parameter names tend to better reflect the mathematical characteristics of the projection, distinguishing between "Latitude of natural origin", "Latitude of projection centre" or "Latitude of false origin"

The currently ongoing task is now to implement a method that takes a PROJ string and builds ISO-19111 corresponding objects. Done for GeographicCRS (+proj=longlat), in progress for Projected CRS. When this will be completed we will have the infrastructure to convert in all directions between PROJ strings, WKT 1 and WKT 2

When digging into PROJ code, I also uncovered a number of issues in the Helmert implementation (confusing for rotational parameters regarding the "Position Vector" vs "Coordinate frame" convention), the handling of the not-so-well-known +geoc flag for geocentric latitudes and the handling of vertical units for geographic CRS with the new PROJ API. All of those fixes have been independantly merged in PROJ master, so as to be available for the upcoming PROJ 5.2.0, which should be released around mid-september (to remove any confusion, this release will not include yet all the WKT 2 related work)