LATEST VERSION: 3.0.0 - RELEASE NOTES
Pivotal GemFire®-Greenplum® Connector v3.0

Using The Connector

To use the connector:

  • Download the single JAR file from the Pivotal GemFire product page. The gemfire-greenplum-3.0.X.jar is for the 3.0 version of the connector, and an appropriate version’s value substitutes for X. Include this JAR in the classpath by setting the CLASSPATH environment variable, substituting the appropriate version’s value for X within the file name:
   export CLASSPATH=$CLASSPATH:/path/to/gemfire-greenplum-3.0.X.jar
  • Specify the correct xsi:schemaLocation attribute in the cache.xml file. For the 3.0.0 connector, use

    http://schema.pivotal.io/gemfire/gpdb/gpdb-2.4.xsd
    
  • Start Pivotal GemFire® locator and server components with the --include-system-classpath option such that they use the environment variable, and can use the connector:

   gfsh>start locator --name=loc1 --include-system-classpath
   gfsh>start server  --name=s1 --cache-xml-file=/path/to/cache.xml --include-system-classpath

When a second server is started, the second server will fail to start under each of the following situations:

  • If two cache.xml files have different GPDB table names mapped to the same GemFire region.

  • If two cache.xml files have different schema attribute values in the specification of the gpdb:pdx element for a single GemFire region.

  • If two cache.xml files map a different set of id fields. This implies that the gpdb:id element must be fully specified; it cannot be empty.

  • If two cache.xml files have different content for their gpdb:field elements. Note that if both have an empty gpdb:fields element, the second server can start. For differing content in the cache.xml files, the situations that cause the second server to fail include:

    • no gpdb:field elements are specified for the first server, and all the gpdb:field elements are specified for the second server
    • no gpdb:field elements are specified for the second server, and all the gpdb:field elements are specified for the first server
    • no gpdb:field elements are specified for the first server, and some gpdb:field elements are specified for the second server
    • no gpdb:field elements are specified for the second server, and some gpdb:field elements are specified for the first server

Connector Requirements and Caveats

  • Export is supported from partitioned GemFire regions only. Data cannot be exported from replicated regions. Data can be imported to replicated regions.

  • The number of Pivotal Greenplum® Database (GPDB) segments must be greater than or equal to the number of Pivotal GemFire servers. If there is a high ratio of GPDB segments to GemFire servers, the GPDB configuration parameter gp_external_max_segs may be used to limit GPDB concurrency. See gp_external_max_segs for details on this parameter. An approach to finding the best setting begins with identifying a representative import operation.

    • Measure the performance of the representative import operation with the default setting.
    • Measure again with gp_external_max_segs set to half the total number of GPDB segments. If there is no gain in performance, then the parameter does not need to be adjusted.
    • Iterate with values of gp_external_max_segs that are half as much at each iteration, until there is no performance improvement or the value of gp_external_max_segs is the same as the number of GemFire servers.

Upgrading Java Applications from Version 2.4 to Version 3.0

API changes for version 3.0.0 require code revisions in all applications that use import or export functionality.

For this sample version 2.4 export operation, an upsert type of operation was implied:

// Version 2.4 API
long numberExported = GpdbService.createOperation(region).exportRegion();

Here is the equivalent version 3.0 code to implement the upsert type of operation:

// Version 3.0 API
ExportConfiguration exportConfig = ExportConfiguration.builder(region)
   .setType(ExportType.UPSERT)
   .build();
ExportResult result = GpdbService.exportRegion(exportConfig);
int numberExported = result.getExportedCount();

For this sample version 2.4 import operation,

// Version 2.4 API
long numberImported = GpdbService.createOperation(region).importRegion();

here is the version 3.0 code to implement the import operation:

// Version 3.0 API
ImportConfiguration importConfig = ImportConfiguration.builder(region)
   .build();
ImportResult result = GpdbService.importRegion(importConfig);
int numberImported = result.getImportedCount();

Please note that the new result objects’ counts are of type int instead of type long. This is for consistency, as the connector internally uses JDBC’s executeQuery(), which supports int.