Data Connectivity Manager User’s Guide

Data managers can turn to LuciadFusion Studio to manage geospatial data and serve it to client applications. LuciadFusion Studio is a web application that allows data managers to centralize their geospatial data in a catalog, manage and style their data, and publish the resulting products through the services hosted on the LuciadFusion Platform.

The LuciadFusion Platform hosts multiple services that allow you to publish your data to any geospatial application through various OGC and custom protocols:

For each service type, except for the catalog and symbology services, you can create multiple instances, each hosting many products. You can set up two WMS services, for instance.

The LuciadFusion Platform is based on the well-known Spring framework for system and application development. Java developers familiar with the Spring framework can quickly familiarize themselves with the LuciadFusion Platform as well. They can also easily customize and extend the platform to suit the needs of their organization.

The LuciadFusion Platform also allows developers to add and remove services, customize services through configuration, and develop and plug in their own services, to perform additional data analysis for example.

The Data Connectivity panel contains the following tabs:

You can enable or disable these tabs from the menu Tools| Fusion. The information and functionality provided by each of these tabs is listed below. A typical use case of the Data Connectivity Manager describes a typical use case of working with the Data Connectivity Manager.

The Data Properties allows you to display and edit metadata of a selected resource in the Metadata Editor.

See the next section for an explanation of the information and functionality provided by the Metadata Editor. A typical use case of the Data Connectivity Manager describes a typical use case of working with the Data Connectivity Manager.

The first step of the sample scenario is connecting to a Data Server and opening a Tile Store from the server.

When the DCM starts up, the Tile Store connection dialog allows you to open or connect to an existing Tile Store on a Data Server.

If you wish to connect to a different Tile Store at a later time, click the Cancel button. In the DCM, click on the Add icon in the Tile Store Manager header. Select Tile Store from the drop-down menu to bring up the Tile Store connection dialog.

The Favorites section shows the Tile Stores that have already been opened once before. If the Favorites section does not list the Tile Store that you want to open, you can enter the URI of the Data Server in the URI box. For example, the URI of the default Data Server is: http://localhost:8081/LuciadFusion/lts. By default, this Tile Store is available in the Favorites section already. Select the Tile Store and click on the OK button to open the Tile Store and add it to the Tile Stores tab.

To see the resources that are contained in the Tile Store tree, click the + expanding icons next to the Tile Store name and the Coverages and Themes folders. Similarly, to see the resources that are part of a coverage or theme, click the + icon next to the name of a coverage or theme. Figure 7, “The opened Tile Store with its resources” shows the Tile Store tab with the opened Tile Store and its resources.

To visualize a resource in the Map panel, select the resource and click on the icon. A layer is created for the resource and is added to the Map layers panel. Double-click on the layer to show the coverage on the map as shown in Figure 8, “The earth elevation layer visualized on the map”.

The layer is by default visualized in a 2D map but you can also visualize the layer in a 3D map. In the menu bar of the Data Connectivity Manager, select File| New| 3D Map. A 3D map opens and the Map layers panel changes to the contents of the 3D map. To copy layers from the 2D map to the 3D map, select the icon in the tool bar of the 3D layer control. Select layers from the dialog box that appears and click on the OK button to add the layers to the 3D map. The added layers appear in the 3D Map layers panel. Double-click on a layer to visualize it on the map. For more information for working with 3D maps, refer to the Lucy User’s Guide.

Non-tiled coverages cannot be visualized, but their assets can, as long as the asset’s location is a valid path for the DCM.

The Data Connectivity Manager is typically used to open data (assets) from different sources and fuse the retrieved data into a new or existing Tile Store. To load new assets, use the File| Open menu item. The Open dialog box allows you to select the files containing the assets you want to load. Click on Open to add the files to the Map layers panel.

If you want to load tile set data from a Web Map Tile Service (WMTS) service, use the File| Connect To…​ menu item, and enter the URL of the WMTS web service in the Connect To…​ dialog. If a Choose a format dialog opens, select the WMTS Client option. Subsequently, the WMTS Client dialog allows you to select one or more of the available WMTS layers. The selected WMTS layers are added to the Map Layers panel.

Table 1, “Default supported file formats” in Appendix A, Supported formats shows the data formats and corresponding file extensions that LuciadFusion supports by default.

The file Ithaca_EastNW_tile3.sid is loaded from the folder \ samples\ resources\ Data\ MrSID. A layer is created for the loaded file and is added to the Map layers panel.

After double-clicking on the layer, the data is displayed in the view as shown in Figure 9, “The file Ithaca_EastNW_tile3.sid visualized on the map”. You can add the data to a coverage to the Tile Store as described in Adding data to a Tile Store.

The default Tile Store contains the following coverages:

In a typical use case, data is retrieved from multiple sources and added to new or existing coverages. In some cases, the asset is in a Tile Store already, and you can add the asset directly from one coverage to another in the Tile Store Manager. If the loaded data is not in a Tile Store yet, you can add the data to a coverage from the Map layers panel.

To add data, consider whether the data is available in a coverage or not:

In the sample scenario, the layer that contains the file Ithaca_EastNW_tile3.sid is added a new coverage. The Tile Store tree colors blue in the locations with uncommitted changes. The new data itself is indicated in green. To commit the changes you need to use the Commit button or icon as described in Committing changes to the Tile Store. Figure 13, “The Ithaca data added to the Tile Store” shows the Tile Store Manager tab with the new Ithaca data.

You can also create a new, empty coverage. Select the Coverages folder, click on the icon, and then select Coverage from the drop-down list. A dialog box appears as shown in Figure 14, “The dialog box for creating a new, empty coverage”. You can change the given ID for the coverage if desired, provide a name for the new coverage (Ithaca), and select the coverage type (IMAGE). Click on the OK button to create the coverage.

This coverage is empty.

The order of assets in a coverage defines the visibility of the assets in the target coverage. Assets high in the list will be fused on top of the lower ones. As long as they have not been committed, it is possible to change the order of the assets in a coverage by dragging and dropping them. Assets cannot be dragged below already committed ones.

When assets are added to a coverage, they are generally inserted according to their natural order defined by their compilation scale, so that the most detailed assets are fused on top of the less detailed ones. The one exception is when an asset is dragged at a specific position under a coverage node. In that case, the asset will be inserted at that exact position.

Figure 15, “Dragging an asset on a coverage will insert it according to its natural order (left), while dragging it under a coverage node will insert it at the mouse pointer, in between assets (right).” illustrates the difference in semantics between dragging an asset on a coverage node and dragging it under a coverage node.

The LuciadFusion server allows exposing a fused raster coverage via the OGC Web Map Tile Service (WMTS) protocol.

WMTS also defines a limited list of well-known scale sets, with a fixed predefined tile pyramid structure. You can make a new, uncommitted raster coverage compliant with the GoogleMapsCompatible scale set by clicking the button on the Metadata Editor tab. As a result, the georeference, the bounding box, the tile layout and tile size will match the GoogleMapsCompatible tile structure, as defined by WMTS. Note that this function is also only available as long as the coverage has not been committed yet.

In the sample scenario, a new coverage has been created containing an asset that covers the area of Ithaca. For illustration purposes, this scenario shows how to add the new coverage to a theme and save the theme in the Tile Store. In this case the theme to which you can add the coverage does not exist yet. The first step is thus creating a theme. Select the Theme folder and click on the icon, and then select Theme from the drop-down list. A dialog box appears that allows you to change the given ID and provide a name for the new theme (Ithaca). Click on the OK button to create the theme. The new theme is added to the Themes folder.

Next, right-click the Ithaca coverage, select Add to theme from the options menu, and select the Ithaca theme in the Choose destination themes dialog. The coverage is added to the theme, as shown in Figure 16, “A newly created theme with the Ithaca coverage”

Changes to a Tile Store that have not been committed yet to the actual Tile Store on the Data Server are indicated in one of the following colors:

Once a change has been made to a Tile Store you can commit the change immediately or you can wait and commit several changes at once. To commit changes to the actual Tile Store on the Data Server, click on the Commit button or icon. A dialog box appears as shown in Figure 17, “The changes made to the Tile Store in the sample scenario”. Since no changes have been committed yet in the scenario, the dialog box shows all changes made to the Tile Store.

After the commit, the color indications will disappear to reflect the committed status of the resources. Move the mouse over the coverage name to display an information box and check the status of the committed change. To run the fusion job for a committed coverage, refer to Running a fusion job.

You cannot change a coverage once it has been committed, aside from its name and its abstract. If you need to make changes to a committed coverage, you need to clone the coverage, and apply your changes in the clone. You can then proceed to commit and fuse the coverage clone.

To clone a coverage, right-click the coverage in the Tile Store Manager, and select the Clone menu item. The coverage clone is immediately added to the Tile Store Manager, and has the same name, with an additional copy number, (1) for instance.

If a change to a coverage has been committed, a fusion job is required as indicated by the fusion (gear) icon. An additional information box with fusion details appears when you move the mouse over the coverage name.

By default, all fusion jobs are executed on the local machine. However, if you are currently connected to a Data Server that is also offering a Luciad Fusion Service (LFS), the Data Connectivity Manager will start any fusion jobs remotely on the Fusion Engine running on the Data Server.

To start the fusion job, right-click on the coverage and select Start fusion job or click on the icon above the tree view. Open the Job Manager panel to view the progress of the fusion job. The progress bar shows the progress of the fusion job in percentages. You can check the following details of the job:

Figure 18, “The running Ithaca fusion job” shows the running Ithaca fusion job.

To cancel the job, click on the icon. Note that this action will delete the job.

To stop the job, click on the icon. You can start the stopped job later by clicking on the icon again. Note that you can even restart a stopped fusion job when the application is restarted.

LuciadFusion allows you to export a coverage from one Tile Store to another. During such an export, LuciadFusion takes a copy of the fused coverage data, and inserts the copy into the other Tile Store. There is no need to commit and fuse the coverage again in the target Tile Store.

To export coverages from one Tile Store to another one:

Note that while the export is ongoing, you can already visualize the result on the target Tile Store by creating a map layer from the coverage.

LuciadFusion allows you to export raster coverages from the Data Connectivity Manager, and save them in the OGC GeoPackage format.

GeoPackage is an OGC standard format for the storage of geospatial information, based on SQLite. It is lightweight, platform-independent, and designed to enhance interoperability between geospatial applications in all kinds of environments. For instance, you can export a LuciadFusion raster coverage as a GeoPackage file, and then use the coverage data on a mobile device.

To save a raster coverage in the GeoPackage format:

The generic type RASTER is a supertype of types IMAGE or ELEVATION. If you select the RASTER type, the image processing API is used to fuse the source data, and you can preserve the band semantics of the source data.

To help you decide whether to select either the RASTER type, one of the other two raster coverage types, or even fusion to an IMAGE coverage, we list its advantages and limitations.

Choose the RASTER coverage type if you want to:

For more information about fusing multi-band and multi-dimensional data, see Fusing multi-dimensional data and multi-band data.

Generic RASTER fusion is more powerful than the other two raster coverage types. The other two types can still be useful to you if you have fusion requirements do not match with the limitations of the RASTER coverage type:

By default, assets in a coverage are ordered according to their pixel density. The most-detailed assets are on top of the less-detailed ones. As long as a coverage has not been committed, it is possible to manually re-order the assets through drag-and-drop. Once a coverage has been committed but has not been fused yet, you can no longer re-order the assets.

When a coverage has finished fusing, you can add assets to it again, but only on top of the already existing assets. You can then fuse the coverage again. This process is known as fusion of additional detail, sometimes also called incremental fusion. You can use it to fuse a low-resolution base layer first, and add a high-resolution imagery on top later on.

Because LuciadFusion is intended for combining assets with heterogeneous geographic references and resolutions into a single coverage, the coverage’s rasters correspond to the different levels of detail, and often do not match exactly with the assets' rasters. LuciadFusion interpolates the input data to fit the rasters of the coverage.

If all assets have the same natural resolution and geographic reference, you can often eliminate interpolation at the most detailed level by aligning the coverage’s raster with the assets' rasters. Doing so will effectively preserve the original resolution and sample values of the input data at the most detailed level. Less detailed levels will still be interpolated. LuciadFusion does not perform this alignment automatically, but you can manually configure the coverage for this.

For example, suppose you want to fuse multiple raster assets with EPSG:4326 as the common geographic reference and a resolution of 1' x 1', aligned to the Earth’s longitude/latitude grid. The following coverage configuration will align the coverage’s raster at level 4:

A tile size of 450 x 450 also results in an aligned raster, but at level 3 instead of level 4. There is always a mathematical solution, but sometimes the resulting tile size is impractically large.

Some raster formats, such as GeoTIFF, can have multiple levels of raster data which are structured in a multilevel raster. LuciadFusion automatically uses the rasters at the appropriate level to produce its own tile pyramid. When the level relationship of the multilevel source data is "overview", the levels are different representations of the same data, and LuciadFusion fuses them in a mutually exclusive way. This means that one level of source data will replace another in the fused coverage. When the level relationship of the multi-level source data is "stack", the levels are unrelated, and LuciadFusion fuses them as unrelated source data. This means that one level of source data may be fused "on top" of another one.

Some raster formats, such as CADRG or ECRG, can have multiple rasters which are not structured in a multilevel raster. This is semantically the same as a multilevel raster with level relationship "stack". LuciadFusion treats it as such: as unrelated source data.

The NetCDF data format describes array-oriented scientific data. NetCDF scientific weather data is stored as gridded datasets. The types of measurement and the dimensional axes along which they are measured determine the structure of the datasets. The measurements are referred to as variables. Temperature, humidity, and cloud cover are all examples of variables. The dimensions of the measurements typically consist of the location coordinates of the measurement, the time of the measurement, and other parameters, such as the altitude at which the measurement was taken. For instance, atmospheric temperature can be measured at a specific location, but at multiple times and vertical positions. Wind data typically comes as two wind component variables, U and V.

These measurements result in a NetCDF temperature dataset, with measurement grids organized in spatial, temporal and vertical dimensions. By default, NetCDF allows multiple sets of variables in a single file, so one NetCDF file can contain both salinity and temperature measurements for example.

Multi-band or multi-spectral images offer data captured on spectral bands other than the visible wavelengths of the electromagnetic spectrum, such as infrared and radar bands. Images captured on the infrared bands, for example, are extremely useful for visualizing and analyzing surface and soil features, such as vegetation, soil moisture levels, crop health and the presence of man-made structures in a natural area.

To fuse a set of NetCDF or GRIB data, select the Raster coverage type when you are setting up the coverage.

LuciadFusion fuses the NetCDF or GRIB data to a raster coverage as follows:

Because fusion as raster coverage preserves the multi-dimensional and multi-band nature of the NetCDF data, the decoded model for a raster coverage will be indistinguishable from the original NetCDF model. You can therefore treat it as a regular NetCDF model for most purposes such as painting, retrieving values, and so on.

It is also possible to fuse NetCDF data to an image coverage.

Because an image coverage is uni-dimensional, you can only select a single combination of possible multi-dimensional values. The image tiles will be rendered for that combination only. You can select this combination using the Dimensional filters panel on the coverage Metadata Editor; you can filter the image coverage for a specific time and a specific vertical value to obtain an image and fuse that single image.

Even though the fusion of NetCDF data to a raster coverage offers more possibilities than the fusion to an image, you can still select an image or elevation coverage type for the following reasons:

You can fuse data consisting of multiple bands to either a raster coverage or an image coverage. If you want to preserve the bands after fusion, select the raster coverage type when you are setting a coverage for your multi-band assets.

Multi-band raster coverages preserve all available bands in the multi-band data set in the coverage. For instance, if you fuse 7-band satellite imagery, the 7 bands will still be available in the coverage for further processing and analysis. To choose what bands in the sets you want to visualize, you can map three of the bands to the Red, Green and Blue color channels in the Band selection drop-down menus of the asset Metadata Editor.

If you do not want to preserve the bands in a multi-band data sets, you can select an image coverage type. On the asset Metadata editor, you can select which three bands will be preserved as colors in the resulting coverage. Map the required three bands to color channels in the Band selection drop-down menus. All information about the other bands will be discarded during fusion.

You can fuse GDAL data to a raster coverage. In this case, the multi-band semantics are preserved.

When you fuse multi-band data to an image coverage, you are projecting the multi-band information onto RGB color space. Therefore, you need to define a filter as if you were visualizing the information: you can choose which bands you want to visualize as an RGB image on the Asset metadata editor. The resulting image coverage will contain RGB images.

If the source data is elevation data, you can fuse it to an elevation coverage. There is no band selection in this case because the source already consists of single-band elevation data.

The first option is to fuse to an image coverage (a raster coverage of type IMAGE). In this configuration, tiles are rasterized with a fixed default styling. The default styling can be configured in the coverage metadata panel, and will only be shown when applicable, as in Figure 22, “The ECDIS display settings in the coverage metadata panel”. See the Lucy User’s Guide for a description of the fields in this panel.

Fusing to an image coverage has the following advantages and limitations:

LuciadFusion supports both the unencrypted S-57 format and the encrypted S-63 format as low-level ECDIS data formats. It will decrypt and encrypt S-63 data as needed while fusing and serving the data.

To decrypt the data, LuciadFusion uses a key based on the S-63 hardware ID, which is part of the LuciadFusion license. In the absence of a hardware ID, the official IHO hardware ID for S-63 test data will be used as a fallback.

The S-63 encryption standard requires that the persisted data is encrypted again, once it has been fused. To meet this requirement, LuciadFusion always automatically encrypts coverages that contain S-63 data, again by means of a key based on the S-63 hardware ID in the license.

Encrypted coverages are decrypted when they are visualized either on the client or on the server, depending on the protocol. If you do not have the correct hardware ID in your license on the system doing the visualization, the visualization will fail.

In general, you need to store a license with a valid S-63 hardware ID on the system that handles the encrypted S-63 data:

To see a list of non-fatal fusion failures, click the exclamation mark button in the toolbar of the job manager panel.

If the exclamation mark button is grayed out, there were no non-fatal failures or the fusion job has not finished yet.

You can override default failure thresholds for raster data. To do so, change the number specified in the system property com.luciad.fusion.engine.rasterFailureThreshold=<whole number>.

To configure Tile Store replication in the DCM:

To leave replication mode at any time, click the Revert button at the bottom of the Tile Store Manager panel.

By default, any coverage that is present on the slave tile story only, will be deleted during replication. In the Tile Store Manager panel, the coverage name will be struck-through. If you want to preserve the slave coverage, right-click the coverage while in replication mode, and select the Revert option.

You can also use this option to prevent that the slave coverage is updated with data from the master coverage, or to exclude a coverage on the master Tile Store from replication to the slave.

You must configure data consistency on the slave, and set its "commit policy" to PER_COVERAGE. You cannot do this through the DCM. You need to configure this in the deployment descriptor web.xml.

A PER_COVERAGE commit policy ensures that the view on the coverage data is consistent during the replication process. As long as replication for that coverage is ongoing, you will be viewing the old data consistently. When replication finishes for that coverage, the view consistently switches to the new data.

Creating a new Tile Store using the Data Connectivity Manager consists of the following steps:

The Tile Store Manager tab adds a new Tile Store to the list including empty folders for coverages, assets, and themes. To add data to the Tile Store you need to:

The general use case of the Data Connectivity Manager is to manage one Tile Store by one person at a time. Nevertheless, it is possible to have multiple sessions of the Data Connectivity Manager running which can access the same Tile Store at the same time. In this case, any changes committed to the Tile Store from one session first have to be ready before changes from another session can be made to the same Tile Store. When it is not possible to commit changes to a Tile Store because of concurrent commits, an error message will appear.