Visualizing large vector data sets

The visualization of a large data set on a map poses 2 big challenges:

  • Preventing visual clutter so that the user can distinguish between the features. Typically, this is achieved by painting less data when the map is zoomed out.

  • Keeping the map and application responsive by preventing that all data is loaded in memory.

Those two requirements go hand in hand: if the number of features that is visible on the map is limited at all times, the amount of data that must be loaded into memory is kept under control as well.

This article describes how you can use a TLcdModelQueryConfiguration to control the amount of data that is loaded by the layer during a paint operation. You can create such a TLcdModelQueryConfiguration through the API or through the use of an SLD styling file. See Limiting data loading through the API and Limiting data loading using SLD for more information about both approaches. The subsequent sections provide illustration with some examples.

For a walkthrough of a concrete example, see How to visualize OpenStreetMap data with LuciadLightspeed.

The role of the TLcdModelQueryConfiguration

When a layer is visualized (= painted) onto a map, the layer needs to retrieve the relevant data from the ILcdModel. It is important that the layer queries the model for as little data as possible.

Data in the model does not necessarily consume JVM memory. For example, when the data from a database is being visualized, the data is stored in a database outside of the JVM. However, once the model sends data to the layer to be painted, the data gets loaded into the JVM memory.

Therefore, the layer should only request data that:

  • Lies within the current bounds of the view. For example, when the map is centered on America, it makes no sense to query the model for data of Europe because that data will not be visible to the user.

  • Will effectively be painted. When a layer is painted, the ILspStyler (for Lightspeed layers) or the ILcdGXYPainter (for GXY layers) can decide to skip a feature instead of visualizing it. If the feature is not going to be painted, there is no need to retrieve it from the model.

While the layer can automatically derive the currently visible bounds from the view, it has no information about the features that need to be visualized. You can provide that information to the layer by configuring a TLcdModelQueryConfiguration on the layer.

The TLcdModelQueryConfiguration defines a filter for each map scale. That filter is used to query the model through the ILcdModel.query method. The filter takes the form of an ILcdOGCCondition, which allows the model to interpret the filter.

For more information about how a filter can influence the performance of a query, see Model query performance notes.

Once a TLcdModelQueryConfiguration has been created, it can be configured on the layer using:

  • TLspLayer.setModelQueryConfiguration or TLspShapeLayerBuilder.setModelQueryConfiguration for Lightspeed layers.

  • TLcdGXYLayer.setModelQueryConfiguration for GXY layers.

Examples of the creation and configuration of such a TLcdModelQueryConfiguration are available in the following sections.

Limiting data loading through the API

You can limit the amount of data that is loaded by the layer during a paint operation by configuring a TLcdModelQueryConfiguration on the layer. As a result, you prevent that the layer needs to requests all data from the model.

For example, when visualizing roads data, you would:

  • Only show the highways when the map has been zoomed out

  • Start including the minor roads when the user zooms in on the map

You can express this in a TLcdModelQueryConfiguration, and configure it on the layer:

Program: Creating and configuring a TLcdModelQueryConfiguration
import static com.luciad.ogc.filter.model.TLcdOGCFilterFactory.*;
import static com.luciad.view.TLcdModelQueryConfiguration.*;

    //Define filters for the different types of data you want to load
    //This code uses the utility methods from TLcdOGCFilterFactory
    ILcdOGCCondition highwaysFilter =
        eq(property("roadType"), literal("highway"));//roadType == highway
    ILcdOGCCondition minorRoadsFilter =
        eq(property("roadType"), literal("minor"));//roadType == minor

    //Create the model query configuration using the Builder class
    //A model query configuration is used to define which data should be loaded at which scale
    TLcdModelQueryConfiguration config =
        TLcdModelQueryConfiguration.newBuilder()
                                   //always include the highways
                                   .addContent(FULLY_ZOOMED_OUT, FULLY_ZOOMED_IN, highwaysFilter)
                                   //include the minor roads only from a certain scale
                                   .addContent(1.0 / 50000, FULLY_ZOOMED_IN, minorRoadsFilter)
                                   .build();

    //Set it on your Lightspeed layer
    ILcdModel model = createModel();
    TLspLayer layer = TLspShapeLayerBuilder.newBuilder()
                                           .model(model)
                                           .modelQueryConfiguration(config)
                                           .build();

    //Or, set it on a GXY layer
    TLcdGXYLayer gxyLayer = createLayer();
    gxyLayer.setModelQueryConfiguration(config);

See Example: visualizing roads data for more details of this example.

Limiting data loading using SLD

Introduction to SLD files

SLD or Styled Layer Descriptor is an OGC standard for describing the appearance of a layer. It allows you to define:

  • Which features to load at a certain scale

  • How to style those features

You can do so by defining SLD rules. For example, the following rule ensures that bicycle roads are visualized only when the user zooms in beyond a scale of 1/2000. The bicycle roads are visualized with a blue dashed line.

Program: Example of an SLD rule
 <Rule>
   <!-- The filter defines which to which features the rule applies -->
   <ogc:Filter>
      <ogc:PropertyIsEqualTo>
         <ogc:PropertyName>roadType</ogc:PropertyName>
         <ogc:Literal>bicycle</ogc:Literal>
      </ogc:PropertyIsEqualTo>
   </ogc:Filter>

   <!-- Minimum and maximum scale denominators define the valid scale range of the rule -->
   <MaxScaleDenominator>2000</MaxScaleDenominator>

   <!-- The symbolizer(s) define the styling to use -->
   <LineSymbolizer>
     <Stroke>
       <CssParameter name="stroke">#0000FF</CssParameter>
       <CssParameter name="stroke-width">3</CssParameter>
       <CssParameter name="stroke-dasharray">5 2</CssParameter>
     </Stroke>
   </LineSymbolizer>
</Rule>

See Styling data with OGC SLD for more information about SLD, and how to use SLD in LuciadLightspeed.

Limit data loading using SLD

To limit the amount of data loaded during a paint operation, you can manually define filters and store them in a TLcdModelQueryConfiguration, as explained in Limiting data loading through the API. However, when you are using SLD to style your data, the SLD rules already define which filter to use for each scale . LuciadLightspeed offers utility methods and classes to re-use that information for the automatic construction of the TLcdModelQueryConfiguration.

The following program illustrates how you can do that:

Program: Configuring a TLcdModelQueryConfiguration through SLD 
//First parse the SLD file
TLcdSLDFeatureTypeStyleDecoder decoder = new TLcdSLDFeatureTypeStyleDecoder();
String source = "/path/to/SLD/file.sld";
TLcdSLDFeatureTypeStyle sldStyle = decoder.decodeFeatureTypeStyle(source);

//Configure the SLD styling on your Lightspeed layer
TLspLayer layer = TLspShapeLayerBuilder.newBuilder()
                                    .model(roadsModel)
                                    .sldStyle(sldStyle)
                                    .build();

//Or, create a GXY layer that uses this SLD
TLcdGXYSLDLayerFactory layerFactory = new TLcdGXYSLDLayerFactory();
ILcdGXYLayer gxyLayer = layerFactory.createGXYLayer(roadsModel, Collections.singletonList(sldStyle));

The utility method TLspShapeLayerBuilder.sldStyle, or the TLcdGXYSLDLayerFactory for GXY layers, ensure that the layer under construction is configured to use the SLD styling for the visualization. It will use the filters defined in the SLD rules to construct the TLcdModelQueryConfiguration and limit the amount of data loading.

When you open data on a Lightspeed view in Lucy, Lucy automatically picks up any SLD files stored next to the source data. This allows you to limit the data loading in Lucy. Consult the Lucy developer’s guide for more information.

Example: visualizing roads data

In this example, we assume that we have a data set containing roads of different types: highways, major roads, minor roads, and so on.

When the map user zooms out and looks at the whole world, we want to show the highways only. At that zoom level, it does not make sense to try and visualize any other road types, because the whole map would be cluttered with data.

When the user zooms in to a scale at which the map shows a single country, we want to show the highways and major roads. As the user zooms in further, we gradually want to include more of the smaller road types.

You can approach this directly through the TLcdModelQueryConfiguration API, or you can use an SLD file.

Using the TLcdModelQueryConfiguration API

You create a TLcdModelQueryConfiguration through a Builder class:

Program: Creating a builder for a TLcdModelQueryConfiguration
TLcdModelQueryConfiguration.Builder builder = TLcdModelQueryConfiguration.newBuilder();

On that builder, we specify when each of the road types should be queried from the model. We start with the highways, which are visible at all times. As such, they must be queried from the model at each scale.

Program: Configuring the filter for the highways
//Create a condition which only accepts features where the "roadType" property has "highway" as value
ILcdOGCCondition highwaysCondition =
  TLcdOGCFilterFactory.eq(TLcdOGCFilterFactory.property("roadType"),TLcdOGCFilterFactory.literal("highway"));

//Put the condition on the filter for the whole scale range
builder.addContent(TLcdModelQueryConfiguration.FULLY_ZOOMED_OUT, TLcdModelQueryConfiguration.FULLY_ZOOMED_IN, highwaysCondition);

The major roads should only be visible when the user starts zooming in.

Program: Configuring the filter for the major roads
ILcdOGCCondition majorRoadsCondition =
  TLcdOGCFilterFactory.eq(TLcdOGCFilterFactory.property("roadType"),TLcdOGCFilterFactory.literal("major_road"));

//Major roads should be visible from a certain scale until completely zoomed in
builder.addContent(1.0 / 700000.0, TLcdModelQueryConfiguration.FULLY_ZOOMED_IN, majorRoadsCondition);

Once we have done the same for all other road types, we build the configuration and assign it to the layer:

Program: Setting the TLcdModelQueryConfiguration on the layer 
TLcdModelQueryConfiguration config = builder.build();

//Lightspeed layer
TLspLayer roadsLayer =
  TLspShapeLayerBuilder.newBuilder().model(roadsModel)
                                    .modelQueryConfiguration(config)
                                    .bodyStyler(...)
                                    .labelStyler(...)
                                    .build();

//GXY layer
TLcdGXYLayer roadsGXYLayer = ...;
layer.setModelQueryConfiguration(config);

Code readability can be improved by using some static imports, and by using the fluent API of the builder class:

Program: Full code
import static com.luciad.ogc.filter.model.TLcdOGCFilterFactory.*;
import static com.luciad.view.TLcdModelQueryConfiguration.*;

ILcdOGCCondition highwaysCondition = eq(property("roadType"),literal("highway"));
ILcdOGCCondition majorRoadsCondition = eq(property("roadType"),literal("major_road"));

TLcdModelQueryConfiguration config =
  TLcdModelQueryConfiguration.newBuilder()
                             .addContent(FULLY_ZOOMED_OUT, FULLY_ZOOMED_IN, highwaysCondition)
                             .addContent(1.0 / 700000.0, FULLY_ZOOMED_IN, majorRoadsCondition)
                             .build();

//Lightspeed layer
TLspLayer roadsLayer =
  TLspShapeLayerBuilder.newBuilder().model(roadsModel)
                                    .modelQueryConfiguration(config)
                                    .bodyStyler(...)
                                    .labelStyler(...)
                                    .build();

//GXY layer
TLcdGXYLayer roadsGXYLayer = ...;
layer.setModelQueryConfiguration(config);

Using SLD

To express the required behavior with SLD, we need to define rules for each road type. For example, highways should always be painted, so we can use a rule that does not include any scale denominators.

Program: Rule for the highways
 <Rule>
  <ogc:Filter>
    <ogc:PropertyIsEqualTo>
      <ogc:PropertyName>roadType</ogc:PropertyName>
      <ogc:Literal>highway</ogc:Literal>
    </ogc:PropertyIsEqualTo>
  </ogc:Filter>
  <LineSymbolizer>
    <Stroke>
      <SvgParameter name="stroke-width">2</SvgParameter>
      <SvgParameter name="stroke">#809bc0</SvgParameter>
    </Stroke>
  </LineSymbolizer>
</Rule>

The major roads should only become visible when the user starts zooming in. Therefore, we specify a minimum scale denominator in the major roads rule.

Program: Rule for the major roads
 <Rule>
  <ogc:Filter>
    <ogc:PropertyIsEqualTo>
      <ogc:PropertyName>roadType</ogc:PropertyName>
      <ogc:Literal>major_road</ogc:Literal>
    </ogc:PropertyIsEqualTo>
  </ogc:Filter>
  <MinScaleDenominator>2500000</MinScaleDenominator>
  <LineSymbolizer>
    <Stroke>
      <SvgParameter name="stroke-width">1</SvgParameter>
      <SvgParameter name="stroke">#97d397</SvgParameter>
    </Stroke>
  </LineSymbolizer>
</Rule>

Once we have rules for each type of road we want to include, we combine them into an SLD file:

Program: Combining the rules in a single SLD file
<?xml version="1.0" encoding="ISO-8859-1"?>
<FeatureTypeStyle
   xsi:schemaLocation="http://www.opengis.net/sld StyledLayerDescriptor.xsd"
   xmlns="http://www.opengis.net/sld"
   xmlns:ogc="http://www.opengis.net/ogc"
   xmlns:xlink="http://www.w3.org/1999/xlink"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

   <!-- Put all the rules created above here -->
   <Rule></Rule>
</FeatureTypeStyle>

You apply the SLD file by decoding it and configuring it on the layer:

Program: Applying the SLD on a layer
TLcdSLDFeatureTypeStyleDecoder decoder = new TLcdSLDFeatureTypeStyleDecoder();
String source = "/path/to/SLD/file.sld";
TLcdSLDFeatureTypeStyle sldStyle = decoder.decodeFeatureTypeStyle(source);

//For Lightspeed layers
TLspLayer layer =
  TLspShapeLayerBuilder.newBuilder().model(roadsModel)
                                    .sldStyle(sldStyle)
                                    .build();

//For GXY layers
TLcdGXYSLDLayerFactory layerFactory = new TLcdGXYSLDLayerFactory();
ILcdGXYLayer gxyLayer =
  layerFactory.createGXYLayer(roadsModel, Collections.singletonList(sldStyle));

Example: limiting visualization to a specific scale range

In this example, we assume that we have a data set containing very detailed features. The data should be visualized on the map only when the user has zoomed in to a sufficiently detailed level. When the data is visualized, all features for the current region must be painted. No extra filtering is required.

We can approach this by using the TLcdModelQueryConfiguration API directly, or by using an SLD file.

This example is the equivalent of calling TLspLayer.setMapScaleRange or TLcdGXYLayer.setMapScaleRange.

Using the TLcdModelQueryConfiguration API

You create a TLcdModelQueryConfiguration through a Builder class:

Program: Creating a builder for a TLcdModelQueryConfiguration
TLcdModelQueryConfiguration.Builder builder = TLcdModelQueryConfiguration.newBuilder();

On this builder, we define that all data must be loaded in the required scale range:

Program: Load everything in the desired scale range
//Use null as condition to indicate that no filtering is needed
ILcdOGCCondition loadAllCondition = null;

//Put the condition on the filter for the desired scale range
builder.addContent(1.0 / 75000.0, TLcdModelQueryConfiguration.FULLY_ZOOMED_IN, loadAllCondition);

Next, we have to indicate that no data should be loaded for any scale outside this range:

Program: Load nothing outside the desired scale range
builder.loadNothingForUndefinedScales();

The configuration can now be put on the layer:

Program: Setting the TLcdModelQueryConfiguration on the layer 
TLcdModelQueryConfiguration config = builder.build();

//Lightspeed layer
TLspLayer layer =
  TLspShapeLayerBuilder.newBuilder().model(model)
                                    .modelQueryConfiguration(config)
                                    .build();
//GXY layer
TLcdGXYLayer gxyLayer = ...;
layer.setModelQueryConfiguration(config);

Code readability can be improved by using some static imports, and by using the fluent API of the builder class:

Program: Full code
import static com.luciad.ogc.filter.model.TLcdOGCFilterFactory.*;
import static com.luciad.view.TLcdModelQueryConfiguration.*;

ILcdOGCCondition loadAllCondition = null;
TLcdModelQueryConfiguration config =
  TLcdModelQueryConfiguration.newBuilder()
                             .addContent(1.0 / 75000.0, FULLY_ZOOMED_IN, loadAllCondition)
                             .loadNothingForUndefinedScales()
                             .build();

//Lightspeed layer
TLspLayer layer =
  TLspShapeLayerBuilder.newBuilder().model(model)
                                    .modelQueryConfiguration(config)
                                    .build();
//GXY layer
TLcdGXYLayer gxyLayer = ...;
layer.setModelQueryConfiguration(config);

Using SLD

To express the required behavior with SLD, we need an SLD file with a single rule. The rule should not contain a filter and apply a maximum scale denominator to ensure that the data becomes visible only when the user zooms in.

Program: The SLD containing a single rule
<?xml version="1.0" encoding="ISO-8859-1"?>
<FeatureTypeStyle
   xsi:schemaLocation="http://www.opengis.net/sld StyledLayerDescriptor.xsd"
   xmlns="http://www.opengis.net/sld"
   xmlns:ogc="http://www.opengis.net/ogc"
   xmlns:xlink="http://www.w3.org/1999/xlink"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Rule>
    <MaxScaleDenominator>750000</MaxScaleDenominator>
    <LineSymbolizer>
      <Stroke>
        <SvgParameter name="stroke-width">1</SvgParameter>
        <SvgParameter name="stroke">#97d397</SvgParameter>
      </Stroke>
    </LineSymbolizer>
  </Rule>
</FeatureTypeStyle>

You apply the SLD file by decoding it and configuring it on the layer:

Program: Applying the SLD on a layer
TLcdSLDFeatureTypeStyleDecoder decoder = new TLcdSLDFeatureTypeStyleDecoder();
String source = "/path/to/SLD/file.sld";
TLcdSLDFeatureTypeStyle sldStyle = decoder.decodeFeatureTypeStyle(source);

//For Lightspeed layers
TLspLayer layer =
  TLspShapeLayerBuilder.newBuilder().model(roadsModel)
                                    .sldStyle(sldStyle)
                                    .build();

//For GXY layers
TLcdGXYSLDLayerFactory layerFactory = new TLcdGXYSLDLayerFactory();
ILcdGXYLayer gxyLayer =
  layerFactory.createGXYLayer(roadsModel, Collections.singletonList(sldStyle));

If you need more control over how the view scale influences a style, you can create your own ILspMapScaleProvider and configure it on the SLD styler. This is explained in more detail in How to lock scale-based styles to a certain zoom level.

Model query performance notes

The retrieval of the model data during a paint operation goes through the ILcdModel.query method. Therefore, the painting performance depends heavily on the performance of that method.

Consult Querying a model for guidelines for influencing this performance.

Other ways to limit data

Apart from specifying a model query configuration on your layer, there are other ways to limit the amount of data loaded in a layer.

Minimum object size

You can specify a minimum object size, in pixels on the screen, for the appearance of objects. For example, if you specify a minimum size of 5, an object is displayed only if its geometry would be about 5x5 pixels large on the screen.

You can achieve an automatic level-of-detail effect in this way: small objects automatically disappear when the user zooms out, but re-appear when the user zooms in.

This is particularly useful for polygon datasets with building or land usage features:

large vector data minsize
Figure 1. Left: with minimum pixel size - Right: without minimum pixel size

For more information, see TLspShapeLayerBuilder.minimumObjectSizeForPainting or TLcdGXYLayer.setMinimumObjectSizeForPainting.

The minimum object size condition is evaluated by the model. As with an OGC condition, the performance depends largely on the model implementation. Database models will evaluate the condition using SQL, WFS clients will send it to the WFS server if the server supports it. SHP models will do a fairly cheap pre-check to avoid loading the entire geometry.

Layer filter

You can specify a layer filter (ILcdFilter) that is evaluated on all loaded elements. For more information, see the TLspLayer.setFilter and TLcdGXYLayer.setFilter reference documentation.

We recommend using TLcdModelQueryConfiguration whenever possible, because that is evaluated inside the model, and the model can optimize it greatly. A layer filter can only be applied inside the layer after all elements have been loaded into memory.

Scales and filters in a 3D view

Unlike a 2D view, a 3D view does not have a single map scale. Each point in the visible area in a 3D view has a distinct scale.

This picture shows the local scale computed at different spots in the view. You can see that the scale near the horizon is vastly different from the scale near the bottom of the screen:

large vector data 3D map scales
Figure 2. Local scale at different spots in a 3D view.

If you have a model query configuration on a layer in 3D, the layer will use different OGC conditions for different regions in your view at the same time. Each scale range in your model query configuration is a contiguous geographic area shaped like an arc band:

large vector data 3D scale region
Figure 3. Arc-shaped area with constant scale in a 3D view.

The layer approximates these areas with a set of tiles, and loads data using the corresponding OGC condition:

large vector data OSM roads
Figure 4. OpenStreetMap roads data example, illustrating the different conditions used for each tile.

Visualizing the data on a GXY view

The concept of limiting the data queried by the layer by configuring a TLcdModelQueryConfiguration on the layer applies to both GXY layers and Lightspeed layers. This is illustrated in all the examples in this article: each example contains code to create a Lightspeed layer and a GXY layer.