In this article, we’ll explain how you can load and display worldwide OpenStreetMap vector data directly from a PostGIS database. You can apply the principles used in this articles to other large data sets, as well as other kinds of data sources, such as other databases and WFS. See the Visualizing large vector data in Lightspeed views article for more information.

LuciadLightspeed also supports loading this data as pre-rendered raster tiles; see the Visualize OSM raster tiles data on a Lightspeed map article for more information.

Why do it?

The OpenStreetMap data set offers worldwide information for roads, points of interest, buildings, and so on. Visualizing such a large data set on a map presents two main challenges:

  • Prevent visual clutter so that the user can distinguish individual features. Visual clutter is typically prevented by painting less data when the map is zoomed out to a smaller scale.

  • Keeping the map and application responsive by not loading all data in memory at once. In XML format, the data for the whole world is around 750GB, and in the binary PBF format 35GB. That is too large to load all at once.

How to do it?

LuciadLightspeed’s MVC architecture addresses these challenges by:

  • Using built-in spatial filtering to only load the data for your current view location

  • Enabling the configuration of additional filters to only load a subset of the data when the map is zoomed out. Together with the model’s capability to apply those filters directly on the database, it is possible to efficiently access and visualize OpenStreetMap vector data in a LuciadLightspeed application.

The following sections discuss the necessary steps in more detail.

Import OpenStreetMap data into a PostGIS database

You can download OpenStreetMap data for the whole world or for smaller regions, typically in the custom binary .pbf format. The PBF format is an exchange format, and not suited for lazy-loading or filtering.

Therefore, we recommend importing the data into a geospatial database so that LuciadLightspeed can easily access it. The database approach offers a few advantages:

  • Lazy and partial loading

  • Spatial or other indexes accelerate queries

In this example, we use PostgreSQL with PostGIS, but the principles demonstrated in this article can be applied to other databases as well.

To import the OpenStreetMap data into a PostGIS database you need to perform the following steps:

  1. Download planet.pbf or a smaller area of your preference from OpenStreetMap.

  2. Create a new database with the PostGIS extension enabled. To enable the PostGIS extension, execute the following query on your new database.

    CREATE EXTENSION postgis;
  3. Import the *.pbf file into the database by using the Imposm or osm2pgsql tool. For more information, see:

    • Imposm documentation

    • osm2pgsql tutorial

      imposm --read --write --connection postgis://osm:passwd@localhost/dbname planet.osm.pbf

      Imposm is a Linux tool, but you can also run it from within the bash shell (Ubuntu on Windows) on Windows 10. It’s the preferred tool to import the OpenStreetMap data, because it requires significantly less memory than osm2pgsql.

      The tool creates various tables for different kinds of data (places, buildings, …​). You can verify the content after the import, in this way for example :

      SELECT COUNT(*) count, type FROM osm_roads GROUP BY type;

      The imposm tool creates several tables with roads (minorroads, majorroads, …​) and a view that combines those tables into one. We will use that view, as it has all road data together.

  4. Define the necessary indices.

    • Spatial index. The import tool should have created a spatial index on the geometry columns already. If that is not the case, you can create one this way:

      CREATE INDEX osm_roads_geometry_geom_idx ON osm_roads (geometry);
    • Primary key index. We will use the osm_id column as primary key in our application as that is the native OpenStreetMap ID:

      CREATE INDEX osm_roads_osm_id_idx ON osm_roads (osm_id);
    • Other indices. We will be filtering on the type column, so it makes sense to have an index for it:

      CREATE INDEX osm_roads_type_idx ON osm_roads (type);
    • Sometimes, a partial index helps a great deal. In our case, we will be filtering on motorways in a certain spatial area, so a spatial index specifically for motorways is useful:

      CREATE INDEX osm_roads_geometry_motorways_idx ON osm_roads (geometry) where type = 'motorways';

Creating a LuciadLightspeed model for the database

You can easily load the data in LuciadLightspeed from the database using TLcdPostGISModelDecoder. This decoder can load a single table or view. You need to pass it a properties file containing the database connection details. The properties file must have the extension .pgs.

driver = org.postgresql.Driver
url = jdbc:postgresql://localhost:5432/osmplanet
user = postgres
password = postgres

table = osm_roads
spatialColumn = geometry

SRID = 3857
primaryFeatureIndex = 0
featureNames.0 = osm_id
featureNames.1 = name
featureNames.2 = type
featureNames.3 = z_order

bounds = -20037508, -34662081,40075016,69324162

maxCacheSize = 0

Feed that file to TLcdPostGISModelDecoder:

TLcdPostGISModelDecoder postgisDecoder = new TLcdPostGISModelDecoder();
ILcdModel model = postgisDecoder.decode("osm_roads.pgs");

No data is actually loaded at this point. The model will only load data at the request of the layer.

Using level-of-detail in a Lightspeed view

The OpenStreetMap data set contains data from across the globe, and an enormous number of objects. We want to keep the performance and interactivity of the application high, but the memory usage low.

Because this data set contains all streets, buildings, and other features of the world, it’s clear that they cannot all be visualized at the same time. Instead, we differentiate between the subtypes of features, and visualize each of them at another map scale by using OGC filters combined with scale ranges. Motorway and secondary road are subtypes of roads, for example:

  • When the map is zoomed out and displaying the whole world, we only want to show the highways. At this zoom level, it does not make sense to try visualizing any other road types or the whole map would be cluttered with data.

  • When the map is zoomed in to a scale at which it shows a single country, we want to show the highways and major roads.

  • When the map is being zoomed in further, we gradually include more and more of the smaller road types.

To do so, we can configure a model query configuration on our layer. A model query configuration TLcdModelQueryConfiguration represents level-of-detail by associating OGC conditions (filters) with scale ranges.

Let’s define a simple configuration to always show motorways:

TLcdModelQueryConfiguration.Builder builder = TLcdModelQueryConfiguration.newBuilder();

// Always show motorways.  Condition is similar to SQL: WHERE type = 'motorway'
builder.addContent(FULLY_ZOOMED_OUT, FULLY_ZOOMED_IN, eq(property("type"), literal("motorway")));

Now let’s show primary roads when the user starts zooming in:

// Major roads should be visible from a certain scale until completely zoomed in
builder.addContent(1.0 / 700000.0, FULLY_ZOOMED_IN, eq(property("type"), literal("primary")));

We can add similar rules for other road types. Once the configuration is complete, we can apply it to a layer using TLspShapeLayerBuilder.modelQueryConfiguration.

TLcdModelQueryConfiguration queryConfig = builder.build();

//Lightspeed layer
TLspLayer roadsLayer = TLspShapeLayerBuilder.newBuilder()
                                            .model(roadsModel)
                                            .modelQueryConfiguration(queryConfig)
                                            .bodyStyler(...)
                                            .labelStyler(...)
                                            .build();

The layer can now query the model with the OGC condition appropriate for the current view scale, along with a BBOX operator. The PostGIS model translates the query into a SQL statement, and lets the database perform the filtering.

GXY layers also support model query configurations. See the TLcdGXYLayer.setModelQueryConfiguration API documentation for details.

Setting a minimum object size for display

You can specify a minimum object size, in pixels on the screen, for objects to appear.

This is particularly useful for OpenStreetMap buildings and land usage polygons.

For example, if you specify a minimum size of 5, an object is displayed only if its geometry is about 5x5 pixels large on the screen, or larger.

In this way, you can get automatic level-of-detail: small objects automatically disappear when zooming out, but re-appear when zooming in.

without minimum pixel size
Figure 1. Without minimum pixel size
with minimum pixel size
Figure 2. With minimum pixel size

For more information, see the API documentation of TLspShapeLayerBuilder.minimumObjectSizeForPainting for Lightspeed layers, or TLcdGXYLayer.setMinimumObjectSizeForPainting for GXY layers.

The minimum object size is evaluated by the model. A PostGIS model will evaluate it using SQL.

Level-of-detail in 3D

Unlike a 2D view, a 3D view does not have a single map scale: each point in the visible area of a 3D view has another scale.

If you have a model query configuration on a layer in 3D, the layer will use different OGC conditions for different regions in your view at the same time. You will have different levels-of-detail within one view at the same time!

3D map scales
Figure 3. This picture shows the local scale computed at different spots in the view. You can see that the scale near the horizon is vastly different from the scale near the bottom of the screen.
3D scale region arc bands
Figure 4. Each scale range in your model query configuration is a contiguous geographic area shaped like an arc band
large road vector data
Figure 5. The layer approximates these areas with a set of tiles, and loads data using the corresponding OGC condition. Small roads nearby and major roads far away!

Using SLD for Level-of-Detail and styling

As an alternative to creating a TLcdModelQueryConfiguration, you can also use OGC SLD. SLD rules consist of a scale, an OGC filter and styling directives.

LuciadLightspeed automatically uses the scale and filter configuration in SLD as a model query configuration. You can use the <sld:Rule> with <sld:Filter> and <sld:MaxScaleDenominator>.

Here’s a very basic OGC SLD file for several road types at several scales:

roads.sld

<FeatureTypeStyle xmlns="http://www.opengis.net/se" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:gml="http://www.opengis.net/gml" xmlns:xlink="http://www.w3.org/1999/xlink"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/filter/1.1.0/filter.xsd http://www.w3.org/2001/XMLSchema http://www.w3.org/2001/XMLSchema.xsd http://www.opengis.net/se http://schemas.opengis.net/se/1.1.0/FeatureStyle.xsd http://www.opengis.net/gml http://schemas.opengis.net/gml/3.1.1/base/gml.xsd http://www.w3.org/1999/xlink http://www.w3.org/1999/xlink.xsd "
                  version="1.1.0">
  <Rule>
    <Filter xmlns="http://www.opengis.net/ogc">
        <PropertyIsEqualTo>
          <PropertyName>type</PropertyName>
          <Literal>motorway</Literal>
        </PropertyIsEqualTo>
    </Filter>
    <MaxScaleDenominator>7000000</MaxScaleDenominator>
    <LineSymbolizer>
      <Stroke>
        <SvgParameter name="stroke-width">3</SvgParameter>
        <SvgParameter name="stroke">#FFC345</SvgParameter>
      </Stroke>
    </LineSymbolizer>
  </Rule>
  <Rule>
    <Filter xmlns="http://www.opengis.net/ogc">
      <Or>
        <PropertyIsEqualTo>
          <PropertyName>type</PropertyName>
          <Literal>trunk</Literal>
        </PropertyIsEqualTo>
        <PropertyIsEqualTo>
          <PropertyName>type</PropertyName>
          <Literal>primary</Literal>
        </PropertyIsEqualTo>
      </Or>
    </Filter>
    <MaxScaleDenominator>700000</MaxScaleDenominator>
    <LineSymbolizer>
      <Stroke>
        <SvgParameter name="stroke-width">3</SvgParameter>
        <SvgParameter name="stroke">#FFFD8B</SvgParameter>
      </Stroke>
    </LineSymbolizer>
  </Rule>
  <Rule>
    <Filter xmlns="http://www.opengis.net/ogc">
        <PropertyIsEqualTo>
          <PropertyName>type</PropertyName>
          <Literal>secondary</Literal>
        </PropertyIsEqualTo>
    </Filter>
    <MaxScaleDenominator>300000</MaxScaleDenominator>
    <LineSymbolizer>
      <Stroke>
        <SvgParameter name="stroke-width">2.5</SvgParameter>
        <SvgParameter name="stroke">#FFFD8B</SvgParameter>
      </Stroke>
    </LineSymbolizer>
  </Rule>
  <Rule>
    <Filter xmlns="http://www.opengis.net/ogc">
        <PropertyIsEqualTo>
          <PropertyName>type</PropertyName>
          <Literal>tertiary</Literal>
        </PropertyIsEqualTo>
    </Filter>
    <MaxScaleDenominator>150000</MaxScaleDenominator>
    <LineSymbolizer>
      <Stroke>
        <SvgParameter name="stroke-width">2</SvgParameter>
        <SvgParameter name="stroke">#F7F496</SvgParameter>
      </Stroke>
    </LineSymbolizer>
  </Rule>
  <Rule>
    <ElseFilter/>
    <MaxScaleDenominator>75000</MaxScaleDenominator>
    <LineSymbolizer>
      <Stroke>
        <SvgParameter name="stroke-width">1.5</SvgParameter>
        <SvgParameter name="stroke">#C0C0C0</SvgParameter>
      </Stroke>
    </LineSymbolizer>
  </Rule>
</FeatureTypeStyle>

To use this SLD file on your layer, use a TLcdSLDFeatureTypeStyle.

Use SLD on your layer

//First parse the SLD file
TLcdSLDFeatureTypeStyleDecoder decoder = new TLcdSLDFeatureTypeStyleDecoder();
TLcdSLDFeatureTypeStyle sldStyle = decoder.decodeFeatureTypeStyle("/path/to/roads.sld");

//Configure the SLD styling on your Lightspeed layer
TLspLayer layer = TLspShapeLayerBuilder.newBuilder()
                                       .model(roadsModel)
                                       .sldStyle(sldStyle)
                                       .build();

//Or, create a GXY layer which uses this SLD
TLcdGXYSLDLayerFactory layerFactory = new TLcdGXYSLDLayerFactory();
ILcdGXYLayer gxyLayer = layerFactory.createGXYLayer(roadsModel, singletonList(sldStyle));

If you place the roads.sld file next to the roads.pgs file, Lucy will automatically pick it up and apply it.