There are only a few standardized formats for exchanging panorama data. LuciadLightspeed and LuciadFusion support the Pegasus Mobile Mapping and E57 formats out-of-the-box, but chances are that your panorama data in a custom, proprietary format. If that is the case, this article is for you. It helps you plug in a decoder for most panorama formats, using our Java API.

To show you how, we load an equirectangular panorama using a custom model decoder, plug it into LuciadFusion, and process it for visualization in LuciadRIA.

Make sure to read the panoramic images overview first, before you continue in this article.

Our example data

In this how-to, we’re going to use a simple panorama captured on a phone. The image itself is equirectangular, and the EXIF tags contain the image GPS location and camera orientation.

If your panorama has no geo-reference information or location, you can still process it. You can locate it on the earth later on, in LuciadRIA. See how to process and visualize non-georefenced panoramas for more information.

The JPEG file has location information. See the GPS Altitude, GPS Position and Pose Heading EXIF tags.

panorama custom input
Figure 1. Our example input data, the Luciad office in Leuven
/path/to/panorama > exiftool PANO_20200929_152253.jpg
File Name                       : PANO_20200929_152253.jpg
File Size                       : 7.1 MB
File Modification Date/Time     : 2020:09:29 15:21:16+02:00
MIME Type                       : image/jpeg
Device Manufacturer             : Google
Is Photosphere                  : True
Projection Type                 : equirectangular
Full Pano Width Pixels          : 8704
Full Pano Height Pixels         : 4352
Pose Heading Degrees            : 10.0
GPS Altitude                    : 30.1 m Above Sea Level
GPS Date/Time                   : 2020:09:29 13:25:35Z
GPS Latitude                    : 50 deg 51' 52.99" N
GPS Longitude                   : 4 deg 40' 7.80" E
GPS Position                    : 50 deg 51' 52.99" N, 4 deg 40' 7.80" E

Setting up the model decoder

To load and process panorama data, we need an ILcdModelDecoder that creates a model that has an ILcdPanoramaModelDescriptor, and ILcdPanorama domain objects.

@LcdService(service = ILcdModelDecoder.class, priority = LcdService.HIGH_PRIORITY)
public static class MyPanoModelDecoder implements ILcdModelDecoder {

  @Override
  public String getDisplayName() {
    return "MyPano";
  }

  @Override
  public boolean canDecodeSource(String sourceName) {
    return sourceName.contains("PANO_");
  }

  @Override
  public ILcdModel decode(String sourceName) throws IOException {
    ILcdModelReference modelReference = new TLcdGeodeticReference();
    ILcdModelDescriptor modelDescriptor = new TLcdPanoramaModelDescriptor(sourceName, "MyPano", sourceName, myDataModel, singleton(myDataType), singleton(myDataType));

    TLcdVectorModel model = new TLcdVectorModel(modelReference, modelDescriptor);
    model.addElement(loadPanorama(sourceName), NO_EVENT);

    model.setModelMetadataFunction(m -> TLcdModelMetadata.newBuilder()
                                                         .fromModel(m)
                                                         .addDataCategory(DataCategory.PANORAMA)
                                                         .build());

    return model;
  }

Some things to note:

  • Annotate the decoder with LcdService, so that it gets picked up by the Java Service Loader used in TLcdCompositeModelDecoder. This way, the decoder works in the samples and in LuciadFusion.

  • The canDecodeSource method checks for a specific pattern. Tailor this to your data.

  • As a model reference, we use a standard lon-lat reference. You can also use an EPSG code in a .epsg file with TLcdEPSGModelDecoder, or WKT text in a .prj file with TLcdWKTModelDecoder.

  • Use a TLcdPanoramaModelDescriptor.

  • Optionally, set the model metadata category to TLcdModelMetadata.DataCategory.PANORAMA.

This is the data model we’re going to use for our domain objects:

private static final TLcdDataModel myDataModel;
private static final TLcdDataType myDataType;

static {
  TLcdDataModelBuilder dataModelBuilder = new TLcdDataModelBuilder("MyPano");
  TLcdDataTypeBuilder typeBuilder = dataModelBuilder.typeBuilder("MyPano");
  typeBuilder.addProperty("location", TLcdShapeDataTypes.SHAPE_TYPE);
  typeBuilder.annotateFromFactory(t -> new TLcdHasGeometryAnnotation(t.getProperty("location")));
  typeBuilder.addProperty("filename", TLcdCoreDataTypes.STRING_TYPE);

  myDataModel = dataModelBuilder.createDataModel();
  myDataType = myDataModel.getDeclaredType("MyPano");
}

Often, the feature information of panoramas comes in a format that LuciadLightspeed and LuciadFusion already support out-of-the-box, a ShapeFile or GeoJson file for example. If that’s the case, you don’t have to write a full decoder yourself.

Instead, delegate to an existing model decoder, the TLcdSHPModelDecoder for example, and wrap its domain objects into ILcdPanorama objects as described further on.

Panorama domain objects

The domain objects in a panorama model must implement ILcdPanorama.

private ILcdPanorama loadPanorama(String sourceName) throws IOException {
  // Hard-coded location & heading, but these can be extracted from the EXIF tags.
  TLcdLonLatHeightPoint location = new TLcdLonLatHeightPoint(4.66883444, 50.86472046, 30.12);
  double heading = 10.0;

  double[] rotationMatrix = getRotationMatrix(heading);

  ILcdModelModelTransformation imageTransformation = TLcdPanoramicTransformationFactory
      .newSphericalTransformation()
      .fov(360, 180)
      .rotation(rotationMatrix)
      .build();

  BufferedImage bufferedImage = ImageIO.read(new File(sourceName));

  ILcdPanoramicImage panoramicImage = new TLcdPanoramicImage(imageTransformation, bufferedImage);

  ILcdPanorama panorama = TLcdPanorama.newBuilder()
                                      .dataType(myDataType)
                                      .location(location)
                                      .image(panoramicImage)
                                      .build();
  panorama.setValue("location", location);
  panorama.setValue("filename", sourceName);

  return panorama;
}

Some thing to note:

  • In this example, the location and the heading are hard-coded. You could extract them from the EXIF tags of course, or read them from a GPX track, but we left that out to keep the code concise.

  • You can include any properties you want with the data object.

If you have many panoramas in your model, it’s better to allow lazy loading by creating and loading the images in your TLcdPanorama through a Supplier function.

Custom transformations

In our example, we use an equirectangular image transformation. These are also known as spherical transformations. You can use it out-of-the-box, just like pinhole (perspective) image transformations for regular photos.

Those are the two most common image projection types, but you can use many other types, such as fish-eye projections with various formulas.

You can plug in your own image transformation to go from image coordinates to world coordinates and back. It takes some effort, so make sure to read the panoramic images overview and the TLcdPanoramicTransformationFactory reference documentation for details.

Camera orientations and poses

Next to the raw image projection transformation, you must also orient your panorama. You need to orient your panorama correctly in the world, so that the directions in the image correspond to the real directions in the world.

In our example, we adapt the Pose Heading to a rotation matrix that rotates the input coordinates in 3D.

First, we need to align the axes to what the spherical transformation expects: the center of the image points east (X). In our input image, the center of the screen is pointing north (Y), though. We can align the axes by rotating counterclockwise around Z by 90 degrees, or you can also view this matrix as swapping X with -Y and Y with X.

Next, we must apply the heading in the EXIF tags. It’s 10 degrees clockwise starting north, so we need to rotate clockwise around the up (Z) axis by 10 degrees.

private double[] getRotationMatrix(double heading) {
  // The input pano image has center of image mapped to Y/North direction,
  // while the default spherical transformation expects center of image to map to X/East.
  // This matrix essentially turns the coordinates 90 degrees counter-clockwise around Z.
  Matrix3d axisSwap = new Matrix3d(
      0, -1, 0,
      1, 0, 0,
      0, 0, 1
  );

  // Heading is defined in degrees, clockwise starting north.
  Matrix3d headingMatrix = new Matrix3d();
  headingMatrix.rotZ(Math.toRadians(-heading));

  // Combine the two rotations
  Matrix3d rotationHeadingNorth = new Matrix3d();
  rotationHeadingNorth.setIdentity();
  rotationHeadingNorth.mul(axisSwap);
  rotationHeadingNorth.mul(headingMatrix);

  return new double[]{
      rotationHeadingNorth.m00, rotationHeadingNorth.m01, rotationHeadingNorth.m02,
      rotationHeadingNorth.m10, rotationHeadingNorth.m11, rotationHeadingNorth.m12,
      rotationHeadingNorth.m20, rotationHeadingNorth.m21, rotationHeadingNorth.m22
  };
}

Inspecting the result

With the Panorama Viewer sample, we can verify if the model decoder works correctly.

panorama custom viewer
Figure 2. The Panorama Viewer sample loads custom panorama data

Processing into Luciad Panorama Format

Now that we can decode our custom format, processing it into the Luciad Panorama Format is straightforward.

We can use our decoder to create a model, and use TLcdLuciadPanoramaModelEncoder to save it into a cubemap.json. This is what the PanoramaConverter sample already does. Because we annotated our model decoder with @LcdService, the sample picks up our decoder and uses it.

Using the custom format in LuciadFusion

LuciadFusion also uses the Java Service Loader and @LcdService annotations to discover model decoders, so it picks up the model decoder if you place it in the classpath.

LuciadFusion discovers your data during crawling, and recognizes it as panorama data.

panorama custom lf
Figure 3. Our custom panorama data crawled by LuciadFusion

For more details, see the panoramic data serving article in the LuciadFusion documentation.

You can create a PANORAMICS service for it, just like for other panorama data, and load the result in LuciadRIA.

panorama custom ria
Figure 4. Our custom panorama data loaded in LuciadRIA.