Periplo documentation

         

Periplo is an open-source console for Delta Lake data lakes, by MassiveDataScope. It discovers the Delta tables under your S3 buckets, shows their schema, statistics and history, runs read-only SQL over them, and follows the ETL deployments of a Prefect orchestrator, all from one container image.

Source code: github.com/MassiveDataScope/periplo.

Overview

• Overview
  • What it does
  • What it is not
  • How the pieces fit
  • License

Getting Started

• Quickstart
  • 1. Run the image
  • 2. Describe your lake
  • 3. Give it credentials and start it
  • 4. Optional: follow your ETLs
  • Running from a checkout
  • Next steps

Configuration

• Environment variables
  • Catalog and sources
  • Queries
  • Table metadata
  • Web console and logging
  • ETL orchestrator
  • Outside Periplo’s own settings
• Sources file
  • A complete example
  • Top level
  • Sources
  • Templates
  • Explorer
  • Discovering again

Architecture

• Architecture
  • Components
  • Life of a request
  • Catalog
  • Queries
  • HTTP API
• Extending Periplo
  • The seven ports
  • Composing a product
  • Rules every extension follows
  • The core actions
  • Filtering collections
  • Product actions
  • Orchestrator capabilities
  • Authorizer requirements
  • Credentials and data planes
  • A complete example
  • Depending on Periplo
  • Building on the container image

ETL

• ETL console
  • Connecting to Prefect
  • What it shows
  • Operating ETLs

Deployment

• Deploying with Docker
  • Images
  • What the image does
  • Running it
  • Building the image
  • Verifying a release
  • Source code for network users

Project

• Security
  • Reporting a vulnerability
  • Security model of a deployment
• Contributing
  • Before you start
  • Development setup
  • Running the checks
  • Documentation
  • Branches, commits and releases
  • License
• Changelog

Reference

• API reference
  • Extensions
  • Identity and tenancy
  • Authorization and audit
  • Storage credentials
  • Data planes
  • ETL orchestrators