Skip to content

Architecture Overview

This page describes the internal structure of the map_data package. The system is organised into four main layers: data ingestion, representation, planning, and visualisation.

Component diagram

MapData

MapData (defined in map_data/map_data.py) is the central data class. It accepts either a path to a .gpx (or .yaml) file or a pre-converted UTM coordinate array, queries the Overpass API for all OSM features inside the derived bounding box, and parses the responses into categorised Way lists.

Overpass queries. Three concurrent HTTP requests are fired in a thread pool:

  • ways query — all OSM ways (roads, footways, barriers, buildings, etc.) plus their constituent nodes
  • relations query — multipolygon relations that reference the above ways
  • nodes query — all standalone point features (obstacle nodes)

The bounding box is the convex hull of the input waypoints expanded by grid_margin metres on all sides (default: 150 m); grid_margin is a per-call override accepted by MapData.__init__, parse_osm_nodes, and separate_ways.

Core attributes after parsing:

Attribute Type Description
roads_list list[Way] Vehicle-intended highway ways, buffered to 7 m wide polygons
footways_list list[Way] Pedestrian footways, buffered to 3 m wide polygons
barriers_list list[Way] Physical barriers, buildings, water, and obstacle nodes
crossroads_list list[Way] Footway intersection points (buffered 1.5 m circles)
nodes_cache dict[int, dict] Mapping of OSM node ID to {lat, lon, tags}
zone_number int UTM zone number inferred from the input waypoints
zone_letter str UTM zone letter inferred from the input waypoints
waypoints np.ndarray (N, 2) UTM easting/northing array of the input waypoints
min_x/max_x/min_y/max_y float UTM bounding box including margins

OSM caching. After a successful Overpass query, the raw responses are saved to a .osm_cache.json sidecar file next to the source .gpx/.yaml file. On the next load, if the file exists and the stored bounding box matches the current one (within 1 × 10⁻⁶ °, ≈ 11 cm), the cache is used and the network round-trip is skipped. The cache is invalidated automatically whenever the bounding box changes.

Serialisation. MapData.save() writes a JSON file (.mapdata extension) using json.dump. MapData.load(path) reads it back. Support for the legacy pickle format has been removed for security reasons; legacy files must be re-parsed from the source GPX/YAML.

Way objects

Every OSM feature is stored as a Way instance (map_data/utils/way.py). A Way holds:

  • id — OSM way/node ID (positive integer) or a synthetic negative integer for merged multipolygon segments or viewer-drawn annotations
  • is_area — True if the geometry is a closed polygon
  • nodes — ordered list of OSM node IDs
  • tags — dict of OSM tag key-value pairs
  • line — Shapely geometry (Polygon for closed/buffered features, LineString for open features)
  • in_out — "outer" or "inner" for multipolygon relation members

All geometry is stored in UTM coordinates (same zone as the input waypoints). Ways are buffered during parsing: roads receive a 7 m half-width buffer and footways a 3 m half-width buffer, converting LineString geometries to Polygon.

Pathsolvers

The package provides two independent path-planning back-ends.

GraphPlanner

map_data/pathsolver/graph_planner.py

Builds an undirected weighted graph directly from the OSM way network (roads_list and/or footways_list). Edge weights are Euclidean distances in metres. Planning uses A* with a straight-line distance heuristic.

Waypoints passed to GraphPlanner.plan() are first snapped to the nearest graph edge using an STRtree spatial index. Viewer-drawn annotation paths (negative-ID ways) are spliced into the graph by projecting their endpoints onto the nearest existing OSM edge and inserting synthetic junction nodes at the projection points.

This planner is well-suited for route planning on well-mapped pedestrian or road networks where staying on designated paths is required.

ReplanPath

map_data/pathsolver/replan.py

Orchestrates cost-grid planning. Grid construction, smoothing, and visualization are handled by dedicated sub-modules:

Module Class / function Responsibility
pathsolver/grid_constructor.py PathGrid Builds the cost raster at cell_size metre resolution; assigns per-cell costs from highway type, surface, and barrier geometry
pathsolver/grid_astar.py grid_astar Fast, optimal A* search on the discrete grid
pathsolver/rrt_star.py RRTStar Sampling-based planner; supports Informed RRT* (ellipsoidal sampling after first path found) and adaptive neighbor radius (γ·√(log n/n)) for asymptotic optimality
pathsolver/smoothing.py smooth_path Gradient-descent path smoothing with optional collision checking
pathsolver/visualizer.py visualize_replan Matplotlib debug visualization of the grid, obstacles, and planned path

Each PathGrid cell cost is determined by:

  • the OSM highway type and surface material of the nearest way within max_path_dist metres (configured via highway_costs and surface_costs in planner_defaults.yaml)
  • a fixed default_off_path_cost for cells not covered by any way
  • barrier polygons inflated by inflate_obstacles metres, set to cost 1.0 (impassable)

Key parameters now exposed in planner_defaults.yaml: grid_cost_weight, obstacle_radius, and buffer_widths (per road type).

Viewer

The viewer (map_data/viewer/) is a single-page web application with three switchable modes.

Back-end (app.py, routes.py, helpers.py, cache.py):

  • Serves GeoJSON representations of the MapData contents and handles annotation CRUD
  • /api/fetch_area runs Overpass queries in a background thread and returns a task ID immediately; the client polls /api/fetch_area/<task_id> for completion, preventing UI hangs on long fetches
  • /export writes a human-readable JSON snapshot of the annotated map
  • cache.py manages the OSM response cache sidecar

Front-end modes (Leaflet + Bootstrap, static/js/):

Mode Entry point Purpose
Viewer ui_handlers.js, layers.js Inspect, edit, annotate, and search OSM features
Planner planner_mode.js Place waypoints, trigger replanning, download GPX (track or waypoint format)
Tracker tracker_mode.js Live robot position overlaid on the map via WebSocket / ROS2

Sidecar annotation files. All viewer edits are persisted alongside the .mapdata file as <stem>.annotations.json. This design keeps the map data immutable while allowing iterative annotation without re-running the Overpass query. The viewer merges the sidecar at load time to produce the rendered view.

ROS2 nodes

Node File Purpose
create_mapdata map_data/create_mapdata.py CLI node: reads a GPX/YAML file, runs MapData.run_all(), writes the .mapdata file
osm_cloud map_data/osm_cloud.py ROS2 node: publishes the parsed OSM features as sensor_msgs/PointCloud2 messages for use in Nav2 or custom navigation stacks
TrackerNode map_data/viewer/ros_node.py ROS2 node embedded in the viewer: subscribes to robot pose and streams it to the Leaflet front-end via WebSocket for live position display

Data flow

A typical session follows this sequence:

  1. Create .mapdata — run create_mapdata with a GPX waypoint file. The node queries Overpass, parses the OSM data, and writes <name>.mapdata to disk. Or download and parse data inside the viewer.
  2. Visualize data — run the viewer and load parsed data in the browser. If the data were parsed through the viewer they will be shown automatically.
  3. Annotate — use the viewer drawing tools to mark obstacles, draw alternative path segments, delete or hide erroneous ways, adjust node positions, and override OSM tags. Changes are saved automatically to <name>.annotations.json.
  4. Export — click the Export button to produce <name>.exported.mapdata, a human-readable JSON snapshot of the annotated map suitable for downstream processing.
  5. Plan path — instantiate GraphPlanner or ReplanPath with the loaded MapData object and call plan() with the desired waypoints.

Project structure

map_data/                          # repository root
├── map_data/                      # Python package
│   ├── __init__.py                # package version
│   ├── map_data.py                # MapData — central data class
│   ├── info.py                    # map_data_info CLI tool
│   ├── create_mapdata.py          # ROS2 node / CLI: parse & save
│   ├── osm_cloud.py               # ROS2 node: PointCloud2 publisher
│   ├── pathsolver/
│   │   ├── astar.py               # graph A* primitives
│   │   ├── graph_planner.py       # GraphPlanner (on-network A*)
│   │   ├── grid_constructor.py    # PathGrid — cost raster builder
│   │   ├── grid_astar.py          # grid A* search
│   │   ├── replan.py              # ReplanPath orchestrator
│   │   ├── rrt_star.py            # RRTStar (Informed + adaptive radius)
│   │   ├── smoothing.py           # gradient-descent path smoother
│   │   └── visualizer.py          # Matplotlib debug visualizer
│   ├── utils/
│   │   ├── config.py              # YAML loading + setup_logging()
│   │   ├── gpx.py                 # GPX / YAML waypoint parsing
│   │   ├── overpass.py            # Overpass API client
│   │   ├── parsing.py             # OSM feature classification & buffering
│   │   ├── serialization.py       # .mapdata JSON read / write
│   │   ├── way.py                 # Way dataclass
│   │   └── points_to_graph_points.py
│   └── viewer/
│       ├── app.py                 # Flask app factory + SocketIO
│       ├── routes.py              # REST API endpoints
│       ├── helpers.py             # GeoJSON conversion helpers
│       ├── cache.py               # OSM response cache
│       ├── ros_node.py            # TrackerNode (optional ROS2)
│       ├── templates/
│       │   └── index.html         # single-page app shell
│       └── static/
│           ├── css/style.css
│           └── js/
│               ├── api.js         # fetch wrappers for all REST calls
│               ├── draw_handlers.js  # Leaflet.Draw event handlers
│               ├── layers.js      # layer management & feature search
│               ├── map_setup.js   # Leaflet map init & mode switching
│               ├── planner_mode.js   # Planner mode (waypoints, replan, GPX)
│               ├── state.js       # shared mutable state
│               ├── tracker_mode.js   # Tracker mode (live robot position)
│               ├── ui_handlers.js    # sidebar & edit panel handlers
│               └── utils.js       # shared helpers
├── config/                        # YAML configuration
│   ├── planner_defaults.yaml      # planner cost & buffer defaults
│   ├── osm_grid.yaml              # osm_cloud grid parameters
│   └── helhest.yaml               # robot-specific parameter overrides
├── data/                          # default location for .mapdata files
├── docs/                          # MkDocs documentation source
├── launch/                        # ROS2 launch files
├── parameters/                    # ROS2 parameter files
├── resource/                      # ROS2 ament resource marker
├── tests/                         # pytest test suite
│   ├── test_astar.py
│   ├── test_core.py
│   ├── test_errors.py
│   ├── test_fill_grid.py
│   ├── test_graph_planner.py
│   ├── test_integration.py
│   ├── test_overpass.py
│   ├── test_parsing.py
│   ├── test_rrt.py
│   ├── test_smoothing.py
│   ├── test_viewer_helpers.py
│   └── test_viewer_routes.py
├── pyproject.toml                 # build metadata, ruff config
├── package.xml                    # ROS2 package manifest
├── setup.py / setup.cfg           # ROS2 ament_python build shims
└── mkdocs.yml                     # documentation site config