Skip to content

Architecture Overview

This page describes the internal structure of the map_data package. The system is organised into four main layers: data ingestion, representation, planning, and visualisation.

Component diagram

MapData

MapData (defined in map_data/map_data.py) is the central data class. It accepts either a path to a .gpx (or .yaml) file or a pre-converted UTM coordinate array, queries the Overpass API for all OSM features inside the derived bounding box, and parses the responses into categorised Way lists.

Overpass queries. Three concurrent HTTP requests are fired in a thread pool, each covering the full bounding box with no server-side tag filtering:

  • ways query — every way in the bounding box (way(bbox); >;), plus a down-recursion that brings in every way's constituent nodes.
  • relations query — every relation that references at least one of those ways (way(bbox); <;). parse_osm_rels() uses these to stamp relation tags onto member ways and mark them inner/outer — this is what makes untagged multipolygon members (lakes, forests, courtyard buildings, whose classifying tag lives on the relation) still show up as obstacles.
  • nodes query — every standalone node in the bounding box (node(bbox)).

An earlier revision filtered these queries server-side by tag family (Overpass ["key"] existence filters) to shrink the download. That was reverted: unioning a dozen-plus separate tag-indexed clauses is markedly more expensive for the Overpass query planner than a single bbox scan, and it made queries slow enough to trip the public instances' rate limiting and timeouts — sometimes returning incomplete results instead of failing loudly. A plain bbox query, while larger, is filtered client-side in run_parse() and is far more reliable in practice.

The bounding box is the convex hull of the input waypoints expanded by grid_margin metres on all sides (default: 150 m); grid_margin is a per-call override accepted by MapData.__init__, parse_osm_nodes, and separate_ways.

Core attributes after parsing:

Attribute Type Description
roads_list list[Way] Vehicle-intended highway ways, buffered to 7 m wide polygons
footways_list list[Way] Pedestrian footways, buffered to 3 m wide polygons
barriers_list list[Way] Physical barriers, buildings, water, and obstacle nodes
crossroads_list list[Way] Footway intersection points (buffered 1.5 m circles)
nodes_cache dict[int, dict] Mapping of OSM node ID to {lat, lon, tags}
zone_number int UTM zone number inferred from the input waypoints
zone_letter str UTM zone letter inferred from the input waypoints
waypoints np.ndarray (N, 2) UTM easting/northing array of the input waypoints
min_x/max_x/min_y/max_y float UTM bounding box including margins

OSM caching. After a successful Overpass query, the raw responses are saved to a .osm_cache.json sidecar file next to the source .gpx/.yaml file. On the next load, if the file exists and the stored bounding box matches the current one (within 1 × 10⁻⁶ °, ≈ 11 cm), the cache is used and the network round-trip is skipped. The cache is invalidated automatically whenever the bounding box changes.

Serialisation. MapData.save() writes a JSON file (.mapdata extension) using json.dump. MapData.load(path) reads it back. Support for the legacy pickle format has been removed for security reasons; legacy files must be re-parsed from the source GPX/YAML.

Way objects

Every OSM feature is stored as a Way instance (map_data/utils/way.py). A Way holds:

  • id — OSM way/node ID (positive integer) or a synthetic negative integer for merged multipolygon segments or viewer-drawn annotations
  • is_area — True if the geometry is a closed polygon
  • nodes — ordered list of OSM node IDs
  • tags — dict of OSM tag key-value pairs
  • line — Shapely geometry (Polygon for closed/buffered features, LineString for open features)
  • in_out — "outer" or "inner" for multipolygon relation members

All geometry is stored in UTM coordinates (same zone as the input waypoints). Ways are buffered during parsing: roads receive a 7 m half-width buffer and footways a 3 m half-width buffer, converting LineString geometries to Polygon.

Pathsolvers

The package provides two independent path-planning back-ends.

GraphPlanner

map_data/pathsolver/graph_planner.py

Builds an undirected weighted graph directly from the OSM way network (roads_list and/or footways_list). Edge weights are Euclidean distances in metres. Planning uses A* with a straight-line distance heuristic.

Waypoints passed to GraphPlanner.plan() are first snapped to the nearest graph edge using an STRtree spatial index. Viewer-drawn annotation paths (negative-ID ways) are spliced into the graph by projecting their endpoints onto the nearest existing OSM edge and inserting synthetic junction nodes at the projection points.

This planner is well-suited for route planning on well-mapped pedestrian or road networks where staying on designated paths is required.

ReplanPath

map_data/pathsolver/replan.py

Orchestrates cost-grid planning. Grid construction, smoothing, and visualization are handled by dedicated sub-modules:

Module Class / function Responsibility
pathsolver/grid_constructor.py PathGrid Builds the cost raster at cell_size metre resolution; assigns per-cell costs from highway type, surface, and barrier geometry
pathsolver/grid_astar.py grid_astar Fast, optimal A* search on the discrete grid
pathsolver/rrt_star.py RRTStar Sampling-based planner; supports Informed RRT* (ellipsoidal sampling after first path found) and adaptive neighbor radius (γ·√(log n/n)) for asymptotic optimality
pathsolver/smoothing.py smooth_path Gradient-descent path smoothing with optional collision checking
pathsolver/visualizer.py visualize_replan Matplotlib debug visualization of the grid, obstacles, and planned path

Each PathGrid cell cost is determined by:

  • the OSM highway type and surface material of the nearest way within max_path_dist metres (configured via highway_costs and surface_costs in planner_defaults.yaml)
  • a fixed default_off_path_cost for cells not covered by any way
  • barrier polygons inflated by inflate_obstacles metres, set to cost 1.0 (impassable)

Key parameters now exposed in planner_defaults.yaml: grid_cost_weight, obstacle_radius, and buffer_widths (per road type).

Viewer

The viewer (map_data/viewer/) is a single-page web application with three switchable modes.

Back-end (app.py, routes.py, helpers.py, cache.py):

  • Serves GeoJSON representations of the MapData contents and handles annotation CRUD
  • /api/fetch_area runs Overpass queries in a background thread and returns a task ID immediately; the client polls /api/fetch_area/<task_id> for completion, preventing UI hangs on long fetches
  • /export writes a human-readable JSON snapshot of the annotated map; /export/geojson writes the same merged data as a GeoJSON FeatureCollection
  • cache.py manages the OSM response cache sidecar

Front-end modes (Leaflet + Bootstrap, static/js/):

Mode Entry point Purpose
Viewer ui_handlers.js, layers.js Inspect, edit, annotate, and search OSM features
Planner planner_mode.js Place waypoints, trigger replanning, download GPX (track or waypoint format)
Tracker tracker_mode.js Live robot position overlaid on the map via WebSocket / ROS2

Sidecar annotation files. All viewer edits are persisted alongside the .mapdata file as <stem>.annotations.json. This design keeps the map data immutable while allowing iterative annotation without re-running the Overpass query. The viewer merges the sidecar at load time to produce the rendered view.

ROS2 nodes

Node File Purpose
create_mapdata map_data/create_mapdata.py CLI node: reads a GPX/YAML file, runs MapData.run_all(), writes the .mapdata file
osm_cloud map_data/osm_cloud.py ROS2 node: publishes the parsed OSM features as sensor_msgs/PointCloud2 messages for use in Nav2 or custom navigation stacks
TrackerNode map_data/viewer/ros_node.py ROS2 node embedded in the viewer: subscribes to robot pose and streams it to the Leaflet front-end via WebSocket for live position display

Data flow

A typical session follows this sequence:

  1. Create .mapdata — run create_mapdata with a GPX waypoint file. The node queries Overpass, parses the OSM data, and writes <name>.mapdata to disk. Or download and parse data inside the viewer.
  2. Visualize data — run the viewer and load parsed data in the browser. If the data were parsed through the viewer they will be shown automatically.
  3. Annotate — use the viewer drawing tools to mark obstacles, draw alternative path segments, delete or hide erroneous ways, adjust node positions, and override OSM tags. Changes are saved automatically to <name>.annotations.json.
  4. Export — click the Export button to produce <name>.exported.mapdata, a human-readable JSON snapshot of the annotated map suitable for downstream processing. Click the GeoJSON button to produce <name>.geojson, the same data for use in QGIS/geojson.io.
  5. Plan path — instantiate GraphPlanner or ReplanPath with the loaded MapData object and call plan() with the desired waypoints.

Project structure

map_data/                          # repository root
├── map_data/                      # Python package
│   ├── __init__.py                # package version
│   ├── map_data.py                # MapData — central data class
│   ├── info.py                    # map_data_info CLI tool
│   ├── create_mapdata.py          # ROS2 node / CLI: parse & save
│   ├── osm_cloud.py               # ROS2 node: PointCloud2 publisher
│   ├── pathsolver/
│   │   ├── astar.py               # graph A* primitives
│   │   ├── graph_planner.py       # GraphPlanner (on-network A*)
│   │   ├── grid_constructor.py    # PathGrid — cost raster builder
│   │   ├── grid_astar.py          # grid A* search
│   │   ├── replan.py              # ReplanPath orchestrator
│   │   ├── rrt_star.py            # RRTStar (Informed + adaptive radius)
│   │   ├── smoothing.py           # gradient-descent path smoother
│   │   └── visualizer.py          # Matplotlib debug visualizer
│   ├── utils/
│   │   ├── config.py              # YAML loading + setup_logging()
│   │   ├── gpx.py                 # GPX / YAML waypoint parsing
│   │   ├── overpass.py            # Overpass API client
│   │   ├── parsing.py             # OSM feature classification & buffering
│   │   ├── serialization.py       # .mapdata JSON read / write
│   │   ├── way.py                 # Way dataclass
│   │   └── points_to_graph_points.py
│   └── viewer/
│       ├── app.py                 # Flask app factory + SocketIO
│       ├── routes.py              # REST API endpoints
│       ├── helpers.py             # GeoJSON conversion helpers
│       ├── cache.py               # OSM response cache
│       ├── ros_node.py            # TrackerNode (optional ROS2)
│       ├── templates/
│       │   └── index.html         # single-page app shell
│       └── static/
│           ├── css/style.css
│           └── js/
│               ├── api.js         # fetch wrappers for all REST calls
│               ├── draw_handlers.js  # Leaflet.Draw event handlers
│               ├── layers.js      # layer management & feature search
│               ├── map_setup.js   # Leaflet map init & mode switching
│               ├── planner_mode.js   # Planner mode (waypoints, replan, GPX)
│               ├── state.js       # shared mutable state
│               ├── tracker_mode.js   # Tracker mode (live robot position)
│               ├── ui_handlers.js    # sidebar & edit panel handlers
│               └── utils.js       # shared helpers
├── config/                        # YAML configuration
│   ├── planner_defaults.yaml      # planner cost & buffer defaults
│   ├── osm_grid.yaml              # osm_cloud grid parameters
│   └── helhest.yaml               # robot-specific parameter overrides
├── data/                          # default location for .mapdata files
├── docs/                          # MkDocs documentation source
├── launch/                        # ROS2 launch files
├── parameters/                    # ROS2 parameter files
├── resource/                      # ROS2 ament resource marker
├── tests/                         # pytest test suite
│   ├── test_astar.py
│   ├── test_core.py
│   ├── test_errors.py
│   ├── test_fill_grid.py
│   ├── test_graph_planner.py
│   ├── test_integration.py
│   ├── test_overpass.py
│   ├── test_parsing.py
│   ├── test_rrt.py
│   ├── test_smoothing.py
│   ├── test_viewer_helpers.py
│   └── test_viewer_routes.py
├── pyproject.toml                 # build metadata, ruff config
├── package.xml                    # ROS2 package manifest
├── setup.py / setup.cfg           # ROS2 ament_python build shims
└── mkdocs.yml                     # documentation site config