PgOSM Flex Update Mode
Running in experimental Update mode enables using osm2pgsql's --append
option.
Note: This is not the
--append
option that existed in PgOSM Flex 0.6.3 and prior.
Testing steps
Important -- Needs higher max connections!
docker stop pgosm && docker build -t rustprooflabs/pgosm-flex .
docker run --name pgosm -d --rm \
-v ~/pgosm-data:/app/output \
-v /etc/localtime:/etc/localtime:ro \
-e POSTGRES_PASSWORD=$POSTGRES_PASSWORD \
-p 5433:5432 -d rustprooflabs/pgosm-flex \
-c max_connections=300
Run fresh import w/ D.C. using --update create
. This ensures osm2pgsql
uses --slim
w/out --drop
. Tested from commit 672d9fd
.
time docker exec -it \
pgosm python3 docker/pgosm_flex.py \
--ram=8 \
--region=north-america/us \
--subregion=district-of-columbia \
--skip-nested --skip-dump \
--update create
...
2022-12-27 09:02:37,654:INFO:pgosm-flex:helpers:PgOSM-Flex version: 0.6.3 672d9fd
...
real 0m43.904s
user 0m0.020s
sys 0m0.012s
Run with a second sub-region using --update append
.
time docker exec -it \
pgosm python3 docker/pgosm_flex.py \
--ram=8 \
--region=north-america/us \
--subregion=new-hampshire \
--skip-nested --skip-dump \
--update append
...
2022-12-27 10:14:26,792:INFO:pgosm-flex:helpers:2022-12-27 10:14:26 osm2pgsql took 1420s (23m 40s) overall.
2022-12-27 10:14:26,832:INFO:pgosm-flex:pgosm_flex:osm2pgsql completed
2022-12-27 10:14:26,832:INFO:pgosm-flex:pgosm_flex:Running with --update append: Skipping post-processing SQL
2022-12-27 10:14:26,832:INFO:pgosm-flex:pgosm_flex:Skipping pg_dump
2022-12-27 10:14:26,832:INFO:pgosm-flex:pgosm_flex:PgOSM Flex complete!
real 23m47.564s
user 0m0.083s
sys 0m0.025s
It seems to work, new output at the end: Skipping post-processing SQL
.
Verified that both New Hampshire and D.C. regions were loaded in osm.place_polygon
.
Smaller test
Put the following into ~/pgosm-data/extracts/colorado-extract.json
:
{
"directory": "/home/ryanlambert/pgosm-data/",
"extracts": [
{
"output": "colorado-boulder-latest.osm.pbf",
"description": "Area extracted around Boulder, Colorado",
"bbox": {
"left": -105.30,
"right": -105.20,
"top": 40.07,
"bottom": 39.98
}
},
{
"output": "colorado-longmont-latest.osm.pbf",
"description": "Area extracted around Longmont, Colorado",
"bbox": {
"left": -105.15,
"right": -105.05,
"top": 40.21,
"bottom": 40.12
}
}
]
}
Create Boulder and Longmont extracts.
osmium extract -c extracts/colorado-extracts.json colorado-2022-12-27.osm.pbf
ryanlambert@tag201:~/pgosm-data$ ls -alh | grep boulder
-rw-rw-r-- 1 ryanlambert ryanlambert 2.4M Dec 27 14:31 colorado-boulder-latest.osm.pbf
ryanlambert@tag201:~/pgosm-data$ ls -alh | grep longmont
-rw-rw-r-- 1 ryanlambert ryanlambert 988K Dec 27 14:31 colorado-longmont-latest.osm.pbf
Takes 11 seconds.
time docker exec -it \
pgosm python3 docker/pgosm_flex.py \
--ram=8 \
--region=north-america/us \
--subregion=colorado-longmont --input-file colorado-longmont-latest.osm.pbf \
--skip-dump --update create
Takes 2 minutes.
time docker exec -it \
pgosm python3 docker/pgosm_flex.py \
--ram=8 \
--region=north-america/us \
--subregion=colorado-boulder --input-file colorado-boulder-latest.osm.pbf \
--skip-dump --update append