[Qgis-user] Failed to import OSM data: XML error: Storing node 1, 920, 141, 933 failed.

[Jochen Albrecht]

I am trying to import a full version (700 GBytes) of OSM into SpatiaLIteand
everything is going smoorthly - until it doesn't. Is there a way to run
this import in a more transparent fashion, so that I can see what the error
is? I was surprised that as the import kept chugging along, no disk space
was consumed on any of my drives (temp, destination).
I know that some folks out there have been working with the full (planet)
XML and am now seeking advice as to how they did it (unlikely that it
worked for them at their first trial).
I am working with QGIS 2.12.1 64-bit on a choice of Windows 8 or Ubuntu.
Eventually, I would like to create a TIN of the road network nodes, so I
need to be able to open the data set sequentially in a database rather than
loading it all into memory.
Cheers,
Jochen

[Andre Joost]

Post by Jochen Albrecht
I am trying to import a full version (700 GBytes) of OSM into SpatiaLIteand
everything is going smoorthly - until it doesn't. Is there a way to run
this import in a more transparent fashion, so that I can see what the error
is? I was surprised that as the import kept chugging along, no disk space
was consumed on any of my drives (temp, destination).
I know that some folks out there have been working with the full (planet)
XML and am now seeking advice as to how they did it (unlikely that it
worked for them at their first trial).

Your attempt is designed to fail with that amount of data. Consider
spatiaLITE to be a lightweight database. Professional work on OSM data
is made on a Postgis populated by osm2pgsql, and the developers are sure
that this does not work for the full planet on Windows due to OS
restrictions:

https://github.com/openstreetmap/osm2pgsql/issues/472

Post by Jochen Albrecht
I am working with QGIS 2.12.1 64-bit on a choice of Windows 8 or Ubuntu.
Eventually, I would like to create a TIN of the road network nodes, so I
need to be able to open the data set sequentially in a database rather than
loading it all into memory.

If you only need the road network, you better filter that out before
using osmosis. This will spare a lot of database size and performing
time. Note that buildings and landuses occupy a lot of the database.

If you still want to work on spatialite, start with a small extract from
geofabrik, and test further with larger ones until the database collapses.

HTH,
André Joost

[Andre Joost]

Thank you Andre.

Please answer to the mailing list, so that others can benefit from the
discussion as well.

PostGIS would have been my next move.
I wanted to use the database to filter out the road network - do you
suggest that I do this with a (Python) script on the XML file?

You have to store all the nodes temporarily to get those of the roads in
a second step, because Openstreetmap data has only geographic
coordinates for the nodes.

Osmosis is designed to do that on the XML file. I have not tested it
myself on the whole planet.

HTH,
André Joost