osm2pgrouting (#24) - Import Error (#261) - Message List

Import Error

I'm attempting to import the Minnesota - United States OSM dataset. I run into the following error during the import process:

root@db:~/osm2pgrouting# ./osm2pgrouting -file ~bobcat/minnesota.osm -conf mapconfig.xml  -dbname osm -user postgres -clean
host=127.0.0.1 user=postgres dbname=osm port=5432
connection success
Trying to load config file mapconfig.xml
Trying to parse config
SE for <configuration>
SE for <type>
...
...
class name = roundabout
class id = 401
class id = 401 name = roundabout added to type name=junction
Trying to load data
Trying to parse data
not well-formed (invalid token) at line 20234194
*** glibc detected *** ./osm2pgrouting: double free or corruption (!prev): 0x0000000000bb3920 ***
======= Backtrace: =========
/lib/libc.so.6[0x7ffa68c32cb8]
/lib/libc.so.6(cfree+0x76)[0x7ffa68c35276]
/lib/libc.so.6(fclose+0x151)[0x7ffa68c22d21]
./osm2pgrouting[0x40f693]
./osm2pgrouting[0x403e42]
/lib/libc.so.6(__libc_start_main+0xe6)[0x7ffa68bd95a6]
./osm2pgrouting[0x4032d9]

It seems like this might be because of in improperly formatted data file, but even so it seems like there could be a more graceful failure and error message.

Carl

  • Message #937

    It seems like this might be because of in improperly formatted data file, but even so it seems like there could be a more graceful failure and error message.

    You're right. We wished Google would have assigned more slots for this year's GSoC to improve the importing tool. It was contributed last year to pgRouting and we would be glad someone could work on this.

    I never saw this error yet. Does it work when you try it with other data than Minnesota? Maybe another country's data?

    • Message #938

      That is ironic - I am trying to import the Minnesota OSM data to aid in my GSoC project.

      I tried the Rhode Island data which imported fine, so did Wisconsin. I tried the Minnesota - Highways file from Cloudmade instead of the whole MN file and it seemed to get further but then run out of memory, going to up the memory for my VM and try again.

      • Message #939

        Hi Carl,

        Congratulations! I just saw that your proposal was accepted through OSM. That's cool! I was already thinking whether it is you or another Carl. Sorry for not having send you some feedback yet. Just too busy recently.

        Well, it seems like you run into this issue: #140
        And unfortunately there is not much we can help at the moment to solve this.

        The problem is that all data is loaded to memory to process, which is not good. Would be nice to write to a temporary table instead before running out of memory.

        Ugly, but what you could do is (if you use Linux) to increase your SWAP partition to as much as you need, so you can load all the data and process it. I once made a 8GB Swap partition and filled it completely when trying to import a complete country. But if you have a big HDD you could just make it bigger.

        • Message #941

          Yes, that is the issue I am having now. My VM died yesterday, I ended up having to reboot the whole system, ended up with a corrupt (virtual) hard drive. Once I get another VM going with pgRouting, I'll try both upping the memory to 2GB or so and giving it plenty if swap space if that is still needed.

          Thanks for the help. I'm hoping to do some cool things with pgRouting and driving data. I might have some contributions to make back to pgRouting too - as there are some functions I will need that could probably be useful for pgRouting in general.

          • Message #947

            Hi Carl,

            Personally I'm very interested in your project, because I'm also active OSM mapper in Yokohama. If you have something to contribute, that would be more than welcome! For an Open Source project it's all about the number of users/developers and its activity. pgRouting will never become that big, but we would like to have more different people activly involved so we might be able to fullfill the requirements to become an official OSGeo project. From our side there is no problem to extend the library with new tools, so if you have any idea here, please let me know.

            My community work is supported also by my company, but of course customer projects must have priority (quite a lot at the moment). Anyway, just ask in the forum or in the mailing list if you have a problem or want to discuss something, and at least Anton or I will try to help.