> Upload size limitations

Hi :),
I am trying to upload a submission and its also says "uploaded image" but i can not find it to select it for submission. Are there any file size limitations? Or do i need to remove old images to free up space? At the moment am trying to upload it again.
Best regards,
Finn

Posted by: FBehrendt @ Oct. 19, 2022, 4:12 p.m.

Hi,
are your submissions about the same size as before? If so, what most likely is happening is that there is something in your Dockerfile that prevents it from building a Docker Image. This was the case for another user. Even though the Docker build worked locally, it did not on the platform. Are your requirements different than before; any new libraries or something like that?

Ideally, the platform would provide a useful error message instead of a 'Successfully uploaded Image' but unfortunately that's currently not the case.

Posted by: kdrukker @ Oct. 19, 2022, 5:26 p.m.

The file is larger than earlier submissions and I had to update one library. An error message would be great but if there is none I will just test around with that library and the file size if needed. So as I understand it correctly there is no direct file size limitation, right?
Best regards
Finn

Posted by: FBehrendt @ Oct. 19, 2022, 5:36 p.m.

Hi Finn,
which library is new or had to be updated?

Posted by: kdrukker @ Oct. 19, 2022, 6:04 p.m.

Timm Library to the newest version

Posted by: FBehrendt @ Oct. 19, 2022, 6:10 p.m.

Was your image zip this: midrc_docker_image_extzoo_v9.zip? I'm trying to start nailing this down.

If the archive you upload isn't fully deleted between uploads on the system, then potentially the file is sticking around and as you upload the same archive name (if you are using the same name), then data bits are being appended to this file rendering it a corrupt zip. Not sure this is happening, but I cleared the zip folder so you can retry and hopefully this isn't an issue still.

I also just added code to store and timestamp uploads for better record-keeping. If you can upload it one more time and paste here the name you uploaded (filename.zip), I can go find it and open it up to see what is happening. Can you do that for now?

Posted by: challenge-organizer @ Oct. 19, 2022, 6:51 p.m.

Thank you for the support!
I just started the upload for the image called midrc_docker_image_extzoo_v9_debug.zip
This will take some time (about 2 hours). Let me know if you can see any issue.
Best regards
Finn

Posted by: FBehrendt @ Oct. 19, 2022, 7:39 p.m.

I uploaded a new submission and the status was "pushing image" before the system said "uploaded image". I will now try to build a smaller zipfile and see if this helps.

Posted by: FBehrendt @ Oct. 20, 2022, 9:21 a.m.

Update: The file (midrc_docker_image_extzoo_v9_v2.zip) did not upload and the process hang at "docker login complete" (>6hrs). I then stopped the upload and tried a very small dummy image. This image hangs at the stage "uploading to host VM". Are you debugging the system at the moment or is there any other reason for that? I need to stop for today and will try it again tomorrow. If you have any suggestions where the problem could be i would be very happy.

Posted by: FBehrendt @ Oct. 20, 2022, 4:25 p.m.

Hi Finn,
sorry you are having trouble uploading. Others have been uploading/running successfully today, so I am unsure what is going on. We will check what is going on. Stay tuned.

Posted by: kdrukker @ Oct. 20, 2022, 5:51 p.m.

So from what I can tell your zip archive is 40GB. Is that right? There are 2 of them and they have exhausted the machine's memory and I have to resize the disk and reboot. So I need a tiny bit of patience there. 40GB seems like an awful lot. I'm happy that the uploads finished but the builds are failing (I think) as there is no more room on the machine for the built docker image that would ensue. Are models typically that large? Do you think it's the docker image itself?

The machine has 250GB of space on disk. Let me see if I can redirect to our cloud storage. I guess if your model needs to be that big then it has to be, but I would ask that you trim as much as possible in the case of uploaded stale or old models or things that aren't needed like test images if that is the case.

Posted by: FBehrendt @ Oct. 20, 2022, 8:18 p.m.

Update,

I changed the destination or archives to cloud storage. Now there is enough room I believe as we aren't storing on the VM itself. Please continue to try to upload and let's see how it goes.

Posted by: challenge-organizer @ Oct. 20, 2022, 10:51 p.m.

Thank you for working on this!
Indeed my modles checkpoints need a lot of space and no other files are responsible for that. My last working submission was about 20GB and the largest i tried was 40GB. I now tried to upload a zipfile of about 37GB. It again failed (haning at "docker login complete").
So i stripped down my solution by removing parts of it. In the end i have a zipfile of about 27gb. Furthermore, i deleted most of my uploaded images. I think this should work now regarding the file size of my image to upload. However, the system will not let me upload the solution at the moment. I assume that it might again need a reboot.
If you could give me a push when the system is ready again for submission id want to try uploading the reduced images.
Apart from the file size, the other change form the latest working submission is one library that i had to update: I updated timm==0.5.4 to timm=0.6.11. I do not think that this should break the build process of the docker but if you think that this could be a problem i can try a solution which does not require the new library.
Again thank you for your effort and sorry for the inconveniences. Hopefully we will find a solution until the deadline.

Posted by: FBehrendt @ Oct. 21, 2022, 1:36 p.m.
Post in this thread