number of files

[dark22]

The game has a huge number of tiny files for its relative small total size, and they can negatively impact disk (de)fragmentation.
Can the game files be packed in larger files for the future ea builds or the release version?

Thank you.

[sven]

The game has a huge number of tiny files for its relative small total size, and they can negatively impact disk (de)fragmentation.
Can the game files be packed in larger files for the future ea builds or the release version?

It could be -- and bundling everything into a few big data files is something I've considered in the past.

That said, implementing bundling isn't trivial (the art is relatively easily bundled, the Lua script files, less so), and it would complicate development a bit. I'm also not certain that the fragmentation and disc size access issues are really likely to be significant on modern hardware and filesystems.

Can you point me to any references that measure the performance impacts of file fragmentation on, say Windows 7+? It's probably best not to start thinking seriously about these kinds of optimizations without evidence that there's a good chance of effecting measurable improvements

[dark22]

Here is a place discussing this: http://superuser.com/questions/808279/why-does-copying-individual-files-take-so-much-longer-than-one-large-file

Moreover, another take on file size and performance:http://gamedev.stackexchange.com/questions/20247/why-are-big-files-better-than-small-files-for-consoles

I am not an expert in the field by any means, so I do not know if their arguments are correct or not.

[SirDamnALot]

Well, file seek times are negligible if you have a SSD drive. It's a "problem" for old harddisks. And only for the first load, after that it should be cached anyway.

A downside to big data blob files is patching. If you do not go through the hassle for binary delta patching, you have to replace the whole blob.
Regardless how small the change is. (Hello Mr. UbiEAsoft, why do I have to download several gigabyte patches every other day ? )

[sven]

Here is a place discussing this: http://superuser.com/questions/808279/why-does-copying-individual-files-take-so-much-longer-than-one-large-file

Moreover, another take on file size and performance:http://gamedev.stackexchange.com/questions/20247/why-are-big-files-better-than-small-files-for-consoles

I am not an expert in the field by any means, so I do not know if their arguments are correct or not.

Based on what testing I've done, my best guess is that in the case of Stars in Shadow, file bundling could decrease load times at startup significantly (as much as 10 seconds). But, I don't think bundling would have much effect on loading times during the game itself. Much of the latency you see in in-game is related to loading very large high-res art files into video memory, and so the limiting factors are disk-speed and video ram bus, not file seeking. (We do load a large number of small Lua files, but that's a one-time cost that happens during program startup. So, reducing those costs via bundling would just make us launch a bit faster.) On some disks, there's probably a chance that fragmentation could cause increase disk-read times, but, on the sorts of lower-spec machines where this is likely to be noticible, I think the more pressing optimization is to provide the option of reducing the resolution of the source textures. However we pack it, cycling 1gb worth of high-res art though an older intel laptop is going to be slow. But if we prune the resolutions to get that number down to 250mb, that in itself will probably dramatically reduce most of the disk-access latency, not to mention making the whole program decidedly less crash prone