NC- APFS slow copy speed [FIXED]

[bchuter]

Hi Guys

I have a Synology DS216+II (higher end spec NAS) for my NAS storage.
I use SMB network shares from my Mac's to the NAS as i have done quite
a bit of reading on forums etc, and it seems Apple themselves are trying
to phase out AFP. My time machine backups would constantly crash when using AFP.
Moving to SMB has completely fixed that issue. I was also not seeing full
network speeds with AFP anyway. But i have fully migrated to using
only SMB on all devices on my network now as this seems to be the default
for Apple (correct me if im wrong, this is just what i can see from reading many forums)

To start i have made sure that im using SMB version 3 for my tests below.
My synology was forcing version 2 for some reason and this created some
crappy behavior when renaming folders etc, whereby it would actually create
a new folder and copy the contents from the old folder to the new folder when
using SMB v2, i fixed this by not forcing SMB v2 in my Synology and now all
shares connect using SMB v3 and this works flawlessly now. None of the
behavior above experienced......yay
Here you can see my share is definitely using SMB v3 now :


Then i read further on some more posts about SMB copy speed and saw many
posts about Client Signing which slows down copies over SMB.
This was introduced in Sierra to take care of vulnerabilities etc.
But im ok disabling this as i dont use SMB over the internet. this is local to my home.
I followed this website : https://dpron.com/os-x-10-11-5-slow-smb/
and specifically applied this :


I also came across another site with some advice on speeding up browsing
on network shares (this one probably does not relate to Nimble but thought
id still add it here in case this helps any of you)
So i noticed that when i open certain folders with allot of files (1000 plus)
it would take some time before the contents are displayed.
Applying this fix, seems to have made it allot faster
Website : https://support.apple.com/en-us/HT208209
What i applied specifically :


Now for my findings on copy speed.

Nimble seems to be allot slower when copying from Network SMB shares
to my local disk.
It will always pause for some time after initiating the copy. Where you will
see the copy process appear in the notification area of Nimble
and it seems paused for quite some time before the copy actually starts.
The copy will also always start around 15/Mb/s and slowly creep up in speed.
So when copying a large 3.3gig file from SMB to local disk it will start copying
at 15MB/s or so and then slowly creep up to about 67MB/s by the end of the copy.
It steadily creeps up from the starting copy speed of 15MB/s. This seems true
for all files but its more noticeable on large files as they take longer and you can
easily see the behavior.
Here is the end of a copy in Nimble from SMB share to my local disk (SSD) :


Doing the exact same file copy in Finder from SMB share to my local disk seems
to yield better speeds.
I copied the exact same file from SMB to my local disk and then monitored the
Disk Write speed in Activity Monitor.
Here you can see the write speed at about half way through the copy.
It also starts copying immediately and there is no momentary pause like there
is in Nimble when initiating the copy. it will also start the copy off with a speed
around 65-70MB/s and quickly move up to the upper 90's MB/s
As seen in this screen capture :


Doing the same file copy in the reverse direction with Nimble from local disk to
SMB network share seems to be not affected....strangely ??
If i copy the exact same file used above from my local SSD disk to the SMB
network share it starts the copy immediately without any pause.
It also starts off immediately with a copy speed around 90-95MB/s which is
exactly what i would expect on a gig network.
As seen in this screen capture :


So Mike is this something you can look into ?
It seems Nimble it doing something extra when copying from SMB to local disk
which it is not doing when copying in the reverse direction.
I can also hear the Synology NAS disk access is allot more active when copying
from SMB to local disk vs copying in the other direction.
I can physically hear the disks are allot more active copying form the NAS vs copying
to the NAS using SMB and Nimble.
The long pause before copy begins is also strange as it only does this when copying
from the SMB share to my local disk, however there is not pause at all when copying
in the other direction and the copy speed is at full speed (or at least at what i would expect to see)

Any assistance would be great.

Thanks

[mike]

Hey bchuter!

It's possible you're overthinking the situation
Those figures various software display highly depend on a methodology used for calculation.

NC does the following:

• start time tracking.
• scan the source (especially painful when dealing with remote volumes).
• do the actual copying.
  update speeds with a simple formula: amount_of_bytes_copied / amount_of_time_elapsed.
  (in reality it's more complicated, but this rough approximation is enough for the picture. for more details you can take a look here: https://github.com/mikekazakov/nimble-c ... istics.cpp)

So what might play a role here is the scanning lag, that's why you don't notice the difference when copying FROM local drive - latencies are order of magnitudes less there.

Also, I'm not sure what Finder does in terms of parallelism. NC's operations are purely sequential.

Regards,
Mike.

[bchuter]

It's possible you're overthinking the situation

Hi Mike

Thanks for the speedy reply!

you possibly right that im overthinking it

However i just did a basic test :

Copy the same file from SMB to Local disk using both Nimble and Finder and I used a stopwatch to measure the copy time for the same file.

File is 3.3gig in size.

Copy in Nimble takes +- 44 seconds
exact same copy in Finder takes +- 32 seconds

Thats a 12 second difference for the same file ? (when copying a bunch of large files this adds up to a big difference in copy time)
Same source and destination folders were used as well.

I noticed that the "copy pause" i mentioned in NC is around 8-9 seconds before the copy even starts. (Finder starts immediately)
Here is a screen capture to see the delay :

a9QJLaPWtj.gif (188.1 KiB) Viewed 16725 times

Thanks again Mike

[mike]

Oh, I see. That's a completely different story then.
Just checked it on my machine - same stuff.
And you might observe the same behavior when copying files inside a local machine.

What's going on here is a preallocation of a space needed for a target file:

Code: Select all

    if( ShouldPreallocateSpace(preallocate_delta, dst_fs_info) ) {
        // tell system to preallocate a space for data since we dont want to trash our disk
        PreallocateSpace(preallocate_delta, destination_fd);
        
        // truncate is needed for actual preallocation
        need_dst_truncate = true;
    }
 ...
bool ShouldPreallocateSpace(int64_t _bytes_to_write, const NativeFileSystemInfo &_fs_info)
{
    const auto min_prealloc_size = 4096;
    if( _bytes_to_write <= min_prealloc_size )
        return false;

    // need to check destination fs and permit preallocation only on certain filesystems
    static const auto prealloc_on = { "hfs"s, "apfs"s };
    return count( begin(prealloc_on), end(prealloc_on), _fs_info.fs_type_name ) != 0;
} 

This mechanism was introduced for HFS+, especially for spinning drives ( yes, I had one previously )
Without telling a filesystem about a resulting file size, many different extents were being allocated for that file.
Preallocation allowed to keep the file data compact (well, as compact as current layout of filesystem blocks permits), thus potentially reducing fragmentation and all negative effects that come with it. Apple always told that HFS+ doesn't suffer from fragmentation problems, because of some kind of magic perhaps, but that's a 100% marketing bullshit.
(here's a old blogpost of mine regarding the situation with HFS+ fragmentation: https://kazakov.life/2013/08/20/file-ma ... tem-stuff/)

Anyway, why is it so slow on APFS/SSD nowadays - might be a good question.
Honestly I don't know much about APFS internals and about how it deals with fragmentation problems, so currently NC does this preallocation for APFS too, say, by default.

[bchuter]

Anyway, why is it so slow on APFS/SSD nowadays - might be a good question.

Ah ok thanks Mike.
So something worth investigating here ?

So i understand, from what you have said above, are you saying SSD's are not affected by this issue ?
If this is true, i know you cant simply remove a function as there are probably still many users that use mechanical drives.
but is it possible to disable this pre-allocation if the local disk is an SSD ?
Or add this in the 'Preferences' somewhere so users have the choice to disable or enable it. but put a big RED banner stating
the results of disabling this if still using mechanical drives ?

Or have a missed the brief completely
and this is not a wise thing to remove at all.

Just thought id ask

Many thanks Mike

[mike]

So something worth investigating here ?

Yup. You're on APFS, right?
Those number you've shown says that this filesystem is actually zeroing the preallocated space: 3300Mb / 8s = ~400Mb/s.
It looks like an additional security measure or some specifics of an implementation of this request in APFS.
I believe HFS+ behaved differently and only dealt with it's metadata, but I'm not sure.

If you check the same on HFS+ - it would be great (it doesn't matter if the drive is attached directly to the bus or via USB).

So i understand, from what you have said above, are you saying SSD's are not affected by this issue ?

It's not that simple. Additional extents placed on a filesystem on SSD will cause additional IOps too, but the difference itself won't be that severe in comparison with a spinning HDD. This all boils down to latency and seek time, not to throughput of a drive.

If this is true, i know you cant simply remove a function as there are probably still many users that use mechanical drives.
but is it possible to disable this pre-allocation if the local disk is an SSD ?

Yes, it looks like a good solution.

[bchuter]

If you check the same on HFS+ - it would be great (it doesn't matter if the drive is attached directly to the bus or via USB).

Hi Mike

Interesting tests here.

So here is a copy to an external SSD via USB formatted as 'Mac OS Extended (Journaled)' :

Copy.gif (183.72 KiB) Viewed 16702 times

It starts the copy immediately without any pause. As you can also see the speed displayed is what i would expect in the 90's MB/s.
I also timed this copy with a stop watch and it was more in line with Finders speed this time round with a copy time of 36 seconds. (Finder is +-32s)
So you are correct that this issue is something to do with APFS

This is my internal SSD on my iMac (my MBP is also APFS) :

Internat SSD.png (51.18 KiB) Viewed 16702 times

The external USB SSD :

External USB.png (47.31 KiB) Viewed 16702 times

[mike]

Fantastic, thank you a lot!
It's definitely an APFS-related problem - they basically broke the semantics of the syscall(F_PREALLOCATE) in this filesystem.
I will turn off the preallocation on APFS then, regardless of a media type (HDD/SSD).

[bchuter]

I will turn off the preallocation on APFS then, regardless of a media type (HDD/SSD).

Great stuff!!

Thanks Mike

[mike]

A bit of follow-up here.
After some further digging, it appears that the problem on APFS is not in F_PREALLOCATE itself, but in a consecutive ftruncate() syscall.
This doesn't change the whole picture, just a detail.
The Internet already has some WTF posts regarding this behavior.