SFTP: max_packet_size

Get help with using the PHP Secure Communications Library.

Moderator: Nuxius

Forum rules
The purpose of this forum is to provide support for phpseclib, a pure PHP SSH / SFTP / RSA library.

Posts by new users are held in a moderation queue and are not publicly visible until the post is approved.

SFTP: max_packet_size

Postby norbyte » Wed Sep 11, 2013 1:57 pm

Hi,

I read some posts here where users had problems with downloading (large) files. The same problem happened to me with phpseclib 0.3.0 and phpseclib 0.3.5.
Then, I found a workaround here for phpseclib 0.3.0 where the packet_size is reduced to "1 << 15". With that workaround my problems were solved.

This problem seems to be known for phpseclib 0.x and a workaround is implemented in the latest phpseclib source code to set the packet_size to "1 << 15" only if the server identifies itself as "SSH-2.0-ROSSSH".
The server I try to connect identifies itself as "SSH-2.0-OpenSSH_5.1p1 Debian-5". Of course it may be possible to add a case also for that server.

But I want to get more details, because WinSCP and Putty are connecting well and are able to download files size-independant.
Then, I took a look at the source code of Putty and found a hard-coded packet_size of 32768 bytes, equal to "1 << 15".

According to the draft of the SFTP protocol (http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13#section-4) all servers SHOULD support packages of at least 34000 bytes to allow reading of 32768 bytes (+overhead).
Section 8.2.1 of that document says, that the SFTP server MAY send less data as requested. But my dumb server denies the packet:

Code: Select all
SFTP Server: SSH-2.0-OpenSSH_5.1p1 Debian-5

SFTP Log
-----------------
<- NET_SFTP_HANDLE (0.0615s)
00000000  00:00:00:04:00:00:00:00                          ........

-> NET_SFTP_READ (0.0002s)
00000000  00:00:00:04:00:00:00:00:00:00:00:00:00:00:00:00  ................
00000010  00:10:00:00                                      ....


SSH2 Log
-----------------
-> NET_SSH2_MSG_CHANNEL_DATA (0s)
00000000  00:00:00:00:00:00:00:1d:00:00:00:19:05:00:00:00  ................
00000010  01:00:00:00:04:00:00:00:00:00:00:00:00:00:00:00  ................
00000020  00:00:10:00:00                                   .....

<- NET_SSH2_MSG_CHANNEL_REQUEST (0.065s)
00000000  00:00:00:02:00:00:00:0b:65:78:69:74:2d:73:69:67  ........exit-sig
00000010  6e:61:6c:00:00:00:00:04:50:49:50:45:00:00:00:00  nal.....PIPE....
00000020  00:00:00:00:00                                   .....

-> NET_SSH2_MSG_CHANNEL_EOF (0s)
00000000  00:00:00:00                                      ....

-> NET_SSH2_MSG_CHANNEL_CLOSE (0s)
00000000  00:00:00:00                                      ....

<- NET_SSH2_MSG_CHANNEL_EOF (0s)
00000000  00:00:00:02                                      ....

<- NET_SSH2_MSG_CHANNEL_CLOSE (0s)
00000000  00:00:00:02                                      ....


Finally my question: Why does phpseclib set a value for "max_packet_size" of "1 << 20" with a special fallback to "1 << 15" for some servers where "1 << 15" is a safe way?
Wouldn't it be better to send smaller packages (like Putty does) and to be more safe and compatible to SFTP server instead of that workaround? (which doesn't work in my case)

Thanks in advance for any advice,
norbyte
norbyte
Traveler
 
Posts: 2
Joined: Wed Sep 11, 2013 9:57 am

Re: SFTP: max_packet_size

Postby TerraFrost » Wed Sep 11, 2013 4:03 pm

I do it for speed. I don't know how it compares to PuTTY but compared to libssh2 phpseclib is faster:

http://phpseclib.sourceforge.net/ssh/compare.html

phpseclib achieves this using two techniques. One of them is a larger packet size, which means less frequent additions of SFTP headers, which means less data to send over. It's kinda the same idea as CSS sprites.

The second method is something the PuTTY docs hint at:

A.6.8 PSFTP transfers files much slower than PSCP.

We believe this is because the SFTP and SSH2 protocols are less efficient at bulk data transfer than SCP and SSH1, because every block of data transferred requires an acknowledgment from the far end. It would in theory be possible to queue several blocks of data to get round this speed problem, but as yet we haven't done the coding. If you really want this fixed, feel free to offer to help.

source: http://the.earth.li/~sgtatham/putty/0.5 ... tml#A.6.18

phpseclib sends 50 packets at a time (ie. queue's 50 packets at a time) and then reads 50 response's back. It used to send all the packets, all at once, and then wait for all the responses (so if it takes 100,000 packets to send something it'd send those 100,000 packets and then try to read 100,000 responses) but that caused problems [1].

Anyway I don't know which has more of an effect. It's possible the latter technique blows any speedups the first technique might result in out of the water. And it's possible the 50 packet number could be increased too idk.

And compatibility aside 1MB may not be the fastest possible packet size either. I haven't experimented too much with optimal packet sizes.

I'll play around with it as time permits.

Thanks!
TerraFrost
Legendary Guard
 
Posts: 12357
Joined: Wed Dec 04, 2002 6:37 am


Return to phpseclib support

Who is online

Users browsing this forum: No registered users and 2 guests

cron