Re: [PLUG] Linux networking really slow on copy

[Andy Wojnarek <andy.wojnarek@theatsgroup.com>]

I would:

1. Take a tcpdump on both sides
2. Loop over netstat/ss and pay attention to the send/recv queues

Check the tcpdump for any tcp window scaling, retransmits, packet fragmentation etc. 

Thanks,
Andrew Wojnarek |  Sr. Systems Engineer    | ATS Group, LLC
mobile 717.856.6901 | andy.wojnarek@TheATSGroup.com


On 11/14/16, 12:31 PM, "plug on behalf of JP Vossen" <plug-bounces@lists.phillylinux.org on behalf of jp@jpsdomain.org> wrote:

    I'm trying to copy about 3TB from a physical machine to a VM running on 
    Debian8 + VMware Workstation 10, and it's going badly.  I'm copying 
    2,000+ MythTV files; half are the tiny PNG thumbnail pics and half are 
    *.mpg ranging up to a single 21G file.
    
    I've had a lot of load and general slowness and since that same host and 
    VMware runs my main "services" server that's not good.  I tried to cut 
    out the overhead of SSH and Rsync as follows:
    	source:	tar cv . | nc -q 1 192.168.80.81 5123
    	target: nc -q 1 -l -p 5123 | pv -pterb -s 3T | tar xv
    
    The data is now moving over a cross-over cable on dedicated NICs on IPAs 
    that are not on my LAN, so I know the traffic is flowing on the x-over 
    cable.  The physical machine has eth1 connected to eth1 on the Debian 
    host.  That NIC has no IP on the host, but is bridged to vmnet2 in 
    VMware and the target VM's eth1 is connected to that.
    
    Target VM: # mii-tool
    	eth0: negotiated 1000baseT-FD flow-control, link ok
    	eth1: negotiated 1000baseT-FD flow-control, link o
    
    The physical machine: # mii-tool
    	eth0: negotiated 100baseTx-FD, link ok
    	eth1: negotiated 1000baseT-FD flow-control, link ok	
    
    eth0 is odd because it's connected to the same core GigE switch 
    everything else is and all the other ports (e.g. the Debian host) are 
    1000baseT-FD as expected.  The 100baseT is also part of the reason to 
    try the dedicated x-over cable, which is 1000baseT-FD as expected.
    
    In ALL cases I've tried so far, transfer speed starts out about as 
    expected but then rapidly tanks to nothing and/or then flails around.
    
    Examples:
    	Originally with rsync:	11MB/s down to ~192KB/s
    	Originally with netcat:	same
    	Cross-over and netcat:	100MB/s to ~192KB/s but now all over
    
    All over is ranging as I watch it from 74MB/s to 850KB/s but often 
    hovering in the 15MB/s or 25MB/s range.
    
    The source is an old PE2950 running Mythbuntu 14.04, the target is a 
    slightly less old R710 running Debian8 + VMWst 10 and the target VM is 
    booted into mythbuntu-14.04.1-desktop-amd64.iso with the appropriate VM 
    file systems mounted into /mnt/.
    
    WTH?  Note I have not tuned the stock network stack on any machine 
    involved.  So this looks like maybe I have a buffering or flow-control 
    problem, but I don't know where or how to fix it.
    
    Clues?
    
    Thanks,
    JP
    --  -------------------------------------------------------------------
    JP Vossen, CISSP | http://www.jpsdomain.org/ | http://bashcookbook.com/
    ___________________________________________________________________________
    Philadelphia Linux Users Group         --        http://www.phillylinux.org
    Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
    General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug
    


___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug