Network Nightmare: gigEnn™ Manual - v2.0.1-b3

http://NetworkNightmare.net/Manuals
http://NetworkNightmare.net/License

  • Headquarters
    1U Ring Inc
    aka NetworkNightmare.net
    PO Box 70958
    Sunnyvale, CA 94086
    408-540-9229
    S a l e s @ NetworkNightmare. net
  • Asia Regional Offices
    Tokyo, Japan
    Sydney, Australia
  • EU Regional Offices
    ?? Germany ??
    ?? UK ??
  • place holder for a pic of the gigEnn box ..

    Bandwidth limiting



      Bandwidth vs RTT delay



      Bandwidth monitoring with netstat.nn

        Bandwidth : ping flood vs iperf vs socat


      • Table of Contents
        • 1.0 Introduction

          gigEnn™ == noun == The gigabit wan simulator
          gigEnn™ == verb == The process of applying network test metrics to simulate the wan at gigE speeds

          The gigEnn™ wan simulator enables you to simulate different network conditions:
          Enter the parameters for the various network conditions by selecting the letter of the configuration item or by using the arrow keys, then press Enter to select.
          Please note that your test results will be highly dependent upon your test network emulating and/or simulating your "tokyo" and "san francisco" connections


          1.1 gigEnn™ Connections

          To connect the gigEnn wan simulator, you can think of gigEnn as a PC:
          The locations of the ethernet ports ( Net2, Net1, M0 ) will vary depending on the hardware model you have purchased.
          Please refer to its connection diagram.
          It is highly suggested that you avoid connecting the wansim onto the internet until you have changed the default configurations in addition to taking other best practices' security precautions.

          You should change the default ip# and passwords by selecting "Configure System", option Z.

          The network test conditions can be defined by a local rs232 console or over the ethernet on Mngt0 port and will be applied onto Net2 and Net1 in bridge mode.

          1.2 gigEnn™ Login and Passwd


          1.3 gigEnn™ Bridge vs Router Sanity PreTesting


        • There is NO physical connectivity differences between bridge mode and router mode.
        • One difference is the kernel parameters applied for bridge mode instead of router mode.

        • The ip# for router mode is defined in /etc/rc.3nic.conf and is automatically copied into /etc/rc.conf at boot time or during dynamic mode changes between router mode and bridge mode.

        • For bridge mode, the /etc/rc.bridge.conf file is automatically and dynamically copied to /etc/rc.conf and the network is restarted.

        • Do NOT edit /etc/rc.conf file as it is automatically overwritten without any warning.

        • you should use your favorite editor to change the ip# and hostnames
        • 1.3.1 For sanity testing, you can also toggle between bridge mode and regular (3nic) router mode.
          During (3nic) router mode, you will have an ip# and routing tables assigned to Net2 and Net1 and you can ping the wansim.
        • 1.3.2 The wansim uses Net2 and Net1 ethernet ports to form a bridge.
          During bridge mode, there is no ip# assigned to Net2 nor Net1 ethernet ports. Similarly, there is no routing table entries for Net2 nor Net1.


          2.0 GigEnn™ Simulation

          • Select B == to define any bandwidth limits

          • Select R == to define round trip latency delays

          • Select L == to define packet loss

          • Select Q == to define queue sizes

          • Select M == to toggle between
            1500 and 9000 MTU
            • B - set the bandwidth limit to effect between networks 1 and 2.
              This setting reflects the lack of bandwidth on a wide area connection. To model a complex network (512 Kbps DSL to the Internet to a T1), it is best to use the speed of the lowest link. Asymmetrical rates are supported through the Advanced mode, select this from the Configure System page to setup.
                enter 45 for 45Mbit/sec bandwidth limit
                enter 155 for 155Mbit/sec bandwidth limit
                enter 622 for 622Mbit/sec bandwidth limit
                R - Enforce a delay between nets 1 and 2.
                This delay ("latency" or "Round Trip Time (RTT)") represents the effects of distance in wide area networks. You can test what your real-life latency is by using "ping" across the network, the average time it reports should be used as the Round Trip Time (or the max it reports, for worst case).
                  enter 3 for 3ms delay
                  enter 12 for 12ms delay
                  enter 123 for 123ms delay
                  L - Loss occurs in some networks
                  This setting enables you to have the Network Nightmare lose a certain fraction of packets at random. This can represent collisions in shared media physical connections, satellite networks, or many other environments. This fraction is the loss for each direction, i.e. 0.01 means 1% loss from 1 to 2 and 1% loss back from 2 to 1.
                    0.01 for ( 1%) 1 packet loss in 100
                    0.001 for ( .1%) 1 packet loss in 1,000
                    0.0001 for ( 0.01%) 1 packet loss in 10,000
                    Q - queue size affects the thruput in some networks
                      enter 0 for infinite queue size ( subject to the ethernet drivers )

                      enter 50 for 50 slot == (50 * MTU) bytes

                      enter -1 for the wansim to calculate the proper queue size for the corresponding Bandwidth * RTT specified

                    Sample settings: These may be representative of your network, they may not.

                    Use                    Bandwidth    RTT     Loss 
                                           (Mbit/sec)   (ms)    (percent)
                    Office-Home(DSL)       512          35      0.01    
                    Office-Data center(T1) 1500         45      0.003
                    Office-Office(cross US) 768         100     0.008
                    Satellite link         2000         650     0.03
                    NY-Tokyo Internet      1000         225     0.02
                    Detroit-new Deli E1    2000         285     0.01
                    SF-London Internet     1500         210     0.02
                    Data Center-Data Center disaster recovery DS3
                                           45000        50      0.00
                    


                    2.1 GigEnn™ Commandline Simulation and Scripting

                  • gigEnn can be used to manually set your network test parameters on the wansimulator:

                  • You should have your test server and client connected and running on Net2 and Net1 of the wan simulator

                  • Your resulting bandwidth thruput test data is extremely dependent upon your test server and test client

                  • You can dynamically change any of your network test parameters with gigEnn or from the wansim gui

                  • You can use the gigEnn.metrics.sh script to run predefined test sequence, please allow say 60 seconds or 60 minutes to allow for the network to stabilize and see the effects of those new test parameters

                  • You can use netstat.nn to view the resulting data in realtime

                  • You can use netstat.xplot.pl to plot the resulting data in a xplot graph

                  • 3.0 GigEnn™ Monitoring

                      T - Traffic monitor
                      ( tcpdump )

                      N - Network monitor

                      I - Interface monitor
                      ( netstat )

                      B - Bandwidth monitor

                      T - Traffic monitor (for experts). This will run the tcpdump command on the bridged interface (network 1). This will print out packets as they go by on the network. You can configure the options for tcpdump on the Configure System menu.

                      If saving tcpdumps to /tmp, note that there is no disk on the Network Nightmare, so you will likely want to limit your tcpdump to just the important traffic for your test, this is available in the configuration options. Alternately, map or mount a remote drive. When doing this, be sure that the route to the remote drive does not pass through network 1's port! (Or else you may end up recording your own monitoring output.)

                      N - Network monitor. This will print information about the network on network 1 (in any of the modes). The information printed is:

                      time CONNECTIONS open: [number of currently open] total: 
                          [total ever open connection] max-sim: [peak # open connections so far]
                      time BYTES tcp: [num tcp bytes]/[num tcp packets] udp: 
                          [num udp bytes]/[num udp packets] other: [num other bytes]/[num other packets]
                      
                      At the end of the run (hit control-c) summary information is printed. Files recording information about your traffic will be placed in /tmp. These include "top talkers," "Port usage," "SMB signing report," and some other summary files. For more information press ? in the menu with the output file selections.

                      Note that in very lightly trafficked networks, this display can fall behind the actual network traffic due to buffering within the NN. Once events are displayed their timestamps will be correct. (The timezone can be set in the Configuration, the NN will use rdate to set its time, if it has access to external networks with timeservers, otherwise the date will be somewhat arbitrary.)

                      I - Interface watch - this watches net1 and shows the number of

                         packets  errs      bytes    packets  errs      bytes colls drops
                      
                      that are input and output on net1

                      W - Watch bandwidth on net1 - this shows the instantaneous datarate, in and out of net1 along with the average (since the Watch process was started) and peak.


                      3.1 GigEnn™ Bandwidth Monitoring netstat.nn

                    • You can monitor the network with netstat.nn , sample output shown below

                    • You can plot netstat.nn's statistics output with netstat.xplot.pl

                    • far right column is the bandwidth thruput for plotting against on the y-axis

                    • left column has time stamp for plotting against on the x-axis

                    • notice the inserted bandwidth parameters ( in comments )
                      and resulting bandwidth thruput ( last column )

                    • 3.2 GigEnn™ Bandwidth Monitoring ntop

                    • You can monitor the network with your browser and ntop , sample output shown below

                    • you will need to have apache running
                    • you will need to have ntop running
                    • you will need to have firefox running:
                      firefox localhost:3000

                    • 3.3 GigEnn™ Bandwidth Monitoring trafshow

                    • You can monitor the network activity with trafshow , sample output shown below

                    • it shows 192.168.2.25 is pinging 192.168.1.14

                    • 3.4 GigEnn™ Bandwidth Plotting


                      4.0 GigEnn™ Sniffing Network Traffic


                      5.0 GigEnn™ Exit/Shutdown

                      ! - drop to shell

                    • X - Logout

                    • S - Shutdown

                    • ! - Shell access (for experts). There are always things that people want to do that the interface will not support. This option provides access to the underlying FreeBSD system. If you break the unit though this access, you will likely need to return it for support. Though try a reboot first!

                      X - Logout (useful for network access to control the simulator, not needed on serial console access)

                      S - Shutdown -- it is always a good idea to be "nice" to the system and shut it down properly.


                      6.1 GigEnn™ System Configuration

                        N - Configure Network IP#

                        P - Set root password

                        W - Set wansimuser password

                        L - Load the previous wansim settings

                        S - Save the current wansim settings

                        c - clear the temp files

                        s - Restore Factory Defaults

                        U - Upgrade to the new gigEnn software

                        N - Configure the IP# and gateway IP#

                        P - Set the root password. Often used to enable a scp to the Network Nightmare of a software upgrade, e.g. scp nn109.img.gz root@192.168.1.101:/var/tmp/ If you ssh to the Network Nightmare as root, you will login with a shell prompt.

                        W - Set the wansimuser password. Often used to enable network control of WAN simulator parameters. Login with "ssh wansimuser@NN-IP" to reach the menu system. There is a default password (1.0.9) of NN2wansim (certain models have a ! at the end of the password)

                        M - Mount/Map a network drive submenu

                        T - Configure TCPdump options. There are very many options, a common use is "host 192.168.1.11" which would record only the traffic to/from this particular host. To see all the options, enter the shell and type: tcpdumpman or select T, then CANCEL instead of selecting OK.

                        D - Set the timezone - menu driven selection of the device timezone.

                        L - Load Wansim Settings - Several files of typical network conditions have been provided (all are for the non-advanced mode). These will set bandwidth/RTT/loss rates.

                        F - Save Wansim settings - create a new template file to reload later.

                        S - Use these correct values as system defaults on reboot. All settings last only until power is removed/the Network Nightmare is rebooted. Use this option to save setting across power cycling. Be sure the settings are correct first!

                        c - Clear Temp files - erase temporary files

                        s - Restore Factory Default - overwrite existing files with factory defaults.
                        CAUTION: this will erase your current settings and previously saved settings.


                        A - Advanced Mode - With 1.0.9 the NN supports two new modes: asymmetric settings and 3way mode. In Asymmetric mode you can work with either bridged interfaces (net1/net2) or routed, but you can emulate different properties in the different directions, such as 1.5 Mbps from net1->net2, but only 386 kbps in the reverse direction. 3way mode works only for routed mode, but enables arbitrary settings in all 6 directions through the device.

                        U - upgrade to a new release of the Network Nightmare firmware. There are two ways to upgrade, copying an image down or mounting it. To mount, see the mount menu, and mount/map a drive with the new image onto the NetworkNightmare (note an image in /tmp will take priority over one in /tmp/mnt). To copy, make sure the Network Nightmare has an IP address and password, then copy to the Network Nightmare (i.e. scp nn109.img.gz root@192.168.1.101:/tmp/)

                        Alternate ways to copy are to: - Just set an IP, then drop to the shell, cd /tmp, and scp from another machine to the NN - Use a terminal program supporting zmodem, drop to the shell, cd to /tmp, execute rz (or zmrx), then send with the terminal program (use binary mode, and it's slow!)

                        After the image has been completely copied, use option U to upgrade. Note that a copied-but-not installed image will be lost in a reboot. A partial copy of an image or a partial install will likely corrupt your device and require you to return it for servicing - possibly at additional cost if you have no Support contract. For safety, an Upgrade will delete all saved settings (IP, password, emulation settings, etc)

                        X - Return to the main menu.


                        6.1 gigEnn™ Configuration: Networking

                        • I - change ip# numbers

                        • D - Set the default gateway

                        • B - The default is bridge mode

                        • R - Switch to routing mode

                        • A - Set Autonegotiate

                        • Q - Query for autonegotiation status
                        • I - change ip# numbers. This enables you to change the management IP address/netmask and gateway. This is the network connection next to power. Note that to use this connection, you must also set a password for the root or wansimuser users. The interface is defined as: IP-address/subnet-mask-length The subnet mask length is how many bits to set, i.e. CIDR notation where 255.255.255.0 is /24. You can then set the default gateway, which is needed if you wish to reach the Network Nightmare other than from the LAN. Finally, you will be prompted for the IP of the nameserver. The defaults are: 192.168.1.101/24, 192.168.1.1, and 127.0.0.1 (none).

                          D - Set the default gateway (also available under I). Note that if you employ routed mode, the gateway may be off of any interface, not just the management interface.

                          B - The default Network Nightmare setting is to Bridge ports 1 and 2. This enables "in path" insertion for simple tests. If you have switched to Routing, then this option will revert to bridging. Use this setting if using port mirroring for tcpdump/network mon.

                          R - Switch to routing mode. To use this mode, you will need to assign IP addresses to ports 1 and 2. The format is as above for (I). Be aware that other routers will also need to know how to get back to the Network Nightmare! No routing protocols are supported, only static routes. You may NOT use the same LAN subnet on any of the 3 interfaces, if you attempt to do so, you will receive an error.

                          A - Set Autonegotiate vs Fixed speed/duplex. By default the Network Nightmare will autonegotiate for speed and duplex settings on its interfaces. If you need to change these to fixed, select this item.

                          Q - Query for autonegotiation status - shows the output of ifconfig. For any interfaces set to autonegotiate, this will reveal what setting they negotiated to.

                          X - Return to the main menu.

                          Configuration: Speed and Duplex

                          Select which setting to use
                            Auto - use the auto-negotiation protocol to learn which setting
                            10H - 10 Mbps, half duplex (old hub)
                            10F - 10 Mbps, full duplex (old switch)          
                            100H: 100 Mbps, half duplex (cheap router)
                            100F: 100 Mbps, full duplex (modern switch/router)
                            1000H: 1000 Mbps, half duplex (cheap gigE router)
                            1000F: 1000 Mbps, full duplex (gigE switch/router)
                          


                          6.2 gigEnn™ Configuration: Mapping/mounting a remote drive

                          • U - Unmount

                          • N - Mount a Network File System

                          • W - Map a Windows network "share" drive

                          • S - Change the default directory

                          • L - List the mounted drive
                          • Mapping/mounting a drive allows one to more easily: save tcpdump information, copy off monitoring statistics files, or upgrade the firmware. (Other options are to drop to the shell to use scp or zmtx, or to remotely scp from /tmp.)

                            Be warned that if mapping a drive, it should be off the network served on network 0 (mgmt), otherwise the network traffic to the drive may interfere with your WAN emulation/WAN monitoring.

                            U - Unmount whatever is mounted

                            N - Mount a Network File System - such as a Linux or Solaris drive which is exported.

                            W - Map a Windows network "share" drive.

                            S - Change the default directory (for tcpdump and network monitor) to use /tmp/mnt (or /tmp).

                            L - List the mounted drive.

                            X - Return to the main menu.


                            6.3 gigEnn™ Advanced Mode

                            • N - normal mode

                            • A - Asymmetric mode

                            • 3 - 3-way mode

                            • I - Infinite queue size

                            • Q - Set fixed queue size

                            • B - Use Bandwidth * delay
                            • The Network Nightmare supports 2 advanced modes - in one case there will be different network characteristics from net1->net2 compared to net2->net1, in the other the third interface is also usable for network emulation. Note that this later configuration (obviously) requires the use of routing mode.

                              N - normal mode - revert to the original way the Network Nightmare operated with just one setting for bidirectional traffic between net1 and net2

                              A - Asymmetric mode you can work with either bridged interfaces (net1/net2) or routed, but you can emulate different properties in the different directions, such as 1.5 Mbps from net1->net2, but only 386 kbps in the reverse direction.

                              3 - 3way mode works only for routed mode, but enables arbitrary settings in all 6 directions through the device. i.e. traffic from net1 to net2, net1 to mgmt, etc.

                              I - Infinite queues (don't drop due to delayed packets) default

                              Q - Set fixed queue size in packets

                              B - Use ( bandwidth * delay ) to determine queue size

                              X - Return to the main menu.


                              7.0 gigEnn™ Statistics (Monitoring)

                              • S - Summary of Traffic
                                (traffic.dat)
                              • P - Traffic Summary by port
                                (traffic_byport.dat)
                              • T - Top Talkers
                                (toptalkers.dat)
                              • A - Top Talking Applications
                                (topapps.dat)
                              • G - Most talking machines
                                (mosttalking.dat)
                              • Y - CIFS/SMB Signing status
                                (smb_conns.dat)
                              • M - Monitor output
                                (monitor.out)
                              • E - Open connections at monitor end
                                (rt_open.dat)
                              • U - Every connection
                                (all_connections.dat)
                              • Only on the PLUS model

                                Note that the monitoring will not begin until a significant amount of traffic has been seen, so it is not suitable for extremely low traffic evaluation. While running the monitor will print its status every 10 seconds, the output looks like:

                                1100650592.535386  CONNECTIONS open: 2 (2)      total: 2 (2)    max-sim: 2 (2)
                                1100650592.535386  BYTES tcp: 1580404/57        udp: 0/0        other: 0/0
                                

                                The first number (1100650592.535386) is a timestamp (seconds.msecs) If you have not used rdate with external access or otherwise set the time, this will be from an arbitrary starting point, but will be useful for relative timing in any case.

                                The Connections line describes the TCP connections currently extant:

                                The Traffic line gives the number of BYTES sent of each type, followed by the number of packets, e.g. bytes/packets. TCP is a reliable protocol, which is most common on networks. UDP is a lossy protocol used for some applications needing low delay, and other comprises all other traffic types, such as GRE or certain VPN protocols.

                                The monitoring process will produce files in /tmp (or /tmp/mnt) to explore different aspects of the traffic it saw. You can browse the files from this menu, or copy them off of the Network Nightmare. In addition it will produce several .csv files which are useful to load into other tools, such as Microsoft Excel(TM).

                                S - Summary of Traffic (traffic.dat) - provides summary statistics for the monitored period. These include the length of trace (secs), number of bytes sent, average speed, average RTT, a breakdown into the number of retransmitted bytes and duplicate acks, #connections, SMB signing statistics, byte and packet counts broken down into TCP/UDP/other, and the peak number of simultaneous connections.

                                P - Traffic Summary by port (traffic_byport.dat) - How much data was sent on each TCP port. Note that BOTH ports of a connection are summarized. i.e. if a client connects to a web-server on port 80, then that traffic counts towards port 80, but it ALSO counts toward the ephemeral port the client received in order to open this connection. Thus each byte sent is represented twice in this summary.

                                T - Top Talkers (toptalkers.dat) - The connections seen, sorted by which sent the most traffic (top 20), plus the rate at which they sent the traffic.

                                A - Top Talking Applications (topapps.dat) - similar to traffic_byport, but just the byte counts and ports.

                                G - Most talking machines (mosttalking.dat) - Identifies the IPs which sent the most data during the session.

                                Y - CIFS/SMB Signing status (smb_conns.dat) - SMB signing is an obscure CIFS setting of interest to WAFS vendors who need to know if traffic has had cryptographic checksums added to it by Microsoft.

                                M - Monitor output (monitor.out) - The recapitulates the output from during the monitoring session.

                                E - Open connections at monitor end (rt_open.dat) - this identifies the connections which had NOT been closed at the end of the monitor run.

                                U - Every connection (all_connections.dat) - displays stats (bytes, time, and rate) for each connection seen.

                                X - Return to the main menu.


                                (C) 2004-2007 NetworkNightmare.net All Rights Reserved.