Fixed/Auto/Forced baudrate for MSTP

MSTP baudrate is always painful for field technician. If the baudrate is wrong, device can’t join a MSTP bus.

Most devices have fixed baudrate. To modify the baudrate setting, technician have to physically access the device and change dip switches. Some devices support changing baudrate by BACnet service. but before that they should already have correctly baudrate setting for BACnet service to access it.

Some vendors implement auto baudrate, but introduce more problem than it solves. There are two types of auto baudrate mechanism:

  • Starting detection: The device detects and adopts baudrate on the bus when it starts. then never changes baudrate.
  • Dynamic detection: The device does same as starting detection type when it starts, but if it find there are error on bus for a predefined time, it considers that the baudrate is changed, it detects baudrate again.

For both types, it is difficult to change baudrate when devices is working. Simply changing baudrate on all fixed baudrate devices can not work, because auto baudrate devices are still working on old baudrate. The solution is to power off all auto baudrate devices, then power on all auto baudrate devices(Don’t power off/on auto baudrate devices one by one)

Our new firmware(>=2.0) introduces new baudrate management mechanism(Patent pending). There are 3 types of baudrate mode for BACRouter : Fixed/Auto/Force:

  • Fixed baudrate mode works as most traditional devices.
  • Auto baudrate mode is same as above-mentioned dynamic detection. The predefined time to re-initiate detection is 10 error frames.
  • Forced baudrate mode is same as auto mode except that when the device get token, it changes baudrate to predefined value.

When there is a device with forced baudrate mode, the baudrate on the bus will be forced to predefined value. Devices with auto baudrate mode will automatically synchronize to predefined baudrate. Devices with fixed baudrate mode but baudrate setting different with predefined value will not be seen on bus (It’s easy to check out in “Recent active devices” field from BACRouter’s runtime info). Devices with starting detection type may run on wrong baudrate, they will not be seen on bus too, but powering off/on them one by one will synchronize them to forced value.

More than one device with forced baudrate mode could coexist on a bus, but the baudrate values on them should be same.

 

Max_info_frames by token occupy time

From firmware ver2.0, BACRouter introduced new “Max_info_frames by token occupy time” feature.

In BACnet standard of MSTP, a master device could hold token until it has sent frames up to Max_info_frames. The default value of Max_info_frames is 1. But for router, this value may be increased to improve bandwidth between networks. Mostly the suggested value for router is between 5 to 20.

MSTP works as a field bus for controllers; sensors and actuators. The data exchanging latency between devices usually should be guaranteed.  We recommend devices get token at least every 1 second.

The APDUs passing router usually have size between 10~50 bytes, but could be up to 480 or 1476 (Extended frame). Larger APDU need more time to send or receive.

For APDU which need a reply from targeted device, router has to wait for reply. Usually the targeted device need more time to handle or generate larger APDU, router has to wait longer.

So the time router holding token could be varied much, which impacts latency guaranty of MSTP bus. To avoid this problem, “Max_info_frames by token occupy time” feature limits router’s token holding time.

The limitation is calculated by:

byte_time * 32 * Max_info_frames

For example, Max_info_frames is set to 10. The baud rate is 76800bps, so the byte_time is 0.13 milliseconds:

0.13 * 32 * 10 = 41.6 milliseconds.

When router founds it have held token for 41.6 milliseconds, it passes token to next station, though the frames it sent may be less than 10.

This feature could be enabled/disabled by user from WebUI.

Solution to MSTP frame desynchronization

Updated on 2020.3.25 for firmware version 3.x

We had discussed BACnet MSTP weakness to frame desynchronization in below:

BACnet MSTP frame lost synchronization

Attack BACnet MSTP by frame desynchronization

But what is BACRouter’s solution to this problem, let’s looking for clue from standard 9.5.3:

Tframe_gap is “The maximum idle time a sending node may allow to elapse between octets of a frame the node is transmitting”, its value is 20 bits time. Almost every MSTP implementation have zero gap between octets in frame.

So BACRouter use a revised RSM:

  1. When there is a idle line longer than 20 bits time, the previous partial frame is aborted.
  2. Idle time on bus greater than or equal to Tturnaround means frame gap. Considering measure error introduced by byte frame desynchronization, Finally we use 30.5 bits idle time to distinguish new frame.
  3. To be compatible with devices not respecting to Tturnaround, any data following valid frame will be regarded as new frame.

In 115200bps, one bit time is only 8.7us. To precisely measure duration of idle line, the timer granularity of BACRouter is set to only 5us. It help to resist to frame desynchronization, and reach 98.8% bandwidth utilization on 115.2kbps because BACRouter no more waste time when 40 bits Tturnaround is over.

Attack BACnet MSTP by frame desynchronization

As pointed out by previous article “BACnet MSTP frame lost synchronization” , BACnet MSTP has a design flaw on frame synchronization, but how to utilize it to perform attack and strictly obey the standard at the same time?

We make some assumptions here:

  1. There are at least 3 devices on the bus with MAC address 1, 8,10. The device 1 is carefully designed to perform attack. Device 8 and 10 are innocent.
  2. Device 1 supports extended frame, device 8 and 10 are not.
  3. The timers of 3 devices is precise enough.

The work flow of device 1 is:

  1. When get token, send out frame A
  2. Pass token to MAC address 2
  3. When get token again, send out frame B
  4. Pass token to MAC address 2
  5. goto step 1 again.

Frame A is a valid proprietary frame (hexadecimal);

55 ff 80 ff 01 00 1d a3 02 2b 72 fe 55 ff 03 08 01 00 11 a0 ff 55 ff 21 01 08 00 09 ce d4 f3 55 ff 00 01 08 00 00 bf

Frame B is also a valid proprietary frame as:

55 ff 80 ff 01 00 1d a3 02 2b fe dc 55 ff 03 0a 01 00 11 b1 ff 55 ff 21 01 0a 00 09 fd 8a 51 55 ff 00 01 0a 00 00 8c

Every thing will go well if there is no frame desynchronization, but after hours running, if device 8 losses synchronization with frame A header (It has same effect if device 10 losses synchronization when device 1 sends frame B) , device 8 find another frame when scan Frame A’s data portion:

55 ff 03 08 01 00 11 a0 ff 55 ff 21 01 08 00 09 ce d4 f3 55 ff 00 01 08 00 00 bf

It’s a Test-Request frame send to device 8,  so device 8 try to reply it after Tturnaround with a Test-Response frame:

55 ff 04 01 08 00 11 ae ff 55 ff 21 01 08 00 09 ce d4 f3 55 ff 00 01 08 00 00 bf

but at the same time, device 1 passes token by sending:

55 ff 00 02 01 00 00 73

So the first 8 bytes of two frames are collided, so device 10 drop invalid header, find data as below:

55 ff 21 01 08 00 09 ce d4 f3 55 ff 00 01 08 00 00 bf

When device 1 finishs sending, it starts receiving data and get the same as:

55 ff 21 01 08 00 09 ce d4 f3 55 ff 00 01 08 00 00 bf

For device 10, it get a valid Not-For-Us frame header, so it enter SKIP-DATA state, there is not enough data to skip, so device 10 will wait until Tframe_abort.

For device 1, it’s a BACnet-Extended-Not-Expecting-Reply frame header, because it support extended frame, so it validate header by procedure described in Addendum 135-2012an. Because the data length is too short, so it abort the frame enter IDLE state again, then find another frame:

55 ff 00 01 08 00 00 bf

It’s a token frame passing token to device 1, so device 1 get token then sending Frame B just after Tturnaround:

55 ff 80 ff 01 00 1d a3 02 2b fe dc 55 ff 03 0a 01 00 11 b1 ff 55 ff 21 01 0a 00 09 fd 8a 51 55 ff 00 01 0a 00 00 8c

As mentioned above, Device 10 still wait 1 byte to skip previous frame (because Tturnaround < Tframe_abort), so it miss this frame header. get wrong frame as:

55 ff 03 0a 01 00 11 b1 ff 55 ff 21 01 0a 00 09 fd 8a 51 55 ff 00 01 0a 00 00 8c

It’s another Test-Request frame send to device 10, thing repeats.

From above, every devices strictly obey standard, but once frame desynchronization occurs, the whole MSTP bus is stalled forever.

Read more on Solution to mstp frame desynchronization

BACnet MSTP frame lost synchronization

There are two concepts of frame:

  1. BACnet MSTP datalink layer frame, it has at least 8 octet bytes, including: 2 preamble bytes of 0x55 and 0xff, frame_type, destination_mac, source_mac, 2 data_len bytes, crc, and omissible data portion.  We call this type of frame as “MSTP frame”.
  2. EIA-485 frame, it is consisted of bits, including start bit, data bit, parity bit, stop bit. BACnet MSTP using non-return to zero (NRZ) encoding with one start bit, eight data bits, no parity, and one stop bit. The start bit shall have a value of zero, while the stop bit shall have a value of one. The data bits shall be transmitted with the
    least significant bit first. We call this type of frame as “byte frame”.

BACnet MSTP Receive Frame Finite State Machine (hereafter refer to RSM) distinguish starting of frame by preamble bytes. If there is no DataAvailable or ReceiveError within a frame for Tframe_abort (60 bits time) , the frame is aborted, RSM search for next frame again.

Because preamble bytes is allowed on other portions of MSTP frame (Extended frame introduced by Addendum 135-2012an is an exception, it use COBS encoding to avoid 0x55 existing in data portion), so RSM may parse MSTP frame beginning from data portion of previous MSTP frame.

The minimum of time gap between MSTP frames is Tturnaround (40 bits time), it is less than Tframe_abort, so if RSM lost syhchronization of MSTP frame, it may parse wrong MSTP frame across actual MSTP frames.

There are several cause of losing synchronization:

  1. Program defect on sending or receiving device. It could be resolved by code review and test.
  2. Timer precision. BACnet MSTP standard only require a 1% precise timer with a resolution of 5 ms or less. So it is hard to check out Tframe_abort. Because there is only 5 ms error space between delay and timeout (Tusage_delay to Tusage_timeout, Treply_delay to Treply_timeout), it is very likely causing collision for a slow responding devices.
  3. Noise on bus line. Noise cause byte frame error and crc error, RSM will abort previous MSTP frame in both situations.

Some may argue that MSTP frame integrity is protected by header crc8 and data crc16. Even not considering malicious devices(Here is a attack example), there are still applications may send valid frame in data portion.

For example, There are lot of serial to IP gateways on market, they usually have
one or more RS485/RS232 serial port with Ethernet or Wifi, remotely
read/write raw serial data by TCP/UDP using predefined protocol. The
serial line maybe Modbus or just ASCII protocol, but the designer doesn’t
even know in advance. They are useful on monitoring remote devices or
simple protocol integration.

Is it reasonable to implement these gateway’s IP side protocol as
BACnet PrivateTransfer service? But what would happen when user attach
this gateway to a MSTP bus, and the BACnet route to remotely access it pass through
another MSTP network?

It is not only possible cause MSTP bus being blocked, but also device
malfunction if the wrong frame is a APDU or breaking whole BACnet
inter-network if the wrong frame is a I-Am-Router-To-Network frame.


For the byte frame, losing synchronization usually is caused by noise, no termination or absence of biasing. The symptoms include regarding data bit of 1 as idle line, regarding data bit of 0 as start bit of new byte. In addition to data error, the losing synchronization of last byte in a MSTP frame will introduce measure error of idle time between MSTP frames.

Read more on Solution to mstp frame desynchronization

How fast device feature of BACRouter improves MSTP token pass rate?

The test criterion is: 115.2kbps, 80m cable. 1 router and 10 mstp master devices, max_master=127, no NPDU traffic on bus, define non used mac as fast device.

Router mac

Fast device timeout config

Other device’s mac

Token pass rate(round/min)

Percentage Base no fast

127

—-

117~126

2301

100%

127

0ms

117~126

4856

211%

127

1ms

117~126

4734

206%

127

2ms

117~126

4473

194%

116

0ms

117~126

4690

204%

116

1ms

117~126

4577

199%

116

2ms

117~126

4343

189%

58

—-

117~126

1777

100%

58

0ms

117~126

4563

257%

58

1ms

117~126

4405

248%

58

2ms

117~126

4071

229%

110

—-

0,11,22,33,44,55,66,77,88,99

1153

100%

110

0ms

0,11,22,33,44,55,66,77,88,99

4037

350%

110

1ms

0,11,22,33,44,55,66,77,88,99

3813

331%

110

2ms

0,11,22,33,44,55,66,77,88,99

3393

294%

BACnet router changelog

3.10  2020.7.1  Download

The config file format is changed much between 2.x and 3.x,  so from 3.00 to 3.09, when upgrade firmware, the upgrading script will install new config file.  This version add a “Keep config” option when upgrade firmware. If config file have something wrong, the WebUI will popup a warning when it is opened.

3.09  2020.6.30

Fixs bug introduced in 3.08 when fetch mstp runtime info.

Some minor WebUi improvements.

3.08  2020.6.28

Bugfix: 1 hop broadcast distribution is optional for BBMD, From version 3.00, BACRouter force other BBMDs to use 1 hop distribution if cross-network broadcast support is enabled. It’s incorrect behavior and fixed in this version.

BDT editing UI is improved. When submitting, BDT entries will be checked to detect possible misconfigure. The checking is also taken when receiving Write_BDT BVLL message.

If multiple BBMD support is not enabled, when receiving forwarded npdu from BBMD not present in local BDT, the npdu will be dropped and logged. It will help distinguishing configure problem of BBMD on most situation. “Report multiple BBMD” function in last verion is covered by this function, so it’s removed.

The filename of downloaded config is set as IP_Version.json, for example: 192.168.100.1_3.08.json

Config uploaded from file will be edited in WebUI before saving to eliminate any potential incompatibility.

3.06  2020.5.22  Download

Ethernet is disabled by default. If BIP and Ethernet are enabled at the same time, a warning of possible circular route will be promoted.

The default capture size increases to 4M.

3.05  2020.5.15  Download

Capture control function is mapped to BACnet.  Each datalink port has its “Capture buf_size” and “Capture command” Multi_State_Value. “Capture buf_size” could be set between “64K” to “16M”. “Capture command” could be set between “Stop&Clean”, “Start”, “Stop”.

3.04  2020.4.28  Download

Password for root could be modified from WebUI now.

3.03  2020.4.23  Download

BACRouter verifies other router’s live by query its routing table with Initialize-Routing-Table message. But addendum 135-2012AL removed the enforced supporting of Initialize-Routing-Table from router. So in this version BACRouter also monitors Reject-Message-To-Network message that could only be sent by router.

BIP BVLL header of forwarded npdu is 6 bytes larger than other npdu, so there was potential bug on packet length verification that is fixed on this release.

3.02  2020.3.26  Download

Minor bugfixs for UI

3.01  2020.3.17

“Download config” and “Upload config” are added to “System setting”

3.00  2020.3.10

Add Packet capture feature:

Supported on all ports(Ethernet, BIP, MSTP).

Because Wireshark could not decode MSTP extended frames, MSTP extended frames are converted to traditional data frames. Fortunately, Wireshark doesn’t complain about overflowed data length.

MSTP packet could be downloaded as “Interval format”. it could be used to show exact idle time before packet by selecting “Seconds Since Previous Captured Pakcet” as “Time Display Format” in Wireshark. It is very helpful to analyze timing and performance problem.

Continuous download is supported. Automatic traffic logging is possible with the API.

MSTP auto addressing feature is removed.  Because the same time online for all devices could not be guaranteed, there is no perfect auto addressing solution. To help determine max_master and unused MAC on bus, “Sniffer mode” may be enabled, then “Current max master” could be obtained from run time info. Unused MAC also could be chosen referred to “Recently active devices”.

MSTP support real time analysis for max master and show it on run time info, if the value is not matched to the setting, “unmatched” is showed after it.

MSTP support sniffer mode. it could analyze and record data without interfering the bus.

The WebUI of MSTP port add options for basic mode and extend mode.

The timeout settings for MSTP are changed from Integer to Float.

MSTP port’s slave proxy is only support broadcasted Who-Is in previous version. Now it support unicast too.

MSTP port’s slave proxy send unicasted I-AM instead of broadcasted one in previous version.

There is a bug in previous version that Who-Is packet readed by MSTP slave proxy module bypass source address routing verify. it’s fixed this time.

BBMD module add option to log error when multiple BBMDs are found in local lan.

Modify feature introduced by version 2.17. “Accept unmatched destination IP” option is provided, if it is selected, only warning message will be logged, else error message will be be logged and the packet will be dropped.

Many WebUI improvements.

2.18  2019.4.16  Download

Improved IP&Ethernet processing performance.

Process 0 in unconfirmed COV notification service is reserved for unsubscribed notificationCOV. Not longer reports error when can not find matched process for 0.

2.17  2019.4.2  Download

In previous version, when BACRouter receive Original-broadcast BVLL from unicast address or Original-unicast BVLL from broadcast address, it would print error message then drop it.  A customer reported that Johnson Controls’ CCT sends Original-broadcast BVLL to unicast address, so now we only print warning message.

2.16  2019.2.25  Download

From version 2.0.8,  only proprietary network layer messages would be relayed. Now all unknown network layer messages will be relayed.

2.15  2017.10.19  Download

Redirect URL from “/” to “/?” to support Edge web browser (avoid keep asking authorization)

MSTP: Fix bug in standard state machine which would cause passing token to self when duplicated token is found.

MSTP: With standard state machine device in sole master mode would send max_info_frames * Npoll frames after polling one master. A more reasonable behavior is implemented that device send max_info_frames * Npoll frames after finish polling all masters.

2.14  2017.10.13  Download

Fixed bug introduced from 2.0, it should not wait reply when it send expecting reply request to mac address 255 on MSTP.

2.13  2017.9.19  Download

Potential bug (barely) may be triggered if MSTP auto mac and baudrate forcing are enabled together.

2.12  2017.7.10  Download

Optimizing route entry protection: Deletion from route hop will bypass protection.

Support devices utilizing 6.5.3 methods 1 and 4 for establishing the address of a BACnet router for a particular DNET.

2.11  2017.6.16  Download

This update help building up routing table quickly in large BACnet inter-network.

When BACRouter startups, it broadcasts out Who_Is_Router_To_Network without network number specified to learn routing info from other routers.

When Who_Is_Router_To_Network/Router_Busy_To_Network/Router_Available_To_Network without network number specified is received, BACRouter reports network number up to 1120 instead of 112 in previous version.

2.10  2017.6.13  Download

Bugfix: ReadRange on property_list should not return object_identifier; object_name; object_type.

Delay 1 second for sending Network_Number_Is when router startup, because if router is a BIP foreign device, the foreign device registering have not completed yet.

2.09  2017.6.10

Protect route entry 60 seconds from last activity. Avoid broadcast storm in circular routing path.

2.08  2017.6.7

Bugfix: Vendor proprietary network message should be correctly relayed.

When relay Reject-Message-To-Network with reason 4(Too long), use unicast address instead of broadcast address.

Relax daemon’s max_apdu_retries to 10, in accord with WebUI.

2.07  2017.6.5

Compliant to BACnet Router and BBMD device profile.

Try to purge invalid route entry if no reply is received for Who_Is_Router_To_Network or Initializing_Routing_Table. This feature helps to rescue from mess network configuration.

Avoid sending too many I_Am_Router_To_Network with circular routing path.

Reduce memory consumption in complicated inter-network.

2.06  2017.6.1

Avoid BBMD broadcast storm. If Forwarded-NPDU is received from broadcast address, do not re-broadcast it.

When relay Reject-Message-To-Network with reason 1/2/4, use broadcast address.

Discard Reject-Message-To-Network with incorrect route info.

Check source net of incoming NPDU.(Trickily handle Initializing-Routing-Table)

2.05  2017.5.23

Bug: global broadcasted initializing-routing-table should be executed.

Bug: WebUI BIP mode select should not be applied without submiting.

Limit number of route entry in runtime info page to avoid request failure in very large internetwork.

2.04  2017.5.11

Fix WebUI Bug for verifying BDT.

Save configuration and restart when receive reinitialize-device service request.

2.03  2017.5.7

BIP BBMD NAT mode support udp port modifying.

BIP BBMD NAT mode do not listen to local net broadcast address and not send broadcast to local net.

BIP foreign device mode do not listen to local net broadcast address.

2.02  2017.5.5

Regards device sent Who-Is-Router-To-Network has no route to that network.(Delete that route entry from route table)

When Initializing-Routing-Table request modify network number of route port, I-Am-Route-To-Network is sent to report new network number.

Fix bug introduced by ver2.0 for saving new BDT modified by BVLL Write-Broadcast-Distribution-Table message.

2.01  2017.5.3

Add upgrade entry “http://ip/app/upgrade for rescue.

More intuitive log.

Improve network reject behavior.

Fix overflow bug when send I-Am-Router-To-Network(if network numbers reported > 738).

Sending I-Am-Router-To-Network at dynamic time interval.

2.0  2017.4.19

Option to enable Device object inside router.

Allow setting modification from BACnet side.

Support What_is_Network_Number and Network_Number_Is message

Option to support MSTP max_info_frames by occupy time.

Introduce MSTP new concept of auto/forced baudrate.

Guarantee of MSTP message delay(10 seconds).

WebUI is more intelligent.

1.22  2016.10.26  Download

Increase max route table entry from 1024 to 65534(unlimited).

1.21  2016.10.18

Slave Proxy: Increase ReadProperty(Multiple) sending rate if remote device is fast device.

BVLL response NAK for BBMD-related BVLL request but BBMD is not enabled. (Addendum 135-2012ax-5)

BVLL response NAK Upon receipt of a Distribute-Broadcast-To-Network message from a un-registered foreign device. (Addendum 135-2010ad-10)

1.20  2016.10.8

BIP provide option to accept buggy broadcast to 255.255.255.255

Reduce minimum value of register interval for BIP Foreign Device Mode from 30 seconds to 15 seconds.

1.19  2016.9.23

Change webui reboot to restart, if IP/Netmask/Gateway/DHCP is not changed, fastly restart self

MSTP add fast device timeout interrupt feature

MSTP fast device minimums timeout is set to 0ms (actually 1.5bits plus 50us after turnaround)

MSTP fast device minimums token pass timeout is limit to 20ms

MSTP runtime info shows devices is active in past 10 seconds

1.18  2016.9.13

BIP BBMD support NAT, add two config parameter: “Accept BDT push from other” and “Accept foreign device register”.

Show Ethernet MAC address on “Setting” page.

1.17  2016.9.8

Requested by some OEM customer, now login authorization domain could be customized when place order.

1.16  2016.9.7

Bugfix: slave proxy should not response to who-is from port that device is not proxied on.

Bugfix: write to remote BBMD BDT fail

Lots UI improving.

1.15  2016.9.1

New slave proxy feature.

MSTP add no_turn_around frame count and padding frame count diagnosis information.

Some UI improved.

Fix decode bug when no turnaround frame following a padding frame.

1.14  2016.8.24

It’s a big release, many new features and bugfix:

MSTP add setting for Treply_timeout and Tusage_timeout.

MSTP add fast device feature, allow user define some nodes as fast device, apply as short as 1 milliseconds to Treply_timeout and Tusage_timeout

MSTP add more diagnosis information as: mac conflict frame count, send conflict count, token lost count, pass token fail count, reply timeout count, frame error count, token duplicate count.

MSTP enhance compatible to some devices no respecting 40bits turnaround requirement.

MSTP fix bug decode extended frame.

MSTP fix bug calculate test response header crc on specified situation.

1.13  2016.8.16

Low level api change, not affected.

1.12  2016.8.12

Add serial no.

1.11  2016.8.10

Firmware filename has version and crc.

Show version on left tree menu.

Auto refresh page after reboot or firmware upgrade .

Other UI bugfixs.

Bandwidth of BACnet MSTP

A common problem of BACnet MSTP is low bandwidth utilization. Some researches disclose that more than 50% traffic on MSTP bus is token frame.

Unfortunately, many poor implement of MSTP make it worse. For example, in standard, the Tturnaroud is 40 bits, that is said, on 115200bps, node only need to wait 0.35 millisecond to send frame when it receive token. But in practice, there are a lot of devices wait at lease several milliseconds.

Our implement of MSTP try best to improve bandwidth. In previous laboratory test, we achieve 98.8% bandwidth utilization on our BACnet router. The test criterion is:

115200bps, 4 node with mac address from 0 to 3, max_master set to 3, no NPDU. in theory, the max token relay rate is 960 per seconds. we get 948 per second.

In another test, we send a read property request (23 bytes frame) each second, then get a Reply_Postponed frame, then a  read property response (29 bytes frame). we still get token relay rate of 943 per second.

Update Feb, 2017, We make a stress test for ReadProperty service on above configuration, set max_info_frames to 10 in all devices, 2 devices request, the other 2 devices reply. The result is 153~154 ReadProperty requests are served each second.

Update March,2017. Thank to our detailed runtime info, user could get token pass rate easily. 9 months after we release first firmware, We perform above test again, then get 956.6 per second, 99.6% bandwidth utilization.