Compress the data that is sent from the server to the client using websockets.

[tzachi-dar]

As traffic from google cloud is limited I have analyzed the data that is passed from google cloud, and got to conclusion that compressing the data before sending it can save around 85% of the data that is being sent. This will also make the site respond faster on slow networks.
The implementation is quite trivial.

Here is an example to the savings that I was able to achieve on my system.
As can be seen, the compression of the big objects is up to 95% while the smaller ones are about 50%.

12/25/2022, 4:57:56 PM xxxxxx dataUpdate obj size = 526003 gzip obj size 51192
12/25/2022, 4:58:00 PM xxxxxx retroUpdate obj size = 6777519 gzip obj size 246483
12/25/2022, 4:58:01 PM xxxxxx dataUpdate obj size = 526003 gzip obj size 51192
12/25/2022, 4:58:16 PM xxxxxx retroUpdate obj size = 6777519 gzip obj size 246483
12/25/2022, 4:58:23 PM xxxxxx dataUpdate obj size = 12816 gzip obj size 2310
12/25/2022, 4:59:16 PM xxxxxx dataUpdate obj size = 60515 gzip obj size 6841
12/25/2022, 4:59:34 PM xxxxxx dataUpdate obj size = 648 gzip obj size 350
12/25/2022, 5:00:23 PM xxxxxx dataUpdate obj size = 4070 gzip obj size 1620
12/25/2022, 5:00:41 PM xxxxxx dataUpdate obj size = 648 gzip obj size 351
12/25/2022, 5:01:17 PM xxxxxx dataUpdate obj size = 4070 gzip obj size 1622
12/25/2022, 5:01:29 PM xxxxxx retroUpdate obj size = 6784536 gzip obj size 247126
12/25/2022, 5:01:33 PM xxxxxx dataUpdate obj size = 648 gzip obj size 350
12/25/2022, 5:02:16 PM xxxxxx dataUpdate obj size = 5388 gzip obj size 1878
12/25/2022, 5:03:16 PM xxxxxx dataUpdate obj size = 5987 gzip obj size 1966
12/25/2022, 5:04:16 PM xxxxxx dataUpdate obj size = 4552 gzip obj size 1752
12/25/2022, 5:04:44 PM xxxxxx retroUpdate obj size = 6777550 gzip obj size 247026
12/25/2022, 5:05:22 PM xxxxxx dataUpdate obj size = 3470 gzip obj size 1370
12/25/2022, 5:06:17 PM xxxxxx dataUpdate obj size = 5214 gzip obj size 1890
12/25/2022, 5:06:33 PM xxxxxx dataUpdate obj size = 2915 gzip obj size 1174
12/25/2022, 5:06:45 PM xxxxxx dataUpdate obj size = 1368 gzip obj size 491
12/25/2022, 5:07:16 PM xxxxxx dataUpdate obj size = 5258 gzip obj size 1935
12/25/2022, 5:07:26 PM xxxxxx dataUpdate obj size = 2302 gzip obj size 996
12/25/2022, 5:07:35 PM xxxxxx dataUpdate obj size = 1368 gzip obj size 490
12/25/2022, 5:08:00 PM xxxxxx retroUpdate obj size = 6765457 gzip obj size 247094
12/25/2022, 5:08:16 PM xxxxxx dataUpdate obj size = 4642 gzip obj size 1776
12/25/2022, 5:08:28 PM xxxxxx dataUpdate obj size = 2302 gzip obj size 997
12/25/2022, 5:08:39 PM xxxxxx dataUpdate obj size = 1368 gzip obj size 488
12/25/2022, 5:09:23 PM xxxxxx dataUpdate obj size = 5726 gzip obj size 2064
12/25/2022, 5:09:30 PM xxxxxx dataUpdate obj size = 2302 gzip obj size 998
12/25/2022, 5:09:41 PM xxxxxx dataUpdate obj size = 713 gzip obj size 432
12/25/2022, 5:10:23 PM xxxxxx dataUpdate obj size = 3988 gzip obj size 1713
12/25/2022, 5:10:36 PM xxxxxx dataUpdate obj size = 2302 gzip obj size 999
12/25/2022, 5:10:47 PM xxxxxx dataUpdate obj size = 713 gzip obj size 433
12/25/2022, 5:11:15 PM xxxxxx retroUpdate obj size = 6774508 gzip obj size 247145
12/25/2022, 5:11:16 PM xxxxxx dataUpdate obj size = 3988 gzip obj size 1717
12/25/2022, 5:11:26 PM xxxxxx dataUpdate obj size = 2807 gzip obj size 1164
12/25/2022, 5:11:38 PM xxxxxx dataUpdate obj size = 713 gzip obj size 433
12/25/2022, 5:12:15 PM xxxxxx dataUpdate obj size = 3988 gzip obj size 1715
12/25/2022, 5:12:29 PM xxxxxx dataUpdate obj size = 2934 gzip obj size 1176
12/25/2022, 5:12:41 PM xxxxxx dataUpdate obj size = 713 gzip obj size 431
12/25/2022, 5:13:17 PM xxxxxx dataUpdate obj size = 4618 gzip obj size 1887
12/25/2022, 5:13:28 PM xxxxxx dataUpdate obj size = 2321 gzip obj size 1004
12/25/2022, 5:13:39 PM xxxxxx dataUpdate obj size = 713 gzip obj size 433
12/25/2022, 5:14:22 PM xxxxxx dataUpdate obj size = 5157 gzip obj size 2069
12/25/2022, 5:14:23 PM xxxxxx retroUpdate obj size = 6769778 gzip obj size 247004
12/25/2022, 5:14:29 PM xxxxxx dataUpdate obj size = 2321 gzip obj size 1004
12/25/2022, 5:14:38 PM xxxxxx dataUpdate obj size = 713 gzip obj size 432
12/25/2022, 5:15:22 PM xxxxxx dataUpdate obj size = 4002 gzip obj size 1731
12/25/2022, 5:15:35 PM xxxxxx dataUpdate obj size = 2321 gzip obj size 1002
12/25/2022, 5:15:47 PM xxxxxx dataUpdate obj size = 713 gzip obj size 433
12/25/2022, 5:16:16 PM xxxxxx dataUpdate obj size = 4510 gzip obj size 1883
12/25/2022, 5:16:30 PM xxxxxx dataUpdate obj size = 2321 gzip obj size 1003
12/25/2022, 5:16:40 PM xxxxxx dataUpdate obj size = 1307 gzip obj size 540
12/25/2022, 5:17:17 PM xxxxxx dataUpdate obj size = 5147 gzip obj size 1987
12/25/2022, 5:17:28 PM xxxxxx dataUpdate obj size = 2321 gzip obj size 1003
12/25/2022, 5:18:17 PM xxxxxx dataUpdate obj size = 5459 gzip obj size 1978
12/25/2022, 5:18:29 PM xxxxxx dataUpdate obj size = 2321 gzip obj size 1005
12/25/2022, 5:19:16 PM xxxxxx dataUpdate obj size = 4298 gzip obj size 1801
12/25/2022, 5:19:26 PM xxxxxx dataUpdate obj size = 4102 gzip obj size 1578
12/25/2022, 5:20:23 PM xxxxxx dataUpdate obj size = 4778 gzip obj size 1881
12/25/2022, 5:20:36 PM xxxxxx dataUpdate obj size = 2321 gzip obj size 1003
12/25/2022, 5:21:16 PM xxxxxx dataUpdate obj size = 4867 gzip obj size 1775
12/25/2022, 5:21:28 PM xxxxxx dataUpdate obj size = 2321 gzip obj size 1002
12/25/2022, 5:22:16 PM xxxxxx dataUpdate obj size = 4829 gzip obj size 1828
12/25/2022, 5:23:16 PM xxxxxx dataUpdate obj size = 2971 gzip obj size 1294
12/25/2022, 5:24:15 PM xxxxxx dataUpdate obj size = 2971 gzip obj size 1295
12/25/2022, 5:24:25 PM xxxxxx dataUpdate obj size = 507 gzip obj size 334
12/25/2022, 5:25:23 PM xxxxxx dataUpdate obj size = 2971 gzip obj size 1294
12/25/2022, 5:26:16 PM xxxxxx dataUpdate obj size = 2971 gzip obj size 1294
12/25/2022, 5:27:17 PM xxxxxx dataUpdate obj size = 5621 gzip obj size 1908
12/25/2022, 5:28:16 PM xxxxxx dataUpdate obj size = 6303 gzip obj size 2034
12/25/2022, 5:29:16 PM xxxxxx dataUpdate obj size = 3257 gzip obj size 1375
12/25/2022, 5:29:31 PM xxxxxx dataUpdate obj size = 593 gzip obj size 374
12/25/2022, 5:30:23 PM xxxxxx dataUpdate obj size = 3257 gzip obj size 1376
12/25/2022, 5:31:17 PM xxxxxx dataUpdate obj size = 4657 gzip obj size 1801
12/25/2022, 5:32:23 PM xxxxxx dataUpdate obj size = 5254 gzip obj size 1954
12/25/2022, 5:33:15 PM xxxxxx dataUpdate obj size = 3312 gzip obj size 1379
12/25/2022, 5:34:16 PM xxxxxx dataUpdate obj size = 3312 gzip obj size 1382
12/25/2022, 5:34:36 PM xxxxxx dataUpdate obj size = 1813 gzip obj size 772
12/25/2022, 5:35:23 PM xxxxxx dataUpdate obj size = 3312 gzip obj size 1380
12/25/2022, 5:36:17 PM xxxxxx dataUpdate obj size = 3856 gzip obj size 1543
12/25/2022, 5:37:15 PM xxxxxx dataUpdate obj size = 3249 gzip obj size 1389
12/25/2022, 5:38:16 PM xxxxxx dataUpdate obj size = 4487 gzip obj size 1698
12/25/2022, 5:39:16 PM xxxxxx dataUpdate obj size = 3243 gzip obj size 1353
12/25/2022, 5:39:38 PM xxxxxx dataUpdate obj size = 1138 gzip obj size 578
12/25/2022, 5:40:23 PM xxxxxx dataUpdate obj size = 3243 gzip obj size 1358
12/25/2022, 5:41:16 PM xxxxxx dataUpdate obj size = 3754 gzip obj size 1516
12/25/2022, 5:42:16 PM xxxxxx dataUpdate obj size = 4502 gzip obj size 1703
12/25/2022, 5:43:16 PM xxxxxx dataUpdate obj size = 3265 gzip obj size 1359
12/25/2022, 5:44:16 PM xxxxxx dataUpdate obj size = 3265 gzip obj size 1361
12/25/2022, 5:44:39 PM xxxxxx dataUpdate obj size = 1144 gzip obj size 577
12/25/2022, 5:45:23 PM xxxxxx dataUpdate obj size = 3265 gzip obj size 1360
12/25/2022, 5:45:39 PM xxxxxx dataUpdate obj size = 2907 gzip obj size 1158
12/25/2022, 5:46:17 PM xxxxxx dataUpdate obj size = 4511 gzip obj size 1700
12/25/2022, 5:46:29 PM xxxxxx dataUpdate obj size = 2799 gzip obj size 1147
12/25/2022, 5:47:16 PM xxxxxx dataUpdate obj size = 3268 gzip obj size 1355
12/25/2022, 5:47:28 PM xxxxxx dataUpdate obj size = 2430 gzip obj size 1011
12/25/2022, 5:48:16 PM xxxxxx dataUpdate obj size = 3268 gzip obj size 1357
12/25/2022, 5:48:29 PM xxxxxx dataUpdate obj size = 2322 gzip obj size 995
12/25/2022, 5:49:15 PM xxxxxx dataUpdate obj size = 3268 gzip obj size 1355
12/25/2022, 5:49:27 PM xxxxxx dataUpdate obj size = 2322 gzip obj size 993
12/25/2022, 5:49:43 PM xxxxxx dataUpdate obj size = 1159 gzip obj size 585
12/25/2022, 5:50:23 PM xxxxxx dataUpdate obj size = 3268 gzip obj size 1359
12/25/2022, 5:50:35 PM xxxxxx dataUpdate obj size = 2322 gzip obj size 997
12/25/2022, 5:50:46 PM xxxxxx dataUpdate obj size = 1824 gzip obj size 474
12/25/2022, 5:51:16 PM xxxxxx dataUpdate obj size = 5059 gzip obj size 1667
12/25/2022, 5:51:29 PM xxxxxx dataUpdate obj size = 2322 gzip obj size 995
12/25/2022, 5:51:41 PM xxxxxx dataUpdate obj size = 1824 gzip obj size 473
12/25/2022, 5:52:16 PM xxxxxx dataUpdate obj size = 5059 gzip obj size 1667
12/25/2022, 5:52:27 PM xxxxxx dataUpdate obj size = 2827 gzip obj size 1155
12/25/2022, 5:53:15 PM xxxxxx dataUpdate obj size = 3268 gzip obj size 1356
12/25/2022, 5:53:29 PM xxxxxx dataUpdate obj size = 2827 gzip obj size 1157

[Suchiman]

You could also turn on compression via socket.io which in turn enables websocket compression (using deflate):
require('socket.io').listen(server, {perMessageDeflate: true});

See also https://github.com/socketio/engine.io/tree/3.5.0#methods-1

[sulkaharo]

I'd much prefer not adding new dependencies and not making the socket.io customized on protocol level as some projects (notably AndroidAPS) uses socket.io. The data should already be compressed over the http long poll transport so I'm curious to hear what the measured effect of this change is? See https://socket.io/docs/v4/server-options/

[tzachi-dar]

I have done two measurments of running once with compression and once without it. The differances are very big:
Here are the sizes of the wireshark captureus for one minute after loading the site:
Without compression:
7,147,824 nightscout_no_compression_displayed.pcapng
8,155,108 nightscout_no_compression_displayed_2.pcapng
With compression:
426,740 nightscout_with_compression_displayed.pcapng
445,968 nightscout_with_compression_displayed_2.pcapng

So, the saving is around 20% of the data.
The site also seems to go up faster.

[tzachi-dar]

tcpdump.zip
Here are the captured files. If anyone wants to get a better insite, I can repeat this with http instead of https.

[sulkaharo]

Ok #7793 updates socket.io to latest release and fixes a bug that caused the standard http long poll compression to be disabled. The standard compression retails the compression table across requests and is much more efficient at the compression than compressing each request separately, so see screenshot with 25x compression on the initial data load.

[sulkaharo]

Closing this as the socket.io fix is in dev. Please test the dev branch.