Node.js 14.x integration tests are failing on response size asserts #769

macbre · 2020-08-25T17:09:32Z

When run locally:

  phantomas:modules:requestsMonitor Response metadata: {"id":"78763.3","url":"http://127.0.0.1:8888/static/jquery-2.1.1.min.js","method":"GET","headers":{"date":"Tue, 25 Aug 2020 17:09:50 GMT","content-encoding":"gzip","last-modified":"Fri, 07 Aug 2020 18:10:57 GMT","server":"ecstatic-3.3.2","etag":"W/\"6294735-29415-2020-08-07T18:10:57.644Z\"","content-type":"application/javascript","cache-control":"max-age=84600","connection":"keep-alive","content-length":"29415"},"bodySize":84245,"transferedSize":29730,"responseSize":29730,"type":"js","protocol":"http","domain":"127.0.0.1","query":null,"stalled":0.447,"timeToFirstByte":3.393,"timeToLastByte":0.017317000001639826,"headersSize":296,"gzip":true,"contentType":"application/javascript","isJS":true,"status":200,"statusText":"OK"} +1ms

When run via GitHub Actions:

2020-08-25T16:36:13.1893140Z   /dom-operations.html
2020-08-25T16:36:13.1955120Z     ✓ should be generated
2020-08-25T16:36:13.1989632Z     ✓ should have "requests" metric properly set
2020-08-25T16:36:13.1990301Z     ✓ should have "gzipRequests" metric properly set
2020-08-25T16:36:13.1990900Z     ✓ should have "htmlCount" metric properly set
2020-08-25T16:36:13.1991401Z     ✗ should have "htmlSize" metric properly set 
2020-08-25T16:36:13.1991817Z         » expected 2385, 
2020-08-25T16:36:13.1992158Z   	got	 2408 (strictEqual) // /home/runner/work/phantomas/phantomas/node_modules/vows/lib/assert/macros.js:14
2020-08-25T16:36:13.1992620Z     ✓ should have "cssCount" metric properly set
2020-08-25T16:36:13.1993080Z     ✗ should have "cssSize" metric properly set 
2020-08-25T16:36:13.1993540Z         » expected 308, 
2020-08-25T16:36:13.1993847Z   	got	 331 (strictEqual) // /home/runner/work/phantomas/phantomas/node_modules/vows/lib/assert/macros.js:14
2020-08-25T16:36:13.1994331Z     ✓ should have "jsCount" metric properly set
2020-08-25T16:36:13.1994747Z     ✗ should have "jsSize" metric properly set 
2020-08-25T16:36:13.1995187Z         » expected 29730, 
2020-08-25T16:36:13.1995501Z   	got	 29753 (strictEqual) // /home/runner/work/phantomas/phantomas/node_modules/vows/lib/assert/macros.js:14
2020-08-25T16:36:13.1995951Z     ✓ should have "domains" metric properly set
2020-08-25T16:36:13.1996353Z     ✓ should have "DOMqueries" metric properly set
2020-08-25T16:36:13.1996775Z     ✓ should have "DOMqueriesById" metric properly set
2020-08-25T16:36:13.1997250Z     ✓ should have "DOMqueriesByClassName" metric properly set
2020-08-25T16:36:13.1997703Z     ✓ should have "DOMqueriesByTagName" metric properly set
2020-08-25T16:36:13.1998115Z     ✓ should have "DOMqueriesByQuerySelectorAll" metric properly set
2020-08-25T16:36:13.1998540Z     ✓ should have "DOMinserts" metric properly set
2020-08-25T16:36:13.1999263Z     ✓ should have "DOMqueriesDuplicated" metric properly set
2020-08-25T16:36:13.1999925Z     ✓ should have "DOMqueriesAvoidable" metric properly set
2020-08-25T16:36:13.2000376Z     ✓ should have "DOMqueriesWithoutResults" metric properly set
2020-08-25T16:36:13.2001575Z     ✗ should have "gzipRequests" offender(s) properly set 
2020-08-25T16:36:13.2002051Z         » Expected values to be strictly deep-equal: 
2020-08-25T16:36:13.2002452Z   + actual - expected 
2020-08-25T16:36:13.2002638Z    
2020-08-25T16:36:13.2002835Z     [ 
2020-08-25T16:36:13.2002965Z       { 
2020-08-25T16:36:13.2003163Z         bodySize: 84245, 
2020-08-25T16:36:13.2003369Z   +     transferedSize: 29753, 
2020-08-25T16:36:13.2003755Z   -     transferedSize: 29730, 
2020-08-25T16:36:13.2011928Z         url: 'http://127.0.0.1:8888/static/jquery-2.1.1.min.js' 
2020-08-25T16:36:13.2012192Z       } 
2020-08-25T16:36:13.2012846Z     ] // /home/runner/work/phantomas/phantomas/test/integration-test.js:83
2020-08-25T16:36:13.2013362Z     ✗ should have "htmlCount" offender(s) properly set 
2020-08-25T16:36:13.2013782Z         » Expected values to be strictly deep-equal: 
2020-08-25T16:36:13.2014172Z   + actual - expected 
2020-08-25T16:36:13.2014363Z    
2020-08-25T16:36:13.2014535Z     [ 
2020-08-25T16:36:13.2014707Z       { 
2020-08-25T16:36:13.2015260Z   +     size: 2408, 
2020-08-25T16:36:13.2015720Z   -     size: 2385, 
2020-08-25T16:36:13.2016127Z         url: 'http://127.0.0.1:8888/dom-operations.html' 
2020-08-25T16:36:13.2016401Z       } 
2020-08-25T16:36:13.2016891Z     ] // /home/runner/work/phantomas/phantomas/test/integration-test.js:83
2020-08-25T16:36:13.2017326Z     ✗ should have "cssCount" offender(s) properly set 
2020-08-25T16:36:13.2017744Z         » Expected values to be strictly deep-equal: 
2020-08-25T16:36:13.2018088Z   + actual - expected 
2020-08-25T16:36:13.2018264Z    
2020-08-25T16:36:13.2018435Z     [ 
2020-08-25T16:36:13.2018562Z       { 
2020-08-25T16:36:13.2018747Z   +     size: 331, 
2020-08-25T16:36:13.2019170Z   -     size: 308, 
2020-08-25T16:36:13.2019920Z         url: 'http://127.0.0.1:8888/static/style.css' 
2020-08-25T16:36:13.2020317Z       } 
2020-08-25T16:36:13.2020834Z     ] // /home/runner/work/phantomas/phantomas/test/integration-test.js:83
2020-08-25T16:36:13.2021265Z     ✗ should have "jsCount" offender(s) properly set 
2020-08-25T16:36:13.2021686Z         » Expected values to be strictly deep-equal: 
2020-08-25T16:36:13.2021985Z   + actual - expected 
2020-08-25T16:36:13.2022172Z    
2020-08-25T16:36:13.2022349Z     [ 
2020-08-25T16:36:13.2022592Z       { 
2020-08-25T16:36:13.2023011Z   +     size: 29753, 
2020-08-25T16:36:13.2023523Z   -     size: 29730, 
2020-08-25T16:36:13.2024898Z         url: 'http://127.0.0.1:8888/static/jquery-2.1.1.min.js' 
2020-08-25T16:36:13.2025160Z       } 
2020-08-25T16:36:13.2025612Z     ] // /home/runner/work/phantomas/phantomas/test/integration-test.js:83

2020-08-25T16:36:13.1893140Z /dom-operations.html 2020-08-25T16:36:13.1955120Z ✓ should be generated 2020-08-25T16:36:13.1989632Z ✓ should have "requests" metric properly set 2020-08-25T16:36:13.1990301Z ✓ should have "gzipRequests" metric properly set 2020-08-25T16:36:13.1990900Z ✓ should have "htmlCount" metric properly set 2020-08-25T16:36:13.1991401Z ✗ should have "htmlSize" metric properly set 2020-08-25T16:36:13.1991817Z » expected 2385, 2020-08-25T16:36:13.1992158Z got 2408 (strictEqual) // /home/runner/work/phantomas/phantomas/node_modules/vows/lib/assert/macros.js:14 2020-08-25T16:36:13.1992620Z ✓ should have "cssCount" metric properly set 2020-08-25T16:36:13.1993080Z ✗ should have "cssSize" metric properly set 2020-08-25T16:36:13.1993540Z » expected 308, 2020-08-25T16:36:13.1993847Z got 331 (strictEqual) // /home/runner/work/phantomas/phantomas/node_modules/vows/lib/assert/macros.js:14 2020-08-25T16:36:13.1994331Z ✓ should have "jsCount" metric properly set 2020-08-25T16:36:13.1994747Z ✗ should have "jsSize" metric properly set 2020-08-25T16:36:13.1995187Z » expected 29730, 2020-08-25T16:36:13.1995501Z got 29753 (strictEqual) // /home/runner/work/phantomas/phantomas/node_modules/vows/lib/assert/macros.js:14 2020-08-25T16:36:13.1995951Z ✓ should have "domains" metric properly set 2020-08-25T16:36:13.1996353Z ✓ should have "DOMqueries" metric properly set 2020-08-25T16:36:13.1996775Z ✓ should have "DOMqueriesById" metric properly set 2020-08-25T16:36:13.1997250Z ✓ should have "DOMqueriesByClassName" metric properly set 2020-08-25T16:36:13.1997703Z ✓ should have "DOMqueriesByTagName" metric properly set 2020-08-25T16:36:13.1998115Z ✓ should have "DOMqueriesByQuerySelectorAll" metric properly set 2020-08-25T16:36:13.1998540Z ✓ should have "DOMinserts" metric properly set 2020-08-25T16:36:13.1999263Z ✓ should have "DOMqueriesDuplicated" metric properly set 2020-08-25T16:36:13.1999925Z ✓ should have "DOMqueriesAvoidable" metric properly set 2020-08-25T16:36:13.2000376Z ✓ should have "DOMqueriesWithoutResults" metric properly set 2020-08-25T16:36:13.2001575Z ✗ should have "gzipRequests" offender(s) properly set 2020-08-25T16:36:13.2002051Z » Expected values to be strictly deep-equal: 2020-08-25T16:36:13.2002452Z + actual - expected 2020-08-25T16:36:13.2002638Z 2020-08-25T16:36:13.2002835Z [ 2020-08-25T16:36:13.2002965Z { 2020-08-25T16:36:13.2003163Z bodySize: 84245, 2020-08-25T16:36:13.2003369Z + transferedSize: 29753, 2020-08-25T16:36:13.2003755Z - transferedSize: 29730, 2020-08-25T16:36:13.2011928Z url: 'http://127.0.0.1:8888/static/jquery-2.1.1.min.js' 2020-08-25T16:36:13.2012192Z } 2020-08-25T16:36:13.2012846Z ] // /home/runner/work/phantomas/phantomas/test/integration-test.js:83 2020-08-25T16:36:13.2013362Z ✗ should have "htmlCount" offender(s) properly set 2020-08-25T16:36:13.2013782Z » Expected values to be strictly deep-equal: 2020-08-25T16:36:13.2014172Z + actual - expected 2020-08-25T16:36:13.2014363Z 2020-08-25T16:36:13.2014535Z [ 2020-08-25T16:36:13.2014707Z { 2020-08-25T16:36:13.2015260Z + size: 2408, 2020-08-25T16:36:13.2015720Z - size: 2385, 2020-08-25T16:36:13.2016127Z url: 'http://127.0.0.1:8888/dom-operations.html' 2020-08-25T16:36:13.2016401Z } 2020-08-25T16:36:13.2016891Z ] // /home/runner/work/phantomas/phantomas/test/integration-test.js:83 2020-08-25T16:36:13.2017326Z ✗ should have "cssCount" offender(s) properly set 2020-08-25T16:36:13.2017744Z » Expected values to be strictly deep-equal: 2020-08-25T16:36:13.2018088Z + actual - expected 2020-08-25T16:36:13.2018264Z 2020-08-25T16:36:13.2018435Z [ 2020-08-25T16:36:13.2018562Z { 2020-08-25T16:36:13.2018747Z + size: 331, 2020-08-25T16:36:13.2019170Z - size: 308, 2020-08-25T16:36:13.2019920Z url: 'http://127.0.0.1:8888/static/style.css' 2020-08-25T16:36:13.2020317Z } 2020-08-25T16:36:13.2020834Z ] // /home/runner/work/phantomas/phantomas/test/integration-test.js:83 2020-08-25T16:36:13.2021265Z ✗ should have "jsCount" offender(s) properly set 2020-08-25T16:36:13.2021686Z » Expected values to be strictly deep-equal: 2020-08-25T16:36:13.2021985Z + actual - expected 2020-08-25T16:36:13.2022172Z 2020-08-25T16:36:13.2022349Z [ 2020-08-25T16:36:13.2022592Z { 2020-08-25T16:36:13.2023011Z + size: 29753, 2020-08-25T16:36:13.2023523Z - size: 29730, 2020-08-25T16:36:13.2024898Z url: 'http://127.0.0.1:8888/static/jquery-2.1.1.min.js' 2020-08-25T16:36:13.2025160Z } 2020-08-25T16:36:13.2025612Z ] // /home/runner/work/phantomas/phantomas/test/integration-test.js:83

macbre · 2020-08-25T17:14:24Z

Locally:

  phantomas:modules:requestsMonitor headers: {"url":"http://127.0.0.1:8888/static/jquery-2.1.1.min.js","status":200,"statusText":"OK","headers":{"Date":"Tue, 25 Aug 2020 17:09:50 GMT","Content-Encoding":"gzip","last-modified":"Fri, 07 Aug 2020 18:10:57 GMT","server":"ecstatic-3.3.2","etag":"W/\"6294735-29415-2020-08-07T18:10:57.644Z\"","content-type":"application/javascript","cache-control":"max-age=84600","Connection":"keep-alive","content-length":"29415"},"mimeType":"application/javascript","connectionReused":false,"connectionId":24,"remoteIPAddress":"127.0.0.1","remotePort":8888,"fromDiskCache":false,"fromServiceWorker":false,"fromPrefetchCache":false,"encodedDataLength":29730,"timing":{"requestTime":22747.397588,"proxyStart":-1,"proxyEnd":-1,"dnsStart":0.273,"dnsEnd":0.279,"connectStart":0.279,"connectEnd":0.355,"sslStart":-1,"sslEnd":-1,"workerStart":-1,"workerReady":-1,"workerFetchStart":-1,"workerRespondWithSettled":-1,"sendStart":0.447,"sendEnd":0.482,"pushStart":0,"pushEnd":0,"receiveHeadersEnd":3.875},"responseTime":1598375390516.801,"protocol":"http/1.1","securityState":"secure","_requestId":"78763.3","dataLength":84245,"chunks":2,"_timestamp":22747.402286} +6ms
  phantomas:modules:requestsMonitor Response compressed with gzip, 82.2705078125 kB -> 29.033203125 kB (x2.8336696939118737) +0ms
  phantomas:modules:requestsMonitor recv: HTTP 200 <http://127.0.0.1:8888/static/jquery-2.1.1.min.js> [application/javascript] +0ms
Response: GET http://127.0.0.1:8888/static/jquery-2.1.1.min.js [application/javascript]
  phantomas:modules:requestsMonitor Response metadata: {"id":"78763.3","url":"http://127.0.0.1:8888/static/jquery-2.1.1.min.js","method":"GET","headers":{"date":"Tue, 25 Aug 2020 17:09:50 GMT","content-encoding":"gzip","last-modified":"Fri, 07 Aug 2020 18:10:57 GMT","server":"ecstatic-3.3.2","etag":"W/\"6294735-29415-2020-08-07T18:10:57.644Z\"","content-type":"application/javascript","cache-control":"max-age=84600","connection":"keep-alive","content-length":"29415"},"bodySize":84245,"transferedSize":29730,"responseSize":29730,"type":"js","protocol":"http","domain":"127.0.0.1","query":null,"stalled":0.447,"timeToFirstByte":3.393,"timeToLastByte":0.017317000001639826,"headersSize":296,"gzip":true,"contentType":"application/javascript","isJS":true,"status":200,"statusText":"OK"} +1ms

GitHub:

2020-08-25T17:09:59.0705102Z 2020-08-25T17:09:59.069Z phantomas:modules:requestsMonitor headers: {"url":"http://127.0.0.1:8888/static/jquery-2.1.1.min.js","status":200,"statusText":"OK","headers":{"Date":"Tue, 25 Aug 2020 17:09:59 GMT","Content-Encoding":"gzip","last-modified":"Tue, 25 Aug 2020 17:09:46 GMT","server":"ecstatic-3.3.2","etag":"W/\"4134010-29415-2020-08-25T17:09:46.309Z\"","content-type":"application/javascript","cache-control":"max-age=84600","Connection":"keep-alive","Keep-Alive":"timeout=5","content-length":"29415"},"mimeType":"application/javascript","connectionReused":false,"connectionId":24,"remoteIPAddress":"127.0.0.1","remotePort":8888,"fromDiskCache":false,"fromServiceWorker":false,"fromPrefetchCache":false,"encodedDataLength":29753,"timing":{"requestTime":64.370311,"proxyStart":-1,"proxyEnd":-1,"dnsStart":0.194,"dnsEnd":0.2,"connectStart":0.2,"connectEnd":0.286,"sslStart":-1,"sslEnd":-1,"workerStart":-1,"workerReady":-1,"workerFetchStart":-1,"workerRespondWithSettled":-1,"sendStart":0.318,"sendEnd":0.364,"pushStart":0,"pushEnd":0,"receiveHeadersEnd":9.356},"responseTime":1598375399057.765,"protocol":"http/1.1","securityState":"secure","_requestId":"2525.3","dataLength":84245,"chunks":2,"_timestamp":64.380629}
2020-08-25T17:09:59.0711937Z 2020-08-25T17:09:59.070Z phantomas:modules:requestsMonitor Response compressed with gzip, 82.2705078125 kB -> 29.0556640625 kB (x2.8314791785702282)
2020-08-25T17:09:59.0716623Z 2020-08-25T17:09:59.071Z phantomas:modules:requestsMonitor recv: HTTP 200 <http://127.0.0.1:8888/static/jquery-2.1.1.min.js> [application/javascript]
2020-08-25T17:09:59.0719035Z Response: GET http://127.0.0.1:8888/static/jquery-2.1.1.min.js [application/javascript]
2020-08-25T17:09:59.0725661Z 2020-08-25T17:09:59.072Z phantomas:modules:requestsMonitor Response metadata: {"id":"2525.3","url":"http://127.0.0.1:8888/static/jquery-2.1.1.min.js","method":"GET","headers":{"date":"Tue, 25 Aug 2020 17:09:59 GMT","content-encoding":"gzip","last-modified":"Tue, 25 Aug 2020 17:09:46 GMT","server":"ecstatic-3.3.2","etag":"W/\"4134010-29415-2020-08-25T17:09:46.309Z\"","content-type":"application/javascript","cache-control":"max-age=84600","connection":"keep-alive","keep-alive":"timeout=5","content-length":"29415"},"bodySize":84245,"transferedSize":29753,"responseSize":29753,"type":"js","protocol":"http","domain":"127.0.0.1","query":null,"stalled":0.318,"timeToFirstByte":8.991999999999999,"timeToLastByte":0.015207000000003745,"headersSize":319,"gzip":true,"contentType":"application/javascript","isJS":true,"status":200,"statusText":"OK"}

encodedDataLength (29730 vs 29753) and headersSize (296 vs 319) differ (by 23 bytes)
content-length matches (29415)

There's an additional Keep-Alive: timeout=5 header sent on GitHub (that's 21 bytes + two bytes for carriage return - \r\n). And only when using Node.js 14.x - nodejs/node#34561

Node 14.x adds "Keep-Alive: timeout=5" response header making response sizes less deterministic.

…s failing" This reverts commit d21b814.

And run curl to check it

macbre added this to the v2.0 - headless Chromium milestone Aug 25, 2020

macbre added bug Chrome-headless Tasks related with migration from PhantomJS to Puppeteer's headless Chrome labels Aug 25, 2020

macbre added 3 commits August 25, 2020 20:22

.github/workflows/tests.yml: run test HTTP server using Node.js 12.x

6c9a117

Node 14.x adds "Keep-Alive: timeout=5" response header making response sizes less deterministic.

Dockerfile: use Node.js 12.x base image

5eafd90

tests.yml: wait a second or two before running the tests

fca5b54

macbre marked this pull request as ready for review August 25, 2020 18:27

macbre added 2 commits August 25, 2020 20:28

Revert "Run an example script to resolve issues with assets size test…

55123dc

…s failing" This reverts commit d21b814.

Wait a bit longer for test HTTP server to start

e74a27d

And run curl to check it

macbre changed the title ~~Run an example script to resolve issues with assets size tests failing~~ Node.js 14.x integration test are failing on response size asserts Aug 25, 2020

macbre changed the title ~~Node.js 14.x integration test are failing on response size asserts~~ Node.js 14.x integration tests are failing on response size asserts Aug 25, 2020

macbre merged commit c504460 into devel Aug 25, 2020

macbre deleted the assets-sizes-issue branch August 25, 2020 18:35

macbre mentioned this pull request Sep 1, 2020

Use nginx to make integration tests more deterministic #773

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Node.js 14.x integration tests are failing on response size asserts #769

Node.js 14.x integration tests are failing on response size asserts #769

Uh oh!

macbre commented Aug 25, 2020 •

edited

Loading

Uh oh!

macbre commented Aug 25, 2020 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Node.js 14.x integration tests are failing on response size asserts #769

Node.js 14.x integration tests are failing on response size asserts #769

Uh oh!

Conversation

macbre commented Aug 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

macbre commented Aug 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

macbre commented Aug 25, 2020 •

edited

Loading

macbre commented Aug 25, 2020 •

edited

Loading