Chromedp - how to get image file size and real dimensions - chromedp

Question

I am trying to retrieve some information like page load time, first paint but also the images and scripts that are being loaded and their sizes.
I am able to detect everything that is being loaded in terms of images and scripts but when I look at the sizes, they do not match the size which I see in my (Firefox) inspector.
Can anyone please explain to me what I am doing wrong?
I would also like to get to know how long it took to load the specific file.
I came up with this code. This shows me the correct name and mimetype, but as said, the file size is not correct.
chromedp.ListenTarget(ctx, func(ev interface{}) {
switch ev := ev.(type) {
case *network.EventResponseReceived:
eventResponseReceived = append(eventResponseReceived, network.EventResponseReceived{
RequestID: ev.RequestID,
LoaderID: ev.LoaderID,
Timestamp: ev.Timestamp,
Type: ev.Type,
Response: ev.Response,
HasExtraInfo: ev.HasExtraInfo,
FrameID: ev.FrameID,
})
case *network.EventLoadingFinished:
eventLoadingFinished = append(eventLoadingFinished, network.EventLoadingFinished{
RequestID: ev.RequestID,
Timestamp: ev.Timestamp,
EncodedDataLength: ev.EncodedDataLength,
ShouldReportCorbBlocking: ev.ShouldReportCorbBlocking,
})
}
})
for i := range eventResponseReceived {
for i2 := range eventLoadingFinished {
if eventResponseReceived[i].RequestID == eventLoadingFinished[i2].RequestID {
fmt.Println(eventResponseReceived[i].Response.URL)
fmt.Println(eventResponseReceived[i].Response.ResponseTime.Time())
fmt.Println(eventResponseReceived[i2].Response.EncodedDataLength)
fmt.Println(eventResponseReceived[i].Response.MimeType)
}
}
}
I found out that I, in some cases, can get the content-length. But for a lot of files the content-length unfortunately is nil.
eventResponseReceived[i2].Response.Headers["content-length"]
So for the files where no content-length was given, I need a solution.

Answer 1

IIUC, you want to measure the response size and the load time. In this case, I think you should check both the Network.responseReceived event and the Network.loadingFinished event. (The corresponding events in chromedp are network.EventResponseReceived and network.EventLoadingFinished).
I have attached an example request to the end of this answer. I will try to answer the question based on this example request.
Size
It's correct to get the size from the Network.loadingFinished event. Please note that this is not the file size. It's the total number of bytes received for this request. And it could be 0 if the response is loaded from a cache. The following fields from the response field of the Network.responseReceived event would tell you why encodedDataLength is 0 (an example event has been attached, read it first):
fromDiskCache: specifies that the request was served from the disk cache.
fromServiceWorker: specifies that the request was served from the ServiceWorker.
fromPrefetchCache: specifies that the request was served from the prefetch cache.
Don't try to read encodedDataLength from the response field. Because it's just the total number of bytes received for this request so far. There could be more data to receive (reported by the Network.dataReceived events, see the screenshot).
And don't try to read the size from the Content-Length header. It's not always provided, and it's not correct to use it to measure network payload.
Time
I will try to list the metrics in the timing tab (see the screenshot) based on my try and test. If not specified, the data points are from the response.timing field of the Network.responseReceived event.
Stalled: dnsStart. It a connection is reused, it could be sendStart.
DNS Lookup: DnsEnd - DnsStart.
Initial connection: connectEnd - connectStart
SSL: sslEnd - sslStart
Request sent: sendEnd - sendStart
Waiting for server response: receiveHeadersEnd - sendEnd
Content Download: timestamp (from the Network.loadingFinished event) - requestTime - receiveHeadersEnd.
I think you want to check the Waiting for server response metric and the Content Download metric most of the time.
Recommentation
Protocol Monitor is a must-have tool if you want to work with Chrome DevTools Protocol.
Example Request
See the screenshot below:
Here is the Network.responseReceived event (some fields are removed):
{
"requestId": "580832.38",
"timestamp": 40037.455632,
"type": "Image",
"response": {
"url": "https://i.ytimg.com/vi/noIxfDrKx_Q/maxresdefault.jpg",
"status": 200,
"mimeType": "image/jpeg",
"connectionReused": false,
"fromDiskCache": false,
"fromServiceWorker": false,
"fromPrefetchCache": false,
"encodedDataLength": 55,
"timing": {
"requestTime": 40036.989753,
"proxyStart": -1,
"proxyEnd": -1,
"dnsStart": 0.662,
"dnsEnd": 81.792,
"connectStart": 81.792,
"connectEnd": 332.482,
"sslStart": 158.76,
"sslEnd": 332.473,
"workerStart": -1,
"workerReady": -1,
"workerFetchStart": -1,
"workerRespondWithSettled": -1,
"sendStart": 332.906,
"sendEnd": 333.142,
"pushStart": 0,
"pushEnd": 0,
"receiveHeadersEnd": 459.986
},
"responseTime": 1668830468734.381
},
"frameId": "83262FAD65C24B78F2B7E6F884B2B146"
}
And here is the Network.loadingFinished event:
{
"requestId": "580832.38",
"timestamp": 40037.577265,
"encodedDataLength": 50766,
"shouldReportCorbBlocking": false
}