In Pagespeed insights, I get the following message in Origin Summary: "Over the previous 28-day collection period, the aggregate experience of all pages served from this origin does not pass the Core Web Vitals assessment."
screenshot of the message in PageSpeed Insights
Does anyone know what % of URLs have to pass the test in order to change this? Or what the criteria is?
Explanation
Lets use Largest Contentful Paint (LCP) as an example.
Firstly, the pass / fail is not based on the percentage of URLs, it is based on the average time / score.
This is an important distinction as you could have 50% of the data fail, but if it only fails by 0.1s (2.6s) and the other 50% of data is passing by 1 second (1.5s) the average will be a pass (average of 2.05s which is a pass).
Obviously this is an over-simplified example but you hopefully get the idea that you could have 50% of your site in the red and still pass in theory, which is why the percentages in each category are more for diagnostics.
If the average time for LCP across all pages in the CrUX dataset is less than 2.5 seconds ("Good") then you will get a green score and that is a pass.
If the time is less than 4 seconds the score will be orange ("Needs improvement") but this will still count as a fail.
Over 4 seconds and it fails and will be red ("Poor").
Passing criteria
So you need the following to be true to pass the web vitals (at time of writing):-
Largest Contentful Paint (LCP) average is less than 2.5 seconds
First Input Delay (FID) is less than 100ms
Cumulative Layout Shift is less than 0.1
If any one of those is over the threshold you will fail, even if the other two are within the green / passes.
FID - when running lighthouse (or Page Speed Insights) on a page you do not get the FID as part of the synthetic test (Lab Data).
Instead you get Total Blocking Time (TBT) - this is a close enough approximation for FID in most circumstances so use that (or run a performance trace).
Related
I am noticing different TTBF values in Chrome network tab vs logged by WebVitals. Ideally it should be exactly same value, but sometimes seeing large difference as much as 2-3 seconds for certain scenarios.
I am using Next.js and using reportWebVitals to log respective performance metrics.
Here is a sample repo, app url and screenshots for reference.
Using performance.timing.responseStart - performance.timing.requestStart is returning more appropriate value than relying on WebVitals TTFB value.
Any idea what could be going wrong? Is is a bug on WebVitals and I shouldn't be using it or mistake at my end in consuming/logging the values?
The number provided by reportWebVitals (and the underlying library web-vitals) is generally considered the correct TTFB in the web performance community (though to be fair, there are some differences in implementation across tools).
I believe DevTools labels that smaller number "Waiting (TTFB)" as an informal hint to the user what that "waiting" is to give it context and because it usually is the large majority of the TTFB time.
However, from a user-centric perspective, time-to-first-byte should really include all the time from when the user starts navigating to a page to when the server responds with the first byte of that page--which will include time for DNS resolution, connection negotiation, redirects (if any), etc. DevTools does include at least some information about that extra time in that screenshot, just separated into various periods above the ostensible TTFB number (see the "Queueing", "Stalled", and "Request Sent" entries).
Generally the Resource Timing spec can be used as the source of truth for talking about web performance. It places time 0 as the start of navigation:
Throughout this work, all time values are measured in milliseconds since the start of navigation of the document [HR-TIME-2]. For example, the start of navigation of the document occurs at time 0.
And then defines responseStart as
The time immediately after the user agent's HTTP parser receives the first byte of the response
So performance.timing.responseStart - performance.timing.navigationStart by itself is the browser's measure of TTFB (or performance.getEntriesByType('navigation')[0].responseStart in the newer Navigation Timing Level 2 API), and that's the number web-vitals uses for TTFB as well.
Recently, after the lighthouse update, significant differences in values appeared in the site measurement, with successive measurements of the same project, the differences reached 30 points, for example, before the update, these differences had a maximum of 3 points.
We are positioning ourselves as a company that gives high pointspeed in pagespeed as an advantage, but for any differences, as a result, we cannot rely on numbers. We want to understand:
Why does such a difference in the resulting values appear?
How can this difference be reduced?
Could this be due to the incorrect operation of your algorithms?
The measurements were carried out on our project in production, which had constant values > 90 points before the update.
Check this.
I doubt this is a CLS (Cumulative Layout Shift) issue. Your CLS numbers are fine.
The variance in Page Speed Insights could be due to your CDN - Content Distribution Network. First-time tests usually are poor as your site gets loaded on CDN cache. Subsequent tests will be faster. Always take the median score from 4-5 runs.
You can also try and disable CDN temporarily. Platforms such as Cloudflare offer developer testing mode to disable CDN temporarily. Try that. You will get more consistent results.
I am trying to understand the "Graph Result" in JMeter I got when I followed the following scenario:
I am hitting www.google.com with:
No. of users: 10,
Ramp up Period is 5 Seconds,
Loop count is 10
I am finding it difficult to read "Graph Result", I have used another listeners too (View Results in Tree, View Results in Table, Summary Report) which are easy to understand but I would like to learn this too.
Please refer the result Image link:
https://www.cubbyusercontent.com/pli/Image.png/_2855385a0bbb40a0b7cd2d31224b521c
Help appreciated.
According to JMeter Help,
Graph results contains,
The Graph Results listener generates a simple graph that plots all
sample times. Along the bottom of the graph, the current sample
(black), the current average of all samples(blue), the current
standard deviation (red), and the current throughput rate (green) are
displayed in milliseconds.
The throughput number represents the actual number of requests/minute
the server handled. This calculation includes any delays you added to
your test and JMeter's own internal processing time.
Basically it shows data,average,median,deviation,throughput i.e.system statistics during test in a graphical format.
These values are plotted runtime thus it updates values at bottom at runtime i.e. total no. of samples are no. of samples occurred till that point of time with deviation at that point of time and similarly other counters represent their values.
Due to its runtime behavior, this listener consumes lot of memory and cpu and it is advised that it should not be used while load test (I think you have used it just to know its use and working.)
While running actual load test you can learn/understand these statistics from aggregate report other reports which can be created in non-ui mode also.
I hope this have cleared what graph result shows and how to read it and when to use it.
I am using Visual Studio TS Load Test for running WebTest (one client/controls hitting one server). How can I configure goal based load pattern to achieve a constant test / second?
I tried to use the counter 'Tests/Sec' under 'LoadTest:Test' but it does not seem to do anything.
I've recently tested against the Tests / Sec, and I've confirmed it working.
For the settings on the Goal Based Load Pattern, I used:
Category: LoadTest:Test
Counter: Tests/Sec
Instance: _Total
When the load test starts, verify it doesn't show an error re: not being able to access that Performance Counter.
Tests I ran for my own needs:
Set Initial User Load quite low (10), and gave it 5 minutes to see if
it would reach the target Tests / Sec target, and stabilise. In my case, it stabilised after about 1 minute 40.
Set the Maximum User Count [Increment|Decrement] to 50. Turns out the
user load would yo-yo up and down quite heavily, as it would keep
trying to play catch-up. (As the tests took 10-20 seconds each)
Set the Initial User Load quite close to the 'answer' from test 1,
and watched it make small but constant adjustments to the user
volume.
Note: When watching the stats, watch the value under "Last". I believe the "Average" is averaged over a relatively long period, and may appear out of step with the target.
Is there a formula that will tell me the max/average # of concurrent users I would expect to have with a population of 1000 users a day using an app for 10 minutes?
1000 users X 10 minutes = 10,000 user minutes
10,000 user minutes / 1440 minutes in a day = 6.944 average # of concurrent users
If you are looking for better estimates of concurrent users, I would suggest putting google analytics on your site. It would give you an accurate reading of highs, lows, and averages for your site.
In the worst case scenario, all 1000 users use the app at the same time, so max # oc concurrent users is 1000.
1000 users * 10 minutes = 10000 total minutes
One day has 24 hours * 60 minutes = 1440 minutes.
Assume normal distribution, you would expect 10000 / 1440 = 6.9 users on average using your app concurrently. However, normal distribution is not a valid assumption since you probably won't expect a lot of users in the middle of a night.
Well, assuming there was a steady arrival pattern, people visited the site with an equal distribution, you would have:
( Users * Visit Length in Minutes) / (Minutes in a Day)
or
(1000 * 10 ) / 1440
Which would be about 7 concurrent users
Of course, you would never have an equal distribution like that. You can take a stab at the anticipated pattern, and distribute the load based on that pattern. Best bet would be to monitor the traffic for a bit, with a decent sampling of user traffic.
That depends quite a bit on their usage pattern and no generic formula can cover it.
As an extreme example, if this is a timecard application, you would have a large peak at the start and stop of each work day, with scattered access between start and stop as people work on different projects, and almost no access outside of working hours.
Can you describe the usage pattern you expect?
Not accurately.
Will the usage be spread evenly over the day or are there events that will cause everyone to use their 10 mintes at the same time?
For example lets compare a general purpose web site with a time card entry system.
General purpose web site will have ebbs and flows throughout the day with clusters of time when you have lots of users (during work hours...).
The time card entry system might have all 1000 people hitting the system within a 15 minute span of time twice a day.
Simple math can show you an average, but people don't generally behave "on average"...
As is universally said, the answer is dependent upon the distribution of when people "arrive." If it's equally likely for someone to arrive at 3:23 AM as at 9:01 AM, max_concurrent is low; if everyone checks in between 8:55 AM and 9:30 AM, max_concurrent is high (and if response time slows depending on current load, so that the 'average' time on the site goes up significantly when there are lots of people on the site...).
A model is only as good as its inputs, but having said that, if you have a good sense of usage patterns, a Monte Carlo model might be a good idea. (If you have access to a person with a background in statistics or probability, they can do the math just based on the distribution parameters, but a Monte Carlo model is easier for most people to create and manipulate.)
In comments, you say that it's a "self-serve reference app similar to Wikipedia," but your relatively low usage means that you cannot rely on the power of large numbers to "dampen out" your arrival curves over 24-hours.