All Posts By

Wisam Abdulaziz

Core Web Vitals Website Speed

How To Get Your Website Ready For Page Experience Core Web Vitals Update

Google has been tweaking their algorithm in the last few years toward user experience elements such as:

  • Website speed
  • Mobile friendliness
  • Secure connection HTTPs

Early 2020 Google Chrome team announced Core Web Vitals, following that Google announced that Core Web Vitals will be added as ranking factors to their algorithm along with the old user experience elements, in what the SEOs call page experience update.

Core Web Vitals measure real world user experience dimensions of web usability, such as load time, interactivity, and the stability of content as it loads. The three core web vitals announced by Google for 2020 are:

  1. Loading: Largest Contentful Paint (LCP)
  2. Interactivity: First Input Delay (FID)
  3. Visual Stability: Cumulative Layout Shift (CLS)

Due to COVID-19 Google will delay launching the new Google algorithm that includes Core Web Vitals (page experience update), and will provide at least six months notice before it rolls out.

How to evaluate Core Web Vitals for a website:

There are many tools that can help with that:

It is very important to understand the difference between lap data and field data before start optimizing for Core Web Vitals.

From Google: "The field data is a historical report about how a particular URL has performed, and represents anonymized performance data from users in the real-world on a variety of devices and network conditions. The lab data is based on a simulated load of a page on a single device and fixed set of network conditions. As a result, the values may differ."

 Field DataOriginal SummaryLap Data
Internet Connection speedDifferent connection speed for real-world usersDifferent connection speed for real-world usersLatency: 150ms Throughput: 1.6Mbps down / 750 Kbps up.    
Machine HardwareDifferent CPU, RAM, GPU, depending or real-world usersDifferent CPU, RAM, GPU, depending or real-world usersMid-Tier Mobile (4X slowdown of a high tier desktop)      
Results timeAverage last 28 daysAverage last 28 daysCurrent
LocationReal-world user’ locationReal-world user’ locationCalifornia
Result forTested URLWhole DomainTested URL

1- Loading - Largest Contentful Paint - LCP

"Largest Contentful Paint (LCP): measures loading performance. To provide a good user experience, LCP should occur within 2.5 seconds of when the page first starts loading."

The Largest Contentful Paint (LCP) metric reports the render time of the largest image or text block visible within the viewport (hero element). Website speed used to be monitored using metrics like Speed Index or First Contentful Paint FCP, those metrics are still important to monitor website speed, but the new focus should be improving LCP.

The best tool to measure and optimize for LCP is Google Chrome Developers Tools, the timestamp number represents LCP, the highlighted area marks the hero element that LCP is calculated when it finishes loading.

It is very important to run LCP test using different internet speed modes, and CPU speed modes, to better understand LCP value for different users based on their internet connection and machine power. Same thing applies to other Core Web Vitals and performance metrics.

blank
blank

How to optimize for LCP?

Optimizing for LCP is optimizing for speed, check my how to optimize for LCP post. A good start will be checking the recommendations provided by Lighthouse:

blank

2- First Input Delay (FID)

"FID measures the time from when a user first interacts with a page (i.e. when they click a link, tap on a button, or use a custom, JavaScript-powered control) to the time when the browser is actually able to begin processing event handlers in response to that interaction."

Not all users will interact with your site every time they visit and the timing of the interaction is different, and not all interactions are relevant to FID. This means some users will have no FID values, some users will have low FID values, and some users will probably have high FID values.

FID is a metric that can only be measured in the field, as it requires a real user to interact with your page. You can measure FID only using Google Speed Insights tool, which is using Chrome User Experience Database, read here to better understand the difference between lab data and filed data. The good news that there is a lab metric (Total Blocking Time) that can help to diagnose and improve FID.

The Total Blocking Time (TBT) metric measures the total amount of time between First Contentful Paint (FCP) and Time to Interactive (TTI) where the main thread was blocked for long enough to prevent input responsiveness. No task should exceed 50 MS, Chrome Developers Tools provides total blocking time along with the tasks that are exceeding the 50 MS mark:

blank

How to optimize for FID:

Optimizing for LCP includes lots of common elements that will improve FID, most common causes of high FID that needs to be worked on are:

  • Unnecessary JavaScript loading, parsing, or execution. While analyzing your code in the Performance panel you might discover that the main thread is doing work that isn't really necessary to load the page. Reducing JavaScript payloads with code splittingremoving unused code, or efficiently loading third-party JavaScript should improve your TBT score.
  • Inefficient JavaScript statements. For example, after analyzing your code in the Performance panel, suppose you see a call to document.querySelectorAll('a') that returns 2000 nodes. Refactoring your code to use a more specific selector that only returns 10 nodes should improve your TBT score.

3- Cumulative Layout Shift (CLS):

"CLS measures visual stability. Have you ever been reading an article online when something suddenly changes on the page? Without warning, the text moves, and you've lost your place. Or even worse: you're about to tap a link or a button, but in the instant before your finger lands—BOOM—the link moves, and you end up clicking something else!"

This could be the easiest Core Web Vital to diagnose, by looking at the thumbnails tracker in Chrome Developers Tools and enabling the Layout Shift Regions it should be easy to find out what is causing a high CLS:

blank

How to optimize for CLS?

CLS diagnosis and optimization could be the easiest among all Core Web Vitals, in most cases CLS issues are visible to the human eye, with some CSS and JS work, creating stability while the page is loaded should be achievable. Check the experience section in Chrome Developers Tools, and find which elements are moving and see if you can:

  • Make them stable on the page
  • Reserve a place holder for them from the beginning wile waiting for them to load.
blank

CLS is measured for the entire life cycle of the page (even if it is kept opened for days by users), for that reason the actual user experience value could be different form Lighthouse or Chorome Developers Tools (where they measure only one trace).

The Marathon Of Good Core Web Vitals score:

Keeping a good Core Web Vitals score is a marathon that needs constant website improvements, there are too many variables that can change the score that used to be a pass before:

  • Website technology changes (hosting and CMS) and content changes.
  • Users technology changes (the type of machines and internet connection they use)
  • Google changing their Core Web Vitals criteria and scoring

Monitoring the Core Web Vitals section in Google Search Console will be key to find any Core Web Vitals issues and start working on them as soon as they are detected.

Core Web Vitals improvement is not only a search engine best practice, it is also a user experience element that can lead to better a conversion rate.

Accessibility News

How To Comply With Website Accessibility Standards In Ontario WCAG

blank

New public websites and significantly refreshed websites were required to be compliant with WCAG 2.0 Level A by January 1, 2014.

Furthermore, by January 1, 2021, all public websites and web content posted after January 1, 2012, must meet WCAG 2.0 Level AA.

All large (50+ employees) private and non-profit organizations, and all public-sector organizations are required to make their websites accessible.

I am not sure what is this going to mean from a legal standpoint and what actions the government will take against non-complying websites, since 2014 1000s of websites were built and refreshed without fully complying with WCAG 2.0 A and I did not hear of any penalties or legal issues.

I have contacted the government several times in the last few months trying to learn more about the tools they will be using to audit websites, and what are the penalties, but I did not get any answers.

Options to comply with WCAG

1- Check with your web developer

Most web developers started to design websites with accessibility in mind, check with your web developer if they have the capability of making the whole website compliant with WCAG 2.0 AA, expect to pay additional cost for that.

If you are redesigning/refreshing your website ask your web developer to include WCAG compliance in the SOW.

2- Request an audit and pass it to your web developer for implementation

There are many audit providers including AODA

Audit providers do not offer implementation in most cases, the implementation must be done by the web developer, which means more cost, you will be paying the audit provider to conduct the audit + the web developer to implement the recommendations in the audit.

The other issue with audits is that they rarely cover the whole website, they will take a sample of pages (home page and other random pages) and use them in the audit.

3- Automated third-party accessibility services

There are many automated accessibility providers that can make websites compatible with WCAG 2.0 AA. There will be a monthly fee to use those services, and the cost will rely on the size of the website (number of pages, and the traffic in some cases), for a small website expect to pay $29+/month to use those services:

Please note that automated solutions may not offer a 100% compliance guarantee, but they will be still far better than leaving the website without any accessibility optimization (in case you do not have the resources to do it).

4- Automated accessibility software/plugins

There are many automated accessibility software that can make websites compatible with WCAG 2.0 AA (especially if you are using WordPress)

Cons and Pros of every method:

1- Web Dev A-Z2- Audit + Web Dev3- Service4- Software
CostHighVery highLowLowest
100% WCAG 2.0 AA complianceGuaranteedGuaranteedNo guaranteedNo guarantee
ImplementationSlow and difficultVery slow and difficultQuick and easyQuick and easy
LongevityMust be updated when major changes in content or website happenMust be updated when major changes in content or website happenNo future updates requiredSoftware updates will be required (it could be automated sometimes)

Tools to check accessibility compliance:

There are many tools that can help you to test your websites compliance with WCAG, I will include 3 of them which happen to have a Google Chrome extension (which normally makes things easier):

I also recommend using screen readers or screen reader simulators (this is what people with impaired vision use to browse the web):

Using screen readers will help you to discover accessibility issues that can not be covered by accessibility tools.

Core Web Vitals

How to Optimize For Largest Contentful Paint (LCP)

LCP (largest contentful paint) is one of the most important core web vitals, it is the only core web vital that is related 100% to speed and hence it is the most difficult one to diagnose and optimize for. LCP could be measured using Google PageSpeed Insights which is powered by Lighthouse.

LCP should load within 2.5 seconds, if 75% of the loads ( regardless of which pages) on a website achieve that number, LCP will be marked as passing assessment. Only elements within the user viewport (above the fold) will be used to calculate LCP.

blank

Google has provided a great tutorial how to optimize for LCP, in this post I will try to provide a check list and actionable directions that you can implement yourself or take to your web developer that is working on improving Core Web Vitals (mainly LCP).

14 improvements that you can do to improve LCP:

  1. Adequate resources on the server
  2. Enable browser caching
  3. Enable GZIP (text compression)
  4. Enable server caching like OPcache or reverse proxy
  5. Keep your software up-to-date (CMS, Plugins, operating system like Ubuntu, control panel like Cpanel, PHP, MySQL and Apache)
  6. Install a server side HTML caching for your CMS
  7. Setup browser side caching using service worker
  8. Minify JavaScript and CSS
  9. Compress images and use the right format for them
  10. Resize images to fit the required dimensions in the style sheet
  11. Remove or defer files and codes that block critical rendering path for above the fold content
  12. Inline critical CSS and critical JavaScript files where possible
  13. Lazy load images below the fold
  14. Use CDN for resources like images and JavaScript files or for the whole website

I will provide more details how to improve each of the items above, most of my examples will work best for WordPress that is hosted on Linux with Apache being the web server.

Adequate resources on the server:

You do not need to be cheap when it comes to web hosting, having more resources than what your website needs is better than lacking resources. Start with a VPS, possibly 2 cores, 4 GB of ram and SSD, and keep eye on your resources usage, your average CPU load must stay one or below and your RAM usage must be less than 50% on average. Add more resources if you did not achieve those numbers.

Enable browser caching:

This setup could be done in most cases using the server's configuration, Apache has mod_expires module that enables you to do that by adding the code below to your .htaccess file:

<FilesMatch "\.(webm|ogg|mp4|ico|pdf|flv|jpg|jpeg|png|gif|webp|js|css|swf|x-html|css|xml|js|woff|woff2|otf|ttf|svg|eot)(\.gz)?$">
<IfModule mod_expires.c>
AddType application/font-woff2 .woff2
AddType application/x-font-opentype .otf
ExpiresActive On
ExpiresDefault A0
ExpiresByType video/webm A10368000
ExpiresByType video/ogg A10368000
ExpiresByType video/mp4 A10368000
ExpiresByType image/webp A10368000
ExpiresByType image/gif A10368000
ExpiresByType image/png A10368000
ExpiresByType image/jpg A10368000
ExpiresByType image/jpeg A10368000
ExpiresByType image/ico A10368000
ExpiresByType image/svg+xml A10368000
ExpiresByType text/css A10368000
ExpiresByType text/javascript A10368000
ExpiresByType application/javascript A10368000
ExpiresByType application/x-javascript A10368000
ExpiresByType application/font-woff2 A10368000
ExpiresByType application/x-font-opentype A10368000
ExpiresByType application/x-font-truetype A10368000
</IfModule>
<IfModule mod_headers.c>
Header set Expires "max-age=A10368000, public"
Header unset ETag
Header set Connection keep-alive
FileETag None
</IfModule>
</FilesMatch>

Enable GZIP (text compression):

This setup could be done in most cases using the server configuration, Apache has mod_deflate module that enables you to do that by adding the code below to your .htaccess file:

<IfModule mod_deflate.c>
AddType x-font/woff .woff
AddType x-font/ttf .ttf
AddOutputFilterByType DEFLATE image/svg+xml
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/x-font-ttf
AddOutputFilterByType DEFLATE x-font/ttf
AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
AddOutputFilterByType DEFLATE font/opentype font/ttf font/eot font/otf
</IfModule>

Enable server caching like OPcache or reverse proxy:

Server caching will save popular pages in the memory of the server so they could be served quickly to users, PHP has OPcache that could be enabled to do that.

Keep your software up-to-date:

There is a lot of technology involved in running a website:

  • The server and its software (Operating system like Linux, web server like Apache, programming language like PHP, database engine like MySQL, control panel like Cpanel and more)
  • CMS, like WordPress.
  • CMS add-ons like Plugins.

Keeping all this software up-to-date is vital for both performance and security.

Install a server side HTML caching for your CMS:

For a CMS like WordPress to generate a page there will be few PHP calls to the database followed by a HTML version of the page passed by the web server (Apache) to the browser to parse it then render it, this is a long journey with too many variables. Caching a copy of every single page of the website in a HTML format will save a lot for work required by the server and enable it to pass a static HTML file to the browser for rendering, this is a big time saver. A plugin like WPfastest cache can take care of that for WordPress.

Setup browser side caching using service worker:

The Service Worker API comes with a Cache interface, that lets you create stores of responses keyed by request. Service Worker can even enable the website to work totally off-line after first load.

Minify JavaScript and CSS:

Spaces and comments can inflate CSS and JavaScript files significantly, minifying those files manually or using a plugin if you have WordPress will improve performance. For WordPress WPcache can do the minification.

Compress images and use the right format for them:

Images have different format and purposes, JPEG, PNG, GIFs, SVG, and each of them also has multiple format like JPEG 2000, choosing the right format and resolution could be a huge size saver. For WordPress WPsuper cache plugin has an image compression module that makes image compression an easy process.

Consider using WEBP, It can reduce your image size by 30% comparing to JPEG or PNG.

Resize images to fit the required dimensions in the style sheet:

The image dimensions should always match the required space on the page, having a 1000X1000 pixel images in 100X100 pixel space will add a lot of unnecessary bytes to the total page size.

Remove or defer files and codes that block critical rendering path for above the fold content:

Files that are blocking the critical rendering path could be checked using the coverage tab in Chrome Developer Tools:

Coverage section chrome developer tools

Files that are not used for critical rendering must be deferred. If you have multiple JavaScript files that are partially used for the critical rendering path, you can combine them in two files, one that has all the codes that are required for the critical rendering path critical.js/critial.css and another one that includes codes that are not required for the critical rendering path non-critical.js/non-critial.css, the second file must be deferred.

Consider also preloading files that are required for critical rendering path:

<head>
<link rel="preload" as="script" href="critical.js">

<link rel="preload" href="critical.woff2" as="font" type="font/woff2" crossorigin>

<link rel="preload" href="critical.css" as="style">

<link rel="preload" href="bg-image-narrow.png" as="image" media="(max-width: 600px)">

</head>

You can defer JavaScript and CSS files using the codes below:

<head>
<script src="demo_async.js" async></script>

<link rel="preload" href="styles.css" as="style" onload="this.onload=null;this.rel='stylesheet'">
<noscript><link rel="stylesheet" href="styles.css"></noscript>
</head>

You can also consider inlining CSS and JS codes that are required for the critical rendering path (read more below).

Inline critical CSS and critical JavaScript files where possible:

Web page rendering will be blocked by the browser until all included resources like JS and CSS are fully loaded, having CSS or JS codes that are required for critical rendering path in files will add an extra step to the rendering process which is requesting those file from the server (extra requests to the server). Inlining codes that are required for critical rendering path and put the rest in deferred files will improve website speed.

Lazy load images below the fold:

Images below the fold contributes to page load time but they are not seen by users during the initial page load, lazy loading those images will reduce load time without negatively affecting user experience.

WordPress 5.4 will natively support lazy load, for other CMS you can use a lazy load library or you can use the browser native lazy load attribution:

<img src="image.png" loading="lazy" alt="…" width="200" height="200">

Use CDN for resources like images and JavaScript files or for the whole website:

CDN has a lot of features that can help with speed:

  • Provide files to users from data centres close to their physical location which improves website speed
  • Reduces server load as most files will be served from the CDN's provider servers (normally powerful servers with load balance)
  • Firewall with DDOS protection
  • Caching content in static HTML (server side caching)

For WordPress Jetpack plugin can provide images and JS files caching, you can also use providers like Cloudeflare which providers a full website caching.

Be aware that optimization for speed is not one and done process, there is always something to improve. Keep monitoring your Core Web Vitals using Google Search Console and make sure your website is having good score.

Core Web Vitals Website Speed

Lab data VS field data VS Origin Summary Core Web Vitals

PageSpeed insights along with Google Search Console are the best tools when comes to assess a website performance against Core Web Vitals (LCP largest contentful paint, FID first input delay and CLS cumulative layout shift). However the terminology used in those tools could be sometimes confusing to users.

One of the most popular questions when it comes to Core Web Vitals assessment is what is the difference between lap data, field data and origin summary, in this post I will try to answer this question.

core web vitals

Lab data:

Things you need to know about lab data:

  • Lab data is empowered by LightHouse technology that simulates mobile throttling with a lower speed. Read more here.
  • Lighthouse uses in most cases a slower CPU than the average CPU available for users "Lighthouse applies CPU throttling to emulate a mid-tier mobile device even when run on far more powerful desktop hardware" .
  • Lighthouse in most cases will use one location (USA).
  • Lab data is generated only for the tested URL in Google PageSpeed Insights tool.
  • Lab data is a live data, it reflects the speed at the time the test was run.
  • Some users interaction elements like FID will not be visitable in the lab data.

When you run Lighthouse in Google Chrome Developer Tools, at the end of the report you will be able to see the settings that are used in that experiment:

lighthouse test

Looking at the settings above we can easily see how Lighthouse is using a slower internet connection and slowing the CPU power by 5 times. The reasons Lighthouse does that is to get the test to cover all online users that use different type of computers and connect to the internet using different levels of speed.

Field data:

Field data is generate by actual Google Chrome everyday users, with:

  • Different computers/phones (different resources like CPU, RAM and GPU).
  • Different internet speed.
  • Different locations.
  • Field data is generated only for the URL that is tested in PageSpeed Insights .
  • Field data is based on Chrome User Experience Report (CrUX), so it is not live data, it the average user data collected in the last 28 days for the tested URL.
  • Google uses the 75th percentile value of all page views to the tested page to produce the score, if at least 75 percent of page views to the tested page meet "good" threshold, the page is classified as having "good" performance for that metric.

So if most of your users coming from a population with a high internet speed and powerful devices, it is normal to see field data better than lab data.

On the other hand if your server is overloaded at the time you are running the test, it is normal to see lab data recording higher numbers than field data.

Original summary:

This is very similar to field data but it represents the average performance of all pages on your website/domain.

Google uses the 75th percentile value of all page views to that site/domain to produce the score, if at least 75 percent of page views to the tested site meet "good" threshold, the site is classified as having "good" performance for that metric.

blank

Which data set to care mostly about?

Google will be using the field data (CrUX) and original summary data to judge your pages/website, which is also data that is available in Google Search Console.

All what you need to monitor in the PageSpeed Insights report is the original summary data, which is the date that will be used by Google algorithm updates in the future. It is ok to use lab data while working on speed improvements as you need an instant feed back.

 Field DataOriginal SummaryLap Data
Internet Connection speedDifferent connection speed for real-world usersDifferent connection speed for real-world usersLatency: 150ms Throughput: 1.6Mbps down / 750 Kbps up.    
Machine HardwareDifferent CPU, RAM, GPU, depending or real-world usersDifferent CPU, RAM, GPU, depending or real-world usersMid-Tier Mobile (4X slowdown of a high tier desktop)      
Results timeAverage last 28 daysAverage last 28 daysCurrent
LocationReal-world user’ locationReal-world user’ locationCalifornia
Result forTested URLWhole DomainTested URL
Tracking & Analytics

How To Measure The Impact Of a TV Campaign on Web Traffic

There are a lot of trackable footprints that could be included in a TV ad, some of them are almost automatically included in the ad (like the brand name) others need to be added:

  1. The brand name, most TV ads will mention the brand name in a sound or tex format.
  2. A trackable dedicated URL or a domain. If your domain is xyz.com you can include in the ad using a URL like xyz.com/tv or use a different domain like xyzpromo.com.
  3. A trackable phone number that is dedicated only for the TV ad.
  4. A unique product name,  brand name + a product name or any statement that can trigger web search (example: cranberry sprite )

The more trackable footprints you have in the TV ad the easier it is for you to track the performance of this ad.

Metrics and tools that can be used to track a TV campaign:

  1. An Increase in search for the brand name  (trackable using Google Search Console).
  2. A significant Increase in search for the product name “i.e. cranberry sprite”  (trackable using GSC).
  3. A CTR improvement. The TV ad will keep the brand at the top of people’s mind so they tend to click more on any page produced by that brand even if it is a blog (trackable using GSC).
  4. An improvement in conversion rate for all products as a result of the brand enhancement (trackable in Google Analytics).
  5. A very significant improvement in sales and conversion rate for the promoted product in the TV campaign (i.e. cranberry sprite).
  6. Increase in brand’s equity (difficult to measure)
  7. More social buzz and inbound links as a result of people talking about the promoted product online (could be measured by tracking social signals and inbound links).

It is very important to discuss trackable footprints in the TV ad with the marketing team to come up with the best tracking options without jeopardizing the marketing part of the ad, after that a custom dashboard could be built using Google Data Studio could be built for that.

Google Tag Manager Tracking & Analytics

Using GTM to Track Shopify Conversions with Facebook and Bing

Shopify has a very nice integration with Google Analytics e-commerce tracking, all what it takes to enable it, is adding the property ID to Shopify and checking the enhanced e-commerce box, then going to Google Analytics to enable e-commerce tracking.

Integrating other codes like Bing or Facebook conversion code is not as easy, you can rely on the thank you page to trigger the pixels for those platforms but that will not be enough to provide very valuable data like the value of the transaction. Thankfully Shopify pushes all transaction variables to the source code of the thank you page using a JavaScript variable Shopify.checkout that has a JSON format:

blank

That means you can add any value of that transaction as a JavaScript variable in Google Tag Manager, something like below:

blank

You can read more about the available checkout variables here

Once you have this variable available in GTM you can use it in the sales tracking trigger (when the value is > 0 + thank you page):

blank

You can also use its value in any pixel tracking code like Bing:

blank
SEO

How to Forecast Natural Organic Traffic Growth

Forecasting organic SEO traffic to fit in a digital marketing plan is a very difficult task to do, there are many variables controlling organic traffic and many of them are not under our control. Let us go through the most popular forces behind organic traffic growth and try to evaluate their impact on our forecast:

Population and internet user growth of the target country:

Most countries grow their population by a certain percentage annually, here in Canada we grow by 1.4% annually, so if all things equal organic traffic should grow by this percentage. Internet user growth (internet penetration) can contribute also to organic traffic growth (please note that there is some overlap between population growth and internet user growth), in Canada our internet users grew by 1.5% last year, for most advanced countries this number will be low, but it could be much higher for developing countries like India where it reached 18% last year.

Do some research and some estimation to come up with a number that you feel comfortable with, if this number is 2.9% and the population of the target country is 1,000,000 then you will get 40,000 extra visits next year through population and internet user growth.

Popularity growth (brand awareness) :

Brand popularity can grow in many different ways:

  • Off-line advertisement (e.g. old media like TV, radio and newspapers).
  • Online advertisement (e.g. Google and Facebook ads).
  • Word of mouth (great products can generate a lot of word of mouth popularity).

First time advertisers in traditional media like TV can expect a large increase in their branded traffic (you can track that number in Google Search Console by monitoring branded keywords impressions and clicks), if branded traffic is 10,000/month a good ad campaign can take it to 20,000/month and that is a 100% YOY increase in branded search, the annual traffic growth as a result of that = (branded traffic grow/last year total organic traffic), let us assume this number is 2%

Please note that an increase in brand awareness can also improve the overall organic traffic for all keywords as people tend to click more on brands that they know in the SERPs (higher CTR is expected in this case).

Authority growth  (link profile):

Link profile's strength is the most important ranking factor with all major search engines, if the website you are working on has:

  • A weak link profile.
  • Large number of quality unique pages.

Then you know that a with a good link building campaign you can possibly double this traffic, on the other hand if you are working on website with a strong link profile, a good link building campaign will have a smaller impact (in the single digit range at best).

Please note that brand popularity growth should help with link growth as more people will be talking about the brand on social media websites, forums and blogs.

I am going to assume that link building can provide a 10% YOY traffic increase.

Content growth:

Websites grow their content by adding more pages on a regular basis, it is very difficult to grow marketing pages (pages where products and services are sold) on a high rate, that is where a blog can be a good tool to keep a high content growth. First you need to check the average traffic on posts that are 1+ month old that were published in the previous year, then multiple that number with the number of posts you are planning to publish next year to get the traffic gain forecast.

Content enhancement:

There are three type of content that you need to optimize and enhance on a regular basis:

  • Pages that are losing traffic YOY, this content could be expiring/outdated content that needs to be refreshed or it is content that is getting a lot of competition and needs enhancement.
  • Content that is not getting any organic traction (could be thin content or low quality content), this content needs to be enhanced otherwise removed or redirected.
  • Content that ranks at the bottom of the first page or at the top of the second page in the SERPs, with some extra work this content can gain few more spots in ranking which can make a big difference in the amount of traffic it can generate.

It will be very difficult to estimate the amount of traffic growth that content enhancement will generate without going back to historical data and calculate the growth, so for this one you need to wait for year 2 to come up with the right amount, for year one you can:

  • Assume you are going to restore the lost traffic for pages that are losing traffic.
  • Use search volume on the keywords you are optimizing for and assume you can double their CTR.

The tally:

Once you have the percentages or numbers from the four items above just add them up to find the organic traffic increase forecast for next year.

Disclaimers you need to include in your forecast when you submit it to your manager for review:

Make sure your manager understands that this is an estimate (educated guess) and there are many variables that can throw this plan off like:

  • Popularity decline as a result of a brand crisis.
  • Algorithmic update (Google does multiple updates every year that can affect rankings positively or negatively).
  • SERPs layout update (Google keeps changing SERPs above the fold area by adding featured snippets or more ads which can negatively affect CTR and traffic).
  • Technical issues (the website can go down for a while or the technical team can make a mistake that can cause traffic loss like blocking Google bots form crawling the website).
  • Production delays (executing the SEO plan is not guaranteed especially in a tight resource situation).

Another disclaimer that you need to include,  the website will go throw a traffic loss situation in a case your SEO plan is not approved. The risk of stopping or slowing SEO efforts could result in:

  • Website/content stagnation which is a bad signal to send to search engines and it can cause reduction in authority (if you are the authority in your space you are expected to produce and refresh content on a regular basis).
  • Expiring/outdated content (e.g. legal related content will lose ranking YOY if stayed unchanged).
Link Building

How to Evaluate the Quality of a Link Quickly Without DA or PA

If you are starting a link building campaign the first question jumps to your mind is how to evaluate the quality and eventually the impact of each link on your ranking, the right evaluation will help you to decide how much efforts is it worth putting to acquire that link.

Back in the day Google used to provide a metric called PageRank (a metric that reflects your website popularity on the web based on the number of quality links pointing back to it), it was publicly available to everyone and ranges from 0 to 10, obviously 10 refers to a very strong website which makes it high quality link for SEOs.

blank

PageRank used to be available as a green bar in Google's tool bar, Google stopped supporting that feature in 2016 leaving SEOs running in the dark looking for quick ways to evaluate the quality of their links. Many companies found an opportunity in that and started crawling the web to gather link data to sell it as a service.

On a side note, Pagerank is still part of Google's ranking algorithm but it is not available anymore for the public.

PageRank alternatives that came to fill the gap for the hidden Google's Pagerank:

Since Google stopped showing page rank publicly many third party link crawlers started offering their own version of page rank:

  • MOZ offers DA (domain authority) and PA (page authority) which become very popular metrics among link traders.

blank

  • Majestic's link quality metrics are TF (trust flow) and CF (citation flow)

blank

  • Ahrefs link quality metrics are DR (domain rating) and UR (URL rating)

blank

  • SEMrush's link quality metrics are Domains Score, Trust Score and Authority Score

blank

But the big question that faced link builders is can I trust those metrics? the metrics above work well for highly trusted websites like cnn.com but they are easy to manipulate by spammer, there are many websites with a DA of 50 but yet they are not even indexed by Google, so what is really the best Pagerank alternative?

The best page rank alternative which is even better that page rank is organic traffic data:

The answer is simply organic traffic and the number of keywords the domain is ranking for in Google, if Google is happy with a website's:

  • Content
  • Link profile
  • Authority
  • Technical elements like website speed

They will simply rank it for more keywords and send it more traffic, and thankfully this data is available for any domain using the same tools that are scoring the link profile (SEOmoz, SEMrush and Ahrefs), my favorite tool for that is SEMrush:

blank

The analysis above shows that this domain gets 209 desktop visits/month and ranks for 527 keywords, not really that good but better than a DA 100 domains with 0 traffic.

Interestingly enough link suppliers like TheHoth finally got that point and started offering links based on traffic (I am not encouraging buying links here):

blank

You can see that links with minimum high traffic (1000 visits/month) worth twice as much as domains with minimum DA which is 10.

Summary:

Link metrics that are not offered by Google are good to use for some competition analysis but when it comes to link acquisition, traffic and number of organic keywords must be the two metrics that should be used to evaluate the value of that link.

 

 

 

 

 

Hosting Technical SEO

How To Block link Cralwers like Majestic Ahrefs Moz and SEMRush

The web has a lot of web crawlers, some of them are good and vital for your website such as Google bot, others can be harmful like email harvesting crawlers and content scrapers. Link crawlers come short of harmful but far from useful. They are not useful for your website, and they are not harmful in way they try to scrape content or anything like that, but they could be consuming your server resources with no benefit.

For SEOs that adopt black hat tactics like PBN (private blog network) those crawlers are a nightmare and can expose the network to competitors if left open, which in most cases will lead to a spam report causing the whole network to be de-indexed + a manual action applied to the money site if not a total deindexation.

The most popular link crawlers are Majestic, Ahrefs, Moz and SEMRush, please note that their crawlers user-agents will not match their brand name and can change in the future, so it is very important to keep an up-to-date list with the user-agents used by those crawlers. I will list below different ways to block them:

Robots.txt:

You add few lines to your robots.txt file that can disallow most popular link crawlers:

User-agent: Rogerbot
User-agent: Exabot
User-agent: MJ12bot
User-agent: Dotbot
User-agent: Gigabot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SemrushBot-SA
Disallow: /

The method above will be very effective assuming:

  • You trust those crawler to obey the directions in the robots.txt file.
  • The crawlers do no keep changing their user-agent's names.
  • The companies that operate those crawlers do not use third party crawling services that come under different user-agents.

.htaccess:

The issue with this method is that it requires your hosting provider to be Apache based, if your host supports htaccess you can use the code below to block most popular link crawlers:

<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} (ahrefsbot|mj12bot|rogerbot|exabot|dotbot|gigabot|semrush) [NC]
RewriteRule .* - [F,L]
</IfModule>

This method is better that robots.txt as the crawlers have no choice but to obey assuming they are not changing their user-agents, or using third party crawlers.

Using PHP:

If you website is built with PHP like WordPress you can add the code below to your header.php to block all link crawlers:

$badAgents = array('rogerbot','mj12bot', 'ahrefsbot', 'semrush', 'dotbot', 'gigabot', 'archive.org_bot');
foreach ($badAgents as $blacklist) {
if (preg_match("/$blacklist/", strtolower($_SERVER['HTTP_USER_AGENT'])) ) {
exit();
} }

This methods is good if your server doesn't support .htaccess , if you are using this method you need to make sure you block also the RSS feed feature in WordPress, you can do that by adding the code below to your function.php file in the theme folder:

function wpb_disable_feed() {
wp_die( __('No feed available,please visit our <a href="'. get_bloginfo('url') .'">homepage</a>!') );
}
add_action('do_feed_xml', 'wpb_disable_feed', 1);
add_action('do_feed', 'wpb_disable_feed', 1);
add_action('do_feed_rdf', 'wpb_disable_feed', 1);
add_action('do_feed_rss', 'wpb_disable_feed', 1);
add_action('do_feed_rss2', 'wpb_disable_feed', 1);
add_action('do_feed_atom', 'wpb_disable_feed', 1);
add_action('do_feed_rss2_comments', 'wpb_disable_feed', 1);
add_action('do_feed_atom_comments', 'wpb_disable_feed', 1);

Aggressive blocking (for PBN users):

If you are a regular webmaster that it is willing to save some server resources by blocking link crawlers, applying any of the methods above should be suffice; however, if you are a webmaster that wants to leave no chance for those crawlers to sneak in, you need to apply harsher measurements.

Robots.txt will be as below:

User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /

This will allow only Google bot to crawl the website assuming the crawlers will obey robots.txt directions. You can also allow other agents used by major search engines like Bing.

If you are using Wodrepss you can hide the links from all user-agents excluding Google using the code below in functions.php:

add_filter( 'the_content', 'link_remove_filter' );
function link_remove_filter( $content ) {
if (!preg_match("/google/", strtolower($_SERVER['HTTP_USER_AGENT'])) && !preg_match('/\.googlebot|google\.com$/i', gethostbyaddr($_SERVER['REMOTE_ADDR'])) ) {
$content = preg_replace('#<a.*?>(.*?)</a>#is', '\1', $content);
}
return $content;
}

This code will allow only Google to see the links, it verifies also that the IP address belongs to Google and it is not faked.

Make sure also to block RSS using the code listed in the previous step, the code above will not be impacted by those crawlers changing their agents or coming with different agent's names.

 

Tracking & Analytics

Integrating Google Ads Lead Form With CRMs

Google has announced the rollout of Lead Form Extensions that enable advertisers to capture form submissions directly from the ad without sending users to a landing page, the submissions will be stored in Google Ads' database and will be downloadable as a CSV file, there is also an option to integrate with CRMs using a webhook, this howGoogle explained the integration:

A webhook is an API that enables you to send lead data to your CRM system in real-time. To set it up, you will need to add a webhook URL and key to your lead form extension. You may need to do some configuration within your CRM system to generate the URL and key.

The URL is the delivery path: after the user submits a lead form, an HTTP POST request is sent to the configured URL, allowing the lead data to go directly into the CRM system. The key is used for validating the leads sent.

The explanation above is confusing for marketers and at least unclear for developers. What is the webhook URL and where to find it? This program (Google ads lead form extension) is in Beta now, and I am not sure if there is any CRM that supports it at the moment, even Salesforce doesn't have a webhook URL for it yet.

How to generate a webhook for the lead form extensions?

The best resource that can help with that is the developer guide provided by Google, anytime the form is submitted to Google a JSON POST request will be sent to the webhook URL added in the form lead extensions, something like below:

{
"lead_id": "lead_id1",
"form_id" : "form_id1",
"user_column_data":[ {"column_name":"Full Name","string_value":"John Doe"},
{"column_name":"User Phone", "string_value":"12345678"},
{"column_name":"User Email", "string_value":"abc@123.com"}],
"api_version":"1.0",
"google_key" : "secret"
}

Firs you need to decide what is the best way to capture this data, for me I am going to use PHP to do that, my webhook could be something like this https://www.wisamabdulaziz.com/webhook.php  with the code below:

$json = file_get_contents('php://input');
$form_data = json_decode($json);
$leadid = $form_data->lead_id ;
$form_id = $form_data->form_id ;
$fullname = $form_data->user_column_data[0]->string_value;
$phone = $form_data->user_column_data[1]->string_value;
$email = $form_data->user_column_data[2]->string_value;

At this point all the values are available in the code and ready to be pushed to any CRM or a local database, most popular CRMs like SalesForce and Hubspot have APIs with PHP libraries (they have also libraries available for the most popular programming languages) making it easy to push the data to those CRMs.

Additional values like campaign and keyword could be available also in the fields and worth saving. Before pushing any values to your CRM make sure to create fields to match all the values you will be saving.