Guides

Integration Guide

In this guide, you’ll learn how to use Middesk’s Website order to receive a detailed analysis of a business’s web presence.

Suggest Edits

Before getting started

  • Contact Sales to enable our improved Web Analysis product on your account
  • Ensure that you can authenticate against the Middesk API and create businesses

Step 1: Create a business and order Web Analysis

Web Analysis is a subproduct of the Website order. To create a business with a website order, you’ll need to submit the following Create Business request:

require 'uri'
require 'net/http'

url = URI("https://api.middesk.com/v1/businesses")

http = Net::HTTP.new(url.host, url.port)

request = Net::HTTP::Post.new(url)
request["Content-Type"] = 'application/json'
request["Authorization"] = '<Your Auth Token>'
request.body = "{\n  \"name\": \"Joe's Bakery\",\n  \"addresses\": [\n\t  {\n\t\t  \"full_address\": \"123 Main Street, Tampa, FL 33626\"\n\t  }\n  ],\n  \"website\": {\n\t  \"url\": \"www.joesbakery.com\"\n  },\n\t\"orders\": [\n\t\t{\n\t\t  \"product\": \"website\",\n\t\t  \"subproducts\": [\"web_analysis\"]\n\t\t}\n\t]\n}"

response = http.request(request)
puts response.read_body

curl --request POST \
  --url https://api.middesk.com/v1/businesses \
  --header 'Authorization: <Your Auth Token>' \
  --header 'Content-Type: application/json' \
  --data '{
  "name": "Joe'\''s Bakery",
  "addresses": [
	  {
		  "full_address": "123 Main Street, Tampa, FL 33626"
	  }
  ],
  "website": {
	  "url": "www.joesbakery.com"
  },
	"orders": [
		{
		  "product": "website",
		  "subproducts": ["web_analysis"]
		}
	]
}'

import http.client

conn = http.client.HTTPConnection("https://api.middesk.com")

payload = "{\n  \"name\": \"Joe's Bakery\",\n  \"addresses\": [\n\t  {\n\t\t  \"full_address\": \"123 Main Street, Tampa, FL 33626\"\n\t  }\n  ],\n  \"website\": {\n\t  \"url\": \"www.joesbakery.com\"\n  },\n\t\"orders\": [\n\t\t{\n\t\t  \"product\": \"website\",\n\t\t  \"subproducts\": [\"web_analysis\"]\n\t\t}\n\t]\n}"

headers = {
    'Content-Type': "application/json",
    'Authorization': "<Your Auth Token>"
    }

conn.request("POST", "/v1/businesses", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

const options = {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    Authorization: '<Your Auth Token>'
  },
  body: '{"name":"Joe\'s Bakery","addresses":[{"full_address":"123 Main Street, Tampa, FL 33626"}],"website":{"url":"www.joesbakery.com"},"orders":[{"product":"website","subproducts":["web_analysis"]}]}'
};

fetch('https://api.middesk.com/v1/businesses', options)
  .then(response => response.json())
  .then(response => console.log(response))
  .catch(err => console.error(err));

Note that the name, address, and website url are all required for Web Analysis.


Step 2: Analyze the outcome

Once your order completes, you can see the results in the GET /business payload or the business.updated webhook, depending on your integration. For a high level summary of the outcome, you can look at the following review tasks:

SummaryKey
Is the website purchased but has no content?website_parked
Is the website online?website_status
Was there a match identified to the submitted office address?web_address_verification
Was there a match identified to the submitted business name?web_business_name_verification
Was there a match identified to the submitted person?web_person_verification
Was there a match identified to the submitted phone number?web_phone_number_verification
Was there a match identified to the submitted email address?web_email_address_verification
What is the web presence quality rating?web_presence_quality

If you’d like to see the results in more detail, you can look at the Website object in the payload as well as the specific Web Analysis Review Tasks in depth.

Additionally, any third party profiles associated with the business will be returned here as Profile objects under the top level profiles key. This includes platforms such as Google, Facebook, LinkedIn, and more.

You can check the business's review tasks by requesting the full business payload using the Retrieve Business Endpoint. For example, you can use the Website Status Review Task to evaluate whether website is online.

require 'net/http'
require 'net/https'
require 'json'

def send_request
	business_id = "<business_id>"
  uri = URI("https://api.middesk.com/v1/businesses/#{business_id}")

  # Create client
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true
  http.verify_mode = OpenSSL::SSL::VERIFY_PEER

  # Create Request
  req =  Net::HTTP::Get.new(uri)
  # Add headers
  req.add_field "Accept", "application/json"
  # Add headers
  req.add_field "Authorization", "Basic <api_key>"

  # Fetch Request
  res = http.request(req)
  
	response = JSON.parse(res.body)
	website_status_task = response['review']['tasks'].find { |task| task['key'] == 'website_status' }
	
	if website_status_task['status'] == 'success'
		puts 'Website online'
	else
    puts 'Website offline'
	end
rescue StandardError => e
  puts "HTTP Request failed (#{e.message})"
end

import requests
from requests.auth import HTTPBasicAuth

def send_request():
    business_id = "<business_id>"
    url = f"https://api.middesk.com/v1/businesses/{business_id}"

    headers = {
        "Accept": "application/json",
        "Authorization": "Basic <api_key>"
    }

    try:
        # Make the request
        response = requests.get(url, headers=headers, auth=HTTPBasicAuth('<api_key>', ''))
        
        # Parse the response JSON
        response_data = response.json()
        website_status_task = next((task for task in response_data['review']['tasks'] if task['key'] == 'website_status'), None)

        if website_status_task and website_status_task['status'] == 'success':
            print('Website online')
        else:
            print('Website offline')

    except requests.exceptions.RequestException as e:
        print(f"HTTP Request failed ({e})")

send_request()

  const businessId = "<business_id>";
  const url = `https://api.middesk.com/v1/businesses/${businessId}`;

  const headers = {
    "Accept": "application/json",
    "Authorization": "Basic <api_key>"
  };

  try {
    // Make the request using fetch
    const response = await fetch(url, { headers });
    
    // Check if the request was successful
    if (!response.ok) {
      throw new Error(`HTTP error! Status: ${response.status}`);
    }

    const responseData = await response.json();
    const websiteStatusTask = responseData.review.tasks.find(task => task.key === 'website_status');

    if (websiteStatusTask && websiteStatusTask.status === 'success') {
      console.log('Website online');
    } else {
      console.log('Website offline');
    }
  } catch (error) {
    console.error(`HTTP Request failed: ${error.message}`);
  }
}

sendRequest();

Step 3: Inspect individual quality indicators

You can leverage web_presence_qualityto determine an overall quality rating of "High", "Moderate", "Low", or "Not Available" of the business's web presence. Going one step further, we can inspect each quality indicator used to determine the overall quality rating and optionally, derive an outcome based on your own defined heuristics.

📘

Indicator Introspection

One example of a potential use case for directly inspecting the indicators is in the case of evaluating new businesses. Naturally a newly formed business may have a recently registered domain and possibly is perfectly okay for the site to not display a high amount of content diversity, given its recent inception. If that is the case, you may wish to explicitly ignore those ratings to determine your desired outcome.

Here's a look at the current quality indicators and their respective keys within the website object. Each indicator includes a rating field of its own, yielding a "positive", "neutral", or "negative" rating for each category type. If the indicator cannot be accurately determined for any reason, the indicator will absent from the payload.

Quality IndicatorDescriptionKey
Broken linksAn evaluation of links found on the website leading to errors or nonexistent pagesbroken_links
Compliance infoAn evaluation of compliance-related information, such as terms of service, privacy policy, etc. can be found on the business's websitecompliance_info
Contact infoAn evaluation of contact information found within the content. You can leverage the other review tasks associated with the Websiteorder to determine matches with submitted contact information.contact_info
Content diversityAn evaluation of the variation and type of information the website presentscontent_diversity
Domain ageAn evaluation of how long as the domain been registered fordomain_age
Filler textAn evaluation of content for any placeholder or filler text, eg. Lorem Ipsumfiller_text
HttpsAn evaluation of HTTPS/TLS supporthttps
Image qualityAn evaluation of images to identify blurry or stock imagesimage_quality
Last updatedAn evaluation of how long ago was the wesbite last modifiedlast_updated
Page countAn evaluation of the total number of internal pages found on the websitepage_count
Spelling and grammarAn evaluation of spelling and grammatical mistakes and inconsistenciesspelling_and_grammar
Top Level DomainAn evaluation of the TLD with respect to the website's qualitytop_level_domain
Update frequencyAn evaluation of how often the website's content is updatedupdate_frequency
US Business PresenceAn evaluation of website signals that suggest the business operates within the United Statesus_business_presence

You can retrieve these additional website details via the https://api.middesk.com/businesses/{id}/website endpoint. Here's an example iterating and printing out the sample business's website's quality indicator ratings:

require 'net/http'
require 'uri'
require 'json'
require 'base64'

def print_indicators_quality_ratings
  # Replace with your actual business ID and API key
  business_id = "51c4b91e-f324-467b-86b5-9e0155bcc251"  # Example business ID
  api_key = "YOUR_API_KEY_HERE"  # Replace with your actual API key

  uri = URI.parse("https://api.middesk.com/v1/businesses/#{business_id}/website")

  # Create the HTTP Basic Auth header
  auth = Base64.strict_encode64("#{api_key}:")
  headers = {
    "Accept" => "application/json",
    "Authorization" => "Basic #{auth}"
  }

  # Create the HTTP request
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true
  request = Net::HTTP::Get.new(uri.request_uri, headers)

  begin
    # Make the request
    response = http.request(request)

    # Raise an error if the response is not successful
    unless response.is_a?(Net::HTTPSuccess)
      puts "HTTP Request failed (#{response.code} #{response.message})"
      return
    end

    # Parse the JSON response
    data = JSON.parse(response.body)

    # Navigate to the indicators
    indicators = data.dig('rating', 'indicators') || []

    if indicators.empty?
      puts "No indicators found in the response."
      return
    end

    # Print each indicator's name and rating
    puts "Indicators' Quality Ratings:"
    puts "----------------------------"
    indicators.each do |indicator|
      name = indicator['name'] || 'Unknown Indicator'
      rating = indicator['rating'] || 'No Rating'
      description = indicator['description'] || ''
      puts "#{name}: #{rating} (#{description})"
    end

  rescue JSON::ParserError
    puts "Error parsing JSON response."
  rescue StandardError => e
    puts "An error occurred: #{e.message}"
  end
end

print_indicators_quality_ratings

# Sample Output:
# Indicators' Quality Ratings:
# ----------------------------
# Domain age: positive (3 to 10 years old)
# Compliance info: positive (Found)
# Spelling and grammar: positive (Good)
# Contact info: positive (Found)
# Content diversity: positive (High)
# Https: positive (Enabled)
# Broken links: negative (Some broken)
# Filler text: positive (Minimal)
# Page count: positive (High)
# Image quality: positive (Good)


import requests
from requests.auth import HTTPBasicAuth

def print_indicators_quality_ratings():
    # Replace with your actual business ID and API key
    business_id = "51c4b91e-f324-467b-86b5-9e0155bcc251"  # Example business ID
    api_key = "YOUR_API_KEY_HERE"  # Replace with your actual API key

    url = f"https://api.middesk.com/v1/businesses/{business_id}/website"

    headers = {
        "Accept": "application/json"
    }

    try:
        # Make the GET request with HTTP Basic Authentication
        response = requests.get(url, headers=headers, auth=HTTPBasicAuth(api_key, ''))

        # Raise an exception if the request was unsuccessful
        response.raise_for_status()

        # Parse the JSON response
        data = response.json()

        # Navigate to the indicators
        indicators = data.get('rating', {}).get('indicators', [])

        if not indicators:
            print("No indicators found in the response.")
            return

        # Print each indicator's name and rating
        print("Indicators' Quality Ratings:")
        print("----------------------------")
        for indicator in indicators:
            name = indicator.get('name', 'Unknown Indicator')
            rating = indicator.get('rating', 'No Rating')
            description = indicator.get('description', '')
            print(f"{name}: {rating} ({description})")

    except requests.exceptions.HTTPError as http_err:
        print(f"HTTP error occurred: {http_err}")  # e.g., 404 Not Found
    except requests.exceptions.RequestException as req_err:
        print(f"Request error occurred: {req_err}")  # Other request-related errors
    except ValueError:
        print("Error parsing JSON response.")
    except KeyError as key_err:
        print(f"Missing expected data in response: {key_err}")

if __name__ == "__main__":
    print_indicators_quality_ratings()

''' Sample output
Indicators' Quality Ratings:
----------------------------
Domain age: positive (3 to 10 years old)
Compliance info: positive (Found)
Spelling and grammar: positive (Good)
Contact info: positive (Found)
Content diversity: positive (High)
Https: positive (Enabled)
Broken links: negative (Some broken)
Filler text: positive (Minimal)
Page count: positive (High)
Image quality: positive (Good)
'''

const axios = require('axios');

async function printIndicatorsQualityRatings() {
  // Replace with your actual business ID and API key
  const businessId = "51c4b91e-f324-467b-86b5-9e0155bcc251"; // Example business ID
  const apiKey = "YOUR_API_KEY_HERE"; // Replace with your actual API key

  const url = `https://api.middesk.com/v1/businesses/${businessId}/website`;

  try {
    const response = await axios.get(url, {
      headers: {
        'Accept': 'application/json',
        'Authorization': `Basic ${Buffer.from(`${apiKey}:`).toString('base64')}`
      }
    });

    const data = response.data;

    // Navigate to the indicators
    const indicators = data.rating && data.rating.indicators ? data.rating.indicators : [];

    if (indicators.length === 0) {
      console.log("No indicators found in the response.");
      return;
    }

    // Print each indicator's name and rating
    console.log("Indicators' Quality Ratings:");
    console.log("----------------------------");
    indicators.forEach(indicator => {
      const name = indicator.name || 'Unknown Indicator';
      const rating = indicator.rating || 'No Rating';
      const description = indicator.description || '';
      console.log(`${name}: ${rating} (${description})`);
    });

  } catch (error) {
    if (error.response) {
      // Server responded with a status other than 2xx
      console.error(`HTTP error occurred: ${error.response.status} ${error.response.statusText}`);
    } else if (error.request) {
      // No response received
      console.error("No response received:", error.request);
    } else {
      // Other errors
      console.error("Error:", error.message);
    }
  }
}

printIndicatorsQualityRatings();

// Sample Output:
// Indicators' Quality Ratings:
// ----------------------------
// Domain age: positive (3 to 10 years old)
// Compliance info: positive (Found)
// Spelling and grammar: positive (Good)
// Contact info: positive (Found)
// Content diversity: positive (High)
// Https: positive (Enabled)
// Broken links: negative (Some broken)
// Filler text: positive (Minimal)
// Page count: positive (High)
// Image quality: positive (Good)

Looking for a deeper dive?

The API also exposes a source object for each indicator to illustrate how the indicator quality rating was derived. This includes a human-readable explanation and, when relevant, examples from the retrieved dataset respective to the indicator returned. For example, the image_quality indicator will return a list of source URLs for any stock images detected, the spelling_and_grammar indicator will list citations for any found mistakes or inconsistencies, and the broken_links indicator will return a list of URLs found on the website that do not resolve.

To access this expanded view, you simply need to add the query parameter includewith the value indicator_details to the aforementioned business website endpoint.

ie.https://api.middesk.com/businesses/{id}/website?include=indicator_details

eg. expanded quality rating indicator with new source key 

 {
                "type": "image_quality",
                "name": "Image quality",
                "rating": "negative",
                "source": {
                    "examples": [
                        {
                            "link": "https://images.unsplash.com/photo-1637684666451-423047d6bf5e?ixid=M3wzOTE5Mjl8MHwxfHNlYXJjaHw4fHxzdGFydHVwfGVufDB8fHx8MTcxNDg3ODI0Nnww\u0026ixlib=rb-4.0.3\u0026auto=format\u0026fit=crop\u0026w=1920",
                            "location": "https://example.com/contact",
                            "classification": "stock_image"
                        },
                        {
                            "link": "https://images.unsplash.com/photo-1588856122867-363b0aa7f598?ixid=M3wzOTE5Mjl8MHwxfHNlYXJjaHw1fHxzdGFydHVwfGVufDB8fHx8MTcxNDg3ODI0Nnww\u0026ixlib=rb-4.0.3\u0026auto=format\u0026fit=crop\u0026w=328\u0026h=332",
                            "location": "https://example.com/",
                            "classification": "stock_image"
                        },
                        {
                            "link": "https://images.unsplash.com/photo-1519389950473-47ba0277781c?ixid=M3wzOTE5Mjl8MHwxfHNlYXJjaHwyfHxzdGFydHVwfGVufDB8fHx8MTcxNDg3ODI0Nnww\u0026ixlib=rb-4.0.3\u0026auto=format\u0026fit=crop\u0026w=1224\u0026h=400",
                            "location": "https://example.com/",
                            "classification": "stock_image"
                        },
                        {
                            "link": "https://images.unsplash.com/photo-1456406644174-8ddd4cd52a06?ixid=M3wzOTE5Mjl8MHwxfHNlYXJjaHw2fHxzdGFydHVwfGVufDB8fHx8MTcxNDg3ODI0Nnww\u0026ixlib=rb-4.0.3\u0026auto=format\u0026fit=crop\u0026w=606\u0026h=304",
                            "location": "https://example.com/about",
                            "classification": "stock_image"
                        },
                        {
                            "link": "https://images.unsplash.com/photo-1553729459-efe14ef6055d?auto=format\u0026fit=crop\u0026w=328\u0026h=264",
                            "location": "https://example.com/company",
                            "classification": "stock_image"
                        }
                    ],
                    "explanation": "We found many stock images."
                },
                "value": "poor",
                "description": "Poor"
            },

Updated 2 days ago