How to write a bot in 100 lines of code that:

• Fires a prebid

• Receives the winning bid: the advertisement

• Passes fraud detection

The digital advertising ecosystem is complex and volatile. Many companies come and go. In order to enable your bids to go to multiple different ad servers of your choice at once you are able to configure and download your unique prebid JavaScript. This is achieved by selectively include adapters and modules and generate your JavaScript on the push of a button.

Besides the prebid, taking care of the bids and responses, you need to configure element ids, tag ids, ad slot sizes, etc. The prebid will fire one or more bids in parallel and waits for their responses, or hits the configured timeout. Last step is to select the won advertisement for an ad-slot and subsequently display the advertisement in the slot.

The question is: Can this be achieved without a browser using a simple Python script?

Spoiler alert: No fraud fraud detection has been seen on prebid while writing and testing the code. If it were, why would a bid made by both the cURL request and the Python code successfully receive the response with a full payload, ie. the advertisement?

Prebid Introduction

Before advertisements are loaded and displayed on a website a bidding process is started. This is called the prebid. Prebid is a standard JavaScript with optional one or more modules by prebid.org. From a technical point of view prebid is a JavaScript that is generated by prebid.org dynamically where you can check one or more bidder adapters, analytic adapters, and modules [1] to be included in the file. That generated JavaScript is included on the website. The last missing piece is a dedicated JavaScript with publisher specific IDs, ad slot sizes, element ids, bid prices, etc. That script orchestrates the bids, won auctions, and finally renders the ads in the slots.

Before looking at a real implementation of prebidding let’s explain where the prebid.js JavaScript comes from. This JavaScript is generated at prebid.org. Before hitting the ‘Get Prebid.js!’ button you can select which modules, bidder adapters, analytics adapters you want to include in the JavaScript.

Now, to the real deal, a real implementation of prebid. The examples are from a local news website here in Amsterdam where I live. I originally started with wsj.com but that site has a too complex setup for a simple explainer article like this. Hence, I reverted to a local news site: www.at5.nl which has a much simpler structure. Figure 2 below show the publisher specific JavaScript defining the ad units and invoking the prebid.js, and if bids are won rendering the ads.

Figure 2 shows the definitions of the adUnits. In total 7 ad units are defined, see the documentation how bidder parameters should be defined [2]. Each ad unit has, an id which corresponds with the id of an element in the web page. The adapters contain the names of the adapter and an integer code next to it. For example, Appnexus, 32592020. This unique number is also used when communicating with the ad server.

Figure 3 shows the prebid.js JavaScript and the modules that are compiled into this file. This might sound weird at a first glance, but it isn’t once you know how prebid.org works. If you go to https://docs.prebid.org/download.html you are able to generate a prebid.js with the modules, adapters, etc. you want.

Figure 4 shows the details of the prebid request sent to Adnexus. The HTTP POST request needs to comply with the prebid standard which can be found at [3].

The request payload in Figure 5 contains four different bids on four tags. In this case only bid 2 will win, which can be seen in Figure 6. The returned answer will contain the id and uuid and some metadata.

Figure 6 shows a lot of data pertaining to the bid and the ad. The one that interest me is the “rtb”. This one contains the banner and trackers URLs. These URLs are generated at the ad server and thus contain the creative with its associated size, and a tracker URL with the correct ids. The data is a full HTML page with HTML tags, styles, and embedded JavaScripts. The data is escaped as it is a JSON string. Hence, the \” and \n characters in the string.

The publisher specific JavaScript will receive, unescape this string and will inject it into the ad-slot iframe. Once injected the HTML is parsed, the JavaScript executed and that means the ad is rendered. If the ad includes any references to verification vendors, then these JavaScripts are loaded as well.

This was the introduction to show what the normal flow is how a bid is placed, how such a bid looks, the answer that is returned upon a bod and what is done with the returned data. Now the question is: Can this be faked? Can these prebid requests be generated out of thin Python air? And should be done with the received answers?

Fake bid request using cURL

The very first test is to see whether the ad server gives a response when the prebid is fired from the command line using cURL. The request URL was copied from the browser using Copy as cURL, and pasted in the terminal window. The response is shown below the purple horizontal line, starting with {“version”:”3.0.0”.

The lines highlighted in yellow are ad verification lines. The content of the advertisement contains a link to DoubleVerify. The querystring parameter rtsurl= contains a link with a large payload. That’s why the total request spans multiple lines.

Fake bid request using Python script

Coping and pasting links from your browser doesn’t scale well. It’s manual and browsers use too much resources anyway, ie. memory and CPU. That’s why fraudsters prefer their operations to be: request based [5]. Although Python is not the most efficient language in terms of memory, execution speed, etc. it surely is much faster than any browser automation and it has a much lower memory footprint.

Figure 8 and Figure 9 show the source code of a Python script using the requests package to fire HTTP requests. The lines 12 to 30 are the initialization of variables. These will be used to dynamically generate the prebid payload at line 64 – 69. Lines 37 – 43 will load a publisher specific JavaScript that contains the IDs needed to include in the prebid. The JavaScript is downloaded using a HTTP GET request. The lines 45 – 60 will perform a simple string extraction to retrieve the ‘tag ids’. The UUIDs are included in a list for simplicity at line 62.

The lines 64 – 70 contain the prebid payload. This payload contains tag ids, UUIDs, information about the sizes of the ads, the user agent, the screen size, consent string and the publisher’s URL, referrer, etc.

Based on this information an auction is run and if one of the slots wins the response will contain the information to show the ads. Lines 77 – 100 will dynamically generate the prebid request URL, the HTTP headers based on the publisher’s URL, and finally execute the HTTP POST request. The response is captured at line 100 in the httpresponse variable.

The response consists of the HTTP response headers, status code, and the returned data. The latter is a big JSON string. When running the Python code it will print the returned data, see Figure 10:

At the top you can see Python3 is started with ‘20241114-prebid-spoofing.py’ as script. The adnsx.com is a parameter given, my intention was to include more ad servers. But, all ad servers I tried work similarly except the ids, required fields and data in the prebid format is different. But, besides that they do answer in a similar way to the bids, and thus this single example is clear enough.

The purple lines highlight important data. At the top right the uuid 110610a135b988de followed by tag id 32592018 can be seen. These ids are provided in the request. The same ids can be seen in the HTTP response just below ‘WON BIDS?’. To confirm, this bid was won and the data printed in the terminal pertains to the won bid. You can also see the costs which has a CPM of $10.000295. The advertisement, a banner in this case, is returned in the response as a raw HTML page, this includes embedded css, links to external images, and links to external JavaScripts. One of the JavaScript is the conditionally loading and executing of the IAS ad verification JavaScript, which is highlighted in yellow. The green line shows that the IAS verification script is only loaded 1% of the times, thus ad verification is being sampled. All other conditions need to be true as well, language and some array check.

You might wonder, what happens with this HTML code returned from the ad server? Technically: Your browser looks for an iframe with a specific elementid in the main webpage, and subsequently the HTML code is injected in that iframe using document.write! This means once injected the browser will parse the HTML and execute the embedded JavaScript and downloads and executes any external JavaScript. If you’re a JavaScript developer, you know how dangerous this is: blindly injecting and executing code from ad servers. It is what it is.

This shows how easy it is, only ~100 lines of Python code, to fire fake prebids and if any inventory is available you’ll win the bid and ‘an advertisement’ is returned. But, that’s not all. In order to be more realistic cookies need to be added. This shows the ad server you’re a ‘real user’ browsing the publisher’s site on the Internet, going from page to page.

I suppose the following thought has crossed your mind: Do any protections exist? Digital advertising is an almost trillion dollar per annum industry, surely some safety controls and checks exist? The answer might be shocking, but it is: No! It’s the open Internet!

The first layer of checks which are a substitute for safety are viewability and completion pixels. Completion pixels are for example fired at specific times when you are watching a video advertisement. Viewability gives the advertiser the data that an advertisement is viewed when an advertisement is loaded in the browser and 50% of the advertisement’s pixels were visible in the viewport for a continuous 1 second [4]. You probably agree with me that these pixels can be fired from a Python script as well.

The second layer of checks is done by the ad verification companies. Once their JavaScript is loaded by the browser it is executed and data is collected and sent to their cloud infrastructure. But, only in 1% of the times; meaning: Not executing the JavaScript will not ring any alarm bells. And don’t you think this can be fooled as well using Python? That means: Downloading their JavaScript, extract some session specific stuff, generate the payload, and return the faked payload to their cloud? Rest assured: Only the professionals do it like this.

Why am I showing you this? Because most people are not coding nerds, understanding bots, digital marketing, etc. and thus have a quite some technical debt when it comes to hands-on fraud prevention. Although, conceptually most do understand what is going on, but they never have reverse engineered an obfuscated protected piece of JavaScript, or have written some Python code to load advertisements, know how different browser internals work, or have written their own puppeteer or playwright bots. They might have asked some tech savvy people, but bots, browser automation, emulating a browser in Python (or whatever language) is quite a specialistic job and this bot world changes very quickly.

So what?

First, let’s clear this up: This isn’t the publisher’s fault, nor a configuration or implementation mistake at their end. It is just how the digital advertising ecosystem works: completely open. No authentication exists on the open Internet. Without any checks and controls ad servers will accept any bid request. How do I know? Why would it return the advertisement if simple fraud checks were in place?

At this point you’re aware that before advertisements are loaded and viewed a bidding process happens. Because the ecosystem is huge with tons of players you have to generate your specific prebid.js by checking the adapters and modules you need. You’ll need your own publisher and/or page specific JavaScript with its own configuration that will invoke the prebid.js in order to fire (pre)bids, and process the response.

It all can be faked

As shown before in 'how to make money using fake apps' again it all can be faked. The code in Figure 8 and Figure 9 is only ~100 lines of Python. This Python code was written on a Wednesday afternoon in about 90 minutes. Now think about what a team of professionals can achieve? They'll be aiming for a few millions at least, and it actually might work if they do a proper job and know their stuff.

If you do care about where and on what your digital marketing budget is spent on you need to protect yourself. Fraud detection really helps. But, not every fraud detection is created equal. I understand that it is hard for a lot of you to determine whether a fraud detection solution actually works, the pros and cons per fraud detection solution. This is not the same as how data is reported and can be segmented, how to integrate and process fraud detection data in your infrastructure? It is just looking at: Does it detect all types of fraud wasting my budget? Those questions are typically not answered with a product features matrix or checklist. It's similar to: How professional fraudsters look at and talk about fraud detection per vendor, eg. whether the detection can be bypassed easily, and how?

You might have questions? Feel free to comment, connect, or DM

#adfraud #digitalmarketing #prebid #frauddetection #CMO

Oxford Biochronometrics

[1] https://docs.prebid.org/download.html

[2] https://docs.prebid.org/dev-docs/bidder-adaptor.html#designing-your-bid-params

[3] https://docs.prebid.org/dev-docs/bidder-adaptor.html#http-simple-requests

[4] https://www.iab.com/wp-content/uploads/2015/06/MRC-Viewable-Ad-Impression-Measurement-Guideline.pdf

[5] https://www.linkedin.com/posts/kouwenhovensander_taylorswift-adfraud-adfraud-activity-7199384350369431552-t8EZ

Device and/or browser fingerprinting. Big Tech prohibits it [4][5][6][7][8]. The EU ePrivacy directive [11][12][13] requires unambiguous consent of the user. The reason: Cookies and fingerprinting are regarded as similar tracking mechanisms. Let’s take a look at fingerprinting. Why it exists. How it works and most importantly: Does fingerprinting in order to detect recurring fraud will meet your expectations?

Why fingerprinting?

Fingerprinting is to track users across multiple websites or applications. This information can be used to build a profile and based on that profile show specific advertisements fitting that profile.

In fraud detection, fingerprints can be associated to fraudulent behavior. The moment a previously flagged or known bad fingerprint appears you can ignore or park generated leads, reviews, or digital sales.

That’s the theory, more on that later.

What is fingerprinting?

Fingerprinting is a form of tracking an individual across multiple websites or applications. These websites and applications have no relation to each other and share no data. A 3rd party JavaScript would be able to associate activities to an individual.

The most easy and prevalent method of tracking is by using the IP address as an unique identifier. It is easy to collect (server side), highly unique, and relatively stable. The browser’s user agent would be another method, though the user agent will change roughly once a month when a new browser version is released.

Fingerprints are obtained by combining multiple settings and properties of resp. the device and browser. Fingerprints are broken in multiple parts. A static part based on hardware and a more volatile part based on the browser configuration. Device fingerprints are based on hardware, e.g. number of screens, screen resolution, color depth, graphics card, audio card, number of CPU cores, etc. Browser fingerprints are based on the attributes of the browser, e.g. language settings, plugins, content encoding, timezone, fonts, browser version aka user agent, etc.

How do regulators look at fingerprinting?

Websites running code that collects device and browser fingerprints must comply with the GDPR in the EU and also comply with the CCPA in California, US. In the EU device fingerprinting falls under European data protection laws and therefore requires consent similar to cookies. To track website visitors using device fingerprints you need unambiguous consent of the user. An exception can be found in article 29 [14]:

(29) The service provider may process traffic data relating to subscribers and users where necessary in individual cases in order to detect technical failure or errors in the transmission of communications. Traffic data necessary for billing purposes may also be processed by the provider in order to detect and stop fraud consisting of unpaid use of the electronic communications service.

GDPR protects consumers. Individuals. Not publishers nor advertisers. So, the question is: Does this exception permit browser and/or device fingerprinting to collect fingerprints on behalf of publishers and advertisers without unambiguous user consent? Can browser and device fingerprinting in such a case be seen as an exception under article 29? I’m not so sure. Anyone?

How does big tech look at fingerprinting?

Both Google and Apple prohibit fingerprinting. Apple prohibits fingerprinting in native iOS Apps [5][6] and has implemented anti-fingerprint mechanisms in Safari [7]. Google prohibits fingerprinting in combination with Google Analytics, see figure 2 [4] or GDN see figure 3 [8] unless the user has knowingly and expressly opted in. I couldn’t find the definition of ‘has knowingly and expressly opted in’.

What do fraudsters think about fingerprinting?

About one and a half year ago I wrote a deep dive into fingerprinting [9] and how JavaScript is used to flag outliers, lies, wrong answers, poorly implemented tricks, etc. The outcome of the fraud detection enables you to flag the visit and/or its fingerprint as malicious.

Fraudsters using browser automation to control a Chromium based browser will typically use special software to patch the browser. Once the browser is invoked and patched in order to prevent being detected by anti-bot vendors they’re good to go! Having a fully fledged undetected browser enables them to go through the entire journey: Load pages with an advertisement, click on the advertisement, load the landing page and fill out lead generation forms at scale. They’ll be using residential proxy servers to circumvent IP blocks, rate limiting, etc. the IP address to location will correspond with the contact address in the lead generation form.

Patchright, made publicly available on Nov 03 2024, is the perfect package to achieve these goals [10]. Figure 4 shows that it is available as Python package and NodeJS package. They claim that its stealth mode is able to bypass many anti-bot vendors as can be seen in figure 5. You might wonder: How on earth is it possible that an open-source package is able to bypass detection of all those vendors? The simple answer: Most bot detection vendors do more or less the same: They read the same properties, look for the same signatures and traces of JavaScript patching, look for CDP runtime enabled, etc. Although each vendor has implemented it differently: A browser is still a browser and there is only so much relevant information available in a browser that can be collected. Once patched within the browser, there’s not much that can be done about it.

In order to prevent a stealth browser to appear exactly the same over multiple requests it will load its settings, attributes and properties of the browser (and cookies, local storage, etc) prior to requesting a web page and executing JavaScripts. This prevents creating a single easy-to-flag fingerprint and that means fraudsters are able to scale.

Where do these settings, attributes and properties come from? They can be bought online (based on real people fingerprints collected at shady websites, pr*n sites, pirated content sites, etc.), or synthetically created by recombining fingerprint data from existing fingerprints.

So, in theory you can flag fraudulent fingerprints. But, be prepared to be disappointed because device and browser fingerprints are disposed after being used once. Just like OTPs (one time passwords) these visitors ues OTFs (one time fingerprints). You’ll never see them again. Except if the fingerprint was illegally obtained from someone using a real legitimate device, then you’re blocking this innocent individual.

Human operated fraud

Human operated fraud is more costly than browser automation, even in low wage countries. Though humans don’t scale like spinning up thousands of bots, from an anti-fraud detection view they need similar protection. They need to refresh their browser fingerprints, change their IP address, stuff the browser cookies and purge and reload local storage, and once they’ve taken a new ‘browser identity with history’ they can do their work.

The details of human operated fraud works, its operation, why use humans at all, will be described in a future article.

Now what? You’re saying that fingerprint doesn’t work?

Collecting fingerprints without any guarantees that a fingerprint is genuine is like accepting passport scans without validating and checking the physical passport. Without the look and feel of the paper, looking at the watermark, using UV light, checking the picture, etc. how do you know it is real?

The rule of thumb is that 99% of the online users will cause less than 1% of the fraud problems, but the remaining 1% will cause more than 99% of the fraud problems. Fingerprinting will not help to improve these ratios. On the contrary, browser automation using synthetic fingerprints or stolen fingerprints from real users are the problem. Human operated fraudsters will use special anti-detect browsers again loading synthetic or stolen fingerprints. They are the problem, flagging these will not stop fraudsters; Only amateurs.

Why would you use fingerprints to catch recurring fraud? If you can detect it the first time, you surely can detect it independently the next time? In the EU having JavaScript code on your website that collects and conveys the device and/or browser fingerprints means the website including the code needs to be compliant with the GDPR, ie. the user needs to provide consent to be fingerprinted [11][12][13][14]. So, why do vendors use device and browser fingerprinting? Because everybody else is doing it, see Patchright above.

Update: Does Google use fingerprinting? And if not, what do they use? AFAIK, They don’t use device fingerprinting to target audiences. They use the Protected Audience API available in Chrome, which uses “on-device ad auctions to serve remarketing and custom audiences, without cross-site third-party tracking.” [15].

To conclude and to confirm. Does Oxford BioChronometrics:

• use tracking cookies, or cookies at all? No

• use device and/or browser fingerprints? No

• collect data? Yes, how else can we detect fraud?

• combine the collected data in order to single out, infer, track or identify users over time? No

Questions? Corrections? Suggestions? Feel free to connect, comment or DM

#frauddetection #browserautomation #fingerprint #CMO #gdpr #digitalmarketing #adfraud

[1] https://blog.google/products/ads-commerce/2021-01-privacy-sandbox/

[2] https://digiday.com/media/googles-opaque-practices-to-restrict-fingerprinting-create-confusion-among-its-ad-tech-partners/

[3] https://abrahamjuliot.github.io/creepjs/

[4] https://support.google.com/analytics/answer/9682282?hl=en

[5] https://developer.apple.com/news/?id=z6fu1dcu

[6] https://developer.apple.com/documentation/bundleresources/privacy_manifest_files/describing_use_of_required_reason_api

[7] https://www.apple.com/newsroom/2023/06/apple-announces-powerful-new-privacy-and-security-features/

[8] https://support.google.com/adspolicy/answer/94230?hl=en#zippy=%2Cpersonally-identifiable-information

[9] https://www.linkedin.com/posts/kouwenhovensander_frauddetection-fingerprinting-activity-7049009901523656705-xrkX

[10] https://github.com/Kaliiiiiiiiii-Vinyzu/patchright

[11] https://www.edpb.europa.eu/system/files/2023-11/edpb_guidelines_202302_technical_scope_art_53_eprivacydirective_en.pdf

[12] https://iabeurope.eu/wp-content/uploads/20161201_IAB-Europe-Position-on-ePrivacy-Directive-Review.pdf

[13] https://www.law.kuleuven.be/citip/blog/no-escape-from-the-eu-privacy-rules-for-the-ones-collecting-device-fingerprinting/

[14] https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=celex%3A32002L0058

[15] https://developers.google.com/privacy-sandbox/private-advertising/protected-audience

Q: What happens if you leave a loophole in online fraud detection?

A: It will surely be exploited!

In 2024 IT security has to be based on the Zero Trust security model [1]. This means: No one is trusted by default. Verification is required in order to be trusted.

This security model works for years when dealing with untrusted files, emails, network connections. Not just in IT, but also in real life. That’s why:

• Email attachments are all scanned before you’re allowed to open them, 100% is scanned

• Your virus scanner scans every file before you’re allowed to open the file, 100% is scanned

• The routing and header information (SPF, DKIM, DMARC) of all emails are checked in order to detect and flag spam, 100% is verified

• Firewalls block any unsolicited inbound connection, 100%

• At US airports all passengers are security checked by the TSA, 100%

• etc.

Does this zero trust security model fail? Yes, sometimes it fails. In most cases because of human error. Zero-day exploits which are not yet detected by the virus scanner are very rare and if you’re a target then you already know what to do. But, clicking on a malicious link in an email, opening a malicious attachment not being flagged are reason it fails, because the human verification failed.

Once clicked on such a attachment or malicious link you’re screwed. This can be: Cyber extortion because of a data breach, a ransomware attack, or an APT will be installed [2]. This is all well known and many companies have implemented a ton of security measures in order to mitigate these threats. But, this is to protect their own infrastructure and the data within their corporate networks. Besides state sponsored actors the majority of bad actors are not really interested in the stolen data. They are interested in the monetary value of the data through extortion and/or blocking access to your private or company’s computer(s) and its files until you pay a hefty sum of money.

Ad fraud totals US market

In 2024 the global digital ad spending will reach ~US $667 billion. At a nine percent growth this means before 2030 the total digital ad spending will cross the US $1 trillion mark. This means a single percent of ad fraud represents US $10 billion. Looking at the US market, which is about 1/3rd of the global market, this means a single percent represents ~US $3.3 billion. If the total fraud is 20% the monetary damage of this fraud in the US will be 20% * ~US $3.3B = ~US $66 billion.

That’s a lot of money! And this implies a lot of responsibility for the vendors fighting this type of fraud.

Q: How many sucessful ransomware installs and payments are required to reach US $66 Billion?

A: At avg. $300 per device that’s 220 million devices and payments!

So, potential ad-fraud profits are way bigger than ransomware and it is less distributed. The following question arises: Does this mean that money spent by brands and/or advertisers is protected better?

Zero-trust in ad-tech?

One of the protections is ads.txt, where ads is an acronym for Authorized Digital Sellers [3][4][5]. If you want to know how such a file looks: go to the location bar in your browser, type in the domain name, a slash, and ads.txt. For example: https://www.wsj.com/ads.txt or https://www.wsj.com/app-ads.txt.

Who enforces these files? The DSPs (Google DV360, The Trade Desk, Centro, etc) will load these text files from a publisher’s domain and only serve ads from authorized monetization partners on these domains. In theory this should protect advertisers from MFA sites and other shady websites, but look at the current situation and how did this work out? Even if you have implemented whitelists of allowed domains fraudsters still find a way around.

But, that’s the supply side. What about the demand side? The demand side are the millions of browsers, apps, and connected TVs downloading a webpage with ad slots requesting ads, or watching a video requesting an in-stream ad. To understand what happens beneath the hood let’s first look at the lifecycle of a typical advertisement in programmatic. Its lifecycle has two logical stages: 1) pre-bid and 2) post-bid.

Pre-bid

The pre-bid stage requests an advertisement. This means the app, connected TV, browser send a requests to the ad-tech infrastructure which will start an auction to determine whether an ad is available in your geo, within your min/ max bid price, device, targeting (your cookies), etc. If the bid is won, an advertisement is returned and the post-bid stage starts.

Post-bid

The post-bid stage will load the advertisement in the ad-slot. If applicable it will start measuring viewability, detect ad-fraud, and fire completion pixels.

Figure 2 shows the (simplified) lifecycle of an advertisement in programmatic as seen from the browser [7]. Each stage is a contact moment between the browser and the ad tech infrastructure. In a zero-trust environment each stage needs to revalidate whether the browser is really the browser it claims to be. And, if not: Failure, bot, or something else you don’t want to pay for!

How is this implemented in pre-bid?

In pre-bid this is achieved by looking at the IP address and the user agent. The IP address roughly tells you where the request comes from. If you’re an US insurance company licensed in 30 states you only want your ads served in those 30 states. The user agent tells you what kind of device is requesting the ad. If the user agent is a bot, eg. HeadlessChrome, bingbot or Applebot you don’t want your ads being served. But, unfortunately, browsers are able to set their own user agent and proxy servers can be used to fake the geo-location. It means these two precautions don’t fit the zero-trust model.

How is this implemented in post-bid?

The typical setup is that a piece of JavaScript code is sent along with the advertisement. Once loaded in the browser it verifies whether the advertisement was seen within the viewport (viewability) and whether the request was made and viewed by a human or a bot. In Connected TV no JavaScript can be executed, only completion pixels can be fired. Whether this fits the zero-trust model can be read below.

What would the fraudster do?

A few weeks ago I posted this article: ‘How to make money using fake browsers’ [6]. This article describes how you can successfully request an advertisement using ~100 lines of Python. One of the images in that article shows the returned advertisement including the post-bid IAS verification loader JavaScript.

Figure 3 shows the prebid response from the ad server. This prebid response contains a field named content which contains the content to be loaded in the ad slot. The content field in Figure 3 is highlighted in red and startes with “content”. This field also contains the IAS post-bid verification tag loader. This small piece of JavaScript starts just before the green marked text after the word <script> and ends just after the yellow marked text before the </script> tag. Its purpose is to load the real verification JavaScript, which is hosted at IAS.

IAS ad-tech: 99% trust?

Looking at the IAS verification tag loader JavaScript code in figure 3: In the green label the following code can be read:

if (Math.random() * 100 < 1 && …. )

This means that only 1% of the browsers will load the verification tag. 99% of the browsers will not verify the advertisements. This is called sampling. But, the decision is made at the client in the browser! This means the bot decides whether it load the verification script or not. The bot decides to be in the 99% trusted group, or not. In this implementation: You don’t know whether the bot didn't load the verification code ON PURPOSE, or the random number was >= 1.

To give IAS some free advice: If you still want to stick with sampling at 1% then please do it well! Move the decision which ad is to be verified and which not to the server side. In this new scenario the server selects which ad is verified, not the browser, not the bot! If a bot still doesn’t load the verification tag and thus nothing is returned, this means: the advertisement was not loaded, not viewed and shouldn’t be paid for.

In post-bid other vendors do sample as well. Human for example. The difference is the decision which client is verified lies at the server side, which is fine if the decision is really random. As far as I know Double Verify doesn’t sample.

Now what?

Is every pre-bid request verified? That depends on the ad-tech infrastructure. Some vendors have partnerships in order to verify all prebid requests, others might do the verification themselves. In post-bid it all depends on the sophistication of the detection (browsers can spoof properties), how well the call back containing the collected data is protected (bots are able to capture and rewrite the returned payload). It also depends whether sampling decisions are made server side (this is the correct way) or client side (this is wrong!). But, how well does the detection solution work? At this moment we just don’t know because no regular independent security audits are being done to verify this.

Prebid

More technical checks need and CAN be to be done in order to verify prebid requests. IP address and user agent are just not enough, as they can be spoofed. Although

Post bid

In order to verify and detect invalid traffic better technology needs to be implemented. You’ll never detect and flag all invalid traffic, but that’s not the goal. The goal of better detection is to make it more expensive to operate for bad actors. When bad actors need to continuously improve their technology, have to route the traffic over expensive residential proxies it will make it much more expensive. The end goal would be to make it economically unfeasible to operate. The first step to achieve this goal is to close all blind spots and loopholes. Unfortunately, at this moment huge blind spots and many loopholes still exist.

The second step is to improve detection technology to detect invalid traffic. This technology exists, but you do have to use it. For example, the current naive detection technology does read many browser properties but take them at face value. In order to improve this your technology needs to be ramped up. If you want to know more about this. Oxford Biochronometrics can help.

Red team audit

Brands and/or advertisers do you want to know how well your digital spend is protected against invalid traffic and/or sophisticated invalid traffic aka fraud? In order to know how well your ad-tech infrastructure protects its inventory and thus your spend you could run an audit. The idea behind an audit like this comes from red team/blue team approach [8]. The bots represent the red team and its goal is to demonstrate what works for the defenders. The verification vendors represent the blue team and must defend against real or simulated attacks.

The red team will unleash a number of different bots. These bots should be different in the level of sophistication. Based on the returned advertisement or a blank HTTP 204 message you know whether the bot was detected or not in pre-bid. The same can be done for post-bid. If the detailed logs contain these bots and show them as human, the bots were not detected. The blue team (the verification vendors) will try to detect the bots. Because the bots differ in sophistication some bots might be detected and some not. If only a subset of the bots is detected you can determine how well the blue team did its work. Of course, if (server side) sampling is used you only have to look at the sample.

The purpose of this audit is NOT to share technology on how to detect bots. Acquiring and improving knowledge is up to the blue team. They’ll need to hire better staff, build a research lab and run internal blue/red simulations within their lab to improve their bot detection technology.

Knowing fraudsters and knowing that if any blind spot or loophole exists they will find it! To prevent this your verification vendor will have to adopt the zero trust security model and verify everything. Just like the TSA, your virus scanner, and your firewall: 100%!

Questions? Feel free to connect, comment and/or DM.

#adfraud #digitalmarketing #prebid #frauddetection #CMO

[1] https://en.wikipedia.org/wiki/Zero_trust_security_model

[2] https://en.wikipedia.org/wiki/Advanced_persistent_threat

[3] https://iabtechlab.com/ads-txt/

[4] https://support.google.com/admanager/answer/7441288?hl=en

[5] https://support.google.com/admanager/answer/9422067?hl=en

[6] https://www.linkedin.com/pulse/how-make-money-using-fake-browsers-sander-kouwenhoven-esgce

[7] https://support.google.com/admanager/answer/7128958?

[8] https://csrc.nist.gov/glossary/term/red_team_blue_team_approach

Human operated fraud uses tools to circumvent detection technologies that would normally block abuse, spam, fraud, etc. These detection technologies work using blacklists. By understanding what exactly is being blacklisted enabled tool makers to build and sell tools that circumvent these lists. These tools are the anti-detect browsers.

Anti-detect browsers are such a handy tool. They avoid being blocked by platforms or inferior fraud detection solutions, by simply changing their appearance using fake fingerprints and rotate their IP address by using proxies. These tools are sold to manage multiple LinkedIn, Facebook, TikTok, Google ads, Instagram, Youtube, Ebay, Etsy, Coinbase, Telegram, X, etc. accounts using a single browser. If these big platforms can't detect them, then are these tools really that good!?

Other purposes of these anti-detect browsers are to generate leads using stolen PII data. Generating leads using bots is still a hard problem for bot makers. Luckily for fraudsters, the profits are enough to hire humans in low wage countries and these humans will fill out the lead generation forms manually.

Many different, 30+ listed in this article, anti-detect browsers are on the market. All claiming to have thousands of users. That means there’s a market for this type of service. The unanswered question is at the expense of what?

Fraud in marketing

By now it should be clear that fraud does exist in the digital marketing ecosystem. Impression and click fraud happens and cost real money. The difference between impression/ click fraud and lead generation/ digital sale fraud is the frequency and the financial risk. If a fraudulent impression is made it will cost a brand a fraction of a penny, but if a fraudulent lead is being generated it costs $10+ and if followed up by your call center you might get sued because the callee didn’t provide express consent. That will cost at least $500 : The settlement costs PER CALL and add to that all the legal hours and other overhead costs.

Fraudster only make money with impression fraud and click fraud at scale. They need millions of impressions to make a few thousands bucks. That’s why they have an army of bots doing that type of work, simply because: Bots scale well and are cheap. Manually clicking on advertisements by humans does happen. But, only at small scale, the most known type of fraud is to burn a competitor’s budget. It’s not to make profit, but to hurt another company. In many other cases humans are too expensive for these types of fraud.

That changes in lead generation and digital sales. Because, the potential profit per lead or sale is in the dollars range and a typical human is able to generate many leads per hour, or to buy many Taylor Swift tickets per hour. Knowing the amount of profit per ticket ($100++/ticket), or per generated lead ($10+/lead) it’s a lucrative business.

Simple bots can be detected using simple technology. Once you know what the difference is between a Python script firing requests and a real browser or app [1][2], browser automation versus a regular browser, etc. bots can be detected with ease. It becomes next level when you’re dealing with human operated fraud.

Human operated fraud

Human fraudsters (human operated fraud) use browsers as intended, using a keyboard and mouse, or touch and a virtual keyboard on a mobile device. So, how does the typical fraud detection vendor try to detect human fraudsters?

• IP address reputation

• Fingerprinting of device and browser

Once fraud has happened an IP address, or fingerprint is put on a blacklist for 30, 60 or even 180 days. Of course this will only solve the fraud problem if fraudsters would use their home IP address and their regular browser. But, they don’t. So, how are these human operated fraudsters able to rotate IP addresses and fingerprints?

How do these human operated fraudsters work?

The goal is to circumvent being blocked by both IP address blacklists and fingerprint blacklists. The poor fraudster’s solution is to use a browser extensions to change the browser’s fingerprint [3]. These browser extensions will be used in combination with free proxies or free VPNs [4]. But, ‘free’ comes with a catch: It can be detected with ease. The usage of browser extensions can be detected from JavaScript. Secondly, free proxies and VPNs operate 99 out of 100 times out of data centers.

Professionals use special anti-detect browsers. These browsers are rented as a service and do cost a monthly fee. Quality has its price. These browsers have built-in features that enable a different proxy server per browser tab, a different device fingerprint per browser tab, a different local storage and cookie jar per browser tab, a different browsing history per browser tab. In other words, each tab looks like a completely separate visitor coming from a distinct location using a completely different device.

Anti-detect browsers support importing and exporting profiles, which means device and browser fingerprint, cookies, local storage, history, etc. can be transferred from desktop A to desktop B and someone else is able to continue a warm session. You might think that this is a lot of manual work? Wrong! Because many of these browsers support a build in REST API [5] in order to automate import and export of profiles. These profiles are typically stored centrally and workers will download such a profile and continue to reach their goal.

Centrally storing profiles also enables bots to warm up profiles by browsing the Internet, collecting cookies, and be tracked over websites of a certain topic groups, etc. Once a profile is ‘warm’ or ‘mature’ implies that it fits an audience and advertisements targeted at this audience will be shown. Now it’s just a matter of browsing to complicit websites and click on the advertisement and finally be redirected to the landing page. This means the attribution goes to the complicit websites.

Humans will fill out the lead generation form, or buy the digital products. Although this can be done by bots, humans are more flexible in unexpected situations when for example lead generation forms are being A/B tested. Also existing fraud detection tools will have a harder time, looking at IP address reputation and blacklisted fingerprints works only after the fact.

Cloud phones

Figure 3 and Figure 4 list 31 anti-detect browsers mostly available for Windows and Mac. Some anti-detect browsers have a Linux or Android version. Some even offer cloud phones, which means that on a desktop you’ll be remote controlling a real physical phone in a data center. The left images in Figure 5 shows that phones in a cloud -which is just a fancy name for a data center- don’t look like your iPhone or Samsung phone with a touch screen, battery, GPS, etc. Nope, these phones are the bare minimum hardware: a circuit board with a CPU, memory, (e)SIM, storage and an installed OS and apps that enable the phone to be remote controlled over the Internet, spoof GPS, and have a remote camera, etc.

Using special anti-detect browsers enables you to use these phones as a service. Connect to them, push your profile (cookies, local storage, device fingerprint, browsing history), and continue whatever session and do you want. And, yes, these phones have SIM cards (virtual eSIMs), so they are able to send/ receive text messages. Figure 5 shows how a box of phones look like and how they can be remote controlled. These are btw real Samsung phones, but without the screen and battery.

Anti-detect browsers can be used for legitimate business. For example, if you manage multiple social media accounts, eg. 5 -- 10 small businesses, you prefer not to have to login/ logout 20 times a day and switch between companies in order to answer questions, post messages, etc. But, the same technology can also be used for less legitimate things. There’s a thin line where privacy ends and a fraudster’s toolbox starts. Do you really need a build-in REST API in your browser to guarantee privacy? Do you really need a marketplace where you can buy a zillion device fingerprints collected from real devices in order to guarantee privacy? Do you really need a unique device fingerprint and proxy per browser tab?

What are they really used for? These anti-detect browsers

What can be achieved with these anti-detect browsers, besides warming up and aging visitor profiles and some lead generation fraud? What are these browsers used for? If you look at how people use these browsers you’ll find that they are used for:

• Ticket / Sneaker scalping

• Online dating scams

• Pig butchering, shazhupan aka investment scams

• Fake reviews

• Lead generation fraud

• Troll farms

• Card testing

• Inflate social media activity and interactions

• Affiliate marketing fraud

• Click fraud

• Poison surveys

• Get free coupons, freebies, promotions, etc.

In Figure 6 an online forum message is shown where someone complains about fraudulent clicks on advertisements and subsequently fraudulent generated leads. Google is not able to detect these clicks, and Google’s algorithm is completely poisoned because it thinks these clicks are legitimate. The last line of the bottom message is the most concerning one. Quote “They essentially refuse to admit there’s any way that invalid traffic can exist outside their ability to detect it”. The arrogance.

Another example of anti-detect browsers can be found on the websites of these browsers, and more specific on the services pages. Figure 7 shows that multilogin can be used for ticket scalping. For those who don’t know what ticket scalping is Figure 8 describes how the ticket scalping solution page explains it. With the best regards to #TaylorSwift

How do fraudsters know these tools work?

How do fraudsters know the claims of these anti-detect browsers are real? They’ll be the worst customers believing any claims. They trust nobody and they know exactly why. Their validation tools are public websites showing their fingerprint and/or VPN/proxy server information.

Whoer.net is a ‘service aimed at vertifying the information your computer sends to the Internet’. They will check the reputation of your IP address, or the VPN/ proxy you are using. Secondly, they’ll check what information is available to be fingerprinted.

These websites will check and show your browser’s fingerprint, and a fingerprint reputation score:

• https://pixelscan.net

• https://browserleaks.com

• https://abrahamjuliot.github.io/creepjs/

In order to validate DNS leaks they’ll use https://dnsleaktest.com/. Some more background info on this: If your webtraffic routed over a proxy server a web server will only see the proxy server’s IP address. DNS traffic from the same host is not routed over that proxy but is sent directly. A session specific DNS query and the difference in IP addresses (proxy IP address vs real IP address) can be detected and may be flagged as fraud.

Other types of leaks (DNS, WebRTC or msleak) can be validated using https://www.perfect-privacy.com/. They offer tests to validate whether your browser leaks your real IP. These are the test URLs:

• https://www.perfect-privacy.com/en/tests/dns-leaktest

• https://www.perfect-privacy.com/en/tests/msleaktest

• https://www.perfect-privacy.com/en/tests/webrtc-leaktest

• https://browserleaks.com/webrtc

• https://browserleaks.com/dns

• https://nordvpn.com/dns-leak-test/

And many more of these tests exist online.

Now what?

Apparently humans using anti-detect browsers bypass all fraud detections. Once they know how to configure their browser, which settings work best per website or platform, which residential proxy providers have the best (fresh) proxies, which device fingerprints look genuine, they bypass all fraud detections. The existing bot detections at these platforms aren’t effective, because these fraudsters are not using bots, but real browsers with a human interface. They’ll use real phone numbers to receive texts in case of 2FA.

These tools are also used in lead generation, or in digital sales. Humans will manually fill out the lead generation forms and the attribution goes to the affiliate or source that provided the lead. If these sources have knowingly or unknowingly generated fraudulent leads they still get paid. The tcpa and potential litigation risks are for the brand buying and following up on these leads.

So, can this type of fraud be detected?

In lead generation the behavior of fraudsters differs from regular people who will see the lead generation form for the first time. It’s like walking in a store straight to the right section, the right aisle in order to get the articles you want.

Routine and experience and time pressure are hard to detect, but not impossible. It’s like looking at the differences in handwriting. A doctor’s handwriting is impossible to read for normal people. Most people have inconsistent handwriting, though the same quirks and style can be seen in their writing. Professional calligraphists have great control over their handwriting because of knowledge, routine and experience. Someone filling out a multi-stage lead generation form for the first time doesn’t know what is coming, has to read the labels carefully, etc. This behavior is completely different compared to someone filling out hits form for the 53rd time.

Behavioral differences and in particular fraudulent behavior is what Oxford Biochronometrics detects besides the regular browser automation and request based bots. Fraud detection at this stage needs to be done with great care, because false positives are very expensive, ie. missing a business opportunity. False negatives are again expensive in terms of potential litigation risks [6]. That’s why accuracy of fraud detection is of great importance. Oxford Biochronometrics encourages their clients to continuously measure the performance of the fraud detection implementation and some of our client have even A/B tested us against other vendors. The result: They keep on being our client, and we keep on being the best!

Want to know more? Questions? Suggestions? Corrections? Comment, connect or DM

#adfraud #leadgeneration #CMO #tcpa #antidetect #frauddetection

2024-11-21

Update 2024-11-22:

• Added section "How do fraudsters know these tools work?"

• Updated title image, now includes synthetic IDs

• Added section "How do fraudsters know these tools work?"

• Updated title image, now includes synthetic IDs

[1] https://www.linkedin.com/pulse/how-make-money-using-fake-browsers-sander-kouwenhoven-esgce

[2] https://www.linkedin.com/pulse/how-make-money-using-fake-android-apps-sander-kouwenhoven-hs1me

[3] https://awesome-privacy.xyz/security-tools/browser-extensions

[4] https://www.linkedin.com/pulse/proxy-servers-root-all-evil-ad-fraud-sander-kouwenhoven-mtt7e

[5] https://en.wikipedia.org/wiki/REST

[6] https://www.linkedin.com/pulse/what-gets-mismeasured-mismanaged-sander-kouwenhoven-2wlee

Imagine an Internet without proxy servers. It would imply that all traffic originates from its own IP address. That also means that when bots in a data center load advertisements, click on advertisements, watch videos, or listen to podcasts you’ll be able to see them simply by looking at the usage per IP address. Normal humans don't load 1000s of ads per minute, or watch hundreds of video streams concurrently. Without proxies excessive usage numbers using automation would be visible.

This means that proxy servers enable bots, and other types of fraud to split their traffic over many different proxies. For example, if you run multiple browsers then each browser connects to a different proxy. At the receiving end you’ll see the incoming requests from many different IP addresses.

Legitimate usage?

Proxy servers do have a legitimate usage. For example, you run different digital campaigns per state and you want to verify the correct ads are served per state. Second example is to verify that advertisements don't serve malware to residential IP addresses only. Third example would be to verify that advertisement really advertise what is being sold on the landing page after the click.

The fraudster technique behind this is 'cloaking' where only residential IP addresses and specific user agents arrive at the gambling, counterfeit products, or illegal goods, etc. website. Visitors from data centers, eg. automated scans by Google, Microsoft, etc., are sent to a different website and thus the automated scan thinks that the advertisement are legit, or no malware is served.

It becomes a gray area when scrapers, crawlers and bots using proxies harvest your content in order to train AI models, collect price information, etc. Though, you could disallow these bots by adding their user agents to your robots.txt, but will these bots adhere these rules? If they even declare themselves as bot. And if not what will you do?

The next valid question would be: How do these legitimate cases explain the vast amount of companies offering zillions of proxies? And those are not data center or ISP proxies, but residential proxies in your neighborhood.

So, Where do these residential proxies come from?

A simple Google search query “high quality proxies proxy servers” shows 19 companies offering proxy services, and there are many more. A screenshot of the results can be seen in Figure 1. Detailed information of the first 10 companies are shown in Table 1. The table contains the company name, URL with details of the residential proxies, the number of proxies worldwide, in the US, in the UK, and in Germany. This gives a good idea how big this ecosystem is, the combined total of these 10 companies world wide is over half a billion proxy servers! Let that sink in!

Table 1 shows the number of IP addresses and the bulk of these are residential IP addresses. An important question would be: How are these residential addresses obtained? The companies themselves claim they are obtained... ethically.

So, how do these proxy companies source their IP address and obtain bandwidth from regular people? By looking at the almost infinite numbers it should be fairly easy.

1. Mobile App creators integrate proxy SDKs in (free) Apps. This enables the developer to earn some extra money [1] [2].

2. Windows or Mac application developers integrate a proxy SDK in their software. This enables the developer to earn some extra money [1] [2].

3. Directly sublet your home broadband to companies that use your bandwidth to proxy their client’s web traffic [3] [4].

4. (free) VPN apps. You’re using a free app and think you are safe. The catch is that the VPN tunnel works in both directions. Your internet connection is used by someone else paying to anonymously access the Internet.

5. Malware. Legitimate companies will typically not use malware, but if they obtain their proxies from other companies without doing a proper checks this might be the case.

6. IP range hijacking. Ancient IPv4 addresses assigned 20+ years ago, forgotten by administrators, no ARIN membership [5], no protections like RPKI or ROA [6] makes them vulnerable to IP range hijacking [7]. Just like malware it's not legal, but IP range hijacking does happen and if successful it generates a lot of money.

You can decide for yourself how ethical each method is. The IP range hijacking and malware proxies are typically used in real criminal activities (hacking, data extraction from companies or government) as criminals can be sure that they don't log anything nor leave a trace.

Breaking down making money for bandwidth

If you sublet your own home broadband you'll get compensated for the bandwidth. But how much do you get compared to the money these companies make?

Figure 2 shows that by sharing your internet connection you'll earn $0.20/GByte and Figure 3 shows that as a proxy user you pay from $4.55/GByte when buying 100GByte of traffic (residential proxies). That’s a nice margin! The earnings with cash for bandwidth are capped at $140/month, but the proxy subscribers will have to pay for each and every GByte.

But, if an App integrates an SDK which allows the app to act as a proxy. Does the owner of the device know this? And is this person compensated for the bandwidth? Or is the App developer the only one being compensated (at $0.20 / GByte) ? [8] How ethical is that?

Again, you can decide for yourself how ethical this compensation is. The example above is just one out of many. If you want to look at some other examples, just search for: Honeygain, Repocket, Earnapp, Packetstream, Loadteam, and there are many more.

Fraud detection of proxy servers

Proxy servers only forward specific traffic, browser traffic or HTTPs traffic. Technically the setup looks like this: Your browser talks to a proxy, the proxy server talks to the web server, and because of that the web server only sees the IP address of the proxy server. Legitimate proxy servers will add HTTP headers, eg. x-forwarded-for, to inform the web server that they did forward the request. But, not all proxy servers do that. Other network traffic, such as DNS queries, UDP are not forwarded to a proxy server by your browser. They are sent directly. This can be leveraged for fraud detection.

WebRTC (Web Real-Time Communication)

Having a zoom video call from your browser uses WebRTC. In order to communicate your browser tries to directly communicate with the zoom video server. Your browser will bypass the HTTPs proxy server and tries to communicate directly. If the user, or fraudster, doesn’t realize this it can be used to determine the true IP address of the client.

DNS (Domain name system)

Domain name resolving works backwards. For example, if you want to resolve the IP address of blablabla.phonyurl.com you ask your local DNS server: what is the IP address of this domain? If it isn’t cached, it will start the full resolve process. First the .com domain is asked: who is phonyurl.com? Then a second DNS query is made and sent to the DNS server of phonyurl.com in order to know what is the the IP address of blablabla.phonyurl.com. If you are the owner of phonyurl.com you also own the local DNS server. Now, let’s generate unique random subdomain names which don't exist yet and thus cannot be cached. In that case the phonyurl.com DNS server will see a query to resolve some random name and thus knows which IP address did try to resolve that name. Tying back the IP address from the DNS resolve to the IP address of the visitor which downloaded a JavaScript file which included this random DNS name is another way of matching the IP addresses. If they are the same it's great, but if they differ that could be because DNS resolving is not sent through a proxy server.

Difference proxy server and VPN

One way to overcome this is to use VPNs which tunnels all network traffic to its endpoint. This includes all protocols like HTTPs, DNS, UDP, NTP, etc. Figure 4 shows the differences between a proxy server and a VPN and how fraud detection at the receiving end would be able to see the IP address(es).

Are (residential) proxies a problem in ad fraud and lead generation fraud?

This is a perfectly valid question. If ad verifcation companies do blacklist all data center IP address ranges, prebid will be ignored, and no advertisements will be served to these IP address ranges. If bots running in a data center don't get advertisements fraudsters in the ad fraud ecosystem will have to use more expensive residential proxies.

I’ll leave it to the reader to ask their ad verification partner whether they upfront flag data center IP address ranges and prevent prebids and ads being served to bots. If they say: “Yes, we do block them!” Can you just trust them? or should you validate their claim using a few random data center proxy servers yourself.

Once you know data center proxies are blocked you might have peace of mind. But, residential and mobile proxies still exist. Though they are more expensive to use than traffic directly from a data center, will ad fraud will still be profitable? If you have to pay $4.55 per GByte, the question is: How many advertisements fit in a GByte? That, of course, differs per ad type. A video ad consumes more bandwidth than a display ad. As you can see, and calculate, these proxies are too expensive for impression ad fraud, especially video ads. Simply calculate how much a site earns per 1000 ads, and calculate the data usage of these ads. Spoiler: Renting residential proxies eats away most, if not all, of the profit made by impression fraud.

In lead generation the volume and thus data usage is smaller and the profits per generated lead are way higher: Dollars instead of fractional pennies. That's why residential proxies in lead generation are common. A second reason is that they when filling out a form the contact address and area code of the phone number has to match the geolocation of the IP address. Proxy providers do programmatically offer you to connect to specific proxies in countries, states, ZIP code, cities or ASN (Autonomous System Numbers) / ISPs. This enables fraudsters to quickly switch to the desired location matching the PII data to be filled out in the contact form. And you already know this, the PII data is bought on the dark web and originates from data breaches.

As mentioned fraudsters continuously rotate IP addresses to keep under the rate-limiting radar, matching IP address with contact data. That combined with the availability of 50 million IP addresses, it will take a very long long time to exhaust all IP addresses or even blacklist them, and these simple facts should worry you.

So, again in different words: Are proxy servers (and VPNs) the enabler of ad fraud?

Without proxies, protecting against ad fraud and lead generation fraud would be so much simpler. Simply flag an IP address when fraud is detected, and use rate limiting per IP address to prevent excessive usage. Only these simple filters will solve 80% of the problems. But, reality is different: Proxies (and VPNs) are part of the Internet. And, thus yes, they are the enabler of many sorts of fraud, downloading TBytes of data from data breaches, web scraping, spamming, buying Taylor Swift tickets, purchase all limited sneakers or playstations at the moment of release, etc.

Can they be detected reliably? It depends. The companies providing these proxy services do improve and because of that it gets harder to detect inconsistencies. But, luckily, many fraudsters are not that technically skilled and make mistakes. The professionals however are a pain and know how to configure their bots to avoid detection at almost all levels. Luckily, in lead generation humans have to interact with the contact form and by looking at the human interactional behavior Oxford Biochronometrics is able to determine whether someone interacts with the contact form, or something interacts with the contact form. A bot replaying a pre-recorded session, or a bot moving the mouse using programmatic lines (eg. b-spline or bezier curves), etc.

Human operated fraud using browsers and residential proxies or VPNs is another level. Luckily their behavior is different from normal humans. Just like an experienced shop owner is able to spot a thief by looking at human behavior, this works the same in the digital world.

Want to know more? Questions? Corrections? Suggestions? Feel free to connect, comment or DM

#proxies #adfraud #leadgen #CMO #residentialproxies

[1] https://bright-sdk.com/

[2] https://infatica.io/sdk-monetization/

[3] https://www.getpaidto.com/quick-points/bandwidth/

[4] https://pawns.app/internet-sharing/

[5] https://www.arin.net/

[6] https://www.arin.net/resources/manage/rpki/roa_request/

[7] https://ipv4.global/blog/hijacked-ip-addresses/

[8] https://repocket.com/sdk

Ad fraud. In this case App fraud. Although no Android phone and installed apps are involved. The fraud is completely generated on servers running a Python script.

You might think: “Sure.. but, we have a blacklist, so we’re safe!”. Wrong! Your blacklist doesn’t contain all possible permutations of fake app names. Fraudsters continuously generate new names, and sometimes even use random sequences of characters. Having and managing a whitelist of allowed appnames would be much safer! But, how safe are you really?

About two months ago Dr. Augustine Fou asked me the following question:

How difficult is it to generate or modify a HTTP web request to retrieve a webpage?After some questions back and forth it boiled down to:

• Can a bot modify or add a referrer to a request? And if so: How?

• Can a bot modify or add the App name (Android)? And if so: How?

• Can a bot modify a bid request? And if so: How?

• Can a bot modify the domain of the website? And if so: How?

To answer these 4 questions upfront: Yes, this is possible! In this article I will show you how you can change HTTP requests in your browser and how it can be done programmatically.

Dr Fou’s post “60% of digital ad spending going to mobile apps is a bad thing” states that apps generate “30 trillion bid requests per week” [4]. Full stop! Let’s break that down: First, let’s assume this number is worldwide. In digital marketing the USA represents about 1/3rd of the worldwide volume. The USA population is according wikipedia 335 million. So, 10 trillion bid requests divided by 335 million = 29,851 bid requests per week per person. Isn’t that a bid much? Even if one in 10 bid requests is a win that’s still 3,000 requests, or 428 advertisements a day! On average, including babies, elderly people, people working, etc. Isn’t this number a bit steep?

Maybe you remember Dr Fou’s post in August 2024 as shown in Figure 1 [1]? This “bot attack” comprised of ~10,000 request to fouanalytics.com claiming to originate from fake, though humoristic, Android apps. This bot attack wasn’t a real attack, it was an experiment where I wrote a program that fired HTTPS requests to https://fouanalytics.com while changing the Android appname by spoofing the x-requested-with HTTP header. And, yes, those app names were truly random generated.

This article will explain the basics how I achieved this in laymans’ terms. It gives you a better understanding of how easy it is to manipulate or generate HTTP requests. Your takeaway after reading this is to not blindly trust anything online. Especially when money is involved. And even more when lots of money is involved, like in the advertising ecosystem.

First, I’ll explain how requests look like and how to change HTTP headers in your own browser by intercepting the requests. This shows exactly what happens under the digital hood. The second part is doing the same programmatically using Python. This can be achieved with only a few lines of code and isn’t any magic. Let’s start with the manual one using a browser extension.

Use browser and extension, such as Requestly

What happens when a browser navigates to a website? The browser sends an initial request which contains the URL of the website, the path, and optionally a querystring and cookies. Using a browser extenstion this can be modified easily

In Figure 2 can be seen that the a Chrome desktop browser’s User Agent is modified to an Android device, more specifically a Facebook App User Agent. It also adds the X-Requested-With HTTP header to make it appear a real App making the request. The Referer and Origin HTTP headers are set to

So, let’s see how traffic from/to https://www.fouanalytics.com looks like when these HTTP header modification rules are applied during live traffic. Figure 3 shows the network traffic to the main FouAnalytics site including the changed HTTP headers in blue. The HTTP headers starting with Sec-Ch-Ua still have their original values, making it obvious that the request has been tampered with.

Modifying HTTP headers is changing only one piece of the total puzzle. It only gives control over the HTTP headers, but not over any low level networking layers (ie. network packets, TLS fingerprinting) and the data captured by the application layer in the browser using JavaScript. That requires more than just modifying HTTP traffic.

But, remember and realize: When you’re advertising in CTV no JavaScript detection can be run. So, if the only detection mechanism is the user agent and the reputation of the IP address you’re easily being conned.

Requestly is a great tool to explain and to show what really happens and how it happens. But, it doesn’t scale! Each ad impression only generates a few pennies at maximum. In order to generate enough money to live a luxury life with big houses, swimming pools, private jets, lazy river pool, luxury sports cars you need a LOT of impressions and clicks. Remember: 30 trillion bid requests per week! That's 30,000,000,000,000!

In a previous post the different types of fraud have been described: request based, browser automation and human operated [1]. Request based is the cheapest way to scale, because it has less overhead. Browser based is way more expensive as browsers allocate and need a lot of memory and CPU. Human operated click fraud is even more expensive. So, let’s quickly go to request based because there is where it happens.

Use Python to automate HTTP requests

As mentioned before, request based automation is lightweight in terms of using resources (CPU and memory). Everyting is command-line based, which means less overhead and thus you can run many instances concurrently.

Figure 4 contains the source code of a minimal Python program that fires HTTPS requests to web servers. These requests are generated out of thin air and contain everything needed to look legitimate at the HTTP level. As a reminder: HTTP requests consists of a method eg. GET, PUT, headers and a body [3].

The code in Figure 4 shows the loading of the wordlists ie. the nouns and adjectives (line 25-26), configuration of the HTTP request (line 28-36), the HTTP headers (line 49-62), the user agent (line 36), and the empty cookie (line 48).

The code in Figure 5 generates a list of 100 fake random Android appnames (line 74-82). At line 87 a loop starts. This loop picks a random name from the just generated appnames list, clears the HTTP headers dictionary, and starts to repopulate it with the headers from Figure 4 (lines 49-62). At line 94 the fake appname is added. At lines 95-96 the cookie is only added if it has any value. At line 97 the URL is constructured of the individual parts. At line 98 the request is fired based on this URL, the querystring parameters, the HTTP headers and a callback function which enables you to capture the response. Last line at 99 prints a logline with time, the HTTP status, method, URL, and the appname used.

This tiny Python script, less than 100 lines, is a boiled down version of the code I used early August. It shows exactly which steps to make in order to construct and fire requests programmatically. And: Yes, these requests don’t match a real Android’s webview TLS fingerprint, they don’t use a mix of residential and mobile proxies, and it loads only the main page of www.FouAnalytics.com. But, if you know how to make code like this by yourself, you also know how to improve this and make it appear like the real thing.

When the Python script is executed three times it generates the following output, see Figure 6. You can see in Figure 5 at line 99 how the output is generated and at each the end of the line are the generated appnames.

To answer the questions at the start of this article. Can a bot, in this case a python script, generate HTTP requests out of thin air, having a freely configurable: domain name, referrer, appname, user agent and HTTP headers with webview, payload, etc. : Yes, of course they can.

The last two questions: Can a bot generate a prebid request? Load the advertisement? Fire the completion pixels? By now you should have been convinced that this is perfectly possible. That also means that once perfected scaling a tech stack like this can be leveraged into making a lot of money. Guess who's indirectly paying for that? The advertisers... ? Nope: You! Because ad fraud means wasted money which is passed on their customers... you. This means less competitive pricing.

Finally, knowing what's technically feasible, the ridiculous amount of prebid requests, and who's paying for it, what can be reasonably done about this?

• Start measuring the quality of your ads, and the quality of visitors arriving at your landing pages [5].

• Perform analytics to see where/ what/ when/ what causes low quality [6].

Measuring and detecting ad fraud is the first step. Analytics the second. The third step is to look at your contracts and see whether you are eligible for refunds or credit traffic in case of fraud, if not then make sure it's included when renewing them. The operational step is to manage the traffic quality and mitigate its impact.

If you have any questions, would like to improve your digital marketing results, suggestions or specific requests feel to connect, comment or DM.

#adfraud #bots #CMO #digitalmarketing #browserautomation #python #analytics

[1] https://www.linkedin.com/posts/augustinefou_another-bot-attack-on-my-site-last-time-activity-7229601769884954625-UEC6?

[2] https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Common_non-standard_request_fields

[3] https://developer.mozilla.org/en-US/docs/Web/HTTP/Messages

[4] https://www.linkedin.com/pulse/why-60-digital-ad-spending-going-mobile-apps-bad-thing-fou-bwbie/

[5] https://www.linkedin.com/posts/kouwenhovensander_adfraud-bots-cmo-activity-7245048677835186178-c6vk

[6] https://www.linkedin.com/posts/kouwenhovensander_adfraud-bots-cmo-activity-7247590450545455105-Wa3L

Ad fraud. A problem ignored by many in the digital advertising ecosystem. Why? Because they make more profit when the problem persists. They are not interested in delivering their clients quality visitors, but invoicing as much as possible is what matters.

Ad-fraud thrives in places where checks and balances cannot be made easily because of the fragmented and distributed nature of the ecosystem.

Financial ad-fraud

One form of fraud is that the reported number of placed advertisement doesn't match reality. The problem of the advertiser is that they can't check every placed advertisement. The result will be an invoice that's 20% too high. Is the advertiser able to check that the reported number and the invoice based on that number has been inflated with 20%? No! Not without detailed per line item logfiles and its analysis.

Another fraud form is: Did the placed advertisements meet the configured constraints and expectations? Like adhering the min max thresholds of bidprices? Is the advertiser able to check this? No! Not without detailed per line item logfiles and its analysis.

MFA (Made for Advertising) sites are fraudulent because they don't meet the advertiser's expectations and minimum standards. Reporting traffic generated at MFA sites as premium URLs is: fraudulent.

Ad-fraud carried out by tech

Besides these forms of inflating numbers or inflating quality which fit financial fraud there's also technical fraud. This type of fraud happens because fraudsters play the ecosystem using advanced technology. The name of this advanced technology is: Bots, or browser automation. The different technical types are described this article 'How does bot and human operated fraud work?' [1]. The previous article in this series described how these can be detected [2]. This article will continue on the previous one by analyzing at the detection results by breaking it down into many different segments.

Ad fraud detection enables advertisers to determine the good players and bad players. But, ad fraud detection on its own isn’t actionable. The output of the fraud detection needs to be broken down by domain, appname. These two are the obvious pivots and enable you to know which domain or app attracts bot traffic to cheat.

Technical fraud hides in averages

Fraud hides in averages. If you run a multimillion dollar campaign and one of your traffic sources contains a high level of fraud it is invisible when you only look at averages. The answer to this lies in analytics. Breaking down the data and using analytics enables you to find fraud patterns in the data.

Ad fraud Analytics

The first step in analytics is data exploration. You know fraud hides somewhere, but you don’t know where because the fraud is hidden in averages. Breaking down the data into segments (source, campaign, country, publisher, ISP, etc) enables you to see which one contains more fraud than others. The only problem with this method is: It takes a lot of time to find the exact segment and the value that contains the fraud.

Analytics: Treeview Examples

Since you don’t know upfront where fraud appears and what its common denominator is you need to start looking at the big picture where you can see all data at once. An efficient way to efficiently look at all data and its fraud is to show the data as a treeview. The treeview shows all data, but broken down to a segment of choice. Each segment is again broken down to human/fraud showing which segment has almost no fraud or a lot of fraud. For example, Figure 1 shows all traffic arriving at a landing page. The visits are broken down to the traffic sources (anonymized in the figure below). Each source has a different fraud level and this single treeview shows them all, sized relative to each other. Human visits are blue and fraudulent visits are toned red. The difference between the red tones is where the fraud has been detected at the request (dark red is request based fraud) or using JavaScript (lighter red).

Figure 1 shows that the three largest paid sources have 11.4%, 12.1% and 9.6% fraud. But it also shows that smaller sources have 37%, 54% and more than 70% fraud. Looking only at the total average would have made these smalles ones invisible as the 3 largest paid sources dont have much fraud and represent ~35% of the total paid volume.

Breaking down the data into states shows where your audience physically resides. This should match your targeting. It also shows where fraud enters the Internet. In Figure 2 can be seen that the state with the largest fraud% is California. This doesn’t tell you much on its own, so we’ll zoom in to the city level.

Figure 3 shows in the right lower corner that Mountain View (CA) has 90+% fraud and represents ~1/3rd of the total fraud in CA. Although this is good information, city as a pivot has some disadvantages: There are too many small cities (the long tail of cities) and as you can see in Figure 3 the left side of the chart isn’t readable in the overview. Although you can zoom and labels will become visible, it is just too much.

Back to Mountain View (CA). Why does it have so much fraud? It’s because a Google Data Center is located over there. This confirms that data center traffic equals bot traffic, which makes perfectly sense. Who lives and browses the Internet from a data center legitimately? Maybe those 2 or 3 individual working in the operations room? Anything else would be VPN traffic, proxy servers, and bots running in those data centers.

To see how fraud is distributed over different ISPs, let’s take a look at Figure 4. This animated GIF zooms in to both Google and Amazon data centers. You can clearly see that almost all traffic from these IP ranges is fraud. Blocking data center IP ranges in you ad infrastructure is a good and cheap way of preventing bots to bid and load your advertisement. Once data centers are excluded bots need to use residential proxies at an extra cost, it adds complexity, the proxies might be detected as fraudulent, and thus affecting their profit margin.

But, what about other added value of breaking down human/ fraud per location or ISP? You could see whether your intended geo-targetting is correct, and you pay for visitors from other countries, or states or even cities. If you target desktop, you don't expect mobile gateways to be used. Secondly, you would be able to see whether fraud originates from a single location outside data centers or new IP ranges recently bought by a data center.

In Figure 5 the same traffic is broken down into appname (Android). The two largest apps in volume don’t have that much fraud, although it would be interesting to see why their percentages differ so much (7.8% newsbreak vs 14.3% facebook).

But, take a look at some of the smaller apps. Those apps have more than 50% fraud, some even 75%. These are the typical apps you want to take a look at and ask yourself: Do I really want to pay for this? Does thise apps really bring new business? Also take a look at that long tail of apps, the left part of the image with all those small rectangles. Although they don’t bring that much traffic, some of those are 100% fraud. Those are the typical ‘non-existing’ apps, with fake app names. Together this long tail of apps ( < 0.5% per App) generates 8.3% of the volume, with ~14% fraud. And, yes, this is data from a client that has been continuously optimizing their traffic quality.

Analytics: Time series examples

The treeview is an excellent way to get a quick overview of what happens, but doesn’t show you the time component of a campaign. You know when you start a campaign, which channel you use, what you have whitelisted (domains and apps) and/or blacklisted (data center Ips), etc. You also know when changes were made and you want to see the effects of those changes as time progresses.

The charts below have the same colorcoding as the previous figures, and they are based on the same data, except the time range. Each bar represent a full day, which smooths things out and shows the bigger trend only.

Figure 1 shows the aggregated results for all sources including the two sources in Figure 6. What can’t be seen in figure 1 is how things progress over time and behave at different volumes. The bottom chart in Figure 6 shows that this source miraculously has 40% fraud at any volume. To me this looks like a configured percentage or lever which sets the percentage of cheap (bot)traffic mixed with real traffic.

The treeview containing the US states showed that California did have a lot of fraud. How does this look when plotting the data as a time series in a chart. Figure 7 shows the Californian traffic. It shows that when the volume goes down the fraud% temporarily goes up.

In Figure 2 the treeview of the states shows that California has a high percentage of fraud, compared to its peers having the same traffic volume. Florida another state with a big volume has a much lower fraud percentage, see Figure 8. So, how do these two compare? Again you can clearly see that the volume decreases around the same date. This time the fraud temporarily goes up a tiny bit, but then goes down to a low level without spikes or other high fraud days. Looking at other states will reveal that states with huge (relatively cheap) data centers were responsible for this temporary increase.

Finally, we’ll take a look at how traffic from Android apps looks like. Figure 5 shows a treeview where you can compare all Android apps to each other, both in volume and in fraud levels per app. But, how does an app look when looking at fraud levels over time in a timeseries. This can be seen in Figure 9, Figure 10, and Figure 11.

Let’s compare the results of the four apps above in the figures 9, 10 and 11. Looking at Figure 5 you can see the relative sizes per app. Newsbreak is the largest one in traffic size per app. Newsbreak is the largest one in traffic size com.particlenews.newsbreak. Second is Facebook’s com.facebook.katana. Other Meta/ Facebook apps are com.facebook.lite and com.facebook.orca which are ranked separately in the treeview. Third place in size is Instagram’s app com.instagram.android. An app with a high percentage of fraud is: com.instabridge.android.

The first three are well known apps, but the fourth one instabridge isn’t something normal users know, at least I didn't know it and I checked it with some colleagues and friends. So, let’s zoom in what this app really is. Figure 12 shows the apps’ information in the app store, the yellow highligted line below says it all: “The Wifi Hunt is over! Fast and Secure Public WiFi Map at no charge”. Perfect for fraudsters needing a temporary new IP address to commit fraud as long as it works. And.. free of charge! OK, apparently some ads do appear otherwise where does this app traffic come from?

Fraudsters love “free” and they even love it more if it is a shared IP address which means also legitimate traffic will originate from the same IP address and/or range. Also, WiFi with a direct connection is faster with a lower latency than using the mobile network or residential proxy servers or a combination. The devices with shady (free) VPN apps that also work as VPN endpoints using these WiFi points will help fraudsters as well. And once it doesn’t work anymore you just move on. Exactly the ingredients fraudsters love.

Let’s take a look again at the bottom chart in Figure 11 which has ~75% fraud. This app, and all other free VPN, free internet connection sharing, and apps with a hidden builtin VPN or proxies without the user knowing are prone to high levels of fraud. The first question you should ask: Does my intended audience use this app? That means the typical users of these apps are freeloaders and only install apps to save some pennies. If you’re Temu, Shein or Aliexpress, then I agree this is your audience! But, if you’re targetting people to sell car insurance, luxury vacations, hearing aids, designer bags, high-quality make-up, skincare and fragrance or just even brand awareness of these products then also agree that the remaining ~25% humans using this app won't fit your ideal customer profile.

Blacklisting these apps might be your first thought, but it takes a week or maybe a month, and they just reappear using a different name. That’s why whitelisting is so much better. Only allow your ads to be shown in the apps you selected. But, do you fear you're missing new apps? Run an A/B test with a small portion of your volume without any black or white lists and see how apps, domains, data centers, etc. perform. If they perform well you can scale up and monitor whether they continue to perform. Rinse and repeat.

Quality of ad-fraud detection is key

One of the key factors to be successful in continuous optimizing your ad spend, campaigns, budget allocation, etc. is the quality of the ad-fraud detection. It doesn’t matter how great or sophisticated the analytics is. GIGO (garbage in = garbage out) will ruin your results. Flagging humans as fraud will cost you business, missing real fraud also hurts your business. And making the wrong decisions based on faulty input is killing your business!

• The collected data is not directly related to browser automation

• The collected data can easily be spoofed at the client side

• The communication from browser to the back-end is not well protected

When that’s the case, and that’s way more often than you might think, can you really trust the collected data? And subsequently the fraud detection engine processing this “faked” data? And subsequently trust the analytics breaking down human/fraud data and displaying It in fancy garbage charts? If there’s a lot of money involved (potentially millions to billions) which can easy be siphoned by fraudsters they will don’t easily give up and continue to hire experts in making bots and other fraud schemes. Your detection company needs to counter that with expertise in the fields of making bots, cyber-security, reverse engineering, analytics. This means fraud detection ideally is: a company with security experts providing a service in the marketing ecosystem, and not the other way around. Because the latter scenario is something we all know as: enshittification [3].

Conclusion

Fraud hides in averages. In order to identify fraud you'll need to look at the fraud detection data from various angles and/or by slicing the data into logical groups. This can be achieved using: analytics. This article has shown how analytics can be applied to reveal fraud, using real data with many examples. Looking at those charts and subcharts enabled you to see that some domains/ apps/ utm_sources/ etc were cheating and some others performed better than average.

• Adopt: Use quality ad-fraud detection. Hint: Oxford Biochronometrics

• Adapt: Adapt your digital spend according the outcome of the ad-fraud analytics

• Improve: Over time improve your results by readapting whenever possible

Feel free to contact me if you’re having doubts about the quality of your current fraud detection vendor and want to run a side-by-side test to compare results.

Corrections? Suggestions? Questions? Feel free to connect, comment or DM

#adfraud #bots #CMO #digitalmarketing #browserautomation #clickfraud #analytics #CFO #CRO

[1] https://www.linkedin.com/posts/kouwenhovensander_taylorswift-adfraud-adfraud-activity-7199384350369431552-t8EZ

[2] https://www.linkedin.com/posts/kouwenhovensander_adfraud-bots-cmo-activity-7245048677835186178-c6vk

[3] https://www.linkedin.com/posts/kouwenhovensander_why-has-digital-marketing-become-enshittified-activity-7105194909032140801-jB0O

[4] Adopt, adapt and improve as speeched by Prince of Wales - Britain's best 'Ambassador of Trade' - at British Industries' Fair. :

Ad fraud. Collectively costing the industry billions/ year, and apparently nobody gives a sh*t. The self-proclaimed and self-certified ‘defenders of the ecosystem’ continuously under-report the problem to the extent of willful ignorance. That makes one wonder: Why? Who profits from this? Where do these billions converge to? And what can be done about this?

A few weeks ago I read a LinkedIn post by John James and he referred to WSJ columnist’s Jason Zweig’s three-part rule [15]:

There are three ways to make a living:

1. Lie to people who want to be lied to, and you’ll get rich.

2. Tell the truth to those who want the truth, and you’ll make a living.

3. Tell the truth to those who want to be lied to, and you’ll go broke.

Certain ad fraud verification companies would clearly fit the 1st category. Those companies “detecting and verifying” fraud supposedly can’t tell you how they do it, because disclosing their secrets would help fraudsters to improve. But, they do send their verification JavaScript to millions of devices, including fraudsters! How does that rhyme? Anyone able to reverse engineer their JavaScript (and that ain’t so hard) knows exactly what data they are collecting to determine fraud. Fraudsters know how to reverse engineer, or hire someone who knows how to. Fraudsters know what data is relevant to detect fraud and actively try to counter it.

Companies hiring fraud verification companies are not IT security companies who can do their own pentesting or code analysis. These companies are large companies (S&P 500) that advertise digitally to promote their products or services across various channels: CTV, programmatic, PPC, social media, email, etc. Apparently, they blindly trust these verification companies. Based on the allegedly percentage of detected fraud, ie. reported is < 1%, one would wonder why they do even verify their digital spends. If there isn’t a big problem, why spend so much money on it? Is it just to check the checkbox in order to be able to claim: We are compliant? If so, that’s probably that company’s most expensive checkbox.

The opposite of ‘ignorance is bliss’ is: Knowledge is power. Having knowledge means being in control. The majority of people working in the digital marketing ecosystem don’t have a master in computer science. Hence, they don’t fully understand what is technically possible, feasible, tell the difference between a poor and a good quality ad fraud verification solution, are easily intimidated with technical terms. Unable to ask the right questions, and without having sound knowledge to see through false promises, makes you indeed the party being lied to of the 1st way to make a living.

Now, you might think: “Whatever...” or “Who cares? That’s the price of doing business, as long as I reach my audience.” But, there are more important things than you reaching your audience. For example, elections. According eMarketer the total US political ad spending will hit $12.32 billion in 2024, up nearly 29% from the prior presidential election in 2020 [16]. Burning only one party’s budget using a botnet preventing to reach humans does make a difference in swing states. And I’m sure that you don’t want your digital marketing budget being spent on funding fake news websites, misinformation in order to change the outcome of elections, indirectly fund bad regimes, or just spend on anything else than your campaign goals.

So, how will this blog series help you? By trying to explain and educate you on the foundations of fraud detection. What types of detections exist, what is possible to achieve and what is not possible. It will show you real scenarios with real output. You don’t have to fully 100% understand everything as long as you know whom to invite to ask these questions during your fraud detection vendor selection process once your contract expires or you’re not satisfied with the current one and you’re going to select a new vendor.

You might think that this post would also educate fraudsters. Yes, maybe the script-kiddies, or students. But, rest assured, the professionals already know the content of this blog by heart. It’s their living.

Oxford Biochronometrics focuses on detecting fraud in lead generation based on human interactional behavioral analysis, but that doesn’t mean we’re not able to detect ad fraud. That’s why this blog describes all fraud detection mechanisms from top to bottom of the marketing funnel. It starts with pre-bid, impressions and video ads where tracking pixels are used. Subsequently, two distinct classes of fraud detection using JavaScript are described. The last part is about fraud detection based on how the application is used by looking at the interaction.

The next part of this series will cover how analytics of the fraud detection data can be transformed in to actionable data.

Ad Fraud detection: The techniques

There are two main types of ad fraud detection at the client side where the browser is involved: Request based aka tracking pixel, and JavaScript based. In Figure 2 can be seen that JavaScript can logically be split in three groups, where each extracts data based on a different extraction method. Besides the data conveyed by the JavaScript also technical features of how the data was sent, ie. TLS fingerprint, network packets, are captures and used to determine ad fraud. All these layers of detection create a complete picture of the client side, the transport and whether that smells like fraud or not.

Technique 1: Tracking Pixel

A tracking pixel, technically an <img> tag in HTML, makes a request to the bidder’s server when a browser loads an advertisement that includes such a tracking pixel. Loading the pixel means that the browser requests and downloads the 1x1 pixel. It is not about displaying the pixel, but about conveying data. That’s why the browser’s requests contains a payload that will contain data such as names or ids of campaign, source, and media details in order to tie back the pixel to the advertisement and make the data actionable.

• a video advertisement is loaded

• at video completion percentages (25%, 50%, 75%, 100%)

• at specific pages to track conversion and/or transaction completion

In order to get a better feeling of what REALLY happens, let’s take a look at Figure 3. It shows how the browser sends such a request, its headers, its payload, the size, the response, etc.

In Figure 3 three purple boxes can be seen. The box with number 1 contains the full URL of the request, where the payload is everything after the question mark. The data is encoded in protobuf format [4]. The box with number 2 shows a positive response: status code 200. The box with number 3 shows that an 1x1 image is returned.

But, as you would expect: Simple pixel requests by a browser can be faked!

Let’s show you how that can be achieved.

The ‘Copy as cURL’ in Figure 4 shows how to generate a full cURL request just like the browser did. The request includes all headers, eg. user agent, referrer, and also all custom headers. This will be copied onto the clipboard, as text. This can be copied to a terminal box and pasted and executed, which can be seen in Figure 5. The only addition I had to make is the --output gen_204.gif, see 3 in Figure 5, because images are binary and thus require a filename.

As you can see both the request by the browser and the request by cURL from a command prompt were accepted by the web server, both times the server returned a HTTP status 200. The status cannot be seen in Figure 5, but running the same command in verbose mode shows the status: HTTP/2 200

This cURL example also shows that a browser is nothing more than a serial HTTPs request machine with some additions, like: session cookies, persistent storage, scripting, APIs, a GUI, an interface. These are important for humans, but not for bots! On the contrary, when committing ad fraud they are expensive in terms of CPU and memory.

Pre-bid

Opening a webpage with advertisements means that your browser starts a sequence of events in order to show these advertisements. The initial event is pre-bidding, aka header bidding. This enables SSPs and exchanges to bid on an ad-slot in a browser. Each of the available advertisements has a price range (min. and max. price), and the one with the highest bid within this range will win the auction and is thus being displayed in the ad-slot.

As you can see in Figure 6 a pre-bid request is, just like a pixel, a simple HTTP(s) request. This means it can be generated from the command line as well, just as easy as the previous example. Fraudsters love command lines, because it is CHEAP and SCALES well. Ever tried to start and open 50 Chrome browsers on your computer? Now try the same with 50 terminal boxes and requests, that’s very doable. It becomes even better when a single terminal box with a program that starts and fires 50 concurrent GET requests. That’s so lightweight, you wouldn’t even notice running this on your laptop or PC.

In a browser you will be using the JavaScript’s implementation of pre-bid. This piece of code will fire prebid requests based on the web page's configuration and configuration on behalf of advertisers, publishers, and middlemen in the ecosystem. The current prebid code size is 176kByte [8] and that’s the minified JavaScript! Although the size is large, AFAIK, it doesn’t do any security checks or data collection on behalf of ad fraud verification or detection. The alleged checks are done server side during pre-bidding, but what these checks and validations exactly are? Maybe ask Baghdad Bob?

Having tested firing prebid requests programmatically using cURL as simple test and Python to create more complex and fully configurable flow of prebidding and loading ads and firing pixels does actually work. In a separate future post I will show how this works step by step and how you can emulate a browser, or pretend to be com.some.arbitraty.random.app doing a prebid, capture the returned answer, load advertisements and fire completion pixels. All without a browser only using Python code and a series of requests. And what I can do without being paid, fraudsters are able to do as well; though they are getting paid, indirectly through companies' digital marketing spend.

CTV

In CTV (connected TV) advertisments can also be requested using pre-bid [6][7]. In CTV they are used to place bids on the available pods (commercial breaks). While the advertisements are shown completion pixels are fired at 25%, 50%, 75% and at 100% completion. I keep repeating myself, but technically these pixels can simply be generated programmatically and are similar to pixel requests.

In CTV no JavaScript can be executed to verify the client’s device and application. This means that the only information available at the verification side to make a decision upon is the incoming request and its payload.

Streaming music, podcasts, and more

Streaming platforms offer music, podcasts, video. About two weeks ago I read a nytimes article about a man that generated tons of new music using AI, placed this music on the platform, and streamed those tracks using bots [9]. He allegedly played his tracks BILLIONS of times using bots. Curiosity killed the cat, and greed killed this one! To me it implies those platforms didn’t have any fraud detection at all, that's why it persisted for such a long time at $110k/month!

A few years back I had a Deezer subscription, which allowed me to listen to music in lossless. Normal usage would be my browser or the Deezer App connecting to the streaming platform and listen to the music. Technically listening to music is simply a series of requests from my device to the platform which returns you some audio based on the request’s payload (ie. IDs, artist, album, song, etc).

An application called Deemix allows you to download audio from streaming platforms and store them as files on your computer. This enables you to make a local copy and/or collection of albums. It also downloads music much faster than the normal listening speed. Downloading 10 albums is a matter a seconds. This confirms that no fraud detection mechanism has been implemented. Why? The next section will explain why.

Technique 1: Tracking Pixel -- Fraud detection

So, how can ad fraud be determined purely on requests? The first question would be: What data is available to make a decision upon? In case of requests that’s not really much; Only data associated to the request:

1. The conveyed payload querystring or HTTP post data

2. HTTP headers such as: user agent, sec-ua headers, origin, referrer, cookie, x-forwarded-for, etc.

3. IP address

4. Network packets, eg. TLS handshake fingerprint, TCP fingerprint

The first two are generated by the browser, and as a fraudster this can be emulated 100% with a few lines of code; You only have to capture some real session data and replay the session, see figure 4 and 5.

Self declared bots don’t change their true user agent. They are considered ‘good bots’ and are often allowed to access the content of web sites. Simply because based on the those bots and their work new customers using search engines will find your website. Examples of self declared bots are:

'DuckDuckBot-Https/1.1; (+https://duckduckgo.com/duckduckbot)'

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/100.0.4896.127 Safari/537.36

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.6533.99 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

IAB Techlab continuously updates a list of known spider, crawler and other known bot user agents, which is available at an annual fee [18]. This list would be the low hanging fraud layer to flag and detect fraud by known bot user agents and thus preventing a bot to successfully prebid and load an advertisement.

Self declared bots typically run in data centers and don’t use residential proxies to hide their IP address. Using a (block)list with IP subnets of data centers would be an additional layer to prevent ads being served to traffic (bots) originating from data centers. They can still be allowed to load the page in order to scan content, but without the ads.

Fraudsters running their bots in a data center to load advertisements will have to change their IP address using residential proxies at an additional cost.

The TLS fingerprint is not hard to change, but requires more technical knowledge. But, more importantly: It prevents easy scaling as different applications and thus browsers (sometimes per version) have different TLS fingerprints. In CTV different TV brands and sticks do have different TLS fingerprint values, differing from version to version. This prevents fraudsters from just simply spoof a user agent (UA), because the combination between UA and TLS fingerprint has to match. It also requires a constant validate and when needed update their list of fingerprints, as new devices, updates of software on those devices, and new browser versions are released monthly.

To illustrate this let’s show you the difference between programmatic requests and real devices used by humans. Figure 8 shows the TLS fingerprint (based on JA3 hash) for different versions of Firefox. Only the last 4 characters are shown per fingerprint and as can be seen the JA3 fingerprint does not change very often. Version 115 and 116 share the same fingerprint, and since version 117 every Firefox release has had the same TLS fingerprint (based on JA3 hash).

But, there’s TLS fingerprint trouble on its way. Chrome started to randomize the order of the ciphers since version 110. The request application cURL exchanged OpenSSL for LibreSSL from version 8.4.0_9 onwards and as a result its TLS fingerprint started to appear as a random value. Figure 9 shows that versions prior to 8.4.0_9 share the same fingerprint (blue), and newer versions show a random fingerprint (red). As cURL is open source some OSes, eg. Windows 10, have their custom compiled version of cURL, and in the case of Windows the cURL fingerprint is static.

Other request based libraries and/or applications have other fingerprints, most of them are relatively static and resemble the fingerprints of the underlying libraries. But, it is just a matter of time before they start appearing as randomized fingerprints.

When applications, like browsers, started randomizing their TLS fingerprint (the JA3 Hash) it made it harder for fraud detection tools to automatically flag and/or block requests. Blindly blocking unknown or unassociated TLS fingerprints may cause a lot of false positives. For example, at large companies your browser on your workstation isn’t able to directly connect to the web sites on the Internet, because there’s a proxy server in between. This proxy server decrypts your secure session, checks the data going back/forward (malware detection, scanning for sensitive company information to be leaked, etc.), and re-encrypts the session to communicate with the web server. The fraud detection at the receiving end will see the TLS fingerprint of the proxy server, which is completely different compared to the fingerprint associated to your browser’s user agent. Automatically flagging without understanding what you're flagging can be a recipe for false positives! If you work in a corporate network and certain sites are randomly blocking you, or harassing you with CAPTCHAs without reason, now you know why!

cURL impersonate

In order to emulate the behavior of real browsers fraudsters have started using special versions of cURL. cURL impersonate [13] is such a version and enables you to emulate the TLS handshake of a real browser. This means that looking at the JA3 hash requests appear to be from genuine browsers. This enables fraudsters to pretend to be a popular browser, and spoof its associated fingerprint. Request based fraud is cheap and easy to scale and making the requests look like normal browser requests is the icing on the cake for fraudsters.

What does this tell us? TLS fingerprint enables fraud detection solutions to determine whether the application is lying about its appearance. Fraudsters spoofing the TLS fingerprint to match the user agent are invisible at this detection level. Randomization of a TLS fingerprint makes browsers stand out because of their uniqueness. Though, at the detection side you can still avoid this by countering the randomization, eg. using sorting, though it is less granular, it is somewhat effective.

So, if an application knocks on the digital door of your Ad server, CTV, API, or music streaming platform and its hash shows that it is a known hash of an old cURL (or: wget, python, golang, electron, jdownloader, deemix, etc) version pretending to be a popular browser you should be aware of that. You might block them by dropping the request, or send them to a slow web server, etc. But, in the end, you should be aware of it and the number of (dropped) bot requests should be in your monthly usage statistics.

Spoofed TLS fingerprints generated by a modified versions of cURL do make TLS fingerprinting ineffective. There are no direct solutions to this, only indirect ones, and by having multiple detection layers, such as: JavaScript.

Technique 2: JavaScript -- Fraud detection

JavaScript can be used to execute your code in order to probe, read and convey information from a browser to the fraud detection server. A common architecture is that the fraud detection the server will make the fraud decision based on the data the client conveyed. In order to make that decision three different classes of data can be collected from the browser.

The three classes of data collection in JavaScript are:

1. Direct. Reading properties from the browser’s configuration and Web APIs

2. Challenge/ Response. This providing a (random) challenge, where the browser generates a response to that specific challenge

3. Human behavior. Recording human interactional data like scrolls, clicks, window resize, visibility, etc.

The paragraphs below contain a description and detailed examples for each class.

JavaScript Class 1: Extracting properties

Every browser has different a different configuration, its unique settings based on the hardware, environment and operating system. These can be read by JavaScript code executed in the browser. The combination of these properties is considered to be unique enough to track an individual over the Internet. Examples of such properties are:

• Screensize, color depth, pixelratio

• Preferred language settings, time zone

• Querying installed fonts, querying browser extensions, codecs

• Number of CPU cores, memory, heapsize, platform

• Permissions of webcam, microphone, GPS

• Applepay available? battery status, supported speechSynthesis languages and voices

• CDP (Chrome devtools protocol) signatures

• And oh so many more properties can be collected

In normal browsers these properties are relatively static and provide a good basis for a persistent fingerprint. But, fraudsters rotate these values, or buy collected fingerprints from real people and simply apply those values in their browsers to match someone else’s fingerprint. This makes their a browser instance able to resemble many many different users.

Fraud detection will of course look at how coherent and logically the collected data is. If a declared iPhone doesn’t support Applepay but does have a Chrome object that’s a big red flag. These basic checks are still necessary and will flag amateuristic fraudsters, just as bots forgetting to spoof their TLS fingerprint.

About a year ago (Apr 2023) Google implemented a new headless mode in Chrome. Prior to that update the fingerprints of a headless and headful browser were quite distinct. But the new mode resembles the headful mode except the value of the property navigator.webdriver. If it returns the value true it is a browser controlled by software, if false a human is interacting with the browser. But, as you can imagine this value will be spoofed and returns false. Every fraudster and their mother knows this!

One of the ways to detect browser automation in this new headless mode is to see whether the chrome driver is enabled and communicates with the browser. The scraping world's answer has been to create ways to remote control a Chrome browser without the driver [17]. This technology can also be used to load pages with advertisements, interact with landing pages, etc.

Unfortunately, you don’t catch professional fraudster with reading properties, because they know exactly what to patch and how to patch or simply bypass it. To catch the professionals you need to go to the next level: Challenge / Response detection.

JavaScript Class 2: Challenge / Response

In security there are three common forms of authentication. Something you know, something you have and something you are. Reading properties (class 1) using JavaScript can be seen as ‘something your browser knows’, though unlike passwords the properties in a browser aren’t secret but change from device to device. The ‘something you have’ can be seen as an authentication cookie, access token, etc. The ‘something your hardware + browser is’ is similar to a challenge / response (class 2).

The challenge is to write a piece of code that can be run on every browser and generates different output based on the configuration of the browser, operating system and hardware. Different systems with the same Operating System (OS) and hardware will generate the same responses per challenge. Because the challenge is randomized it is hard -or near impossible- to predict the associated response.

Figure 10 shows the implementation flow of the JavaScript challenge/ response. The JavaScript will execute a test that depends on the configuration of the OS and/or type of hardware. At (1) the server generates a unique JavaScript that is sent to the browser. The browser needs the OS and/or hardware to execute this task (2). Based on the how the browser has implemented this, the configuration of the device, and the available hardware an answer is returned (3). That answer is captured and conveyed to the server that generated the challenge. In this scenario there is no right or wrong, an answer will always be generated and returned. Whether the response can be associated to known bots, normal browsers that’s up to the fraud detection engine.

An example: Different operating systems have different fonts installed. Some fonts are available on Mac, some on Windows, etc. As fonts are vector based they are rendered by the browser on the spot. This means that different browsers might render the same font differently, the availability of a glyph [14] in that font, and the font fallback when the requested font is not available on the device.

This becomes apparent when rendering emoticons. If available, the emoticon is rendered, if not a box is rendered. It depends on the browser and available fonts whether a full color, simple color or black/white version is rendered. The code example in Figure 11 shows how the unicode character 128522 is displayed in a browser using the Sans Serif font. Subsequently, the size of the bounding box is displayed as width x height below the emoticon. This challenge can be randomized by changing the font, fontsize and the emoticon.

Would such a simple piece of code generate different output using different browsers? Yes, it does! The test doesn’t even count the number of colors used, or calculate a hash of the emoticon. Figure 12 shows how different this single smiley is rendered based on the few devices and browsers I have. At scale, looking at millions and millions of devices, these tests will show what is human, what is a bot, or is an outlier and needs further inspection.

Another example of challenge / response tests is Google’s Picasso: Lightweight Device Class Fingerprinting for Web Clients [11]. This research shows that by looking at the individual pixels when different browser versions render the exact same shapes (the challenge) and how the results differ per browser engine and version (the response). Based on the response the verification solution knows exactly which browser and engine version did run the test, which can be checked against the provided user agent. The next level to this type of test is to include the differences per GPU in the mix. And as we all know scaling different types of hardware is expensive.

The fraudster’s answer to this type of tests has been to stop spoofing user agents and use the browser associated to the reported user agent. Keep this version up to date which makes you blend in with what everybody is using. But, their problem is the variety of hardware. As they typically run their browsers in the cloud, preferably in containers, they use the cloud's hardware. This means they're caught between a rock and a hard place and to prevent detection randomization of the challenge/ response result has been the answer. This means each fraud detection running in their browsers becomes an outlier. To prevent false positives outliers cannot be blocked/ flagged blindly; but can only be blocked using rate limiting rules way above normal human thresholds.

The last few years this has been a whack-a-mole – cycle of updating detection by fraud detection companies and subsequently patching their browsers by fraudsters to avoid detection, etc. The benefit of forcing fraudsters into this corner is that the browser based solution doesn’t scale well at very low costs. Using rate limiting per collected fingerprint per IP address and/or IP subnet forces fraudsters to use proxies which is again increasing costs.

JavaScript Class 3: Capturing human interaction behavior

The upper part of the marketing funnel, ie. pre-bid, impressions, clicks on impressions, does not have much human interaction. At most some visibility (in viewport/ out of viewport) data can be collected and hopefully a click.

The lower in the funnel the more the prospect will interact. At the landing page you expect the prospect to read about your product and once convinced the lead generation form is filled out. Filling out a form will have a lot of user interaction: clicking, typing, touching, zooming, etc.

Behavioral analysis =/= Behavioral statistics

Counting the number of clicks, or the click:scroll ratio, avg mouse speed per second, mouse acceleration, mouse trajectory distance, etc. is creating a series of behavioral statistics. This isn’t capturing the underlying behavior. It’s like measuring and trying to classify a chess game by measuring the distance each piece has made over the board: it says nothing about the strategy of the game and its outcome. Using these simplified behavioral statistics is prone to false positives.

Behavioral analysis goes much further than flattening and averaging behavioral data. This is the area where Oxford Biochronometrics excels, and where the company name comes from: Bio = life, Chronos = time, Metrics = measuring stuff. Measuring (human) behavior over time. Automated behavior from a bot over time differs completely from human behavior. Human operated fraud also differs from regular human behavior.

Compare it with simple handwriting differences. Where typewriters are the bots and handwriting belongs to a human. The difference between the two is like day and night. The differences between human operated fraud and regular human usage are more subtle. Showing these differences would be a long post on its own and beyond the goal of this article. If there’s enough demand for such a post, I’ll might write an article about OxBio’s former behavioral analytics models, let me know in the comments.

Final words

Your takeaway would be to realize ad fraud exist and is a lot more than the reported <1% simply because to fraudsters it's lucrative and almost risk free.

The opposite of ‘ignorance is bliss’ is: Knowledge is power. Having knowledge means being in control.

In prebid, CTV, streaming, TLS fingerprinting is a must in order to prevent fraudsters to scale easily and massively like that dude streaming his own AI songs. But, the pre-bid filtering, ad verification or fraud detection will at minimum have to collect and validate these fingerprints. Though, firing requests using modified versions of cURL are possible, there are other techniques that would add layers of detection, eg. TCP fingerprinting and still enable quality detection of fraud.

By educating marketers and other interested folks the foundations of fraud detection, ie. What types of detections exist, what is possible to achieve and what is not possible, is the first step for brands to become more resilient. You don’t have to fully 100% understand everything in this post and know how to do this yourself. As long as you know whom to invite to assess the vendors technically and ask difficult but necessary questions during the ad fraud vendor selection process, you’re safer than believing the sales representatives of those vendors.

Next article in this series will be about how analytics is used to make the detection actionable.

Corrections? Suggestions? Questions? Feel free to comment, connect, or DM

Appendix

Links mentioned in the article. They are not listed in order of appearance.

[1] https://iabtechlab.com/wp-content/uploads/2022/04/OpenRTB-2-6_FINAL.pdf

[2] https://support.google.com/admanager/answer/6123557?hl=en

[3] https://helpcenter.integralplatform.com/article/publisher-verificatiohttps://support.google.com/admanager/answer/6123557?hl=en

[3] https://helpcenter.integralplatform.com/article/publisher-verification-implementation-guide

[4] https://protobuf.dev/overview/

[5] https://docs.prebid.org/prebid/prebidjs.html

[6] https://docs.prebid.org/formats/ctv.html

[7] https://files.prebid.org/docs/Prebid_for_CTV-OTT.pdf

[8] https://docs.prebid.org/download.html

[9] https://www.nytimes.com/2024/09/05/nyregion/nc-man-charged-ai-fake-music.html

[10] https://en.wikipedia.org/wiki/List_of_typefaces_included_with_Microsoft_Windows

[11] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45581.pdf

[12] https://daniel.haxx.se/blog/2022/09/02/curls-tls-fingerprint/

[13] https://github.com/lwthiker/curl-impersonate

[14] https://en.wikipedia.org/wiki/Glyph

[15] https://jasonzweig.com/three-ways-to-get-paid/

[16] https://www.emarketer.com/press-releases/2024-political-ad-spending-will-jump-nearly-30-vs-2020/

[17]

[18] https://iabtechlab.com/software/iababc-international-spiders-and-bots-list/

The typical flow of a conversation during the first call with a new customer having fraud problems can be boiled down to this:

Client: “What are bots capable of in 2024?”

Me: “Quite a lot! With a big laugh.”

Client: “Haha, but... realistically, what exactly?”.

Whenever I talk to new clients the conversation will have this sequence of questions. That’s why I made this FAQ from a technical point of view. What can bots fake, spoof, manipulate, alter, etc. in order to appear as a real genuine browser and a genuine human sitting behind the screen.

After the initial questions made by the marketers, the tech people jump in. They ask the more difficult questions which are very specific and based on research they did to solve their problem or research prior to the call. I have collected and combined the most asked question into a FAQ.

This article contains in-depth and detailed answers to 10 questions about bots, you’ve always wanted an answer to. It has been written for marketing professionals looking for ad-fraud and/or lead generation-fraud solutions where this FAQ can be used as a guideline in order to ask these vendors: "How does your solution detect a bot faking ... ?"-FAQ

1. Are bots able to fake the domain name of the website they are accessing?

The domain name can be read from the browser using JavaScript by reading the values window.location.origin, window.location.host, window.location.hostname and window.location.href properties [1]. Browser automation, ie. advanced bots, are capable to load webpages and execute JavaScript. These bots will remote control the browser in all its glory. This is achieved by starting the browser and controlling each tab using the CDP protocol (Chrome Dev Tools), giving instructions to click, scroll, type, etc. This works for Chromium based browsers, eg. Chrome, MS Edge, etc. but also Firefox and Safari have similar protocols, though less common.

The example in figure 1 is a post-bidding example. But, in pre-bidding the browser will send one or more bid-requests for an advertisement. This request is again fired from the same browser and contains plain text, which can be faked to anything the bot wants.

So, can bots fake the domain they are on? Yes, they can both pre-bid and post-bid.

2. Are bots able to fake the referrer URL?

The referrer URL informs 3rd party web sites which URL is visited when a resource is requested. For example, if you are browsing to www . usatoday . com and a JavaScript is loaded from a different domain, eg. static.adsafeprotected.com, the referrer header is set. Figure 2 shows the referrer header and its value. These values can be changed by bots by intercepting the network traffic within the browser, overriding the value to whatever they want, and send the modified traffic.

3. Are bots able to fake the Mobile App name?

Advertisements in Mobile Apps are displayed using a web browser that is embedded within the App. This embedded browser is called WebView. Within this webview an advertising platform loads and refreshing ads based on your cookies, geo-location, etc. For example, Google’s Admob is such a platform [2][3], InMobi, AppLovin, Glispa, Amobee are prominent alternative platforms.

When an App accesses a website using WebView the appname is conveyed in the HTTP header x-requested-with. Figure 3 shows the communication captured with a MITM (man in the middle) proxy between the FT App installed on a real Android phone and Google. The highlighted blue line shows the x-requested-with HTTP header. This is of course only a single request made from the App. Another interesting fact which can be seen in Figure 3 is the User Agent (UA). Apps have full control over the UA and in this case you’ll see the Appname and full version are included in the UA as well.

4. Can bots fake utm_ parameters and other forms of link decoration?

Querystring parameters like utm_source, utm_campaign, gclid and/or fbclid are technically nothing more than a ampersand separated string appended to the GET request. When bots load the advertisement in the browser, they already know what the target link will be. They can simply change the parameters upfront, and click and let the browser do its work. Another method is to dynamically change the parameters using request interception. If a bot already uses this technique to change HTTP headers, referrer it is fairly easy to add some rules to rewrite destination URLs. So, yes they can. Easy!

5. Can bots fake cookies in the browser?

Browsing to a website means that all cookies in the browser associated with that URL are sent along with the request. Figure 5 shows how this looks at the website

So, can bots fake cookies? Yes, they can and will warm browsing session in order to maximize their profit.

6. Can bots fake the fingerprint of the browser?

Using CDP (Chrome Devtools Protocol) any property or value in the browser can be overridden. This enables fraudsters to change values like: Screen resolution, keyboard language, available plugins, time zone, webGL vendor and renderer, etc. Combining these property values (and many others) are called the browser's fingerprint. This also means if a value changes, the fingerprint changes. CreepJS is an open source tool to calculate your browser's fingerprint. When developing a bot creepJS is typically your litmus test.

In addition to fingerprint a series of static values, it is also possible to fingerprint responses to challenges. For example, WebGL shapes are drawn and because different OSes, browsers, videocards and its drivers may use different anti-alias methods, have different IEEE754 floating point implementations, use different rounding modes, etc. the color values of individual pixels may thus differ per video card type [4][5].

7. Can bots fake the TLS fingerprint of the browser?

TLS fingerprints are generated server side. It is based on the client-server handshake prior to the encryption of the communication. In this handshake the browser sends: Hello, I support these encryption cipher suites. The server answers with the selected cipher and key (simplified) [7]. Different browsers on different OSes support different cipher suites. This is most relevant for request based bots as by default their fingerprint does not resemble any browser [23]. That’s why request based bots will have to use special clients and tools. For example curl-cffi (see Figure 7), curl-impersonate, AzureTLS and CycleTLS [8][9][10].

Browser based bots need to connect to a proxy in order to change the fingerprint. In such a setup a proxy server will setup the secure connection to the web server with the publisher site and/or landing page. The proxy will forward the requests on behalf of the automated browser. In this case the web site (and its fraud detection) will fingerprint the proxy server's requests.

8. Can bots fake (prevent) WebRTC from leaking your real IP address?

WebRTC (Web Real-Time Communication) is the technology that enables videoconferencing from a browser [11]. WebRTC uses point to point communication bypassing proxy servers configured in the browser. In fraud detection WebRTC can be used to detect the real internet facing IP address of the client, even if the client is using a proxy or VPN. The detection can be split in two parts: Capturing the IP address server side and extracting the local IP address(es) at the client using JavaScript.

WebRTC Server side

In order to detect bots and fraudsters using residential proxies anti-bot detection companies have setup their own WebRTC infrastructure. Cheap and low quality VPN clients allow anti-bot and fraud detection companies to extract the true external IP address the bot or fraudster uses. Premium quality (non-free) VPN and proxy software typically don’t have this issue.

Figure 8 shows a screenshot of the browserleaks’ WebRTC test page [13]. Both the IPv4 and IPv6 addresses in the screenshot are located in New York, United States. I made this screenshot in the Netherlands, so my local IP address did not leak while I was using ProtonVPN. Other VPN clients or residential proxy services may have different results.

WebRTC Client Side

In 2015 Daniel Roesler exposed a WebRTC vulnerability on his github page [12], see also Figure 9. This vulnerability enables code running at the client to know its external IP address, even if the client is part of a local infrastructure with local addresses, eg. a corporate network or your home network, behind a firewall.

Depending on your network configuration the JavaScript code on the github will be able to extract your local IP addresses of your device. In case of IPv4 the code will typically extract your internal network IP address, which is in most cases is a NAT (Network Address Translation) address, eg. 192.168.x.x or 10.x.x.x. However, with IPv6 the need for NAT disappeared. That means in many cases this technique reveals your true IPv6 address, and thus your true location, even though you are connected through a VPN, and behind a router/ firewall.

9. Can bots solve CAPTCHAs?

With the rise of AI you would expect that bots will be able to solve all CAPTCHAs automatically. That is correct up to a certain degree [14][15]. Images containing text, or simple image recognition is achievable with high accuracy. Figure 10 shows examples of CAPCHAs which can be solved automatically.

More recent CAPTCHAs have become puzzles based on knowledge, where you need to have some subject knowledge in order to solve the CAPTCHA. Sometimes they even resemble an IQ test. Figure 11 contains a few example CAPTCHAs that require general knowledge, eg. animals that lay eggs, the usage of objects eg. vehicles on paved roads, and/or the monetary value of goods.

10. Can bots fake human interactions like mouse movements, clicks, scrolls and/or touches ?

When a browser is controlled by browser automation software it is able to move the mouse to new locations. In CDP (Chrome Devtools Protocol) mouse movements, clicks and scrolls are controlled by dispatchMouseEvent [16]. This enables a developper to fully control the mouse and its buttons and wheels. The same accounts for touch events by using dispatchTouchEvent, which can be used to emulate mobile behavor [17].

In order to simulate human behavior humanlike mouse paths must be generated instead of straight lines. This can be achieved using b-splines [18] or bezier curves. The software is able to generates a series of X,Y points based on a starting and an end-point based on the coordinates of elements in a page. The second step is to calculates a spline curve and timestamps how fast the mouse should move from the starting coordinate to the destination coordinate and at what time resolution. This technique enables fraudsters to perform humanlike mouse movements. This is exactly what mouse synthesizer [19] (see Figure 13) and ghost cursor do [20]. But, don't worry, of course this can be detected as no human is able to make perfect round curves using a mouse.

Conclusion

You might think why didn’t you write something about blacklisting IP addresses? If you’re able to create a sophisticated bot able to buy #taylorswift tickets then you’ll KNOW how to spoof, fake and/or emulate browser functionality. You are well aware that you have to use residential proxies, which means IP blacklisting will cause false positives. Filtering on IP addresses only works to exclude traffic from outside your country, but don’t forget if you’re in the US to include overseas territories like: Guam, American Samoa, Virgin Islands, etc.

You are aware that bots are able to fake (almost) anything. They can’t fake dynamic WebGL / GPU challenges but they will poison these challenges with noise in order to hide their headless appearance. CAPTCHAs do work to a certain degree, but they do annoy humans and thus cause friction to their journey.

So, can these sophisticated bots be detected? Sure they can. Once you know how bots override properties, fake answers, and hide their true appearance you know where and what to look for. Lastly, browser automation will cause “browser automation”-leakage. Spotting these leakages and traces of automation will reveal the true nature of the bot accessing your website.

Questions? Corrections? Remarks? Need help with bots and fraud? Feel free to connect, comment or DM

#adfraud #bots #CMO #digitalmarketing #browserautomation #clickfraud

[1] https://developer.mozilla.org/en-US/docs/Web/API/Location

[2] https://developers.google.com/admob/ios/browser/webview/api-for-ads

[3] https://developers.google.com/admob/android/browser/webview

[4] https://elie.net/static/files/picasso-lightweight-device-class-fingerprinting-for-web-clients/picasso-lightweight-device-class-fingerprinting-for-web-clients-paper.pdf

[5] https://cdn.elie.net/static/files/picasso-lightweight-device-class-fingerprinting-for-web-clients/picasso-lightweight-device-class-fingerprinting-for-web-clients-slides.pdf

[6]

https://privacybadger.org/

[7] https://en.wikipedia.org/wiki/Cipher_suite#TLS_1.0%E2%80%931.2_handshake

[8] https://github.com/lwthiker/curl-impersonate

[9] https://github.com/Danny-Dasilva/CycleTLS

[10] https://github.com/Noooste/azuretls-client

[11] https://en.wikipedia.org/wiki/WebRTC

[12] https://github.com/diafygi/webrtc-ips

[13] https://browserleaks.com/webrtc

[14] https://arxiv.org/abs/2307.12108

[15] https://arxiv.org/abs/2307.10239

[16] https://chromedevtools.github.io/devtools-protocol/tot/Input/#method-dispatchMouseEvent

[17] https://chromedevtools.github.io/devtools-protocol/tot/Input/#method-dispatchTouchEvent

[18] https://en.wikipedia.org/wiki/B-spline

[19] https://github.com/MIMIC-LOGICS/Mouse-Synthesizer/tree/main

[20] https://github.com/Xetera/ghost-cursor

[21] https://www.mimic.sbs/antibot/On-Anti-Bot-Biometric-Protections.md/

[23] https://www.linkedin.com/posts/kouwenhovensander_taylorswift-adfraud-adfraud-activity-7199384350369431552-t8EZ

[24] https://github.com/salesforce/ja3

[25] https://chromestatus.com/feature/5124606246518784

[26] https://github.com/yifeikong/curl_cffi

How to scale your lead generation campaigns without scaling problems such as fraud, TCPA violations and its risks?

Most B2C companies generating leads have never experienced the luxury of knowing which lead may convert and which lead will never ever convert, simply because it is fraudulent. Some companies have added bot detection, but these detections are still circumvented by human operated fraud. Some have added CAPTCHAs to their landing pages to prevent bots from filling out contact forms. Unfortunately, in lead generation the profits are high enough that fraudsters are able to afford paid CAPTCHA solver services, eg. 2captcha, and you didn’t solve your fraud problem. Humans, however, will be annoyed by CAPTCHAs as it affects their user experience. It only creates friction and frustration. Some hate CAPTCHAs so much that they immediately close their browser tab and you’ve lost a prospect.

So, what does work?

Frictionless fraud detection that detects both bots and human operated fraud and conveys the fraud status in real-time with low latency to you. The fraud status needs to be conveyed to you prior to making the call. This prevents you from following up on fraudulent leads. The fraud detection needs to be accurate, preventing flagging real humans as fraud (ie. false positives) and thus missing business. It also needs to prevent fraud slipping through (ie. false negatives). To some this may sound like science fiction or wishful thinking, but this is what our clients experience. The result is:

• You know upfront which generated lead is real, and which lead is fraudulent

• Your call center only contacts real leads submitted by humans interested in your product or service

• Your campaign data and analysis of the campaign’s audience and performance will be based on ‘clean data’

• Clean data means you will be optimizing towards humans

• Fraudulent leads often result in contacting someone without their written consent, violating the TCPA act. The callees are simply the victim of data breaches and you are the victim of lead generation fraud.

Mitigating risks enables you to scale your lead generation campaigns without scaling the problematic parts and thus becoming an unmanageable risk. Though, the problematic parts will scale similarly, flagging them in advance as fraud prevents you from calling these fraudulent leads. This prevents you from becoming a victim of litigation costs. We all know that TCPA settlements are steep, and nobody wants to be on the frontpage of tcpaworld.com having a headline: ‘... sued in a TCPA class action’!

In short, what are your risk mitigation mechanisms:

• Include lead generation fraud detection on your contact forms as the first line of defence

• Check prospect’s contact data: Is it on the litigator scrub list? In the Do Not Call Registry?

• Check whether the contact data does checkout with verification services

• Obtain ‘proof of consent’ for every generated lead you are going to follow up

How does it work?

Figure 1 shows the four parties actively involved when generating leads in B2C, these are:

1. You. You want human prospects attracted by and interested in your product or service.

2. Your in-house marketing campaigns, affiliates, agencies, social media platforms, search engines, linear TV, CTV, programmatic, mailing lists, magazines, SMS / text messaging, etc.

3. The visitor arriving at your landing page. This can be organic traffic, or paid traffic (2)

4. Fraud detection by Oxford BioChronometrics

As you (1) only want to pay for human visitors viewing your ads and arriving at your landing page (3), you’ll need to verify these visitors using a fraud detection (4). This enables you to assess whether the affiliates, social media platforms, search engines, etc. (2) bring you real humans or bots, fraudsters and crooks.

• A. You will run your digital marketing campaigns, a weekly mailing, buy keywords and pay per click, or hire 3rd parties and pay per click or generated lead, etc.

• B. Visitors, prospects clicking on your ads, links in emails, etc. are redirected to your landing page

• C. The landing page contains a contact form to be filled out

• D. The Oxford BioChronometrics fraud detection tag is present on the contact form and inspects the session

• E. The findings of the fraud detection tag are conveyed to Oxford BioChronometrics for analysis

• F. Interested prospects fill out the contact form with their contact details.

• G. You will receive the fraud detection status associated with the contact form prior to making the call. Your call center will only contact the prospect to close the deal if the lead is genuine. Besides the fraud status you will also record proof of consent, check the do-not-call list, check the lead data with verification services.

• H. You aggregate the numbers per source per month and only pay for human traffic. You either get a refund or credit traffic. You refuse to pay for fraud, make sure for each source your contract contains such a clause.

How does this setup help you?

You save up to 10% - 20% of your digital marketing budget, depending on your fraud levels. Secondly, you don’t follow up on fraudulent generated leads. These fraudulent leads truly look genuine, but the contact data is typically based on stolen PII data. That means you are contacting someone without express consent, which is costly in case you call someone who will sue you. The bare minimum of a TCPA settlement is $500 but in reality it is much much more. At scale, if only 1% of the fraudulent generated leads is going to sue you (that’s ~1% of ~1% and thus ~1 in 10,000 generated leads), you have a serious problem. Mitigating this is a must!

The most undervalued one is you get clean data, this is your long-term gain: Analysis will enable you to know what works to attract quality: Humans, not fraudsters. Knowing what drives your audience, what works and what not (with real humans), that’s what also enables you to scale!

I’m sure you still might have questions like: How to implement the detection tag? How to connect to the fraud status API? How can the fraud status be associated with a specific contact form? How to integrate the fraud status in your pipeline? How to setup a test in order to validate your claims? How to know which source is fraudulent?

Just connect, comment or send a DM

Appointment HERE.

#adfraud #leadgeneration #B2C #frauddetection #TCPA #howtoscale

According statista.com1 the number of bots in 2022 on the Internet was ~50%. This bot traffic can be split in two parts:

1. Good bots: Search engines, internet archive, malware scans, etc.

2. Bad bots: Scraping, scalping, ad fraud, DDoS attacks, etc.

Respectively 17.3% and 30.2 of the total traffic [1] are good and bad bots. The main technical difference between good bots and bad bots is that bad bots try to blend in with human traffic by technically changing their appearance, ie. user agent, use residential proxies, change their TLS fingerprint in order to match the provided user agent, prevent browser automation leaks, etc. Good bots declare themselves, see the table 1 for some examples.

According statista.com2 the number of bots in 2022 on the Internet was ~50%. This bot traffic can be split in two parts:

1. Good bots: Search engines, internet archive, malware scans, etc.

2. Bad bots: Scraping, scalping, ad fraud, DDoS attacks, etc.

Respectively 17.3% and 30.2 of the total traffic are good and bad bots. The main technical difference between good bots and bad bots is that bad bots try to blend in with human traffic by technically changing their appearance, ie. user agent, use residential proxies, change their TLS fingerprint in order to match the provided user agent, prevent browser automation leaks, etc. Good bots declare themselves, see the table 1 for some examples.

One valid question would be: What are all these bots doing? Bots are used to monetize something. This can be scraping information, polling the website for the exact sneaker release moment, but also loading ads, clicking ads, etc. There are more flavors, but in general the bot world can technically be split into these two major groups:

1. Scraping and scalping information, purchase limited goods, tickets

2. Ad fraud, click fraud, and lead gen fraud

Interestingly, vendors protecting businesses from these two bot classifications can also be split in two groups. The vendors protecting retail websites and ticket sales actively block bots when they see them. Vendors in the advertising world don’t block bots, but passively detect bot traffic.

Passive bot detection has advantages: The biggest advantage is that you don’t provide a direct feedback loop to the bot makers. Bot developers simply don’t know whether their bots are flagged or not, at least not instantly. It also has disadvantages, for example: Having an ecommerce site with millions of articles means that bots will scrape product info round the clock, costing a lot of bandwidth and CPU cycles especially when bots scrape the long tail of products as the information and images are typically not cached. That’s a valid reason why bots are actively blocked, even though the effect is that it forces bot makers to quickly evolve.

In digital marketing active blocking bots would imply that detected bots are not able to load advertisements while the business model of all players in the ecosystem is based on volume. Publishers or websites showing advertisements, ad verification vendors, and all middleman in the ecosystem, make money based on volume. More ads means more impressions and ad verifications which means more money to both the verification companies and the publishers or websites where the ads were shown! Solving the bot problem by actively blocking bots would cost them 20%, 30%, 40% and in some cases over 50%.

Technology wise: How do bots work?

In order to run bots at scale the right technology stack needs to be chosen, the less overhead the better. The cheapest way to run a bot is by generating and firing the HTTPS requests directly, without a browser or App. This is how scrapers were able to scrape price and seat information from airlines, buy limited edition sneakers and PS5s at the release date, concert tickets, and sports event tickets.

In response to scraping anti-bot vendors started to block bots by simply looking at the combination of the user agent, TLS fingerprint and some basic JavaScript challenge-response tests. If the returned payload didn’t match the expected answer to the challenge, or was inconsistent, contained traces of browser automation, etc. the WAF (web application firewall) would simply block access and would return: HTTP status 4033 or 4294 and blacklist the IP address for 10, 20 or 30 minutes.

Checking for bot traffic at each request is expensive. That’s where access tokens, aka. access cookies, appear. Once a browser is approved by the anti-bot’s backend it will receive an access token, which expires after eg. 10 minutes or a max number of requests. Each subsequent request within the 10 minutes providing this access token will be allowed and receive a normal response.

This is exploited by bot makers. They will obtain an access token using a fully fledged browser and once a valid token is obtained, the bot switches to pure HTTP requests and continues until the token expires. The reason is, browsers are slow cost of lot of CPU and memory resources preventing to scale on a single node, etc. Of course the anti-bot answer is to validate whether the client requests all non  essential components in a webpage, whether the client’s traversal path through the website is a normal human path, etc.

The stakes are high, that’s why services offering to bypass anti-bot vendors are popular and very profitable. Their services are often offered as an API where you send your request to the API, which forwards the request like a proxy server to the target website or App backend. The API controls a browser or software fully emulating the detection JavaScript, network packets and TLS fingerprint in order to return a perfect and correct payload to the anti-bot vendor, which will respond with an access token allowing free passage to the next several requests.

Knowing that the most difficult part of scalping and scraping is being maintained by the “10 best reverse engineers in the world”, see Figure 1 will give you peace of mind when you are trying to setup an organization to resell #taylorswift tickets, #PS5 consoles or limited #nike sneakers.

Luckily there are many more of these APIs available, so there’s always a backup.

So, what do Ticketmaster, Nike, Sony, but also poker and gambling sites do about this? Because they surely know that these anti-anti-bot services exist and only keep amateur scrapers and hobbyist scalpers away. Their answer is to stack vendors. Some have multiple anti-bot vendors protecting their sites and backend APIs with which their Apps communicate and the bad guys needs to bypass each of the anti-bots. From the brand’s perspective it probably is: The majority vote counts. If 2 out of 3 anti-bot vendors say it’s human, it probably is. Bots have evolved quickly because of the direct feedback loop, ie. blocked access when detected. That enabled bot makers to quickly evolve their bots and now the best ones are winning, and monetizing their hard work.

The number of checks which can be done at the request itself, the network stack, TLS, the browser object models, audio and webgl canvas fingerprinting, etc. is limited5 , while browsers become more restricted and lose entropy due to privacy and anti-tracking. The answer is that anti-bot vendors will randomly change the challenges over time, again it seems that isn’t affecting the services of these API providers.

Anti-bot vendors have tried to make it more difficult by implementing virtual machines in JavaScript. This means that JavaScript code is not executed directly, but is implemented in bytecode and has to be fetched, decoded and executed by JavaScript code. Over time the opcodes, encryption methods and keys, etc. will change making static responses useless. Again, this only raises the bar and will weed out amateurs while the professionals laugh about it. Though, if it becomes too hard to bypass and the profits are high enough low wage workers and farms will be used to buy sneakers and tickets manually. The same farms are used to solve new types of CAPTCHAs which cannot be solved yet by software, eg. sliding puzzles.

Anti-bot vendors VS. Ad fraud detection

Why does this ‘anti-bot vendor versus scrapers and scalpers’ fight look so different when comparing it to the fight in digital advertising ecosystem? It can’t be explained by the amount of dollars involved. The answer can be found in what clients expect from their anti-bot vendor to protect their assets or spend. When #Sony released the #PS5 a large portion of the consoles were being flipped at markup of 200% and that isn’t good for the brand’s reputation. The same with Taylor Swift tickets: A lot of unhappy fans having to pay more than double the orginial ticket price.

In digital advertising a direct feedback loop doesn’t exist. If an advertisement has 1,000,000 impressions, it will have between 2,000 and 5,000 clicks at a click-through-rate of 0.2% to 0.5%. The number of conversions to leads or sales is again between 2% to 8% of the clicks. This means that 1 million impressions will convert to 40 (2% of 2,000) [low estimate] and 400 (8% of 5,000) [high estimate] leads or sales. But, if half of the impressions (500,000) were shown to bots and the other half (500,000) converted to 100 leads or sales the campaign would be considered successful, where it could have been 200 leads or sales.

The difference is that people will start complaining on social media when tickets, playstations or sneakers are sold out instantly and only available at a 200% - 300% markup, while the people working in digital marketing didn’t even realize that they could have doubled the business outcome.

Technically, the differences between ad fraud detection vendors in the digital marketing ecosystem and anti-bot vendors protecting brands against scraping and/or scalping are enormous. The anti-bot vendors protecting websites from scrapers and scalpers do block access to bots and thus are required to evolve quicker and find creative ways of detecting bots in order to keep up with the bot makers. In digital marketing vendors keep on using the same detection for years, while on the offensive side the bot creators use the experience from the scraping and scalping world and thus evolve continuously. Ad fraud detection vendors don’t have a feedback loop telling them what works and what not.

So, How do ad fraud detection vendors without a feedback loop know if their detection works? They don’t and if they detect high rates of fraud it is most likely based on assumptions. That’s why most ad fraud detections in digital marketing are laughable: “These aren’t the bots you’re looking for. Move along, move along”.

In lead generation our clients will contact the prospects and will know whether Oxford BioChronometrics' SecureLead did catch the fraud, or not. If our detection would be based on assumptions it would cost them business due to false positives6, and by not catching the real fraud it would increase litigation risk simply because following up on generated lead and thus calling people without their consent is risky. Some of these callees will start legal action which has to be settled and will cost a ton of money.

Now what?

If you want to be sure your ad fraud vendor detects fraud and fraud only, you should be asking the right questions. Questions like:

• On what criteria does your detection flag bots and/or fraud?

• Do you have a feedback loop to know what works?

• If not, how do you know your detection works accurately?

The past few years have shown many examples that you simply cannot trust any vendor in the digital marketing ecosystem, eg. MFA sites, MFA on publisher’s subdomains, long tail websites sold as premium, audience networks, fake Apps loading ads in the background, free gaming apps where the gamer has to click on an ad to continue, should I go on? An ad fraud detection vendor showing a dashboard with decreasing numbers is like the butcher certifying its own meat. Quarterly reports with low bot percentages smells like willful ignorance.

What value do such charts or claims really have? Not much without the option to see why individual bots and/or fraud were flagged (the discrete decision). When a decreased ad fraud percentage can’t directly be tied to increased business outcomes like sales and increased quality of generated leads, you still don’t know whether the fraud detection fails to flag fraud, or the quality of traffic improved.

I can assure you browser automation and bot detection at scale is discrete and thus a yes / no. Human operated fraud detection has more shades as it is based on interactional behavior and looking at flow. But, luckily, percentage wise human operated fraud is relatively small simply because it is expensive to scale.

Questions? Feel free to comment, connect or DM

#adfraud #leadgeneration #CMO #botdetection

Let’s talk about Meta's Facebook again.

Last week’s post showed that Oxford Biochronometrics detected large numbers of fraudulent clicks coming from the Android Facebook app. Without a fraud detection tag on your landing pages these fake clicks look like regular clicks and your campaigns running on Facebook would look successful. Again this proves in digital it is easy to fake things and without any form of protection you would think your campaigns get traction. Yeah, by bots, fraudsters and imposters.

Because we all know talk is cheap, this second post will show some hard data and evidence, real results and breakdowns of data. The first example is based on a large company running many campaigns from many sources simultaneously. The second example is a small company (hospitality), they don’t have much budget so they try to spend it wisely.

How to read the charts and data

Before showing charts, it is also important to know how to read the chart and what information is available in the chart. The charts have the same colors as FouAnalytics, because I like that colorscheme and it is also colorblind proof. I’m not colorblind but have shown these charts to two friends who are red-green color blind.

The top section of a chart is a percentage stacked barchart where each color shows the amount of traffic of that kind. At the left upper corner the clientname is listed, which can be anonimized when choosing not to disclose the clientname. The code next to the name is the time frame. Possible time frames are M2, M5, M15, M30 which stands respectively for 2, 5, 15 and 30 minutes. H1, H4, H12 is one hour, 4 hours, 12 hours, and D1 which is a daily barchart. Each bar represents one time frame, which is listed at the left upper corner. Changing time frames allows to zoom in and zoom out when something happened in the data and you want to look at details. The last part of the label is the pivot value and determines how the data is broken down into segments. The pivotvalue can be stacked to create a compound pivot (maximal 4), eg. you could look at Facebook traffic from Canada on Android devices, or Google traffic from Wisconsin using a specific search term.

The green barchart is the volume of the traffic. It follows the same time frame and is vertically automatically scaled. Between these charts some date-time labels are printed. These are rounded down to the time frame.

The legend would be:

The difference between FouAnalytics and Oxford Biochronometrics is that Oxford Biochronometrics has less ‘looks like bot’ and ‘looks like human’ aka human-ish classification. Only a fraction of the visits will fit this ‘gray area’ where the detection is uncertain what it really is.

The juice

First we’ll start with a large client running multiple campaigns concurrently. The chart is based on traffic arriving at their landing pages (after the click). This means that looking at the totals both paid and organic traffic ar included in the data.

FTSE 100 company

The first chart shows all traffic arriving at their landing pages. It shows some levels of fraud, but also a real good portion human traffic.

Of course our clients are only interested in: How much paid traffic was fraudulent? So, we’ll need to split that combination chart into organic and paid.

This is how the organic traffic looks like. It contains a lot of bot traffic. Though these bots might have some impact on the infrastructure and costs, but that’s a fraction of the attribution costs.

The same chart but now paid traffic. And this looks much quieter than the organic traffic.

As we would like to know how Facebook traffic looks like, we’ll have to zoom in to only paid traffic and where its source is Facebook. The chart below shows exactly that, all paid facebook traffic arriving at the landing pages.

As you can see it does contain fraud, but as this is still a combination of Mac, Windows, iOS and Android and whatever OSes, we’ll have to zoom in again. Let’s compare the two mobile OSes, as we’re interested in traffic originating from apps.

In the chart above you can see a MEGA-difference between the topchart (iOS) and the bottom one (Android): iOS is almost without fraud, except for that little spike on October 17. The Android traffic is ~40% fraudulent.

You might also notice that during the weekend paid traffic from Facebook is disabled. This makes perfectly sense, as this client runs their lead generation campaigns only during the weekdays and non holidays.

SME company

First step is to look at all data. The chart below contains daily bars (D1 timeframe) and shows what traffic arrived at their landing pages during the last 6 months. The client does have some fraud, but based on this chart you still don’t know anything.

It is also clear from the bars that the traffic volume is not bound to weekdays and holidays. Let’s zoom in to the paid sources, because that’s what is most relevant. Ah, it seems that they only buy Facebook traffic.

This looks not that good. The amount of fraudulent clicks originating from Facebook is about half, even when the traffic volume is large, around 2023-Nov-21 and in April 2024. Of course the question here is again, which Facebook traffic? So, let’s break down the data to operating system and Facebook traffic only:

What a surprise!? NOT! The iOS traffic looks very clean, but the Android traffic is a marketing bloodbath, as ~90% of your paid visitors arriving on your landing page(s) are flagged as fraud. What a waste of budget!

I can give many more examples of Facebook traffic and the differences in fraud levels between iOS and Android. But, over and over again the pattern is the same. If people like these charts and examples, not restricted to Facebook, let me know, it depends on the amount of traction this post gets.

Takeaway

Your takeaway: Target iOS, try to avoid Android or do it very selectively. It might be more expensive, but better 50 real humans looking at your product than 1000 bots wasting energy, bandwidth and ruining your campaign goals.

#cmo #facebook #adfraud #digitalmarketing

At Oxford Biochronometrics we regularly organize a bothunt, although a better term would be fraudhunt. This means analyzing, scrutinizing, creating anomaly statistics, etc. on raw data in order to spot outliers and irregularities and if you’re lucky find a new type of fraud and win the bothunt medal of honor.

Over time it has become harder and harder to spot fraud, as most low hanging fraud is detected automatically the remaining portions look very clean and human-like. When dealing with large institutions and corporates running massive campaigns and thus having tons of human visitors the remaining fraud -if it even exists- hides well in aggregates. One trick is to start filtering per source or affiliate, because each of them is paid independently and there's always some source or affiliate that feels the need to blend some inferior cheap traffic into the stream of visitors to siphon your spend.

When advertising on Meta Facebook you pay for impressions in the user’s feed or for clicks on your advertisements [1]. Fraud is directly related to how you buy your media. If you pay for clicks (CPC), fraudsters are incentivized to only generate clicks. The same logic applies when buying to leads (CPL / CPA), except in lead generation you have a feedback loop. Following up on the leads gives a good picture of the amount of fraud, eg. if 80% of the callees deny that they have filled out your lead generation form you not just know but also have confirmed you were hit by fraudsters. That’s why buying media in CPL / CPA in combination with 1st class fraud detection (hint: Oxford Biochronometrics ) is the optimal setup: You will know the fraud status prior to making the call, return generated leads with reason: fraud, thus not paying for those thus lowering your marketing costs and mitigating litigation risks.

This time Meta’s Android "Facebook app"

The casus: Small business has fraudulent flagged visitors originating from "Facebook app"

Facebook traffic mostly (>90%) originates from the Facebook app, only a fraction of the traffic comes from desktop. There are multiple Facebook apps. The most common one is the regular Facebook app and the other one using less (mobile) data is the Facebook Lite app.

We'll be looking at traffic originating from the Facebook app which can be recognized by looking at the user agent (UA). Besides the UA many other properties of the browser are recorded which give an overview of who and what is loading your landing page originating from the Facebook app.

Looking at the UA reveals which mobile operating system is used: Android or iOS (marked in yellow). Secondly, the UA shows which browser version (in gray) and also which Facebook app version were used (in red). The UA may be spoofed by the fraudster, but for a JavaScript based fraud detection this is easy to detect and flag. In these two examples the UA belongs to the reported browser and OS. Let’s take a look at two UA examples:

and

The Facebook app appends a string to the useragent marked in red showing which OS and which app version made the request. Android adds FB_IAB/FB4A which translates to FaceBook InAppBrowser / FaceBook for Android. IOS has a similar string FBAN/FBIOS. The added FBAV/xxx.x.x.xx.xxx portion shows the Facebook app version.

FBCLID

Each click from within the Facebook app to an external website gets an FBCLID appended to the URL as a querystring key/value. This FBCLID is the FaceBookCLickID. The value is a per-click unique (Facebook server side) generated hash [2]. Its length is between roughly between 60 and 155 characters and includes hyphens and underscores. The same ID may appear multiple times at your landing page, in such a case the FBCLID is cached and the clicks occurred quickly after another. In this breakdown we’ll be looking at unique FBCLIDs only, as you only pay once per visitor that clicked.

Platform

In this analysis we’ll be looking at Android traffic only. Besides the user agent much more information can be extracted from the browser. For example, the navigator.platform [3]. This resembles the CPU architecture of the device. On Android the most common values are: Linux armv6l, Linux armv7l, Linux armv8l, Linux armv81 and Linux aarch64. These values have to match the phone model, which is marked orange in the Android user agent example above.

HTTP Headers

At each request the browser sends a set of HTTP headers. These headers contain information which which accepted encoding, accepted language, referrer, cache control, character sets, session cookie, user agent, etc. (see: RFC 7231). Although these headers are standardized different browsers render these headers slightly different. This can be fingerprinted and outliers can be spotted [4].

Bothunt fraud details

When looking at the unique clicks in a small businesses' campaigns (eg. travel, hospitality) having an unique fbclid we observe that 61% of the clicks comes from Android. The remaining 39% clicks comes from iOS (iPhone and iPad). A stunning ~95% of the Android clicks is flagged as fraud, and all clicks have the exact same type of fraud. The same behavior is also observed at larger companies and campaigns, though the fraud percentages are lower because their volume is bigger and thus relatively more real humans are attracted to the landing pages. It is easier to get high fraud percentages at low volumes, and small companies have no weapons against this type of fraud.

So, in short how does this fraudulent Facebook traffic look like?

• Traffic originates from the intended and targeted geolocation

• The traffic comes from real physical mobile phones, directly making the request

• The requests are made by an Android webview browser, most recent version: Chrome/123.x.x.x

• User agent contains FB_IAB/FB4A indicating Android Facebook app

• The requests have slightly incorrect HTTP headers

• The same reported platform for each fraudulent visit of this type

• The browser reports a spoofed renderer and not its real renderer

And that, my dear LinkedIn friends, is how this apparent Facebook app originated fraud burns through your ad spend. In this travel agency’s example a few hundred dollars a week. Besides complaining about poor conversion ratios, ie. low amount of bookings, there’s not much else a small business can do.

I'd love to create a 100+ page Adalytics -like report with methodology, many screenshots, show in detail which HTTP headers are not correct and why, how to determine a spoofed renderer, and corroborate evidence from different businesses being affected, etc. But, in contrary to legitimate businesses having to correct their behavior or be punished by the markets; fraudsters will thank me for such a free improvement report, and not even thank me personally nor financially but thank me by improving themselves and making advertisers' lives even more miserable.

Why?

You might be wondering why would this happen? Who benefits from this? Let’s assume you are running a campaign at scale, besides buying traffic directly, you also use affiliates to generate traffic and pay per click. The goal is to see visitors arriving at your landing page, of which a subset buys a car insurance, books a vacation, test drive in a new Renault, or buys your virus scanner, etc. But, as always the majority of clicks don’t convert to a generated lead or sale. Yet, you still have to pay for the clicks unless you know and would have flagged them as fraud, if not: the affiliate wins and the invoice is paid.

Another reason would be when an agency running your campaign, buys cheap traffic in order to artificially boost numbers in order to claim success. Although in 2024 it's common knowledge that bought traffic == bot traffic.

Since when?

Once you know the pattern to look for it becomes easy to search for fraud in historical data. Based on the data collected by Oxford Biochronometrics we could trace back this fraud type back to September 2023. With Oxford Biochronometrics ' limited view of all traffic originating from this apparent Facebook app one might wonder what the true scale of this fraud is.

Now what?

So, you are a small business owner: What can you do? If your site has less than 10,000 visits a month, we don’t charge for our service, we never have. Though, the free service comes without a real-time feedback, which most small businesses don’t need. You typically just want a monthly overview, which can be extracted from our dashboard, and use this data to get a refund or credit traffic for the coming month.

Lastly, if anyone knows a good contact at Meta/ Facebook HMU, because it’s their reputation on the table; I do have many more details and the FBCLIDs which Facebook should be able to track. I’m sure a lot more companies from SME to FSTE 100 are affected and to my understanding this type of fraud is not detected by legacy fraud detection vendors [5].

Sharing, liking and comments are appreciated. Feel free to connect or DM with questions

#facebook #clickfraud #cmo #digitalmarketing #adfraud

[1] https://www.facebook.com/business/help/716180208457684?id=1792465934137726

[2] https://dl.acm.org/doi/pdf/10.1145/3543507.3583311

[3] https://developer.mozilla.org/en-US/docs/Web/API/Navigator/platform

[4] https://www.linkedin.com/posts/kouwenhovensander_frauddetection-fingerprinting-activity-7049009901523656705-xrkX

[5] https://www.linkedin.com/posts/kouwenhovensander_adfraud-b2c-digitalmarketing-activity-7125112905296957441-O4Wp

What are the root causes and effects of poor performing digital marketing campaigns? In many cases you’ll think poor performance is caused by your creative, positioning, the messaging, the intended audience, etc. But, the real issues are often deeper and more fundamental.

When your business growth depends on digital bringing in new customers many executives focus on the symptoms of poor growth and not the root cause of the growth problem. Over time the numbers get worse and the exec team and board get bogged down in managing the unexpected legal costs, external counsel, non-performing digital marketing campaigns, and how to communicate the disappointing results to the shareholders.

Though these effects are important and need to be addressed, the real problem often is that the fundamental aren’t right. If the root cause of the problems aka the fundamentals aren’t right no marketing team, no matter how smart or dedicated, will be able to deliver the results the business is expecting.

However, when the fundamentals are right, it makes any marketing team deliver great results. That’s why companies should consider the 5 layers of digital marketing problems:

1. Buying inferior and cheap traffic quality. Low quality traffic without fraud detection to filter out the bad apples = no marketing optimization will ever lead to sustainable success.

2. If you attract poor quality traffic without 1st class fraud detection you’ll get an inflated picture of your audience. You think your ads had a great click-through-rate by your audience, and visitors generate leads or buy digital products. But, a chunk of your visitors were fraudsters presenting themselves as your perfect audience. These fraudulent visitors will cost you easily 20% - 25% of your digital spend. After removing fraud: Good traffic = you are in control of your true audience. Oxford Biochronometrics offers 1st class fraud detection.

3. Low quality leads will never convert to a sale, and are typically not generated by interested humans, but by fraudsters to attempt to skew conversion percentages of the traffic source directly or indirectly paying them. Without detecting fraud, filtering for known litigators, and documenting proof of consent, you might be contacting prospects without their express consent. In case you did document consent: It were fraudsters providing the consent, not the prospects. Contacting prospects without their consent, mostly victims of stolen PII data, allows them to sue you for violating the TCPA act and some will.

4. Marketing problems become business problems when your CAC:CLV ratio becomes lower than ~1:3. This implies the marketing efforts haven’t achieved their goals, and the business as a whole will suffer due to lack of growth. Marketing problems become a legal problem when your call center did contact prospects without their express consent. Fraudulent generated leads using stolen data is the main reason and primary source of this problem. So, brace yourself for a big wave of TCPA demand letters and the associated settlement costs. It might become worse when you get sued and a class action is started. At this point you’ll really need external counsel charging $2,100/hr. I'm sure that doesn’t help your business results.

5. The doom scenario unfolds when a class action lawsuit gets certified. In such a scenario a costly litigation process starts, which might end in a hefty multimillion settlement.

This all happens because of buying cheap traffic, not filtering out fraudulent generated leads, not validating against known litigators, and not documenting proof of consent, and thus not being compliant. Let that sink in.

You might think: We’ll just outsource layers 1 and 2 and only pay for generated leads in the form of affiliate fees (CPL / CPA) and we’ll use a call center. Good thought, but unfortunately you can’t outsource liability. The TCPA holds businesses accountable for the actions of those acting as their agents. This means you must exercise diligence in selecting and monitoring third-party entities engaged in telemarketing activities to prevent TCPA violations.

If the input (inferior traffic) isn’t good, how can the outcome (business growth) be great?

That’s why each layer in figure 1 requires its own checks and validations before the collected information is allowed to move to the next layer. These checks and validations act as a filter removing the bad apples. Fraud detection enables you to know the fraud status per generated lead in real-time before making the call, ActiveProspect's TrustedForm enables you to document proof of consent, Contact Center Compliance’s litigator scrub enables you to scrub known litigators. If this works correctly you generally don’t need legal counsel. But, if you think these services are too expensive, Troutman Amin LLP will only be charging you $2,100/hr attempting to solve your TCPA problems, though we all agree they are your best shot at winning and thus worth every penny.

Focus on fundamentals first

You simply need to focus on fundamentals first: By addressing root causes in lower layers problems in subsequent layers will be mitigated or completely disappear. Without these problems your digital marketing becomes successful and your business can build a solid foundation for growth.

#B2C #digitalmarketing #fraud #leadgeneration #tcpa

In B2C growing your business using digital marketing comes with many challenges and pitfalls. In order to attract prospects you run digital advertising campaigns, at the landing page you nurture your prospects in order to close the deal, or get their contact details in case you’re generating leads. That’s nothing new and simply how the marketing funnel works.

Zooming in to those challenges and pitfalls, which can be placing your ads on MFA sites, your ads being loaded and “viewed” by bots, click fraud, and fraudulent generated leads. The former cause that your campaigns underperform, the latter opens litigation risk once the contact data appears to genuine and you start calling the prospect. But, how much is this risk? And how much money would be at stake? How does it affect your business’ performance? That’s what this article is about, and what you can do to mitigate those risks and losses.

First step is to visualize the costs of digital marketing. Figure 1 below shows a number of colored and labeled rectanges and a legend for a more elaborate explanation. The surface of each rectangle corresponds with the US dollar value. The largest expense in Figure 1 is the digital marketing budget, a 1000x1000 blue/red square and each pixel represents a visitor at the landing page, which on average costs $1.50. This can be buying clicks at an average of $1.50, or advertisements and based on the CTR each visitor arriving at your landing page costs on average: $1.50, etc.

At the left of Figure 1 the big 1000x1000 blue and red square represents $1,500,000 dollars, the digital marketing budget. This amount is considered the investment. All expenses have to be paid by the return on this investment. If web is your #1 revenue generator the profit made by this generated also needs to cover most of the other company expenses.

Both genuine and fraudulent traffic will convert to leads, or sales. This makes fraudulent traffic so hard to pinpoint, as it completely hides in aggregate numbers. The effects, however, are very noticable. Once you receive a dozen TCPA demand letters a day which you have to settle each at a hefty price. These settlements, legal hours and external counsel costs will increase your costs and thus lower your ROI. This explains why the lower your fraud% is the lower the unforseen overhead costs will eat away your profit.

How fraud losses, costs and legal side effects relate to each other can be seen in Figure 1. The model to calculate these ratios uses the following input values. Without fraud detection:

• 20% of the traffic is fraudulent.

• 8% of the visitors converts to a generated lead.

• 25% of the generated leads converts to a paying customer (2% of the generated leads).

• 2% of the generated leads are fraudulent.

• Only 1% of the fraudulent generated leads converts to a TCPA demand letter. That’s 0.02% of the volume of generated leads.

• On average each TCPA settlement will cost you $2000, as you might have made multiple calls.

• If someone doesn’t settle and starts a legal fight your costs will increase (diagonal stripes)

• Your legal department spends on average one hour per letter.

The gray rectangle on the right in Figure 1 represents the costs of Oxford Biochronometrics ’ real-time fraud detection at this volume, which can be used to 1) prevent contacting fraudulent leads and also 2) to claim back fraudulent traffic.

Figure 1 shows that the price of Oxford Biochronometrics real-time fraud detection is roughly an additional 2%, but the increase in the marketing efficiency is +20% and it prevents and mitigates litigation risk at again 20%. Talking about ROI; that’s a 20:1 return on the investment! But, of course you’re thinking: “How do different percentages of fraud influence my digital marketing ROI? And what are the effects of the additional 2% when the fraudlevels have decreased to 12%? or 10%?” That’s what will be discussed in the next section.

Based on the numbers above 1 in 50 generated leads is fraudulent ( 2% / 0.04% = 50 ). That small subset of generated leads accounts can be found in the ~20% fraudulent traffic. A second degree effect (settlements, legal costs and litigation risk) is based on a tiny subset (about 1%) of these these fraudulent leads and still this adds another ~20% to your costs. You can probably imagine what will happen at a 25% or 30% fraud!?

Make fraud detection actionable

The damaging effects of fraud is the reason why you need real-time fraud detection, and the real-time portion means a real-time feedback loop. This enables you to know the fraud status of a generated lead prior to making the call. It also enables you to know which sources and which campaigns contain fraud and allows you to get a refund or credit traffic. Ofcourse, your contract needs to contain a refund or credit traffic clause.

In order to see the first and second degree effects of different levels of fraud we’ll take a look at Figure 2. This is an animation showing how fraud affects the performance of your digital marketing campaign. The animation is based and created upon the following configuration settings:

• Average fraud percentage (variable: 5%, 10%, 15%, 20%, 25%, 30%, 35%)

• The customer lifetime value. The average $ of a new customer

• The volume 1,000,000 visitors

• The avg cost of a single visitor, at $1.50

• Conversion from click to lead, at 8%

• Conversion from lead to sale, at 25% which means 2% converts from click to sale

• Percentage fraudulent leads, at 2% which means 2% of 8% which is 0.0016% of the volume

• The percentage fraudulent leads that start litigation, at 1% which means 1% of 2% of 8% = 0.000016% (16 claims out of a million visitors)

• On average each claim is settled at $2,000

• Your legal department spends 1 hour at $250 per letter

These configuration settings have been put into a model that creates a sankey flow chart showing the breakdown of the revenue of the digital marketing campaigns into its profit and cost (and loss) components. This enables you to see how the performance of your digital marketing campaigns is affected by fraud, but also by a too high cost per click, or a too low conversion percentage, or a too low CLV (Customer Lifetime Value), etc.

Figure 2 shows how the ratio of the profit versus costs and losses changes when the fraud percentage increases. The harsh reality is that with less humans arriving at your landing page it becomes harder and harder to achieve your business goals. You just can’t perfect your content and multi-stage forms to compensate with these losses.

Table 1 below contains the same information and based on the same default values as used in the animation. The calculated ROI, profit% of revenue, and costs/ losses% of revenue can be seen in the table below by only changing the fraud%:

The data in Figure 1 shows that at a fraud rate of 30% your marketing investment doesn’t generate profit, ie. the costs are equal to the profit. This means when your landing page is your only stream of revenue your company as a total makes a loss. The two charts in Figure 3 below show how the ROI and profit% plummet when a smaller and smaller portion of the prospects generate business in combination with growing legal overhead costs.

The two charts above emphasize again that your digital marketing campaigns need to be healthy, where the definition of healthy is maximum of ~12% fraud. Having this level of fraud the impact of TCPA claims is still relatively low and thus manageable, and you’re able to get credit traffic or a refund on the portion of your digital marketing budget which was flagged as fraud.

Make your digital campaigns healthy again

So, how can you make your digital marketing campaigns healthy again? It’s not just adding a fraud detection tag. It’s an organizational change on how your business handles the ingestion of generated leads, follows up on the fraud numbers in your digital marketing campaigns, and monitoring and managing the output of your campaigns at a strategic level. This has been explained in detail in a previous post, see: [1].

Of course, fraud detection is only one reason why digital campaigns are performing sub-optimal. Though, you have to agree it is a large reason and if not dealt with it will flourish and thus seriously affect your business outcome, increase your CAC, murk your view on which campaigns work, which audiences did respond, and in worst case a class action is started which you have to settle or fight against.

That’s why you will have to safeguard and continuous monitor your largest revenue stream by adding a real-time fraud detection, and filter out fraudulent generated leads. The price you’ll be paying for real-time fraud detection is almost nothing compared to the value it provides. As mentioned before: You get 20 times the value compared to the price you pay.

Want more info? The link to the interactive sankey chart? Leave a comment, connect, or DM.

[1] https://www.linkedin.com/posts/kouwenhovensander_digitalmarketing-b2c-fraud-activity-7168958124249321472-SqcO

#digitalmarketing #b2c #fraud #leadgeneration #tcpa

Appendix A

This appendix contains a few extra examples how fraud percentages affect the performance of a digital marketing campaign. To keep things clear each example only changes one single variable compared to the example in the main article.

Table 2 below shows how different fraud percentages affects the ROI and profit/ costs percentages relative to the revenue. The only different value used to create Table 2 compared to table 1 in the article is: $2.00 per visitor instead of $1.50.

Table 3 below shows how different fraud percentages affects the ROI and profit/ costs percentages relative to the revenue. The only different value used to create Table 3 compared to table 1 in the article is: 20% of the leads convert to a sale instead of 25% used in the article. The average price per visitor is again $1.50.

Table 4 below shows how different fraud percentages affects the ROI and profit/ costs percentages relative to the revenue. The only different value used to create Table 4 compared to table 1 in the article is: The CLV is $250 instead of $300 using in the article. The average price per visitor is again $1.50, the conversion leads to sale is set at the default: 25%.

Glossary

B2C - Business to Consumer

CAC - Customer Acquisition Costs

CLV - Customer Lifetime Value

CTR - Click through rate

MFA - Made For Advertisement

ROI - Return on investment

TCPA - Telephone Consumer Protection Act

The moments after the Oxford Biochronometrics ’ fraud detection tag has been added to a client’s B2C website(s) a clear fraud picture emerges and by simply looking at those numbers we again ask ourselves the question: “How can it be that it took so long that someone started asking questions and acted accordingly?” In this case the client’s #1 revenue channel is their website and it had ~25% of fraudulent paid traffic. Of course this paid traffic has a variety of fraud, mainly fraudulent clicks, fraudulent stuffed leads, fraudulent digital sales, etc. which is fully blended within the human audience.

Knowing your business and the predicted outcomes and expectations of your digital marketing program might probably raise this question:

“How much does this click and lead generation fraud affect my ROI and the CAC (customer acquisition costs) and its payback time?”

Or,

“Could this be the reason why our digital marketing campaigns have been performing worse than expected?”

Or, in worst case,

“Is this the reason why our legal department receives that many TCPA demand letters?”

How to make fraud detection actionable?

How can you make fraud detection actionable in order to improve your ROI, lower the CAC and mitigate litigation risk?

1. At the (strategical) macro level:

• Is your digital marketing program both effective and efficient based on its ROI?

• Does your digital marketing program expose litigation risks to your company?

• Is your digital advertising CAC, and your total CAC, ie. including all marketing and sales expenses, way too high? And what is the associated payback time in months? Unrealistic? How on earth can you lower these?

Answering and addressing the questions above will create a sustainable competitive advantage.

2. At the (tactical) meso level:

• Which digital channels or campaigns are driving conversion and contribute to the ROI?

• Which digital channels or campaigns require improvement? More budget? Or need to be removed entirely? See also [6]

• Which digital channels or campaigns expose the company to e.g. TCPA litigation risks?

• Periodically benchmark different real-time feedback fraud detection vendors, e.g. using A/B testing. This should answer which vendor flags fraud and fraud only! See also [5]

Answering the questions above will make your digital marketing program more efficient and thus more profitable by identifying and removing the poor performing parts. It also identifies where the TCPA litigation risks originate from.

3. At the (operational) micro level:

• Continuously monitor your campaigns and traffic sources using real-time feedback fraud detection.

• The fraud detection provides invaluable information at the exact moment you need it.

• This enables you to know the fraud status prior to making the call and thus prevents you from calling a prospect who didn’t consent to be contacted.

• This avoids the risk of receiving TCPA demand letters and thus a costly settlements.

• Fraud detection prevents you paying for fraudulent affiliate fees, and fraudulent clicks, the saved money can be spend on your intended human audience.

• Besides the fraud status also collect: Where did this click or lead originate from. Did the caller pick up the phone and denied to have consented?

• A positive side effect of fraud detection is that the quality of your campaign data improves and over time you’ll get a clearer picture on how your intended audience responded to your campaigns.

Of course, fraud detection is only one reason why digital campaigns are performing sub-optimal. Though, you have to agree it is a large reason and if not dealt with it will flourish and thus seriously affect your business outcome, increase your CAC, murk your view on which campaigns work, which audiences did respond, etc.

If your #1 revenue stream is through digital advertising and converting clicks to leads and customers the negative effects of fraud and the associated risks can be enormous. In many cases new Oxford Biochronometrics clients already use some sort of fraud detection. This can be IP address reputation scoring which typically flag data centers, TOR and VPN exit points, but not residential proxies running on consumer devices [1]. In other cases they use vendors which are able to flag bots and browser automation, but not human operated fraud.

Human operated fraud is able to fully recreate a lead’s path and the form data would checkout with verification services, and the behavior at your landing pages is not ‘robotic' or 'static’ but just like any human would interact. This sophisticated humanoid fraud is low in frequency but has a high impact in terms of litigation risk and the associated settlements. To reassure you Oxford Biochronometrics is able to flag this type of sophisticated human operated fraud, in real-time.

Figure 1 shows that fraudulent clicks will cost you a dollar, maybe a few dollars, per click. Lead generation fraud is more costly, but also happens at a lower frequency. TCPA claims are way more costly at a bare minimum of $500 per settlement. Having made too much calls based on fraudulent generated leads will cost you a fortune or even worse a class action is started and if certified you will have a serious problem.

That’s why you will have to safeguard and continuous monitor your #1 revenue stream by including a real-time fraud detection. On a side note you should also include proof of consent in order to protect you in the event of litigation, see [2][3][4]. These two measurements will drastically boost your marketing ROI and mitigate litigation risks at a fraction of the costs.

Want more info? Leave a comment, connect, or DM.

[1] https://www.linkedin.com/posts/kouwenhovensander_residential-backconnect-proxies-are-available-activity-7044282431310286848-g117

[2] https://www.linkedin.com/posts/kouwenhovensander_leadgen-leadgeneration-digitalmarketing-activity-7019280429815885825-5YRS

[3] https://www.linkedin.com/posts/kouwenhovensander_leadgen-leadgeneration-digitalmarketing-activity-7021834418898001920-uzPE

[4] https://www.linkedin.com/posts/kouwenhovensander_marketing-can-be-seen-as-a-financial-investment-activity-7024358269049561088-LFop

[5] https://www.linkedin.com/posts/kouwenhovensander_adfraud-b2c-digitalmarketing-activity-7125112905296957441-O4Wp

[6] https://www.linkedin.com/posts/kouwenhovensander_leadgen-ecommerce-digitalmarketing-activity-7033787920691949568-0znz

#digitalmarketing #b2c #fraud #leadgeneration #tcpa

What’s the fraud situation so far this year?

How much lead and ad fraud is there really?

Which affiliates had the best (and worst) results?

What can you do to make sure you’re getting the best results from your affiliate program?

With this new Affiliate Performance Benchmark Report based on real data from the second half of 2023, you’re just a click away from all the answers you need to make the most out of your lead generation programs.

The Free Report Now!

Marketing is an investment. An investment in potential new clients or customers generating revenue. Unfortunately, the road to acquiring new clients has many potholes, blind curves, oil slicks and bandits. Configuring and continuous monitoring your campaigns well will avoid the usual dangers, except the most dangerous one: the bandits aka fraudsters in the digital world. To address ad-fraud, click fraud, lead gen fraud sucessfully you’ll need specialistic tools which again have to be updated continuously in order to keep up with the evolving threats.

The one million dollar question is ‘what is successfully’ when addressing these types of fraud?

Flagging 100% of all visitors as fraud will definitely solve any fraud problem. But, you don’t have any business left. Flagging only 1% of the visitors will just catch low hanging fraud. Unfortunately, catching simple fraud doesn’t stop compliance issues, and in case you're generating leads flagging simple fraud won’t stop your legal department receiving a ton of TCPA demand letters [6]. Both scenarios are far from ideal, but what is?

The ideal situation is when fraud is flagged as fraud and humans aren’t! Simple as that. Except, that’s not what the average fraud detection vendor delivers. This means on one side the trade-off of a fraud detection not flagging fraud is the financial damage of the unflagged fraud. On the other side the trade-off of a fraud detection flagging too much is the financial damage of missed business. The question is: What is worse? Accept some fraud or accept some missed business, and how much is acceptable?

Legacy ad-fraud detection

Figure 1 shows how legacy fraud detection separates human visitors from the fraudulent visitors. This is based on their ~1% fraud reported quarterly. We all know fraud is more prevalent, though the reported percentage remains ~1%. Using 12% of fraud as the ‘true percentage of fraud’ this means their detection misses 11/12th of the fraud. That looks bad, but the real question is: How costly is this, financially from a business perspective?

Trigger happy ad-fraud detection

Abraham Kaplan called it “the law of the instrument” and it may be formulated as follows: “Give a small boy a hammer, and he will find that everything he encounters needs pounding.” [1] . The same behavior can be seen in fraud detection solutions too eager to find and flag fraud, anything out of the comfort zone (eg. browsers with an Asian language), anything out of the ordinary (eg. custom fonts installed), anything deviating from the default setting (eg. other javascripts on the website override certain browser functionality and/or polyfills [2]) affecting the detection, will be considered fraud.

This causes fraud detection solutions to flag a much higher percentage of fraud compared to the true fraud percentage. These incorrect flagged visitors are false positives. This means a portion of your human audience will be ignored by these trigger happy fraud detection vendors. Ignoring humans will cost you business and will affect your sales numbers.

Figure 2 shows how trigger happy fraud detection has a high false positives rate. The main reason is that these fraud detection solutions use “soft metrics” to flag fraud. Without having a groundtruth a baseline cannot be established and thus you’re just guessing who is a normal human visitor and what isn’t. As we all know “Assumpsion is the mother of all f-ups”, then why do these vendors flag like this? Here, I’m assuming it’s because of enshittification [3], hubris, and self flattery :-).

But, back to business. A high number of false positives sounds really bad, but again the real question is: How costly is this, financially from a business’ perspective?

Business impact

In order to model the business impact an Excel document has been created. This contains all the parameters and metrics of a typical digital marketing campaign. As campaigns differ over time, per vertical and per country, state, city, the sheet enables you to configure price (CPM), click-through-rate (CTR), conversion to sale, customer lifetime value (CLV), ad-fraud detection costs, and how much ad-fraud is detected. Based on these parameters it calculates the cost per click (CPC), customer acquisition costs (CAC), the CLV:CAC ratio, the false negative rate, and the false positive rate.

Figure 3 shows the comparison between a legacy vendor (1% fraud) and a trigger happy vendor (30% fraud). Both fraud detection solutions are equal priced and the model doesn’t take include any refunds or credit traffic based on the fraud percentage. On the other side it also doesn’t include the business costs of poor data quality, and potential litigation risks. The model shows that missed business (18% FP) affects the ROI performance of your campaign(s) much more than the wasted dollars on fraudulent impressions and/or clicks (11% FN) . That makes perfectly sense, simply because any business has to outweigh the campaign investment.

But, I can hear you thinking, what if the legacy fraud detection vendor is much more expensive? Let’s triple the costs and recalculate the campaign using an ad-fraud detection cost of 0.0006 per verification. This will affect the pricing, and increases the CAC from $62 to $66 and respectively lowers the CLV:CAC ratio from 4.52 to 4.24, as can be seen in Figure 4. In this scenario, pricewise, flagging only 1% of the fraud and thus missing 11%, is still 14% cheaper than flagging 18% too much.

In order to compare different fraud levels a lookup table has been constructed. The horizontal x-axis contains the true fraud percentage. As you may have read in the Oxford Biochronometrics ’ affiliate benchmark report [5]: In normal circumstances the fraud averages out at 12%, but may differ greatly per source. The first step is to look up the fraud percentage of a source (eg. Facebook, Bing, etc.) in the benchmark report, and write down the percentage. The second step is to look up what percentage your current fraud detection vendor flags and reports. Now you have two percentages: On the x-axis go to the column using the percentage of step 1, and subsequently go to the row using the percentage of step 2. This cell contains the performance value as a CLV:CAC ratio based on your true and vendor reported percentage combination. The table only contains percentages below 40%. The table has been generated using the exact same logic and ad-verification pricing as shown in Figure 4.

Based on the table it can be seen that having a vendor reporting a ~1% fraud percentage is (financially) better for the ROI of your marketing campaign than having a trigger happy fraud detection overreporting and make you ignore human traffic. Of course when refunds,credit traffic, data poisoning, (TCPA) [6] litigation risks are added to the mix these results will change slightly. In the end: The best scenario is to flag and report fraud accurately, where the reported fraud matches the real fraud, without FPs and without FNs. Okay, granted, digital marketing campaigns without any fraud would be even better, but we all know that’s wishful thinking.

How does Oxford BioChronometrics know the real fraud percentage?

Oxford Biochronometrics has been founded by individuals NOT having a background in digital marketing. Our background has been in cybersecurity, designing and building a real-time solution to detect and flag fraudulent transactions in Internet banking sessions. The experience of designing, building and embedding a real-time fraud detection product into a large bank with all its regulations, compliance, etc. has set the standard for creating the Oxford Biochronometrics fraud products. Yes, we had to learn how digital marketing in general works, how the ecosystem, the attribution, pixel tracking, viewability, calculating the ROI, ROAS, etc. works. Most of these are based on technical solutions mapped to a marketing functionality, financial instruments, or similar to stock markets and stock exchanges, which we were already familiar with.

“Keep your friends close, but your enemies closer” -- the Godfather part II [3]. As a company you would like to know what your competitors are doing. This is why every now and then you’ll take a look at your competitors’ websites, blogs, results, etc. and one of the most interesting freely available piece of information is their public part of the fraud detection software: The JavaScript collecting data from the browser. Anybody will be able to download this, read the code and based on that be able to understand what data points the JavaScript code collects and conveys. If you have a programming and cyber background this is a fairly simple exercise, mentally similar as baking an apple pie! Although some companies try to protect their JavaScript and try to make your life miserable by obfuscating and encrypting the JavaScript code, with the proper tools and knowledge that can be reversed, and then it’s still doable [4].

Real results is what counts. Having good feedback from your clients, “your detection saves us a ton of money and provides us clean data”, is nice. But, firsthand seeing how one of your clients grows from a startup to a scaleup and then ~18 months later they are being bought for multiple billion dollars, while their #1 revenue comes from digital marketing on the Internet to get customers to acquire their products. That’s gold and has been the ultimate confirmation that Oxford Biochronometrics has been able to protect their business, provide their data analytics and ML teams with clean data to work with and mitigate litigation risks by filtering out fraudulent leads. This is the long term feedback loop that shows that we consistently keep on performing over a longer period of time.

Combining our cyber background, our freshly acquired marketing knowledge, competitor intelligence and confirmation that we are right on track enables us to know what works and what doesn’t and also what will never ever work. This allows us to understand which fraud detection vendor collects good data and runs good tests in the browser, versus vendors having essential code with design flaws (due to incompetence), ‘soft metrics’ which are interpretable and thus prone to false positives, and the lack of cyber security knowledge to protect their collection mechanism. When you mismeasure, ie. the noise is louder than the signal, you simply can’t distinguish fraud from humans. Any usage of AI or advanced machine learning will not help against that, as the collected data is processed unsupervised and the volumes are just too big to flag it manually. This means an arbitrary line is drawn (the conservative or trigger happy line) to cluster the data into human and fraud clusters. Mismeasured results in a mismanaged outcome: Drawing incorrect conclusions will seriously affect the growth and thus the performance of your business in the long run.

Final words

The costs of doing real fraud detection are high when you need large neural nets and/or machine learning models to separate human from fraudulent traffic. It is not just detection in real-time, but these models needs also to be updated continuously to include new fraud types, new modus operandi of fraudsters, etc. Simply because a fraud detection vendor you don’t want any time gaps between the appearance of a new type of fraud and the detection of that fraud.

Performing real-time fraud detection is a complex process, and hard to do it right. This is much more complex than writing a JavaScript collecting data from the browser. When this JavaScript contains elementary flaws, it sets the standard for their overall detection quality. We can’t see or know how the back-end processes work and how well that software is written and configured, but based on the public available JavaScript code I don’t expect it will suddenly be super-sophisticated high-tech, on the contrary I’m afraid it just isn’t.

10 years ago, in 2013, separating fraud from human traffic was a relatively simple problem. You could just look at the user agent, look at the existence of variables and objects in the browser, webdriver flag, look at some webdriver residues, or rely on a pixel being fired, etc. These days it is much harder and only few understand this entirely throughout the tech stack. Some vendors take advantage of this and sell whatever you want to hear (good marketing, poor product), and unfortunately it is too complex and too technical and thus too time consuming for most companies to verify what really is needed and what happens at their digital frontdoor. Back to the one million dollar question ‘What is successfully in addressing these types of fraud?‘ Answer: Overreporting (ie. higher fraud percentages) isn’t better, underreporting (ie. almost no fraud) is also sub-optimal. Ad-fraud detection has to be accurate. Period!

2023-10-31

Would like the excel sheet? Want to know more? Leave a comment, connect or DM

[1] https://en.wikipedia.org/wiki/Law_of_the_instrument#Kaplan

[2] https://en.wikipedia.org/wiki/Polyfill_(programming)

[3]

[4] https://www.linkedin.com/posts/kouwenhovensander_javascript-frauddetection-adfraud-activity-7052261359882752000-fHTT

[5] https://oxford-biochron.com/affiliate-performance-report/

[6] https://en.wikipedia.org/wiki/Telephone_Consumer_Protection_Act_of_1991

How much fraud did Oxford Biochronometrics measure in 2023 Q3’s PAID traffic? In order to provide a clean answer we’ll break down the traffic into two groups: (1) Desktop traffic and (2) Mobile traffic. The ratio is can be seen in figure 1 where desktop traffic is just shy of 20% (1 in 5 visitors) of the total traffic. The other ~80% (4 in 5 visitors) were using a mobile device.

The next paragraphs will break down the traffic into fraud and human by operating systems and browser types.

Desktop traffic

How much fraud did appear (1) during 2023Q3, (2) PAID traffic only, (3) desktop only, (4) measured at the landing page(s)?

The most inner ring in figure 2 contains the total percentages for both groups: fraud and/or human. Each of these groups are broken down to the operating systems and subsequently to the browser which has been used to access a landing page. The percentages in each ring add up to 100%.

Desktop traffic in 2023Q3 had ~33.3% fraud: Exactly 1 in 3 visitors! That is ~6% more than measured during the previous quarter. Almost than (~61%) of desktop fraud presented itself as MS Windows, then Mac OS (~27%) then ChromeOS (~7%) and Linux (~5%). Within the human group different percentages can be seen: ~63% of the traffic is MS Windows, Mac OS (~30%), ChromeOS (~6%) and Linux (~0.3%).

When looking at the browser types the majority of fraud had ‘Windows with Chrome’ as User Agent. This doesn’t mean that those devices are really Windows machines with Chrome, they only claim to be. It can also be seen that volume wise traffic from Linux is ~32 times more likely to be fraudulent (0.17% human vs. 5.28% fraud).

Mobile traffic

Looking only at mobile traffic in 2023Q3, we can see in figure 3 that this type of traffic had ~12.5% fraud, roughly 1 in 8 visitors. An increase of ~3% compared to the previous quarter. It is again a completely different picture compared to desktop traffic in 2023Q3 where 1 in 3 visitors was flagged as fraud. The ratio Android / iOS in human traffic is about 55:45, the exact numbers are: Android (56.39%), iOS (43.61%). The picture changes when looking at fraudulent traffic, where the ratio is 80:20, the exact numbers are: Android (80.57%), iOS (19.43%).

This shows -just like previous quarters- that Android traffic contains more fraud than iOS. Also, nearly all iOS fraud doesn’t originate from real iOS devices. The fraud originates from Android or (special) desktop browsers pretending to be an iOS device by faking the user agent and device fingerprints.

Year-over-Year comparison of the results

Comparing 2023Q3 with the same quarter last year 2022Q3 provides a comparison without seasonal effects. In 2022Q3 the fraud% on desktop (34.71%) is similar to this year’s percentage (33.27%). Looking at mobile traffic shows a different picture. In 2022Q3 fraud on mobile was 19.01% (~1 in 5 visitors), but in 2023Q3 it is 12.5% (1 in 8 visitors).

Conclusion

Knowing that the desktop traffic is only 1/5th of the total volume shows that the desktop fraud percentage is only ~6.5% of the total volume. This is still too much, but not as bad as it looks in the chart. The ~12.5% fraud in mobile traffic is lower than last year’s fraud. It is the result of the following loop: measuring, optimizing campaigns and sources, measuring, optimizing campaigns and sources, etc. [1]

Note: The numbers are based on traffic measured at landing pages, microsites, lead gen forms, digital sales, check-out pages using Oxford Biochronometrics’ SecureLead.

Legend of the donut charts:

🟩 GREEN indicates human MOBILE traffic,

🟦 BLUE indicates human DESKTOP traffic

🟥RED indicates fraudulent traffic, mobile or desktop.

[1] https://www.linkedin.com/posts/kouwenhovensander_leadgen-ecommerce-digitalmarketing-activity-7033787920691949568-0znz

#digitalmarketing #leadgeneration #adfraud #ios #android #b2c

Earlier this week I read an article in Het Financieele Dagblad (fd.nl) a Dutch financial website [2][3]. It was about bots scraping their content in order to build a vast dataset upon which large language models can be trained. The first paragraph ends with ‘Attempts to block websites don’t suffice’ and a bit further in the article again it is iterated that it’s ‘technically difficult to shield websites against bots’, and ‘they discovered that bots copy articles protected with a paywall’.

This doesn’t surprise me at all. Some bots have become sophisticated and without specialized software they are indistinguishable from normal human visitors. When the stakes are high, the potential profit is enormous, and when you’re able to pay a thousands of $ a month to hire a team of specialists able to make and continuous update the best content scraping bots known to mankind, then.. then you’re king of your bot-army. Ah, and for those wondering: These bots will happily ignore the robots.txt, terms and conditions, etc.

Content scraped from millions of news articles end up in LLMs (Large Language Models) like ChatGPT, Bard and a zillion lesser known LLMs. These models, once trained, make big bucks without any compensation to the original content writers and owners. To make it even more bitter “copycat websites” use bots to scrape the content from news sites, top publishers, then rewrite the content as an “rewritten by an LLM”-version on their “news websites” (read: MFA news websites[8]), place ads on these websites to monetize the “hard work of the bots”, as can be read in this Bloomberg News article [4][5]. These content farms don’t add value, on the contrary they host a plagiarized version of the news, fake news and/or just clickbait, only and only to lure people (and don’t worry: also bots) to their websites where these ads are shown. The reason why they use LLMs to rewrite the articles is that it’s hard to detect plagiarism using automated checker tools. Ironically, these LLMs (re)generating copycat news articles are trained on the content from the very same news websites.

The bills and *cough* hard work of these copycat websites are indirectly paid by advertisers. By having lots of advertisement slots available, they promise advertisers and agencies the world! Luckily, the URLs of these websites will be relatively new and aren’t relevant reputation wise. So, they can typically be found in the so called “long-tail” of websites. Exactly where your advertisements shouldn’t be placed. One more reason to use an inclusion whitelist of domains where your ads are allowed.

Web Archiving

Another form of content stealing is done by archive.today [1]. The website claims to be a time capsule for webpages. At a very first glance a very noble mission statement! Except that this service can be used to capture content from news publishers, paywalled websites, etc. and once captured and archived the articles can immediately be accessed free of charge, even articles behind a paywall.

Be able to achieve this implies archive.today has subscriptions to most of the newspapers, magazines, and not just limited to the fd.nl but also ft.com, nytimes.com, wsj.com, reuters.com, bloomberg.com, also many other non-financial news websites and also in many other languages. Try it for yourself and go to https://archive.today with an URL of an news article behind a paywall, copy/paste the URL in the dark blue search bar to search the archive. If the URL has not been archived yet, you can request their bots to make a capture of the URL by copy/pasting the URL in the red bar.

This implies that all of these publishers and news websites don’t have any bot-detection present. And if they do, clearly it doesn’t work!

So, how does the archive.today bot work? And with this one many others, because most bots share 95% of the technology stack, only the remaining 5% makes it either an easy-to-detect bot, or a pain-in-the-neck bot. To validate this, I requested a few URLs to be added to the archive. To make my life easier I added an unique suffix to each URL to recognize the specific request made by the bot. The websites hosting these URLs contain Oxford Biochronometrics ’ fraud detection technology and thus I should be able to isolate this “visitor”.

How do their bots look from Oxford Biochronometrics ’ perspective? All recorded visits by their scraping bot used a similar technology stack:

1. Based on the IP addresses the bots all reside in a data center, though each in a different country in my small sample test

2. The used User Agent was: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36

3. The browser engine used was indeed a true Chrome/92 browser. If it were a spoofed userAgent, the detection would have flagged it because of that. Spoofing is a red flag!

4. The navigator.webdriver was not existent, in a real Chrome/92 this value should be ‘false’, or ‘true’ when the browser is being controlled remotely. Again a red flag!

The list above can be seen as a low hanging fraud checklist [6] and contains a lot of useful information. The first bullet point shows that the bot resides in data centers, probably all around the world. The first question you should ask yourself is: Why are data center visitors allowed to read premium articles? Which visitors would ever visit your web site from a data center? The 24/7 monitoring and maintenance team? The only possible reason I can think of is VPN users. That’s because VPN endpoints reside in data centers ensuring high bandwidth and low latency from/to the Internet.

Chrome/92 was released in July 2021 [7]. That’s over two years ago! The percentage of vistitors using this browser in Sep 2023 on a desktop is near zero, and if a Chrome/92 visitor is coming from one or more data centers, all sharing the same membership account, you know what to do! Another question of course is: Why hasn’t archive.today updated this browser to a more modern Chrome version? Simply, because it still works!

The forth bullet point would be a technical check. This should be validated in the browser using Javascript at the moment the webpage is loaded and accessed. This shows that no bot detection is present at these web sites, as this is the one of the most basic checks which reveals the browser IS a bot. BTW. It’s also very easy for the bot to have this flag set to the value ‘false’, so don’t rely on this too much! Besides this technical checks more clues are available in the data that reveal this visitor is a bot. This also accounts for other bots from other ‘indexing’ or ‘scraping’ services. Making a bot using browser automation isn’t that hard. The part where it becomes difficult is: How to make the bots blend in AT SCALE! That’s the hard part.

Because subscriptions are used to access paywalled and/or premium articles, a simple analysis can be run: which accounts accesses all our articles, how often, 24/7, and what percentage of those visits comes from a data center? And data centers in multiple countries? To know which IP ranges are data centers: Databases with metadata and information per IP address range can be bought, and once bought they should be updated monthly, to include newly bought data centers IP ranges.

The blog started with some quotes: ‘Attempts to block websites don’t suffice’, ‘technically difficult to shield websites against bots’, and ‘they discovered that bots copy articles protected with a paywall’ from the fd.nl article talking about bots scraping their content. Apparently, it appears that with some reasoning, some basic analytics and some IT knowledge it isn’t that difficult to spot bots. So, how to fight these bots?

Analytics, analytics and analytics.

Without knowing who is loading your content, what devices are visiting you, where your visitors are coming from you will always be guessing. To be able to separate your legitimate audience from bots and fraud you’ll have to slice the data, segment it into groups. Having read this article you’ll know what to look for, it’s fairly simple, see the 4 point checklist above.

Once bots are blocked, it is just a matter of time until they adapt and/or update their technology stack. It would be wishful thinking to think that they just disappear. So, once adapted they should reappear in the analytics. Where? Why? Because, as mentioned before: It’s hard to make bots at scale and blend in with the regular human audience. Due to the large scale these bots have to visit your website it automatically becomes a group of statistical outliers and thus will be visible in the analytics. The underlying question is: What makes them unique this time? And next time? And the next-next time? etc.

Use common sense

Although, I do see the added value of an Internet archive or an Internet Museum preserving content, screenshot webpages and archive entire web sites, at this moment the service is abused to read premium content for free. The simplest and cleanest way without affecting news publishers (and many others) too much and still adhering to their mission would be: a 30 day non-viewing period for newly archived web pages.

In the other case where fake news websites are launched either to spread fake news based on real actual content, and/or to monetize visitors using advertisements. These bots will work similar to the archiving bot described above. Will they adapt when data center IP ranges are blocked? Yes, they will. Will they start using residential proxies? Yes they will, but at a price: > $2 per Gigabyte. Once you add a bot detection layer and make their life even more miserable, the price of running and continuously updating an infrastructure with scraping bots will again increase.

The goal is: Make it economically unfeasible! Advertisements will not provide that much profit to cover all these expenses, especially when advertisers start using common sense and avoid advertising on MFA sites by using domain whitelists, and also whitelists of Apps since this phenomenon surely will shift -or already exists- to fake mobile news apps.

In the end it is all about common sense and the use analytics to see where and how the scraping bots move over time. Content isn’t a high-value product like limited Nike sneakers, Taylor Swift tickets, NFL tickets, etc. so making it economically unfeasible is doable and will work.

Questions? Comment, connect and/or DM

Jan Fred van Wijnen

Sandra Olsthoorn

NDP Nieuwsmedia

News Media Europe

#adfraud #fakenews #digitalmarketing #MadeForAdvertising

[1] https://en.wikipedia.org/wiki/Archive.today

[2] https://fd.nl/bedrijfsleven/1490101/nederlandse-media-willen-plundering-door

[3] https://archive.ph/hCzVe (archive.today archived version of article at [2])

[4] https://www.bloomberg.com/news/articles/2023-08-24/ai-chatbots-help-web-content-farms-copy-work-from-top-publishers-report-says

[5] https://archive.ph/LpnlL (archive.today archived version of article at [4])

[6] https://www.linkedin.com/pulse/data-exploration-how-help-you-catch-low-hanging-fraud-kouwenhoven

[7] https://support.google.com/chrome/a/answer/10314655?hl=en

[8] MFA (Made for advertising), https://en.wikipedia.org/wiki/Scraper_site#Made_for_advertising