Detecting headless browsers

Headless Browser
Hide & Seek
Sergey Shekyan, Bei Zhang
Shape Security

Who We Are
• Bei Zhang
Senior Software Engineer at Shape Security, focused on analysis
and countermeasures of automated web attacks. Previously, he
worked at the Chrome team at Google with a focus on the Chrome
Apps API. His interests include web security, source code analysis,
and algorithms.
• Sergey Shekyan
Principal Engineer at Shape Security, focused on the development of
the new generation web security product. Prior to Shape Security, he
spent 4 years at Qualys developing their on demand web
application vulnerability scanning service. Sergey presented
research at security conferences around the world, covering various
information security topics.

What Is a Headless Browser and How it Works
Scriptable browser environment that doesn’t require GUI
• Existing browser layout engine with bells and whistles
(PhantomJS - WebKit, SlimerJS - Gecko, TriﬂeJS - Trident)
• Custom software that models a browser (ZombieJS,
HtmlUnit)
• Selenium (WebDriver API)

What Is a Headless Browser and How it Works
Discussion will focus on PhantomJS:
• Backed by WebKit engine
• Cross-platform
• Popular
• True headless

PhantomJS World
PhantomJS
JavaScript Context
QWebFrame
QtWebKit
Web Page
JavaScript
Context
Control
Callback
Injection
PageEvent
Callbacks are
serialized
var page = require('webpage').create();
page.open(url, function(status) {
var title = page.evaluate(function() {
return document.title;
});
console.log('Page title is ' + title);
});

Legitimate uses and how you can beneﬁt
• Web Application functional and performance
testing
• Crawler that can provide certain amount of
interaction to reveal web application topology,
Automated DOM XSS, CSRF detection
• SEO (render dynamic web page into static
HTML to feed to search engines)
• Reporting, image generation

Malicious Use of Headless Browser
• Fuzzing
• Botnet
• Content scraping
• Login brute force attacks
• Click fraud
• Bidding wars
Web admins tend to block PhantomJS in production,
so pretending to be a real browser is healthy choice

How It Is Different From a Real Browser
• Outdated WebKit engine (close to Safari 5 engine, 4 y.o.)
• Uses Qt Framework’s QtWebKit wrapper around WebKit
• Qt rendering engine
• Qt network stack, SSL implementation
• Qt Cookie Jar, that doesn’t support RFC 2965
• No Media Hardware support (no video and audio)
• Exposes window.callPhantom and window._phantom
• No sandboxing

Headless Browser Seek
• Look at user agent string
if (/PhantomJS/.test(window.navigator.userAgent)) {
console.log(‘PhantomJS environment detected.’);
}

Headless Browser Hide
• Making user-agent (and navigator.userAgent) a
“legitimate” one:
var page = require(‘webpage').create();
page.settings.userAgent = ‘Mozilla/5.0 (Macintosh; Intel Mac
OS X 10.9; rv:30.0) Gecko/20100101 Firefox/30.0';

Score board
Phantom Web site
User Agent String Win Lose

• Sniff for PluginArray content
if (!(navigator.plugins instanceof PluginArray) ||
navigator.plugins.length == 0) {
   console.log("PhantomJS environment detected.");
  } else {
   console.log("PhantomJS environment not detected.");
  }

• Fake navigator object, populate PluginArray with whatever values you need.
• Spooﬁng Plugin objects inside the PluginArray is tedious and hard.
• Websites can actually create a plugin to test it.
• CONCLUSION: Not a good idea to spoof plugins.
page.onInitialized = function () {
   page.evaluate(function () {
   var oldNavigator = navigator;
   var oldPlugins = oldNavigator.plugins;
   var plugins = {};
   plugins.length = 1;
   plugins.__proto__ = oldPlugins.__proto__;
   window.navigator = {plugins: plugins};
   window.navigator.__proto__ = oldNavigator.__proto__;
   });
};

Score board
Phantom Web site
Inspect PluginArray Lose Win

• Alert/prompt/conﬁrm popup suppression timing
detection
var start = Date.now();
  prompt('I`m just kidding');
  var elapse = Date.now() - start;
  if (elapse < 15) {
   console.log("PhantomJS environment detected. #1");
  } else {
  }

• Can’t use setTimeout, but blocking the callback by
all means would work
page.onAlert = page.onConﬁrm = page.onPrompt = function ()
{
   for (var i = 0; i < 1e8; i++) {
   }
   return "a";
};

Score board
Phantom Web site
Timed alert() Lose Win

• Default order of headers is consistently different
in PhantomJS. Camel case in some header
values is also a good point to look at.
PhantomJS 1.9.7
GET / HTTP/1.1
User-Agent:
Accept:
Connection: Keep-Alive
Accept-Encoding:
Accept-Language:
Host:
Chrome 37
GET / HTTP/1.1
Host:
Connection: keep-alive
Accept:
User-Agent:
Accept-Encoding:
Accept-Language:

• A custom proxy server in front of PhantomJS
instance that makes headers look consistent with
user agent string

Score board
Phantom Web site
HTTP Header order Win Lose

• PhantomJS exposes APIs:
• window.callPhantom
• window._phantom // not documented.
if (window.callPhantom || window._phantom) {
} else {
}

• store references to original callPhantom, _phantom
• delete window.callPhantom, window._phantom
page.onInitialized = function () {
   page.evaluate(function () {
   var p = window.callPhantom;
   delete window._phantom;
   delete window.callPhantom;
   Object.deﬁneProperty(window, "myCallPhantom", {
   get: function () { return p;},
   set: function () {}, enumerable: false});
   setTimeout(function () { window.myCallPhantom();}, 1000);
   });
};
page.onCallback = function (obj) { console.log(‘proﬁt!'); };
Unguessable name

Score board
Phantom Web site
window.callPhantom Win Lose

• Spooﬁng DOM API properties of real browsers:
• WebAudio
• WebRTC
• WebSocket
• Device APIs
• FileAPI
• WebGL
• CSS3 - not observable. Defeats printing.
• Our research on WebSockets: http://goo.gl/degwTr

Score board
Phantom Web site
HTML5 features Lose Win

• Signiﬁcant difference in JavaScript Engine: bind() is
not deﬁned in PhantomJS prior to version 2
(function () {
   if (!Function.prototype.bind) {
   return;
   }
  })();
Function.prototype.bind = function () {
var func = this;
var self = arguments[0];
var rest = [].slice.call(arguments, 1);
return function () {
var args = [].slice.call(arguments, 0);
return func.apply(self, rest.concat(args));
};
};

• Detecting spoofed Function.prototype.bind for
PhantomJS prior to version 2
(function () {
   return;
   }
   if (Function.prototype.bind.toString().replace(/bind/g, 'Error') !=
Error.toString()) {
   return;
   }
  })();

• Spooﬁng Function.prototype.toString
function functionToString() {
   if (this === bind) {
   return nativeFunctionString;
   }
   return oldCall.call(oldToString, this);
}

• Detecting spoofed Function.prototype.bind for
PhantomJS prior to version 2
(function () {
   return;
   }
   if (Function.prototype.bind.toString().replace(/bind/g, 'Error') !=
Error.toString()) {
   return;
   }
   if (Function.prototype.toString.toString().replace(/toString/g, 'Error') != Error.toString()) {
   return;
   }
  })();

• Spooﬁng Function.prototype.toString.toString
function functionToString() {
   if (this === bind) {
   return nativeFunctionString;
   }
   if (this === functionToString) {
   return nativeToStringFunctionString;
   }
   if (this === call) {
   return nativeCallFunctionString;
   }
   if (this === apply) {
   return nativeApplyFunctionString;
   }
   var idx = indexOfArray(bound, this);
   if (idx >= 0) {
   return nativeBoundFunctionString;
   }
   return oldCall.call(oldToString, this);
}

Score board
Phantom Web site
Function.prototype.bind Win Lose

• PhantomJS2 is very EcmaScript5 spec compliant,
so checking for outstanding JavaScript features is
not going to work

• Stack trace generated by PhantomJs
• Stack trace generated by SlimerJS
• Hmm, there is something common…
at querySelectorAll (phantomjs://webpage.evaluate():9:10)
at phantomjs://webpage.evaluate():19:30
at phantomjs://webpage.evaluate():20:7
at global code (phantomjs://webpage.evaluate():20:13)
at evaluateJavaScript ([native code])
at global code (/Users/sshekyan/Projects/phantomjs/spoof.js:8:14)
Element.prototype.querySelectorAll@phantomjs://webpage.evaluate():9
@phantomjs://webpage.evaluate():19
@phantomjs://webpage.evaluate():20

var err;
try {
null[0]();
} catch (e) {
err = e;
}
if (indexOfString(err.stack, 'phantomjs') > -1) {
} else {
console.log("PhantomJS environment is not detected.");
It is not possible to override the thrown TypeError
generic indexOf(), can be spoofed, deﬁne your own

• Modifying webpage.cpp:
QVariant WebPage::evaluateJavaScript(const QString &code)
{
QVariant evalResult;
QString function = "(" + code + ")()";
evalResult = m_currentFrame->evaluateJavaScript(function,
QString("phantomjs://webpage.evaluate()"));
return evalResult;
}
at
at
at
at
at
• Produces

• Spoofed PhantomJs vs Chrome:
at querySelectorAll (Object.InjectedScript:9:10)
at Object.InjectedScript:19:30
at Object.InjectedScript:20:7
at global code (Object.InjectedScript:20:13)
at
TypeError: Cannot read property '0' of null
at HTMLDocument.Document.querySelectorAll.Element.querySelectorAll [as
querySelectorAll] (<anonymous>:21:5)
at <anonymous>:2:10
at Object.InjectedScript._evaluateOn (<anonymous>:730:39)
at Object.InjectedScript._evaluateAndWrap (<anonymous>:669:52)
at Object.InjectedScript.evaluate (<anonymous>:581:21)

Score board
Phantom Web site
Stack trace Lose Win

Score board
Phantom Web site
Stack trace Lose Win
SCORE: 4 4

How to turn a headless browser against the
attacker
The usual picture:
• PhantomJS running as root
• No sandboxing in PhantomJS
• Blindly executing untrusted JavaScript
• outdated third party libs (libpng, libxml, etc.)
Can possibly lead to:
• abuse of PhantomJS
• abuse of OS running PhantomJS

• In a lot of cases --web-security=false is used in
PhantomJS
var xhr = new XMLHttpRequest();
xhr.open('GET', 'ﬁle:/etc/hosts', false);
xhr.onload = function () {
console.log(xhr.responseText);
};
xhr.onerror = function (e) {
console.log('Error: ' + JSON.stringify(e));
};
xhr.send();

• Obfuscate, randomize the output, randomize the modified API
call
var _0x34c7=["x68x74x6D
x6C","x71x75x65x72x79x53x65x6Cx65x63x74x6F
x72x41x6Cx6C","x70x72x6Fx74x6F
x74x79x70x65","x73x74x61x63x6B","x70x68x61x6E
x74x6Fx6Dx6Ax73","x61x70x70x6Cx79"];var
html=document[_0x34c7[1]](_0x34c7[0]);var
apsdk=document[_0x34c7[1]];Document[_0x34c7[2]]
[_0x34c7[1]]=Element[_0x34c7[2]][_0x34c7[1]]=function ()
{var _0xad6dx3;try{null[0]();} catch(e)
{_0xad6dx3=e;} ;if(indexOfString(_0xad6dx3[_0x34c7[3]],_0
x34c7[4])>-1){return html;} else {return apsdk[_0x34c7[5]]
(this,arguments);};};

Tips for Using Headless Browsers Safely
• If you don’t need a full blown browser engine, don’t
use it
• Do not run with ‘- -web-security=false’ in production,
try not to do that in tests as well.
• Avoid opening arbitrary page from the Internet
• No root or use chroot
• Use child processes to run webpages
• If security is a requirement, use alternatives or on a
controlled environment, use Selenium

Tips For Web Admins
• It is not easy to unveil a user agent. Make sure you want
to do it.
• Do sniff for headless browsers (some will bail out)
• Combine several detection techniques
• Reject known unwanted user agents (5G Blacklist 2013
is a good start)
• Alter DOM API if headless browser is detected
• DoS
• Pwn

Start Detecting Already
Ready to use examples:
github.com/ikarienator/phantomjs_hide_and_seek

References
• http://www.sba-research.org/wp-content/uploads/publications/jsfingerprinting.pdf
• https://kangax.github.io/compat-table/es5/
• https://media.blackhat.com/us-13/us-13-Grossman-Million-Browser-Botnet.pdf
• http://ariya.ofilabs.com/2011/10/detecting-browser-sniffing-2.html
• http://www.darkreading.com/attacks-breaches/ddos-attack-used-headless-browsers-
in-150-hour-siege/d/d-id/1140696?
• http://vamsoft.com/downloads/articles/vamsoft-headless-browsers-in-forum-spam.pdf
• http://blog.spiderlabs.com/2013/02/server-site-xss-attack-detection-with-modsecurity-
and-phantomjs.html
• http://googleprojectzero.blogspot.com/2014/07/pwn4fun-spring-2014-safari-part-
i_24.html

Thank you!
@ikarienator
@sshekyan

Detecting headless browsers

More Related Content

What's hot (20)

Similar to Detecting headless browsers (20)

Recently uploaded (20)

Detecting headless browsers