Configure a Firefox web extension from Selenium

I’ve been wanting to add automated tests for Referer Modifier for a while, and now I finally got around to implementing some using Selenium (which lets you remote-control a browser). One tricky question to solve was: How do I automatically configure the freshly installed Firefox add-on?

Selenium has interfaces to open a page in the browser, find elements, click on them, and so on. It also has a way to install an add-on, in my Python unittest code it looks like this:

    def setUp(self):
        self.browser = webdriver.Firefox(options=self.options)
        self.browser.install_addon(str(self.addon_path), temporary=True)

The self.addon_path is the path to the ZIP archive containing the add-on, setting temporary=True is necessary because Firefox refuses to install unsigned add-ons permanently. But how do I configure the settings I want to test?

Semi-obvious answer: also automate the options page. Which leads to the next question: How to I open the options page? If you use Firefox with add-ons you may have noticed that add-on configuration pages have URLs like this:

moz-extension://[some UUID goes here]/page.html

Hm, okay, but what’s the right UUID? It’s definitely not the add-on ID set in the web extension manifest. As it turns out, the UUIDs are randomly generated on each computer to make them hard to guess, and a mapping from add-on IDs to the local UUIDs is stored in the extensions.webextensions.uuids preference. Next question: How do I get that mapping?

Answer: Not at all, it seems Selenium has no way to access Firefox preferences once the browser is running. But it has something else: It lets you set up a Firefox profile (including preferences) before starting the browser… If I can’t read the mapping, maybe I can just provide my own? 🤔

    @classmethod
    def setUpClass(cls):
        cls.ext_dir = Path(sys.argv[0]).parent
        with open(cls.ext_dir / 'manifest.json') as fh:
            manifest = json.load(fh)

        cls.addon_path = (cls.ext_dir /
                          f'referer-mod-{manifest["version"]}.zip').resolve()
        addon_id = manifest["browser_specific_settings"]["gecko"]["id"]
        addon_dyn_id = str(uuid.uuid4())
        cls.config_url = f'moz-extension://{addon_dyn_id}/options.html'
        print(f'Dynamic ID: {addon_dyn_id}')

        profile = webdriver.FirefoxProfile()
        # Pre-seed the dynamic addon ID so we can find the options page
        profile.set_preference('extensions.webextensions.uuids',
                               json.dumps({addon_id: addon_dyn_id}))
        # [...]
        cls.options = FirefoxOptions()
        cls.options.profile = profile

And it works! What this code does is:

  1. Read the manifest file of the add-on.
  2. Retrieve the static add-on ID from the manifest.
  3. Create a random UUID.
  4. Create a new Firefox profile and store the mapping from the add-on ID to the UUID as JSON in the extensions.webextensions.uuids preference.

The resulting cls.options is what’s used as the options parameter in the first snippet above. The pre-configured mapping is used when the add-on is installed, and because my test knows the UUID it generated (still randomly!) it can later load the config_url generated above, and import the configuration I want to test:

        test_config = (self.ext_dir / 'test_config.json').resolve()
        self.browser.get(self.config_url)
        import_file = self.browser.find_element_by_id('import_file')
        import_file.send_keys(str(test_config))
        import_button = self.browser.find_element_by_id('import_button')
        import_button.click()

And done! Of course if your add-on doesn’t support configuration import, or you actually want to test the add-on UI (good idea!) a few more actions will be necessary. If you want to see the full test, look at test.py in the Referer Modifier repository.

Testing Python log output

Imagine you have a module and want to make sure everything’s working, so you test. For functions the idea is usually pretty simple: Provide some inputs, see if the output looks as expected. But what the function is also supposed to write some log messages and you want to check if they look the way they should? I was asking myself that question on the weekend and found a solution that I think is fun: Adding a custom log handler!

Maybe you’ve already used the logging module? Logging messages is pretty simple, for example:

import logging
logger = logging.getLogger(__name__)

def add(*vargs):
    logger.debug('Calculating the sum of %s', vargs)
    res = sum(vargs)
    logger.debug('Result is %d', res)
    return res

Let’s assume you have that in a file add.py, and are going to write tests in a separate file. A simple unittest-based functionality test might look like this:

import add
import unittest

class AddTest(unittest.TestCase):
    def test_add(self):
        self.assertEqual(add.add(1, 2, 3), 6)

But what if you want to check if the debug messages are logged as expected? I realized you can tap into the logging framework for that! The logging module uses handlers to send log messages wherever they are supposed to go, and you can add as many as you want, so why not add one that sends output to the test?

First you’re going to need a logger for the target module:

        logger = logging.getLogger('add')
        logger.setLevel(logging.DEBUG)

Setting the level of the logger is necessary to ensure you really get all messages (unless that has been set elsewhere already), but keep in mind that that’s a process-wide setting. In a simple unit test like here that’s not going to cause trouble. Now that we have the logger we need to add a handler.

Also I want to have the input parameters and expected result in variables, so I can use them later for comparing with the log messages:

        params = (1, 2, 3)
        expected_sum = 6

Option one: A temporary log file

        with tempfile.SpooledTemporaryFile(mode='w+') as log:
            handler = logging.StreamHandler(stream=log)
            logger.addHandler(handler)
            try:
                self.assertEqual(add.add(*params), expected_sum)
            finally:
                logger.removeHandler(handler)
            log.seek(0)
            logdata = log.read()

The logging.StreamHandler can write to all sorts of streams, its default is sys.stderr. I’m using a tempfile.SpooledTemporaryFile as the target because it is automatically cleaned up as soon as it is closed, and the amount of log data will be small, so it makes sense to keep it in memory.

The try/finally block around the function I’m testing ensures the handler is always removed after the function call, even in case of an exception (including those from failed assertions).

In the end you just have to read the file and check if the output looks like it should.

        lines = logdata.splitlines()
        self.assertEqual(lines[0], f'Calculating the sum of {params!s}')
        self.assertEqual(lines[1], f'Result is {expected_sum}')

This also shows the disadvantages of this method: You end up with a wall of text that you have to parse. With two lines it’s not too bad, but with a lot of output it may get messy.

You can remedy that somewhat by attaching a Formatter to the handler, which as the name indicates lets you format the log messages, including adding some metadata.

Option two: A message queue

        q = queue.SimpleQueue()
        handler = logging.handlers.QueueHandler(q)
        logger.addHandler(handler)
        try:
            self.assertEqual(add.add(*params), expected_sum)
        finally:
            logger.removeHandler(handler)

This code is a bit shorter, because there’s no file to open. Instead the log messages are added to the queue, and I can retrieve them message by message:

        self.assertEqual(q.get_nowait().getMessage(),
                         f'Calculating the sum of {params!s}')
        self.assertEqual(q.get_nowait().getMessage(),
                         f'Result is {expected_sum}')

This has two advantages:

  1. I always get complete messages, no need to worry about splitting lines and newlines in messages.
  2. The objects in the queue are not strings, they are LogRecord objects, which hold all the metadata of the log message. Though in this example I’m just like “give me the message” and that’s it.

Conclusion

Turns out the Python logging module is easier to use than I had thought when I started figuring this out, and is fun to play with. Of course with more complex tests this kind of analysis might get more complex, too: You might not want to look at every message (maybe a Filter helps?), or you might not be sure which order they arrive in.

Have fun coding, and if you want you can find my full example code on GitHub.

Fun with Javascript, or: Setting document.referrer

It’s a bit of an odd thing for me to say, but yesterday I had some fun playing with Javascript. I maintain a little Firefox add-on called “Referer Modifier” to modify the HTTP Referer header, an since Web Extensions became the standard for browser extension Javascript is simply the language of choice for that.

In case you don’t know, when you click a link by default most browsers will send the URL of the site on which the link was to the site you’re going to as the “Referer” (the spelling is a typo that made it into the specification). There are some uses for this, for example if you get to this blog via a link I can see that in the statistics, but it can also be a privacy issue. That’s why I wrote an add-on that can change, or simply remove that information.

A while ago someone asked if the add-on would also support changing the Javascript document.referrer property (oh hey, spelled correctly there!). It’s basically the same thing, except instead of being sent to the server it’s available to scripts running on the site, with the same possible privacy issues. At the time I didn’t have time to look into it, but later someone else was so kind to look into it and leave a comment: Firefox used to make document.referrer match the HTTP Referer, but there’s a bug since version 69 that stopped that. The comment also included a link to how another add-on called “Smart Referer” solved the problem.

So yesterday I set out to find my own solution. From looking at the Smart Referer code I knew I’d need to add a content script to Referer Modifier, code that would run in the context of each loaded site. But I didn’t like their approach of getting configuration to the content script, which runs in a limited environment: They generated the script (as text) at runtime, and included the configuration in that text as JSON. I guess it works, but it seems overly complicated.

Looking around MDN (highly recommended for any kind of web stuff!) I found that there’s a messaging mechanism specifically for communication between different parts of an add-on: runtime.sendMessage(). It’s perfect for my purposes: The content script can send a message to the background script of the extension containing the URL of the current page and the pre-set document.referrer, and the response will tell it how to overwrite the latter. That way there’s no need to generate code, and I could just write it as a (pretty short) script file that Firefox will load when a page is loaded. Also the code that checks the configuration is literally the same, no unnecessary complexity there!

In the existing background script I just had to add a listener for incoming messages, and write a function to send the responses. It took a little while though to figure out why I’d get “invalid URL” errors when trying to parse the default document.referrer and it was unset. Turns out while I was checking for null, “unset” is actually an empty string in this case.

That was a fun little exercise, and here’s the patch with everything put together if you’re interested. 😉

So, about THAT GnuTLS session ticket bug (CVE-2020-13777)

It’s been almost two weeks ago now that I discovered the bug in GnuTLS now known as CVE-2020-13777. The report is issue #1011 on the GnuTLS bug tracker. Here I want to talk a bit about how I discovered the bug and some thoughts on its impact.

Session resumption for mod_gnutls

I was working on session resumption support for proxy connections in mod_gnutls. To test I had a setup with two Apache servers with mod_gnutls:

  1. a front-end server, which should cache session tickets and try to resume sessions, and
  2. a back-end server which should issue and use the session tickets.

After session resumption worked as it should, I restarted the back-end server to invalidate the session ticket the front-end server had cached. On server stop mod_gnutls wipes the primary key used to protect session tickets from memory, so the freshly started server should have had no way to decrypt the old ticket. But somehow both sides still reported successful session resumption. That absolutely shouldn’t happen, so I started investigating.

At first I worried about a bug in mod_gnutls: Maybe the log output about resumption wasn’t correct? Or, way worse, was the key not used correctly? Over time I ruled those out, and instead started to suspect a bug in GnuTLS.

Testing GnuTLS itself

To check that I tested gnutls-serv, a relatively simple TLS server included with GnuTLS.

As the client I had to use OpenSSL’s s_client, because contrary to gnutls-cli it offers a way to cache sessions to disk and load them. I needed that because for my test I needed to change the server between initial session and resumption, and the --resume flag in gnutls-cli is way too fast to do that manually.

This brought me to the steps I described in the bug report to reproduce the problem:

  1. Start a server with valid credentials: gnutls-serv --x509keyfile=authority/server/secret.key --x509certfile=authority/server/x509.pem
  2. Connect to the server and store resumption data: openssl s_client -connect localhost:5556 -CAfile authority/x509.pem -verify_return_error -sess_out session.cache
  3. Stop the server started in step 1.
  4. Start a server with bogus credentials at the same address port (imagine a real attacker redirecting connections only if the client is attempting resumption): gnutls-serv --x509keyfile=rogueca/mitm/secret.key --x509certfile=rogueca/mitm/x509.pem
  5. Connect again, using the stored resumption data: openssl s_client -connect localhost:5556 -CAfile authority/x509.pem -verify_return_error -sess_in session.cache

That worked flawlessly on the first try. At that point it sunk in that I had a serious security issue on my hands that allowed man-in-the-middle (MITM) attacks on GnuTLS servers, and did a round of cycling to clear my mind.

After a few more tests I wrote up the bug report. It was clearly too bad to sit on until I had all the details, what I had was already more than bad enough to require an immediate fix. A quick attempt to reproduce the problem with TLS 1.2 had failed (looking back I must’ve made some mistake in the commands), so I guessed there was some difference in how GnuTLS handled tickets for TLS 1.3 and 1.2.

It gets worse

Daiki Ueno commented on my report:

“Looking at the code path, ticket encryption key and decryption key are all-zero, until the first rotation happens. In TLS 1.3, that can only bypass the authentication, but in TLS 1.2, it may allow attackers to recover the previous conversations.”

With that hint I was able to find the relevant GnuTLS code pretty quickly, and indeed there was no hint of different ticket encryption for the different TLS versions. So I tried to reproduce the problem with TLS 1.2 again, and indeed it worked. With a bit of fiddling I was able to get the server to write the key to disk during the resumed session, and decrypt the previous, initial session. Mind that that was just the easiest way to do it, it is possible to get the same data just from captured network traffic.

Allowing MITM with TLS 1.3 is already really bad, but effectively unprotected tickets with TLS 1.2 are way worse, because it means it’s possible to decrypt past recorded sessions. If the NSA has such a TLS connection from a year ago stored somewhere? Whoops, they can decrypt it now.

This is because of design flaws in TLS 1.2:

  • Resumed sessions use exactly the same secret keys as the initial session. So if you get access to the secrets at any point (like from a not properly protected ticket here), you can decrypt all the sessions.
  • Session tickets are sent from the server to the client right before starting to use the negotiated encryption. For this bug that means TLS 1.2 sessions are vulnerable as soon as the server sends a ticket, not only on resumption.

TLS 1.3 mostly avoids those issues by doing a fresh Diffie-Hellman exchange during resumption (simply put, it negotiates new keys, and just keeps authentication in place – which allowed the MITM here), and sending the ticket to the client encrypted. During session resumption the client naturally has to send the ticket in the clear.

I wrote “mostly” above, because there is one part where TLS 1.3 should be vulnerable to passive decryption, that I haven’t seen discussed much yet: Early data. I say “should” because I haven’t tested it, but based on what I know about how early data works it should be. Early data is sent with the request for session resumption to speed up communication (hence the name), so it cannot be protected by the new Diffie-Hellman exchange yet and uses key material from the ticket, similar to what TLS 1.2 does for the whole session. “Early data” tends to be small (if it is used at all, mod_gnutls does not support it), but if that small piece of data contains, say, an authentication token… Not good.

Consequences

First of, big thanks to the GnuTLS team for handling the bug report and fix well!

Secondly, this demonstrates why the design flaws in TLS 1.2 I described above are so bad. If a ticket is compromised, all associated sessions are. And these are not the only flaws, and even though most can be avoided with careful implementation: Don’t rely on TLS 1.2 (much less older) any more if you can avoid it! From the client side that’s hard to enforce (for example, way too many websites still offer only TLS 1.2), but seriously consider it if you’re operating a TLS server. Although I understand that may be difficult too: All modern browsers support TLS 1.3, but I have no idea how many legacy devices, like phones and IoT devices that might not have seen updates in a long time, might still need your servers.

After this it gets murky. Should you still use session tickets with TLS 1.3? I think it’s reasonable to, assuming of course that the implementation is sound. Should you use early data? I’m leaning towards no, and have no plans to support it with mod_gnutls, because it doesn’t offer forward secrecy in case of ticket compromise (unlike the rest of TLS 1.3), and I doubt the slight speedup is worth the complexity.

The issue with session ticket key rotation

A particularly interesting case is the session ticket key rotation as implemented in GnuTLS. What I learned while looking at the code that caused this bug I learned that the rotation doesn’t in fact rotate the primary key (as generated using gnutls_session_ticket_key_generate()) based on time, it only derives the key used for a particular session from it in a time-based manner. That means it does not protect against an attacker who is able to steal the primary key from server memory, which I consider to be the main risk for tickets. That is because if you can steal that key, you can decrypt all tickets, and at the very least do MITM attacks.

Some have called the rotation scheme entirely useless because of that. I’m not yet sure if I agree with that, but I do want to implement a rotation of the primary key in mod_gnutls to protect against server compromise. However, I also see that it is extremely difficult for a library like GnuTLS to offer a key rotation mechanism that reliably works for all possible use cases. For example with mod_gnutls I’d have to synchronize a truly random new key across multiple server processes. Maybe clarifying the documentation on exactly what is and isn’t rotated is the best GnuTLS can do on the library side.

NSA Admins and Privacy

While reading through an Intercept article on the NSA’s XKEYSCORE program (simply put: a search engine for data captured by the NSA), I came across this gem:

When systems administrators log into XKEYSCORE servers to configure them, they appear to use a shared account, under the name “oper.” Adams notes, “That means that changes made by an administrator cannot be logged.” If one administrator does something malicious on an XKEYSCORE server using the “oper” user, it’s possible that the digital trail of what was done wouldn’t lead back to the administrator, since multiple operators use the account.

Behind the Curtain, The Intercept, 2015-07-02

It’s common knowledge that shared accounts are extremely bad practice from a security standpoint. It’s difficult to revoke access for a specific person without causing a fuss for everyone else, or attribute actions to a specific person. And that’s exactly why I would want a shared administrator account if I wanted to avoid responsibility. “Someone ran an illegal query? Wasn’t me, and you can’t prove otherwise!” See, NSA admins know about privacy… They’re just selective about who should have it. 🙄

New in Java 8: Catching Integer Overflows

I’ve recently discovered a nice new feature in Java 8: methods to properly handle integer overflows. Consider the following example:

public class OverflowTest
{
	public static void main(String[] args)
	{
		int a = Integer.MAX_VALUE;
		int b = 1;

		int c = a + b;
		System.out.println(a + " + " + b + " = " + c);
	}
}

When you compile and run it, this is the result:

$ javac OverflowTest.java
$ java OverflowTest
2147483647 + 1 = -2147483648

Quite obviously, this can’t be mathematically right. The problem occurs because an int has a limited size of 4 byte. When this size is too small to store a number, the value will overflow and wrap around from the largest to the smallest possible value (or vice versa if the number is too small).

If you wanted to catch possible overflows in previous Java versions, you had to write your own checks. In Java 8, java.lang.Math offers new methods that will take care of that for you.

public class OverflowTest
{
	public static void main(String[] args)
	{
		int a = Integer.MAX_VALUE;
		int b = 1;

		int c = Math.addExact(a, b);
		System.out.println(a + " + " + b + " = " + c);
	}
}
$ javac OverflowTest.java
$ java OverflowTest
Exception in thread "main" java.lang.ArithmeticException: integer overflow
	at java.lang.Math.addExact(Math.java:790)
	at OverflowTest.main(OverflowTest.java:8)

If an overflow occurs, Math.addExcact(int,int) throws an ArithmeticException, which you can catch and handle. Similar methods exist for other operations and the long type. What to do in case of an exception depends on your application, and may be quite complicated. This post is just about mentioning these shiny new methods. 😉

Anyway, if I want to print the correct result in my example, I can simply fall back to long. At 8 bytes length it can definitely store the result of adding two ints. Note that I have to cast at least one summand to long before adding them, or the intermediate result would still be an int and overflow.

public class OverflowTest
{
	public static void main(String[] args)
	{
		int a = Integer.MAX_VALUE;
		int b = Integer.MAX_VALUE;

		try
		{
			int c = Math.addExact(a, b);
			System.out.println(a + " + " + b + " = " + c);
		}
		catch (ArithmeticException ex)
		{
			System.err.println("int is too small, falling back to long.");
			long c = (long) a + (long) b;
			System.out.println(a + " + " + b + " = " + c);
		}
	}
}
$ javac OverflowTest.java
$ java OverflowTest
int is too small, falling back to long.
2147483647 + 2147483647 = 4294967294

If you need a way to handle numbers of (almost) unlimited length, take a look at java.math.BigInteger. Also note that catching possible overflows influences performance, so using *Exact everywhere instead of simple operators is probably a bad idea. Anyway, this is not one of the biggest news in Java 8 (lambda expressions!), but I think it’s neat, and also about the right scope for a quick blog post. 😀

Peculiar Ethernet Timings

Have you ever tried sending one Ethernet packet every 78 microseconds? If not, what would you expect to happen? Actually, I did that kind of experiment (and many others) last year in my graduation thesis “Development of a Scalable and Distributed System for Precise Performance Analysis of Communication Networks“, which is now published. For the thesis I developed a system called the Lightweight Universal Network Analyzer (LUNA), which can generate packets at precise times and record their arrival times, among other things. When I tested it on different hardware, I got some surprising results, as you can see in the figure below.

Graphs of IAT Distributions with Different Ethernet Hardware
IAT distributions from tests with 78 μs IST (Figure 8.12)

The diagram shows packet inter arrival times (IAT) on the x-axis. I had configured the packet source to send a packet every 78 microsecond, and the IAT measurement shows at which intervals they actually arrived. The y-axis shows how frequently a certain IAT occurred, note that it has a logarithmic scale. The differently colored curves are from different measurements:

  • The measurement for the red curve was done between two hosts equipped with Realtek RTL8111/8168B Gigabit Ethernet controllers,
  • the cyan one between two hosts with Intel Gigabit Ethernet controllers (82567LF and 82579LM, to be precise),
  • and the dark blue one via the loopback interface on one of the hosts for reference.

The hosts were sufficiently similar in processing power (for details, see chapter 8 of the thesis).

The loopback measurement looks as expected, with a strong peak at 78 µs IAT and a packets distributed around it. In both measurements with real hardware some packets were transmitted in rapid succession, probably after some of them were stalled. The really interesting thing, however, is the different behavior at and above the intended IAT. The measurement with Intel hardware led to a peak around 78 µs, although much wider than the loopback one. Using the Realtek cards, almost no packet arrived with the intended intervals, instead, there is a very wide peak around approximately 250 µs. All three measurements showed average IATs of 77 µs, though.

If you now think that the Intel hardware followed the timing pattern created by the software much better, well, it’s not that easy. Yes, the distribution looks more like the one I wanted, but the maximum deviation from the intended inter arrival time was actually much larger. For the red curve, representing the measurement with Realtek hardware, the rightmost signal (328 µs IAT) in the graph is indeed the maximum deviation. The largest IAT recorded in the Intel measurement, however, was 1922 µs. These outliers are not shown in the figure because otherwise the peaks would be very difficult to distinguish. You can find the detailed numbers in Table 8.8 in the thesis.

The hardware for this experiment was essentially just what was available at the lab. 😉 Nonetheless the results show that networking hardware can have an impressive impact on the timing behavior of packet transmissions. I’d really like to see some studies on other devices! Also, it may be interesting to check in what part the difference is caused by the hardware itself, and what influence the hardware drivers have on the results.

Quick Update on Kyūbunhama

Sorry to jump from sightseeing in Ise to a serious topic so suddenly, but I’m in Sendai right now, just got back from dinner with friends from the time I spent here as an exchange student, and one of them is the coordinator of the volunteer work I participated in after the tsunami in 2011. Of course, I used the opportunity to ask her about the current situation in Kyūbunhama and the surrounding area.

Heavy machines flattening ground

The gist of what she told me is this: People live mostly in temporary housing, while higher ground is being prepared for permanent buildings. In some places construction of permanent homes might start this year, but she didn’t seem sure about that. Life in the temporary buildings brings some troubles, including lack of space and bad insulation, leading to cold homes in winter and hot ones in summer. A big social issue is that existing neighborhoods have been broken up, which means that people — especially older people — are at risk of getting socially isolated.

They currently try to help by offering tea time with everyone who wants to come to give people an opportunity to talk, and distributing vegetables (I assume to encourage healthy eating, or maybe just to be nice).