In the ever-evolving world of connected TV (CTV) advertising, embracing the most rigorous approach to outcome measurement is crucial. As the landscape becomes more complex and competitive, marketers and advertisers must find reliable ways to validate the impact of their campaigns. A/B testing with incremental lift studies offers a powerful solution that is superior to alternatives like multi-touch attribution and media mix modeling (MMM).

The drawbacks of multi-touch attribution and media mix modeling

Before diving into the perks of A/B testing, let's quickly address the drawbacks of two attribution models: multi-touch attribution and MMM. Multi-touch attribution tries to cover various touchpoints in the customer journey, but falls short due to its reliance on arbitrary assumptions and focus on correlations instead of causality.

Meanwhile, MMM leans heavily on historical data to assess marketing channels but overlooks individual user-level interactions, making it susceptible to confounding factors. While both models may reveal intriguing correlations, they fail to demonstrate causality, leaving advertisers uncertain about their campaigns' true impact.

Additionally, it is important to recognize that CTV conversions may involve different devices and could be delayed compared to traditional digital advertising. To address these challenges surrounding measurements specific to CTV, advertisers can effectively leverage incremental lift studies to drive impactful and measurable campaign results.

How do CTV advertisers calculate incremental lift?

A/B testing, the same methodology used in clinical trials for vaccines and life-saving drugs, offers exceptional rigor for determining incremental lift.

In CTV advertising, A/B testing involves randomly assigning viewers to a test group (exposed to your ad campaign) or a control group (either exposed to a placebo creative or generated synthetically). These two groups are statistically identical, except for their exposure to your campaign. By comparing their behaviors and outcomes, you can establish a direct causal link between your campaign and its impact on viewer behavior.

However, this gold standard of advertising efficacy is not without its challenges. Ensuring adequate sample sizes is essential, as is focusing on experimental design and tracking relevant metrics. Advertisers must also consider factors such as suitable identifiers for accurate targeting and measurement.

Beyond measurement, failure  to correctly detect incrementality can lead to inefficient budget allocation to audiences with high baseline conversion rates therefore missing the opportunity to bring new incremental conversions with CTV.  It is crucial to distinguish between conversions driven by efforts such as retargeting and those that would have occurred anyway. In short, understanding incrementality is key for making smarter investments and improving campaign performance while avoiding overallocation in high baseline audiences.

Advertisers must carefully consider several execution options in CTV: PSA testing, geo testing, ghost bids, and synthetic groups. Each method has its unique advantages and drawbacks.

: This method is among the easiest to set up.  It involves running public service announcements (PSAs) as a control, allowing for a straightforward comparison against your campaign's performance. The downside is dedicating a portion of your ad budget to non-promotional content.

: This approach involves splitting your target audience into test and control groups based on geographical regions. However, implementing this approach can be complex and may not always account for regional differences that could skew the results.

:This advanced technique involves placing non-winning bids alongside your actual campaign to gather a clean sample without additional costs. The challenge lies in ensuring your non-winning bids accurately represent the same audience segments, which can present some technical challenges

: This method creates a control group that mirrors the test group using advanced statistical techniques and extensive user data. It aims to match demographics, behavior, and other factors for accurate campaign performance comparison against a group not exposed to the advertising.

We recommend starting with PSA testing for its simplicity and ease of implementation, particularly for those new to A/B testing and incrementality measurements in the context of CTV. As your expertise grows, consider exploring ghost bids or synthetic groups as advanced, cost-effective options for measuring the incremental lift of your CTV advertising campaigns.

It's time to ditch the guesswork and activate your campaigns with Simulmedia, the cutting-edge ad-tech company that delivers scientifically proven results. Experience the power of incremental lift studies and prove the impact of your advertising efforts with absolute certainty. Join the advertising revolution and 

Marketers and advertisers must find reliable ways to measure their connected TV campaigns. Our chief scientist explains why A/B testing with incremental lift studies is a powerful solution. 

How CTV Advertisers Can Better Measure Incrementality

Open AI’s ChatGPT, released in November of 2022, has taken the world by storm. Questions concerning the ethics of AI have surfaced, as well as the prospect of replacing countless jobs with this new technology. In fact, according to 

a study conducted by economists from Oxford University

, Artificial Intelligence could replace nearly 50% of jobs in the United States.

There is an upside to the innovation and ingenuity behind ChatGPT, though. The AI-based chatbot has the ability to exponentially increase efficiency, especially in the world of tech. As a company operating in the AdTech space, 

 have already found ways to utilize ChatGPT to optimize for efficiency, boost productivity, and deliver industry-leading results 

One of the truly mind-blowing aspects of Artificial Intelligence is that it is improving exponentially with each new query. According to Reuters, ChatGPT has already reached 

more than 100 million monthly active users

 since its release in November of last year — a new record. Although there have been issues with bandwidth, due to an excessive number of users overloading Open AI’s servers, ChatGPT’s user base will continue to grow, and so will its capabilities.

ChatGPT isn’t the only competitor in the space. This past Tuesday, Google unveiled its newest chatbot, Bard, to a less-than-promising start. After giving an incorrect answer in a promotion on Twitter, Bard’s mistake cost Google’s parent company, Alphabet, 

over $100 billion in market capitalization

 as shares plunged on the bad news. Even this small mistake denotes the current limitations of Artificial Intelligence.

Microsoft, which is a primary stakeholder in Open AI, has already found a use case for ChatGPT: Adding it to the search engine, Bing, in an effort to supercharge its capabilities. More than that, Microsoft is looking towards a future that will include ChatGPT’s functionality in Outlook, Office 365, and Microsoft Teams — software tools used by millions every day. These are just a few examples of how the leaders of Big Tech are hoping to leverage AI to drive their products, and their customers, further.

It’s safe to say that Artificial Intelligence won’t be taking over our lives tomorrow, but it will certainly pervade nearly every aspect of our day-to-day lives sooner than we might think. From coding to writing to Zoom meetings, AI chatbots like ChatGPT will be a mainstay, and its creators envision a world in which human efficiency will be dramatically increased by this burgeoning technology.

, our Data Science team is working with large sets of data on a daily basis, building models to analyze and deliver performant advertising campaigns for clients. Analyzing large datasets is an area in which ChatGPT excels, as its computing power allows for incredibly fast results, as well as the automation of certain routine tasks.

Alex Papiu, Chief Scientist at Simulmedia, explains, “The underlying brain behind ChatGPT is a massive language model known as GPT-3.5. This language model is a probabilistic system that has been trained on a vast amount of text data. Its main objective during training is to predict the next word in a sentence, given the context of the previous words.”

In fact, Alex has already found uses for ChatGPT in his work at Simulmedia. “I mostly use ChatGPT to generate code for generic, exploratory data analysis. This usually involves getting data from SQL, aggregating it, and plotting the results. It’s also fairly easy to have ChatGPT define a simple Machine Learning setup in Python - although, for more complicated Machine Learning tasks, it tends to decrease in quality.”

Below is an example of how ChatGPT uses its predictive capabilities to generate code in Python based on prompts entered by Alex.

It’s indisputable that Open AI’s chatbot is a significant leap forward in the world of tech. The prowess of ChatGPT is even more impressive when considering the fact that it contains more than 175 billion parameters. ChatGPT is in a constant state of learning, too, as its language model continues to improve as more data is ingested and evaluated.

Despite its impressive capabilities, Open AI’s chatbot is not without limitations. Alex notes, “It's important to understand that the model is probabilistic and does not possess true comprehension of reality or truth. As a result, it may produce occasional outputs that sound plausible, but are incorrect.” In other words, ChatGPT won’t replace Software Engineers and Data Scientists in the near future; human discernment, namely domain expertise, is still necessary to determine the validity of the chatbot’s responses.

The future looks bright for Artificial Intelligence, and ChatGPT is just the beginning. As leading tech companies invest billions in the race to dominate the realm of AI, the underlying technology, and its ability to learn and progress, will continue its exponential growth. The future of technology is coming. Are you ready?

Simulmedians Utilize ChatGPT for Data Analysis

The UI team at Simulmedia recently made the decision to upgrade its version of Ember, our front-end framework, to utilize Octane, an update that adds myriad new features for cleaner, more effective code. Let’s take a look at some of the changes that Octane brings to the Ember experience:

One of the first notable changes is the way in which we create components from the command line. Although this doesn’t represent a huge change, it is important for a few reasons.

Previously, we would generate a component in Ember with the following command:


This command is pretty straightforward and helps us to generate three files: a JavaScript file, a Handlebars template, and an integration test for the component that we just generated.


In Octane, we have a slight modification that allows us to customize the files that we generate with this command. For example, we can append an option to generate a 

, which is what we call components in Octane. Additionally, we can add an option to generate a classic component, which refers to the components in previous versions of Ember, like the one we generated above. Let’s take a look at how these commands look.

There are a few key differences between the two commands. First, adding the 

flag indicates that we are using a Glimmer component, which entails using new syntax and features in Octane. Conversely, if we want to use syntax and features from previous Ember versions, we can add the ‘

Another new feature of Octane is the new syntax for rendering our components in templates. Rather than using the traditional, double curly bracket syntax, Octane utilizes the new bracket notation, which is analogous to the syntax found in the React framework.

Nested components in Octane are specified as such:

Although not a huge change, I truly believe that it makes the code more readable, and, ultimately, that is one of the primary goals and advantages of upgrading to Ember Octane.


Passing Arguments from a Parent Component

One of my favorite changes implemented in Ember Octane is the notation that we use to specify an argument that has been passed from a parent component, as well as the way in which a component utilizes its own properties.


Octane syntax for passing down an argument:


Likewise, the way in which we utilize a component’s properties in the template is slightly different. Consider the following:

In my opinion, this is a welcome change, as it helps us to differentiate between a component’s own properties, and those that have been passed down as arguments from a parent component. Just another example of how Octane makes code more readable!


A change that definitely takes getting used to is the use of decorators in components, controllers, and routes in Octane. They’re everywhere! From the new tracked property to actions to classic components, Octane makes good use of decorators, 

As we discussed above, with the implementation of Octane, we can utilize two types of components: Classic components and Glimmer components. In Octane, this can be accomplished by adding the 

This indicates to developers that Classic Ember syntax will be utilized. Glimmer components do 

have a decorator; additionally, there are several other ways in which Glimmer and Classic components are different, but this is the most-obvious.


However, the use of decorators does not end there! Decorators in Octane are used with actions, 

, importing services, and more. Let’s take a look at how a typical Glimmer component might look:

Although not a massive change, I am a proponent of decorators in Octane, as I feel that they again add a bit more context and readability to our code. Also note that computed properties have a new look, and, as a general rule, Octane advocates for the use of simple getters over computed properties, although there are some exceptions.


To sum up, I am a fan of a lot of Octane’s new features, although it did take a bit of time to fully understand and appreciate them. The UI Team at Simulmedia still faces a big task ahead in updating our code base to reflect the latest conventions in Ember. Overall, sentiment is good; as developers, we have accepted the changes with open arms, and the general consensus is that Octane really does make Ember code more readable and easier to understand. So, are you ready to turn up the heat with Octane?

The UI team at Simulmedia recently made the decision to upgrade its version of Ember, our front-end framework, to utilize Octane, an update that adds myriad new features for cleaner, more effective code.

Simulmedia’s UI Team Upgrades to Ember Octane

Here at Simulmedia, we recently decided to take on a new project using a different front-end framework. So when I was asked to set up the tests for this new project, I started thinking about the current testing infrastructure and whether it would make sense in this new environment. Our current automated testing repo is built on Selenium WebDriver written in Python that utilizes Pytest. Since the front-end team was branching out and trying new technologies, I decided to take some time to see what else is out there, and if it would make sense for me and the rest of the developers.

I had never had the need or interest to abandon Selenium WebDriver. Why would I want to move away from something that is tried and true? Why would I want to switch programming languages after writing in Python for the past few years? One of the more likable parts of writing automated tests using Selenium WebDriver is that you have the freedom to build a repo from scratch and truly customize it to your needs. You get to choose your language, your test runner and how and where your tests are run. So why would I want to start writing tests using an out of the box, all-in-one, single-language testing tool called Cypress? Well, the answer to these questions can be boiled down to: curiosity and the desire to find alternatives that will better fit our overall development strategy.

On my journey there were the obvious challenges of brushing off the old JS knowledge that I had and learning the nuances of the Cypress syntax, along with the do’s and don'ts and best practices. But as a whole, Cypress and Selenium share the same dialect, which means the core concepts of testing are still the same. But here are some of the speed bumps that I hit during this transition:

First off and maybe one of the most frustrating parts of automated testing is waiting. WebDriver gives the user a few different ways that you can accomplish waiting: Fluent, Implicit and Explicit. Each having to be declared somewhere either in your set up or directly injected into a step of your test. With Cypress, you do not have to worry about declaring any kind of waits. Most of Cypress’ custom commands have built-in waits that will logically wait for that element to reach a usable state. For example, I have this custom click command using WebDriver:

It now becomes a single line call in Cypress:

This was a refreshing take on the sometimes-tedious task of finding creative ways to wait in WebDriver.The next challenge was figuring out a way to write tests when you are not allowed to assign an element to a variable. Cypress works asynchronously, meaning that each Cypress command returns immediately and then appends to a queue of commands that execute at a later time. So if you want to use an element and its value, you’ll need to use Cypress’ closures (native promises), which are pretty foreign coming from Python and WebDriver. This makes you rethink the way you would write a test. Instead of having access to everything in your test no matter where you're calling it from, you now need to think of the test as dominoes falling in line.

Lastly and maybe mostly importantly is working at a different pace. Running tests in parallel no matter where you are running them can add a bit of latency to your test run. WebDriver is built to send your test commands through HTTP to your test browser. Whereas Cypress executes in the browser and in the same run loop as the device under test. This means there are no network connection issues allowing the tests to run at a faster speed then they’re otherwise used to.

After writing a few tests and running them in headless mode, I kept getting failures. At first I could not figure out what was going on. I stepped through the test in debug mode and the test would pass every time. I then turned to watching the video of the test failure, which is where I noticed in the Cypress console that the PATCH request to the API was not executing fast enough. Saving a change, reloading the page and asserting the change was executed before the API returned a 200 on the PATCH. That a page refresh would finish before an API request was pretty new to me.

But since Cypress is working within the browser, it has access to all the network traffic -- as you can see in this example of their test logs:

This is a very versatile advantage to have when running any automated test. So to combat the milliseconds of time I needed to wait for the API, I was able to easily intercept the API request. Once the API request is intercepted, Cypress allows you to assign that request to an alias. Then you can set a wait on that alias (which you can see next to the API calls in the logs):

The next step will not execute before the API request to return a status code.

The ability to intercept requests and wait on them to execute was a learning experience for sure. But those two lines of code saved me from adding one of the dreaded static waits that slow down your test and ultimately shows that you have waved the white flag on your problem. But now my tests will execute on time and with less flakiness since I am able to target exactly what I need for my test to pass.

So has Cypress converted me from Selenium WebDriver? That is still to be determined... stay tuned.

Should our developers abandon Selenium WebDriver in favor of the single-language testing tool Cypress? The only way to find out was to write tests and wait.

Challenges of Switching From Selenium to Cypress

Shortly after completing a web development bootcamp in New York, I was thrilled to learn that I’d received an offer to join Simulmedia full-time as a software engineer. I felt a sense of achievement knowing that the long and grueling months spent studying various programming languages and their respective frameworks had paid off. I felt a sense of relief knowing that I didn’t have to spend seemingly endless weeks studying for assessments anymore. Perhaps most of all, I was excited to take the next step and actually use the skills I’d learned – this time as a working professional rather than a student.

Yet, as I’d soon come to realize, those two aren’t mutually exclusive. The nature of software engineering indeed proves otherwise; in such a massive and quickly evolving industry, you never really stop studying. You never really stop being a student. And it’s one of the reasons why my passion for the field only grows. 

Finishing school and beginning work as an engineer didn’t mean the days of research and studying had come to an end. It meant that they had just begun.

My first week at Simulmedia was a busy one. Day one began with meetings with various departments within the company to get an understanding of how Simulmedia manages targeted TV advertising, which was quickly followed by a demonstration of what my day-to-day would look like. By the second day, I had already pushed code to production – adding pagination to a list component in one of our Ember UIs.

Surely, coding in an entirely new tech stack in an industry unfamiliar to me would be enough to keep me busy. But there was more: “I want to organize another hackathon,” my manager said to me on my third day. We were headed back to the office from a one-on-one meeting over lunch. “We’ve had several hackathons in the past; they’re a chance to show off your skills as an engineer - some hackathon projects even become part of our tech stack.” The fast-paced environment typical to startups like Simulmedia was made apparent, and I realized how much I had to learn.

As the first engineering hire at Simulmedia without a traditional computer science degree, I wondered how my abilities to pick up the job quickly and contribute might be perceived. The freedom and, as a result, ambiguity associated with the hackathon made me weary about if I could successfully participate; I wondered if everything I’d done to prepare myself was enough - not just enough to participate in Simulmedia’s fast paced startup culture, but enough to actively contribute to it.

We’d been warned about imposter syndrome - it’s the fear of incompetence, rooted in the lack of significant real-world experience that comes with acquiring a new skill. It’s a bit more prevalent among us bootcampers, particularly because most of us don’t have a typical four-year computer science degree, which is still considered by many to be the absolute minimum in terms of required experience. To add fuel to the fire, the coding bootcamp itself is a relatively new concept - the first bootcamps were founded in 2011, and graduates of these programs still comprise just a small fraction of the larger pool of engineers. Some more traditional companies continue to prefer engineers with conventional backgrounds, and a few don’t consider bootcamp graduates at all.

However, their popularity continues to increase. The relatively short-term curriculum and tuition model has both been a magnet to students and has inspired other educators to follow suit. Since the first few bootcamps, dozens more have opened their doors to students. More recently, traditional educational institutions such as Ivy League universities have opened their very own bootcamps in an effort to capture a share of the market. In fact, the demand for software engineers is 

projected to increase 21% by 2028, whereas other industries are projected to increase at only 5%.

 is not the only reason bootcamps have been largely successful in recent years. Many of the students who graduate from these programs are skilled and prepared enough to meet job requirements and hit the ground running - and Simulmedia isn’t the only company seeking us out. Apple, Google, Facebook and others have removed bachelor’s degrees in computer science from their application requirements altogether. In fact, here at Simulmedia, almost all of our new engineering hires are now from coding bootcamps.

This is partly due to the intensity of their programs, and the valuable skills that engineers pick up as they progress through the programs. Despite the lack of forced physical activity, using the term ‘bootcamp’ to describe them is no dramatization. App Academy, like others, is extremely rigorous (students at App Academy who fail more than one of the weekly assessments are asked to leave the program.) And it’s incredibly fast-paced.

The 12-week curriculum begins with programming language fundamentals in Ruby but escalates very quickly. By the end of day one, students have begun to practice writing algorithms and by the end of the week are taught more advanced concepts like Big-O notation, recursion, object-oriented programming (OOP), and relational database architecture.

The first half of the curriculum culminates in building a reddit-like blog app using Ruby on Rails, and the second half immediately begins with Javascript fundamentals. Ultimately, students are expected to build a fully functional web app - such as Facebook, Instagram, or Robinhood (my personal choice) - using everything learned in the curriculum: Ruby on Rails as the backend and a React frontend, with Redux as the state manager. Essentially, we were provided the tools to think for ourselves, react on our feet, then prove what we know through frameworks and projects.

Within a matter of months, participants go from knowing little to nothing about web development, to building a full-stack web application from scratch. Ironically, I actually found this fact to be a source of self doubt, fueling sentiments related to the imposter syndrome mentioned earlier. How could I possibly perform successfully as an engineer, I thought at times, when just months prior I knew close to nothing about software development? Well, the first step was to stop thinking that way. Reframing my point of view on the situation, it turns out, was half the battle. Rather than focusing on what I didn’t know before, and discrediting myself because of it, I focused on how much I had learned in those short months and let this anchor my thoughts whenever feelings of uncertainty resurfaced. If anything, being able to learn such a vast amount of material in a relatively short amount of time serves as a testament to my abilities - and how much I could contribute to Simulmedia in a similar amount of time.

A Word of Advice for Fellow Bootcamp Graduates

As bootcamp graduates, perhaps our strongest advantage is our diversity – in all senses of the word. The tech industry in its current form significantly lacks racial and gender diversity, but this is beginning to change. As we continue to enter the workforce en masse, and as experienced bootcampers continue to move up in their positions, we are beginning to quite literally 

. As an example, nearly 35% of bootcampers are women, compared to only 19% in traditional undergraduate computer science degrees. We’re also diverse from a professional perspective. The engineer that sits to my left at our NYC office, a fellow bootcamp graduate, was once a drummer. The engineer across from me comes from a background in law, while I myself come from a background in finance. This is all to say that you can - and should - bring your diverse set of skills to the workplace so that they may benefit you. Don’t let your background make you feel illegitimate or incompetent. Know that with diversity comes a list of advantages and talents that you can tap into to propel yourself forward. You’ll find that not only is your diversity beneficial to you personally, but your team as well.

“Normal people believe that if it ain't broke, don't fix it. Engineers believe that if it ain't broke, it doesn't have enough features yet.” -Scott Adams

As engineers, it is our nature to be curious, inquisitive, and eager to build. Sometimes, a little too eager; it’s natural that there will be bugs and errors along the way. Don’t be discouraged by this; there is no shame in overcoming hurdles. Instead, see it as part of becoming a better engineer. The best of the best will err, and you are no exception. If you expect to grow as an engineer without making mistakes along the way, you will quickly be humbled. Understand that pride is not the remedy for shame, it is its source. It is with humbleness and humility that one overcomes the shame (wrongly) associated with ignorance. Look at uncertainty or lack of knowledge on a subject as an opportunity to learn.

3. Accept your destiny as a perpetual student

Remember that as an engineer, you will continue to learn as your career progresses. You must, in order to keep up with the rapidly evolving industry. But you should also want to, so that you can better yourself. A versatile engineer is more valuable than one whose knowledge is limited to a few areas. One of the best ways to expand your skill set is by learning a tech stack that you’re not yet familiar with. Certainly do not be discouraged by foreign tech stacks; though I was trained in Ruby on Rails and React + Redux, I currently work primarily with Ember.js and Roda.

While bootcamps teach you the skills you need to begin work as an engineer, they don't teach you everything. Notably, container platforms such as Docker are often left out of the curriculum, usually because there’s no better teacher for this than the actual engineers using it on the job; container platforms and deployment systems are notoriously difficult to teach to beginners due to the specificity in implementation.

If there’s one thing you should take away from this, it’s that you should never discredit yourself for having a non-traditional background. Regardless of what industry you may be in, we’ve seen that talent can come from unexpected places. Stay focused, and with time you will improve.

One year after joining Simulmedia, a hackathon finally took place. Only this time, I wasn’t thinking about if I'd be able to perform. Rather, I was thinking about how I might challenge myself to excel. I chose to work with a team on a project that was heavily data science based, which is something I likely wouldn’t have considered a year prior. In this brief moment, I realized how much I’d grown. 

In retrospect, any doubts about my abilities because of my background were silly. I was hired because of my background, not in spite of it.

Brikend Behrami is a Software Engineer in Simulmedia’s New York City office. You can get in touch with him at 

Our very own Brikend Behrami discusses his journey to becoming a software engineer and shares his advice for success in tech.

An Engineer’s Advice for Success and Fulfillment in Tech

Set operations like unions and intersections on a collection of data points are useful for a variety of scenarios. We here at Simulmedia also rely on them heavily. However using them on a very large collection presents a significant challenge.

Sets are implemented in most programming languages but are not natively available widely in popular data stores such as relational databases. Redis for large datasets has been a popular option. However because it is an in-memory datastore, it cannot manage a large number of sets at a reasonable cost. Other solutions such as Pilosa have recently been implemented to solve that need but they require a long learning curve.

On the other hand, Postgres which has been a popular relational database that can accomodate billions of rows does not provide support for sets by default.

In this blog post we explore how to use set operation efficiently in a Postgres. There are multiple ways to do it:

Using roaring bitmap (extension that needs to be built)

In order to illustrate these two different approaches, we will use the 

 that was open-sourced in 2017. It contains over 3 millions of anonymized orders placed by over 200,000 users on around 50,000 products.

The data preparation is described in this page: 

https://gist.github.com/darthbear/332318706ba7ac56e2a19bdf5f4daf1e#file-simulmedia-roaringbitmap-post-md

Intarray allows a compact storage of integer arrays in Postgres and to compute operations on them. It is supported in most Postgres distributions (AWS, Google Cloud and Microsoft Azure).

In Postgres, let's use the extension intarray:

And create the purchases table using intarray:

Now, let's create sets of users for each products:

We can see that the most popular products are bananas, strawberries, spinach and lemons. In our example, let's try to see how many people are buying bananas (organic and non-organic) and strawberries (organic and non-organic):

Which translates as a set intersection query as follows:

This query ran almost instantly. If we had to run it in a normal query with Postgres, we would have needed to use INTERSECT which would have taken much more time.

Using a table that contains all the products per user:

We can select all the user_ids that match the requirement as follows:

This takes 0.330 seconds which is about 6 times as slow. And the difference is getting bigger as the arrays grow larger.

Storing integers as intarray is very efficient when the sets are small. However for larger sets, it takes more space than storing the set as a bitmap array. For example, if you consider the set (0, 2, 4, 5, 6, 7) to be stored as an integer array, It will take much more space than storing it as the bit array “10101111”.

Another way to compress even further is to use Run Length Encoding (RLE) which stores groups of 1s. For instance, 00011110000011000 will be stored as (3, 4), (12, 2) that denotes a group of 4 1s at position 3 and a group of 2 1s at position 12.

Roaring Bitmap takes the best of the 3 worlds by splitting a set in blocks of 65,536 numbers. In this case, in each block, numbers will be from 0 to 65,635 and can be encoded using 16 bits.

If there are less than 4,096 numbers in the set, then encode the set as an integer array. It will take at most 4,096 x 16 bits = 65,536 bits

If not, consider a bitmap of 65,536 bits to store the members of the set.

If the set is dense and/or have numbers packed together, RLE might be a better choice.

In order to install the roaringbitmap extension, first build the extension and install it as follows:

Let's create a new table purchases_roaring:

Creating the sets of users in the purchase table is similar to how we built the intarray previously:

Again we can select all the users that bought bananas and strawberries. The query is the same as the one with intarrays.

Compared to the intarray solution which took 0.052 seconds, roaring bitmap performed twice as fast which is about 15 times faster than using a normal table structure and doing set intersection in normal SQL (with INTERSECT). In the experiments section we will show that they perform even better than Redis for large datasets.

We will compare the performance of these two approaches with Redis which is a common datastore to compute set operations.

Medium size sets: 100,000 users with a total of 10,000 products:

 sparse with 10% of popular products purchased by between 25% and 50% users

 dense with 50% of popular products purchased by between 25% and 50% users

Large size sets: 1,000,000 users with a total of 1,000 products:

For non popular products, they will be purchased by under 10% of the users.

For our tests, we use a MacBook Pro with 2.9 GHz 6-Core Intel Core i9 and 32GB of RAM.

We use Redis 6.0.1 (To install and start Redis on MacOS you can use the following commands: 

In order to load the data into Redis we can use a simple script that will send a lot of SADD commands. For example: SADD 1 100 101 102 103

It will create a set with the key 1 containing the elements 100, 101, 102 and 103.

We use Postgres 12.1 with the extensions intarray 1.2 and roaringbitmap 0.5.

When we generate the data, instead of generating a row for each (product_id, user_id) pair, we generate the data in a format that can be ingested directly into intarrays or roaring bitmap. Each row will look as follows:

Where the first column is the product_id and {1, 10, 70} is the array of user_ids.

The process is similar and the data imported use the same format as for intarray.

We create the table purchases_intarray as follows:

We can see that Postgres stored the data in intarray in a compact way being less than 7% the size needed in Redis. Using roaringbitmap, the footprint decreased even more to 1-2% of what Redis uses.

We show for each the RAM required to run 1 light query / heavy query. Running 10 at the same time will take 10 times as much RAM.

Using the Redis open source version, all the data is loaded in memory so the RAM required is basically the same as the footprint.

In order to determine the RAM required by Postgres we run postgres in docker using different memory sizes. We can see that storing the data in Postgres takes significantly less space than Redis. Also Roaring Bitmap really shines when we consider large sets.

Note that Redis loads the data in memory which quickly becomes a limitation when it comes to big dataset. Postgres, on the other hand, uses considerably less disk space and RAM. As queries are becoming more and more complex Postgres intarray consumes more and more RAM but using Roaring Bitmap does not increase RAM consumption much.

For our experiments we consider 2 kinds of workload:

Light queries that consist of 10 intersections of 10 unions each

Heavy queries that consist of 10 intersections of 100 unions each

For each of the experiments, we measure the time taken to run each query using different concurrency levels (between 1 and 25).

Medium size sets (100,000 users and 10,000 items)

 We represent here the response time that was obtained for 10,000 medium size sets that are sparse for the 2 different workloads.

Using set operations on very large data sets presents a significant challenge. Click to learn more about how to use set operation efficiently in a Postgres.

Running Set Operations on Large Data Sets Efficiently

As we grew our Docker infrastructure at Simulmedia, we started finding Docker’s default logging to be inadequate. We wanted to find a way to harness the features and flexibility of traditional logging systems, and so we began researching alternate paths that could help meet our needs.

The primary requirements of a logging system come from the developers. The most common use cases involve looking at logs to determine what happened or what is happening on the system. We want our developers to have easy access to current and historical logs. They should be able to follow logs, including logs from multiple containers at the same time. Ideally, logs should be available in standard formats, so that they are easy to search, filter, and follow. Many developers prefer using standard Unix tools and scripting to access log files, and so a solution that allows that is preferred.

Docker’s default JSON-file style of logging works well for a single container, but there are no great options for watching multiple containers. If you have 4 instances of a service running under a load balancer, following all their logs requires 4 commands, which may each need to be run on different hosts.

Ideally, logs should be archived for at least a month, if not longer. This makes it possible to analyze the historical usage of a system and provides a way to get answers to questions about past usage and behavior.

Another issue with Docker’s logging is that viewing a container’s logs requires access to the Docker socket, which is effectively root access. In our ideal world, developers should only access production servers via deployment systems. Docker’s default logging makes this type of security standard difficult.

In the Docker landscape, each problem seems to have a dizzying number of possible solutions. Logging is no exception. A variety of commercial services exist to help these needs be met, as well as open source options, such as FluentD or the ELK stack (which combines Elasticsearch, Logstash, and Kibana.)

In traditional system administration, powerful and extremely flexible logging systems have existed for decades. Newer tools like the ELK stack provide different feature sets which add some very nice capabilities for log querying and analysis.

We decided at the core that best way of doing logging with Docker would be to use the same proven logging systems that are used elsewhere in our infrastrastructure. For us, this was rsyslog, a popular implementation of the syslog protocol with support for many extensions, basic scripting, encryption, and a number of reliability features.

While we would use rsyslog to collect and archive the logs, this didn’t mean we were leaving behind the newer solutions and their features. From rsyslog it is possible to forward the collected logs into other systems like Fluentd or an ELK stack. Sending logs from rsyslog to an extra endpoint only requires small configuration file changes, either on the central syslog server, or on the clients. This lets us have a proven and familiar system at the core, while still benefiting from newer log analysis tools.

Docker has various logging drivers that provide mechanisms for forwarding logs. The recommended way of generating logs is by writing lines of text to STDOUT or STDERR. Each line is a new logging message. When using the syslog driver, these messages are converted to the syslog protocol and forwarded to a server.

For the first step, we opted to have each host that would run Docker containers also running rsyslog locally. Docker containers send their logs to the local rsyslog daemon, which is configured to write them to the host’s local file system. Each container gets its own logs by name. We also provide a combined log file that has all containers on a host, so that they can be viewed in-line with one another.

After writing out the logs to local files, the rsyslog server then forwards them to the Docker centralized logging server. We forward all logs for the host, rather than just the Docker logs. The central server then has archived copies of the host’s system logs, which are very useful when researching and diagnosing issues. On the central server, logs are stored by host, date, and container name.

When starting up, rsyslog reads /etc/rsyslog.conf, which on many systems, such as Debian, includes the directive $IncludeConfig /etc/rsyslog.d/*.conf. The imported files are processed in order, based on the filename, and so it is a common practice to include a number at the beginning of the filename, as an easy way of defining the order.

To start, we will create a config file that will handle local Docker centralized logging.

When rsyslog creates new log files it uses mode 0600 as the default permissions, meaning that files are readable and writable by root, and nobody else. With this example configuration, we make the log files world readable. In some environments this may be acceptable, but if further controls are needed, group permissions can be set, and logic could be written to set different permission for different services.

To set the permissions for our logs we start with $FileCreateMode 0644, which makes the file readable and writeable by root as well as world and group readable. At the end of the configuration, we set $fileCreateMode 0600, which restores the default for configurations read in after this file.

Templates are used by rsyslog to define formats for filenames, log entries, or writing to other systems. In this case, the DockerLogFileName template is used to define the name of the local Docker log files. Logs are created on the host at /var/log/docker/NAME/docker.log. The NAME is taken from the syslog tag, which internally is docker/TAG. A regex is used to extract only the TAG portion.

When writing out the logs, we chose to write to a file named docker.log within a directory named after the syslog tag. Currently, the docker.log will be the only file in the directory. When writing logs from Docker you can use STDOUT and STDERR. Right now, with the syslog driver, both streams become indistinguishable from one another. Our hope in making this a directory is that at some point in the future it will be possible to tell these apart. When that happens, we can add a docker-error.log into the directory, so that errors can be viewed separately from regular log entries.

After defining the template it is time to write the actual logs. We start by writing logs from any container on the host to /var/log/docker/combined.log, where they can all be viewed in-line. After that we write the container logs. If a name is provided via a syslog tag, we use the template to create the proper file. In cases where the tag isn’t defined, we use no_tag in place of a name, so that the logs aren’t lost.

There may be cases where certain types of events are logged regularly, but serve no value. In these situations, you can filter out log entries to prevent them from creating noise in the logs. Filters can be performed using parameters or regex comparisons. For example, code like this can be placed before lines that write or send logs, to filter out matching entries.

Next, we will configure the writing of logs to the central server. With this configuration, all system and Docker logs will be sent to the server.

To start, we define a template RemoteForwardFormat. Rsyslog does have a built-in format for syslog forwarding, but it truncates the syslog tag at 32 characters. Our RemoteForwardFormat is identical, except for the removal of that limitation.

An action is used to send the logs to the server. For reliability, a queue is configured. This is helpful if the central server goes down, since messages can remain queued locally until the server comes back. Rsyslog has a lot of options for sending messages. In this example, we are using TCP for reliability and for its improved capability to send large messages. The target syslog.example.com should be updated with the host that will be acting as a centralized syslog server.

There are many more options worth exploring here to further increase reliability. While it is beyond the scope of this article, rsyslog also supports fully encrypted messaging and trusted host verification via the GTLS stream driver, and it could be very worthwhile to enable support for that.

With the above configurations, logs will be written locally on the hosts running Docker and sent to the central syslog server. The central syslog server now needs to be configured to receive these logs.

This config file is a bit longer, so we will introduce it one section at a time. The file is named 

To start, we load the imtcp module, and define a new input that listens on port 514. This input specifies that it will use a ruleset named remote. This means that when logs come through the input, rather than treating them the same as the local logs, an alternate set of rules will be used.

/var/log/remote/HOST/YYYY/MM/DD/docker/NAME/docker.log

. These are used for the logs from individual containers. The 

 creates a file that includes logs from all Docker containers on a host, in-lined with each other. This goes into 

/var/log/remote/HOST/YYYY/MM/DD/docker/combined.log

At the current time, it is useful for us to organize containers by host. With systems where the host that a container runs on can vary greatly, these rules can be modified to store logs by container name at the base level instead of (or in addition to) doing it by host.

Now we begin to define a ruleset for the incoming docker logs. Everything inside the braces of the ruleset will be used when evaluating logs that come in through the TCP socket.

Similar to the client side rules, we change the $FileCreateMode to make the files readable by world. Since we are creating directories as well, we also add in a $DirCreateMode 0755, which allows anyone to access the newly created directories.

There are some clear benefits to using Docker centralized logging through rsyslog. Learn more from Simulmedia about the setup and why it is an improvement.

Join the Waitlist for Skybeam: Be First to Access Our Self-Service CTV Tool!

Why We Use Docker for App Deployment