Approach and challenges in implementing householding ad tech solutions in the real world

Learn more about Lineate’s custom solutions for AdTech

In the previous articles of this series (here and here), we discussed householding, how it serves as an effective interim solution during the ongoing transition to a cookieless environment, and the challenges of its implementation. In this article, we will delve into the technical intricacies of implementing a householding solution in a specific real-world scenario and the challenges we encountered in getting an implementation that performs well.

Firstly, let’s define household traffic. Household traffic refers to cookieless requests that are identified as belonging to real households and are technically enriched with additional data. In this case, we defined a BuyerUID field in our user profile, which is filled with a list of user cookie mappings. 

Since we planned to transform cookieless traffic into cookied traffic, we had to use previously collected cookie mappings. These mappings were for traffic coming from the same IP address, on devices that had cookies enabled.

The use of these existing mappings was relatively straightforward in principle. When processing an ad request with a householding IP address, we simply loaded the corresponding cookie mapping associated with that IP address. This action transformed the traffic from cookieless to cookied. Of course, the reality was somewhat more complex. We needed to ensure that the IP addresses were valid proxies for households, for example. In general, we have discovered that the initial steps involving data preparation are crucial and nontrivial, warranting a closer examination.

To prepare householding data for integration into the OpenRTB pipeline, we followed these steps:

Step 1 - Identification of household traffic

The first step involved determining whether the traffic should be classified as householding traffic. We used a third-party service, Tapad, which can identify whether the IP address from the current ad request belongs to a household by returning a Boolean flag. To ensure system efficiency, we implemented a caching layer based on Hadoop Distributed File System (HDFS), which stored the information about whether previously seen IP addresses were household ones. To keep this information up to date, we implemented date refreshing through scheduled jobs.

Step 2 - Selection of appropriate cookie mapping for household IP addresses

Given that our system already maintains cookie mappings for all demand partners, we have services with simultaneous access to both IP addresses and cookies. For householding IP addresses, we associated one cookie with the corresponding IP address. Since multiple cookies may be associated with a single IP address, we applied special logic to select the most suitable one based on heuristic criteria: We avoided choosing very old cookies (due to potential expiration) or the newest ones (since user profiles may not be fully informative, or mappings may not be synced with demand partners yet).

Step 3 - Optimization of access to household mapping for performance

Next, we mapped IP addresses to cookies. Ad requests typically contain payloads with lists of IP address–and-cookie pairs. In OpenRTB, it’s crucial to respond quickly, within a very short time frame of usually up to 350 ms. To meet this requirement, we cached all data that was used during this pipeline. However, user mappings generate terabytes of data, making it impractical to store everything in memory. Therefore, all previously collected cookie mappings in our application were stored under one namespace in the Aerospike database, chosen for its rapid data access capabilities. When the householding API was invoked for cookie traffic, the application used the request to find appropriate cookie mappings in the cookie namespace, then copied them to a separate householding namespace, with IP addresses from the request serving as keys. As a result, for all future cookieless traffic, we will be able to efficiently access cookie mappings by IP address and meet the performance requirements.

Step 4 - Actualization of old household mapping

During step 1, we identified whether a specific IP address belonged to a household. However, this information is overall less static than one would hope and may pose future issues. For example, previously identified and cached household IP addresses may be reassigned frequently by the ISP to an office or other public place. Fortunately, the Tapad service may return IP addresses that were previously marked as householding but no longer belong to a household. We used this capability to address potentially outdated information. For example, in the process of identification, if a specific IP address was associated with a household, we could possibly encounter data indicating that said IP address was formerly affiliated with a household but was presently unlinked. In such instances, we promptly expunged the cookie mappings tied to this IP address from the Aerospike database.

Results

Through the implementation described above, we achieved approximately double the revenue and triple the bid rate of our demand partners from the same traffic relative to pure cookieless traffic. As we calculated, in this process, about 24 percent of cookieless traffic was transformed to householding traffic, and about 2 percent of previously collected cookie mappings were used by the householding feature. Those results significantly surpassed our customer’s expectations. We proceeded to work on the improvement of this feature, and our client sees an opportunity to improve it even further and use all remaining time before the world finally switches to the fully cookieless approach.

Learn more about Lineate’s custom solutions for AdTech