In-product awareness

Mid-2023 to Present

This page is divided into a few sections:

1. An extremely brief summary, made up of a few statements to provide context and impact infographics.

2. A brief summary of my specific contributions to the effort.

3. A full explanation of the project, from inception to where we’re at today.

Extremely brief summary

How it started: We identified a clear lack of user awareness around new feature releases.

How we responded: We developed in-product awareness patterns to experiment with, in order to address that lack of awareness.

Outcomes: Comparing HUX surveys from 2022 (before the patterns) and 2025 (after the patterns), there are clear signs they had a positive impact:

A: Average awareness for individual releases in RE NXT increased from 33% to 65.6%, with awareness of specific features reaching as high as 85%.

B: In 2022, only 57% of respondents were aware of any major feature releases. In 2025, that jumped to 100%.

C: User satisfaction rating for features and capabilities increased from 21% to 33%, the biggest single jump in over six years.

What’s next: We’re working across teams to move these patterns from being one-off experiments into being a platform-level aspect of notifications, lower cost of development and improving consistency.

What I contributed

• I led the strategy and execution for this project, including:

  • Which components to develop and include

  • Guidelines and governance for when to use for a given feature release (and how)

  • How they should behave in terms of cadence and proximity to one another

  • How to measure their success

• After drafting their guidelines and governance, I communicated them out internally with product managers, UX leadership, customer success, engineering, on up through executive leadership.

• Coordinated and contributed to tracking for active, upcoming, and past experiments (the patterns had to be developed one-at-a-time), and communicated out the results of those experiments once we had a critical mass of test cases.

• Worked with our user research team to make sure questions were included in a HUX survey that matched the questions we’d asked back in late 2022 that had brought leadership’s attention to our issues with awareness, to allow us to compare the results to the initial problem.

• I’m currently helping lead the transition from individual experimentation to platform service, now that we’ve proven success, and drafting new guidelines and governance to match their move into a service. I’m also working (with another designer) to expand our strategy from just awareness around new features to include awareness around feature trials, package upsells, and more.

The whole story – 

Background

In Q4 2022, Blackbaud’s user experience research team conducted a HUX (Happiness and User Experience) survey for Raiser's Edge NXT, Blackbaud’s flagship product. One of the topics users were asked about was their awareness of newly released features. The responses gathered shined a clear light on an issue we'd previously observed in discovery calls, customer conversations at bbcon, and elsewhere: Our users were not nearly as aware as we'd like them to be of new features available in their solution. 

For each of the three major feature releases we’d asked about, only 33% of responding users* knew those were things they had access to. To add another dimension to the problem, 43% of users were not aware of any of the three new features.

(*Quick note: Each individual feature had the same percentage of user awareness at 33%, but that doesn't mean each feature had the exact same population of users who knew about it. The populations just happen to be the same sizes. That's how 57% of users can be aware of any of the features, even though only 33% are aware of any given one.)

It goes without saying, but a lack of user awareness around high value new features is an issue for the business, and the organizations who use our software. Our customers cannot capitalize on the value of a feature set that they remain unaware of. A lack of awareness around features released also gives the impression that Blackbaud is innovating less quickly than we are, making us appear less valuable and less competitive. And if the features being released are blockers for an organization to move from more deeply engage and move to our newer offerings, than a lack of awareness will also result in slower migration away from our older solutions.

The opportunity, we hypothesized, is that we'd be able to raise users' perception of product value if we could make sure they became more reliably aware of the features they already had access too, as well as more quickly aware of any new features coming in. 

Development

Myself and other designers began digging into a strategy for potential awareness patterns. In parallel, I was involved with helping design our latest online donation forms, which were identified as an appropriate feature set to test the “loudest” of the awareness patterns we'd been thinking through, the “awareness modal”. 

The work had to be done very carefully, because poorly implemented awareness patterns had (and have) the potential to confuse or frustrate the user they appear to, and the experiments needed to be built one-off to allow for lightweight development and flexibility before we could justify developing them into any type of service. But in January 2024, the first awareness modal lit up, in support of the release of our aforementioned donation form updates. 

In the time since, we've run multiple experimental use cases for the the enhanced announcement, the awareness overlay, and the awareness modal.

EXAMPLE EXPERIMENT

This chart shows a daily view of weekly active users for a major feature we released called query – so it’s a daily count of anyone who’s done anything within query, within the last seven days. I’ve added some labels to show the end-of-year lull, and then the “before” and “after” states of our first out-of-product messaging, followed by a month of no awareness activity (to allow the number of users to stabilize), followed by an in-product awareness pattern. We observed that the email resulted in approximately 2,051 new weekly active users adopting query, and that the in-product awareness modal added another 4,045, meaning in one sense it outperformed the email blast by nearly 100%.

Results like these were very exciting to see, but they also all needed to be taken with a grain of salt. Factors like timing with other awareness methods, interest in the feature itself, time of year, size of user base, etc all created learnings that felt actionable and definitive in one sense, but apples and oranges in another.

Understanding positive impact through survey comparison

That “apples and oranges” feeling changed with the release of our Q3 2025 HUX Pulse survey for Raiser's Edge NXT. The user experience research team was able to help us compare apples to apples, by asking users the same awareness questions they’d been asked back in 2022. Zooming out and comparing statistically significant trends in awareness, as well as overall satisfaction for features and capabilities, was the best way to understand whether or not our awareness work had had a meaningful impact for our customers.

The Q4 2022 HUX survey had shown that 33% was the HIGHEST level of awareness a feature achieved, and 43% of 2022's users were not aware of ANY new features. 

Compare that to feature awareness data from the Q3 2025 HUX survey: 

• 85% of users aware of query (awareness modal)

• 72% of users aware of online giving (awareness modal + overlays) 

• 67% of users aware of prospect insights (overlay + on-page trial UI)

• 56% of users aware of pledge (awareness modal)

• 48% of users aware of Constant Contact (awareness modal)

• 0% of users were not aware of any. 

Query awareness (85%) was a 52% improvement on Q4 '22 (when the highest awareness for a single feature was 33%), our lowest (Constant Contact) was still a 15% improvement, and crucially – a complete lack of awareness of new features has moved from 43% of users to 0% of them, meaning 100% of users knew about at least one of the major feature releases we asked them about.

We also asked respondents to rate their satisfaction in RE NXT is for features and capabilities. This is something we ask in every HUX survey, so we can look backwards and observe trends over time. 

• To start off with, it went up! By quite a bit. A 12% absolute increase from 21% to 33%. (And a 57% relative increase.)

• Looking back from 2019 to now, our other increases have been modest steps of 3%. When the score has dropped, it's dropped by 8% in both cases. A positive change of 12% represents an increase that is four times greater than any other jump up in satisfaction since 2019. 

• User satisfaction for RE NXT is at a four-year high (highest since this time in 2021).

Looking at the 32.6% average increase in individual feature awareness from the ‘22 survey to the ’25 survey, as well as the increase in awareness of ANY feature release from 57% to 100%, and then looking at this unprecedented jump of 12% in satisfaction for features and capabilities: in-product awareness experimentation had a significant positive effect on user awareness, as well as overall user satisfaction for features and capabilities.

Next steps

The small team working on these capabilities are now partnering with our platform teams, to move these patterns from operating as one-off experiments to operating as a core part of our notifications service.

This work will allow for a few important things: it will help ensure consistent governance, it will encourage consistency in design, release, and measurement, it massively lowers engineering cost, and it will allow us to work with these patterns at a greater scale. 

As that work concretizes into part of our platform, relationships we’ve formed with product management, user education, customer success, and beyond have allowed us a genuine opportunity to keep a hand on the wheel and evolve from thinking only about new feature awareness and into a vision of a strategic plan for all in-product awareness, including a new model for how we offer in-product trials and upgrade opportunities.