First Gen AI Implementation

As enterprises strive to embed AI into their workflows and improve efficiency, ServiceNow has emerged as a market leader in delivering AI-driven solutions for the enterprise. With their recent advancements in Gen AI, ServiceNow is setting the standard for how AI can transform the workplace.

I recently wrapped up my first Now Assist implementation on ServiceNow and wanted and wanted to share some insights from the experience. The project was a huge success, and the customer was extremely pleased with the new Gen AI capabilities we rolled out for them.

The project scope included the following features:

• Case Summarization
• Case Resolution
• Chat Summarization
• Knowledge Article Generation
• AI Search
• Now Assist for Virtual Agent
  • LLM for Virtual Agent (from NLU)
  • Microsoft Teams Integration/Virtual Agent Now Assist
  • Genius Results
  • Multi-Turn Catalog Ordering

As with any implementation, there were challenges along the way. Below are the key findings and lessons learned to help others navigating future Now Assist projects.

Genius Results

If there’s one key takeaway here, its that the the quality of your Knowledge Base data matters. Unlike platforms such as ChatGPT, Anthropic, or Perplexity, ServiceNow’s LLM does not connect to the open internet for generating responses. It relies entirely on local data within your Knowledge Base.

One of the struggles during this implementation was insufficient Knowledge data. Without matching relevant content, the LLM produced subpar results for certain user utterances. To address this, we pushed the customer to focus on top incident issues their employees frequently needed help with. By analyzing incident and search data, we updated Knowledge Articles to improve Genius Results.

However, customers with a robust Knowledge Base, especially around self-help instructions and resolutions for common issues, will see significantly more value. In short, a strong Knowledge Base is a critical dependency for success.

Record Summarization

Out of all the Gen AI features we implemented, Record Summarization proved to be the most impactful and valuable. One of its biggest strengths is its simplicity, it’s incredibly fast to set up and provides immediate value. For organizations looking to implement Gen AI with minimal upfront effort, this feature is a clear quick win.

Time Saving

Record Summarization eliminates the need for agents to manually craft handoff summaries or resolution notes, which can take several minutes per incident. On average, writing a 20-word summary takes about 30 seconds to a minute depending on the complexity. If an agent handles 20 cases a day, this feature can save them approximately 10-20 minutes daily. For a team of 50 agents, that translates to a savings of 16+ hours per day or nearly 4,000 hours per year across the team. These savings not only improve productivity but also enable agents to focus on higher-value tasks.

Challenges and Lessons Learned

While the benefits of Record Summarization are undeniable, we did encounter a few limitations during the implementation:

1. Dot-Walking Limitations:
   The customer required related record data for summaries. Unfortunately, current input templates do not support dot-walking to related fields. As a workaround, we used a Business Rule to sync related record data behind the scenes. From what I’ve read, this limitation will soon be addressed in upcoming releases.
2. Formatting Constraints:
   While the out-of-the-box summarization does a great job, customers with complex use cases may need customizable summarization formats. This will be crucial as implementations scale.
3. Relevant Field Selection:
   A critical lesson was identifying additional fields beyond the base input templates. In testing, we found that including more relevant fields (e.g., Work Notes, Resolution Notes) dramatically improved summarization accuracy and completeness. The default fields (Short Description, Description, Priority) were adequate but often lacked sufficient context for handoffs or closure notes.
   

Knowledge Article Generation

This was another standout feature and addressed a long-standing challenge for customers: Knowledge Base maintenance. In most implementations I’ve worked on, customers struggle with:

• Outdated or irrelevant articles
• Lack of content creation and curation

By automatically generating Knowledge Articles based on historical incident data, this challenge is minimized. Additionally, this ties directly to Genius Results: as more articles are created, the Virtual Agent continuously improves over time.

One limitation we found is the inability to select multiple related records when generating a Knowledge Article from Now Assist in Workspace. Currently, this is only possible when manually creating a new article outside Workspace.

Pro Tip: While generated articles are often solid when incidents have sufficient comments, work notes, and resolution data, they still require heavy editing when input data is lacking.

Multi-Turn Catalog Ordering

From a user experience standpoint, Multi-Turn Ordering solves a long-standing issue where users had to navigate out of a Virtual Agent chat to complete a catalog request. With this feature, users can fill out requests directly in the chat, improving continuity.

However, not all catalog items are ideal for Multi-Turn Ordering.

Key Lesson:

• Catalog items with 5 or fewer questions work seamlessly and provide a smooth user experience.
• Catalog items with >5 questions tend to create a clunky experience, making it slower for users compared to completing requests via a web form.

Migration from NLU to LLM in Virtual Agent

Previously, topic selection in Virtual Agent relied on NLU with a combination of utterances and intents. The transition to LLM in Virtual Agent has been a game changer:

• Topic matching improved by 3x.
• Set-up time decreased by approximately 60 hours.

This is a significant improvement for both implementation timelines and user experience. The LLM-driven Virtual Agent not only delivers better results but also simplifies the configuration process.

Conclusion

The Now Assist implementation for ITSM delivered tremendous value to our customer, particularly through features like Record Summarization, Knowledge Article Generation, and Virtual Agent enhancements. While challenges like knowledge dependency arose, we found effective workarounds and learned valuable lessons for future implementations.

As Generative AI capabilities within ServiceNow continue to evolve, I’m excited to see how these features will further enhance customer experiences. For organizations planning a Now Assist rollout, the key success factors are:

1. Ensuring a strong, curated Knowledge Base.
2. Optimizing field selection for Record Summarization.
3. Leveraging Multi-Turn Ordering for the right catalog items (<5 questions).

With thoughtful planning and execution, Now Assist is well-positioned to transform ITSM workflows.