Site icon Efficient Coder

STORM & Co-STORM: AI-Powered Knowledge Curation for Wikipedia-Style Content Generation

高效码农

STORM & Co-STORM: Your AI-Powered Knowledge Curation Assistants

In today’s information age, efficient knowledge creation and organization are more critical than ever. STORM (Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking) and its advanced version Co-STORM, developed by Stanford University, serve as intelligent assistants that can craft Wikipedia-like articles from scratch. This article will provide an in-depth yet easy-to-understand introduction to these tools and guide you through their installation and usage.

What Are STORM and Co-STORM?

STORM is an AI system based on large language models (LLMs) that can conduct internet research, generate outlines, and produce full-length articles with citations. Think of it as a digital researcher, editor, and writer combined into one tool, capable of handling the preliminary work of article writing.

Co-STORM takes this concept further by enabling human-AI collaboration. It’s like having a conversation with a knowledgeable expert who can answer your questions, ask follow-up questions, and guide you through complex topics. This interactive knowledge curation makes it easier to grasp intricate subjects.

How Does STORM Work?

STORM simplifies the process of writing long articles by breaking it down into two main steps:

  1. Pre-writing Stage: The system conducts internet research to gather references and generates an outline. This is akin to the initial stage of writing a paper when you gather literature and determine the structure of your article.
  2. Writing Stage: Using the outline and references, the system generates the full article with citations.

To enhance the depth and breadth of its questions, STORM employs two strategies:

  1. Perspective-Guided Question Asking: By analyzing existing articles on similar topics, STORM identifies different perspectives to guide its questioning process.
  2. Simulated Conversation: STORM simulates conversations between a Wikipedia writer and a topic expert, grounded in internet sources. This allows the language model to update its understanding of the topic and ask more insightful follow-up questions.

How Does Co-STORM Enhance Collaboration?

Co-STORM introduces a collaborative discourse protocol that facilitates smooth interaction among different participants:

  1. Co-STORM LLM Experts: These AI agents generate answers based on external knowledge sources or raise follow-up questions based on the conversation history.
  2. Moderator: This agent generates thought-provoking questions inspired by information discovered by the retriever but not directly used in previous turns. Question generation can also be grounded in external knowledge.
  3. Human User: You can either observe the conversation to gain a deeper understanding of the topic or actively participate by injecting your own comments to steer the discussion.

Co-STORM also maintains a dynamically updated mind map that organizes collected information into a hierarchical concept structure. This mind map helps build a shared conceptual space between the human user and the system, reducing cognitive load during lengthy and in-depth discussions.

How to Install STORM and Co-STORM?

Direct Installation of the Precompiled Package

Simply run the following command in your terminal:

pip install knowledge-storm

This method is ideal for users who want to quickly experience the functionality, as it automatically installs all dependencies.

Installation from Source Code

  1. Clone the project repository:
git clone https://github.com/stanford-oval/storm.git
cd storm
  1. Create and activate a Python environment:
conda create -n storm python=3.11
conda activate storm
  1. Install the dependencies:
pip install -r requirements.txt

This method is suitable for users who wish to customize the code according to their specific needs, such as adjusting the AI models or retrieval modules.

How to Use STORM?

Basic Usage Example

import os
from knowledge_storm import STORMWikiRunnerArguments, STORMWikiRunner, STORMWikiLMConfigs
from knowledge_storm.lm import LitellmModel
from knowledge_storm.rm import YouRM

# Configure language models
lm_configs = STORMWikiLMConfigs()
openai_kwargs = {
    'api_key': os.getenv("OPENAI_API_KEY"),
    'temperature': 1.0,
    'top_p': 0.9,
}
gpt_35 = LitellmModel(model='gpt-3.5-turbo', max_tokens=500, **openai_kwargs)
gpt_4 = LitellmModel(model='gpt-4o', max_tokens=3000, **openai_kwargs)

lm_configs.set_conv_simulator_lm(gpt_35)
lm_configs.set_question_asker_lm(gpt_35)
lm_configs.set_outline_gen_lm(gpt_4)
lm_configs.set_article_gen_lm(gpt_4)
lm_configs.set_article_polish_lm(gpt_4)

# Configure retrieval module
engine_args = STORMWikiRunnerArguments(...)
rm = YouRM(ydc_api_key=os.getenv('YDC_API_KEY'), k=engine_args.search_top_k)

# Create runner
runner = STORMWikiRunner(engine_args, lm_configs, rm)

# Run
topic = input('Enter topic: ')
runner.run(
    topic=topic,
    do_research=True,
    do_generate_outline=True,
    do_generate_article=True,
    do_polish_article=True,
)
runner.post_run()
runner.summary()

Parameter Explanation

  • do_research: Whether to conduct internet research to collect information. If set to False, it will load previous results.
  • do_generate_outline: Whether to generate an article outline.
  • do_generate_article: Whether to generate a full article based on the outline and collected information.
  • do_polish_article: Whether to polish the article, such as adding a summary section or removing duplicate content.

How to Use Co-STORM?

Basic Usage Example

from knowledge_storm.collaborative_storm.engine import CollaborativeStormLMConfigs, RunnerArgument, CoStormRunner
from knowledge_storm.lm import LitellmModel
from knowledge_storm.logging_wrapper import LoggingWrapper
from knowledge_storm.rm import BingSearch

# Configure language models
lm_config = CollaborativeStormLMConfigs()
openai_kwargs = {
    "api_key": os.getenv("OPENAI_API_KEY"),
    "temperature": 1.0,
    "top_p": 0.9,
}

question_answering_lm = LitellmModel(model='gpt-4o', max_tokens=1000, **openai_kwargs)
discourse_manage_lm = LitellmModel(model='gpt-4o', max_tokens=500, **openai_kwargs)
utterance_polishing_lm = LitellmModel(model='gpt-4o', max_tokens=2000, **openai_kwargs)
warmstart_outline_gen_lm = LitellmModel(model='gpt-4o', max_tokens=500, **openai_kwargs)
question_asking_lm = LitellmModel(model='gpt-4o', max_tokens=300, **openai_kwargs)
knowledge_base_lm = LitellmModel(model='gpt-4o', max_tokens=1000, **openai_kwargs)

lm_config.set_question_answering_lm(question_answering_lm)
lm_config.set_discourse_manage_lm(discourse_manage_lm)
lm_config.set_utterance_polishing_lm(utterance_polishing_lm)
lm_config.set_warmstart_outline_gen_lm(warmstart_outline_gen_lm)
lm_config.set_question_asking_lm(question_asking_lm)
lm_config.set_knowledge_base_lm(knowledge_base_lm)

# Configure retrieval module
topic = input('Enter topic: ')
runner_argument = RunnerArgument(topic=topic, ...)
logging_wrapper = LoggingWrapper(lm_config)
bing_rm = BingSearch(bing_search_api_key=os.environ.get("BING_SEARCH_API_KEY"), k=runner_argument.retrieve_top_k)

# Create runner
costorm_runner = CoStormRunner(lm_config=lm_config, runner_argument=runner_argument, logging_wrapper=logging_wrapper, rm=bing_rm)

# Start and interact
costorm_runner.warm_start()

# Observe the conversation
conv_turn = costorm_runner.step()

# Actively guide the conversation
costorm_runner.step(user_utterance="Could you provide more details on the X section?")

# Generate report
costorm_runner.knowledge_base.reorganize()
article = costorm_runner.generate_report()
print(article)

Interaction Modes

  1. Observation Mode: Run costorm_runner.step() to let the AI experts and moderator engage in conversation first. You can observe how they explore the topic from different angles.
  2. Active Guidance: Use costorm_runner.step(user_utterance="Your question") to insert your own thoughts at any time. For example, you can request more detailed explanations of a particular section or steer the discussion in a different direction.

How to Customize STORM and Co-STORM?

Customizing STORM

STORM’s engine consists of four modules:

  1. Knowledge Curation Module: Collects comprehensive information on the given topic. You can adjust its search strategy, such as increasing the diversity of search keywords.
  2. Outline Generation Module: Organizes the collected information into a hierarchical outline. You can modify its logic to align with specific requirements, such as the structure of academic papers.
  3. Article Generation Module: Populates the outline with content based on the collected information. You can adjust the writing style to be more formal or conversational.
  4. Article Polishing Module: Optimizes and enhances the generated article. You can incorporate your own rules, such as emphasizing certain key points.

The interfaces for these modules are defined in knowledge_storm/interface.py, and their implementations are located in the knowledge_storm/storm_wiki/modules/* directory. You can customize these files according to your specific needs.

Customizing Co-STORM

Co-STORM features a unique collaborative discourse protocol:

  1. AI Agents: It includes multiple AI agent types, such as expert agents and moderator agents. Their behaviors are defined in knowledge_storm/collaborative_storm/modules/co_storm_agents.py. You can adjust the strategies of these agents, such as enabling the moderator to ask more challenging questions.
  2. Discourse Turn Management: The conversation flow is controlled by DiscourseManager, which is defined in knowledge_storm/collaborative_storm/engine.py. You can modify this logic to adjust the rhythm and direction of the conversation.

Datasets Provided by STORM and Co-STORM

FreshWiki Dataset

This dataset comprises 100 high-quality Wikipedia articles focusing on the most frequently edited pages from February 2022 to September 2023. It serves as a benchmark for evaluating the quality of articles generated by STORM. You can download it directly from Hugging Face FreshWiki Dataset. To ensure data timeliness, the project also open-sources the code for the data construction process, enabling future replication of experiments.

WildSeek Dataset

Created using data from STORM’s web research preview, this dataset records users’ topics and goals for in-depth searches. It helps researchers understand users’ interests in complex information search tasks. You can access it at Hugging Face WildSeek Dataset.

Quick Start with Example Scripts

The project provides example scripts to help you quickly get started with STORM and Co-STORM.

Running STORM Example

  1. Create a secrets.toml file to configure API keys:
OPENAI_API_KEY="your_openai_api_key"
BING_SEARCH_API_KEY="your_bing_search_api_key"
  1. Run the following command:
python examples/storm_examples/run_storm_wiki_gpt.py \
    --output-dir ./output \
    --retriever bing \
    --do-research \
    --do-generate-outline \
    --do-generate-article \
    --do-polish-article

Running Co-STORM Example

  1. Ensure that secrets.toml includes the Bing Search and encoder API keys.
  2. Run the following command:
python examples/costorm_examples/run_costorm_gpt.py \
    --output-dir ./output \
    --retriever bing

Future Outlook

The development teams of STORM and Co-STORM are currently focusing on:

  1. Enhancing Human Participation: Enabling users to play a more active role in the knowledge curation process, such as directly editing outlines or articles.
  2. Developing Information Abstraction Capabilities: Transforming curated information into various presentation formats beyond traditional articles, such as mind maps and tables.

If you encounter any issues or have suggestions while using these tools, you can provide feedback through the project’s contact information. STORM and Co-STORM are not only valuable for researchers in organizing literature but also for students writing papers, professionals conducting industry research, and even content creators generating knowledge-based materials. We hope this introduction helps you better understand and utilize STORM and Co-STORM to enhance the efficiency and enjoyment of knowledge creation.

Exit mobile version