../assets/audio/notebooklm_question_asking.wav

Overview

This blog post walks you through the implementation of a question-asking game built using Streamlit. The game is designed to encourage curiosity and critical thinking by allowing users to ask questions about a randomly chosen phenomenon. The teacher (an AI model) provides answers that provoke deeper inquiry. The game also evaluates the quality of the questions asked.

The project is divided into the following sections:

1. Game Setup and Sidebar Configuration: Setting up the game parameters and user inputs.
2. Phenomenon Selection and Explanation: Randomly selecting a phenomenon and generating an explanation.
3. Conversation Management: Initializing and managing the AI-driven conversation.
4. Evaluation of Questions: Analyzing the user’s questions based on knowledge graph coverage and Bloom’s Taxonomy.
5. User Interface: Displaying the chat interface and handling user interactions.

Here’s a demo on streamlit.

Let’s dive into each section with code snippets and detailed explanations.

1. Game Setup and Sidebar Configuration

The game begins with a sidebar where users can configure the game settings, such as the number of questions and the subject of the phenomenon. This setup ensures that the user has control over the game parameters, making the experience customizable and engaging.

import streamlit as st
import phenomena

st.title("Question Asking Game", help="Open the sidebar to start a new game.")
st.session_state.setdefault('questions_asked', 0)

subjects = phenomena.phenomena.keys()
with st.sidebar:
    password = ""  # User password input (currently disabled)
    question_limit = st.number_input('Question Limit', min_value=1, max_value=10, value=2, step=1)
    subject = st.selectbox('Choose a subject', subjects, label_visibility='collapsed')

Here, the st.sidebar component is used to create a sidebar where users can input their preferences. The st.number_input and st.selectbox widgets allow users to set the question limit and select a subject, respectively. These inputs are stored in the session state for use throughout the game.

2. Phenomenon Selection and Explanation

Once the game starts, a random phenomenon is selected from the chosen subject. The AI model then generates a high-level explanation of the phenomenon to provide context for the user.

import numpy as np
from langchain.chat_models import ChatOpenAI
from langchain.prompts.chat import ChatPromptTemplate
from langchain.chains import LLMChain

def get_paradox():
    chosen_paradox = np.random.choice(phenomena.phenomena[subject])
    chat_prompt = ChatPromptTemplate.from_messages([game_description, "{text}"])
    chain = LLMChain(llm=ChatOpenAI(), prompt=chat_prompt)
    paradox_explanation = chain.run(
        f"Describe the following phenomena in 2 paragraphs: {chosen_paradox}"
    )
    return chosen_paradox, paradox_explanation, ""

This function uses the numpy library to randomly select a phenomenon and the LangChain library to generate an explanation. The explanation is designed to be accessible to a layperson while still being thought-provoking.

3. Conversation Management

The core of the game is the conversation between the user (student) and the AI (teacher). This section initializes the conversation with a prompt template that sets the context and rules for the interaction.

from langchain.chains.conversation.memory import ConversationSummaryMemory
from langchain.prompts.prompt import PromptTemplate
from langchain import OpenAI, ConversationChain

def init_gpt(game_description):
    chosen_paradox, paradox_explanation, paradox_rdf = get_paradox()
    template = game_description + """
Teacher: Please ask questions about """ + chosen_paradox + """
{history}
Student: {input}
Teacher:"""
    PROMPT = PromptTemplate(input_variables=["history", "input"], template=template)
    llm = OpenAI(temperature=0, model_name="gpt-3.5-turbo")
    conversation = ConversationChain(
        prompt=PROMPT,
        llm=llm,
        verbose=True,
        memory=ConversationSummaryMemory(llm=llm, ai_prefix="Teacher", human_prefix="Student"),
    )
    return conversation, paradox_explanation, paradox_rdf

The ConversationChain object manages the dialogue, using memory to track the conversation history. This ensures that the AI’s responses are coherent and contextually relevant.

4. Evaluation of Questions

At the end of the game, the user’s questions are evaluated based on their depth, coverage, and alignment with Bloom’s Taxonomy. This provides valuable feedback to the user on their questioning skills.

def evaluate():
    questions = "\n".join([f"Q{idx+1}. {q}" for idx, q in enumerate(st.session_state.past)])
    evaluate_rdf_prompt = f"""Estimate how comprehensive my questions have been...
    Questions:
    {questions}"""
    evaluate_blooms_prompt = """Use Blooms Taxonomy to classify my questions...
    Questions:
    {questions}"""
    with eval_tab:
        st.header('Knowledge Graph Coverage')
        kg_coverage = st.session_state.chain.run(evaluate_rdf_prompt)
        st.markdown(kg_coverage)
        st.header('Blooms Taxonomy')
        blooms_taxonomy = st.session_state.chain.run(evaluate_blooms_prompt)
        st.markdown(blooms_taxonomy)

This function uses prompts to analyze the user’s questions. The results are displayed in the evaluation tab, providing insights into the user’s performance.

5. User Interface

The user interface is designed to be intuitive and interactive. It displays the conversation and allows the user to input their questions.

from streamlit_chat import message

with chat_tab:
    if st.session_state.get('generated'):
        for i, response in enumerate(st.session_state['generated']):
            if i - 3 >= 0 and i - 3 < len(st.session_state['past']):
                message(st.session_state['past'][i-3], is_user=True, key=f"{i-3}_user")
            message(response, key=f"{i}")
    if password == pwd:
        st.text_input("Ask a question:", on_change=on_input_change, key="user_input")

The streamlit_chat library is used to render the chat interface. The st.text_input widget captures the user’s questions, which are then processed by the AI.

Conclusion

This project demonstrates how to build an interactive question-asking game using Streamlit and LangChain. The game encourages curiosity and critical thinking while providing insightful evaluations of the user’s questions. By combining modern AI tools with an engaging interface, this project showcases the potential of educational technology.

Find the complete code on GitHub and a demo on Streamlit

../assets/audio/notebooklm_dig_deeper.wav

Overview

This blog post explores the creation of a prototype browser plugin that analyzes the YouTube Shorts you watch. The goal is to encourage users to actively search and “dig deeper” into topics, rather than passively consuming content from the platform’s recommendation system. The plugin achieves this by generating transcripts of videos, feeding them to a language model, and providing a chat interface for interaction.

The plugin has three main components:

1. Transcripts: Extracting audio from YouTube videos and converting it into text.
2. GPT API Integration: Using OpenAI’s GPT to generate curiosity-provoking suggestions based on the video content.
3. Chat Interface: A simple browser-based chatbox for user interaction.

Transcripts

The first step is to extract audio from YouTube videos and convert it into text using OpenAI’s Whisper model. Below is the code for downloading audio, converting it to a suitable format, and generating transcripts.

Downloading Audio

const fs = require('fs');
const ytdl = require('@distube/ytdl-core');

The ytdl-core library is used to download audio from YouTube. The downloadAudioOnly function filters the video to download only the audio stream and saves it to a specified path.

function downloadAudioOnly(url, audio_path, callback) {
  const stream = ytdl(url, 
    {
      filter: 'audioonly',
      requestOptions: {
        headers: {
          'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
        }
      }
    }
  );

  stream.pipe(fs.createWriteStream(audio_path))
    .on('finish', () => {
      callback(null, 'Download and save completed');
    })
    .on('error', (err) => {
      callback('Error writing to file: ' + err.message);
    });

  stream.on('error', (err) => {
    callback('Error downloading audio: ' + err.message);
  });
}

This function ensures that only the audio is downloaded, which reduces the file size and processing time.

Converting Audio to WAV Format

Once the audio is downloaded, it needs to be converted to a WAV format compatible with the Whisper model. This is done using the ffmpeg-static library.

function convertMP3ToWAV(audio_path_mp3, audio_path_wav){
  const ffmpeg = require('ffmpeg-static');
  const spawn = require('child_process').spawn;

  const ffmpegProcess = spawn(ffmpeg, ['-i', audio_path_mp3, '-ar', '16000', audio_path_wav]);

  ffmpegProcess.on('exit', () => {
    console.debug('Audio converted successfully!');
  });

  ffmpegProcess.on('error', (err) => {
    console.error('Error converting audio:', err);
  });
}

This function uses ffmpeg to convert the MP3 file to a WAV file with a sample rate of 16,000 Hz, which is required for Whisper.

Generating Transcripts

The ytTranscript function orchestrates the process of downloading audio, converting it to WAV, and generating a transcript using Whisper.

async function ytTranscript(url) {
  return new Promise((resolve, reject) => {
    video_id = get_video_id(url);
    audio_path_mp3 = path.join(process.cwd(), 'output/', video_id + '.mp3');
    audio_path_wav = path.join(process.cwd(), 'output/', video_id + '.wav');

    downloadAudioOnly(url, audio_path_mp3, (error, message) => {
      if (error) {
        console.error('Error downloading or saving the video:', error);
        reject(error);
      } else {
        console.debug(message);
        convertMP3ToWAV(audio_path_mp3, audio_path_wav);
        const whisper = require('whisper-node');

        (async () => {
          try {
            const transcript = await whisper.whisper(audio_path_wav, {modelName: "base.en", word_timestamps: false});
            const speechText = transcript.map(item => item.speech).join(' ');
            const cleanedSpeechText = speechText.replace(/ ([',.!?])/g, '$1');
            console.debug(cleanedSpeechText);
            resolve(cleanedSpeechText);
          } catch (error) {
            console.error('Error:', error);
            reject(error);
          }
        })();
      }
    });
  });
}

This function:

1. Downloads the audio from the YouTube video.
2. Converts the audio to WAV format.
3. Uses Whisper to generate a transcript of the audio.

GPT API Integration

The next step is to integrate OpenAI’s GPT API to generate curiosity-provoking suggestions based on the video transcript. The following code sets up an Express server to handle requests.

const express = require('express');
const OpenAI = require('openai');

const app = express();
const port = 3000;

app.use(express.json({ limit: '10mb' }));

const openai = new OpenAI({
  apiKey: <OPENAI_KEY>,
});

app.post('/oracle', async (req, res) => {
  const { query, page } = req.body;

  try {
    const completion = await openai.chat.completions.create({
      model: 'gpt-4o',
      messages: [
        { role: 'system', content: 'You are a fact-finding curiosity machine...' },
        { role: 'user', content: `Context: ${context}, Question: ${query}` },
      ],
    });

    const response = completion.choices[0].message.content;
    res.json({ response });
  } catch (error) {
    console.error('Error querying OpenAI:', error);
    res.status(500).json({ error: 'Failed to get a response from OpenAI' });
  }
});

app.listen(port, () => {
  console.log(`Oracle server listening at http://localhost:${port}`);
});

This server:

1. Accepts a query and video URL from the client.
2. Retrieves the transcript for the video.
3. Sends the transcript and query to GPT to generate a response.
4. Returns the response to the client.

Chat Interface

Finally, a simple chat interface is implemented in the browser to allow users to interact with the plugin.

document.getElementById("send").addEventListener("click", () => {
  const query = document.getElementById("input").value;
  chrome.tabs.query({ active: true, currentWindow: true }, (tabs) => {
    chrome.tabs.query(
      { url: ["*://*.youtube.com/*"] },
      async (youtubeTabs) => {
        if (youtubeTabs.length == 0) {
          console.log("No YouTube tab found");
        } else {
          const currentTab = youtubeTabs[0];
          fetch("http://localhost:3000/oracle", {
            method: "POST",
            headers: {
              "Content-Type": "application/json",
            },
            body: JSON.stringify({ query, page: currentTab.url }),
          })
            .then((response) => response.json())
            .then((data) => {
              const response = data.response;
              const chatDiv = document.getElementById("chat");
              chatDiv.innerHTML += `<p><strong>You:</strong> ${query}</p>`;
              chatDiv.innerHTML += `<p><strong>Oracle:</strong> ${response}</p>`;
              document.getElementById("input").value = "";
              chatDiv.scrollTop = chatDiv.scrollHeight;
            })
            .catch((error) => {
              console.error("Error:", error);
            });
        }
      }
    );
  });
});

This script:

1. Captures user input from a chatbox.
2. Sends the query and current YouTube video URL to the server.
3. Displays the response from GPT in the chatbox.

Conclusion

While the prototype demonstrates how AI can be used to encourage curiosity and deeper exploration of topics, there are so many limitations:

1. The chatbox isn’t persistent
2. It requires prompting
3. It doesn’t interrupt the user with suggestions
4. It relies on ytdl, which can be rate limited
5. It requires running OpenAI’s whisper model locally
6. It requires an unreasonable degree of permission and access to user data in the browser
7. There is a fair amount of latency when waiting for responses

Despite all that, it’s pretty neat that you can run Whisper locally on a CPU and have it chat to you about what you watch. It looks like this:

For the full code, checkout this Github repository.

../assets/audio/notebooklm_network_diffusion.wav

Overview

1. NDlib at a Glance
2. Generating Diffusion Plots Per Timestep
3. FuncAnimation to Animate Plots Over Time
4. NDlib Model and Network Configuration
5. Observing Code Updates in the Widget
6. Voila Webapp

NDlib at a Glance

NDlib is a Python library designed for modeling and simulating diffusion processes over complex networks. It provides a variety of epidemic and information diffusion models, such as SIR, SIS, and Independent Cascade, which can be applied to networks created using libraries like NetworkX. NDlib simplifies the process of configuring, running, and visualizing these models, making it an excellent tool for researchers and educators.

For example, here’s how we define a simple SIR model using NDlib:

import networkx as nx
import ndlib.models.ModelConfig as mc
import ndlib.models.epidemics as ep

# Create a graph
g = nx.erdos_renyi_graph(10, 0.2)

# Define the SIR model
model = ep.SIRModel(g)

# Configure the model parameters
cfg = mc.Configuration()
cfg.add_model_parameter('beta', 0.5)  # Infection rate
cfg.add_model_parameter('gamma', 0.1)  # Recovery rate
cfg.add_model_parameter("fraction_infected", 0.2)  # Initial infected fraction
model.set_initial_status(cfg)

# Run the model for a number of timesteps
iterations = model.iteration_bunch(20)

Generating Diffusion Plots Per Timestep

NDlib provides tools to visualize the diffusion process over time. For example, we can use the DiffusionTrend class to plot the number of susceptible, infected, and recovered nodes at each timestep.

from ndlib.viz.mpl.DiffusionTrend import DiffusionTrend

# Generate trends from the model's iterations
trends = model.build_trends(iterations)

# Visualize the trends
viz = DiffusionTrend(model, trends)
viz.plot()

This produces a static plot showing how the diffusion evolves over time.

FuncAnimation to Animate Plots Over Time

To make the diffusion process more interactive, we use Matplotlib’s FuncAnimation to animate the network’s state over time. Each node is colored based on its status (e.g., blue for susceptible, red for infected, green for recovered).

import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

# Define node colors based on their status
def get_node_colors(status):
    colors = []
    for node in status.keys():
        if status[node] == 0:
            colors.append('blue')  # Susceptible
        elif status[node] == 1:
            colors.append('red')  # Infected
        elif status[node] == 2:
            colors.append('green')  # Recovered
    return colors

# Prepare the animation
fig, ax = plt.subplots()
pos = nx.spring_layout(g)
nodes = nx.draw_networkx_nodes(g, pos, node_color=get_node_colors(iterations[0]['status']), ax=ax)
edges = nx.draw_networkx_edges(g, pos, ax=ax)

def update(frame):
    current_state = iterations[frame]['status']
    nodes.set_facecolor(get_node_colors(current_state))
    return nodes,

animation = FuncAnimation(fig, update, frames=len(iterations), interval=500, blit=True)
plt.show()

For a basic random graph as in the example above, you should get something like this:

NDlib Model and Network Configuration

To make the simulation interactive, we use Jupyter widgets. Users can modify the model’s parameters and the number of timesteps using a text area and a slider.

import ipywidgets as widgets
from IPython.display import display

# Text area for modifying the model code
code_input = widgets.Textarea(
    value="""def get_iterations(g, time_steps):
    model = ep.SIRModel(g)
    cfg = mc.Configuration()
    cfg.add_model_parameter('beta', 0.5)
    cfg.add_model_parameter('gamma', 0.1)
    cfg.add_model_parameter("fraction_infected", 0.2)
    model.set_initial_status(cfg)
    return model.iteration_bunch(time_steps), model
""",
    layout=widgets.Layout(width='100%', height='300px')
)

# Slider for adjusting the number of timesteps
time_steps_slider = widgets.IntSlider(
    value=20,
    min=1,
    max=100,
    step=1,
    description='Time Steps:'
)

# Display the widgets
display(code_input)
display(time_steps_slider)

These widgets enable users to experiment with different network configurations and model parameters without modifying the underlying code.

Observing Code Updates in the Widget

The line code_input.observe(run_code, names='value') is used to observe changes in the code_input widget. Whenever the user modifies the code in the text area, the run_code function is triggered. This is necessary because:

1. Dynamic Updates: It allows the application to dynamically re-run the simulation whenever the user changes the code or parameters.
2. Interactivity: Without this observation, the user would need to manually trigger the simulation, reducing the interactivity of the application.
3. Error Handling: The run_code function can handle errors in the user-provided code, ensuring the application remains responsive even if invalid code is entered.

Here’s how the observation is set up:

def run_code(change):
    with output:
        try:
            clear_output(wait=True)
            local_vars = {}
            exec(change['new'], globals(), local_vars)
            get_iterations = local_vars['get_iterations']
            iterations, model = get_iterations(g, time_steps_slider.value)
            # Additional logic for updating plots and animations...
        except Exception as e:
            print("Error executing code:", e)

# Observe changes in the code input
code_input.observe(run_code, names='value')

This ensures that the application reacts immediately to user input, providing a seamless interactive experience.

Voila Webapp

Finally, we use Voila to convert the notebook into a standalone web application. Voila renders the notebook as an interactive dashboard, hiding the code cells and displaying only the widgets, plots, and animations. This makes it easy to share the application with others, even if they don’t have Python or Jupyter installed.

To run the web app, simply execute the following command in the terminal:

voila [network_animation.ipynb](https://github.com/PeterCarragher/diffusion_animator/blob/master/network_animation.ipynb)

This will launch a browser window where users can interact with the diffusion simulation, modify parameters, and observe the results in real-time.

By combining NDlib, Matplotlib, Jupyter widgets, and Voila, this notebook provides a powerful and interactive tool for visualizing diffusion processes in networks. It serves as both an educational resource and a platform for experimentation. For the full code, checkout this Github repository.