Data Extraction

Figranium offers powerful visual and script-based tools to extract structured data from web pages. The extraction process executes automatically after all visual action blocks complete.

Visual Field Mapping (Default)

Every task opens its extraction editor in Visual mode by default. Instead of manually writing JavaScript parsers, you define the fields and repeating groups you want to capture and target elements directly on the page. Figranium automatically generates and maintains the underlying execution script. Visual mode is ideal when:

Capturing named fields (e.g., title, price, author, URLs, element visibility) from a target page.
Extracting repeating structured lists (e.g., product cards, search results, table rows).
Building field mappings visually by picking elements directly in the Headful Browser.
Avoiding custom JavaScript boilerplate for common web scraping tasks.

Field Configuration

In Visual mode, click Add Field in the Extraction Script block on the canvas or the Extraction tab in Task Settings. Each field supports six configuration parameters:

Input	Description
Name	The key name in the returned JSON object (e.g., `title`, `price`, `is_in_stock`).
Selector	A standard CSS selector passed to `document.querySelector` / `document.querySelectorAll`. Supports dynamic template variables like `{$query}` or `{$itemId}`.
Attribute Type	The DOM property or extraction logic to execute on matched elements (see below).
Attribute Name	Visible only when Attribute Type is set to `Attribute`. Specifies the target attribute key (e.g., `data-sku`, `alt`, `aria-label`).
Multiple (List)	Toggle On to query all matching elements on the page (`document.querySelectorAll`) and return an array of values. Toggle Off to extract the first match (`document.querySelector`) or `null`.

Supported Attribute Types

Figranium provides seven specialized attribute extraction types:

Text (text): Reads and trims text content (el.textContent.trim()).
HTML (html): Extracts inner HTML (el.innerHTML).
Input Value (value): Reads el.value for form elements (inputs, textareas, selects), falling back to el.textContent.trim() for non-form elements.
Attribute (attr): Reads any explicit DOM attribute via el.getAttribute(attrName).
Element Exists (exists): Evaluates whether a matching element exists in the DOM (document.querySelector(selector) !== null). Returns a boolean true or false.
Image URL (image): Resolves absolute image URLs from <img> elements or CSS background images.
Link URL (link): Resolves absolute link URLs from anchors or clickable elements.

Advanced Media Extraction Types

Image URL Extraction (`image`)

Extracting image URLs from modern websites requires handling lazy loading, responsive srcset definitions, and CSS background images. When an attribute type is set to Image URL, Figranium executes an inline resolver that:

Checks el.currentSrc, el.src, data-src, data-lazy-src, or data-original attributes.
Inspects srcset attributes, parsing candidate URLs and returning the highest-density candidate.
Fallbacks to computed CSS background images (getComputedStyle(el).backgroundImage) by extracting url(...) declarations.
Resolves relative paths against the current page context (location.href) to return a fully qualified absolute URL.

Link URL Extraction (`link`)

When set to Link URL, Figranium checks href, src, or data-href attributes on the matched element and converts relative paths into canonical absolute URLs using the browser’s location.href.

Repeating Groups (`extractionGroups`)

Web pages frequently contain repeating structures such as product listings, search result cards, tabular rows, or comment feeds. Figranium supports Repeating Groups to extract nested, structured array data without requiring manual loops.

How Repeating Groups Work

Container Selector: Defines the parent repeating element on the page (e.g., .product-card, table.results > tr).
Sub-Fields: Defines individual fields (e.g., title, price, image_url) scoped to each container instance.
Relative Scope: Each sub-field selector is executed relative to its parent container element (container.querySelector(...)).

Data Structure Example

Extracting a repeating group named products with container selector .product-card yields an array of structured objects:

{
  "products": [
    {
      "name": "Wireless Headphones",
      "price": "$99.99",
      "in_stock": true,
      "image": "https://example.com/images/headphones.jpg"
    },
    {
      "name": "Mechanical Keyboard",
      "price": "$149.00",
      "in_stock": false,
      "image": "https://example.com/images/keyboard.jpg"
    }
  ]
}

Under the Hood: Script Generation

When running in Visual mode, Figranium translates field definitions and repeating groups into an optimized JavaScript extraction function executed via Playwright’s page.evaluate().

Generated Script Example

For a task configured with top-level fields (title, banner_image, is_sale) and a repeating group (items), Figranium generates code equivalent to:

return {
  "title": (() => { const el = document.querySelector(".page-title"); return el ? el.textContent.trim() : null; })(),
  "banner_image": (() => { const el = document.querySelector(".hero-banner"); return el ? (() => {
        const src = el.currentSrc || el.getAttribute('src') || el.getAttribute('data-src') || el.getAttribute('data-lazy-src') || el.getAttribute('data-original');
        if (src) return new URL(src, location.href).href;
        const srcset = el.getAttribute('srcset');
        if (srcset) {
          const candidates = srcset.split(',').map(s => s.trim().split(' ')[0]).filter(Boolean);
          if (candidates.length) return new URL(candidates[candidates.length - 1], location.href).href;
        }
        const bg = getComputedStyle(el).backgroundImage;
        const match = bg && bg.match(/url\((['"]?)(.*?)\1\)/);
        if (match && match[2]) return new URL(match[2], location.href).href;
        return null;
      })() : null; })(),
  "is_sale": document.querySelector(".sale-badge") !== null,
  "items": Array.from(document.querySelectorAll(".item-card")).map(container => ({
      "name": (() => { const el = container.querySelector(".item-name"); return el ? el.textContent.trim() : null; })(),
      "price": (() => { const el = container.querySelector(".item-price"); return el ? el.textContent.trim() : null; })()
  }))
};

Picking Selectors with Headful Inspector

Next to selector inputs for both individual fields and repeating group containers is a target icon that launches the Headful Browser element inspector.

Click the target icon next to any field or group container selector.
Click any element on the live browser canvas.
Figranium inspects the element hierarchy and generates robust CSS selectors.
A list of candidate selectors is displayed, allowing single-click substitution.

Extraction scripts run using native browser methods (document.querySelector), which do not support Playwright-specific pseudo-selectors like :has-text(...). Figranium automatically filters out incompatible pseudo-selectors from inspector candidate lists when configuring visual extraction fields.

Switching to JavaScript Mode

The mode toggle at the top of the Extraction editor allows switching between Visual and JavaScript modes.

Visual Mode: Manages field mappings and repeating groups visually.
JavaScript Mode: Exposes the full JavaScript editor for writing custom extraction logic, data transformation pipelines, or post-processing algorithms.

Switching from Visual to JavaScript mode preserves your field definitions and displays the compiled JavaScript code. You can modify the code directly in JavaScript mode. However, if you switch back from JavaScript mode to Visual mode, the script will be regenerated from the current visual field definitions.

Extraction Script Environment (JavaScript Mode)

In JavaScript mode, the code executes inside the browser page context and has access to:

document: The DOM of the active page.
$$data.html(): A utility returning the cleaned page HTML string.
variables: Task runtime variables accessible as a key-value object.

Example: Custom JavaScript Extraction

// Extract all product cards and filter by price
const cards = Array.from(document.querySelectorAll(".product-card"));

const data = cards.map(card => {
  const title = card.querySelector(".title")?.textContent.trim();
  const rawPrice = card.querySelector(".price")?.textContent.trim();
  const price = rawPrice ? parseFloat(rawPrice.replace(/[^0-9.]/g, '')) : 0;
  const link = card.querySelector("a")?.href;

  return { title, price, link };
}).filter(item => item.price > 0 && item.price < 100);

return {
  total: data.length,
  items: data
};

DOM Cleaning and Shadow DOM

Before extraction runs, Figranium optimizes page HTML to reduce payload size and eliminate noise.

Stripped Elements

The following tags are removed during cleaning: script, style, link, meta, noscript, svg, canvas, iframe, object, embed, applet, param, source, track

Preserved Attributes

Identification: id, class, name
Links & Media: href, src, alt, title, value, placeholder, content, datetime
Accessibility & Forms: aria-label, type, for, action, method, selected, checked, disabled
Tables: colspan, rowspan, scope
Custom Data Attributes: All data-* attributes (e.g., data-id, data-sku, data-price)

Shadow DOM Support

If a web page utilizes open Shadow DOM trees (common in Web Components), Figranium serializes shadow roots into <template data-shadowroot="open"> elements inside cleaned HTML. Shadow DOM inclusion can be toggled on or off in Task Settings.

When scraping pages built with Web Components, enable Shadow DOM support in Task Settings so selectors can traverse shadow boundaries through the serialized template elements.

AI Extraction Script Generation

Figranium includes AI assistance to write custom extraction scripts from natural language prompts. For detailed instructions on model settings and prompt guidelines, refer to the AI Extraction Script Generation guide.

Output Formats and Dynamic Content

Handling Dynamic Content

If a page loads data asynchronously via AJAX or client-side rendering, include wait or wait_selector action blocks prior to the extraction step to ensure all DOM elements are present before extraction runs.

CSV Output Formatting

When setting the task output format to CSV:

Ensure the extraction result returns an Array of Objects (or an object containing a single array property).
Object keys in the array elements automatically become CSV header columns.
Values are formatted, quoted, and escaped automatically according to RFC 4180 standards.

Single Repeating Group Unwrapping

When the extraction result is an object with a single key whose value is an array of objects (the shape produced by a lone repeating group), Figranium unwraps that array before serialization. Each group item becomes its own CSV row instead of a single row containing a stringified array in one cell. For example, an extraction that returns:

{
  "products": [
    { "name": "Wireless Headphones", "price": "$99.99" },
    { "name": "Mechanical Keyboard", "price": "$149.00" }
  ]
}

is exported as:

name,price
Wireless Headphones,$99.99
Mechanical Keyboard,$149.00

Add any top-level field alongside the repeating group to opt out of unwrapping and get one row containing all keys.

Extraction Script Generation

Generate custom extraction scripts with AI assistance.

JavaScript Execution

Write custom mid-flow logic and data transformation.

CSV Processing

Parse and iterate over CSV data in your tasks.

Selectors

Learn how to target elements for extraction and actions.

Intro

Installation

Basics

Features

Agent Logic

API

JavaScript SDK

Integrations

Ops

Changelog

Visual Field Mapping (Default)

Field Configuration

Supported Attribute Types

Advanced Media Extraction Types

Image URL Extraction (`image`)

Link URL Extraction (`link`)

Repeating Groups (`extractionGroups`)

How Repeating Groups Work

Data Structure Example

Under the Hood: Script Generation

Generated Script Example

Picking Selectors with Headful Inspector

Switching to JavaScript Mode

Extraction Script Environment (JavaScript Mode)

Example: Custom JavaScript Extraction

DOM Cleaning and Shadow DOM

Stripped Elements

Preserved Attributes

Shadow DOM Support

AI Extraction Script Generation

Output Formats and Dynamic Content

Handling Dynamic Content

CSV Output Formatting

Single Repeating Group Unwrapping

Extraction Script Generation

JavaScript Execution

CSV Processing

Selectors

​Visual Field Mapping (Default)

​Field Configuration

​Supported Attribute Types

​Advanced Media Extraction Types

​Image URL Extraction (image)

​Link URL Extraction (link)

​Repeating Groups (extractionGroups)

​How Repeating Groups Work

​Data Structure Example

​Under the Hood: Script Generation

​Generated Script Example

​Picking Selectors with Headful Inspector

​Switching to JavaScript Mode

​Extraction Script Environment (JavaScript Mode)

​Example: Custom JavaScript Extraction

​DOM Cleaning and Shadow DOM

​Stripped Elements

​Preserved Attributes

​Shadow DOM Support

​AI Extraction Script Generation

​Output Formats and Dynamic Content

​Handling Dynamic Content

​CSV Output Formatting

​Single Repeating Group Unwrapping

Extraction Script Generation

JavaScript Execution

CSV Processing

Selectors

Visual Field Mapping (Default)

Field Configuration

Supported Attribute Types

Advanced Media Extraction Types

Image URL Extraction (`image`)

Link URL Extraction (`link`)

Repeating Groups (`extractionGroups`)

How Repeating Groups Work

Data Structure Example

Under the Hood: Script Generation

Generated Script Example

Picking Selectors with Headful Inspector

Switching to JavaScript Mode

Extraction Script Environment (JavaScript Mode)

Example: Custom JavaScript Extraction

DOM Cleaning and Shadow DOM

Stripped Elements

Preserved Attributes

Shadow DOM Support

AI Extraction Script Generation

Output Formats and Dynamic Content

Handling Dynamic Content

CSV Output Formatting

Single Repeating Group Unwrapping