Superfund Database
A comprehensive system for managing and analyzing investment data. Import CSV files, classify investments, perform FIGI lookups, and run SQL queries on your data.
Getting Started
1. Request Access
To use the Superfund Database, you need to be logged in as a superuser.
-
Click the "Login" link in the top navigation
- If you don't have an account, approach a member of the AFR datajournalism team
- Once logged in, you'll have access to all features of the application
2. Import Your Data
Start by importing investment data from CSV files.
-
Navigate to Import CSV in the main menu
- Upload your CSV file containing investment data
- The system will validate and import the data into the database
- Monitor the import progress in real-time using the task panel
3. Process Your Data
After importing, classify your investments and perform FIGI lookups.
-
Go to Classify Import in the main menu
- Select the import you want to process
-
Run the classification tasks in sequence:
-
Classify Investments - Groups unclassified investments by security ID and name, creating Classification records
-
FIGI Lookup - Looks up unlinked classifications against existing entities or the OpenFIGI API
-
Perfect Name Match - Matches unlinked classifications to existing entities using Nilsimsa score for medium confidence
-
Fuzzy Name Match - Groups remaining unprocessed classifications by fuzzy name similarity with low confidence
- Monitor progress and view statistics for each processing step
4. Query Your Data
Use the SQL Query Tool to analyze your processed data.
-
Navigate to Query Tool in the main menu
- Write and execute SELECT queries against your data
- View results in a paginated table format
- Export query results to CSV for further analysis
- Use the query history to reuse previous queries
Security & Performance Notes
- Only SELECT queries are allowed in the Query Tool
- Queries are limited to 45 seconds execution time and 100,000 rows
- All data processing happens asynchronously in the background
- Real-time progress updates are provided via WebSocket connections
Quick Links
Data Tables
The system manages the following main data entities:
-
Investments - Individual investment records
-
Classifications - Categorized investment types with FIGI data
-
Entities - Companies and organizations
-
Imports - CSV import batches and metadata
System Architecture
The Superfund Database is built on Django with the following key components:
-
PostgreSQL - Primary data storage
-
Redis - Task queue and caching
-
Celery - Background task processing
-
Django Channels - Real-time WebSocket updates
-
Bootstrap - Responsive UI framework
All data processing tasks run asynchronously, providing real-time progress updates through WebSocket connections. The system is designed for high performance and reliability when handling large datasets.
Version Information
- App Version
- 2.2.0
- Core Version
- 1.1.6
- Git Commit
80e4cc6
- Dirty State