Data Processing Agent | Automated Data Pipeline

1

Data Collection

Gather data from various sources

Data Sources

CSV Files

Databases (SQL, NoSQL)

APIs (REST, GraphQL)

Web Scraping

Upload Dataset Files

Click to browse or drag and drop your files

0 files selected

2

Data Preparation

Structure the raw data for processing

The agent will analyze your data structure and suggest appropriate preparation steps.

Data Format

Encoding

Header Row

Delimiter

3

Data Cleaning

Identify and handle missing, incorrect, or inconsistent data

Handling Missing Values

Remove rows with missing values

Impute with mean (numeric)

Impute with mode (categorical)

Mark missing values as 'Unknown'

Handling Outliers

Remove statistical outliers (z-score > 3)

Winsorize (cap extreme values)

Apply log transformation

Text Normalization

Convert to lowercase

Trim whitespace

Remove special characters

Run data cleaning to visualize the cleaning process

4

Data Transformation

Convert data into appropriate formats and scales

Feature Scaling

Standard Scaling (Z-score)

Min-Max Scaling (0-1)

Robust Scaling (median/IQR)

No Scaling

Encoding Categorical Variables

One-Hot Encoding

Label Encoding

Target Encoding

Frequency Encoding

Date/Time Features

Extract year

Extract month

Extract day

Extract day of week

Extract hour

Run data transformation to see before/after comparison

5

Data Integration

Combine data from multiple sources

Upload additional datasets to integrate with your main dataset.

Upload Additional Datasets

Click to browse or drag and drop your files

0 additional files

Integration Method

Join Parameters

Join Type

Left Join Key

Right Join Key

Run data integration to see the combined dataset

6

Data Reduction

Reduce data size while preserving important information

Dimensionality Reduction

PCA (Principal Component Analysis)

t-SNE (t-Distributed Stochastic Neighbor Embedding)

UMAP (Uniform Manifold Approximation)

No Dimensionality Reduction

PCA Settings

Number of Components

Feature Selection

Remove low variance features

Remove highly correlated features (>0.9)

Select K best features (statistical tests)

Select K Best Features Settings

Number of Features (K)

Scoring Function

Run data reduction to see the feature importance and components

7

Exploratory Data Analysis

Understand patterns, relationships, and statistics

Analysis Types

Summary Statistics

Feature Distributions

Correlation Analysis

Missing Values Analysis

Outlier Analysis

Time Trends (if applicable)

Run EDA to generate interactive visualizations and insights

8

Feature Engineering

Create new informative features from existing data

Feature engineering suggestions are based on your data characteristics from EDA.

Automated Feature Creation

Polynomial Features (degree 2)

Interaction Features

Binning/Numeric Discretization

Date/Time Features

Custom Feature Creation

Run feature engineering to see feature importance rankings

9

Data Splitting

Divide data into training, validation and test sets

Splitting Strategy

Default (70% train, 15% validation, 15% test)

Custom Ratios

Time-Based Split

Group-Based Split

Custom Split Ratios

Training Set (%)

Validation Set (%)

Test Set (%)

Time-Based Split Settings

Date/Time Column

Training Period

Validation Period

Group-Based Split Settings

Group Column

Stratification

Preserve target variable distribution across splits

Target Column (For Stratification)

Run data splitting to see the distribution of your datasets

10

Modeling

Build and evaluate machine learning models

The agent will automatically select appropriate models based on your problem type.

Problem Type

Classification

Regression

Clustering

Anomaly Detection

Classification Models (Auto-Selected)

Random Forest

XGBoost

Logistic Regression

SVM

Regression Models (Auto-Selected)

Random Forest Regressor

XGBoost Regressor

Linear Regression

Cross-Validation

Hyperparameter Tuning

Run modeling to see model performance comparison

11

Export & Deployment

Save your processed data and models

Export Options

Processed Dataset