---
title: "earthUI User Guide"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{earthUI User Guide}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

## Introduction

`earthUI` is a graphical user interface for the R `earth` package, which
implements Multivariate Adaptive Regression Splines (MARS). It runs as a
local Shiny application --- no login, no server, no accounts. You launch it
from R, import a dataset (CSV or Excel), configure your model, and fit it
interactively.

### Three Purpose Modes

When you launch earthUI, a **Purpose** radio button at the top of the sidebar
lets you choose one of three modes:

- **General** --- Earth regression for any dataset. The default mode.
- **For Appraisal** --- Tailored for real estate appraisal with subject
  property handling, special column designations, RCA adjustments, and Sales
  Comparison Grid generation.
- **Market Area Analysis** --- For analyzing groups of properties with optional
  subject row exclusion.

Switching purpose clears all state (imported data, results, settings). When
you reopen the project, previously saved settings for that project and purpose
combination are automatically restored.

### Getting Started

```{r eval=FALSE}
library(earthUI)
launch()
```

This opens the Shiny app at `http://localhost:7878`.

## Locale & Regional Settings

earthUI supports international number, date, and CSV formatting conventions
through a country-based locale system. The **Country** dropdown in Section 1
of the sidebar (below the file upload) selects a preset for 31 supported
countries. Each preset configures:

- **CSV separator** --- comma (`,`) for US/UK/Japan or semicolon (`;`) for
  most of Europe, where the comma is used as a decimal mark.
- **Decimal mark** --- period (`.`) or comma (`,`).
- **Thousands separator** --- comma (US/UK/Japan), period (Germany/Italy/Spain),
  space (Finland/France/Poland/Baltics/Ukraine/Russia), or apostrophe
  (Switzerland).
- **Date format** --- MM/DD/YYYY (US), DD/MM/YYYY (most of Europe), or
  YYYY-MM-DD (Sweden/Lithuania/Japan/Canada).
- **Paper size** --- Letter (US/Canada/Mexico) or A4 (everywhere else).

### Override Dropdowns

Below the country selector, four override dropdowns let you change individual
settings without switching countries:

- **Paper** --- Letter or A4 (affects PDF report page size)
- **CSV sep** --- comma or semicolon (used when importing CSV files)
- **Decimal** --- period or comma (used in number display on plots and axes)
- **Date** --- MDY, DMY, or YMD (controls the order in which date formats
  are tried when parsing date columns)

When you change the country, all overrides reset to that country's defaults.
Changing an override only affects that one setting.

### Saving Defaults

Click **Save as my default** to store your locale preferences globally. These
defaults apply to all future sessions regardless of which project you open.
Project settings (target, predictors, parameters) are saved separately in the
project database via the **Save current as default** buttons, scoped per
project and purpose. Locale defaults persist across all projects via the same
database.

## MLS Input Data Requirements

For real estate workflows, your input data typically comes from an MLS export.
earthUI accepts CSV and Excel files. On import, column names are automatically
converted to `snake_case`, e.g. "Area ID" gets converted to "area_id". The CSV
separator and decimal mark used during import are determined by the locale
settings (see "Locale & Regional Settings" above).

### Required Columns for Appraisal Mode

The full appraisal workflow (RCA + Sales Grid) benefits from these columns:

| Column | Special Type | Purpose |
|:-------|:-------------|:--------|
| Sale Price (or equivalent wording) | (target) | Response variable |
| Contract Date | `contract_date` | Computes `sale_age` from effective date |
| Listing Date | `listing_date` | Fallback DOM = contract date - listing date |
| Days on Market | `dom` | Displayed in Sales Grid |
| Concessions | `concessions` | Net SP = Sale Price - Concessions |
| Living Area (SF) | `living_area` | Per-SF residuals (`residual_sf`, `cqa_sf`) |
| Lot Size | `lot_size` | Grouped in "Site Size" row |
| Site Dimensions | `site_dimensions` | Grouped with lot size |
| Latitude | `latitude` | Rounded to 3 dp; proximity calc; Location group |
| Longitude | `longitude` | Rounded to 3 dp; proximity calc; Location group |
| Area ID | `area` | Grouped in "Location" row |
| Actual Age | `actual_age` | Grouped in "Age" row |
| Effective Age | `effective_age` | Grouped with actual age |
| Address | `display_only` | Shown in grid; excluded from model |

Spreadsheet column names can be in a foreign language, the "special" names are 
in English so that the R program can give them special treatment.  Otherwise, 
the given column names show up in the regression models, graphs and, if doing
appraisals, the Intermediate Sales Grid.

Not all columns are required. earthUI adapts --- if a column is missing, the
corresponding feature is simply omitted.  However, that being said, for real 
estate pricing models certain columns are highly recommended to achieve acceptable
fit:

1.  "Sale Age,"  which is the number of days between the contract sale date and
the effective date of the appraisal or analysis.  If multi-year sales history is 
being used, especially for periods over 5 years, sale_age often plays a central 
role in estimating the sale price. In fact it is often so important that
without it, earthUI fails to provide any model at all.

2.  "Living Area" which also goes by names such as "Living Sqft,"  "GLA" (gross
living area) and so on.  This is also another leading determinant of sale price.

3.  "Total Bath Count" is the total number of full, quarter, half and 3/4
bathrooms. For example, two full baths and one half-bath would be a value of 2.5.

4.  "Garage Bays" or "Garage Area" --- the number of garage spaces or the
garage square footage.

5.  "Lot Size" --- the land area of the property, typically in square feet or
acres.

6.  "Longitude," "Latitude," and if available "Area ID." Adjustments for these
will be combined under a single Location adjustment in the Sales Grid.

### Data Quality

- **Missing values**: Rows with NAs in predictors or target are removed before
  fitting.
- **Date columns**: Should be in a format R can parse (e.g., `2025-06-15`).
- **Numeric columns**: Sale price, living area, etc. must be numeric (no
  dollar signs or commas).

### Subject Row Placement

In **Appraisal** mode, row 1 must be the subject property. In **Market**
mode, placing the subject in row 1 is optional (use the "Skip first row"
checkbox). In **General** mode, all rows are treated equally.

## The Sidebar Workflow

The sidebar is organized into numbered sections:

1. **Project** --- pick or create a project; sets the input/output locations
2. **Import Data** --- choose a CSV or Excel file from the project's input folder
3. **Variable Configuration** --- target, effective date, predictors, data types, special columns
4. **Earth Call Parameters** --- degree, penalty, pruning, CV, subset filters.
   Recommended values are displayed below key parameters (see below).
5. **Fit Earth Model** --- green button to run the model
6. **Download Output** --- Excel file with predictions and diagnostics
7. **Calculate RCA Adjustments** (appraisal only)
8. **Generate Sales Grid** (appraisal only)
9. **Download Report** --- HTML, Word, or PDF via Quarto

## Variable Selection

### The Predictor Table

Each column is listed with:

- **Type** --- data type dropdown (numeric, integer, character, factor, Date, etc.)
- **Inc?** --- include as predictor
- **Special** --- special column designation (appraisal/market only)
- **Factor** --- treat as categorical
- **Linear** --- force linear entry (no hinges)
- **NAs** --- missing value count

### Special Column Types

In appraisal and market modes, each predictor can be assigned a special type:

**Date & Time:**

- `contract_date` --- triggers `sale_age` computation
- `listing_date` --- fallback for DOM calculation
- `dom` --- Days on Market column

**Monetary:**

- `concessions` --- used in Net SP formula (Sale Price - Concessions)

**Size & Location:**

- `latitude` / `longitude` --- rounded to 3 dp; proximity and Location group
- `area` --- grouped with lat/lon in Location row
- `living_area` --- enables per-SF residuals
- `lot_size` / `site_dimensions` --- grouped in Site Size row

**Age:**

- `actual_age` / `effective_age` --- grouped in Age row

**Display:**

- `display_only` --- in exports but excluded from model (multiple allowed)

Only one column per special type (except `display_only`). A blue badge shows
the assigned type next to each variable name.

## Allowed Interactions (Degree 2+)

When `degree >= 2`, the **Allowed Interactions** matrix appears under
Earth Call Parameters. This upper-triangular matrix controls which
variable pairs can form interaction terms (hinge products).

- **Check a cell** to allow the interaction `h(x1) * h(x2)`
- **Left-click a variable name** (row or column) to toggle all its
  interactions at once
- **Allow All / Clear All** buttons toggle the entire matrix
- Selections are saved with the earth parameters when you click **Save current
  as default** in Section 4 (stored in the project database, per project and purpose)

### Block from Degree 1 (Interaction Only)

**Right-click** a variable name in the Allowed Interactions matrix to
block it from entering the model as a main effect (degree-1 hinge).
A bold **" 1"** indicator appears after the variable name.

When blocked:

- The variable **cannot** enter as `h(variable - knot)` (main effect)
- The variable **can** enter through interactions like
  `h(variable - k1) * h(other_var - k2)` (degree 2+)
- Right-click again to unblock

**Use case:** When modeling time adjustments that should scale with
property size, block `sale_age` from degree 1 and allow it only in an
interaction with `living_area`. This forces earth to discover
`h(sale_age - k1) * h(living_area - k2)` -- a time appreciation rate
that varies by house size -- rather than a flat $/day rate that applies
equally to all properties.

```{r eval=FALSE}
# Programmatic equivalent using the allowed function:
mat <- build_allowed_matrix(c("sale_age", "living_area", "lot_size"))
func <- build_allowed_function(mat, block_degree1 = "sale_age")
result <- fit_earth(df, "sale_price", predictors,
                    degree = 2L, allowed_func = func)
```

## Recommended Parameter Values

earthUI displays recommended values in blue text below key parameters in
Section 4. These update reactively as you load data and select predictors,
based on the number of fitting rows (*n*) and predictors (*p*).

### Summary Table

| Parameter | Formula | n=200, p=10 | n=1500, p=10 |
|:----------|:--------|------------:|-------------:|
| **nk** | min(100, max(21, 2p+1, n/10)) | 21 | 100 |
| **minspan** | min(16, 5 + n/50) | 9 | 16 |
| **endspan** | min(16, 5 + n/28) | 12 | 16 |
| **penalty** | 3 if degree>1, else 2 | 2 | 2 |
| **pmethod** | backward | backward | backward |
| **nprune** | leave empty | NULL | NULL |
| **nfold** | min(15, max(10, n/100)) | 10 | 15 |
| **ncross** | max(3, 100/n) | 3 | 3 |
| **varmod.method** | lm | lm | lm |
| **newvar.penalty** | 0.1 if collinear predictors | 0.1 | 0.1 |

### Key Rationale

- **nk**: Earth's default (`max(21, 2p+1)`) does not account for dataset
  size. For large datasets, `n/10` provides enough forward-pass capacity
  for the greedy search to explore all predictors with multiple knots.
- **minspan / endspan**: Earth's auto-calculation scales as log(n*p),
  which grows too slowly for large *n*. The linear formulas target
  approximately 100 candidate knot locations per continuous predictor,
  preventing the forward pass from fitting noise. Endspan is set slightly
  larger for additional boundary protection.
- **pmethod = backward**: GCV-based pruning is deterministic --- no
  dependence on random seeds, which is critical for reproducibility.
- **nfold / ncross**: With `pmethod = backward`, CV only provides the
  diagnostic CVR^2^ and residuals for the variance model. The formulas
  balance accuracy with computational cost.
- **newvar.penalty = 0.1**: When predictors are correlated (e.g., living
  area with bedroom count), a small penalty biases the forward pass toward
  reusing existing predictors, producing simpler and more interpretable
  models.

### Earth Output File

After every fit, earthUI writes `<filename>_earth_output_<timestamp>.txt`
to the output folder containing the model, summary statistics, variance
model, and trace log --- useful for comparing parameter configurations
across runs.

See the earthUI User Guide, Chapter 7 for detailed derivations and
additional scaling tables.

## Downloading Data (Step 6)

After fitting, download an Excel file with predictions and diagnostics.
This output is used in Step 7 (RCA) to assign a CQA (Condition/Quality/Appeal)
rating to the subject property.  The output is sorted by `residual_sf` and
`cqa_sf` to help you assess where the subject falls in the ranking.  If the model is good quality, then
the properties should be ranked from lowest appealing to most appealing based
on residual features that did not go into the regression.  The middle value
should be approximately 0, the lower half negative values and the upper half
positive values.  You should find the worst quality homes, or "fixers" near the
bottom of the ranking and the nicest homes at the top.  There will usually be
exceptions for anomalies such as foreclosures, short sales, probate (inheritance
related) sales, and quick sales needed for job change or other reasons.  
Investigation of anomalies usually turns up a pertinent reason for the price
anomaly.

### Output Columns

| Column | Description |
|:-------|:------------|
| `est_<target>` | Model prediction (e.g., `est_sale_price`) |
| `residual` | Actual - predicted |
| `cqa` | Comparative Quality Analysis score (0-10 scale) |
| `residual_sf` | Residual / living area (if designated) |
| `cqa_sf` | CQA calculated from ranking via residual_sf |
| `<var>_contribution` | Per-g-function contribution |
| `basis` | Intercept value contribution (same for all properties) |
| `calc_residual` | Verification column |

### Column Ordering & Formatting

The ranking columns are placed leftmost: `residual_sf`, `cqa_sf`, `residual`,
`cqa`. Excel formatting:

- `residual_sf` --- numeric, 2 decimal places
- `cqa_sf` --- numeric, 2 decimal places
- `residual` --- numeric, 0 decimal places
- `cqa` --- numeric, 2 decimal places

### CQA Scores

CQA ranks each row's residual against all others on a 0--10 scale:

- **High CQA (~9-10)**: sold for much more than predicted
- **Low CQA (~0-1)**: sold for much less than predicted
- **CQA ~5**: near the median

In appraisal/market modes, rows are sorted by `residual_sf` descending.

## mgcvUI Auto-Export

On every successful fit with `degree <= 2`, earthUI automatically saves the
full result object as an `.rds` file to the project's output folder. The filename
follows the pattern `<datafile>_earthUI_result_<YYYYMMDD_HHMMSS>.rds`.

This file can be loaded by [mgcvUI](https://github.com/wcraytor/mgcvUI) (a
companion Shiny app for GAM modeling) using `readRDS()`. mgcvUI uses the
earth model's knot locations and basis functions as starting points for GAM
smooth terms, enabling a seamless transition from MARS to GAM modeling.

Models with `degree > 2` are skipped because mgcvUI only supports pairwise
interactions. A manual **Export for mgcvUI** button is also available in the
sidebar for on-demand export.

## RCA Calculations (Step 7, Appraisal Only)

### The RCA Dialog

- Choose **CQA** or **CQA per SF** score type. If you choose "CQA per SF", then 
based on the CQA score you assign the subject, its residual score will be its 
living_area times the residual_sf that matches the given CQA_SF score.
- Enter the subject's CQA score (0.00--9.99, default 5.00)
- Click **Generate** to compute adjustments

### How It Works

1. Comparables' CQA scores and residuals are sorted
2. Linear interpolation maps your CQA value to a residual
3. Subject value = model prediction + interpolated residual

### Output Columns

| Column | Description |
|:-------|:------------|
| `subject_value` | Model prediction + interpolated residual |
| `<var>_adjustment` | Subject contribution - comp contribution |
| `residual_adjustment` | Subject residual - comp residual |
| `net_adjustments` | Sum of all adjustments |
| `gross_adjustments` | Sum of absolute adjustments |
| `adjusted_sale_price` | Comp sale price + net adjustments |

## Sales Comparison Grid (Step 8, Appraisal Only)

### Overview

The Sales Grid is a multi-sheet Excel workbook with the subject and selected
comparables in a structured format. Excel formulas compute adjustments and
adjusted sale prices. Output: `SalesGrid_<timestamp>.xlsx`.

### Comp Selection

A modal dialog shows:

- **Recommended comps** (pre-checked): gross adjustment < 25% of sale price,
  sorted by gross adjustment % ascending
- **Additional comps** (unchecked): larger adjustments
- Maximum 30 comps / 10 sheets (3 comps per sheet)


### Grid Layout

Each sheet has 20 columns: subject + 3 comps (5 columns each). Rows from top
to bottom:

1. **Title** and **Headers**
2. **Address**
3. **APN | MLS# | DOM | Subj.Prox** --- parcel, listing ID, days on market,
   Haversine distance (miles)
4. **Sales Price | Concess. | Net SP** --- sale price, concessions, Net SP
   formula
5. **Regression Features** header
6. **BASE VALUE** --- model intercept
7. **Date of Sale | OffMkt | OnMkt** --- contract date, sale age, DOM
8. **Grouped rows** (conditional):
   - "Loc: Long | Lat | Area" --- combined VC from location variables
   - "Site Size | Dimensions" --- combined VC from site variables
   - "Actual Age | Effective Age" --- combined VC from age variables
9. **Model variable rows** --- one per predictor (excluding grouped vars)
10. **CQA|Residual** --- CQA score + remaining residual formula
11. **Residual feature rows** --- named + blank rows for appraiser entry
12. **Total VC / Net Adjustment**
13. **Net/Gross Adjustment %**
14. **Adjusted Sale Price** --- formula row

### Adjusted Sale Price Formula

- **Subject**: SUM of all Value Contribution cells from BASE VALUE through
  the last residual feature row
- **Comps**: Net SP + SUM of all Adjustment cells above Total VC row

### Sheet Protection

Sheets are protected. Only the **residual feature VC input cells** (light
yellow background) under CQA|Residual are unlocked for appraiser entry.
All formulas, data values, and labels are locked.

### Working with the Grid

1. Open in Excel
2. Review regression-derived adjustments (locked)
3. Allocate the residual in the yellow cells (Location, View, Condition,
   Quality, etc.)
4. Watch the Remaining Residual decrease and Adjusted Sale Price update
5. Goal: allocate until Remaining Residual is near zero

## Downloading Reports (Step 9)

Three formats via Quarto:

- **HTML** --- with KaTeX math rendering
- **Word** --- `.docx` for editing and distribution, with page numbers
- **PDF** --- typeset with LuaLaTeX (paper size follows locale: Letter or A4)

Reports include: dataset description, model specification, allowed
interactions, summary metrics, model equation, coefficients, variable
importance, g-function plots, correlation matrix, diagnostics, ANOVA, and
raw earth output.

Report rendering runs in the background --- a modal dialog shows elapsed
time and Quarto progress, and the app remains responsive.

All operations are logged to `<filename>_earthui_log.txt` in the output
folder with start/end timestamps and elapsed times for troubleshooting.

## Demo Dataset: Appraisal_1.csv

earthUI includes a demo MLS dataset for exploring the appraisal workflow.
Load it with:

```{r eval=FALSE}
demo_file <- system.file("extdata", "Appraisal_1.csv", package = "earthUI")
df <- import_data(demo_file)
```

Or import it directly through the Shiny app file upload.

### Description

The file contains 1,502 residential sales (plus 1 subject property in row 1)
from a simulated MLS export. The data represents single-family home sales
in a multi-area market with a range of property sizes, ages, and locations. 

This is not real data, but is based on a realistic neighborhood in Northern 
California.  All identification information has been altered or removed.

### Columns

| Column | Type | Special Type | Description |
|:-------|:-----|:-------------|:------------|
| `weight` | numeric | --- | Observation weight (0 = exclude from fitting) |
| `id` | numeric | `display_only` | Internal record ID |
| `property_id` | numeric | `display_only` | MLS property identifier |
| `listing_id` | character | `display_only` | MLS listing number |
| `parcel_number` | character | `display_only` | County assessor parcel number (APN) |
| `street_address` | character | `display_only` | Property address |
| `city_name` | character | `display_only` | City |
| `postal_code` | character | `display_only` | ZIP code |
| `county_name` | character | `display_only` | County |
| `contract_date` | Date | `contract_date` | Sale contract date (computes `sale_age`) |
| `sale_age` | numeric | --- | Days from contract date to effective date (pre-computed) |
| `coe_date` | Date | `display_only` | Close of escrow date |
| `listing_status` | character | `display_only` | Listing status (e.g., "Sold") |
| `sale_price` | numeric | (target) | Sale price --- response variable |
| `rent` | numeric | --- | Monthly rent (for multi-target models) |
| `list_price` | numeric | `display_only` | Listing price |
| `original_list_price` | numeric | `display_only` | Original listing price |
| `living_sqft` | numeric | `living_area` | Gross living area in square feet |
| `beds_total` | integer | --- | Number of bedrooms |
| `baths_total` | numeric | --- | Total bath count (e.g., 2.5 = 2 full + 1 half) |
| `lot_size` | numeric | `lot_size` | Lot size in square feet |
| `area_id` | integer | `area` | MLS area identifier |
| `area_text` | character | `display_only` | Area name |
| `age` | numeric | `actual_age` | Property age in years |
| `year_built` | integer | `display_only` | Year of construction |
| `latitude` | numeric | `latitude` | Latitude (rounded to 3 dp for model) |
| `longitude` | numeric | `longitude` | Longitude (rounded to 3 dp for model) |
| `latitude6` | numeric | `display_only` | Full-precision latitude |
| `longitude6` | numeric | `display_only` | Full-precision longitude |
| `garage_spaces` | integer | --- | Number of garage bays |
| `fp_count` | integer | --- | Number of fireplaces |
| `no_of_stories` | numeric | --- | Number of stories |
| `style` | character | --- | Architectural style |
| `view` | character | --- | View type (e.g., "Neighborhood", "Hills") |
| `days_on_market` | integer | `dom` | Days on market |
| `listing_date` | Date | `listing_date` | Listing date |
| `sale_concessions` | numeric | `concessions` | Seller concessions |

### Suggested Quick Start

1. Launch earthUI: `earthUI::launch()`
2. Import `Appraisal_1.csv` via the file upload
3. Set Purpose to **For Appraisal**
4. Select `sale_price` as the target
5. Assign special types as shown in the table above
6. Include predictors: `sale_age`, `living_sqft`, `baths_total`, `lot_size`,
   `area_id` (as factor), `age`, `latitude`, `longitude`, `garage_spaces`
7. Set degree to 1, click **Fit Earth Model**
8. Download intermediate output (Step 6), review the CQA ranking
9. Compute RCA adjustments (Step 7) with a CQA score of ~5.00
10. Generate the Sales Comparison Grid (Step 8)

## System Requirements & Troubleshooting

### Supported Platforms

earthUI runs on **macOS**, **Windows**, and **Linux**. RStudio Desktop
(2023.06+) is strongly recommended — it bundles **pandoc** needed for
reports.

### Optional Dependencies

- **LaTeX** (TinyTeX, MiKTeX, or MacTeX): PDF report generation only. If
  not detected, the PDF option is automatically hidden. Install with:
  `tinytex::install_tinytex()`
- **sysfonts + showtext**: Roboto Condensed font for plots. Falls back to
  system sans-serif automatically if unavailable or offline.

### Platform Notes

**macOS**: Fewest issues. Install TinyTeX for PDF:
`tinytex::install_tinytex()`

**Windows**: Works well with RStudio. Corporate/locked-down machines may
have temp directory restrictions — the app will warn in the console but
continue without settings persistence.

**Linux**: May need system libraries:
`sudo apt install libcurl4-openssl-dev libssl-dev libxml2-dev
libsqlite3-dev libfontconfig1-dev`

### Graceful Degradation

| Missing Component | Behavior |
|---|---|
| No LaTeX | PDF option hidden. HTML and Word still available. |
| No internet / fonts fail | System sans-serif used. Console message logged. |
| No RSQLite / read-only filesystem | Settings don't persist. App runs normally. |
| Temp directory not writable | Report generation fails with clear error. |

### Troubleshooting

**"PDF option not available"**: Run `tinytex::install_tinytex()` in R,
restart app.

**"Settings will not persist"**: Check permissions on the user data
directory.

**Port 7878 already in use**: Run `lsof -ti:7878 | xargs kill`
(macOS/Linux).
