Skip to content
Newsfork Docs

This document was automatically generated from the README in the test folder.

NewsFork Seeds - Test Directory

Section titled β€œNewsFork Seeds - Test Directory”

This directory is a separate project for testing GitHub Actions-based automation pipelines.

πŸ“ Directory Structure

Section titled β€œπŸ“ Directory Structure”
test/
β”œβ”€β”€ .github/                    # GitHub Actions μ›Œν¬ν”Œλ‘œμš°
β”‚   β”œβ”€β”€ workflows/              # CI/CD μ›Œν¬ν”Œλ‘œμš°
β”‚   β”‚   β”œβ”€β”€ research-engine.yml    # Phase 1: URL 발견 & Dataset 생성
β”‚   β”‚   β”œβ”€β”€ analysis-engine.yml    # Phase 2: Seed 후보 뢄석
β”‚   β”‚   β”œβ”€β”€ contract-engine.yml    # Phase 3: Seed JSON 생성
β”‚   β”‚   └── ci.test.yml            # 톡합 CI ν…ŒμŠ€νŠΈ
β”‚   β”œβ”€β”€ scripts/                # CI/CD 슀크립트
β”‚   β”‚   β”œβ”€β”€ ci.sh
β”‚   β”‚   └── steps/
β”‚   └── README.md
β”‚
β”œβ”€β”€ cli/                        # CLI μ‹€ν–‰κΈ°
β”‚   β”œβ”€β”€ research-engine.ts      # Research 단계 μ‹€ν–‰κΈ°
β”‚   β”œβ”€β”€ analysis-engine.ts      # Analysis 단계 μ‹€ν–‰κΈ°
β”‚   └── contract-engine.ts      # Contract 단계 μ‹€ν–‰κΈ°
β”‚
β”œβ”€β”€ schemas/                    # Zod μŠ€ν‚€λ§ˆ (Single Source of Truth)
β”‚   β”œβ”€β”€ domain.schema.ts        # Domain/Seed ID 검증
β”‚   β”œβ”€β”€ research.schema.ts      # Research/Liveness 검증
β”‚   β”œβ”€β”€ seed.schema.ts         # Enhanced Seed V1 검증
β”‚   └── index.ts               # 톡합 export
β”‚
β”œβ”€β”€ services/                   # 순수 ν•¨μˆ˜ λΉ„μ¦ˆλ‹ˆμŠ€ 둜직
β”‚   β”œβ”€β”€ domain.service.ts       # 도메인 μ •κ·œν™” (Pure Functions)
β”‚   β”œβ”€β”€ research.service.ts     # λ¦¬μ„œμΉ˜ 발견 (Pure Functions)
β”‚   β”œβ”€β”€ seed.service.ts         # μ‹œλ“œ 생성 (Pure Functions)
β”‚   └── index.ts               # 톡합 export
β”‚
β”œβ”€β”€ utils/                      # 핡심 μœ ν‹Έλ¦¬ν‹°
β”‚   β”œβ”€β”€ domain-normalizer.ts   # λ ˆκ±°μ‹œ ν˜Έν™˜μ„±
β”‚   └── kv-deduplication.ts     # KV λ ˆμ§€μŠ€νŠΈλ¦¬
β”‚
β”œβ”€β”€ research/                   # Research κ²°κ³Ό (ν…ŒμŠ€νŠΈ 데이터)
β”‚   β”œβ”€β”€ datasets/
β”‚   β”œβ”€β”€ liveness/
β”‚   β”œβ”€β”€ blocked/
β”‚   └── dead/
β”‚
β”œβ”€β”€ analysis/                   # Analysis κ²°κ³Ό (ν…ŒμŠ€νŠΈ 데이터)
β”‚   └── country=sg/
β”‚
β”œβ”€β”€ results/                    # μ΅œμ’… κ²°κ³Ό (ν…ŒμŠ€νŠΈ 데이터)
β”‚   └── seeds/drafts/
β”‚
β”œβ”€β”€ services.test.ts            # μ„œλΉ„μŠ€ λ ˆμ΄μ–΄ ν…ŒμŠ€νŠΈ
β”œβ”€β”€ kv-deduplication.test.ts    # KV μ‹œμŠ€ν…œ ν…ŒμŠ€νŠΈ
└── README.md                   # 이 파일

πŸš€ GitHub Actions Workflow

Section titled β€œπŸš€ GitHub Actions Workflow”

1. Research Engine (research-engine.yml)

Section titled β€œ1. Research Engine (research-engine.yml)”

Trigger: Daily at 2:00 AM UTC or manual execution

Stages:

  • Phase 1-A: Liveness Check (Domain Accessibility Verification)
  • Phase 1-B: URL Discovery (Google Search, crt.sh, Domain Directory)
  • Dataset Generation (Generate Immutable Research Dataset)

Output: test/research/datasets/ , test/research/liveness/

CLI: npm run research:full -- --country=SG --category=news

2. Analysis Engine (analysis-engine.yml)

Section titled β€œ2. Analysis Engine (analysis-engine.yml)”

Trigger: Auto-exec after Research Engine completion or manual execution

Steps:

  • Load Research Registry
  • Domain analysis (KV duplicate check)
  • Generate Seed Candidates
  • Candidate validation

Output: test/analysis/country=*/

CLI: npm run analysis:full -- --country=SG --category=news

3. Contract Engine (contract-engine.yml)

Section titled β€œ3. Contract Engine (contract-engine.yml)”

Trigger: Manually triggered by Analysis Engine or manually executed

Steps:

  • Load Seed Candidates
  • Collect R2 Assets (robots.txt, sitemap.xml, homepage.html)
  • Generate Enhanced Seed V1
  • Perform Legal Compliance Check
  • Upload to R2 Storage

Output: test/results/seeds/drafts/ , Cloudflare R2

CLI: npm run contract:full -- --country=SG --category=news --fetch-r2=true

πŸ› οΈ Local Development and Testing

Section titled β€œπŸ› οΈ Local Development and Testing”

Run Individual Steps

Section titled β€œRun Individual Steps”
Terminal window
# Research 단계
npm run research:liveness -- --country=SG --category=news
npm run research:discovery -- --country=SG --category=news --max-results=50
npm run research:dataset -- --country=SG --category=news


# Analysis 단계
npm run analysis:registry -- --country=SG --category=news
npm run analysis:domains -- --country=SG --category=news
npm run analysis:candidates -- --country=SG --category=news


# Contract 단계
npm run contract:generate -- --country=SG --category=news --fetch-r2=true
npm run contract:validate -- --country=SG --category=news
npm run contract:legal -- --country=SG --category=news

Run Full Pipeline

Section titled β€œRun Full Pipeline”
Terminal window
# 전체 Research νŒŒμ΄ν”„λΌμΈ
npm run research:full -- --country=SG --category=news


# 전체 Analysis νŒŒμ΄ν”„λΌμΈ
npm run analysis:full -- --country=SG --category=news


# 전체 Contract νŒŒμ΄ν”„λΌμΈ
npm run contract:full -- --country=SG --category=news --fetch-r2=true

Test Execution

Section titled β€œTest Execution”
Terminal window
# μ„œλΉ„μŠ€ λ ˆμ΄μ–΄ ν…ŒμŠ€νŠΈ
npm test test/services.test.ts


# KV μ‹œμŠ€ν…œ ν…ŒμŠ€νŠΈ
npm test test/kv-deduplication.test.ts


# λͺ¨λ“  ν…ŒμŠ€νŠΈ
npm test test/

πŸ“Š Data Flow

Section titled β€œπŸ“Š Data Flow”

1. Research Phase

Section titled β€œ1. Research Phase”
  • Input: Country/Category Parameters
  • Processing: URL Discovery, Domain Normalization, Liveness Check
  • Output: Immutable Research Dataset (JSON)

2. Analysis Phase

Section titled β€œ2. Analysis Phase”
  • Input: Research Dataset
  • Processing: Build domain registry, KV duplicate check, Generate Seed candidates
  • Output: Seed Candidates (JSON)

3. Contract Phase

Section titled β€œ3. Contract Phase”
  • Input: Seed Candidates
  • Processing: R2 Asset Collection, Enhanced Seed V1 Generation, Legal Verification
  • Output: Seed Contracts (GitHub) + R2 Assets (Cloudflare R2)

βš™οΈ Environment Variable Setup

Section titled β€œβš™οΈ Environment Variable Setup”

Secrets used in GitHub Actions:

Terminal window
# Cloudflare KV (Analysis 단계)
CLOUDFLARE_KV_ACCOUNT_ID
CLOUDFLARE_KV_NAMESPACE_ID
CLOUDFLARE_API_TOKEN


# Cloudflare R2 (Contract 단계)
CLOUDFLARE_R2_ACCOUNT_ID
CLOUDFLARE_R2_ACCESS_KEY_ID
CLOUDFLARE_R2_SECRET_ACCESS_KEY
CLOUDFLARE_R2_BUCKET


# NewsFork μ„€μ •
NEWSFORK_DEDUP=true

πŸ—οΈ Principal Architect Design Principles

Section titled β€œπŸ—οΈ Principal Architect Design Principles”

SOLID + GitHub Actions Optimization

Section titled β€œSOLID + GitHub Actions Optimization”

1. Single Responsibility

  • Each workflow handles only one step
  • CLI scripts have clear command separation

2. Dependency Inversion

  • Service layer implemented as pure functions
  • GitHub Actions calls services only via CLI

3. Open/Closed

  • Adding new countries/categories requires only parameter changes, no code modification
  • New Research Methods can be added

4. Separation of Concerns

  • Research (discovery) ↔ Analysis (analysis) ↔ Contract (contract) are completely separated
  • Each step can be executed and tested independently

🎯 Advantages

Section titled β€œπŸŽ― Advantages”
  1. Automation: Daily automated execution for latest domain discovery
  2. Scalability: Parallel processing possible by country/category
  3. Stability: Verification and error handling at each stage
  4. Traceability: All results version-controlled on GitHub
  5. Cost Efficiency: Utilizes GitHub Actions free quota
  6. Legal Safety: Automated compliance checks

πŸ“ Notes

Section titled β€œπŸ“ Notes”

This directory is the main project’s test pipeline. Actual tests for the main project run in the .test.ts files within the src/ directory.

Main project test execution:

Terminal window
# ν”„λ‘œμ νŠΈ λ£¨νŠΈμ—μ„œ
pnpm test
Section titled β€œRelated Documentation”
Help