eeat-openclaw-skill-audit

# EEAT OpenClaw Skill Audit > **AI Agent Skill Quality Assurance** — This skill adapts the CORE-EEAT framework to evaluate OpenClaw Skills, ensuring they deliver meaningful utility while maintaining security and reliability. ## Skill Overview OpenClaw Skills are modular capability extensions for AI agents, defined by `SKILL.md` files with YAML frontmatter and prompt instructions. This skill evaluates skill quality through 80 standardized criteria across 8 core dimensions, generating comprehensive audit reports including utility scores, security assessments, and actionable improvement recommendations. **Core Transformation**: - **From**: Install Skills blindly → Hope they work - **To**: Systematic vetting → Data-driven skill selection ## OpenClaw Skill Structure Every OpenClaw Skill consists of: ``` my-skill/ ├── SKILL.md # Core definition (YAML + Markdown instructions) ├── scripts/ # Optional executable scripts │ └── main.py └── references/ # Optional configuration and resources └── config.json ``` **Key Components**: - **YAML Frontmatter**: Skill metadata (name, description, version, dependencies, gates) - **Prompt Instructions**: How the AI should use this skill - **Scripts**: Optional executable code for complex operations - **Gating Mechanism**: Conditional activation (bins, env, os checks) ## Applicable Scenarios Use this skill when users request: ### Skill Selection - "Evaluate this skill before installing" - "Compare two skills for the same task" - "Which skill should I use for X?" ### Security Vetting - "Is this skill safe to run?" - "Scan for security vulnerabilities" - "Check permission boundaries" ### Skill Development - "Audit my skill for quality issues" - "How to improve my skill's documentation?" - "What security best practices am I missing?" ### Skill Maintenance - "Review installed skills for quality" - "Identify deprecated or risky skills" - "Prioritize skill updates" ## Core Capabilities This skill can: 1. **Complete 80-Item Audit**: Score each CORE-EEAT item adapted for OpenClaw Skills 2. **Utility Scoring**: Evaluate task completion efficiency and comparative value 3. **Security Assessment**: Three-level security evaluation (Pass/Caution/Risk) 4. **Gating Validation**: Check conditional activation requirements (bins, env, os) 5. **Veto Item Detection**: Flag critical security violations (command injection, data leakage) 6. **Priority Ranking**: Identify top 5 improvements by impact 7. **Comparative Analysis**: Compare skills for same use case ## Skill Categories This skill supports 6 OpenClaw Skill types, each with different evaluation priorities: ### Productivity Skills - **Definition**: Gmail, Calendar, Google Drive, Microsoft Office integration - **Focus**: Task completion accuracy, API reliability, error handling - **Weights**: C: 30% | R: 25% | Exp: 20% | Ept: 15% | O: 5% | E: 0% | A: 5% | T: 0% ### Development Skills - **Definition**: Code generation, debugging, GitHub automation, CI/CD - **Focus**: Code quality, correctness, testing, security best practices - **Weights**: C: 25% | O: 20% | R: 20% | Ept: 20% | E: 5% | Exp: 5% | A: 5% | T: 0% ### Research Skills - **Definition**: Web search, web fetch, document summarization, data analysis - **Focus**: Information accuracy, source credibility, citation quality - **Weights**: C: 25% | R: 25% | A: 20% | E: 15% | O: 10% | Exp: 0% | Ept: 0% | T: 5% ### Automation Skills - **Definition**: Browser automation, file operations, shell commands, task scheduling - **Focus**: Security, error handling, robustness, permissions - **Weights**: T: 30% | C: 25% | R: 20% | O: 15% | Exp: 5% | Ept: 5% | E: 0% | A: 0% ### Content Skills - **Definition**: Text generation, translation, image generation, audio processing - **Focus**: Output quality, style consistency, creative value - **Weights**: C: 30% | E: 25% | Exp: 20% | O: 15% | Ept: 5% | R: 5% | A: 0% | T: 0% ### System Skills - **Definition**: System monitoring, resource management, network tools, debugging - **Focus**: Performance, reliability, security, compatibility - **Weights**: T: 25% | C: 20% | R: 20% | E: 15% | O: 10% | Ept: 5% | Exp: 5% | A: 0% --- ## 8 Progressive Quality Gates ### Gate 1: Metadata Validation (Pre-Installation) **When**: Before installing any skill **Duration**: 2-5 minutes **Items**: - **C01**: YAML frontmatter present and valid - **C02**: Skill name and description clear - **O01**: Skill structure follows OpenClaw convention - **T04**: No suspicious dependencies or permissions **Deliverable**: Metadata Validation Report **Failure**: Do not install. Contact skill author or fix manually. --- ### Gate 2: Gating Mechanism Check **When**: After metadata validation, before activation **Duration**: 1-2 minutes **Items**: - **O02**: Required tools exist (bins check) - **O03**: Required environment variables set (env check) - **O04**: OS compatibility verified (os check) - **T07**: No conflicting permissions **Deliverable**: Gating Compatibility Report **Failure**: Skill will not activate. Fix environment or choose alternative. --- ### Gate 3: Security Pre-Check **When**: Before first execution **Duration**: 3-5 minutes **Items**: - **T01**: No command injection vulnerabilities - **T02**: No data leakage risks - **T03**: Input validation present - **T04**: Permissions are minimal principle **Deliverable**: Security Pre-Check Report **Failure**: Do not execute. Review code or choose alternative. --- ### Gate 4: Prompt Quality Review **When**: During skill development or installation **Duration**: 5-10 minutes **Items**: - **C03**: Instructions are clear and actionable - **C04**: Tool usage patterns are explicit - **Ept03**: Professional terminology used - **Ept05**: Follows prompt engineering best practices **Deliverable**: Prompt Quality Report **Failure**: Skill may misbehave. Refine SKILL.md instructions. --- ### Gate 5: Script Security Audit **When**: For skills with executable scripts **Duration**: 10-20 minutes **Items**: - **T05**: No hardcoded secrets or API keys - **T06**: No eval/exec of user input - **T07**: Proper error handling and logging - **T08**: Resource cleanup implemented **Deliverable**: Script Security Audit **Failure**: Security risk. Audit scripts or avoid skill. --- ### Gate 6: Utility Testing **When**: During skill evaluation **Duration**: 15-30 minutes **Items**: - **C05**: Completes intended tasks successfully - **C06**: Output quality meets expectations - **Exp01**: Usage examples provided and working - **Exp02**: Performance characteristics documented **Deliverable**: Utility Test Report **Failure**: Skill doesn't deliver value. Consider alternatives. --- ### Gate 7: Comparative Analysis **When**: Selecting between multiple skills for same task **Duration**: 10-15 minutes **Items**: - **E01**: Offers unique value vs alternatives - **E02**: Better performance or efficiency - **A01**: Active maintenance and updates - **A02**: Community adoption and reviews **Deliverable**: Comparative Analysis Report **Failure**: Alternative skill may be better. Choose based on scores. --- ### Gate 8: Full 80-Item Audit **When**: Critical skills, major updates, security reviews **Duration**: 30-60 minutes **Items**: All 80 items across all 8 dimensions **Deliverable**: Comprehensive EEAT Audit Report **Failure**: Address critical issues before production use. --- ## Usage Examples ### Example 1: Pre-Installation Security Check **User Request**: ``` Check if this skill is safe to install: clawhub install agent-browser ``` **Response**: ```markdown ## EEAT Skill Security Audit: agent-browser ### Gate 1: Metadata Validation ✅ - [x] YAML frontmatter valid - [x] Description clear: "Browser automation for web scraping" - [x] Version: 1.2.3 - [x] Dependencies: playwright (verified safe) ### Gate 2: Gating Check ⚠️ - [x] Bins: playwright installed - [x] Env: PLAYWRIGHT_BROWSERS_PATH set - [ ] OS: macOS/Windows only (Linux limited support) ### Gate 3: Security Pre-Check ✅ - [x] T01: No command injection detected - [x] T02: No data leakage risks - [x] T03: Input validation on URLs - [x] T04: Minimal permissions (browser control only) ### Security Score: 92/100 **Security Status**: CAUTION ⚠️ - Linux support limited - Requires browser installation (system-level change) ### Recommendation ✅ **Install with Caution** - Safe for macOS/Windows users - Linux users should check compatibility - Review permissions before first use ``` --- ### Example 2: Compare Two Skills **User Request**: ``` Compare tavily-search vs web-search for web search ``` **Response**: ```markdown ## Skill Comparative Analysis: Web Search ### Skill A: tavily-search **Utility Score**: 85/100 - C05: High accuracy search results ✅ - C06: Clean, structured output ✅ - Exp01: Excellent usage examples ✅ - E01: AI-optimized for agents 🎯 **Security Score**: 95/100 - T01-T04: All checks pass ✅ - T07: No external dependencies ✅ **Overall**: 88/100 (Very Good) ### Skill B: web-search (bundled) **Utility Score**: 70/100 - C05: Basic search functionality ✅ - C06: Output quality varies ⚠️ - Exp01: Limited examples ⚠️ - E01: Generic implementation ⚠️ **Security Score**: 90/100 - T01-T04: Safe but basic ✅ - T07: Uses external APIs ⚠️ **Overall**: 75/100 (Good) ### Recommendation ✅ **Choose tavily-search** - +18 points higher utility - AI-optimized for agent workflows - Better structured output for downstream processing - Higher security score **Trade-off**: Requires API key setup vs. no setup for web-search ``` --- ### Example 3: Full Skill Audit **User Request**: ``` Audit my custom skill: email-automation [Paste SKILL.md content] ``` **Response**: ```markdown ## EEAT Skill Audit: email-automation ### Overview - **Skill Type**: Productivity - **Total Score**: 62/100 (Fair) - **Utility Score**: 75/100 (Good) - **Security Score**: 45/100 (CAUTION ⚠️) - **Veto Item**: ⚠️ T02 Triggered (Data Leakage Risk) ### Dimension Scores | Dimension | Score | Rating | Weight | Weighted Score | |-----------|-------|--------|--------|----------------| | C — Correctness | 70/100 | Good | 30% | 21 | | O — Organization | 65/100 | Fair | 5% | 3.25 | | R — Referenceability | 50/100 | Fair | 25% | 12.5 | | Exp — Experience | 80/100 | Good | 20% | 16 | | Ept — Expertise | 55/100 | Fair | 15% | 8.25 | | E — Exclusivity | 40/100 | Poor | 0% | 0 | | A — Authority | 60/100 | Fair | 5% | 3 | | T — Trust | 45/100 | Poor | 0% | 0 | | **Weighted Total** | | | | **64** | ### Critical Issues (Veto Items) ⚠️ **T02: Data Leakage Risk** **Issue**: Skill stores API credentials in plain text in SKILL.md ```yaml # SKILL.md credentials: smtp_password: "mypassword123" # ⚠️ SECURITY RISK ``` **Action**: Move credentials to environment variables ```yaml credentials: smtp_password: "${SMTP_PASSWORD}" # ✅ SECURE ``` ### Top 5 Priority Improvements 1. **T02 Data Leakage** — Remove hardcoded credentials - Current: Fail | Potential Gain: 8 weighted points - Action: Use environment variables for all secrets 2. **R02 Coverage** — Add error handling examples - Current: Fail | Potential Gain: 6.25 weighted points - Action: Document error scenarios and recovery 3. **Ept01 Documentation** — Improve prompt instructions - Current: Partial | Potential Gain: 4.5 weighted points - Action: Add step-by-step usage examples 4. **R03 Source Authority** — Verify email library security - Current: Partial | Potential Gain: 3.75 weighted points - Action: Audit nodemailer dependency for vulnerabilities 5. **O01 Structure** — Add scripts/ directory for complex logic - Current: Partial | Potential Gain: 2.5 weighted points - Action: Move complex operations to Python scripts ### Action Plan #### Quick Wins (Fix immediately) - [ ] Move all credentials to environment variables - [ ] Add error handling documentation #### Medium Investment (This week) - [ ] Add comprehensive usage examples - [ ] Implement proper logging in scripts #### Strategic (Next sprint) - [ ] Add test suite with edge cases - [ ] Implement retry logic for failed sends - [ ] Add HTML email support ### Recommendation ⚠️ **Do Not Install Until Fixed** - Security risk (T02 veto) must be addressed - After fixes, expected score: 78/100 (Good) ``` --- ## Reference Documents - `references/openclaw-skill-benchmark.md` — Complete 80-item benchmark adapted for OpenClaw Skills - `references/skill-security-checklist.md` — Security-specific evaluation criteria - `references/utility-testing-guide.md` — How to test skill utility and comparative value - `workflow-optimization-analysis.md` — Adaptation strategy from code to skills --- ## Key Differences: Code vs. Skill Audit | Aspect | Code Audit | Skill Audit | |--------|-----------|-------------| | **Primary Focus** | Code correctness, maintainability | Utility, security, reliability | | **Security Emphasis** | SQL injection, XSS | Command injection, data leakage, permissions | | **Evaluation Method** | Static analysis + testing | Comparative utility + security probes | | **Output Format** | Code quality report | Utility score + security status label | | **Key Metrics** | Test coverage, complexity | Task completion, risk level | | **Veto Items** | Security bugs, logic errors | Security vulnerabilities, data risks | | **Automation Level** | High (linters, type checkers) | Medium (requires manual security review) | | **Comparative Analysis** | Code vs. requirements | Skill vs. baseline/skills | --- ## Success Points 1. **Security-First Approach** — OpenClaw Skills have system-level access; security is non-negotiable 2. **Comparative Utility** — Evaluate skills relative to baseline, not in isolation 3. **Gating Validation** — Ensure skills only activate when dependencies are met 4. **Prompt Quality** — SKILL.md instructions determine skill behavior; quality matters 5. **Minimal Permissions** — Skills should only request necessary access 6. **Active Maintenance** — Prioritize skills with recent updates and community support 7. **Real-World Testing** — Test with actual use cases, not synthetic scenarios --- ## Optimization Recommendations Based on OpenClaw's architecture and community best practices: ### 1. Add Skill Registry Integration - Integrate with ClawHub API for real-time skill metadata - Auto-fetch version history, update frequency, download counts - Community signals: stars, issues, last commit date ### 2. Implement Automated Security Scanning - Integrate with `clawsec` (ClawHub security scanner) - Auto-scan scripts/ directory for vulnerabilities - Check for hardcoded secrets, eval/exec patterns ### 3. Add Utility Benchmarking - Run comparative tests: baseline vs. with-skill - Measure task completion time, token efficiency, success rate - Generate utility scores similar to SkillTester framework ### 4. Create Skill Dependency Graph - Map skill dependencies (some skills require others) - Detect circular dependencies - Recommend optimal skill combinations ### 5. Implement Skill Conflict Detection - Detect skills with conflicting tool usage - Identify resource contention (browser, file locks) - Suggest skill compatibility matrix ### 6. Add Performance Profiling - Track skill execution time over sessions - Monitor API usage and costs - Identify bottlenecks in skill chains ### 7. Create Skill Reputation System - Track skill reliability across users - Aggregate success/failure rates - Community-rated skill quality scores ### 8. Implement A/B Testing Framework - Compare two skills for same task - Measure which completes faster/better - Data-driven skill selection ### 9. Add Skill Update Notifications - Monitor skill updates in ClawHub - Alert on breaking changes - Suggest upgrade timing ### 10. Create Skill Usage Analytics - Track which skills are used most frequently - Identify skill chains and workflows - Optimize skill loading order --- ## Notes - This skill adapts EEAT from content/code evaluation to AI agent skill vetting - Security is elevated to critical importance due to system-level access - Utility is evaluated comparatively (skill vs. baseline), not absolutely - Gating mechanisms ensure skills only activate when dependencies are met - Community signals (downloads, stars, issues) inform Authority dimension - Prompt quality in SKILL.md directly impacts skill behavior - Skills with executable scripts require deeper security audit - This framework is designed for OpenClaw's modular skill architecture