Skip to content

[ALERT] SystemdUnitFailed —  #454

Open
Open
[ALERT] SystemdUnitFailed — #454
Labels
alertAuto-created alert investigationcategory:generalAlert category: generalseverity:criticalCritical severity alert

Description

@dmzoneill
dmzoneill
opened on Aug 17, 2026

Alert Details

  • Alert: SystemdUnitFailed
  • Status: FIRING
  • Severity: critical
  • Instance: unknown
  • Category: general
  • Summary:

[FIRING:5]
• alertname: SystemdUnitFailed
• instance: localhost:9100
• job: node_exporter
• name: pmcd.service
• severity: critical
• state: failed
• type: forking
• alertname: SystemdUnitFailed
• instance: localhost:9100
• job: node_exporter
• name: pmie.service
• severity: critical
• state: failed
• type: forking
• alertname: SystemdUnitFailed
• instance: localhost:9100
• job: node_exporter
• name: pmlogger.service
• severity: critical
• state: failed
• type: forking
• alertname: SystemdUnitFailed
• instance: localhost:9100
• job: node_exporter
• name: pmproxy.service
• severity: critical
• state: failed
• type: forking
• alertname: DeadMansSwitch
• severity: none
[FIRING:5] | http://feeditout:9093/#/alerts?receiver=default

Auto-created by FIO Bot at 2026-08-17 15:33 UTC

Activity

  1. dmzoneill
    added
    alertAuto-created alert investigation
    severity:criticalCritical severity alert
    category:generalAlert category: general
    on Aug 17, 2026

  2. github-actions commented on Aug 17, 2026

    @github-actions
    github-actionsBot
    on Aug 17, 2026 – with GitHub Actions
    Contributor

    Issue Analysis

    This alert indicates 5 PCP (Performance Co-Pilot) systemd services have failed on the monitored host:

    1. pmcd.service - Performance Metrics Collector Daemon
    2. pmie.service - Performance Metrics Inference Engine
    3. pmlogger.service - Performance Metrics Logger
    4. pmproxy.service - Performance Metrics Proxy

    Root Cause

    All failed services are type forking, suggesting a common startup issue. Likely causes:

    • PCP not properly installed - Missing binaries or dependencies
    • Configuration errors - Invalid /etc/pcp/ configs
    • Permission issues - Missing user accounts or directory permissions
    • Port conflicts - Default ports (44321, 44322, 44323) already in use

    Recommended Actions

    Immediate:

    # Check service status details
systemctl status pmcd pmie pmlogger pmproxy

# View recent logs
journalctl -u pmcd -u pmie -u pmlogger -u pmproxy --since "1 hour ago"

    Resolution:

    # Reinstall PCP packages
sudo dnf reinstall pcp pcp-system-tools  # RHEL/Fedora
# or
sudo apt reinstall pcp  # Debian/Ubuntu

# Reset and restart services
sudo systemctl reset-failed pmcd pmie pmlogger pmproxy
sudo systemctl restart pmcd pmie pmlogger pmproxy

    If Ansible-managed:
    Ensure your PCP role/playbook includes proper service enablement and startup validation. Add handlers to restart services after config changes.

  3. dmzoneill commented on Aug 17, 2026

    @dmzoneill
    dmzoneill
    on Aug 17, 2026
    OwnerAuthor

    Alert still firing at 2026-08-17 16:34 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  4. dmzoneill commented on Aug 17, 2026

    @dmzoneill
    dmzoneill
    on Aug 17, 2026
    OwnerAuthor

    Alert still firing at 2026-08-17 17:34 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  5. dmzoneill commented on Aug 17, 2026

    @dmzoneill
    dmzoneill
    on Aug 17, 2026
    OwnerAuthor

    Alert still firing at 2026-08-17 19:35 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  6. dmzoneill commented on Aug 17, 2026

    @dmzoneill
    dmzoneill
    on Aug 17, 2026
    OwnerAuthor

    Alert still firing at 2026-08-17 20:36 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  7. dmzoneill commented on Aug 17, 2026

    @dmzoneill
    dmzoneill
    on Aug 17, 2026
    OwnerAuthor

    Alert still firing at 2026-08-17 21:36 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  8. dmzoneill commented on Aug 17, 2026

    @dmzoneill
    dmzoneill
    on Aug 17, 2026
    OwnerAuthor

    Alert still firing at 2026-08-17 22:37 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  9. dmzoneill commented on Aug 17, 2026

    @dmzoneill
    dmzoneill
    on Aug 17, 2026
    OwnerAuthor

    Alert still firing at 2026-08-17 23:37 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  10. dmzoneill commented on Aug 18, 2026

    @dmzoneill
    dmzoneill
    on Aug 18, 2026
    OwnerAuthor

    Alert still firing at 2026-08-18 00:38 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  11. dmzoneill commented on Aug 18, 2026

    @dmzoneill
    dmzoneill
    on Aug 18, 2026
    OwnerAuthor

    Alert still firing at 2026-08-18 01:38 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  12. dmzoneill commented on Aug 18, 2026

    @dmzoneill
    dmzoneill
    on Aug 18, 2026
    OwnerAuthor

    Alert still firing at 2026-08-18 02:39 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  13. dmzoneill commented on Aug 18, 2026

    @dmzoneill
    dmzoneill
    on Aug 18, 2026
    OwnerAuthor

    Alert still firing at 2026-08-18 03:39 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  14. 826 remaining items

  15. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 04:18 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  16. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 05:18 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  17. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 06:19 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  18. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 07:19 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  19. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 08:20 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  20. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 09:20 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  21. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 10:21 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  22. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 11:21 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  23. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 12:22 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  24. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 13:22 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  25. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 14:23 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  26. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 15:04 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  27. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 15:06 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  28. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 16:06 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:

  29. dmzoneill commented on Sep 21, 2026

    @dmzoneill
    dmzoneill
    on Sep 21, 2026
    OwnerAuthor

    Alert still firing at 2026-09-21 16:07 UTC

    • Instance: unknown
    • Severity: critical
    • Summary:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    alertAuto-created alert investigationcategory:generalAlert category: generalseverity:criticalCritical severity alert

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions