verint_integration

Source: ZoosGlobal/verint_integration Visibility: Private This page is automatically synchronized from the repository README. Do not edit this generated file directly.

---

Verint → Datadog Pipeline

Fetches data from Verint’s Engagement Data Hub (EDH) for three tables, computes KPIs, and pushes them as metrics to Datadog. Idempotent and restartable.

---

Quick start

# 1. Install dependencies
pip install -r requirements.txt

# 2. Copy and fill in the env template
cp config/.env.example .env
# Edit .env with your Verint credentials, Datadog API key, etc.

# 3. (Optional) Edit config/config.yaml to add org/site name mappings

# 4. Run
python main.py

# Process a single tenant
python main.py --tenant my-tenant-instance

# Process a single table
python main.py --table edm_interactions

# Force re-processing of all files (wipes idempotency state)
python main.py --reset-state

---

Configuration

Secrets (.env)

Variable	Required	Description
VERINT_EDH_URL	Yes	Base URL of the Verint EDH instance
VERINT_TENANT_ID	Yes	Azure AD tenant GUID
VERINT_CLIENT_ID	Yes	Azure AD app client ID
VERINT_CLIENT_SECRET	Yes	Azure AD app client secret
VERINT_CLIENT_SCOPE	Yes	OAuth scope (e.g. api://xxx/.default)
DATADOG_API_KEY	Yes	Datadog API key
DATADOG_APP_KEY	Yes	Datadog application key
VERINT_ACCOUNT_ID	Yes	Account identifier (used as a Datadog tag)
DATADOG_SITE	No	Datadog site (default: datadoghq.com)
ENABLE_METADATA_TAGS	No	true (default) / false — see Tagging Rules
STATE_DB_PATH	No	Path to SQLite state DB (default: ./pipeline_state.db)

Non-secret config (config/config.yaml)

• datasource_mapping: The 906003201 ↔ 3201 QM/WFM datasource ID pair. Do not change unless Verint instructs you to.
• org_site_map / site_map: Map numeric org/site IDs to human-readable names for Datadog tags. The pipeline falls back to the raw numeric ID if an entry is missing.
• pipeline: Worker count, retry settings, batch size, abandon threshold, WFM sentinel value.
• tables: Which tables to process and their metric prefixes.

---

Tagging rules

Core tags (always present on every metric)

tenant:{tenant_instance_id}
account:{account_id}
table:{table_name}
direction:{inbound|outbound}   (where applicable)

Bounded metadata tags (controlled by ENABLE_METADATA_TAGS)

org:{org_name}
site:{site_name}
queue_name:{name}
media_type:{type}
datasource:{id}

Setting ENABLE_METADATA_TAGS=false is a true kill-switch: these tags are suppressed even if a caller tries to pass them in via the build_metadata_tags function. This is enforced in kpi/base.py.

Queue tag exception

queue:{queueid} is always present on wfm_queuehistorytimeseries metrics regardless of the metadata kill-switch. The queue breakdown is the entire purpose of that table.

---

⚠️ NEVER-TAG LIST — do not add these as Datadog tags

The following fields are permanently excluded from tagging. No config switch can override this. They are either PII or high-cardinality identifiers that would explode Datadog custom metric counts and expose personal data.

Field	Reason
agent_name	PII
pbx_login_id	PII / high cardinality
employee_upn	PII
ani	PII (caller phone number)
dnis	High cardinality
caller_number	PII
switch_call_id	High cardinality
contact_id	High cardinality
sid_key	High cardinality
interaction_id	High cardinality

The enforce_tags() function in kpi/base.py strips any tag matching these keys and logs a warning. Tests in tests/test_tagging_rules.py assert this holds under all configs.

If you need per-agent or per-call lookups, build a separate record-search feature — do not solve it by tagging these fields.

---

Datasource ID mapping

qm_analytics_datasource_id = 906003201   (edm_interactions)
wfm_datasource_id          = 3201        (wfm_queuehistorytimeseries)

These IDs are a foreign-key relationship: QM analytics data and WFM data can be correlated downstream only if both IDs are consistent. Every edm_interactions record is validated against 906003201 at ingest time. Records with mismatched IDs are logged as errors (not dropped) so they can be investigated.

---

WFM sentinel masking

wfm_queuehistorytimeseries uses -2147483648 as a null sentinel on all numeric fields. The pipeline masks this to NaN before any aggregation in kpi/wfm_queue.py::_mask_sentinel(). This is unconditional — if it were ever skipped, every aggregate (AHT, ASA, etc.) would be wildly incorrect. Tests in tests/test_sentinel_masking.py cover this.

---

Project structure

verint_integration/
├── config/
│   ├── config.yaml          # Non-secret structured config
│   └── .env.example         # Secret template (copy to .env)
├── src/
│   ├── config_loader.py     # Loads all config; exposes Config dataclass
│   ├── verint_client.py     # Verint EDH API client (auth, pagination, retries)
│   ├── datadog_client.py    # Datadog metrics pusher (official SDK)
│   ├── org_site_cache.py    # Org/site ID → name resolver
│   ├── state_tracker.py     # SQLite idempotency tracker
│   └── kpi/
│       ├── base.py          # Tagging helpers, never-tag enforcement
│       ├── edm_interactions.py
│       ├── wa_call_event.py
│       └── wfm_queue.py
├── tests/
│   ├── test_sentinel_masking.py
│   ├── test_tagging_rules.py
│   └── test_kpi_transforms.py
├── main.py                  # Orchestrator
├── requirements.txt
├── README.md
└── ROADMAP.md

---

Running tests

pytest tests/ -v

Key test coverage:

• test_sentinel_masking.py — WFM -2147483648 is masked before aggregation
• test_tagging_rules.py — metadata kill-switch, never-tag PII enforcement, queue-always-present, core-always-present
• test_kpi_transforms.py — KPI value correctness for all three tables

---

Assumed field names

The Verint Postman collection has no response examples. Field names in the KPI modules are inferred from the KPI spec and common Verint EDH conventions. Check the actual schema against these names before going to production:

• edm_interactions: See top-of-file docstring in src/kpi/edm_interactions.py
• wa_call_event: See top-of-file docstring in src/kpi/wa_call_event.py
• wfm_queuehistorytimeseries: See top-of-file docstring in src/kpi/wfm_queue.py

---

Extending to new tables

See ROADMAP.md. In brief: add a new file under src/kpi/, register it in main.py::_KPI_FUNCTIONS, add a row to config.yaml::tables, and write tests following the same patterns.